Technology Lead - Data Engineering

As a Lead Data Engineer, you will be responsible for designing, developing, and maintaining complex data pipelines, ensuring smooth data flow and quality across various systems. This role focuses heavily on Azure services like Azure Data Factory, Azure Databricks, and SQL Database. You will work closely with cross-functional teams to implement big data solutions, ensure compliance with data security standards, and continuously optimize data infrastructure for better performance and scalability.

In this position, you’ll monitor workflows, troubleshoot issues, and drive innovations to improve system functionality. Your experience with big data processing frameworks like Apache Spark and advanced proficiency in SQL will be essential for success. An Azure Data Engineer certification is mandatory, and experience in DevOps and CI/CD practices is highly valued.

• Design, develop, and maintain robust data pipelines using Azure Data Factory and Databricks workflows.
• Implement and manage big data solutions using Azure Databricks.
• Design and maintain relational databases using Azure Delta Lake.
• Ensure data quality and integrity through rigorous testing and validation processes.
• Monitor and troubleshoot data pipelines to ensure seamless operations.
• Implement data security measures in line with industry standards.
• Collaborate with cross-functional teams to continuously improve data infrastructure for scalability and performance.
• Maintain and enhance CI/CD pipelines in the data engineering space.
• Stay updated with the latest trends and best practices in data engineering and cloud technologies.
• Good in understanding business requirements and design to build
• Excellent communication skills with ability to directly interact with business stakeholders
• Ability to manage a small team and handle conflicting priorities
• Escalation management