Data Engineer Lead(Azure, Python or Java)

We are seeking an experienced Data Engineer Lead to lead a team in designing, developing, and implementing high-performance data solutions across Azure, AWS, and Google Cloud platforms. The ideal candidate will have a strong technical background, leadership skills, and experience in building scalable data pipelines and analytics solutions to meet business requirements.

Key Responsibilities:

Leadership & Strategy:

• Lead a team of data engineers to deliver robust, high-quality data solutions across multi-cloud environments.
• Define best practices, processes, and standards for data engineering in Azure, AWS, and Google Cloud.
• Collaborate with cross-functional teams (data scientists, analysts, product managers) to define and implement comprehensive data strategies.

Solution Design & Development:

• Architect and implement data pipelines using Azure Data Factory, AWS Glue, and Google Dataflow.
• Design and manage data lake architectures on Azure Data Lake Storage (ADLS), Amazon S3, and Google Cloud Storage.
• Develop data integration, data modeling, and ETL/ELT pipelines using tools like Azure Synapse, Redshift, and BigQuery.

Operational Excellence:

• Optimize workflows for performance, scalability, and cost-efficiency across cloud platforms.
• Implement CI/CD pipelines for data engineering workflows using Azure DevOps, AWS Code Pipeline, and Google Cloud Build.
• Monitor, troubleshoot, and optimize pipelines to ensure resilience and address performance issues.

Governance & Security:

• Ensure compliance with data governance, security policies, and regulations across cloud platforms.
• Utilize tools like Azure Purview, AWS Glue Data Catalog, and Google Data Catalog for metadata management.
• Implement role-based access control (RBAC), encryption, and managed identities to secure data.

Communication:

• Stakeholder Collaboration: Translate business requirements into technical solutions and clearly communicate project updates to both technical and non-technical stakeholders.
• Team Leadership: Mentor and guide team members, foster collaboration, and provide constructive feedback.
• Cross-Functional Coordination: Align data strategies with product managers, data scientists, and engineers.
• Documentation: Maintain comprehensive documentation for data workflows, project decisions, and architecture.
• Conflict Resolution: Address team and stakeholder conflicts effectively, leveraging strong interpersonal skills.
• Communication Tools: Use collaboration platforms (Slack, Jira) to streamline communication.
• Thought Leadership: Advocate for innovative data engineering initiatives and share insights on industry trends.
• Adaptability: Adjust communication style to effectively engage with both technical and non-technical audiences.

Required Skills and Qualifications:

• Multi-Cloud Expertise: Proficiency in at least one cloud platform (Azure, AWS, Google Cloud) with a proven track record of using Azure Data Factory, AWS Glue, Google Dataflow, Azure Synapse, BigQuery, and related tools.
• Streaming Services: Experience with services like Azure Event Hubs, AWS Kinesis, and Google Pub/Sub for real-time data streaming.
• Programming and Scripting: Strong programming skills in Python, SQL, and optionally Scala for big data processing.
• Big Data Frameworks: Expertise in Apache Spark, Scala, Java, and distributed computing frameworks for large-scale data processing.
• Data Modeling & ETL/ELT: Expertise in designing efficient data models for data lakes and warehouses. Hands-on experience building scalable ETL/ELT pipelines.
• CI/CD & DevOps: Experience with multi-cloud CI/CD tools like Azure DevOps, AWS Code Pipeline, and Google Cloud Build. Familiarity with Infrastructure as Code (IaC) tools like Terraform or Biceps.
• Data Governance & Security: Knowledge of Azure Purview, AWS Glue Data Catalog, Google Data Catalog, and encryption/key management solutions like Azure Key Vault, AWS KMS, and Google Cloud KMS.
• Visualization Tools: Familiarity with tools like Power BI, Tableau, or Looker.
• Real-Time Data Streaming: Understanding of real-time data streaming platforms like Kafka.

Preferred Certifications:

• AWS Certified Data Analytics Specialty
• Microsoft Certified: Azure Data Engineer Associate
• Google Professional Data Engineer
• Microsoft Certification: DP-600 Microsoft Fabric