Eminds.ai - Lead Data Engineer - ETL (7-11 yrs)

Job Title : Data Engineer Lead - Expertise in either (Azure, AWS, Google Cloud)

Job Location : Remote

Experience : 7-10 years of experience

Qualifications : Bachelor's in engineering (preferably computer science)

Job Summary :

We are seeking a highly skilled and experienced Data Engineer Lead to oversee the design, development, and implementation of data solutions across- Azure, AWS, and Google Cloud platforms. The ideal candidate will lead a team of data engineers, ensuring the creation of scalable, high-performance data pipelines and analytics solutions to meet business needs.

Key Responsibilities :

Leadership & Strategy :

- Lead a team of data engineers to deliver high-quality data solutions across multiple cloud environments.

- Define best practices and standards for multi-cloud- data engineering.

- Collaborate with cross-functional teams, including data scientists, analysts, and product managers, to define comprehensive data strategies.

Solution Design & Development :

- Architect and implement data pipelines using Azure Data Factory, AWS Glue, and Google Dataflow.

- Design and manage data lake architectures on Azure Data Lake Storage (ADLS), Amazon S3, and Google Cloud Storage.

- Implement data modeling, data integration, and ETL/ELT pipelines using tools like Azure Synapse, Redshift, and Big Query.

Operational Excellence :

- Optimize workflows to ensure performance, scalability, and cost-efficiency across cloud platforms.

- Implement CI/CD pipelines for data engineering workflows using Azure DevOps, AWS Code Pipeline, and Google Cloud Build.

- Monitor and troubleshoot pipelines to address performance issues and ensure resilience.

Governance & Security :

- Ensure compliance with data governance, security policies, and regulations across cloud providers.

- Use tools like Azure Purview, AWS Glue Data Catalog, and Google Data Catalog for metadata management.

- Implement best practices for securing data using role-based access control (RBAC), encryption, and managed identities.

Communication :

- Stakeholder Collaboration : Translate business needs into technical solutions and present updates clearly to non-technical audiences.

- Team Leadership: Mentor team members, foster collaboration, and provide constructive feedback.

- Cross-Functional Coordination : Work with product managers, data scientists, and engineers to align data strategies.

- Documentation : Maintain clear documentation for workflows and project decisions.

- Conflict Resolution : Resolve team and stakeholder conflicts effectively with strong interpersonal skills.

- Communication Tools : Use collaboration platforms (e.g., Slack, Jira) to streamline communication.

- Thought Leadership : Advocate for data engineering initiatives and share insights on industry trends.

- Adaptability : Tailor communication style to suit technical and non-technical audiences.

Required Skills and Qualifications :

- Multi-Cloud Expertise (Optional - At least one cloud with proven skill)

- Proficiency in Azure (Data Factory, Databricks, Synapse Analytics), AWS (Glue, Redshift, S3), and Google Cloud (Dataflow, Big Query, Cloud Storage).

- Knowledge of streaming services like Azure Event Hubs, AWS Kinesis, and Google Pub/Sub.

Programming and Scripting Languages :

- Strong programming skills in Python, SQL, and optionally Scala for big data processing.

Big Data Frameworks and Tools :

- Expertise in Apache Spark, Scala, Java and other multi-cloud implementations.

- Skilled in distributed computing frameworks for large-scale data processing.

Data Modeling and ETL/ELT Design :

- Expertise in designing efficient data models for data lakes and warehouses.

- Hands-on experience creating scalable ETL/ELT pipelines.

CI/CD and DevOps :

- Hands-on experience with multi-cloud- CI/CD tools (Azure DevOps, AWS CodePipeline, Google Cloud Build).

- Knowledge of Infrastructure as Code (IaC) tools like Terraform, Biceps.

Data Governance and Security :

- Experience with Azure Purview, AWS Glue Data Catalog, and Google Data Catalog.

- Knowledge of encryption and key management solutions like Azure Key Vault, AWS KMS, and Google Cloud KMS.

Additional Knowledge :

- Familiarity with visualization tools like Power BI, Tableau, or Looker.

- Understanding of real-time data streaming with Kafka or equivalent.

Certifications (Preferred) :

- AWS Certified Data Analytics - Specialty

- Microsoft Certified: Azure Data Engineer Associate

- Google Professional Data Engineer

- MS Fabric DP-600