Lead Data Engineer - R/Python (8-12 yrs)

Primary Skills : Strong Experience in Erwin, Azure Synapse, Azure Databricks, Azure DevOps, SQL, Power BI, Spark, Python, R for (8 to 12 years)

Job Description :

Lead Data Engineer utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies to handle a distribution of data types and volumes in support of data architecture design.

A Senior Data Engineer designs and oversees the entire data infrastructure, data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.

Key Responsibilities :

- Design and overseeing the entire data architecture strategy.

- Mentor junior data architects to ensure skill development in alignment with the team strategy.

- Design and implement complex scalable, high-performance data architectures that meet business requirements.

- Model data for optimal reuse, interoperability, security and accessibility.

- Develop and maintain data flow diagrams, and data dictionaries.

- Collaborate with stakeholders to understand data needs and translate them into technical solutions.

- Ensure data accessibility through a performant, cost-effective consumption layer that supports use by citizen developers, data scientists, AI, and application integration.

- Ensure data quality, integrity, and security across all data systems.

Qualification :

- Experience in Erwin, Azure Synapse, Azure Databricks, Azure DevOps, SQL, Power BI, Spark, Python, R.

- Ability to drive business results by building optimal cost data landscapes.

- Familiarity with Azure AI/ML Services, Azure Analytics: Event Hub, Azure Stream Analytics, Scripting Ansible

- Experience with machine learning and advanced analytics.

- Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).

- Understanding of CI/CD pipelines and automated testing frameworks.

- Certifications such as AWS Certified Solutions Architect , IBM certified data architect or similar are a plus.