Azure Data Engineer - Data Factory/Databricks (7-10 yrs)

Experience : 7-10 Years

Location : Chennai, Coimbatore, Hyderabad, Kochi, Pune, Mumbai

Joining : Immediate Joiners Preferred

About the Role :

We are seeking a highly motivated and experienced Azure Data Engineer to join our growing data team. You will play a crucial role in designing, building, and maintaining robust and scalable data pipelines on the Azure cloud platform.

You will work closely with data scientists, analysts, and other engineers to deliver high-quality data solutions that drive business insights and decision-making.

Responsibilities :

- Design, develop, and maintain data pipelines using Azure Data Factory, Databricks, and other Azure data services.

- Implement data ingestion, transformation, and loading (ETL) processes for batch, streaming, and near real-time data.

- Develop and optimize complex SQL queries for data extraction, manipulation, and analysis.

- Utilize Python (preferred) for data manipulation, automation, and scripting.

- Ensure data quality and integrity through data profiling, validation, and testing.

- Implement and maintain data governance and security best practices.

- Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.

- Participate in all phases of the software development lifecycle (SDLC) using Agile methodologies.

- Utilize Git for version control and collaborative development.

- Automate data validation and testing processes using SQL and Python.

- Document technical designs, implementations, and operational procedures.

- Troubleshoot and resolve data pipeline issues and performance bottlenecks.

- Stay up-to-date with the latest Azure data services and technologies.

- Contribute to the continuous improvement of our data engineering practices.

Required Skills & Experience :

- Azure Expertise : Extensive experience with Azure Data Factory, Azure Databricks, and building data pipelines (batch, streaming, and near real-time). Deep understanding of Azure data services such as Azure Blob Storage, Azure Data Lake Storage Gen2, Azure SQL Database, Azure Synapse Analytics (preferred), etc.

- SQL Proficiency : Strong SQL skills, including writing complex queries, performance tuning, and data profiling. Ability to optimize SQL queries for performance and scalability.

- Python (Preferred) : Proficiency in Python for data manipulation, automation, and scripting. Experience with data science libraries like Pandas and NumPy is a plus.

- SDLC & Agile : Experience working in an Agile environment and participating in all phases of the software development lifecycle.

- Version Control : Proficiency with Git for code management and collaboration.

- Automation : Experience automating data validation and testing processes using SQL and Python.

- Communication & Documentation : Excellent communication and documentation skills.

- Ability to clearly articulate technical concepts to both technical and non-technical audiences.

- Problem-Solving Skills : Strong analytical and problem-solving skills.

- Ability to identify and resolve data pipeline issues and performance bottlenecks.

- Experience with data warehousing concepts and dimensional modeling is a plus.

- Experience with CI/CD pipelines is a plus.

- Experience with cloud security best practices is a plus.

Qualifications :

- Bachelor's degree in Computer Science, or a related field