Azure Data Engineer - ETL/Python (6-9 yrs)

Work Hours : team working hours like 3 PM - 12 AM IST

Rotational shift (6-3am IST) one weekend work either Sat/Sun

Experience Level : 6 Years to 9 Years

Location; Indore/Hyderabad

Key Responsibilities :

1. Develop and optimize T-SQL queries, stored procedures, and functions to support data extraction, transformation, and loading (ETL) processes.

2. Design, implement, and maintain data pipelines using Databricks, ensuring efficient processing and analysis of large volumes of data.

3. Automate data-related tasks and processes using Python or Java programming languages, enhancing efficiency and scalability.

4. Build and maintain RESTful APIs for data access and integration, ensuring security, scalability, and performance.

5. Monitor data pipelines and integrations, proactively identifying and resolving issues to minimize downtime and ensure data integrity.

6. Implement data governance and quality control measures to ensure the accuracy, completeness, and consistency of data.

7. Document data models, processes, and procedures, ensuring knowledge transfer and compliance with best practices.

8. Stay up-to-date with emerging technologies and industry trends, evaluating new tools and methodologies to improve data engineering practices.

Required Skills :

- Proficiency in TSQL query writing in SQL Server.

- Develop robust and scalable data pipelines and workflows on azure with technologies like Data bricks, Spark, and Python.

- Knowledge of data modeling and database design principles.

- Proven experience working with SQL Server Integration Services (SSIS) and SQL Server Management Studio (SSMS).

- Experience in third party API integration.

- Experience with Databricks.

- Experience in optimizing and tuning ETL processes and SQL queries for performance.

- Develop and maintain Python scripts for data processing, transformation, and automation.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Strong understanding of data warehousing concepts, ETL best practices, and database administration.

- Understanding of data governance, quality, and security practices.

Nice-to-Have Skills :

- Experience with GCP data services such as Apache Airflow, BigQuery, Dataflow, and Data prep.

- Understanding of database security and compliance tools.

- Experience in DataStage

- Azure or GCP cloud data engineering certification

Qualifications :

- Bachelor's degree in computer science, engineering, or related technical field.

- Four or more years of experience in data engineering, data system development, or related roles.