Azure Data Engineer - SQL/Python (5-13 yrs)

Job Description :

Main Roles and Responsibilities :

- Design, develop, and implement ETL pipelines for the continuous deployment and integration.

- Collaborate with other data engineers and data scientists to understand data requirements and optimize data solution processes.

- Assure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements

- The ability to automate tasks and deploy production standard code (with unit testing, continuous integration, versioning etc.)

- Load transformed data into storage and reporting structures in destinations including data warehouse, high speed indexes, real-time reporting systems and analytics applications

- Build data pipelines to collectively bring data together

- Other responsibilities include extracting data, troubleshooting and maintaining the data warehouse

- Implement best practices for pipeline building and governance.

- Troubleshoot and resolve issues related to pipeline deployment and performance.

- Ensure compliance with security and data privacy standards

- Solution design using Microsoft Azure services and other tools

Required skills and experience :

- 2-3 years of experience in Azure Data Analytics, Synapse, DevOps.

- Proficiency in programming languages such as Python, SQL, Py-Spark, Spark-SQL.

- Experience with Azure: ADLS, Synapse, Stream Analytics, SQL DW, Analysis Services, Serverless Architecture, ARM Templates

- Experience with cloud platforms like Azure, AWS or and their respective services.

- Knowledge of CI/CD pipelines, automation tools and version control systems like Git.

- Strong problem-solving skills and ability to troubleshoot complex issues.

- Experience with monitoring tools.

- Ability to work collaboratively in cross-functional teams.

Nice to have:

- Should have Data Engineer certification from any discipline - Azure, AWS.

- Azure Synapse Studio, Data bricks, Fabric

- Familiarity with agile software development lifecycle (SCRUM, Kanban, etc.)