Azure Data Engineer - Python/ETL (5-15 yrs)

Position Overview :

We are seeking a highly skilled Azure Data Engineer to join our team. The ideal candidate will have experience in designing, implementing, and maintaining data pipelines and solutions using Microsoft Azure's ecosystem. The role requires expertise in Azure Data Factory, Azure Synapse Analytics, and other data engineering tools to process, analyze, and store large-scale datasets effectively.

Key Responsibilities :

Data Pipeline Development :

- Design, build, and maintain scalable and reliable data pipelines using Azure Data Factory, Databricks, and Synapse Analytics.

- Optimize data flows and integration between on-premise and cloud environments.

Data Modeling & Management :

- Develop and implement data models to support business requirements.

- Ensure data integrity, quality, and performance in Azure SQL Database, Cosmos DB, or Data Lake Storage.

Data Integration :

- Integrate structured and unstructured data from multiple sources using Azure tools.

- Implement ETL/ELT processes and ensure efficient data movement and transformation.

Analytics Support :

- Enable advanced analytics by implementing solutions with Azure Synapse, Power BI, and other analytical tools.

- Collaborate with data analysts and scientists to support business insights.

Performance Optimization :

- Monitor and optimize data solutions for performance and cost-efficiency.

- Troubleshoot and resolve issues with pipelines, storage, and related systems.

Security & Compliance :

- Implement best practices for data governance and security, including RBAC and encryption.

- Ensure compliance with industry standards such as GDPR, HIPAA, or SOC2.

Required Skills and Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or a related field.

- 3+ years of experience in Azure data services and related tools.

- Proficiency in Azure Data Factory, Databricks, Synapse Analytics, and Azure Data Lake.

- Strong understanding of SQL, Python, and/or Scala for data manipulation.

- Experience with big data technologies such as Hadoop or Spark.

- Knowledge of CI/CD practices and tools like Azure DevOps.

- Familiarity with data visualization tools like Power BI.

- Strong problem-solving and analytical skills.

Preferred Skills :

- Certifications in Azure Data Engineering (e.g., DP-203).

- Experience with real-time streaming using Azure Event Hub or Kafka.

- Familiarity with machine learning tools and frameworks.

Soft Skills :

- Strong communication and collaboration skills.

- Ability to work in a fast-paced and dynamic environment.

- Attention to detail and commitment to quality.