Data Engineer - Synapse Analytics (4-5 yrs)

Job Description :

We are seeking a skilled and experienced Azure Data Engineer to join our team on a contract basis. The ideal candidate will have 4+ years of relevant experience with a deep understanding of Azure data tools and expertise in designing and implementing scalable data solutions.

Key Responsibilities :

- Design, build, and maintain scalable data pipelines and workflows using Azure Data Factory.

- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for organizing and transforming data.

- Develop and optimize SQL data warehousing solutions, including schema design, indexing, and query optimization.

- Set up and manage incremental refreshes for large-scale transactional data tables.

- Collaborate with stakeholders to understand data requirements and provide effective solutions.

- Ensure data quality, consistency, and accuracy throughout the data lifecycle.

- Monitor and troubleshoot data pipelines, addressing performance and reliability issues.

- Document data workflows, pipelines, and architecture for knowledge sharing and maintenance.

Required Skills and Experience :

Azure Expertise :

- Proficient in Azure Data Factory for orchestrating ETL/ELT workflows.

- Experience with Azure Data Lake Storage, Azure SQL Database, and Synapse Analytics.

Medallion Architecture :

- Hands-on experience implementing Medallion (Bronze, Silver, Gold) architecture.

- Strong understanding of data transformation and enrichment processes at each layer.

SQL Data Warehousing :

- Proficient in designing and managing large-scale SQL data warehouses.

- Expertise in writing complex queries, stored procedures, and optimizing query performance.

Incremental Data Processing :

- Demonstrated experience implementing incremental data refreshes for large datasets.

- Familiarity with handling change data capture (CDC) and delta updates.

Data Engineering :

- Strong understanding of ETL/ELT concepts and best practices.

- Experience with data modeling and working with large transactional data tables.

Tools and Frameworks :

- Familiarity with Databricks, PySpark, and other data processing tools is a plus.

Qualifications :

- Bachelor's degree in Computer Science, Data Engineering, or a related field.

- 4+ years of hands-on experience as a Data Engineer, with a focus on Azure data solutions.

- Strong problem-solving skills and the ability to work independently in a fast-paced environment.

- Excellent communication and collaboration skills.