Azure Data Engineer - Data Factory/Synapse Analytics (6-8 yrs)

Key Responsibilities :

- Design and Implement Data Solutions : Build, maintain, and optimize data pipelines, data lakes, and data warehouses using Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure SQL Database, and Azure Synapse Analytics.

- ETL/ELT Development : Develop efficient ETL/ELT pipelines to extract, transform, and load data from various sources into structured and unstructured formats in Azure.

- Big Data Processing : Work with Azure Databricks, Azure HDInsight, and Apache Spark for large-scale data processing and analytics.

- Data Integration : Integrate disparate data sources from on-premises, hybrid, and cloud-based systems, ensuring smooth data flow.

- Database Management : Manage data storage solutions, optimize database performance, and ensure data security and compliance.

- Real-Time Data Processing : Set up and manage Azure Stream Analytics, Azure Event Hub, or similar tools for real-time data processing and event-driven architecture.

- Collaboration with Data Scientists and Analysts : Collaborate with data scientists and business analysts to ensure data pipelines support business intelligence, reporting, and machine learning needs.

- Data Governance & Security : Ensure that data governance policies and security protocols are followed, including encryption, access control, and compliance with regulatory standards (e.g., GDPR, HIPAA).

- Performance Monitoring & Troubleshooting : Monitor data pipeline performance, identify bottlenecks, troubleshoot issues, and implement improvements.

- Documentation : Document data architectures, pipelines, processes, and best practices for future reference.

Required Skills & Qualifications :

- Bachelor's Degree in Computer Science, Engineering, Information Technology, or a related field.

- Proven experience (X+ years) in Azure cloud technologies, including Azure Data Factory, Azure SQL Database, Azure Data Lake, Azure Databricks, Azure Synapse Analytics, and related services.

- Strong experience in ETL/ELT processes, including building and optimizing data pipelines.

- Proficiency in SQL and T-SQL for querying and managing relational databases.

- Experience with big data technologies such as Apache Spark, Azure HDInsight, Hadoop, and NoSQL databases (e.g., Cosmos DB).

- Hands-on experience with Power BI or other Business Intelligence (BI) tools for reporting and visualization.

- Familiarity with Azure DevOps, CI/CD, and Infrastructure as Code (e.g., Terraform).

- Experience with Python, Scala, or Java for data processing.

- Understanding of data governance, data security, and compliance best practices.

- Knowledge of containerization (Docker) and Kubernetes is a plus.

- Strong problem-solving skills and the ability to troubleshoot complex data-related issues.

- Excellent communication and collaboration skills.

Preferred Qualifications :

- Microsoft certifications, such as Microsoft Certified : Azure Data Engineer Associate or Microsoft Certified : Azure Solutions Architect Expert.

- Familiarity with machine learning workflows in Azure (e.g., Azure Machine Learning).

- Experience with real-time analytics using Azure Stream Analytics or similar platforms.

- Understanding of serverless computing in Azure.