Azure Data Engineer

Job Title: Azure Data Engineer

Job Description:

We are seeking a highly skilled and experienced Azure Data Engineer to join our team. The ideal candidate will have a strong background in cloud-based data integration, data transformation, and analytics solutions, with a focus on Azure services. This role involves designing, implementing, and maintaining robust data pipelines and analytics frameworks to support business intelligence and advanced analytics initiatives.

Key Responsibilities:

1. Data Engineering and Development:
• Design and implement scalable and efficient ETL/ELT processes using Azure Data Factory and Azure Databricks (PySpark).
• Manage and optimize data storage and access in Azure Data Lake Storage.
2. Data Architecture:
• Develop and maintain a Lakehouse architecture for data analytics and reporting.
• Ensure best practices in data organization, partitioning, and access control in Azure Data Lake.
• Collaborate on the design and implementation of Microsoft Fabric architecture to enhance data accessibility and governance.
3. Data Analysis and Reporting:
• Create and optimize reports and dashboards using Power BI to enable data-driven decision-making.
• Develop T-SQL scripts and stored procedures for data transformation and analysis.
4. Cloud Infrastructure and DevOps:
• Leverage knowledge of Azure IaaS services for data platform setup and maintenance.
• Implement and manage CI/CD pipelines using Azure DevOps to automate data pipeline deployments.
5. Technical Expertise and Collaboration:
• Provide strong technical guidance in Spark architecture and ensure best practices for PySpark code development.
• Collaborate with cross-functional teams to align on data strategies and solutions.
• Troubleshoot and resolve complex data engineering challenges efficiently.

Mandatory Skills:

• Proficiency in Azure Data Factory, Azure Data Lake Storage, and Azure Databricks (PySpark).
• Strong understanding of ETL and ELT principles.
• Deep knowledge of Lakehouse architecture and its implementation.
• Expertise in PySpark and Spark architecture.
• Solid understanding of Azure Data Lake architecture and access control mechanisms
• Strong command of T-SQL for advanced querying and data manipulation..

Good to Have:

• Experience with Power BI for visualization and reporting.
• Familiarity with Microsoft Fabric architecture.
• Knowledge of Azure IaaS services.
• Understanding of CI/CD processes and tools like Azure DevOps.

Qualifications:

• Bachelors degree in Computer Science, Information Technology, or a related field.
• 5+ years of experience in data engineering with a focus on Azure services.
• Proven expertise in designing and implementing data pipelines and architectures in cloud environments.
• Strong problem-solving skills and ability to work collaboratively in a team.

Preferred Certifications:

• Microsoft Certified: Azure Data Engineer Associate.
• Microsoft Certified: Azure Solutions Architect Expert.

Microsoft Certified: Power BI Data Analyst Associate