Senior Data Engineer - Delta Lake & Azure Fabric Specialist

Overview:

We are seeking a Senior Data Engineer with deep expertise in Delta Lake, Azure Data Lake Storage Gen2 (ADLS Gen2), and a strong command of T-SQL, Azure Data Factory (ADF), and Spark Notebooks. The ideal candidate will play a critical role in building and optimizing our next-generation data architecture, transitioning from Synapse to Microsoft Fabric, and driving efficient, scalable, and reliable data pipelines and analytics solutions.

Key Responsibilities:

1. Data Architecture & Design:

• Develop, implement, and maintain a robust, scalable data architecture using Delta Lake on OneLake or ADLS Gen2.
• Design and implement optimized schema and partitioning strategies for large-scale data processing.

2. Data Pipelines & Orchestration:

• Create, monitor, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Dataflows Gen2.
• Automate data processing tasks and schedule orchestrated workflows to support real-time and batch workloads.

3. Data Aggregation & Transformation:

• Develop highly efficient data transformation processes for hourly, daily, and custom aggregations using T-SQL, Delta Lake, and Spark Notebooks.
• Optimize complex SQL queries and Spark jobs for performance and reliability.

4. Real-Time & Incremental Processing:

• Implement solutions for incremental data ingestion and processing, leveraging Delta Lakes ACID transactions and streaming capabilities.
• Design solutions to handle real-time data pipelines for campaign analytics, touchpoint performance, and client-specific KPIs.

5. Collaboration & Documentation:

• Work closely with data scientists, analysts, and business stakeholders to understand data needs and ensure solutions align with organizational goals.
• Document data engineering workflows, pipelines, and best practices.

6. Monitoring & Optimization:

• Implement monitoring systems for pipeline health, latency, and error rates using Azure tools.
• Continuously optimize data storage, compute resource utilization, and data retrieval performance.

Qualifications:

1. Technical Skills:

• Expertise in Delta Lake and ADLS Gen2 for large-scale data storage and processing.
• Advanced proficiency in T-SQL for complex query optimization and transformations.
• Hands-on experience with Azure Data Factory (ADF) for orchestrating ETL/ELT pipelines.
• Proficiency in Spark and experience with PySpark or Scala for building distributed data processing solutions.
• Familiarity with Microsoft Fabric and its integration with OneLake, Dataflows Gen2, and other components.

2. Experience:

• 5+ years of experience as a Data Engineer or in a similar role.
• Proven experience working with data lakes, real-time streaming, and batch processing systems.
• Strong background in designing and maintaining data warehouse solutions, preferably using Delta Lake and Azure technologies.

3. Soft Skills:

• Excellent problem-solving and debugging skills.
• Strong communication and documentation skills to collaborate with cross-functional teams.
• Ability to work in a fast-paced, dynamic environment while maintaining a focus on delivering high-quality solutions.

Preferred Skills:

• Experience with Azure Synapse Analytics or transitioning from Synapse to Microsoft Fabric.
• Familiarity with Power BI for designing and integrating data models with reporting tools.
• Understanding of CI/CD pipelines for data workflows and Azure DevOps.

Why Join Us:

• Opportunity to work on cutting-edge technologies in the Azure ecosystem, including Microsoft Fabric.
• Collaborative work environment with a focus on innovation and professional growth.
• Competitive compensation and benefits, including access to continuous learning resources.

Job Type: Full-time

Location: Remote or [Specify Location]

Compensation: Competitive, based on experience

How to Apply:

Submit your resume and a cover letter outlining your experience with Delta Lake, Azure Fabric, and related technologies to [email/contact details].

This job description highlights the specific expertise youre seeking, focusing on the tools and platforms (Delta Lake, ADLS Gen2, ADF, etc.) central to your current architectural goals and transition to Microsoft Fabric.