Affine - Lead Data Engineer - Azure Platform (9-15 yrs)

Roles and Responsibilities :

- Resource and Project Timeline Estimation : Accurately estimate project timelines and resource allocation for Azure-based cloud projects.

- Defining Project and Program Management Process : Establish project and program management frameworks for Azure environments.

- Task Prioritization and Resource Management : Allocate tasks across resources by assessing skills and areas for improvement, with a focus on developing resource capabilities.

- Contribution to Organizational Initiatives : Involvement in sales collateral development, proposal design, leading Proof of Concepts, and training teams on Azure cloud technologies.

- Team Leadership : Recruit, train, develop, and supervise a team of Associates & Consultants, with a focus on cloud and data technologies.

- Specialization in Cloud Technologies : Expertise in Azure cloud and data services such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and other Azure-based data solutions.

- End-to-End Data Pipeline Design : Understanding upstream data and designing efficient data pipelines on Azure, building data marts in Synapse Analytics, and making transformed data accessible to downstream applications.

- Data Pipeline Optimization : Build and optimize high-performance data pipelines using Azure Databricks, with ingestion, cleansing, and curation into Azure Data Lake, Azure SQL, or other Azure data platforms using Spark with Python, Scala, or Java.

- Scalable Frameworks : Develop scalable, reusable frameworks for data ingestion and transformation, leveraging Azurenative services like Azure Data Factory and Azure Functions.

- Data Quality Assurance : Ensure data quality and integrity across the entire Azure data pipeline, implementing robust validation and cleansing mechanisms.

- Data Warehouse/Mart Design : Experience designing and building data warehouses or data marts using Azure Synapse or SQL Database.

- RDBMS Expertise : Strong knowledge of SQL Server, able to write and understand complex SQL queries.

- Documentation & Agile Delivery : Expertise in requirement gathering, technical design, and functional documentation, proven experience delivering projects using Agile frameworks (Scrum, Kanban).

- Client and Stakeholder Engagement : Lead discussions with clients and stakeholders, including on-site and remote engagements.

- Problem Solving and Adaptability: Strong analytical skills and the ability to quickly adapt to new challenges and technologies.

- Communication Skills : Excellent verbal and written communication skills.

Experience : 10 to 15 years of relevant experience in data engineering, with a focus on Azure-based cloud and data projects.

Technical Capabilities :

- Cloud Expertise : Experience managing projects on Azure Data Services, including Azure Data Lake, Azure Synapse Analytics, and Azure Databricks.

- Programming Languages : Expertise in Python, PySpark

- Databricks : Extensive experience working with Databricks on Azure.

- Database Expertise : Proficiency in SQL, with knowledge of SQL Server and Azure Cosmos DB (NoSQL).

- ETL/ELT Processes : Deep understanding of ETL/ELT processes and how to implement them using Azure Data Factory.

- DevOps & CI/CD Pipelines : Knowledge of CI/CD pipelines and DevOps practices using Azure DevOps or GitHub Actions.

- Agile Tools : Familiarity with Jira or similar tools used in agile environments.

Location : Gurugram - Hybrid Mode