Lead Azure Data Engineer - Data Factory/Databricks (4-6 yrs)

Company Description :

TalentOnLease is an innovative online platform in Gurugram, created by industry veterans with the mission of providing the right IT resources at the right time and cost.

The platform connects a wide range of clients with urgent IT requirements to verified IT partners for critical projects.

TalentOnLease aims to facilitate collaboration between clients and partners through a trusted and audited platform.

Job Description :

We are seeking a skilled Azure Data Engineer to join our team.

The ideal candidate will have a strong understanding of cloud-based data solutions, with hands-on experience in Azure services.

You will be responsible for designing, implementing, and optimizing data pipelines and solutions to meet business needs.

Key Responsibilities :

- Design and develop scalable and robust data solutions on the Azure platform.

- Build and maintain data pipelines using tools such as Azure Data Factory, Azure Databricks, or Synapse Analytics.

- Implement data ingestion, transformation, and integration workflows using Azure technologies.

- Optimize and maintain data storage solutions such as Azure Data Lake, Azure Blob Storage, or Azure SQL Database.

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.

- Ensure data security, compliance, and governance best practices using tools like Azure Purview.

- Monitor and troubleshoot data pipelines to ensure high availability and performance.

- Write and optimize complex SQL queries for data processing and analysis.

- Implement CI/CD practices for data pipelines using tools like Azure DevOps.

Required Skills and Qualifications :

- Proven experience in designing and building data solutions on the Azure platform.

- Experience handling a team size of at least 4.6 years of experience in building end-to-end business solutions using big data and data engineering.

- Expertise in core Microsoft Azure Data Services (e., Azure Data Factory, Azure Databricks, Azure.)

- Proficiency in Azure services such as Azure Data Factory, Azure Databricks, Synapse Analytics, Azure Data Lake, and Azure Functions.

- Strong knowledge of programming languages like Python, Scala, or SQL.

- Experience with distributed data processing frameworks such as Spark.

- Understanding of ETL/ELT processes and data modeling principles.

- Familiarity with containerization and orchestration tools like Docker and Kubernetes is a plus.

- Excellent problem-solving skills with a focus on optimizing performance and scalability.

- Experience with version control systems such as Git.

- Knowledge of data security, compliance standards (GDPR, HIPAA), and governance practices.

- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).

Preferred Qualifications :

- Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect.

- Knowledge of machine learning workflows and integration with data pipelines.

- Experience working with Power BI or other data visualization tools.

- Familiarity with REST APIs and data integration with third-party systems.

- Microsoft Azure Certifications are a plus.