Lead Data Engineer

Job Title: Lead Data Engineer

Location: Hyderabad

Job Type: Full-Time

Job Description:

Vector India is seeking an experienced and motivated Lead Data Engineer to join our dynamic team. As a Lead Data Engineer, you will play a pivotal role in shaping and driving the data engineering strategies across the organization. You will be responsible for leading a team of talented engineers, working on Azure-based cloud platforms, and utilizing cutting-edge technologies to build scalable data pipelines.

Key Responsibilities:

• Lead and manage a team of data engineers, providing technical guidance, support, and mentorship.
• Design, build, and optimize data pipelines using Azure services such as Azure Databricks, Azure Data Lake Storage (ADLS), and Azure Data Factory (ADF).
• Lead the development of distributed data processing solutions using Apache Spark and PySpark.
• Ensure high availability and performance of data systems while ensuring compliance with industry standards.
• Collaborate with cross-functional teams to understand business requirements and translate them into data-driven solutions.
• Review and optimize SQL queries for performance and scalability.
• Drive best practices for data engineering, code quality, and documentation.
• Troubleshoot and resolve complex data-related issues.
• Help design scalable data architecture solutions on the Azure platform.
• Maintain and enforce best practices for data engineering workflows, from ingestion to transformation and storage.

Skills and Qualifications:

• Experience: Around 10 years of experience in data engineering with at least 5 years of experience working on Azure platform.
• Azure Services: Expertise in Azure Databricks, Azure Data Lake Storage (ADLS), Azure Data Factory (ADF).
• Programming Languages: Strong experience with PySpark and Apache Spark for distributed data processing.
• SQL: Strong SQL skills with experience in writing efficient and optimized queries.
• Leadership: Proven experience in managing small teams and providing technical leadership and guidance.
• Problem-Solving: Ability to address complex technical issues, design solutions, and improve data processing workflows.
• Collaboration: Excellent communication skills with the ability to collaborate with multiple stakeholders.
• Education: A degree in Computer Science, Engineering, or a related field is preferred.

Certifications (Preferred but not mandatory):

• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert
• Databricks Certified Data Engineer