Senior Lead Data Engineer - Spark/Hadoop (8-10 yrs)

Role : Senior Lead Data Engineer

Years of Experience : 8+yrs

Location : Chennai(Hybrid)

No of Opening : 1

Job Description :

We are seeking a highly skilled Senior Lead Data Engineer to oversee the design, development, and optimization of our data infrastructure and integration processes. This role requires extensive experience in data engineering, leadership abilities, and proficiency in modern data technologies. Join us to drive innovation and lead a dynamic team in delivering robust data solutions.

Primary Skills : Azure, ADF, Databricks, DBT

Secondary Skills : AWS, Snowflake

Role Description :

The Lead Data Engineer will be responsible for overseeing the design, development, and maintenance of our data infrastructure and pipelines. This role requires strong technical expertise, leadership skills, and the ability to work collaboratively with cross-functional teams to ensure efficient and reliable data processes.

Role Responsibility :

- Lead the design and architecture of scalable and robust data integration solutions on-premise and in the cloud.

- Ensure solutions meet business requirements and industry standards.

- Engage with executive leadership and key stakeholders to understand business needs and translate them into technical solutions.

- Define data models, schemas, and structures to optimize data storage, retrieval, and processing for analytical workloads.

Role Requirement :

- Proficient in basic and advanced SQL.

- Proficient in using Python for data integration and data engineering tasks.

- Proficiency in ETL/ELT tools such as Informatica, Talend, SSIS, DBT, Databricks or equivalent.

- Experience with relational databases (likeSQL Server, Oracle, MySQL, PostgreSQL) and NoSQL databases (like MongoDB, Cassandra), cloud databases (RedShift, Snowflake, Azure SQL).

- Familiarity with big data technologies like Hadoop, Spark, Kafka, and cloud platforms such as AWS, Azure, or Google Cloud.

- Solid understanding of data modeling, data warehousing concepts, and practices.

- Good Knowledge and Understanding of Data warehouse concepts (Dimensional

- Work with data architects and solution

architects toensure alignment with

overall data strategy andarchitecture

principles.

Lead a team of data engineers by

providing technical guidance,

mentorship, and support. Plan,

prioritize, and manage data

engineeringprojects, tasks, and

timelines.

Design, develop, and maintain data

integrationpipelines and ETL/ELT

workflows.

Lead more than one project or manage

a largerteam that might have sub-

tracks.

Modeling, change data capture, slowlychanging

dimensions etc.).

Knowledgeable in performance tuningand optimization

Experience in Data Profiling and Datavalidation

Experience in requirements gatheringand

documentation processes and performing unit testing.

Understanding and Implementing QA andvarious testing

process in the project.

Knowledge in any BI tools will be anadded advantage.

Sound aptitude, outstanding logicalreasoning, and

analytical skills.

Willingness to learn and take initiatives.Ability to adapt to

fast-paced Agile environment.

Relevant certifications in data engineering or cloud platforms

are a plus.

Additional Requirement:

Minimum of 8-10 years of experience in data engineering, focusing on data integration, data warehousing, and big data

technologies.

At least 3-5 years in a leadership role, demonstrating the ability to manage and mentor engineering teams and lead complex

projects.