Lead Data Engineer - ETL/Data Pipeline (8-10 yrs)

Job Title : Data Engineer Lead

Location : Chennai, Bangalore, Pune, Hyderabad

Experience : 8-10 Years

Skills Required : SQL, Data Modelling, Python

Job Description :

We are seeking an experienced and highly skilled Data Engineer Lead to join our growing team. In this role, you will lead data engineering initiatives, utilizing your expertise in data modeling, Python, and SQL to design and implement efficient data pipelines and systems.

You will work closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to ensure that high-quality data is available for analysis and decision-making.

Key Responsibilities :

- Lead Data Engineering Projects : Oversee the design, development, and implementation of robust, scalable, and efficient data pipelines, ensuring smooth data flow from various sources to data warehouses and lakes.

- Data Modeling : Develop and optimize data models (logical and physical), ensuring they support business analytics and reporting needs.

- Design relational, dimensional, and NoSQL models for efficient querying and analysis.

- Team Leadership : Lead, mentor, and guide a team of data engineers, providing technical expertise, setting best practices, and ensuring the team's success in meeting project objectives and deadlines.

- Database Management : Use SQL to write complex queries and optimize database performance.

- Work with relational and non-relational databases to ensure data is stored, processed, and retrieved efficiently.

- Automation & Optimization : Leverage Python for automating data-related tasks, including data extraction, transformation, and loading (ETL) processes.

- Optimize existing data pipelines for better performance and lower costs.

Requirements :

- Education : Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.

- Experience : 8+ years of experience in data engineering, with at least 8 years in a leadership or senior technical role.

Technical Skills :

- Expertise in Python for data engineering tasks, including building ETL pipelines, data processing, and automation.

- Strong proficiency in SQL for data manipulation, query optimization, and database management.

- Solid experience with data modeling techniques for relational, dimensional, and NoSQL databases.

- Experience with cloud platforms (AWS, Azure, Google Cloud) and relevant data engineering services (i.e., Redshift, BigQuery, Data Lake, S3).

- Familiarity with data warehousing and data lake architectures, ensuring data integration across systems.

- Experience with data pipeline orchestration tools (eApache Airflow, Luigi, Prefect) for automating workflows.

Tools & Technologies :

- Hands-on experience with data processing frameworks like Apache Spark, Hadoop, or similar big data technologies.

- Experience with data storage and query technologies such as MySQL, PostgreSQL, Snowflake, MongoDB, and others.

- Familiarity with version control systems like Git.

- Exposure to Docker, Kubernetes, or similar containerization/orchestration tools