Lead Data Engineer - Python/PySpark (8-12 yrs)

Office Location : Chennai (Work from Office)

Job Type : Full-time

Experience : 8+ Years

The tech stack in brief :

- Strong experience in Python, Pyspark with SQL.

- Augmentation of existing ELT pipelines or building new pipelines from scratch.

- DE cloud exp with Azure synapse, ADLS, ADF, Data bricks, and Snowflake.

- Big Data experience with the Hadoop ecosystem.

- Building data streaming pipelines in Kafka or similar tools.

- Someone who can own the design, development and deployment of data engineering solutions at scale.

Job Description :

As a lead data engineer you will oversee data architecture, ETL processes, and analytics pipelines, ensuring efficiency, scalability, and quality.

Key Responsibilities :

- Working with clients to understand their data.

- Based on the understanding you will be building the data structures and pipelines.

- You will be working on the application from end to end collaborating with UI and other development teams.

- You will be working with various cloud providers such as Azure & AWS.

- You will be engineering data using the Hadoop/Spark ecosystem.

- You will be responsible for designing, building, optimizing and supporting new and existing data pipelines.

- Orchestrating jobs using various tools such Oozie, Airflow, etc.

- Developing programs for cleaning and processing data.

- You will be responsible for building the data pipelines to migrate and load the data into the HDFS either on-prem or in the cloud.

- Developing Data ingestion/process/integration pipelines effectively.

- Creating Hive data structures,metadata and loading the data into data lakes / BigData warehouse environments.

- Optimized (Performance tuning) many data pipelines effectively to minimize cost.

- Code versioning control and git repository is up to date.

- You should be able to explain the data pipeline to internal and external stakeholders.

- You will be responsible for building and maintaining CI/CD of the data pipelines.

- You will be managing the unit testing of all data pipelines.

Tech Stack :

- Minimum of 5+ years working experience with Spark, Hadoop eco systems.

- Minimum of 4+ years working experience on designing data streaming pipelines.

- Should be an expert in either Python/Scala/Java.

- Should have experience in Data Ingestion and Integration into data lake using hadoop ecosystem tools such as Sqoop, Spark, SQL, Hive, Airflow, etc..

- Should have experience optimizing (Performance tuning) data pipelines.

- Should have minimum experience of 3+ years on NoSQL and Spark Streaming.

- Knowledge of Kubernetes and Docker is a plus.

- Should have experience with Cloud services either Azure/AWS.

- Should have experience with on-prem distribution such as Cloudera/HortonWorks/MapR.

- Basic understanding of CI/CD pipelines.

- Basic knowledge of Linux environment and commands.

Preferred Qualifications :

- Bachelor's degree in computer science or related field.

- Proven experience with big data ecosystem tools such as Sqoop, Spark, SQL, API, Hive, Oozie, Airflow, etc..

- Solid experience in all phases of SDLC with 10+ years of experience (plan, design, develop, test, release, maintain and support)

- Hands-on experience using Azure's data engineering stack.

- Should have implemented projects using programming languages such as Scala or Python.

- Working experience on SQL complex data merging techniques such as windowing functions etc..

- Hands-on experience with on-prem distribution tools such as Cloudera/HortonWorks/MapR.

- Should have excellent communication, presentation and problem solving skills.

Interview process :

- 2 technical interviews and 1 or 2 Managerial rounds.

If the candidate is from Chennai then one of the managerial rounds will be a face to face discussion at our office in Chennai.