Senior Data Engineer - Python/PySpark (8-12 yrs)

Job Description :

Job Title : Senior Engineer and Data Engineer.

Job Location : Hybrid (Bengaluru/Noida).

Work Timing : IST Shift.

Job Type : Full-Time Employment.

Exp level : Mid to high (8+yrs Min).

Skill set/Description:

- 8 + years Experience for a Senior engineer and 6 + years for a data engineer Experience in Python/Pyspark scripting, Glue/lambda, Snowflake(mandate),

- Airflow or equivalent orchestration, AWS Cloud formation, Informatica (Secondary).

Responsibilities :

- Design, develop, and maintain efficient data pipelines using Python/PySpark, Glue, Lambda, and Snowflake.

- Implement data ingestion and transformation processes to extract, clean, and load data into data warehouses and data lakes.

- Optimize data pipelines for performance and scalability.

- Collaborate with data analysts and data scientists to understand their data needs and provide effective solutions.

- Monitor data pipelines for errors and anomalies, and implement corrective actions.

- Leverage AWS Cloud Formation to automate infrastructure provisioning and configuration.

- Work with AWS services like S3, EMR, and Redshift to build scalable data solutions. Ensure data security and compliance with industry standards.

- Utilize Airflow or equivalent orchestration tools to schedule and manage data pipelines.

- Implement data quality checks and monitoring mechanisms. Design and implement data models that are efficient and scalable.

- Collaborate with data analysts and business stakeholders to understand their data requirements and translate them into technical solutions.

- Identify and troubleshoot data quality issues and performance bottlenecks. Analyze complex data problems and propose innovative solutions.

- Work closely with cross-functional teams, including data analysts, data scientists, and software engineers.

- Share knowledge and best practices with team members.

Requirements :

- Strong proficiency in Python and PySpark.

- Experience with data engineering tools like Glue, Lambda, and Snowflake.

- Expertise in AWS cloud services, including S3, EMR, and Redshift.

- Knowledge of data orchestration tools like Airflow.

- Understanding of data warehousing and data lake concepts.

- Familiarity with SQL and data modeling techniques.

- Strong problem-solving and analytical skills.

- Excellent communication and interpersonal skills.

- Ability to work independently and as part of a team.

- A passion for data and a desire to learn new technologies.

Additional Considerations :

- Experience with Informatica is a plus.

- Experience with data visualization tools like Tableau or Power BI is beneficial.

- A strong understanding of data governance and security best practices is essential.