Senior Data Engineer - Spark/Python (5-7 yrs)

Job Description :

Role & Responsibilities :

About the Role :

We are seeking a highly skilled Senior Data Engineer with 5-7 years of experience to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in data warehouse architecture, data modeling, ETL processes, and building both batch and streaming pipelines. The candidate should also possess advanced proficiency in Spark, Databricks, Kafka, Python, SQL, and Change Data Capture (CDC) methodologies.

Key responsibilities :

1. Design, develop, and maintain robust data warehouse solutions to support the organization's analytical and reporting needs.

2. Implement efficient data modeling techniques to optimize performance and scalability of data systems.

3. Build and manage data lakehouse infrastructure, ensuring reliability, availability, and security of data assets.

4. Develop and maintain ETL pipelines to ingest, transform, and load data from various sources into the data warehouse and data lakehouse.

5. Utilize Spark and Databricks to process large-scale datasets efficiently and in real-time.

6. Implement Kafka for building real-time streaming pipelines and ensure data consistency and reliability.

7. Design and develop batch pipelines for scheduled data processing tasks.

8. Collaborate with cross-functional teams to gather requirements, understand data needs, and deliver effective data solutions.

9. Perform data analysis and troubleshooting to identify and resolve data quality issues and performance bottlenecks.

10. Stay updated with the latest technologies and industry trends in data engineering and contribute to continuous improvement initiatives.

Ideal Candidate :

1. Bachelor's degree in Computer Science, Engineering, or a related field.

2. 5-6 years of hands-on experience in data engineering roles, with a focus on building and maintaining data warehouse solutions.

3. Strong proficiency in data modeling concepts and experience with relational and dimensional data models.

4. Extensive experience with ETL processes and tools for data ingestion, transformation, and loading.

5. In-depth knowledge of Spark and Databricks for large-scale data processing and analytics.

6. Proficiency in Kafka for building real-time streaming pipelines and event-driven architectures.

7. Solid programming skills in Python for data manipulation, scripting, and automation tasks.

8. Expertise in SQL for querying and analyzing data from relational databases and data warehouses.

9. Experience with Change Data Capture (CDC) techniques and tools for capturing and propagating data changes.

10. Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.

11. Strong communication and collaboration skills, with the ability to work effectively in a team environment.