Country Delight - Data Engineer - Spark/Python (1-4 yrs)

Position Overview :

We are seeking a talented and experienced Data Engineer with expertise in Apache Spark, Python / Java and distributed systems. The ideal candidate will be skilled in creating and managing data pipelines using AWS.

Key Responsibilities :

- Design, develop, and implement data pipelines for ingesting, transforming, and loading data at scale.

- Utilize Apache Spark for data processing and analysis.

- Utilize AWS services (S3, Redshift, EMR, Glue) to build and manage efficient data pipelines.

- Optimize data pipelines for performance and scalability, considering factors like partitioning, bucketing, and caching.

- Write efficient and maintainable Python code.

- Implement and manage distributed systems for data processing.

- Collaborate with cross-functional teams to understand data requirements and deliver optimal solutions.

- Ensure data quality and integrity throughout the data lifecycle.

Qualifications :

- Proven experience with Apache Spark and Python / Java.

- Strong knowledge of distributed systems.

- Proficiency in creating data pipelines with AWS.

- Excellent problem-solving and analytical skills.

- Ability to work independently and as part of a team.

- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).

- Proven experience in designing and developing data pipelines using Apache Spark and Python.

- Experience with distributed systems concepts (Hadoop, YARN) is a plus.

- In-depth knowledge of AWS cloud services for data engineering (S3, Redshift, EMR, Glue).

- Familiarity with data warehousing concepts (data modeling, ETL) is preferred.

- Strong programming skills in Python (Pandas, NumPy, Scikit-learn are a plus).

- Experience with data pipeline orchestration tools (Airflow, Luigi) is a plus.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

Preferred Qualifications :

- Experience with additional AWS services (e.g., AWS Glue, AWS Lambda, Amazon Redshift).

- Familiarity with data warehousing and ETL processes.

- Knowledge of data governance and best practices.

- Have a good understanding of the oops concept.

- Hands-on experience with SQL database design

- Experience with Python, SQL, and data visualization/exploration tools