Data Engineer - Scala/Spark (8-10 yrs)

Job Title : Data Engineer

Location : Noida/Gurugram

Experience : 8+ Years

Job Overview :

We are seeking an experienced Data Engineer with strong expertise in data technologies and AWS Cloud services. The ideal candidate will have a deep understanding of OLAP systems like ClickHouse and experience with distributed data processing frameworks. If you are skilled in Java/Scala and Python and thrive in a collaborative environment, this is an excellent opportunity for you.

Key Responsibilities :

Data Engineering and Development :

- Design, develop, and optimize scalable data pipelines and data warehouses.

- Implement OLAP systems, focusing on ClickHouse or similar technologies.

AWS Cloud Services :

- Leverage AWS Cloud services for data storage, processing, and analytics.

- Ensure secure and efficient data management using AWS best practices.

Distributed Data Processing Frameworks :

- Work with frameworks like Spark (Core, Streaming, SQL), Storm, or Flink for processing large-scale datasets.

- Optimize frameworks for high performance and low latency.

Programming and Scripting :

- Develop robust applications using Java/Scala.

- Write scripts and utilities using Python for data transformations and automation.

Collaboration and Optimization :

- Collaborate with cross-functional teams to understand and fulfill data requirements.

- Optimize data workflows and systems for scalability and performance.

Skill Set Requirements :

Technical Expertise :

- 8+ years of IT experience with 5+ years in data-related technologies.

- At least 3+ years of expertise in AWS Cloud services related to data engineering.

- 3+ years of experience with OLAP systems, specifically ClickHouse or similar.

Programming Languages :

- Proficiency in Java/Scala for building data-intensive applications.

- Strong skills in Python for data manipulation and scripting.

Distributed Data Processing Frameworks :

- Hands-on experience with frameworks like Spark, Storm, or Flink.

- Proficiency in Spark modules: Core, Streaming, and SQL.

AWS Cloud Expertise :

- Experience with services such as AWS S3, Redshift, Glue, Lambda, EMR, and Kinesis.

Qualifications :

- Bachelor's/Master's degree in Computer Science, Information Technology, or a related field.

- Proven experience in designing and optimizing scalable data solutions.

- Strong analytical and problem-solving skills.

- Excellent communication and collaboration skills.