Data Engineer - Scala/Python (8-12 yrs)

Job Description :

We are looking for an experienced and skilled Data Engineer to join our team. The ideal candidate will have 8+ years of IT experience, with a focus on designing, developing, and maintaining scalable data infrastructure and pipelines. This role demands strong expertise in AWS cloud services, OLAP systems like ClickHouse, and proficiency in programming languages such as Java, Scala, and Python.

Key Responsibilities :

Data Infrastructure and Pipeline Development :

- Design, develop, and optimize scalable ETL pipelines for data ingestion and processing.

- Implement data transformation processes to ensure high data quality and availability.

Cloud Services Management :

- Leverage AWS cloud services (e.g., S3, Redshift, Lambda, Glue, EMR) for data storage, processing, and analytics.

- Design and manage cloud-native data architectures.

OLAP System Expertise :

- Work with ClickHouse or similar OLAP systems to optimize query performance and data modeling.

- Develop and maintain OLAP cubes for efficient analytics.

Distributed Data Processing :

- Utilize distributed processing frameworks such as Apache Spark (Core, Streaming, SQL), Apache Flink, or Apache Storm.

- Optimize the performance of big data solutions for real-time and batch processing.

Programming and Development :

- Write efficient and scalable code in Java/Scala and Python for data processing and transformation.

- Automate workflows and deployments using scripting and DevOps tools.

Collaboration and Best Practices :

- Collaborate with cross-functional teams, including data analysts, data scientists, and business stakeholders.

- Promote best practices in coding, data modeling, and system design.

Skills and Qualifications :

Mandatory Skills :

- Overall Experience : 8+ years in IT with a strong focus on data-related technologies.

- Cloud Expertise : 3+ years of hands-on experience with AWS data services.

- OLAP Systems : 3+ years of experience working with ClickHouse or other OLAP systems.

- Programming : Proficiency in Java/Scala and Python for data-centric development.

- Distributed Frameworks : Experience with at least one distributed data processing framework (e.g., Spark, Flink, Storm).

Additional Skills :

- Knowledge of DevOps practices and CI/CD pipelines for data workflows.

- Strong understanding of data warehousing concepts and database optimization.

- Familiarity with monitoring and logging tools for data platforms.

Preferred Skills :

- Certifications in AWS or big data technologies.

- Experience with other cloud platforms (e.g., Azure, GCP) is a plus.