KrtrimaIQ Cognitive Solution - Data Engineer - Python/Spark/Hadoop (5-12 yrs)

Job Description :

We are looking for an Only immediate joiner and experienced Data Engineer with a strong background in Kafka, PySpark, Python/Scala, Spark, SQL, and the Hadoop ecosystem. The ideal candidate should have over 5 years of experience and be ready to join immediately. This role requires hands-on expertise in big data technologies and the ability to design and implement robust data processing solutions.

Key Responsibilities:

- Design, develop, and maintain scalable data processing pipelines using Kafka, PySpark, Python/Scala, and Spark.

- Work extensively with the Kafka and Hadoop ecosystem, including HDFS, Hive, and other related technologies.

- Write efficient SQL queries for data extraction, transformation, and analysis.

- Implement and manage Kafka streams for real-time data processing.

- Utilize scheduling tools to automate data workflows and processes.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.

- Ensure data quality and integrity by implementing robust data validation processes.

- Optimize existing data processes for performance and scalability.

- Knowledge of data warehousing concepts and best practices.

- Familiarity with machine learning and data analysis tools.

- Understanding of data governance and compliance standards.