Streaming Data Engineer - Apache Flink/Spark (4-7 yrs)

As a Streaming Data Engineer, you will play a critical role in designing, developing, and maintaining real-time data processing pipelines.

You will be responsible for ingesting, transforming, and analyzing high-velocity data streams to deliver valuable insights in real-time.

Key Responsibilities :

- Design and develop real-time data pipelines using Apache Flink or Apache Spark Streaming.

- Ingest data from various sources, including Kafka, messaging queues, and streaming APIs.

- Transform and enrich streaming data using stream processing techniques.

- Develop and maintain data lakes to store and process large volumes of streaming data.

- Optimize data storage and retrieval for high-performance analytics.

- Utilize Apache Airflow or similar tools to orchestrate and schedule data pipelines.

- Monitor and troubleshoot data pipelines to ensure data quality and reliability.

- Implement data quality checks and validation rules.

- Leverage cloud platforms like AWS (with services like S3, Athena) for data storage and processing.

Required Skills and Experience :

- Strong Proficiency in Python : Extensive experience in Python programming.

- Stream Processing : Hands-on experience with Apache Flink or Apache Spark Streaming.

- Data Ingestion : Experience with Kafka or other message queuing systems.

- SQL Expertise : Ability to write complex SQL queries for data manipulation and analysis.

- Data Lake Development : Experience with data lake architectures and technologies.

- Data Orchestration : Experience with Apache Airflow or similar orchestration tools.

Preferred Skills :

- Experience with Trino/Athena.

- Experience with Snowflake.

- Experience with data visualization and analytics tools.

- Knowledge of containerization technologies (Docker, Kubernetes)