Streaming Data Engineer - ETL/Data Pipeline (4-7 yrs)

Streaming Data Engineer

About the Role :

We are seeking highly skilled and experienced Streaming Data Engineers to join our growing team.

As a Streaming Data Engineer, you will be responsible for designing, developing, and maintaining real-time data pipelines to extract, transform, and load (ETL) high-velocity data streams from various sources. You will play a crucial role in building and optimizing data pipelines that deliver valuable insights in real-time.

Key Responsibilities :

- Design, develop, and implement real-time data pipelines using Apache Flink or Spark Streaming.

- Ingest data from various sources, including Kafka, messaging queues, and streaming APIs.

- Develop and maintain data lakes to store and process large volumes of streaming data.

- Optimize data storage and retrieval for high-performance analytics.

- Utilize Apache Airflow (preferred) or other orchestration tools to schedule and manage data pipelines.

- Implement data quality checks and validation rules.

- Work with SQL databases and write complex SQL queries for data manipulation and analysis.

- Experience with NoSQL databases (i.e., Cassandra, MongoDB) is a plus.

- Experience with cloud platforms (AWS, Azure, GCP) is a plus.

- Utilize cloud-native services for data storage, processing, and analysis (i.e., AWS S3, AWS Athena).

Qualifications :

Experience : 4+ years of experience in data engineering, with a strong focus on stream processing.

Technical Skills :

- Must-have : Expertise in Python, Apache Flink or Spark Streaming, Apache Kafka, and SQL.

- Preferred : Experience with Apache Airflow, Trino/Athena, Snowflake, and cloud platforms (AWS, Azure, GCP).

- Education : Bachelor's degree in Computer Science, Engineering, or a related field.

Other Skills :

- Excellent problem-solving and analytical skills.

- Strong communication and interpersonal skills.

- Ability to work independently and as part of a team.

- Strong work ethic and a passion for data