Streaming Data Engineer - Python/SQL (5-10 yrs)

Job Summary :

You will be responsible for building and maintaining data pipelines that allow our organization to process and analyze real-time data streams.

Location : Anywhere in India

Notice period : Immediate to 30 days

Third-party payroll

Work from office

Key Responsibilities :

- Build and optimize real-time data pipelines to process large amounts of streaming data efficiently.

- Implement solutions to ingest, process, and analyze data from various real-time sources (IoT devices, logs, sensors, etc.).

- Integrate streaming systems with backend databases and data lakes to ensure smooth data flow.

- Develop real-time data transformation and enrichment logic.

- Ensure data is processed accurately and within the required timeframes.

- Work closely with cross-functional teams including data scientists, analysts, and software engineers to support various data use cases.

- Ensure the data quality and consistency in real-time processing environments.

Skills and Qualifications Required :

Streaming Data Engineer :

- Expertise in Python Language is a MUST.

- SQL (should be able to write complex SQL Queries) is MUST

- Hands-on experience in Apache Flink Streaming Or Spark Streaming MUST

- Hands-On expertise in Apache Kafka experience is MUST

- Data Lake Development experience.

- Orchestration (Apache Airflow is preferred).

- Spark and Hive : Optimization of Spark/PySpark and Hive apps

- Trino/(AWS Athena) (Good to have)

- Snowflake (good to have).

- Data Quality (good to have).

- File Storage (S3 is good to have)