Data Engineer (Streaming Technologies)

Job Title: Streaming Data Engineer

Experience Level: 6+ Years

Job Summary: We are seeking a highly skilled Streaming Data Engineer to join our team. The ideal candidate will have extensive experience in developing, optimizing, and managing streaming data pipelines and platforms. This role requires strong expertise in Python, SQL, and cutting-edge streaming technologies to build scalable and efficient data solutions.

Key Responsibilities:

• Design, develop, and maintain real-time data streaming pipelines.
• Build scalable solutions using Apache Flink Streaming or Spark Streaming.
• Integrate and manage Apache Kafka for event-driven data architectures.
• Develop and maintain data lakes to support diverse data storage and processing needs.
• Optimize Spark/PySpark and Hive applications for performance and scalability.
• Implement orchestration workflows, preferably using Apache Airflow.
• Collaborate with cross-functional teams to ensure data quality, reliability, and availability.
• Contribute to architecture decisions, ensuring alignment with business requirements.

Required Skills:

• Programming Expertise: Advanced proficiency in Python.
• SQL Skills: Ability to write and optimize complex SQL queries.
• Streaming Technologies: Hands-on experience with Apache Flink Streaming or Spark Streaming.
• Event Streaming: Expertise in Apache Kafka.
• Data Lake Development: Proven experience in designing and maintaining data lakes.
• Workflow Orchestration: Experience with Apache Airflow or similar tools.
• Big Data Optimization: Strong knowledge of optimizing Spark/PySpark and Hive applications.

Preferred Skills (Good to Have):

• Experience with Trino or AWS Athena.
• Familiarity with Snowflake.
• Understanding of data quality frameworks and implementation.
• Knowledge of file storage solutions like Amazon S3.

Qualifications:

• Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.
• Proven experience in developing and managing streaming data solutions.
• Strong problem-solving skills and the ability to work in a fast-paced environment.