Senior Data Engineer - Spark/Apache Flink (5-8 yrs)

About the Role :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing team. In this role, you will be responsible for designing, developing, and maintaining high-performance data pipelines and applications. You will leverage your expertise in big data technologies like Spark, Flink, Kafka, and cloud platforms like Azure to extract, transform, and load (ETL) large volumes of data from various sources.

Key Responsibilities :

Data Engineering :

- Design, develop, and implement data pipelines using Spark, Flink, Kafka, and other big data technologies.

- Develop and maintain Spark applications for batch and stream processing.

- Experience with Spark Streaming or Flink Streaming with Kafka projects (hands-on).

- Develop and optimize Spark and Hive applications.

- Experience with data lake development and data warehousing solutions.

Data Integration & Transformation :

- Extract, transform, and load data from various sources (e.g., databases, APIs, cloud storage) into target systems.

- Perform data cleansing, transformation, and enrichment.

- Ensure data quality and integrity throughout the data pipeline.

Cloud Technologies :

- Experience with cloud platforms like Azure and its relevant services.

- Experience with cloud storage services like Azure Blob Storage or AWS S3.

Data Warehousing :

- Experience with data warehousing technologies like Snowflake.

Scripting & Development :

- Strong proficiency in Python, including core Python concepts (e.g., file handling, data structures, algorithms).

- Proficiency in SQL, including advanced SQL queries (joins, aggregations, window functions).

- Experience with scripting languages like Shell.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 5+ years of hands-on experience as a Data Engineer.

- Strong proficiency in Python.

- Solid understanding of big data technologies: Spark, Flink, Kafka, Hive.

- Experience with data warehousing concepts and technologies (e.g., Snowflake).

- Experience with cloud platforms, preferably Azure.

- Experience with data orchestration tools (e.g., Apache Airflow).

- Strong analytical and problem-solving skills.

- Excellent communication and interpersonal skills.

- Ability to work independently and as part of a team