Data Engineer - Scala/Spark (2-4 yrs)

Job Description :

Responsibilities :

- Design, develop, and maintain efficient and reliable data pipelines.

- Design and implement event/streaming-based architectures.

- Identify and implement process improvements, automate manual tasks, and optimize data delivery.

- Build and maintain the infrastructure for data extraction, transformation, and loading (ETL) from diverse sources using SQL and AWS cloud technologies.

- Develop data tools and solutions to empower our analytics and data science teams, contributing to product innovation.

- Collaborate with data scientists to enhance the functionality and performance of our data systems.

Requirements :

- Experience in a Product Development or Data Engineering role.

- Experience with big data tools such as HDFS/S3 Spark/Flink, Hive, HBase, Kafka/Kinesis.

- Proficiency in SQL and experience with NoSQL databases like Elasticsearch and Cassandra/MongoDB.

- Familiarity with data pipeline and workflow management tools (e., Airflow, Luigi, Azkaban).

- Experience with AWS cloud services.

- Experience with stream-processing systems (e., Spark Streaming, Flink).

- Working knowledge of object-oriented/functional scripting languages (e., Java, Scala, Python).

- Experience building and optimizing data pipelines and datasets.

- Strong analytical skills and experience working with structured and unstructured data.

- Understanding of data transformation, data structures, dimensional modeling, metadata management, schema evolution, and workload management.

- Working knowledge of message queuing, stream processing, and scalable data stores.

- Experience collaborating with cross-functional teams in a dynamic environment.

- A passion for building high-quality, scalable data solutions.

Bonus Points :

- Contributions to open-source projects.

- Experience working at a product company.

- Active GitHub profile showcasing personal projects.

- Working knowledge of a backend programming language (e., Python, Go)