Senior Data Engineer - Hadoop/Hive/Scala (8-9 yrs)

About the job

Company Description

Founded in early 2016, Sentienz is an IoT, AI, and Big Data company based in Bengaluru.

The company specializes in creating internet scale platforms, petabyte scale digital insights platforms, machine learning models, and advanced analytics platforms.

Sentienz Akiro is the AI-powered connectivity platform that enables IoT M2M communication and real-time connectivity.

Job Overview :

- We are seeking a Senior Data Engineer with expertise in Apache Spark, Scala, and Python to join our growing data team.

- As a Senior Data Engineer, you will be responsible for designing, implementing, and maintaining complex data processing systems to support our data pipelines, analytics, and machine learning models.

- You will collaborate with data scientists, analysts, and other engineers to build efficient and scalable data solutions.

Key Responsibilities :

- Design, develop, and optimize data pipelines using Apache Spark to process large volumes of structured and unstructured data.

- Write efficient and maintainable code in Scala and Python for data extraction, transformation, and loading (ETL) operations.

- Collaborate with cross-functional teams to define data engineering solutions to support analytics and machine learning initiatives.

- Implement and maintain data lake and warehouse solutions using cloud platforms (e.g, AWS, GCP, Azure).

- Ensure data workflows and distributed systems' performance, scalability, and reliability.

- Perform data quality assessments, implement monitoring, and improve data governance practices.

- Assist in migrating and refactoring legacy data systems into modern distributed data processing platforms.

- Provide technical leadership and mentorship to junior engineers and contribute to best practices in coding, testing, and deployment.

Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 8+ years of hands-on experience in data engineering, with strong skills in Apache Spark, Scala, and Python.

- Experience with distributed data processing frameworks and real-time data processing.

- Strong experience with big data technologies such as Hadoop, Hive, and Kafka.

- Proficient with relational databases (SQL, PostgreSQL, MySQL) and NoSQL databases (Cassandra, HBase, MongoDB).

- Knowledge of CI/CD pipelines and DevOps practices for deploying data workflows.

- Strong problem-solving skills and experience with optimizing large-scale data systems.

- Excellent communication and collaboration skills.

- Experience with orchestration tools like Airflow

- Experience with containerization and orchestration (e.g, Docker, Kubernetes)