Big Data Developer - Spark/Scala/Hive (4-8 yrs)

Skills :

Big Data Technologies :

- Extensive experience with Apache Spark and Scala

- Proficient in Hive and HDFS

- Hands-on experience with Oozie or similar workflow management tools

Programming Languages :

- Strong expertise in Scala and object-oriented programming concepts

Data Modeling & Processing :

- Solid understanding of data models, data mining techniques, and data partitioning strategies

Database Skills :

- Experience with SQL databases (e.g , MySQL, PostgreSQL)

DevOps & Tools :

- Proficient in CI/CD tools like Maven, Git, and Jenkins

- Experience with code quality and static analysis tools like SonarQube (SONAR)

Responsibilities :

- Develop and maintain scalable big data pipelines using Spark, Scala, Hive, and other related technologies.

- Design and implement data ingestion, transformation, and processing logic.

- Optimize data processing performance and ensure data quality.

- Implement data partitioning and other techniques for efficient data storage and retrieval.

- Collaborate with data scientists and analysts to understand data requirements.

- Develop and maintain data models and data dictionaries.

- Use CI/CD tools to automate build, testing, and deployment processes.

- Ensure code quality and maintainability using static analysis tools.

- Troubleshoot and resolve data-related issues.

- Stay up-to-date with the latest big data technologies and trends.

Qualifications:

Bachelor's degree in Computer Science or a related field.

- 4-8 years of experience in big data development.

- Strong understanding of big data concepts and architectures.

- Hands-on experience with Spark, Scala, Hive, HDFS, and Oozie.

- Proficiency in SQL and database technologies.

- Experience with CI/CD and code quality tools.

- Excellent problem-solving and communication skills.

Preferred Qualifications :

- Experience with cloud-based big data platforms (e.g , AWS EMR, Azure HDInsight, GCP Dataproc).

- Knowledge of other big data technologies (e.g , Kafka, HBase, Cassandra).

- Experience with data visualization tools