Omind - Big Data Engineer - Spark/Scala (6-11 yrs)

We are seeking a highly skilled Big Data Engineer with expertise in distributed systems, the Hadoop ecosystem, and advanced programming techniques. The ideal candidate will possess strong technical knowledge, thought leadership, and a passion for creating scalable, high-performance systems.

Key Responsibilities:

- Design, develop, and optimize distributed systems and Big Data solutions.

- Implement and maintain batch and streaming pipelines using Scala and Spark.

- Leverage experience in the Hadoop ecosystem, including technologies like Hive, Oozie, and Kafka.

- Build and maintain CI/CD pipelines to ensure efficient code deployment and integration.

- Apply design patterns, optimization techniques, and locking principles to enhance system performance and reliability.

- Scale systems and optimize performance through effective caching mechanisms.

- Demonstrate strong computer science fundamentals, logical reasoning, and problem-solving abilities.

- Collaborate with cross-functional teams to influence and drive innovative solutions.

- Provide technical leadership, mentorship, and presentations to guide team members and stakeholders.

Required Qualifications:

- Proven experience working with distributed systems and Big Data technologies.

- Proficiency in Scala, Spark, and related tools.

- In-depth understanding of design patterns and system optimisation principles.

- Hands-on experience with batch and streaming data pipelines.

- Familiarity with CI/CD pipelines and modern software development practices.

- Strong understanding of system scaling and caching techniques.

- Excellent communication and presentation skills, with the ability to inspire and influence others.

Preferred Qualifications :

- Scala, spark(preferably scala should be an experienced spark exposure

- Good understanding of Spark Architecture including Spark Core, Spark SQL, data Frames, Spark Streaming, Driver Node, Worker Node, stages, Executors and Tasks.

- Should have experienced with Hadoop ecosystem-HDFS, hive (must), Oozie, sqoop etc.

- Should have prior experience working with big data ingestion from Various sources handling TB' of data.

- Excellent Knowledge of SQL and coding skills.

- Experience with scripting languages such as Shell.

- Effective working in a team as well as working independently.

- Experience of working in Agile delivery.

We would love to hear from you if you are passionate about working on complex systems and leading the charge in cutting-edge Big Data solutions.