Data Engineer IV - Big Data / Spark

Must have skills :

- Minimum of 5-7 years of experience in software development, with a focus on Java and infrastructure tools.

- Min 6+ years of experience as a Data Engineer.

- Good Experience in handling Big Data Spark, Hive SQL, BigQuery, SQL.

- Candidate worked on cloud platforms and GCP would be an added advantage.

- Good understanding of Hadoop based ecosystem including hard sequel, HDFS would be very essential.

- Very good professional knowledge of PySpark or using Scala.

Responsibilities :

- Collaborate with cross-functional teams such as Data Scientists, Product Partners and Partner Team Developers to identify opportunities for Big Data, Query ( Spark, Hive SQL, BigQuery, SQL ) tuning opportunities that can be solved using machine learning and generative AI.

- Write clean, high-performance, high-quality, maintainable code.

- Design and develop Big Data Engineering Solutions Applications for above ensuring scalability, efficiency, and maintainability of such solutions.

Requirements :

- A Bachelor or Master's degree in Computer Science or a related field.

- Proven experience working as a Big Data & MLOps Engineer, with a focus on Spark, Scala Spark or PySpark, Spark SQL, BigQuery, Python, Google Cloud,.

- Deep understanding and experience in tuning Dataproc, BigQuery, Spark Applications.

- Solid knowledge of software engineering best practices, including version control systems (e.g Git), code reviews, and testing methodologies.

- Strong communication skills to effectively collaborate and present findings to both technical and non-technical stakeholders.

- Proven ability to adapt and learn new technologies and frameworks quickly.

- A proactive mindset with a passion for continuous learning and research.