AU Small Finance Bank - Spark Engineer - Big Data (5-8 yrs)

Job Description :

Responsibilities :

1. Design and Develop Spark Applications :

- Understand the current Apache Spark platform scala code and nuances related to the same

- Optimize and fine-tune Spark jobs for performance and efficiency.

2. Python Programming :

- Utilize Python for scripting, data manipulation, and integration with Spark applications.

- Collaborate with cross-functional teams to integrate Python-based solutions into Spark workflows.

3. Big Data Processing :

- Work with large-scale distributed systems for processing and analyzing big data.

- Implement best practices for data processing, storage, and retrieval.

4. Performance Optimization :

- Identify bottlenecks and optimize Spark applications for enhanced performance.

- Implement caching and partitioning strategies to improve data processing speed.

5. Collaboration and Communication :

- Collaborate with data scientists, analysts, and other engineering teams to understand business requirements and translate them into technical solutions.

- Communicate effectively with team members and stakeholders.

6. Monitoring and Troubleshooting :

- Implement monitoring solutions for Spark jobs and proactively address any issues.

- Troubleshoot and resolve performance and operational issues.

7. Documentation :

- Maintain comprehensive documentation for Spark applications, workflows, and processes.

- Provide training and knowledge transfer to junior team members.

Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 5 to 8 years of experience working with Apache Spark, Scala.

- Familiarity with AWS Cloud platform

- Strong proficiency in Python programming.

- Experience with big data processing frameworks and distributed computing.

- Solid understanding of data structures, algorithms, and data modeling.

- Excellent problem-solving and analytical skills.

- Effective communication and collaboration skills