Spark Developer

Job Description

Responsibilities:

1. Design and Develop Spark Applications:

• Understand the current Apache Spark platform scala code and nuances related to the same
• Optimize and fine-tune Spark jobs for performance and efficiency.

2. Python Programming:

• Utilize Python for scripting, data manipulation, and integration with Spark applications.
• Collaborate with cross-functional teams to integrate Python-based solutions into Spark workflows.

3. Big Data Processing:

• Work with large-scale distributed systems for processing and analysing big data.
• Implement best practices for data processing, storage, and retrieval.

4. Performance Optimization:

• Identify bottlenecks and optimize Spark applications for enhanced performance.
• Implement caching and partitioning strategies to improve data processing speed.

5. Collaboration and Communication:

• Collaborate with data scientists, analysts, and other engineering teams to understand business requirements and translate them into technical solutions.
• Communicate effectively with team members and stakeholders.

6. Monitoring and Troubleshooting:

• Implement monitoring solutions for Spark jobs and proactively address any issues.
• Troubleshoot and resolve performance and operational issues.

7. Documentation:

• Maintain comprehensive documentation for Spark applications, workflows, and processes.
• Provide training and knowledge transfer to junior team members.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 5-7 years of experience working with Apache Spark, Scala.
• Familiarity with AWS Cloud platform
• Strong proficiency in Python programming.
• Experience with big data processing frameworks and distributed computing.
• Solid understanding of data structures, algorithms, and data modeling.
• Excellent problem-solving and analytical skills.
• Effective communication and collaboration skills