Senior Data Engineer - Python/PySpark (5-10 yrs)

We are seeking a highly skilled Senior Data Engineer to join our team and lead data engineering initiatives. The ideal candidate will have extensive experience in designing and building scalable data pipelines, optimizing data workflows, and working with modern cloud-based technologies. This role involves coding, debugging, code refactoring, and collaborating with cross-functional teams to deliver robust data solutions. Key

Responsibilities :

- Design, build, and maintain scalable data pipelines and data assets.

- Develop high-quality code using Python, PySpark, and Boto3, adhering to best practices and frameworks.

- Enhance performance and efficiency through optimized Spark SQL and PySpark code.

- Modernize and refactor legacy codebases to improve readability, maintainability, and performance.

- Leverage services like S3, EC2, Lambda, Redshift, and CloudFormation to build and deploy data solutions.

- Implement unit tests (TDD approach) to ensure code quality and troubleshoot complex bugs, including performance and concurrency issues.

- Manage code versioning using tools like Git and artifact management with JFrog Artifactory.

- Work with cross-functional teams to gather requirements and deliver data-driven solutions tailored to business needs.

Required Skills & Qualifications :

- Minimum of 5+ years of experience in data engineering, with strong technical and problem-solving capabilities.

- At least 4+ years of hands-on experience with Python, PySpark, and Boto3, including frameworks and libraries.

- Solid understanding of AWS services such as S3, EC2, Lambda, Redshift, and CloudFormation, with the ability to explain their benefits and use cases.

- Strong ability to optimize code for performance and scalability, especially using Spark SQL and PySpark.

- Proficiency in writing unit tests and adopting TDD practices.

- Expertise in isolating and resolving complex code issues.

- Experience with Git and JFrog Artifactory for managing and maintaining codebases.

- Familiarity with data streaming tools and frameworks such as Kafka or Kinesis.

- Experience with CI/CD pipelines and DevOps practices.

- Knowledge of distributed data processing systems and big data technologies.

- Certifications in AWS or related fields.

- Work on challenging projects using the latest cloud and data technologies.

- Opportunities for upskilling, certifications, and career advancement.

- Be part of a supportive and innovative team. Develop data solutions that drive decision-making and business value.