Pyspark Developer

Job Description

Position: PySpark Developer

Location:Pune Job Summary: We are seeking an experienced PySpark Developer to join our team for a migration project. The primary responsibility will be to migrate Teradata tables and the Data Warehouse to Spark 3.x and Delta Lake. The ideal candidate will have extensive experience with PySpark, Teradata, Control M and Delta Lake, and a strong understanding of data processing and big data technologies. Key Responsibilities: Analyze existing Teradata tables and Data Warehouse structures. Design and implement a strategy to migrate data from Teradata to Spark 3.x and Delta Lake. Develop PySpark scripts to extract, transform, and load (ETL) data from Teradata to Delta Lake. Optimize and refactor existing SQL queries and ETL processes for performance in Spark. Collaborate with data engineers, database administrators, and business stakeholders to ensure data accuracy and integrity. Perform data validation and reconciliation to ensure the integrity of migrated data. Monitor and troubleshoot data migration processes to resolve any issues that arise. Document the migration process, including any changes made to data structures and ETL processes. Stay updated with the latest features and best practices in Spark, Delta Lake, and big data technologies. Design, develop, and implement job scheduling and automation processes using Control-M. Monitor and troubleshoot job schedules and automation workflows to ensure smooth operations. Perform regular maintenance and updates to Control-M job schedules and shell scripts. Stay updated with the latest features and best practices in Control-M and shell scripting. Required Qualifications: BE / B.Tech / MCA, Computer Science, Information Technology, Data Science, or a related field. 5+ years of experience in developing and maintaining ETL processes using PySpark. Strong experience with Teradata and Delta Lake. Proficiency in SQL and experience with query optimization. Knowledge of data warehousing concepts and best practices. Experience with data migration projects. Strong analytical and problem-solving skills. Excellent communication and collaboration skills. Ability to work independently and as part of a team. Preferred Qualifications: Experience with other big data technologies and frameworks. Knowledge of cloud-based data platforms (e.g., AWS, Azure, GCP). Experience with Spark and Delta Lake architectures and best practices. Certification in Spark or related technologies.