Pyspark Developer

Develop and maintain data pipelines and ETL processes using Apache Spark (PySpark) to process large volumes of structured and unstructured data.Implement and optimize data transformations, cleansing, and aggregation logic to deliver insights.Collaborate with data architects and other stakeholders to design scalable and efficient data processing solutions.Build and maintain end-to-end data pipelines for batch and real-time data processing.Work with data storage systems such as HDFS, Hive, and Amazon S3 for scalable data storage and retrieval.Ensure data quality and integrity by implementing data validation and error handling mechanisms.

Optimize PySpark jobs for performance by leveraging best practices for distributed processing.Tune Spark jobs to improve performance, scalability, and reduce runtime.Monitor and troubleshoot PySpark job failures and resource management issues.Work closely with data scientists, business analysts, and product teams to understand data requirements.Present technical designs and solutions to non-technical stakeholders, ensuring clear communication.Provide mentoring and support to junior team members.Write unit tests for PySpark jobs and ensure code coverage and quality.Document data workflows, processes, and design specifications for future reference.PySpark Hands-on experience in designing, developing, and deploying PySpark applications.Big Data Technologies Experience working with Spark, HDFS, Hive, Kafka, and other big data tools.Programming Languages Strong proficiency in Python and SQL.

 Familiarity with Java/Scala is a plus.Data Warehousing Experience with data warehousing concepts and systems like Amazon Redshift, Google BigQuery, or Snowflake.Cloud Platforms Experience with cloud technologies like AWS (S3, EMR, Redshift), GCP, or Azure is a plus.Databases Strong knowledge of relational (SQL) and NoSQL databases like PostgreSQL, MySQL, Cassandra, MongoDB.ETL Tools

Experience with ETL frameworks such as Apache Airflow, Luigi, or Oozie is preferred.Ability to analyze and optimize Spark job performance in a distributed environment.Familiar with partitioning strategies, caching, and data shuffling in PySpark.Knowledge of version control systems like Git.Experience with data pipelines, data orchestration, and scheduling systems.Strong analytical and problem-solving skills, with the ability to troubleshoot issues in complex data systems.Familiarity with Agile/Scrum methodologies.