PySpark Engineer

Responsibilities ? Design, develop, and maintain scalable and efficient data pipelines using PySpark and related big data technologies. ? Work with large datasets (structured and unstructured) and develop data models for consumption by data scientists and analysts. ? Optimize Spark jobs for better performance and resource management on large-scale clusters. ? Design and implement data integration workflows between various data sources such as HDFS, S3, and relational databases. ? Troubleshoot and resolve issues related to data pipelines, ensuring reliability and stability. ? Collaborate with cross-functional teams to understand business requirements and deliver solutions. ? Ensure the quality and cleanliness of data using data validation and transformation techniques. ? Write and maintain efficient, scalable code in Python and PySpark to process big data. ? Work with cloud platforms like AWS or Azure to manage data storage, computation, and scaling.. Requirements Must Have: ? Bachelor?s degree in computer science, Engineering, or a related field. ? 2 to 4 years of experience with PySpark for data processing and manipulation on large-scale datasets. ? Solid understanding of Spark architecture, including RDDs, DataFrames, and Datasets. ? Strong programming experience in Python, including libraries such as pandas, numpy, and matplotlib. ? Experience with Hadoop, Hive, and NoSQL databases (e.g., Cassandra, MongoDB). ? Working knowledge of cloud computing services (e.g., AWS, Azure, or Google Cloud). ? Familiarity with batch and stream processing (using Kafka, Flink, Spark Streaming). ? Strong problem-solving skills and attention to detail. ? Excellent communication and teamwork skills. Good to have skills: ? Experience with Apache Airflow or other orchestration tools for managing workflows. ? Familiarity with Docker or Kubernetes for containerized data environments. ? Experience in implementing and managing Continuous Integration (CI) and Continuous Delivery (CD) pipelines, with a focus on automating, testing, and deploying code in a fast-paced development environment.