PySpark Data Engineer

Role & responsibilities

Key Responsibilities: Data Engineering Development:

• Design, develop, and test PySpark-based applications to process, transform, and analyze large-scale datasets from various sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
• Implement efficient data transformation and aggregation using PySpark and relevant big data frameworks.
• Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
• Optimize PySpark jobs for performance, including partitioning, caching, and tuning of Spark configurations.

Data Analysis and Transformation:

• Collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions.
• Analyze and interpret data structures, formats, and relationships to implement effective data transformations using PySpark.
• Work with distributed datasets in Spark, ensuring optimal performance for large-scale data processing and analytics.

Data Integration and ETL:

• Design and implement ETL (Extract, Transform, Load) processes to ingest and integrate data from various sources, ensuring consistency, accuracy, and performance.
• Integrate PySpark applications with data sources such as SQL databases, NoSQL databases, data lakes, and streaming platforms

Qualifications and Skills:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
• Strong understanding of data processing principles, techniques, and best practices in a big data environment.
• Proficiency in PySpark, Apache Spark, and related big data technologies for data processing, analysis, and integration.
• Experience with ETL development and data pipeline orchestration tools (e.g., Apache Airflow, Luigi).
• Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
• Excellent communication and collaboration skills to work effectively with data analysts, data architects, and other team members.