Manager Pyspark Data Engeer

A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge.

• Lead and manage a team of Data Engineers specializing in Hadoop and PySpark to design, build, and optimize scalable data processing pipelines and architectures.
• Manage the development of endtoend data ingestion, transformation, and ETL workflows using Hadoop, PySpark, and related big data technologies.
• Oversee the design and implementation of distributed computing frameworks and realtime data processing solutions using Spark and PySpark.
• Ensure the effective management and operation of Hadoop clusters (HDFS, YARN, Hive, HBase) and related big data tools, ensuring high availability, security, and performance.
• Collaborate with Data Scientists, Business Analysts, and other teams to understand data requirements and deliver appropriate solutions that support analytical models and reporting.
• Develop and maintain data pipelines to process large volumes of structured and unstructured data, ensuring data consistency, integrity, and efficiency.
• Provide technical guidance, mentorship, and career development for junior engineers and team members in the Hadoop and PySpark ecosystem.
• Lead the design and implementation of data models and data storage strategies, including integration with cloud platforms such as AWS, Google Cloud, or Azure.
• Optimize performance of data pipelines and queries, troubleshooting any issues related to data processing and performance bottlenecks.
• Monitor and manage cloudbased Hadoop environments, including cost optimization, scaling, and performance tuning.
• Ensure adherence to best practices in code quality, data governance, and security standards.
• Communicate complex technical challenges and solutions effectively to both technical and nontechnical stakeholders.
• Drive innovation by staying uptodate with the latest big data technologies and incorporating relevant advancements into the platform.

• Bachelor s or Master s degree in Computer Science, Engineering, Information Technology, or a related field.
• 6 years of experience in Big Data Engineering, with a strong background in Hadoop and PySpark (at least 23 years in a leadership or managerial role).
• Proven experience in designing and implementing data pipelines using PySpark for largescale data processing.
• Strong understanding and handson experience with the Hadoop ecosystem, including HDFS, MapReduce, Hive, HBase, and YARN.
• Proficiency in Python and PySpark, including working with distributed computing frameworks and data processing workflows.
• Expertise in working with ETL processes, data lakes, data warehousing, and cloud platforms (AWS, GCP, or Azure).
• Strong knowledge of SQL and experience in data modeling, querying, and optimization.
• Experience in realtime data streaming using tools such as Kafka, Flink, or Spark Streaming is a plus.
• Familiarity with DevOps and CI/CD practices for managing big data workflows and infrastructure.
• Excellent leadership and team management skills, with the ability to mentor and guide engineers in technical best practices.
• Strong problemsolving skills, analytical mindset, and ability to work under pressure in fastpaced environments.
• Excellent communication skills, with the ability to explain complex technical concepts to both technical and nontechnical stakeholders.