Data Engineer - Python/PySpark (4-6 yrs)

Responsibilities :

- Design, develop, and implement efficient data pipelines using Python, PySpark, and distributed processing frameworks.

- Identify opportunities for process improvement and automate manual tasks to enhance efficiency and scalability.

- Collaborate with cross-functional teams (data analysts, product managers, software engineers) to understand data needs and translate them into technical solutions.

- Develop and implement comprehensive monitoring and alerting mechanisms to proactively identify and address potential issues within the platform.

- Continuously analyze and optimize data pipelines for performance and resource utilization.

- Provide technical support and guidance to users of the platform's data services.

- Stay up-to-date with the latest advancements in data engineering tools and technologies.

Qualifications :

- 4+ years of demonstrably successful experience as a Data Engineer or similar role.

- Strong experience building and optimizing data pipelines in a distributed environment (i.e., Spark, Flink).

- Proficiency in Python, PySpark, and advanced SQL for data manipulation and analysis.

- Experience working in a Linux environment and utilizing scripting languages (Bash).

- Familiarity with version control systems (Git/Bitbucket) and CI/CD tools (Jenkins/CodeBuild, CodePipeline).

- Experience with platform monitoring and alerting tools (i.e., Prometheus, Grafana).

- Excellent problem-solving, analytical, and critical thinking skills.

- Strong communication, collaboration, and interpersonal skills