Data Engineer - SQL/Python/PySpark (10-12 yrs)

About the Role :

We are seeking a highly skilled and experienced Data Engineer to join our growing team.

In this role, you will play a critical role in developing and enhancing our data platform, ensuring efficient and reliable data processing, orchestration, and monitoring.

You will collaborate closely with product and technology teams to design and implement innovative solutions that drive data-driven decision making.

Responsibilities :

Data Platform Development & Enhancement :

- Design, develop, and maintain robust and scalable data pipelines using open-source technologies, AWS services, and GitLab automation.

- Optimize data processing and transformation workflows for performance and efficiency.

- Implement data quality checks and validations to ensure data accuracy and integrity.

Collaboration & Design :

- Collaborate with product managers, data scientists, and software engineers to understand business requirements and translate them into technical solutions.

- Participate in the design and validation of data platform capabilities, ensuring they meet the evolving needs of the business.

Process Improvement :

- Identify opportunities to automate manual processes, optimize for usability, and re-design for greater scalability.

- Drive continuous improvement initiatives to enhance the efficiency and effectiveness of data operations.

Technical Support & Guidance :

- Provide technical support and guidance to users of our data platform services.

- Troubleshoot and resolve data-related issues effectively and efficiently.

Monitoring & Alerting :

- Develop and refine metrics, monitoring, and alerting mechanisms to gain deep insights into the performance and health of our production services.

- Proactively identify and address potential issues before they impact data availability or quality.

Qualifications :

Essential Skills :

- Minimum 10 years of experience as a Data Engineer.

- Strong proficiency in SQL, Python, and PySpark.

- Experience building and optimizing data pipelines in a distributed environment (e.g, AWS, Hadoop).

- Experience with cloud platforms, particularly AWS (e.g, S3, EMR, Redshift, Glue).

- Experience with version control systems (e.g, Git, Bitbucket).

- Experience with CI/CD tools (e.g, Jenkins, CodeBuild, CodePipeline).

- Experience with data integration and ETL/ELT processes.

- Experience with data warehousing and data lake architectures.

- Strong analytical and problem-solving skills.

- Excellent communication and collaboration skills.

- Experience with Palantir technologies.

- Experience with containerization technologies (e.g, Docker, Kubernetes).

- Experience with data visualization tools (e.g, Tableau, Power BI).

- Knowledge of data governance and security best practices