ATAI - Data Platform Lead (8-10 yrs)

Job Description :

We are seeking an experienced Data Platform Lead to spearhead the design and development of a pioneering federated platform for data sharing based on International Data Spaces (IDSA) standards.

In this role, you will collaborate with cross-functional teams to shape the platform's design and architecture, while leading a team of engineers to iteratively build and enhance the platform.

The ideal candidate will have deep expertise in modern data platforms such as data lakes or lakehouses, proficiency in open-source technologies like the Hadoop ecosystem, and a solid understanding of federated systems and data sovereignty.

Key Responsibilities :

- Lead the design and development of a federated data sharing platform based on IDSA standards.

- Collaborate with various teams to define the architecture and design of the platform.

- Oversee the iterative build-out of the platform, ensuring performance, scalability, and security.

- Build and maintain high-performance batch and stream processing data pipelines.

- Ensure alignment with enterprise architecture standards, particularly in areas such as identity, security, and federated data.

- Provide technical leadership and mentorship to a team of engineers, helping them grow and succeed in their roles.

- Contribute to the development of best practices for the platform and advocate for continuous improvement.

Required Skills & Experience :

- 8+ years of experience in data engineering, with a proven track record of building at least one end-to-end solution using a modern open-source tech stack.

- Expertise in architecture and design patterns within the data engineering space.

- Strong experience in building high-performance batch and stream processing pipelines.

- In-depth knowledge of enterprise architecture, particularly in identity, security, and federated data.

- Hands-on experience with Python-based data frameworks.

- Proven ability to build, manage, and lead a team of engineers.

- Strong communication skills with the ability to collaborate, influence, and build consensus across teams