Principal Data Plane Engineer

Responsibilities

Data Pipeline Design and Implementation:

Architect, develop, and maintain real-time and batch data pipelines using tools like Kafka, Snowflake, and Oracle Data Streams.

Implement Delta Sharing to facilitate secure and scalable data sharing across systems and organizations.

Data Integration and Streaming:

Design robust integration solutions for streaming and processing high-throughput data using Apache Kafka and similar technologies.

Build pipelines that integrate seamlessly with Snowflake, ensuring optimal performance for analytics and reporting workloads.

Performance Optimization:

Monitor and optimize the performance of data streams and pipelines.

Implement strategies to reduce latency and improve throughput for real-time data processing.

Troubleshooting and Support:

Diagnose and resolve complex data pipeline and integration issues, ensuring high availability and reliability of data flows.

Collaborate with DevOps and platform teams to maintain stable environments for data processing workloads.

Collaboration and Innovation:

Work closely with data architects, engineers, and analysts to align on business requirements and deliver scalable solutions.

Research and implement the latest tools and technologies to enhance the data plane infrastructure.

Data Governance and Security:

Ensure data security and compliance with industry standards while implementing data sharing solutions using Delta Sharing and Snowflake.

Maintain robust logging, monitoring, and alerting systems for all data pipelines.

Ideal Candidate Will Have

Educational Qualifications:
Bachelors or masters degree in computer science, Engineering, or related technical field.
Technical Expertise:
Strong experience with Delta Sharing, enabling scalable and secure data exchange.
Advanced knowledge of Apache Kafka, including cluster setup, topic management, and real-time processing.
Hands-on expertise with Snowflake, including schema design, query optimization, and Snowpipe integration.
Experience with Oracle Data Streams or similar streaming solutions.
Programming and Scripting Skills:
Proficiency in programming languages like Python, Java, or Scala for building and managing data pipelines.
Familiarity with SQL for data manipulation and analysis.
Data Infrastructure Skills:
Experience with cloud platforms such as OCI, AWS, or Azure for deploying data infrastructure.
Knowledge of tools like Spark, Flink, or Beam for distributed data processing is a plus.
Soft Skills:
Excellent problem-solving skills and attention to detail.
Strong communication skills for collaborating with technical and non-technical stakeholders.

Preferred Qualifications

Certification in data technologies such as Snowflake, Kafka, or Delta Sharing.
Experience with real-time analytics systems or machine learning data pipelines.
Familiarity with DevOps practices and tools for CI/CD in data pipelines.

Why Join Us

Work with cutting-edge technologies to solve complex data challenges.
Opportunity to design and manage high-performance data systems for enterprise-grade applications.
Be part of an innovative and collaborative environment with growth opportunities.

If you are passionate about building scalable and reliable data solutions and have expertise in Delta Sharing, Kafka, Snowflake, and Oracle Data Streams, we encourage you to apply for this exciting role