Principal Data Engineer

Responsibilities

Data Pipeline Design and Implementation:

Designs and implements data solutions using ETL processes, report generation. Develops dashboards and reports, integrates data from various sources, and ensures data is available for analysis.

Performance Optimization:

Monitor and optimize the performance of data streams and pipelines.

Implement strategies to reduce latency and improve throughput for real-time data processing.

Analyzes business data to uncover insights, prepares data for reporting, and creates visualizations for reports or dashboards

Builds and maintains data pipelines, manages ETL processes using SSIS , and ensures that data is correctly processed, stored, and accessible for reporting and analysis. Works on data architecture and supports BI teams by ensuring the availability of clean and accurate data.

Troubleshooting and Support :

Diagnose and resolve complex data pipeline and integration issues, ensuring high availability and reliability of data flows.

Collaborate with DevOps and platform teams to maintain stable environments for data processing workloads.

Collaboration and Innovation:

Work closely with data architects, engineers, and analysts to align on business requirements and deliver scalable solutions.

Research and implement the latest tools and technologies to enhance the data plane infrastructure.

Data Governance and Security:

Maintain robust logging, monitoring, and alerting systems for all data pipelines.

Ideal Candidate Will Have

Educational Qualifications:
Bachelors or masters degree in computer science, Engineering, or related technical field.
Technical Expertise:
Strong experience with Delta Sharing, enabling scalable and secure data exchange.
Advanced knowledge of Apache Kafka, including cluster setup, topic management, and real-time processing.
Hands-on expertise with Snowflake, including schema design, query optimization, and integration.
Experience with Oracle Data Streams or similar streaming solutions.
Programming and Scripting Skills:
Proficiency in programming languages and managing data pipelines.
Familiarity with SQL for data manipulation and analysis.
Data Infrastructure Skills:
Experience with cloud platforms such as OCI, AWS, or Azure for deploying data infrastructure.
Knowledge of tools like Spark, Flink, or Beam for distributed data processing is a plus.
Soft Skills:
Excellent problem-solving skills and attention to detail.
Strong communication skills for collaborating with technical and non-technical stakeholders.

Preferred Qualifications

Certification in data technologies such as Snowflake, Kafka, or Delta Sharing.
Experience with real-time analytics systems or machine learning data pipelines.
Familiarity with DevOps practices and tools for CI/CD in data pipelines.