Data Engineer

Position Summary:
As a Data Engineer, you will be responsible for designing, building, and maintaining data pipelines, ensuring the delivery of reliable and scalable solutions. You will collaborate with cross-functional teams to optimize data workflows and contribute to the organizations data-driven strategies. While high-performance metrics are expected, we understand these benchmarks will require consistent effort and growth in the role.

Key Responsibilities:

1. Develop and maintain ETL pipelines aligned with the architectural framework and best practices, aiming for a 99% success rate for daily data loads. Implement data quality checks based on established standards, targeting 98% accuracy for key data tables over time.

2. Collaborate with cross-functional teams to deliver 5+ complex data models per quarter, ensuring alignment with the overall architectural vision and contributing to business goals.

3. Optimize data processing workflows in line with recommended practices to progressively reduce execution time by 20% within six months.

4. Implement and maintain CI/CD pipelines for all data projects, striving to achieve a goal of 95% on-time deployments while following established workflows.

5. Work closely with technical leadership and cross-functional teams to plan and execute the migration of on-premise data/solutions to the cloud, contributing to at least one major migration project annually while adhering to the overarching architectural principles.

Skills and Qualifications:

1. At least 4 years of experience in Data Engineering, including 2.5+ years with Spark-based tools (e.g., Databricks, PySpark).
2. Proficiency in SQL and Python, with experience in writing complex queries and scripts.
3. Strong knowledge of Azure or AWS, including services like Data Factory, Glue, or S3.
4. Experience with data modeling, performance tuning, and big data frameworks.
5. Familiarity with data governance tools like Apache Atlas or Azure Purview is a plus.
6. Excellent problem-solving skills and the ability to handle large datasets (1TB+).