AWS Data Engineer (Lead)

We are seeking a Data Observability Engineer to join our team and play a pivotal role in ensuring data reliability, quality, and governance across our organization. The ideal candidate will possess a strong understanding of Data Observability principles, have hands-on experience implementing and managing DataDog as a Data Observability tool, and demonstrate expertise in Data Governance and Data Quality frameworks.

Key Responsibilities

Data Observability Implementation and Monitoring

Design and implement Data Observability solutions using DataDog, ensuring proactive monitoring and alerting for data pipelines, workflows, and systems.

Set up automated anomaly detection, monitoring, and logging to identify data issues before they impact business operations.

Collaborate with data engineers and analysts to integrate observability practices into existing data infrastructure.

Data Quality Assurance

Develop frameworks and processes for assessing and improving Data Quality, including validation, reconciliation, and issue resolution.

Implement metrics, SLAs, and KPIs to ensure data meets organizational standards.

Data Governance

Work with cross-functional teams to establish and maintain Data Governance policies and procedures.

Assist in creating data cataloging, lineage, and classification systems to ensure data is discoverable, accessible, and compliant.

Tool Administration

Administer and optimize DataDog configurations, dashboards, and alerting systems to meet business requirements.

Stay updated with new features and capabilities of DataDog to continuously improve observability practices.

Collaboration and Stakeholder Engagement

Partner with data teams, business users, and leadership to drive a culture of observability and data excellence.

Provide training and documentation to stakeholders on Data Observability tools and best practices.

Qualifications

Technical Expertise:

Hands-on experience with DataDog for Data Observability, including setup, configuration, and administration.

Strong understanding of Data Observability frameworks and best practices.

Knowledge of data pipeline architectures (ETL/ELT), cloud data platforms (e.g., Snowflake, Databricks, AWS, GCP, Azure), and data integration tools.

Data Governance and Quality:

Familiarity with Data Governance frameworks and tools.

Experience in developing and managing Data Quality processes and metrics.

Programming and Scripting:

Proficiency in programming languages like Python, SQL, or similar for automation and data processing.

Experience with infrastructure as code tools (e.g., Terraform, CloudFormation) is a plus.

Soft Skills:

Strong problem-solving and analytical skills with attention to detail.

Excellent communication and collaboration skills to work effectively with technical and non-technical teams.

Preferred Qualifications

Certifications in DataDog, Data Governance, or Cloud Platforms.

Experience with machine learning or AI-driven monitoring for data systems.

Knowledge of regulatory and compliance requirements related to data (e.g., GDPR, CCPA).

EXPERIENCE

• 8-11 Years

SKILLS

• Primary Skill: Data Engineering
• Sub Skill(s): Data Engineering
• Additional Skill(s): Python, AWS-Apps, SQL