Etl Architect

Role & responsibilities

Preferred candidate profile

• Design Data Quality Architectures:

Architect scalable, automated, and efficient solutions for data validation, quality monitoring, and issue detection. Build frameworks that ensure data quality throughout the data lifecycle (from extraction to transformation to loading).

• Establish Data Quality Processes:
  Develop and enforce data quality processes, including data profiling, anomaly detection, data cleansing, and validation checks. Define and implement standardized methods for measuring data quality and KPIs.
• Ensure Data Quality Automation:

Drive the automation of data quality tests, including the creation of automated data validation scripts for both batch and streaming data. Ensure continuous integration and delivery (CI/CD) practices for data pipelines incorporate automated data quality checks.

• Monitor and Track Data Quality Metrics:
  Develop monitoring dashboards and reporting mechanisms to track data quality trends, issues, and performance. Provide regular updates to leadership and stakeholders on data quality metrics.
• Data Quality Testing:
  Lead the effort to create comprehensive test plans, frameworks, and tools for data quality testing. Ensure that data is validated across various stages of ETL processes (Extraction, Transformation, Loading).
• Mentor and Lead Team Members:

Provide technical leadership and mentorship to junior data quality engineers and team members. Build a strong data quality team by hiring, training, and guiding staff on best practices and tools.

• Experience:
• At least 10+ years of experience in data engineering, data quality engineering, or data testing, with at least 3-5 years in a leadership or architect role.
• Strong experience with data quality frameworks, ETL/ELT processes, data pipelines, and data governance.
• Expertise in building data quality testing frameworks and implementing data validation strategies across large-scale data systems.
• Technical Skills:
• Proficiency in Data Engineering Tools such as Apache Spark, Apache Kafka, Hadoop, SQL, Python, Scala, etc.
• Experience with data quality automation tools like Talend, Informatica, DataStage, dbt, or custom-built solutions.
• Expertise in data warehousing platforms (e.g., Snowflake, Redshift, Google BigQuery).
• Strong working knowledge of ETL tools and technologies such as Apache Nifi, Airflow, or similar.
• Familiarity with data profiling, anomaly detection, data reconciliation, and data cleansing techniques.
• Testing Expertise:
• Hands-on experience in data quality testing, including data profiling, validation testing, and regression testing of data pipelines.
• Proficiency in test automation using scripting languages (e.g., Python, Java, SQL) and frameworks (e.g., pytest, JUnit).
• Communication and Leadership:
• Excellent verbal and written communication skills, with the ability to clearly communicate complex technical concepts to non-technical stakeholders.
• Strong leadership and collaboration skills to work cross-functionally with other teams (engineering, business, etc.).

Location: Pune

Perks and benefits