Lead I - Data Engineering

This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions.

Outcomes:

Measures of Outcomes:

Outputs Expected:

Code Development:

Documentation:

Configuration:

Testing:

Domain Relevance:

Project Management:

Defect Management:

Estimation:

Knowledge Management:

Release Management:

Design Contribution:

Customer Interface:

Team Management:

Certifications:

Skill Examples:

Knowledge Examples:

Knowledge Examples

Design, develop, and maintain scalable data pipelines using Apache Spark and PySpark for processing large volumes of data. Implement data workflows and orchestration using Apache Airflow to ensure reliable and efficient data pipeline execution. Optimize data storage and retrieval processes by leveraging very good SQL skills and understanding of NoSQL databases (e.g., MongoDB, Cassandra). Collaborate with data scientists and analysts to understand data requirements and implement solutions that meet analytical needs. Monitor and troubleshoot data pipelines to ensure data quality, reliability, and performance. Design and implement data models and schemas that support business requirements and ensure data consistency. Evaluate and recommend new technologies and tools to enhance data processing and analytics capabilities. Document technical designs, procedures, and operating processes related to data engineering activities.