Data Engineer

Data Engineering Skills:

Data Handling: Manage and optimize data within dimension and fact tables.

Data Pipeline Development: Create and maintain data pipelines for integrating and ingesting data from various sources.

Data Cleaning and Aggregation: Clean, enrich, and aggregate data within the Common Data Platform, leveraging the Bronze, Silver, and Gold layers.

Consumer Data Pipelines: Build and manage data pipelines to ensure the smooth flow of data to consumers.

E-R Model and Data Cataloging: Update the Entity-Relationship (E-R) model and catalog data to maintain data integrity and accessibility.

Data Platform Migration: Lead the migration of local data platforms to the Common Data Platform (CDP) using tools like Talend or Informatica.

ETL Transformation Replication: Replicate existing ETL transformations from local sources to the CDP.

Quality Assurance: Support both Technical QA and Business QA processes to ensure the accuracy and reliability of data.

Technical Skills:

Must Have:

PySpark SQL: Advanced proficiency in PySpark and SQL for data processing and querying.

Should Have:

Scripting (Python Bash): Proficiency in scripting with Python and Bash for automation and data manipulation.

AWS Knowledge: Strong understanding of AWS services and cloud architecture.

CI/CD (Github Actions): Experience with Continuous Integration/Continuous Deployment (CI/CD) pipelines using Github Actions.

Could Have:

Terraform: Familiarity with Terraform for infrastructure as code.

Airflow: Knowledge of Airflow for orchestrating complex data workflows.

Neptune or Neo4j: Experience with graph databases like Neptune or Neo4j.

Medallion Architecture: Understanding of Medallion Architecture principles.

Atlassian or Unity Catalog: Experience with data governance tools like Atlassian or Unity Catalog.

Soft Skills:

Communication: Collaborate effectively with data source technical teams to gather requirements and manage data.

Collaboration: Work with data consumer teams to resolve queries related to data handling and usage.

Change Management: Assist in the deprecation of legacy systems and onboard business users to the new data environment.