Data Engineer - ETL/Teradata (2-7 yrs)

Job Description :

- Design and implement scalable data migration pipelines from Relational databases, Teradata to AWS Databricks.

- Implement ETL processes, data transformations, and performance tuning using Spark and SQL.

- Support automation, data validation, and quality checks to ensure accurate data migration.

- Work closely with senior engineers and architects to enable a secure and efficient migration process.

KEY RESPONSIBILITIES :

- Data Migration Pipeline Development:

- Design and implement scalable data migration pipelines from relational databases and Teradata to AWS Databricks.

- ETL and Data Transformation :

- Implement ETL processes using Spark and SQL.

- Perform data transformations and performance tuning to optimize data processing.

- Automation and Data Quality:

- Support automation efforts for data ingestion and processing.

- Implement data validation and quality checks to ensure data accuracy.

- Collaboration and Support:

- Work closely with senior data engineers and architects to facilitate a secure and efficient migration process.

- Contribute to the overall success of the data migration project.

Required Skills and Experience :

Technical Skills:

- Data Migration:

- Experience in designing and implementing data migration pipelines.

- Familiarity with migrating data from relational databases (e.g., Oracle, SQL Server, DB2) and Teradata to AWS Databricks.

- Databricks and Spark:

- Proficiency in AWS Databricks and Apache Spark (SQL, Python, or Scala).

- Ability to implement ETL processes and data transformations using Spark.

SQL :

- Strong SQL skills for data manipulation and querying.

- ETL/ELT:

Understanding of ETL/ELT concepts.

- Cloud Computing (AWS):

Familiarity with AWS services relevant to data storage and processing.

- Automation:

Experience with scripting or tools for automation (e.g., Python, shell scripting).

- Data Quality:

- Understanding of data validation and quality checks.

Experience :

- Experience in data engineering or related roles.

- Experience in building and maintaining data pipelines.

- Familiarity with cloud environments.

- Experience working in a team environment.

Soft Skills :

- Strong problem-solving and analytical skills.

- Good communication and collaboration skills.

- Ability to work independently and as part of a team.

- Attention to detail.

- Willingness to learn.