Senior Data Engineer - ETL/Spark (7-10 yrs)

Job Description :

- Design and implement scalable data migration pipelines from Relational databases, Teradata to AWS Databricks.

- Experience using IBM Data Stage, FiveTran, Airflow, Databaricks Workflow

- Optimize ETL processes, performance tuning, and data transformations using Spark and SQL.

- Develop automation for data ingestion, validation, and quality checks in a cloud environment.

- Collaborate with architects and stakeholders to ensure a secure, efficient, and seamless migration.

KEY RESPONSIBILITIES :

- Design and Implement Scalable Data Migration Pipelines:

- Develop robust and efficient data migration pipelines from relational databases and Teradata to AWS Databricks.

- Ensure pipelines are scalable to handle large volumes of data.

Utilize Diverse Data Integration Tools :

- Employ IBM DataStage, Fivetran, Airflow, and Databricks Workflows for data ingestion and orchestration.

- Select and apply the appropriate tools for specific data migration needs.

- Optimize ETL Processes and Performance:

- Fine-tune ETL processes for optimal performance using Spark and SQL.

- Conduct performance tuning and data transformations to improve efficiency.

- Develop Automation for Data Operations:

- Create automation scripts and processes for data ingestion.

- Implement automated data validation and quality checks within the cloud environment.

- Ensure Data Quality and Security:

- Implement data quality checks, and monitoring.

- Work with security teams to implement needed security measures.

- Collaborate with Architects and Stakeholders:

- Work closely with data architects to align data pipelines with overall architecture.

- Collaborate with stakeholders to understand requirements and ensure a seamless migration.

- Communicate clearly and effectively with both technical and non technical stakeholders.

- Cloud Environment Expertise:

- Work within the AWS cloud environment.

- Understand how to work with data within the AWS ecosystem.

REQUIRED SKILLS AND EXPERIENCE :

Technical Skills :

Data Migration :

- Extensive experience in designing and implementing data migration pipelines.

- Specific expertise in migrating data from relational databases (e.g., Oracle, SQL Server, DB2) and Teradata to AWS Databricks.

Databricks Expertise :

- Strong proficiency in AWS Databricks, including Spark (SQL, Python, or Scala).

- Experience with Databricks Workflows for orchestration.

ETL/ELT Tools :

- Hands-on experience with:

- IBM DataStage

- Fivetran

- Apache Airflow

Database Knowledge :

- Advanced SQL skills.

- Understanding of relational database systems and Teradata.

Cloud Computing (AWS):

- Familiarity with AWS services relevant to data storage, processing, and migration.

Programming/Scripting :

- Proficiency in at least one programming language (Python, Scala, etc.).

- Scripting skills for automation (e.g., shell scripting).

Data Optimization :

- Experience in optimizing ETL processes and performance tuning.

- Ability to perform complex data transformations.

Automation :

- Ability to develop automation scripts and processes for data ingestion, validation, and quality checks.

- Data Quality and Security:

- Experience implementing data quality checks.

- Understanding of data security principles in cloud environments.

Experience :

- Significant experience in data engineering roles, with a focus on data migration and ETL/ELT.

- Proven track record of designing and implementing scalable data pipelines.

- Experience working in cloud environments, preferably AWS.

- Experience collaborating with data architects and stakeholders.

- Experience in the full life cycle of data projects.

Soft Skills :

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills.

- Ability to work independently and as part of a team.

- Strong attention to detail.

- Ability to communicate complex technical information to non technical users.