Data Engineer - Azure Databricks (4-12 yrs)

Data Engineer (Teradata to Snowflake and Databricks on Azure Cloud).

Position Overview :

We are looking for a talented Data Engineer with strong expertise in data migration from Teradata to Snowflake and Databricks on Azure Cloud.

This position will be pivotal in migrating legacy data systems to cloud-based solutions, ensuring a smooth transition with minimal disruption.

The ideal candidate will be responsible for designing, building, and maintaining robust data pipelines that support seamless migration, as well as ensuring the quality, scalability, and reliability of the data environment post-migration.

Responsibilities :

Lead Data Migration Efforts: Take the lead in migrating data systems from Teradata to Snowflake and Databricks on Azure Cloud, ensuring the integrity, quality, and security of data throughout the migration process.

ETL Migration and Optimization: Design, build, and optimize ETL pipelines to facilitate the migration of large datasets from Teradata to cloud-based systems, ensuring high performance and data consistency.

Cloud Data Architecture: Architect cloud data solutions using Azure, including Azure Databricks, Snowflake, and Azure Data Lake, for optimized data storage and processing in the cloud.

Data Modeling and Schema Migration: Manage and oversee the migration of data models, schemas, and data structures from Teradata to Snowflake and Databricks, ensuring efficient performance in the cloud environment.

Post-Migration Data Integrity: Ensure the accuracy and completeness of data after migration by performing thorough validation, testing, and troubleshooting to resolve any issues in data flows.

Performance Tuning : Focus on optimizing the performance of data pipelines and ensuring the efficient use of cloud resources post-migration.

Collaboration : Work closely with cross-functional teams (e.g, data scientists, analysts, and business stakeholders) to ensure data systems are properly integrated into the cloud environment and align with business needs.

Governance and Compliance : Implement and monitor data governance best practices, including security, privacy, and compliance policies, to ensure the integrity and protection of data throughout and after the migration.

Automation and Scalability : Automate and scale migration processes to ensure that data migration efforts are repeatable, efficient, and cost-effective.

Required Skills :

- 4-12 years of experience in data engineering, with specific expertise in data migration from Teradata to Snowflake and Databricks.

- Proficient in Azure Cloud tools, including Azure Databricks, Azure Data Lake, Azure SQL Database, and Azure Synapse Analytics.

- Strong experience with migrating and integrating data systems to cloud-based platforms, specifically Snowflake and Databricks on Azure.

- Proficient in building and optimizing ETL pipelines for cloud environments using tools like Azure Data Factory, Databricks, and Apache Spark.

- Expertise in SQL and Python for data migration, transformation, and automation tasks.

- Solid understanding of data modeling and experience migrating complex data schemas to cloud environments.

- Familiarity with cloud security protocols, encryption, and data privacy concerns in cloud-based data systems.

- Ability to troubleshoot and resolve complex migration issues and ensure seamless integration across platforms.

Preferred Skills :

- Azure certifications (Azure Data Engineer, Azure Solutions Architect) or other relevant cloud certifications.

- Experience with CI/CD pipelines for automating and deploying data migration processes.

- Knowledge of data governance and regulatory requirements for data migration to cloud environments.

- Familiarity with Delta Lake and Lakehouse architecture for data storage and management.