Azure Data Bricks Software Engineer

Key Responsibilities

• The Databricks Engineer will be responsible for developing and maintaining data pipelines to read partner files from AWS S3 transform the data using Databricks and assist in the migration from Snowflake to Azure Databricks.
• This role requires working closely with the Databricks Architect to ensure the successful implementation of data solutions
• Develop and manage data pipelines to read and process partner files from AWS S3 using Azure Databricks.
• Implement transformations using Python and Spark to prepare data for loading into Snowflake and Azure Databricks
• Assist with the migration of data objects schema and databases from Snowflake to Azure Databricks
• Work with the Databricks Architect to ensure efficient design and implementation of data transformation and migration activities
• Perform unit testing and troubleshooting of data pipelines
• Ensure best practices are followed for coding security and data quality

Required Skills

• Experience in developing data pipelines with Azure Databricks Spark and Python.
• Strong understanding of AWS S3 Snowflake and data migration concepts.
• Proficiency in SQL for data querying transformation and analysis.
• Experience with unit testing debugging and troubleshooting data pipelines.
• Good communication skills and ability to work in a collaborative team environment.