Senior Data Engineer - PySpark/Databricks (5-8 yrs)

About the Role :

We are seeking a highly skilled and motivated Sr. Data Engineer to join our growing team. In this role, you will play a critical part in designing, developing, and maintaining robust and scalable data pipelines and solutions.

You will leverage your expertise in big data technologies like Spark, Azure, and Databricks to extract, transform, and load data from various sources into our data lake.

Key Responsibilities :

- Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks.

- Design and build ETL pipelines using Azure Data Factory (ADF).

- Build and maintain a Lakehouse architecture in Azure Data Lake Storage (ADLS) / Databricks.

- Perform data preparation tasks, including data cleaning, normalization, deduplication, type conversion, and more.

- Work closely with the DevOps team to deploy and maintain solutions in production environments.

- Monitor and control data processes, proactively identifying and resolving errors.

- Collaborate with Data Science and Business Intelligence teams to share knowledge, leverage best practices, and deliver impactful data-driven solutions.

- Lead and participate in projects, mentoring junior team members as needed.

- Implement and manage change management processes, including training, communication, and documentation for upgrades, changes, and data migrations.

Must-Have Skills :

- Strong proficiency in PySpark and Spark-SQL

- Hands-on experience with Azure Databricks

- Experience with Azure Data Factory (ADF)

- Solid understanding of ETL/ELT processes and data warehousing concepts

- Proficiency in SQL

Good-to-Have Skills :

- Experience with change management tools

- DevOps experience (i.e., CI/CD pipelines, containerization)

- Knowledge of cloud-native technologies and services

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field

- 5-8 years of experience as a Data Engineer or a similar role

- Excellent analytical and problem-solving skills

- Strong communication and interpersonal skills

- Ability to work independently and as part of a team