Data Engineer - Python/PySpark (5-8 yrs)

Primary Skills : Strong experience in Azure Data Factory (ADF) , Azure Databricks (ADB) , PySpark , Delta Lake, with strong development skills for 5-8 years in Data Engineering Projects

Job Description :

Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies to handle a distribution of data types and volumes in support of data architecture design. A Data Engineer designs data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.

Key Responsibilities :

- Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks,Azure Fabric).

- Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage).

- Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions.

- Optimize data pipelines in the Azure environment for performance, scalability, and reliability.

- Ensure data quality and integrity through data validation techniques and frameworks.

- Develop and maintain documentation for data processes, configurations, and best practices.

- Monitor and troubleshoot data pipeline issues to ensure timely resolution.

- Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge.

- Manage the CI/CD process for deploying and maintaining data solutions.

Qualifications :

Learning agility :

Technical Leadership :

- Consulting and managing business needs

- Strong experience in Python is preferred but experience in other languages such as Scala, Java, C#, etc is accepted.

- Experience building spark applications utilizing PySpark.

- Experience with file formats such as Parquet, Delta, Avro.

- Experience efficiently querying API endpoints as a data source.

- Understanding of the Azure environment and related services such as subscriptions, resource groups, etc.

- Understanding of Git workflows in software development.

- Using Azure DevOps pipeline and repositories to deploy and maintain solutions.

- Understanding of Ansible and how to use it in Azure DevOps pipelines.