Azure with Azure Data Factory & Pyspark - Lead

Data Pipeline Development: Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance.

Develop and maintain ETL workflows for seamless data processing. Azure Cloud Expertise.

Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering. Implement and manage data storage solutions on Azure.

Data Transformation with PySpark Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.

Implement data cleansing, enrichment, and validation processes using PySpark.

Performance Optimization Optimize data pipelines, queries, and PySpark jobs to enhance overall performance and scalability.

Identify and address performance bottlenecks within data processing workflows.

Proven experience as a Data Engineer, emphasizing expertise in Azure.

Proficiency in Azure Data Factory and other relevant Azure services.

Expertise in PySpark for data processing and analytics is a must.

Experience with data modeling, ETL processes, and data warehousing.