Azure with PySpark - Architect

Data Pipeline Development :

• Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance.
• Develop and maintain ETL workflows for seamless data processing.

Azure Cloud Expertise :

• Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering.
• Implement and manage data storage solutions on Azure.

Data Transformation with PySpark:

• Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.
• Implement data cleansing, enrichment, and validation processes using PySpark.

Performance Optimization:

• Optimize data pipelines, queries, and PySpark jobs to enhance overall performance and scalability.
• Identify and address performance bottlenecks within data processing workflows.

Requirements:

• Proven experience as a Data Engineer, emphasizing expertise in Azure.
• Proficiency in Azure Data Factory and other relevant Azure services.
• Expertise in PySpark for data processing and analytics is a must.
• Experience with data modeling, ETL processes, and data warehousing.