Glorious Insight - Azure Data Engineer - PySpark (3-10 yrs)

Position Overview :

We are seeking an experienced Azure Data Engineer with a strong background in Azure Databricks, Azure Data Factory (ADF), PySpark, SQL, and Azure Data Lake Storage (ADLS). The ideal candidate will be responsible for designing and implementing robust, scalable data pipelines and workflows to support our data-driven initiatives.

Key Responsibilities :

- Develop and deploy end-to-end data pipelines using Azure Databricks, PySpark, and Azure Data Factory (ADF).

- Design and optimize data storage solutions in Azure Data Lake Storage (ADLS) for efficient querying and processing.

- Create, schedule, and monitor data workflows in Azure Data Factory to automate data movement and transformation.

- Implement ETL/ELT processes to ingest, transform, and prepare data from diverse sources (structured, semi-structured, and unstructured).

- Write and optimize complex SQL queries for data processing, analysis, and reporting.

- Integrate various Azure services (e.g., Synapse Analytics, Event Hubs, Logic Apps) into data workflows.

- Ensure data quality, integrity, and security through robust monitoring and error-handling mechanisms.

- Troubleshoot performance and reliability issues in data workflows and pipelines.

- Collaborate with stakeholders to gather requirements, understand business objectives, and deliver technical solutions.

- Document workflows, processes, and system configurations for knowledge sharing and operational support.

Qualifications :

Mandatory Skills :

- Azure Data Factory (ADF) : Expertise in building and orchestrating data pipelines using ADF.

- Azure Databricks : Hands-on experience in designing and managing Databricks clusters and workflows.

- PySpark : Proficiency in writing efficient PySpark scripts for distributed data processing.

- Azure Data Lake Storage (ADLS) : Deep understanding of storage architecture, data partitioning, and security management.

- SQL : Advanced skills in creating and optimizing queries for data manipulation and analysis.

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 3+ years of professional experience in data engineering with a focus on Azure cloud technologies.

Preferred Qualifications :

- Certification in Microsoft Azure Data Engineer Associate (DP-203) or related Azure certifications.

- Experience with Azure Event Hubs, Azure Functions, and Logic Apps for real-time and event-driven data processing.

- Familiarity with Apache Spark architecture and distributed computing principles.