Azure Big Data Engineer

Overall, 4 to 8 years of experience in IT Industry.

Min 4 years of experience working on Data Engineering using Azure Databricks, Synapse, ADF/Airflow.
At least 3 Project experience in Building and maintaining ETL / ELT pipelines for large data sets, complex data
processing, transformations, business logics, cost monitoring & performance optimization, and feature
engineering processes.

Must Have skills:
Extensive experience with Azure Databricks (ADB), Delta Lake, Azure Data Lake Storage (ADLS), Azure Data
factory (ADF), Azure SQL Database (SQL DB), SQL, ELT / ETL Pipeline Development in Spark based
environment.
Extensive Experience with SparkCore, PySpark, Python, SparkSQL, Scala, Azure Blob Storage.
Experience in Real-Time Data Processing using Apache Kafka/EventHub/IoT, Structured Streaming and Stream
analytics.
Experience with Apache Airflow for ELT Orchestration.
Experience with infrastructure management, Infrastructure as code (e.g. Terraform)
Experience with CI/CD, Version control tools like GitHub, Azure DevOps.
Experience with Version control tools, building CI/CD pipelines.
Experience with Azure cloud platform.

Good to have:
Experience / Knowledge on Containerizations - Docker, Kubernetes
Experience working in Agile Methodology

Qualifications
Qualifications - BE, MS, M.Tech or MCA
Additional Information
Certifications: Azure Big Data, Databricks Certified Associate.