Azure Data Engineer

Job Description: Data Engineer / Sr. Data Engineer - Azure

Mandatory Skills- PySpark, SQL, ADF/data factory, Databricks

As a Big Data Engineer - Azure, you will build and learn about a variety of analytics solutions & platforms, data lake, modern data platforms, data fabric solutions etc. using different Open Source, Big Data and Cloud technologies on Microsoft Azure. More specifically, you will:

• Design and build scalable & metadata driven data ingestion pipelines (For Batch and Streaming Datasets)
• Execute high-performance data processing for structured and unstructured data, and data harmonization
• Schedule, orchestrate, and validate pipelines
• Design exception handling and log monitoring for debugging
• Make tech stack and tools related decisions
• Collaborate with business consultants, data scientists, and application developers to develop analytics solutions

Required Experience, Skills & Competencies:

• 4 to 8 years of total IT experience with 2+ years on big data engineering and Microsoft Azure.
• Strong Hands-on experience in implementing Data Lake with technologies like:
• Mandatory: Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
• Optional: Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview.
• Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java.
• Some experience of using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search.
• Good Understanding of different file formats like Delta Lake, Avro, Parquet, JSON and CSV.
• Experience of working in Agile projects and following DevOps processes with technologies like Git, Jenkins & Azure DevOps.
• Good to have:
• Experience of working on Data Lake & Lakehouse projects
• Experience of building REST services and implementing service-oriented architectures.
• Experience of supporting BI and Data Science teams in consuming the data in a secure and governed manner.
• Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE)