Azure Big Data Engineer

Data Engineer / Sr. Data Engineer - Azure

Chennai/Bangalore/Gurugram

As a Big Data Engineer - Azure, you will build and learn about a variety of analytics solutions & platform, data lake, modern

data platforms, data fabric solutions etc. using different Open Source, Big Data and Cloud technologies on Microsoft Azure.

More specifically, you will:

• Design and build scalable & metadata driven data ingestion pipelines (For Batch and Streaming Datasets)

• Execute high-performance data processing for structured and unstructured data, and data harmonization

• Schedule, orchestrate, and validate pipelines

• Design exception handling and log monitoring for debugging

• Make tech stack and tools related decisions

• Collaborate with business consultants, data scientists, and application developers to develop analytics solutions

Required Experience, Skills & Competencies:

• 4 to 8 years of total IT experience with 2+ years on big data engineering and Microsoft Azure.

• Strong Hands-on experience in implementing Data Lake with technologies like:

• Mandatory: Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
• Optional: Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview.

• Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java.

• Some experience of using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search.

• Good Understanding of different file formats like Delta Lake, Avro, Parquet, JSON and CSV.

• Experience of working in Agile projects and following DevOps processes with technologies like Git, Jenkins & Azure

DevOps.

• Good to have:

• Experience of working on Data Lake & Lakehouse projects
• Experience of building REST services and implementing service-oriented architectures.
• Experience of supporting BI and Data Science teams in consuming the data in a secure and governed manner.
• Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE)