Technical Lead-ADF/spark/Kafka

Role

Sr.Data Engineer

Years of Experience 5-7 Yrs

Preferred Location PAN INDIA

Shift Timing (IST) 11:00 AM - 08:30 PM

Short Description

Develop batch and real-time streaming data pipelines using Python, Spark/Scala, Kafka and Cloud storage on the Azure Cloud platform, as well as implement and integrate APIs using Microservice architecture in SpringBoot/Java

Engagement Duration (in Months) 18

Engagement & Project Overview

IDStrat Transformation and ClinicalInsights

Primary Responsibilities

Build batch and real-time streaming data pipelines using Python, Spark/Scala, Kafka and Cloud storage on Azure Cloud platform

End-to-End ownership, implement to production deployment

Schedule ETL jobs, monitor the pipeline, resolving the issues

Contribute in designing the data solutions including integration, transformation and loading

Must Have Skills

Should have minimum 5-7 years of experience in Data Engineering and ETL

Minimum 2 years of experience in building data solutions on Azure Cloud

Experience in developing batch and real-time streaming data pipelines using Python, Spark/Scala, Kafka and Cloud storage on Azure Cloud platform

Hands on experience in implementing and integrating APIs using Microservice architecture in SpringBoot/Java

Experience in Azure Data Factory to streamline orchestration and data movement tasks and integrate with Databricks and optimize Spark ETL jobs

Experience in integrating, transforming, and consolidating data from various structured and unstructured data

Experience with Azure services such as ADLS, Databricks, AKS, Azure EventHub, CosmosDB

Strong hold on SQL and NoSQL databases

Translate complex functional and technical requirements into detailed design

Expert in CI/CD including automating data pipelines, monitoring data processes, and managing deployments

Propose best practices/standards

Excellent communication skills.

Nice To Have Skills

Snowflake