OSB Group - Azure Data Engineering Manager - Python/PySpark (12-17 yrs)

Job Description :

- Overall 12-15 years of IT experience preferably in Data and Analytics implementation.

- 7-8 years of Data engineering (Both On-prem and Azure) delivery experience including Architecture / analysis / development / configuration/ deployment.

- 4-5 years of implementation experience in Azure Data engineering projects

- Mandatory to have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end Big Data solutions on Azure Cloud.

- Expert in Python/SQL/SparkSQL/PySpark

- Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, Data bricks, purview and Azure App Service is required.

- Extensive hands-on experience implementing data migration and data processing using Azure services : Serverless Architecture, Azure Storage, Azure SQL DB/DW, Data Factory, Azure Stream Analytics, Azure Analysis Service, Databricks Azure Data Catalog, Cosmos Db, Azure Functions, ARM Templates, Azure DevOps, CI/CD etc.

- Expert in at least one distributed data processing framework : Spark (Core, Streaming, SQL), Storm or Flink, etc.

- Expert in distributed messaging and ingestion frameworks (Kafka, Pulsar, Pub/Sub, etc) and good to know traditional ETL tools like Informatica, SSIS, etc

- Should have worked on any NoSQL solutions like Mongo DB, Cassandra, HBase, Cosmos DB etc, or Cloud-based NoSQL offerings like DynamoDB, Big Table, etc.

- Good Exposure in development with CI / CD pipelines. Knowledge of containerization, orchestration, and Kubernetes engine would be an added advantage.

- Designing and building of data pipelines using API ingestion and Streaming ingestion methods.

- Knowledge in Azure Databricks, Azure IoT, Azure HDInsight + Spark, Azure Stream Analytics, Power BI is desirable.

- Strong problem solving and troubleshooting skills.

- Experience in working in a fast-paced agile environment.

- Proven experience in People management, hiring and capability building.

- Strong experience in common data warehouse modelling principles including Kimball, Inmon.

- Experience with data visualization tools such as Power BI.

- Experience in Streaming and batch architecture (e.g., Kafka, Kafka streams, Spark, Flink)