Lead Architect - Azure Databricks (8-13 yrs)

Position : Lead /Architect - Azure Databricks

Overall IT experience : 8 to 13 years

Primary Skills : Azure Databricks, Hadoop, Hive, Pig, Impala, ETL, Data warehouse

Secondary Skills : U-SQL, Spark, Scala, Python

Work Location : Bengaluru, Hyderabad

Notice Period : 0 to 30 Days

Essential experience:

- Overall 10+ years of experience Cloud data engineering, out of which minimum of 7+ years experience in Azure Databricks, PySpark.

- Proficient in SQL and must worked on complex joins, Subqueries, functions, procedure and able to perform SQL tunning and query optimization without support.

- Ideally 5 to 6 years of solid experience in SQL queries

- Design, develop, and maintain ETL pipelines using Databricks, PySpark and ADF to extract, transform, and load data from various sources.

- Experience in implementing large & complex hosted solution using Azure (ADF, Azure Data Lake Storage and databricks).

- Optimize and fine-tune existing ETL workflows for performance and scalability.

- Must have experience on working with large data set

- Experience working with batch and streaming data processing (Good to have).

- Ability to develop reliable, secure, compliant data processing systems.

- Work closely with cross-functional teams to support data analytics, reporting, and business intelligence initiatives.

One should be self-driven and work independently without support :

1. Azure Cloud knowledge

A. Integrations (mandatory)

a) Azure Data Factory

b) Synapse

c) LogicApps

d) Storage

B. Messaging system (Mandatory)

a) IoT Hub/IoT Central

b) Event Hubs

c) Reporting

2. DevOps (mandatory) :

- CI/CD pipelines

- ADF pipelines

- Databricks pipelines

- Infrastructure as code (IaC)

3. Data Platform :

- Databricks (Good to Have)

- Modeling

- Metadata notebooks

- Deltalake

4. Others :

- Design documents

- Solutioning

- Python (any one of 3 .4 should aware)

- Pysql

5. Reporting : Power BI