Solutions Architect - Azure Databricks

As a Databricks Solution Architect, you will play a crucial role in designing, building, and maintaining scalable data pipelines and infrastructure. You will work with a variety of technologies, including Scala, Python, Spark, AWS (Amazon Web Services) services, and SQL, to support our data processing and analytics needs.

Prior experience working for Databricks COE (Center of excellence) would be a huge plus.

Responsibilities: -

• Develop robust data platforms, contributing to the organization's data-driven growth.
• Utilize expertise in Apache Airflow for orchestrating and automating complex data workflows.
• Help the other team members in developing solutions.
• Design solutions on databricks including using delta lake/delta table, data warehouse etc.
• applying best practices while developing the solutions.
• work towards creating reusable components.
• ensure that the pipelines are designed to keep the operating cost low.

Experience: -

• Good hands-on experience in designing, developing, and optimizing scalable large volume pipelines to support the processing of structured and semi-structured data.
• In depth knowledge in SQL, Pyspark, Databricks, delta lake, delta table , Azure data factory, Azure services Python, DevOps & GIT, Relational Databases (Oracle, SqlServer etc.)
• Understanding in designing and implementing data solutions involving real-time streaming technologies, such as Apache Kafka, AWS MSK, Kinesis and Azure Event Hubs, ensuring seamless integration of streaming data into our processing pipelines
• Exposure on the CICD pipelines for the azure and AWS resources
• Possess excellent written and verbal communication skills, with the ability to effectively convey complex analytical findings and tell a compelling story with data.
• Understanding of Finops/ observability patterns, data governance best practices

Location: -

• Any location in India.

Educational Qualifications: -

• Engineering Degree BE / ME / BTech / M Tech / B.Sc. / M.Sc.
• Technical certification in multiple technologies is desirable.

Mandatory Technical skills

• Databricks
• AWS/GCP/Azure
• Pyspark
• SQL

Good to have skills:

• CI/CD (DevOps)