Azure DataBricks Engineer

About the Company :

Headquartered in California, U.S.A., GSPANN provides consulting and IT services to global clients. We help clients transform how they deliver business value by helping them optimize their IT capabilities, practices, and operations with our experience in retail, high-technology, and manufacturing. With five global delivery centers and 1900+ employees, we provide the intimacy of a boutique consultancy with the capabilities of a large IT services firm.

Role: Azure Databricks Engineer

Experience: 5+yrs

Skillsets: Azure + Databricks + Unity Catalog + Python+SQL

Work Location: Hyderabad/ Pune/Gurgaon/Bangalore

Job Description

Job Summary:

We are seeking a highly skilled Azure Databricks Engineer to join our growing team. In this role, you will be responsible for managing and integrating Databricks services with Azure to build and maintain robust data pipelines. You will play a key role in ensuring our data infrastructure is efficient, scalable, and secure.

Responsibilities:

• Manage Databricks clusters, including provisioning, configuration, and optimization.
• Manage Databricks users in workspace and services used by them such as notebooks, sql, workflows, mlflow etc.
• Integrate Azure services like Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics with Databricks for a comprehensive data processing solution.
• Write high-performance Spark code for data manipulation and analysis.
• Monitor and troubleshoot Databricks jobs and pipelines for performance and stability.
• Monitor usage of Databricks workspaces and generate reports.
• Collaborate with data analysts, data scientists, and software engineers to design and implement data solutions.
• Stay up-to-date on the latest advancements in Azure and Databricks technologies.

Qualifications:

• Proven experience working with Microsoft Azure and Azure Databricks.
• Good understanding of data warehousing and data lake concepts.
• Experience with Apache Spark for large-scale data processing.
• Proficiency in programming languages like Python (PySpark).
• Experience with SQL and relational databases.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Ability to work independently and as part of a team.

Benefits:

• Competitive salary and benefits package.
• Opportunity to work on challenging and impactful projects.
• Collaborative and supportive work environment.
• Continuous learning and development opportunities.