Azure Databricks Administrator - Cloud Infrastructure & Data Engineering (11-16 yrs)

Job Summary :

We are seeking a highly experienced and skilled Azure Databricks Administrator to join our team. The ideal candidate will have a deep understanding of Azure Databricks, cloud infrastructure, and data engineering principles. You will be responsible for the design, deployment, and management of our Azure Databricks environment, ensuring its reliability, security, and performance. This role requires a strong technical background, excellent problem-solving skills, and the ability to lead and mentor junior team members.

Responsibilities :

Azure Databricks Environment Management :

- Design, deploy, and manage Azure Databricks clusters, workspaces, and related resources.

- Optimize Databricks configurations for performance, scalability, and cost-effectiveness.

- Monitor Databricks cluster health and performance, identifying and resolving issues proactively.

- Manage user access and permissions, ensuring security and compliance.

- Implement and maintain Databricks security policies and best practices.

Infrastructure as Code (IaC) and Automation :

- Implement and manage infrastructure as code (IaC) using tools like Terraform or Azure Resource Manager (ARM) templates.

- Automate Databricks cluster provisioning, configuration, and maintenance.

- Develop and maintain scripts for automating routine administrative tasks.

Data Engineering Support :

- Collaborate with data engineers and data scientists to support their Databricks workloads.

- Provide technical guidance and support for data ingestion, processing, and analysis.

- Troubleshoot Databricks-related issues and provide timely resolutions.

- Optimize data pipelines and workflows for performance and efficiency.

Security and Compliance :

- Implement and maintain security best practices for Azure Databricks.

- Ensure compliance with data governance and security policies.

- Monitor Databricks logs and audit trails for security incidents.

- Implement and manage encryption and access controls.

Performance Tuning and Optimization :

- Monitor and analyze Databricks performance metrics.

- Identify and resolve performance bottlenecks.

- Optimize Databricks configurations and workloads for efficiency.

- Implement and manage caching and indexing strategies.

Disaster Recovery and High Availability :

- Implement and manage disaster recovery and high availability solutions for Azure Databricks.

- Develop and test backup and restore procedures.

- Ensure business continuity and data resilience.

Collaboration and Communication :

- Collaborate with cross-functional teams, including data engineers, data scientists, and IT operations.

- Communicate effectively with team members and stakeholders.

- Provide technical documentation and training.

- Lead and mentor junior team members.

Required Technical Skills :

Azure Databricks :

- Extensive experience in deploying, configuring, and managing Azure Databricks environments.

- Deep understanding of Databricks architecture, components, and features.

- Experience with Databricks clusters, workspaces, notebooks, and jobs.

- Proficiency in using Databricks APIs and command-line tools.

Azure Cloud Platform :

- Strong understanding of Azure cloud services, including compute, storage, networking, and security.

- Experience with Azure Resource Manager (ARM) templates and Azure CLI.

- Knowledge of Azure Active Directory (AAD) and role-based access control (RBAC).

Infrastructure as Code (IaC) :

- Proficiency in using IaC tools such as Terraform or ARM templates.

- Experience in automating infrastructure provisioning and management.

Scripting and Automation :

- Proficiency in scripting languages such as Python, Bash, or PowerShell.

- Experience in automating routine administrative tasks.

Data Engineering :

- Understanding of data engineering principles and best practices.

- Experience with data ingestion, processing, and analysis.

- Knowledge of data warehousing and data lake concepts.

Security and Compliance :

- Strong understanding of security best practices for cloud environments.

- Experience with implementing and managing security policies and controls.

Monitoring and Logging :

- Experience with monitoring and logging tools such as Azure Monitor, Log Analytics, or Grafana.

- Ability to analyze logs and metrics to identify and resolve issues.

Networking :

- Strong understanding of networking concepts and protocols.

- Experience with Azure Virtual Network (VNet) and network security groups (NSGs).

Preferred Skills :

- Experience with Apache Spark and related technologies.

- Knowledge of big data processing frameworks such as Hadoop or Kafka.

- Experience with data governance and data quality tools.

- Experience with CI/CD pipelines and tools.

- Azure certifications (e.g., Azure Solutions Architect Expert, Azure Data Engineer Associate).

Soft Skills :

- Strong problem-solving and analytical skills.

- Excellent communication and interpersonal skills.

- Ability to work independently and as part of a team.

- Strong leadership and mentoring abilities.

- Ability to learn and adapt to new technologies.

- Strong attention to detail and a commitment to quality.

- Proactive and self-motivated.

Education : Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Benefits :

- Competitive salary and benefits package.

- Opportunity to work on challenging and innovative projects. -

- Collaborative and supportive work environment.

- Opportunities for professional growth and development. -