Databricks Platform Engineer

Below is the detailed JD -

Azure Platform Engineer (Databricks):

Platform Design:

Define best practices for end-to-end databricks platform.

Work with databricks and internal teams on evaluation of new features of databricks (private preview/ public preview)

Ongoing discussions with databricks product teams on product features

Platform Infra:

Create new databricks workspaces (premium, standard, serverless) and clusters including right sizing

Drop unused workspaces

Delta Sharing: Work with enterprise teams on connected data (data sharing)

User Management

Create new security groups and add/delete users

Assign Unity Catalog permissions to respective groups/teams

Manage Quantum Collaboration platform sandbox for enterprise teams for ideation and innovation.

Troubleshooting Issues in Databricks

Investigate and diagnose performance issues or errors within Databricks.

Review and analyze Databricks logs and error messages.

Identify and address problems related to cluster configuration or job failures.

Optimize Databricks notebooks and jobs for performance.

Coordinate with Databricks support for unresolved or complex issues.

Document common troubleshooting steps and solutions.

Develop and test Databricks clusters to ensure stability and scalability.

Governance

Create dashboards to monitor job performance, cluster utilization, and cost.

Design dashboards to cater to various user roles (e.g., data scientists, admins).

Use Databricks APIs or integration with monitoring tools for up-to-date metrics.

Establish alerts for performance degradation or unexpected resource usage.

Ensure dashboards are user-friendly and provide actionable insights.

Regularly review and update dashboards based on user feedback and changing requirements.

Security:

Fix Issues Reported by Databricks Security Compliance Report

Analyze security findings from Databricks compliance reports.

Apply necessary configurations or updates to address security concerns.

Track and verify the effectiveness of applied security measures.

Work with cyber on any security changes that needs to be implemented.