Sr. Engineering Manager - Serverful Compute

P-932

The last 10 years was the decade of the cloud. The next 10 years will be the decade of Data and AI. This market will be over $3T by 2030. In a market this big, the winner(s) will win on the strength of a rich product portfolio combined with strong platform play.

Databricks is uniquely positioned in this market as both a powerful platform built on open source and open standards, and boasting of a rich product portfolio spanning verticals such as SQL, model lifecycle, data engineering, analytics, and observability for data and AI. Databricks has always been a visionary in this space, embracing AI before it was cool. Our platform story is the result of years of investment in a general purpose platform that is driving the hockey stick growth in our user community and customers.

The Compute Fabric team in Databricks powers this rocketship. Compute Fabric is organized into and Serverful Compute is Databricks original Compute offering, still powering most of Databricks revenue at EOY 2024. It s the most sophisticated, powerful, and scalable platform for orchestrating data-centric compute on the planet.

The charter of the Serverful Compute team, aka Clusters Compute Team, or Clusters Team, is to create the most reliable, cost-competitive, and easy to use compute platform for the modern data cloud, which . Towards that goal, we are scaling our platform for the next order of magnitude of growth, while raising the bar significantly on reliability, TCO, capabilities, and performance. We work on complex problems at the forefront of distributed systems, cloud computing, ML infrastructure, and compute.

In serverful, the end user is typically doing two things: (1) using a Databricks product (e.g. jobs, notebooks), and (2) managing compute (clusters) for the product. In serverless, compute is completely transparent to the user and is automatically managed through the user-facing product (e.g. SQL). Although this distinction is orthogonal to hosting model, for historic and business reasons we offer serverful in customer account (customer-hosted) and serverless in Databricks account (self-hosted). Serverful has thus gotten coupled with the customer-hosted offering.

Serverful Compute Team in BLR will fully own the following areas from the above table by end of FY26:

The impact you will have:

What we look for: