I've noticed that most modern cloud platforms have an interesting feature: they display costs, but in an overly aggregated and complex manner. Sometimes they hide features behind SKU combinations. Databricks has a standard Dashboard for this, as well as the Governance Hub. The Hub displays costs at a high level, across all workspaces:

As a result, an entire industry of FinOps tools has grown around cloud spending. At DAIS 2026, I counted more than 10 similar products. There are solutions for monitoring, cost allocation, anomaly detection, and infrastructure optimization. But each new tool is another interface that needs to be learned, configured, and regularly accessed. I don't have time to learn the new features of Databricks, let alone third-party tools...

I had the idea to try making a widget, but Ilya came up with a more interesting solution. What if we added prices and costs directly to the Databricks interface via a plugin? It sounded crazy, but after a couple of experiments, we got it working.

This is how FinOpsWay was born —a free browser extension that displays costs directly alongside Databricks resources. Ultimately, it complements the Databricks Governance Hub.

Initially, we added a mini-panel with a list of expenses:

Then the ideas started coming one after another. First, we added a display next to the clusters. That was simple. Then, we added the cost when selecting a cluster, which was more interesting because now the engineer could see the actual cost, not just abstract DBUs:

But I was always interested in how much each launch cost. Copying the ID each time and going to the dashboard was too much. But now I can see it right next to each launch:

We gradually added cost display for various objects. At some point, someone asked to add costs for Notebooks. It wasn't difficult to implement, but ultimately, the LinkedIn post about this feature got the most likes. I still don't understand why they need to see it...

What else can you see? We've also added a cost display next to Databricks Apps. In the current Dashboard, they're hidden as a combination of SKU and Billing_SKU and displayed as All-purpose serverless compute:

Also, for fun, we added this to Lakebase Postgres during setup, as well as to branches and databases:

It was a funny effect. Ilya used to think Lakebase was expensive and avoided using it, but after this feature, he changed his mind.

Before launching this product, we discussed at length whether to release it as a paid product, a free product, or a subscription. Unfortunately, any Chrome extension can be downloaded from the browser and the full code can be viewed. Therefore, we decided to make this product free for everyone by activating sponsorship through GitHub . Currently, we have about 250 active users and no donations, so I'm not sure how long this initiative will last. We'd be interested in your opinions on the idea and what features might be missing.

This extension also works on Databricks Free Edition, and you can track the latest updates on the LinkedIn page .

Databricks costs directly within the interface — FinOpsWay extension was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why I built it

I wanted to build an enterprise-scale, self-hosted lakehouse platform with end-to-end governance, using open source tools.

The picture

Six layers, with requests entering at the top and data sitting at the bottom.

The journey, left to right

One table travels from raw files to a dashboard in four steps, and at every step Unity Catalog decides who touches it.

Step 1 of 4: Dagster starts the run

Dagster, the orchestrator, runs analytics_team_job every day at 06:00 UTC, or when an engineer launches it.

The job carries the tag team: analytics, so each of its steps logs in as one of the analytics team's service accounts, never as a person. Dagster's UI sits behind oauth2-proxy, a login gate in front of tools with no login of their own, which lets in the data-engineer and platform-admin roles only, and launching a team-tagged job also needs the team's own role, team-analytics.

Governance check: the analyst holds neither role, so the orchestrator stays closed to them.

Step 2 of 4: dlt lands raw data in bronze

dlt reads the raw CSV files and writes them into bronze as managed Delta tables.

dlt is a library with no login of its own. The ingest step logs in to Keycloak as sa-team-analytics-ingest, with a secret Dagster reads from Vault, and exchanges the token for a Unity Catalog token. dlt reads orders.csv and customers.csv and hands each batch to uc_managed, a dlt destination I wrote, which writes analytics.bronze.orders and analytics.bronze.customers through Spark Connect, the client-server protocol every tool uses to reach the one Spark server. On released Unity Catalog only Spark writes catalog-managed tables, so the destination goes through Spark. Unity Catalog vends the storage credentials for each write, and no static storage key exists anywhere in the compute plane.

Governance check: the ingest account holds the ingestion-bot persona, a named set of grants, so Unity Catalog lets it write bronze and nothing else. op-lake's test script logs in as each identity against the running stack and tries what it should and should not be able to do. Reading gold as the ingest account fails with:

Failed to load table analytics.gold.customer_order_summary (HTTP 403)

Step 3 of 4: dbt builds silver and gold

dbt turns bronze into cleaned silver tables and one gold table for reports.

dbt has no login of its own either. Dagster’s dbt step mints a Unity Catalog token for sa-team-analytics-build, which holds the data-engineer persona, and hands it to dbt as DBT_UC_TOKEN. The dbt-spark profile sets it as the Spark session's catalog token, so every model runs under the build account's grants. dbt builds analytics.silver.stg_orders and analytics.silver.stg_customers, then analytics.gold.customer_order_summary. Every table is a managed Delta table, so Unity Catalog owns its storage path under s3://lakehouse/managed, and dbt's CREATE OR REPLACE works because UC allows it on managed tables.

Governance check: in the same test run, the build account read all 10 orders in bronze, and dbt built silver and gold with 14 models and tests passing and no errors.

Step 4 of 4: Superset shows gold to the analyst

Superset and JupyterLab read gold live, as the person who logged in.

The analyst signs in once through Keycloak at the launchpad. Superset’s connector exchanges the analyst’s Keycloak token for a Unity Catalog token, and the Spark Connect session carries it, so Unity Catalog checks the analyst’s own grants on every query. No serving copy of gold exists: dashboards and notebooks read the table where dbt wrote it.

Governance check: the same analyst, in the same session, asks for bronze, and Unity Catalog refuses.

The tools hold no identity of their own. Every step runs under a named identity, and Unity Catalog checks each one against its grants.

Who enforces it

• Keycloak 26.1: single sign-on and every token. Users, groups and roles live in Postgres.
• Unity Catalog v0.5.0: grants, token exchange (RFC 8693) and storage credential vending. Its own metadata sits in an H2 file, because UC 0.5.x issues 1364 and 1385 break managed Delta tables on Postgres.
• iam-reconciler: turns Keycloak group membership into per-person grants every 60 seconds, and provisions the service accounts declared in personas.yaml.
• Vault 1.18: every service secret, including the service-account secrets Dagster reads at run time.
• Spark Connect: starts with no default Unity Catalog token, so a session without its own token is denied. A pre-shared token behind a TLS proxy (nginx 1.27, port 15003) gates the connection itself.

How a person gets grants

1. An admin adds the person to a Keycloak group, in the admin UI or with the reconciler’s add-member command.
2. Within 60 seconds, the reconciler reads the group’s members from Keycloak.
3. It expands the group’s persona or team grants and writes one grant per person in Unity Catalog.
4. When the person leaves the group, or gets disabled, the next pass revokes the grants.

Result: nobody edits a grant by hand.

Around the data

• Front door: Caddy 2.8 terminates HTTPS for every UI, and oauth2-proxy v7.7.1 gates the tools with no login of their own. The launchpad lets in any signed-in user, the admin console only platform-admins.
• Three network tiers: default for the apps, data for MinIO and Spark, admin for Vault and the socket proxy. Superset and the other UIs never reach MinIO directly.
• Managed tables only: no persona gets an external location, a storage credential or CREATE EXTERNAL TABLE, so nobody points a new table at another table’s files.
• Three published ports: 80 and 443 on Caddy, and 15003 on the Spark Connect proxy.
• Fresh storage credentials: uc-cred-refresher restarts Unity Catalog every 6 days, so it mints a new 7-day storage session, and it reaches Docker only through docker-socket-proxy, which allows restarts and nothing else.

Reference

Tools and versions

Where state lives

A crash, a restart or a recreate loses nothing.

Only docker compose down --volumes wipes this, and it's meant to.

Moving to a cloud

You change .env and server.properties. Models, assets, personas and the catalog layout stay the same.

What this build leaves out

• Lineage and observability (OpenLineage, Marquez), CI/CD for the platform itself, and cloud IaC (Helm, Terraform) are deferred to later phases by design. The test script runs by hand.

Read the code

1. docker-compose.yml, 964 lines: every service and its network tier.
2. infra/iam/personas.yaml, 198 lines: the catalog layout, personas, groups and teams.
3. infra/superset/spark_connect_uc/: the Superset connector.
4. dagster/data_platform/uc_managed_destination.py: the dlt destination writing through Spark Connect.
5. infra/unitycatalog/uc-entrypoint.sh, 163 lines: the storage session minted at every start.

Everything is on GitHub at sathwikbs/op-lake.

Next: open docker-compose.yml at line 938 and read the networks block. It's about 10 lines, and the comments explain all three tiers.

Sources

• op-lake on GitHub (source code)
• Unity Catalog (open source) documentation
• Apache Spark: Spark Connect overview
• RFC 8693: OAuth 2.0 Token Exchange
• Unity Catalog issue 1364 and issue 1385

Built on open source: Unity Catalog, Apache Spark, Delta Lake, Keycloak, Vault, MinIO, Superset, JupyterHub, Dagster, dlt and dbt. Opinions are my own.

op-lake: an enterprise-scale, self-hosted lakehouse with end-to-end Unity Catalog governance on… was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Deploying a Databricks project reliably comes down to three components: code, resources and an execution environment. How well a team manages these three determines whether a project moves from development to production in a controlled, repeatable way, or requires manual intervention at every stage.

This article defines those three components, outlines how a project is meant to progress through development, staging, and production, and explains how Declarative Automation Bundles (DABs) were built to manage that progression.

What’s inside a Databricks project?

A Databricks project is built from three kinds assests.

Code is the logic itself: notebooks, Python wheels, JARs, dbt models, SQL scripts. This is the part everyone thinks of first because it’s the part you write by hand or nowadays an LLM helps with.

Resources are the Databricks objects that let your code run as something managed: Lakeflow Jobs, Spark Declarative Pipelines, the MLflow tracking server and model registry. A notebook by itself is just a file. Attach a job resource to it and Databricks can schedule it, retry it, and alert you when it fails.

Execution environment covers the workspace, the Unity Catalog namespace, and the compute configuration the code runs on. The same notebook can behave completely differently depending on whether it’s running on a single-node cluster against dev tables or on serverless compute against production tables.

How much of each you need depends on what you’re building. A simple report might be one notebook on a small cluster and nothing else. A full MLOps pipeline needs code, several resource types, and a much more deliberate execution environment, because the deliverable is more complex.

The path to production

Code doesn’t jump straight to production. It moves through development, staging, and production, and version control drives that movement: a pull request pushes code from a dev branch into stage, a release pushes it from stage into main. Each branch deploys into its matching environment, and each environment gets its own configuration, not just its own data.

In development, you’re running on a single-node cluster under your own account, against data that’s fine to break.

In staging, the setup gets closer to the real world: serverless compute, and a service principal instead of your personal login, because the job needs to run the way an automated process runs, not the way a person clicking around a UI does.

In production, you’re still on serverless, still under the service principal principle, and now the job runs on an actual schedule against real data for real consumers.

The important is that the same code has to behave correctly under three different sets of rules, and something has to manage that translation every time you deploy.

Three ways engineers did this before Declarative Automation Bundles

There were three ways to orchestrate this journey, and each one has a real cost:

1. Doing it through the UI is the easiest thing to learn. It’s also slow, easy to get wrong because a human is clicking through every step by hand, and not something you can plug into an automated pipeline.
2. Doing it through the REST API or SDK gives you low-level control. You’re also learning a large set of endpoints and classes, and for writing the deployment logic yourself, which usually ends up looking the same for every project you build.
3. Doing it through Terraform is powerful, and it’s what most Databricks administrators already use for infrastructure. It’s also a separate tool with its own syntax and state management, which most data engineers and data scientists never signed up to learn to deploy a pipeline.

None of these three were built specifically for the thing data teams actually need: write a project once, then deploy it to dev, staging, and production with the right settings applied automatically in each place.

What was missing

If you laid out what an ideal tool would do, it would look something like this:

• Store code and every environment’s configuration together, in a format people can read and define, like YAML.
• Define Databricks resources using the same parameters the REST API already uses.
• Keep deployments isolated per user during development.
• Let you override specific settings per environment, like compute size, identity, or schedule, without duplicating the whole project.

Declarative Automation Bundles (DABs)

Declarative Automation Bundles (formerly known as Databricks Asset Bundles) are a tool to facilitate the adoption of software engineering best practices, including source control, code review, testing, and continuous integration and delivery (CI/CD), for your data and AI projects. Bundles provide a way to include metadata alongside your project’s source files and make it possible to describe Databricks resources such as jobs and pipelines as source files. Ultimately a bundle is an end-to-end definition of a project, including how the project should be structured, tested, and deployed. This makes it easier to collaborate on projects during active development.

Bundles are Databricks’ infrastructure-as-code approach to managing projects. The core trigger for using them: multiple contributors, automation, and a CI/CD requirement. Because a bundle is just YAML files versioned next to your source code, it fits naturally wherever infrastructure-as-code already fits.

Four cases stand out:

• Team development. Bundles keep source files organized across contributors, so collaboration doesn’t turn into chaos.
• Faster ML iteration. Training jobs, batch inference, and other pipeline resources get managed as part of the project from day one, following production practices instead of bolting them on later.
• Org-wide standards. Teams can build custom bundle templates with default permissions, service principals, and CI/CD settings baked in, so every new project starts from the same baseline.
• Regulatory compliance. Because a bundle keeps a versioned history of code and infrastructure together, it supports governance and audit requirements in regulated industries.

Files of particular interest include the following:

• databricks.yml: This file specifies the bundle's programmatic name, includes references to the bundle's files, defines catalog and schema variables, and specifies settings for target workspaces.
• resources/sample_job.job.yml and resources/<project-name>_etl_pipeline.yml: These files define the job that contains a pipeline refresh task, and the pipeline's settings. For information about pipeline settings, see pipeline.
• src/: This folder contains the sample pipeline's source files, explorations, and transformations.

Conclusion

• A Databricks project is made of code, resources, and an execution environment. Moving that project into production means running all three correctly across development, staging, and production, with the right configuration applied at each stop.
• The CI/CD journey promotes a project across development, staging, and production, driven by version control, with per-environment configuration.
• The UI, REST API/SDK, and Terraform can each orchestrate deployment, but each has real drawbacks for data engineering CI/CD.
• Declarative Automation Bundles (DABs) co-version code and configuration in YAML and deploy to multiple environments easily.

Useful Links:

• CI/CD workflows on Databricks | Databricks on AWS
• Develop pipelines with Declarative Automation Bundles | Databricks on AWS

What Makes Up a Databricks Project? was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.