From a Unity Catalog Page to a trusted Genie answer: the dltHub on Databricks walkthrough

“How much do we spend on CI, and how much of it is waste?”

Ask a coding agent and you have a wrong number in minutes. Perhaps it can figure out the spend, but how should it know how you define “waste”?

Getting a trusted answer

An agent might guess that “waste” means failed jobs, but failed jobs means failed tests, which is the entire purpose of testing. Our definition of waste is “Jobs we paid for but whose outcomes we didn’t use. A wasted job is a job that was re-run before it could finish, or got cancelled or timed out.”

So how do we get trusted answers, from day one? By making sure the definitions are verified and consistent before generating the pipelines that grab the data and answer the question.

This is the technical walkthrough of the Private Preview of dltHub working with Databricks Unity Catalog Pages and Genie Ontology. Read the announcement.

Meaning comes from the Unity Catalog

So when an agent answers a business question, where can it find the definition? The definition has to live somewhere centrally, in a place that humans own and every tool reads.

On Databricks, that place is Unity Catalog, where your team writes its definitions down as Pages, governed glossary entries, so this is your single source of truth for meaning. Genie reads them to answer questions about your data, and so does the dltHub agent, which keeps the data pipeline and the answer in sync.

This post shows how dltHub’s agent enables Databricks Genie to self-serve with new pipelines and data in order to answer the questions it is asked, as defined in Databricks Pages.

▶ In the video: where the definition should live (0:17)

The tl;dr: data and meaning need to be in sync

Data without definition is meaningless. Definition without data is a wish.

On Databricks, the definition is a Page in Unity Catalog. In the demo it’s “CI waste,” written by the team that owns the CI budget, with the rule the GitHub API cannot know: a failed job is not waste. Genie reads that Page and cites it. Ask it about CI waste today and it knows exactly what you mean, and cannot answer, because the data isn’t there. That is the wish.

Genie can now fulfil it, by asking dltHub’s data pipeline agent. They share the same source for meaning, governed on Databricks, so there’s no sprawl, no second source of truth, no governance hole.

Formerly, going from requirements to pipeline to data model to data semantics and finally to an answer was human labor intensive. Now Genie provides the requirements, the dltHub agent builds the pipeline into your Databricks platform all the way from ingestion to semantics, and Genie answers from it.

When meaning and data stay in sync on their own, engineers stop babysitting the platform and can instead spend their time on what only people can do: framing new problems to enable solving them, and judging what a good solution looks like.

▶ In the video: what Databricks already gives you, and the CI waste Page (0:51)

What happens, technically

Our example case is “How much do we spend on CI, and how much of it is waste?” The four steps below are the four parts of the demo, and each one links to its part of the video.

1. Getting valid data from the trusted definition.

“Waste” is defined in Databricks Unity Catalog Pages. The dltHub agent finds it there when asked “I want to know how much wasted CI runs on GitHub Actions cost us”. The dltHub AI harness agent raises uncertainty by asking 2 questions, records the complete context as “business rules” serving as requirements, then builds the ingestion pipeline from them and checks the loaded data against the rules. These requirements are later synced back into the Pages. ▶ 3:00

2. Building the model from definition.

The dltHub AI harness agent builds the canonical data model, ontology and metric view from the same business rules. ▶ 5:40

3. Deploy and answer on Databricks.

The requirements and associated pipeline are deployed into Databricks. New terms go out as a Page proposal for people to publish, the pipeline gets deployed and executed, the metric view gets deployed to Unity Catalog, and Genie can now answer the question. ▶ 7:07

4. Maintenance.

If the pipeline breaks, the dltHub background agent reads Databricks logs for dltHub pipelines and offers a fix. Human decides everything. ▶ 10:14

Here’s a diagram of the entire demo flow.

Outcomes of this setup

1. One single source of truth from definition to code. “Waste” is written once, in a Unity Catalog Page, and the pipeline, the data model, the Databricks metric view and Genie’s answer all happen as a consequence.
2. Trusted data before anyone looks at it. The agent clarifies uncertainty upfront and checks the loaded data against the business rules before it builds anything on top. In the demo it found two gaps the Pages definition had not covered and asked, instead of deciding on its own.
3. Trusted answers from day one. Genie answers from a metric view built from your definitions and cites the Page definition it used.
4. Governed by default. Everything the agent builds lands in Unity Catalog with owner, tags, descriptions and permissions, and as reviewable code in git. The jobs run under your access controls, so the agent cannot read or write what the user cannot.
5. Low upkeep. When a load fails, the dltHub background agent diagnoses it in about two minutes and proposes the fix. When a definition changes, the harness agent rebuilds the chain from that point. A person approves every change.

Databricks owns the meaning. dltHub owns the build.

The definition of CI waste is written once, by the engineering team, as a Page in Unity Catalog. The dltHub agent reads it through Genie, turns it into business rules, and builds the pipeline, the model, the transformation and the Databricks metric view from those rules, all as code in one repo. A Lakeflow job deploys it into Unity Catalog. Genie answers from the metric view and cites the Page. When a load fails, a dltHub agent finds the cause and proposes the fix, and a person reviews it.

The definition goes in at the top and comes back out in the answer. It is not locked in one agent conversation. It lives in the data, and nobody retypes it in between.

▶ In the video: the trusted-answer loop (9:31)

Get access

The Private Preview is open to dltHub and Databricks customers. If you run Databricks and want this on your own definitions, contact us to get access.