Trusted answers on new data from day one, with dltHub on Databricks

Today we announce a Private Preview of dltHub integration with Databricks Unity Catalog Pages and Genie Ontology. Together, dltHub and Databricks can provide trusted answers on new data from day one.

Where we come from

In May we announced dltHub. dltHub provides the agentic tools and managed infrastructure for teams and AI agents to build and operate data pipelines at scale, on top of any warehouse. Customers run it on BigQuery, ClickHouse, Databricks, DuckDB and Snowflake. At its core is dlt, the open-source Python library used in production by more than 22,000 companies, 9% of the Fortune 2000.

The question we keep hearing

How do I make the business users in my company work effectively with our internal agents? How do I let humans and agents reach any data they need, and still get trusted answers?

When an agent builds a pipeline and answers a business question, whose definition does it use and how do I manage governance?

The trend: data loaded for AI agents is growing very quickly every month

Last month, dlt users and agents loaded data from 53,623 sources for AI agents, RAG pipelines and web scraping. That is 198% more than the month before.

Every one of those sources brings new semantics with it: what a field means, what counts as a metric, which table is the one to trust. Each agent that touches the data has to answer the same question: whose definition do I use?

Companies keep that knowledge in many places. A markdown file in the repo. A Notion page. A Jira ticket. A semantic layer. A gold layer of modeled tables. A platform catalog.

dlt sits in the middle of all of it. Teams and agents load data from any source and build whichever layer fits.

Agents with curated context outperform agents with raw data access

For complex work, agents perform better with a curated layer of company knowledge and data. That means work that:

• combines data from different sources, and
• needs people to trace an answer to its source.

The layer can be a semantic layer or a gold layer of modeled tables. It lives alongside the company's knowledge base and ontology. The progressive AI and data teams we talk to are already building it. Most enterprises have a semantic layer covering a fraction of their data. We expect extending it to become a top priority for data teams in 2027.

From a markdown file to the platform catalog

Because the number of sources and agents grows, the place where definitions live starts to matter.

A small team can keep its definitions in a markdown file. A few sources, a few people, one place to look. That works, and dlt works with it.

Enterprises, and companies that depend on trusted, auditable answers, have more complex needs. They have more sources, more agents and more people who must be able to trust and audit an answer. They want the layer in the platform catalog. Here is why.

1. The people who audit the data already work there

Governance, ownership and audit already sit in the catalog. A definition in a Unity Catalog Page is reviewed where the data it describes is reviewed.

dltHub does not add a second governance layer for context and semantic definitions. A second governance layer is any other place where a definition gets its own owner, its own review and its own access rules, next to the catalog's. Each of the other places creates an additional governance layer:

With the first four, the business definition of "waste" is governed in one place and the data it describes in another. Two owner lists, two review processes, two sets of access rules. They drift apart, and an auditor has to check both.

With a Page, there is one governance layer. The dltHub AI harness reads the Page, and the context file in the repo is written from it. dltHub adds no owner list, no review process and no access model of its own.

2. An answer can be traced

The definition, the tables and the metric view sit side by side in one platform. A person can follow an answer from Genie back to the definition and to the data.

3. The business decides what a term means

People write the definitions in Unity Catalog Semantics Pages. The agent proposes new terms as Page proposals, and people publish them.

Nothing becomes a definition because an agent guessed it.

4. The data matches the definition

The coding agent using the dltHub AI harness checks the dlt pipeline against the definition. It builds the model and the metric view.

The pipeline is Python, version-controlled in git, and every change is a reviewable diff. The cost of governing the context is paid once, at build time, not on every question.

The difference to MCP on raw data

MCP on raw data gives an agent access to the data. It does not give it the meaning. It re-reads the raw source on every question, so the cost shows up per question, and answers can vary from run to run.

Raw access works for demos and small tasks. For answers a company has to trust, the layer needs a home that its governance already covers. dltHub builds the data to match it and feeds what it builds back into the Ontology.

What you can do as a dltHub on Databricks customer from today onward

dltHub customers can write their company knowledge in Unity Catalog Pages. From there:

1. The dltHub AI harness in Claude Code reads the team's Pages through Genie and writes the definitions into a context file in the repo.
2. It checks the dlt pipeline against that file, then builds the model and the transformation.
3. It deploys a metric view to Unity Catalog.
4. New terms go out as a Page proposal for business users to publish.
5. Genie answers from the same definitions your dlt pipelines were built from.
6. A dltHub agent diagnoses failed loads.

The demo uses a real question: how much of our GitHub Actions CI spend is waste? "Waste" is defined once, in a Page. The pipeline, the model, the metric view and the Genie answer all use that definition.

See it

Adrian, my co-founder, walks through the whole flow step by step here. Watch the full technical demo here.

Get access

The Private Preview is open to dltHub and Databricks customers. Contact us at sales at dltHub to get access.

Matthaus Krzykowski
Co-founder and CEO, dltHub