One runtime, one agentic context, end to end with dltHub Transformations

Context has always been the bottleneck.

Today's stacks split ingestion, transformation, orchestration, metadata, and access into separate runtimes and while the data flows end-to-end, context doesn't. In order to make this work, these tools rely on re-adding the context agents need at every boundary.

That fragmentation creates invisible operational seams.

• the orchestrator schedule knows when data is produced but ingestion doesn't
• ingestion knows schema drift but transforms don't
• transformation knows how data joins but the semantic layer doesn't
• semantic layers know metrics but not which ingestion source the data came from
• AI systems see what the table is for but not if the pipeline to fill it already ran today.

Agents break at those seams.

dlthub runs ingestion, transformation, data quality, lineage, metadata propagation, verification, visualisation and agent introspection inside the same execution context:

• same Python process
• same auth
• same runtime
• same metadata graph
• same verification loop

That changes what agents can reliably do.

Most transformation tools weren't going to get us there. Because, they were built for humans working on an island called transformation. Ingestion happens on one island. The transformation model lives on another. The orchestrator sits on a third. The architecture diagram and documentation in yet another tool. Each island has its own metadata format, its own credentials, its own deployment story, its own view of what a column means. The bridges between them are JSON build artifacts and ticket queues. Often, they are out of sync.

What's worse, throughout the data movement chain, meaning is added to data which is then lost before it reaches the consumer - we capture tribal knowledge as business rules, we map reality into data, and then the end user should somehow know how to map back. We call this data literacy and it's still an unsolved problem despite decades of organisational attempts.

Picture this: Your data team uses tribal knowledge to give data meaning and put it on the cave wall. But your business team doesn't have the same tribal knowledge to decode.

The end to end agentic experience needs end-to-end tools. Same metadata flowing from logical modeling to ingest to canonical to projection to agent answer. Same Python environment. Same auth. Same lineage. Composable stacks created flexibility, but also fragmented execution context and metadata ownership. The architecture has to start from "this is one system" instead of "these are seven systems we glue together."

For the sake of clarity I will start using the word "ontology" as "tribal knowledge & business context relevant to our data that LLMs can use". It's essentially docs about what the data represents - your business context and rules documented somewhere so agents can read it, curated by a human for precision. By writing out the ontology for LLMs, they are able to understand how the business world connects to the data and they are able to work with, curate and answer based on this data.

The end-to-end agentic experience: an LLM that reasons about your business with the context a senior analyst would have

The agentic data stack means talking to an agent that knows your data well enough to answer business questions correctly, with the same context a senior analyst would have. An agent that reasons about your business because it has a faithful, machine-readable model of how the business actually works and how the world works: a business context, tribal knowledge, ontology.

But much deeper than that, when the tooling has a single context, it enables a LLM to cross between stakeholder request, data model, and data source, and build its own stack - either ad hoc as queries, or better - as a stable and curated canonical model in which business definitions are written in code and validated by humans.

Picture this: By documenting your tribal knowledge, the LLM can use it to give business meaning to data, and also map the data back to business questions.

That's the end to end agentic experience. You ask, the system answers, having understood, not just retrieved. LLMs are finally good enough that the bottleneck moved from execution to understanding, or business context curation. With "understanding" as the bottleneck, the answer lies not in better transformation tooling, but in knowledge management and semantic continuity.

What it takes: a clean canonical model, a business ontology, and metadata flowing end-to-end

Two things have to be in place at once to make the agentic stack work end to end. A clean data model (canonical model) with its business ontology and a good end to end metadata flow that enables semantic continuity, or single-context tooling. This enables an agent to analyze the business data, understand it and even act on it in agentic workflows.

Imagine this: Semantics enables retrieval, Ontology enables understanding and action.

A clean canonical data model with an ontology is a virtual knowledge graph (actually, more) for your LLM - every business concept has one place to live, with a stable definition, so there's no doubt what you're asking about.

Metadata flowing end-to-end, or semantic continuity ensures the model is aware of the dependency and state across layers. If it's a primary key in canonical, the agent can put a data quality check on the field pre-ingestion. Whatever ingestion knows about the data, makes it to where the agent reads it, instead of being stripped at every layer boundary.

When we started building a data model with an LLM in the loop, we realized something that surprised us: capturing the ontology, the tribal knowledge senior modelers used to keep in their heads, is not hard. You don't need a six-month engagement with a data architect. You need a structured conversation with someone who knows the business, and a tool that turns that conversation into something the rest of the stack can read. Something like twenty questions are enough to get started, or arguably excessive.

The interesting consequence is that the focus of the work shifts. We used to think of data work as transforming data. The new framing is giving it meaning.

The old workflow had humans implementing transformations and tools supporting them: better SQL editors, better testing, better orchestration. All of it for the human at the keyboard.

The new workflow has agents implementing what humans tell them: they read the meaning, they write the SQL, they verify the output, they extend the data model when a new question shows up. They need a different kind of support.

Agents are good at things humans are bad at. Holding ten thousand columns in working memory. Tracking which entity a field came from. Surfacing where the model is uncertain rather than smoothing it over. Applying the rules the team agreed on, every single time.

They're worse at things humans are good at: knowing what should be true, what matters, when a number "feels off," when a relationship the data suggests is actually wrong. The split is clear: agents do the volume, humans do the judgment. The stack has to talk end-to-end so that the volume work and the judgment work meet in the same artifact.

In our current AI Workbench, this is already working in development workflows - ask about an entity that isn't yet represented and the agent inspects the source APIs, adds the endpoints, pulls the data, extends the canonical, projects the answer, the entire loop. Of course, this happens during a development workflow for a data person, not live with business users - you wouldn't want to have complex problems solved by guesswork.

That's the move from team transform to team outcome. Team transform was a craft team that took requirements and produced models. Team outcome doesn't write models — it ensures the agent can produce the right outcomes from the logical model that exists, and curates into precision the cases where the agent's first attempt isn't good enough.

What stays human is the accuracy of clarification - Humans speak in high semantic language, while code is highly accurate. To bridge the semantics to accuracy gap, humans act as a precision layer that ensure the relationship between code and semantics is correctly implemented. An example: when the agent merges HubSpot contacts with Luma event guests on email to build a lead table, which name wins if they differ? In our case, the HubSpot one, because HubSpot is our CRM source of truth. That rule belongs in code and it's the kind of precise detail that would normally be easily ommited or confused in a human language prompt.

Bolted-on meaning starts too late: semantic layers add it after the context has been stripped

Semantic layers, BI chatbots, agent sidekicks — they're all the same shape. Meaning bolted on at the end of a pipeline that already threw the context away.

Semantic layers come closest to trying. They sit on top of the warehouse and add back a small amount of meaning: names for metrics, definitions of joins, calculation paths. That's enough to let an end user type a question in natural language instead of clicking through a menu. It's not enough to let an agent reason about a business, because the semantic layer talks about the data model, not about the business.

A semantic model defines how to retrieve business-named metrics from the data model. The data model is a subset of your data. Your data is a recorded subset of facts about your business. Your business is a subset of businesses from your vertical that do something similar. This vertical operates in a subset of "world rules" in which its actions and decisions mattter.

At its core, the semantic model adds back to the data model a small amount of semantics that were discarded during the construction process. The vast majority of meaning is lost and the data literacy problem remains - if you only know about your business, but do not understand what the data model measures, getting your metrics retrieved won't necessarily help.

What do LLMs do with a semantic layer that defines names, table joins, metric calculation paths of a data model? They enable end users to get data in natural language instead of picking them from a menu. That's nice, but lacks substance. The LLM cannot interpret the data correctly because they still don't know about the business this data model belongs to.

Without understanding what your data means, the LLM will lack the data literacy, meaning it will not be able to answer how to use the data to operate in the world.

A hallucination engine by design:

• Your semantic layer describes a small part of data.
• Data is a small part of your business.
• The business is a small part of the world we operate in.
• But you're asking the agent how to operate your business, in the world, based on data.

In order to make that work, an agentic data stack cannot be built as an afterthought. An agentic data stack has to start with the beginning: your business, the goals you are trying to achieve, and implicitly the definition of the data you want to measure, and go from there.

Let's look how agentic memory for sparse or unstructured data is automatically built - An ontology is applied to data. The result is stored in a graph database as a knowledge graph. The graph now contains the ontology relevant to the data space packed together with the data points.

In the classic business intelligence stacks, our data is neither sparse nor unstructured - it's tabular. Here, to create the same AI memory mechanism we simply create a knowledge graph, for instance as a separate markdown file, that describes the tables and their relationships (and more, but later). The agent can first inspect the graph and then decide what query to write to retrieve the data it wants to know.

If we then put the data in a "canonical" shape, it allows us to do two things: First, precisely create and capture the definitions of "how" that should happen. For example, we create a customer table by defining in code how should users between 3 platforms be merged, with precision. Second, it creates a clean knowledge graph for our data. This reduces any possible semantic confusion, and creates a "truth layer" it can check instead of leaving it to hallucinate rules. Such, the agent reports [more] honestly when a question is out of scope instead of confidently lying to you.

Imagine this: An agentic data stack has to be world-aware by design.

• The agent knows the world (has access to search, etc)
• We know and define our business (Ontology captured from business talk)
• We tell the agent what data sources available.
• It designs its own virtual knowledge graph/canonical data model to answer "what should the business do to have better outcomes in the world"

Beyond retrieval and data usage, an LLM agent is actually able to do much more.

• It can infer a business ontology from questions, web searches, or from your org knowledge.
• It can then create a logical canonical data model (theoretical, ideal model, if we were not constrained by real data availability).
• It can then search the available sources and create REST API, db or file pipelines to self serve with data to create a canonical data model.
• It understands where data belongs in a business and how to build a stack on-demand for what it might need (don't build it on demand, you want to have a human review precision in code implementation).
• it understands how to reason over the data in the context of the business of the world.
• It can basically act as an end to end "data team prosthetic", from analyst to architect and engineer, for a real human who ensures correctness.

This essentially adds full stack data skills, even to non developer humans (recently chatted to a freelancer who has serious test harnesses so he doesn't have to read any code, only validate correctness of data flow).

With a semantic layer, you enable your data team to share their work outcomes faster. With an agentic data stack that uses a canonical model and ontology, you enable your data team to stop pushing the cart and start directing it - outcome managers.

dltHub Transformation is one Python decorator @dlt.hub.transformation away from end to end implementation.

(transformation docs)

Giving meaning to data is an incremental process. At ingestion, data starts with minimal metadata. For example, the world's interchange format, JSON, doesn't have a timestamp time and instead relies on strings and conventions which may or may not have been followed by the data producer.

The first enrichment step usually happens before ingestion in tools like dlt: Schema is discovered, and data types are inferred. This grants LLMs the first level of information - what the data contains, what format it's in, etc. dlt also includes checks a LLM can run like "is this column fully populated?", or "does it meet the conditions to be a primary key?"

The second enrichment step is giving the data meaning - we usually start from a logical data model or business canonical model we want to build, and then we label the data we have with the right taxonomy: A hubspot contact is an account, a Luma event attendee is an account, so you probably put them in the same table. Of course, to do this step you need to know

1. What data do we have? (structural info from dlt)
2. What business concepts do we want to represent? (business requirements)

And finally, if you're already using dlt, you want to continue to solve the problem, instead of stopping, bootstrapping a transformation project, adding credentials, local/prod setup, copying over agentic context, changing your own context and possibly even handing over to another team member.

That's where dlthub Transformations comes in, it is the engine that compiles the canonical from raw data and the ontology, runs transforms over it, and exposes the lineage and metadata the agent needs to reason about its own outputs.

What it does, concretely:

• Python or SQL transforms in the same Python process as ingestion. Same auth, same configuration, same observability. No second tool to deploy or coordinate.
• Engine-agnostic execution. The same transform code runs on Snowflake, BigQuery, Postgres, DuckDB, or filesystem (Parquet/Iceberg/Delta via DuckDB). Develop locally on DuckDB, deploy to Snowflake, no rewrite.
• Schema-aware joins. When dlt normalized nested data into parent/child tables, it knew the join keys. T uses them — dataset["users"].join("users__orders") works without an ON clause.
• Lineage propagated. _dlt_load_id carries from ingest through every transform layer. The agent can ask which load batch produced this row and get an answer in the same Python session.
• Ontology-driven canonical generation. Using the AI workflows, data modeling isn't architected by hand; it's bootstrapped from the ontology over the source schemas and polished by your data team. When sources or requirements change, the ontology updates, which can update the canonical.
• Same-session introspection. dataset["customers"].df() works in the same Python process the transform ran in. The agent verifies its own outputs without parsing build artifacts or opening a separate warehouse connection.
• Agent extends, system extends. When a question requires data that isn't yet ingested, the agent searches the context library (dlthub maintains 10,000+ REST API configurations), sets up the ingestion, runs it, extends the ontology mapping, recompiles canonical, projects the answer.

The thing that's qualitatively different from a separate-tool stack is what happens in the seam between transform and verify. The metadata is in your hands, in the same Python session you're running. The agent's in the same place. There's no impedance between "the agent ran the transform" and "the agent verified what came out." It's one motion.

dlthub transformations is a transformation framework that together with the AI Workbench, the runtime, the canonical compilation, the ontology toolkit, and dlt itself underneath make end to end agentic work possible in one chat session.

In practice: how this looks at Navit

Navit is a classic case. Two years ago they stood up a first-generation pipeline with an affordable contractor: dlt for ingestion, Airflow for orchestration, a SQL transformation layer on top orchestrated in dbt core and Metabase on top. It worked well enough to unblock the business.

By year two, the tech debt had compounded. SLAs plateaued in the 80–90% range. The data model grew in tables over time. The mapping between data and business meaning lived partly in code and partly in the contractor's head.

Image: Most data stacks reach a delivery plateau before they clean up the stack or extend the team.

The default move for a company in Navit's position is to hire a data team and kick off a greenfield rebuild. New salaries, a long recruiting cycle, and a bet that the people you hire stay long enough to pay it back.

For Navit, dltHub Forward Deployed Engineers ran the AI Workbench ontology-driven transformations toolkit against their existing pipelines. The toolkit reverse-engineered the SQL into a draft ontology, consolidated stray tables and fields into a small set of canonical concepts (Person, Account, Interaction, Deal, Product Event), generated a clean transformation layer from the ontology, moved execution onto the dltHub runtime (retiring the Airflow setup and its operational overhead), and lit up Chat-BI on the same semantic model that powers the dashboards.

The code came out clean and explicit, so Navit's team could validate it.

The numbers: SLA from ~85% to 99%+. Time-to-new-metric from days to hours. Zero new hires. Cost slightly down.

The most consequential change wasn't either of those. It was that the meaning of the data finally lives in a versioned ontology Navit owns, instead of as a bus-factor-1 dependency on one contractor. Adding a field, adjusting a metric, onboarding a new HubSpot property is a small, well-scoped change anyone on the team can make.

And Chat-BI behaves like an analyst with context, not a text-to-SQL guessing about business context.

Navit's "year two" problem is the default state of every mid-market data stack built in the last two years. HubSpot, an operational DB, a hand-maintained T-layer, SLAs in the 80s, and an open req for a senior data engineer that's been sitting on someone's desk for three months. If that's your stack, this is your migration.

What comes next

Look at how you actually work today: you're typing into a chat box.

Single context agentic data engineering is an architecture you can try today, not a roadmap promise for next year. Start by running this command in your CLI: uvx dlthub-start@latest

Read more about dlthub transformations on our docs.