Building pipelines has become easy. Knowing if they work correctly is now easy, too
We are releasing our new observability UI, the interface to the dltHub context graph, to General Availability. Find out how this will help you solve missing data when everything looks right at the surface from your pipelines.
Adrian Brudaru
Co-Founder & CDO
5 min read
Your agents use the data that arrived. But did all of it arrive?
AI agents and chat-based analytics tools rely on the data your pipelines deliver. If a run finishes successfully but loads incomplete data, an agent can give the business a wrong answer before the data team knows anything is missing.
A successful pipeline run doesn’t always mean you have the data you expected. An API can return fewer records, source data might be late, or a pipeline might load only the first page of results without flagging missing data.
Nothing in the logs immediately suggests a problem, so you start deep diving on the data: how many rows arrived, when and what's their last timestamp, and did anything else change along the way?
Even when the data turns out to be perfectly fine, establishing that truth can take serious time, or it’s a lose-lose scenario where there’s an actual deeper issue at play, that may derail a day’s work.
Getting a pipeline into production has become easy. Understanding its health has now become even easier.
Today, we are releasing the new observability UI, the interface to the dltHub context graph, to general availability.
With new pipeline views and configurable alerts in the new UI, dltHub brings more of that information into the everyday work of running pipelines. You can track load volumes and duration over time, inspect individual runs down to their tables, and receive failure notifications in Slack or email.
The work that follows the successful deployment
Coding agents are making it easier to build pipelines for sources that previously required substantial custom work. The dltHub AI harness gives engineers and their agents the tools and context to create and deploy pipelines with far less human involvement required.
But every new pipeline also brings something to look after. Sources change, credentials expire, and a load that made sense during development can behave differently a month later.
As teams build more pipelines, they need to keep track of their health. Taking every deeper question back to the person who wrote a pipeline is inefficient at best and risky at worst. That person may remember why a Friday load is smaller, or why one table takes twice as long as the others. But teams and their agents need a way into that knowledge too.
The dltHub context graph solves this problem. A useful run history with pipeline traces and server logs, easily accessible to engineers via UI and to agents via MCP, gives the team a shared starting point. Combined with dltHub background agents, it helps someone unfamiliar with a pipeline investigate issues easily and delegate follow-up work to a coding agent.
A ‘successful’ run, and the story behind it
Consider a pipeline that finishes without errors but loads only the first page of API results. From the execution status, everything looks fine. For someone using the data, most of it is missing.
Comparing row counts across runs can help you spot changes worth investigating. A sudden drop might mean some source data hasn’t arrived; an unexpected spike could mean the source has sent records again. Neither necessarily causes the pipeline to fail, but both give you a reason to check what changed.
That’s a much more useful starting point than “the dashboard looks a bit off,” and gives a more informed perspective on the data before the CFO sees it in a dashboard.
Six data problems, and how to investigate them with dltHub
These examples involve two different signals: a failed run, which can trigger a notification, and an unexpected load, which you can investigate through the new views.
| Pipeline problem | What you see in dltHub | What you do next |
|---|---|---|
| Source not ready. The upstream export is still being written when the schedule fires. The run is green, with 40% fewer rows. | The rows chart on the Pipelines page dips. The run page shows which pipeline came up short, and the pipeline run shows which table. | Re-run from the web UI once the export has landed. |
Empty load. The data selector points at data.results, but the API returns a bare array. Green run, 0 of 6 rows. | A green run with zero rows for that table. | Let your agent read the logs on the same run page, fix the selector, redeploy. |
Paginator stops at page one. It reads next from the response body; the API puts it in the Link header. 100 of 935 rows. | The row count sits at exactly one page, run after run. | Fix the paginator, then reload the missing pages. |
The cursor has changed. A refactor swaps updated_at for created_at, and changes to existing rows stop arriving. | Rows step down from the first run after the deploy and stay down. | Compare runs before and after the deploy, then roll the change back. |
| The source re-sends everything. A reset upstream turns an incremental load into a full one. | Rows, bytes and duration jump on the same run. | Open the run to see which table grew. |
| The run fails. Expired credentials, a crashed container, a timeout, a memory limit, a dependency that won't install. | An email or Slack alert to the owners or the whole team, with the error and a link to the run and its logs. | Click through to the run. Add job-inspector and a diagnosis follows in Slack within minutes. |
What’s new in dltHub
New features doing the work in dltHub for data observability:
- The overview page. A simple overview into the health of your workspace, highlighting all job runs that need attention.
- The pipeline view. For each pipeline: success rate, rows loaded and duration over time, with charts for runs, rows, bytes and duration.
- For each run: every pipeline the job executed, with its row count and status.
- For each pipeline run: every table it loaded, with rows, bytes and duration, next to the logs.
- Alerts. Under Settings > Alerts: alerts on failed runs, and on successful ones if you want them. Email to the workspace owners or to every member, posts to one or more Slack channels, scoped to all jobs or to the pipelines you pick.
- Every alert links to the run and its logs.
John Barlow, the Senior Analytics Engineer at Pinter describes the work this saves:
“Hey guys, thanks for shipping notifications, I've been hacking together stuff like this for years and it's so nice that it just works! I also built a daily digest across all my workspaces using the dltHub MCP. It gives me the visibility to manage by exception, and real peace of mind.”
Get started with better observability
Open Overview and Pipelines in the dltHub web app to explore the new views, then head to Settings → Alerts to configure notifications.
Running dlt yourself? Start a free dltHub trial, run uvx dlthub-init@latest in an existing dlt project and deploy one pipeline with uv run dlthub deploy. Put its first week of runs next to what you have today.
