One runtime for ingestion, transformation and quality
Schedule all three on the same platform, including the sources your GUI tool has no connector for. The orchestrator you were about to stand up is one you no longer need, and the one you already run keeps working.
Consolidate three tools into one runtime
Ingestion, transformation and data quality run on the same scheduler, including sources a GUI tool has no connector for.
Keep the orchestrator you already have
Already on Airflow or dbt? dltHub runs beside them: one dlt pipeline, one Airflow task per resource, your DAG still driving incremental loads.
Compute that runs only while your job does
Pick a machine size per job. dltHub starts it, runs it, shuts it down and bills only those seconds, so there is no idle cluster between loads and no base image to patch.
You are paying for ingestion twice
Once for the tool, and again for the team keeping the pipelines it cannot reach alive beside it. Only one of those shows up on the invoice.
On the invoice
Per-row or per-connector pricing that grows with the business, and a renewal that gets harder to sign each year.
On the payroll
The engineers writing and maintaining the Python pipelines your GUI tool has no connector for, running alongside it.
- 10+ analysts
- Build and maintain ingestion, where only the Staff Data Engineer could troubleshoot the old connectors.
- Zero new hires
- SLA went from roughly 85% to over 99%. The plan it replaced was three senior hires and a six-month rebuild.
- 66% faster
- Development time, while consolidating 53 data tools toward a target of 16.
Publicly traded, highly regulated
Stellantis
Managed infrastructure
The sources your GUI tool has no connector for
Your agent writes the dlt pipeline that reaches them. dltHub runs it: scheduled, observable, with secrets and regions handled. One runtime covers the ingestion, the orchestration and the operations around both.
The agentic layer
Your coding agent writes the pipeline, through the AI harness and the context catalog.
AI harness →operates ↓
Hard to reach
10,100+ REST API sources, plus any SQL database or filesystem
dltHub managed infrastructure
Runs your pipelines, so you don’t
Your warehouse
Wherever it already is
The pipeline stays plain Python you can read, diff and own. What changes is that you no longer run the infrastructure underneath it.
Scheduled, and deployed from your repo
Cron and event-driven triggers with follow-up chains, and a deploy that runs from the CI you already have.
Scheduling
Cron and event-driven triggers, with follow-up chains that wait for the loads above them.
| Job | Type | Profile | Trigger | Next Run | Last Run |
|---|---|---|---|---|---|
load_salesforce_objects crm_raw | Batch | prod | At 06:00 AMTag: extraction | in 8h 9/15/2026, 6:00:00 AM GMT+2 | 16h ago |
load_stripe_payouts finance_raw | Batch | prod | At 06:00 AMTag: extraction | in 8h 9/15/2026, 6:00:00 AM GMT+2 | 16h ago |
build_revenue_cdm finance_core | Batch | prod | After load_salesforce_objectsAfter load_stripe_payoutsTag: transformation | — | 16h ago |
CI/CD
Pipelines are Python in your repository. Deploy them from your own CI with a workspace API token, reviewed and merged like the rest of your code.
- checkoutorders-pipeline @ 8f21c4e
- dlthub profile use ciworkspace token
- dlthub deployworkspace · finance
- dlthub job runload_orders
You find out before your stakeholders do
Runs, schemas, lineage and logs on one surface, and a failure that reaches the person who owns it with the diagnosis already attached.
Observability
Runs, schemas, lineage and logs on one surface, with the reason a run failed attached to it.
| Pipeline | Started | Duration | Rows | Status |
|---|---|---|---|---|
| load_salesforce_objects | 06:00 | 2m 14s | 184,220 | |
| load_stripe_payouts | 06:00 | 48s | 12,004 | |
| load_oracle_bicc | 06:01 | 6m 02s | — | |
| build_revenue_cdm | 06:08 | 1m 31s | 96,412 |
Alerting
Failures, freshness breaches and recoveries reach Slack or email with the diagnosis attached.
| Event | Pipeline | Sent to | At | |
|---|---|---|---|---|
| Load failed | load_oracle_bicc | Slack · #data-alerts | 06:07 | |
| Freshness breached | finance_core | Email · data-oncall | 06:15 | |
| Recovered on retry | load_oracle_bicc | Slack · #data-alerts | 06:22 |
Scale out and up, in Python
Another job scales out. One argument scales up. Neither needs a console.
Deployment manifest
Your infrastructure is Python in the same repo. Adding a job is a function; giving one a bigger machine is an argument. Both go through review, and dlthub deploy ships them.
from dlthub import run
@run.pipeline(load_orders)
def orders():
...
@run.pipeline(load_oracle_bicc, schedule="0 * * * *")
def oracle():
...
@run.pipeline(
backfill_2019_2024,
require={"instance": {"size": "xlarge"}},
execute={"timeout": "6h"},
)
def backfill():
...
Instance sizesPreview
One argument on the job decides the machine. require={"instance": {"size": "large"}} gives that job 8 vCPU and 16 GiB. Omit it and it runs small. The size lives in the pull request, not in a console someone changed last quarter.
| Size | vCPU | Memory | Budget |
|---|---|---|---|
| small | 2 | 4 GiB | 1× |
| medium | 4 | 8 GiB | 2× |
| large | 8 | 16 GiB | 4× |
| xlarge | 16 | 32 GiB | 8× |
Performance toolkit
Skills that profile a slow pipeline and name the change that makes it fast, with a playbook per source type: REST, SQL and filesystem.
- Serial extractionParallelise 8 resources4.1x
- Full refresh each runAdd incremental cursor9.7x
- Batch size 500Raise to 50,0002.3x
Drive it from your own tooling
The CLI ships today and is how most people drive the platform. The API and SDK are in private preview, for teams that want to trigger and monitor runs from Airflow, dbt or their own scheduler.
CLI
Run, inspect and debug from your terminal, with the same commands your CI uses. No dashboard round trip to find out what a pipeline did.
dlthub job logs load_orders -f
06:01:12INFOextract oracle_bicc 14 resources
06:01:48INFOnormalize 184,220 rows
06:02:31WARNschema amount: int -> decimal
06:03:04INFOload snowflake 184,220 rows
06:03:04INFOdone 33s
API and SDKPreview
Trigger and monitor runs from Airflow, dbt or your own scheduler, with a workspace token. In private preview, so ask us for access.
- POST/v1/jobs/{id}/runsTrigger a run
- GET/v1/runs/{id}Poll status and rows
- GET/v1/workspaces/{id}/runsList recent runs
- PUT/v1/secrets/{key}Rotate a credential
Explore the rest of dltHub
Transformations
Annotate your sources, generate the canonical model, and keep your dbt models running.
See the modelsData quality
Expectations, schema contracts and PII redaction, running on the way into the warehouse.
See the checksAI harness
The skills, commands and MCP servers your coding agent calls to build and fix pipelines.
See the harnessSecurity and compliance
Access controls, credential handling, encryption, data residency and audit, with the control set published.
See the controlsPricing
Pick a machine size per job and pay by the second, with the meter running only while the job does.
See pricing