Data pipelines that are just Python.

The open source Python library for moving data out of any API, database, or file and into any warehouse, lake, or vector store. Schema inference, incremental loading, and normalization included.

OPEN SOURCE · APACHE 2.0

dlt

Write the pipeline in Python and run it wherever Python already runs: a laptop, a notebook, Airflow, Dagster, a Lambda, a container. No backend, no agent to install, no data leaving your infrastructure.

pip install dlt

Loads into

Read the docs GitHub5,400+ stars · 6M+ downloads a month

AGENTIC DATA + MANAGED INFRA

dltHub

Paste this prompt into Claude, Codex, or Cursor. The agent writes the pipeline and deploys it to managed production, with schedules, alerts, and run history.

Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub

Works with

Book a demo Explore dltHub

Used by data teams at

Pick a logo to read how they did it, or browse every case study.

5,400+: GitHub stars
6M+: Downloads per month
180+: Contributors
5,900+: Community members

Agents now build 10x more dlt data pipelines than humans

In January 2025, the dlt community created 2,400 pipelines, almost entirely by hand. By January 2026, that number had grown to 81,000. The shift is not just the 34x year-over-year volume increase: agents now build roughly 10x more pipelines per month than human developers, accounting for ~91% of all new pipelines. The same inflection seen with databases such as Neon or Supabase is happening with data pipelines, only faster.

How dlt works

Every pipeline is a Python file you own

One library, one shape, whatever the source. Declare where the data comes from and where it goes; dlt works out the schema, the types, and what changed since the last run.

Declare the endpoints and the auth. dlt handles pagination, retries, and incremental state.

github_pipeline.py

import dlt
from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {
        "base_url": "https://api.github.com/",
        "auth": {"token": dlt.secrets["github_token"]},
        "paginator": "header_link",
    },
    "resource_defaults": {"write_disposition": "merge", "primary_key": "id"},
    "resources": ["events", "issues", "stargazers"],
})

pipeline = dlt.pipeline(
    pipeline_name="github",
    destination="snowflake",
    dataset_name="github_raw",
)
print(pipeline.run(source))

dlt handles

Pagination, rate limits, and retries with backoff
Incremental cursors persisted between runs
Nested JSON unnested into typed, related tables

REST API source docs

Point dlt at a connection string. Any of 100+ engines, replicated with merge or SCD2 history.

erp_pipeline.py

import dlt
from dlt.sources.sql_database import sql_database

source = sql_database(
    "postgresql://reader@erp.internal:5432/erp",
    table_names=["customers", "orders", "order_items"],
    backend="pyarrow",
)

pipeline = dlt.pipeline(
    pipeline_name="erp",
    destination="databricks",
    dataset_name="erp_raw",
)
print(pipeline.run(source, write_disposition="merge"))

dlt handles

Arrow and ConnectorX backends for bulk throughput
Reflected source types mapped to destination types
Merge, append, or SCD2 history per table

SQL database source docs

Read from S3, GCS, Azure, or SFTP. Write Parquet, Delta, or Iceberg back out.

clickstream_pipeline.py

import dlt
from dlt.sources.filesystem import filesystem, read_parquet

files = filesystem(
    bucket_url="s3://acme-events/clickstream/",
    file_glob="**/*.parquet",
) | read_parquet()

pipeline = dlt.pipeline(
    pipeline_name="clickstream",
    destination="filesystem",
    dataset_name="lakehouse",
)
print(pipeline.run(files.with_name("events"), table_format="delta"))

dlt handles

CSV, Parquet, JSONL, XLSX, and PDF readers built in
New and changed files picked up incrementally
Delta and Iceberg table formats at the destination

Filesystem source docs

No API spec, no connector, no problem. A generator function is a dlt source.

support_pipeline.py

import dlt

@dlt.resource(table_name="tickets", write_disposition="merge", primary_key="id")
def tickets(updated_at=dlt.sources.incremental("updated_at")):
    page = 1
    while True:
        rows = fetch_tickets(page, since=updated_at.last_value)
        if not rows:
            return
        yield rows
        page += 1

pipeline = dlt.pipeline(
    pipeline_name="support",
    destination="bigquery",
    dataset_name="support_raw",
)
print(pipeline.run(tickets))

dlt handles

Yield dicts, dataframes, or Arrow tables
Schema inferred from the data, then enforced
The same loading engine as every built-in source

Build a custom source

What you stop building yourself

Every team that hand-rolls ingestion rebuilds the same four things. dlt ships them tested, in a library you can read.

Schema inference and evolution

dlt reads the data, creates the tables, and adapts when a source adds or renames a field. You get an alert and a migration instead of a failed load at 3am.

Incremental by default

Declare a cursor and a primary key once. State persists between runs, so a re-run loads what changed rather than the whole table again.

Runs where Python runs

A laptop, a notebook, Airflow, Dagster, a Lambda, a container. No backend to host, no agent to install, and no data leaving your own infrastructure.

Any destination, same code

Change one argument to go from DuckDB to Snowflake, BigQuery, Databricks, Postgres, Iceberg, or a vector store. The pipeline itself does not change.

Sources and destinations, already written

Start from a verified source and customize it, generate one from an OpenAPI spec, or write your own in a few lines. Load into Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB, ClickHouse, Athena, MotherDuck, a filesystem lake, or a vector store.

Explore the ecosystem Generate from an OpenAPI spec

dlt gets the pipeline running. dltHub keeps it running.

Writing the pipeline is the part that takes an afternoon. Keeping it healthy for two years is the part that takes a team. Promote the file you already wrote and dltHub takes over the operational half, with no rewrite and no proprietary runtime.

What dltHub takes over

Deploy: One command promotes a local pipeline to managed production.
Schedule: Cron runs, retries, and backfills with no orchestrator to operate.
Observe: Run history, row counts, and lineage for every load.
Alert: Schema drift and stale data reach you before they reach consumers.
Scale: Compute sized per pipeline, from a small API to a full replication.
Govern: Environments, secrets, and access control your platform team can sign off.

Your data team stays the owner. The dltHub AI harness gives your coding agent the skills and run context to fix, maintain, and extend these pipelines in production, so Day 2 does not need a platform team.

Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub

Paste it into Claude, Codex, or Cursor. Your code stays yours, Apache 2.0, and it still runs locally.

Book a demo Explore dltHub See pricing

In production at Hugging Face, Sparebank 1, and Harness

Julien Chaumond
CTO/Co-Founder at Hugging Face

Alex Butler
Senior Data Engineer at Harness

Apache 2.0, and free to run forever

dlt is open source under Apache 2.0, commercial use included. Read it, fork it, vendor it. The paid platform runs this same library, so adopting it now commits you to nothing.

Read the docs Join the community

Apache 2.0

pip install dlt

No license key, no seat count, no call with sales
Runs entirely inside your own infrastructure
180+ contributors and 5,900+ people in the community Slack