The open source Python library for moving data out of any API, database, or file and into any warehouse, lake, or vector store. Schema inference, incremental loading, and normalization included.
OPEN SOURCE · APACHE 2.0
Write the pipeline in Python and run it wherever Python already runs: a laptop, a notebook, Airflow, Dagster, a Lambda, a container. No backend, no agent to install, no data leaving your infrastructure.
pip install dltLoads into
AGENTIC DATA + MANAGED INFRA
Paste this prompt into Claude, Codex, or Cursor. The agent writes the pipeline and deploys it to managed production, with schedules, alerts, and run history.
Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub
Works with
Used by data teams at
Pick a logo to read how they did it, or browse every case study.
How dlt works
One library, one shape, whatever the source. Declare where the data comes from and where it goes; dlt works out the schema, the types, and what changed since the last run.
Declare the endpoints and the auth. dlt handles pagination, retries, and incremental state.
import dlt
from dlt.sources.rest_api import rest_api_source
source = rest_api_source({
"client": {
"base_url": "https://api.github.com/",
"auth": {"token": dlt.secrets["github_token"]},
"paginator": "header_link",
},
"resource_defaults": {"write_disposition": "merge", "primary_key": "id"},
"resources": ["events", "issues", "stargazers"],
})
pipeline = dlt.pipeline(
pipeline_name="github",
destination="snowflake",
dataset_name="github_raw",
)
print(pipeline.run(source))Point dlt at a connection string. Any of 100+ engines, replicated with merge or SCD2 history.
import dlt
from dlt.sources.sql_database import sql_database
source = sql_database(
"postgresql://reader@erp.internal:5432/erp",
table_names=["customers", "orders", "order_items"],
backend="pyarrow",
)
pipeline = dlt.pipeline(
pipeline_name="erp",
destination="databricks",
dataset_name="erp_raw",
)
print(pipeline.run(source, write_disposition="merge"))Read from S3, GCS, Azure, or SFTP. Write Parquet, Delta, or Iceberg back out.
import dlt
from dlt.sources.filesystem import filesystem, read_parquet
files = filesystem(
bucket_url="s3://acme-events/clickstream/",
file_glob="**/*.parquet",
) | read_parquet()
pipeline = dlt.pipeline(
pipeline_name="clickstream",
destination="filesystem",
dataset_name="lakehouse",
)
print(pipeline.run(files.with_name("events"), table_format="delta"))No API spec, no connector, no problem. A generator function is a dlt source.
import dlt
@dlt.resource(table_name="tickets", write_disposition="merge", primary_key="id")
def tickets(updated_at=dlt.sources.incremental("updated_at")):
page = 1
while True:
rows = fetch_tickets(page, since=updated_at.last_value)
if not rows:
return
yield rows
page += 1
pipeline = dlt.pipeline(
pipeline_name="support",
destination="bigquery",
dataset_name="support_raw",
)
print(pipeline.run(tickets))Every team that hand-rolls ingestion rebuilds the same four things. dlt ships them tested, in a library you can read.
dlt reads the data, creates the tables, and adapts when a source adds or renames a field. You get an alert and a migration instead of a failed load at 3am.
Declare a cursor and a primary key once. State persists between runs, so a re-run loads what changed rather than the whole table again.
A laptop, a notebook, Airflow, Dagster, a Lambda, a container. No backend to host, no agent to install, and no data leaving your own infrastructure.
Change one argument to go from DuckDB to Snowflake, BigQuery, Databricks, Postgres, Iceberg, or a vector store. The pipeline itself does not change.
Start from a verified source and customize it, generate one from an OpenAPI spec, or write your own in a few lines. Load into Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB, ClickHouse, Athena, MotherDuck, a filesystem lake, or a vector store.
Writing the pipeline is the part that takes an afternoon. Keeping it healthy for two years is the part that takes a team. Promote the file you already wrote and dltHub takes over the operational half, with no rewrite and no proprietary runtime.
Your data team stays the owner. The dltHub AI harness gives your coding agent the skills and run context to fix, maintain, and extend these pipelines in production, so Day 2 does not need a platform team.
Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub
Paste it into Claude, Codex, or Cursor. Your code stays yours, Apache 2.0, and it still runs locally.
The current machine learning revolution has been enabled by the Cambrian explosion of Python open-source tools that have become so accessible that a wide range of practitioners can use them. As a simple-to-use Python library, dlt is the first tool that this new wave of people can use. By leveraging this library, we can extend the machine learning revolution into enterprise data.
Julien Chaumond
CTO/Co-Founder at Hugging Face
Our tests have proven that dlt meets our requirements regarding performance, customization and data privacy. It fits easily into our existing hosting and security infrastructure making the production rollout cost effective. The team behind dlt has a culture of supporting enterprise customers and we were able to get help and advice quickly.
Erling Brandvik
Sparebank 1 SR-Bank
dlt has enabled me to completely rewrite all of our core SaaS service pipelines in 2 weeks and have data pipelines in production with full confidence. We also achieved data democracy for our data platform. Our product, business, and operation teams can independently satisfy a majority of their data needs through no-code self-service. The teams built multi-touch attribution for how Harness acquires customers, and models for how Harness customers utilize licenses. If the teams want to build anything else to push the company forward, they don't need to wait for permission or data access to do it.
Alex Butler
Senior Data Engineer at Harness
dlt is open source under Apache 2.0, commercial use included. Read it, fork it, vendor it. The paid platform runs this same library, so adopting it now commits you to nothing.
Apache 2.0
pip install dlt