Release highlights: 1.25
Breaking changes
pipeline.dataset()now includes every schema.pipeline.dataset()without aschemaargument used to return only the default schema's tables. With multiple schemas it now includes them all by default, so queries may see extra tables or rows in shared table names. Restore the old behavior withpipeline.dataset(schema=pipeline.default_schema_name)(#3770).
New lance destination
Load data into the Lance columnar format on local disk or s3, gs, and az object storage, with optional vector embeddings generated through lancedb (#3810). It uses the Lance Directory Namespace V2 catalog and supports branching. Install with pip install "dlt[lance]", then point a pipeline at it. This complements the existing lancedb destination, which targets LanceDB Cloud.
import dlt
pipeline = dlt.pipeline(
pipeline_name="movies",
destination="lance",
dataset_name="movies_db",
)
info = pipeline.run(
[{"id": 1, "title": "Blade Runner", "year": 1982}],
table_name="movies",
)
Query every schema from one dataset
A pipeline that loads several sources now exposes all their tables through one pipeline.dataset(), so you can query across sources together (#3770). Tables that share a name across schemas have their columns merged and rows combined, with missing columns filled as NULL. Pass a schema name to scope to one source, and read its load history per schema.
import dlt
pipeline = dlt.pipeline(pipeline_name="my_pipeline", destination="duckdb")
# includes tables from every schema the pipeline loaded
dataset = pipeline.dataset()
# scope to a single schema (the pre-1.25.0 default)
one_source = pipeline.dataset(schema="github")
# load history is tracked per schema
load_ids = dataset.load_ids(schema_name="github")
Load metrics persist and record follow-up jobs
Load metrics are now persisted to the load package, so a load that is interrupted and resumed keeps the metrics of already-completed jobs instead of losing them, and each job records the follow-up jobs it creates. That follow-up graph is saved into the pipeline trace, closing the long-standing #853 (#3768).
Shout-out to new contributors
Big thanks to our newest contributors:
Full release notes