Release highlights: 1.29
Explicit joins in the dataset API
Relation.join() now takes an on= predicate, so you write the join condition yourself instead of relying on dlt schema references (#3868). This covers joins between unrelated top-level tables, self-joins, and joins across two datasets. Passing a Relation from another dlt.Dataset joins across datasets, and destination configs now declare which relations can be joined together (#3905).
import dlt
crm = dlt.pipeline("crm", destination="duckdb", dataset_name="crm_data")
sales = dlt.pipeline("sales", destination="duckdb", dataset_name="sales_data")
# explicit predicate between two tables without a dlt schema reference
by_email = crm.dataset()["users"].join(
"logins", on="users.email = logins.email", kind="left"
)
# pass a Relation from another dataset to join across datasets
users_purchases = crm.dataset()["users"].join(
sales.dataset()["purchases"], on="users.id = purchases.user_id"
)
AWS Secrets Manager config provider
dlt can now resolve config and secrets from AWS Secrets Manager, alongside the existing Google Secret Manager provider (#4162). Enable it with enable_aws_secrets. By default dlt looks up secrets under the dlt/ name prefix so they stay namespaced in a shared vault. The change also stops re-creating the Google secrets client on every lookup.
[providers]
enable_aws_secrets = true
[providers.aws_secrets.credentials]
aws_access_key_id = "..."
aws_secret_access_key = "..."
region_name = "eu-central-1"
ClickHouse staging-optimized replace strategy
ClickHouse now supports the staging-optimized replace strategy (#3927). It swaps the destination and staging tables with EXCHANGE TABLES, so a full replace is atomic and never leaves the table half-loaded. The database must use the Atomic or Shared engine, otherwise the load fails fast before any data is extracted.
[destination.clickhouse]
replace_strategy = "staging-optimized"
Atomic, metadata-preserving replace on BigQuery
Turn on enable_atomic_replace and the truncate-and-insert strategy replaces a table with a single WRITE_TRUNCATE_DATA load job from GCS staging (#4130). The data is swapped in place, so descriptions, labels, column policy tags, primary keys, and dependent materialized views survive the refresh with no empty-table window. It requires a GCS staging destination.
import dlt
pipeline = dlt.pipeline(
"bq_pipeline",
destination=dlt.destinations.bigquery(enable_atomic_replace=True),
staging="filesystem",
)
Cross-batch schema evolution for Arrow sources
Safe Arrow type promotions such as float32 to float64 now work across flush batches instead of crashing once the data spans more than one flush (#3896). Set arrow_concat_promote_options to default to rotate a new parquet file on a type change, or permissive to cast batches into the first schema. The default none still requires identical schemas but now fails with a clearer error.
[sources.data_writer]
arrow_concat_promote_options = "default"
Runner instance size for platform jobs
Declare a job's runner resources with require.instance, an open dict the platform reads (today size) (#4262). The legacy machine key still works but now emits a deprecation warning pointing to instance.
from dlt.hub.run import job
@job(require={"instance": {"size": "medium"}})
def train():
...
REST paginators respect their stop conditions
A REST source paginator that has already hit a stop condition (maximum_offset, maximum_page, the response total, or a missing cursor) is no longer restarted when the API also returns has_more=true (#4227). Configured page limits and cursor checks now act as hard stops.
Shout-out to new contributors
Big thanks to our newest contributors:
Full release notes