---
title: "Release highlights: 1.29"
description: Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update.
keywords: [dlt, data-pipelines, etl, release-notes, data-engineering]
---

# Release highlights: 1.29

## Explicit joins in the dataset API

`Relation.join()` now takes an `on=` predicate, so you write the join condition yourself instead of relying on dlt schema references ([#3868](https://github.com/dlt-hub/dlt/pull/3868)). This covers joins between unrelated top-level tables, self-joins, and joins across two datasets. Passing a `Relation` from another `dlt.Dataset` joins across datasets, and destination configs now declare which relations can be joined together ([#3905](https://github.com/dlt-hub/dlt/pull/3905)).

```py
import dlt

crm = dlt.pipeline("crm", destination="duckdb", dataset_name="crm_data")
sales = dlt.pipeline("sales", destination="duckdb", dataset_name="sales_data")

# explicit predicate between two tables without a dlt schema reference
by_email = crm.dataset()["users"].join(
    "logins", on="users.email = logins.email", kind="left"
)

# pass a Relation from another dataset to join across datasets
users_purchases = crm.dataset()["users"].join(
    sales.dataset()["purchases"], on="users.id = purchases.user_id"
)
```

## AWS Secrets Manager config provider

dlt can now resolve config and secrets from AWS Secrets Manager, alongside the existing Google Secret Manager provider ([#4162](https://github.com/dlt-hub/dlt/pull/4162)). Enable it with `enable_aws_secrets`. By default dlt looks up secrets under the `dlt/` name prefix so they stay namespaced in a shared vault. The change also stops re-creating the Google secrets client on every lookup.

```toml
[providers]
enable_aws_secrets = true

[providers.aws_secrets.credentials]
aws_access_key_id = "..."
aws_secret_access_key = "..."
region_name = "eu-central-1"
```

## ClickHouse `staging-optimized` replace strategy

ClickHouse now supports the `staging-optimized` replace strategy ([#3927](https://github.com/dlt-hub/dlt/pull/3927)). It swaps the destination and staging tables with `EXCHANGE TABLES`, so a full replace is atomic and never leaves the table half-loaded. The database must use the Atomic or Shared engine, otherwise the load fails fast before any data is extracted.

```toml
[destination.clickhouse]
replace_strategy = "staging-optimized"
```

## Atomic, metadata-preserving replace on BigQuery

Turn on `enable_atomic_replace` and the `truncate-and-insert` strategy replaces a table with a single `WRITE_TRUNCATE_DATA` load job from GCS staging ([#4130](https://github.com/dlt-hub/dlt/pull/4130)). The data is swapped in place, so descriptions, labels, column policy tags, primary keys, and dependent materialized views survive the refresh with no empty-table window. It requires a GCS staging destination.

```py
import dlt

pipeline = dlt.pipeline(
    "bq_pipeline",
    destination=dlt.destinations.bigquery(enable_atomic_replace=True),
    staging="filesystem",
)
```

## Cross-batch schema evolution for Arrow sources

Safe Arrow type promotions such as `float32` to `float64` now work across flush batches instead of crashing once the data spans more than one flush ([#3896](https://github.com/dlt-hub/dlt/pull/3896)). Set `arrow_concat_promote_options` to `default` to rotate a new parquet file on a type change, or `permissive` to cast batches into the first schema. The default `none` still requires identical schemas but now fails with a clearer error.

```toml
[sources.data_writer]
arrow_concat_promote_options = "default"
```

## Runner instance size for platform jobs

Declare a job's runner resources with `require.instance`, an open dict the platform reads (today `size`) ([#4262](https://github.com/dlt-hub/dlt/pull/4262)). The legacy `machine` key still works but now emits a deprecation warning pointing to `instance`.

```py
from dlt.hub.run import job

@job(require={"instance": {"size": "medium"}})
def train():
    ...
```

## REST paginators respect their stop conditions

A REST source paginator that has already hit a stop condition (`maximum_offset`, `maximum_page`, the response total, or a missing cursor) is no longer restarted when the API also returns `has_more=true` ([#4227](https://github.com/dlt-hub/dlt/pull/4227)). Configured page limits and cursor checks now act as hard stops.

## Shout-out to new contributors

Big thanks to our newest contributors:

* [@filipesilva](https://github.com/filipesilva)
* [@cbility](https://github.com/cbility)
* [@DRACULA1729](https://github.com/DRACULA1729)
* [@oystersuki](https://github.com/oystersuki)
* [@davidmyriel](https://github.com/davidmyriel)
* [@nizar-zerrad](https://github.com/nizar-zerrad)

**Full release notes**

[View the 1.29.0 release notes](https://github.com/dlt-hub/dlt/releases/tag/1.29.0)