Release highlights: 1.24

Breaking changes

• Incremental metrics moved in the pipeline trace. Incremental metrics are now stored in table format in the trace. Counters like unfiltered_items_count and initial_unique_hashes_count no longer sit as scalar keys under a resource's custom_metrics. They now live in an incremental child table of the extract resource_metrics. Update code that reads them from the trace (#3718).

New insert-only merge strategy

A fourth merge strategy joins delete-insert, scd2, and upsert. insert-only inserts a record when its primary key is not yet in the destination and skips the record when the key already exists, never updating or deleting. It suits append-only data like events, logs, and transactions, and gives idempotent, faster loads by skipping UPDATE operations. Works on all SQL destinations, Delta, Iceberg, and LanceDB (#3741).

import dlt

@dlt.resource(
    write_disposition={"disposition": "merge", "strategy": "insert-only"},
    primary_key="event_id",
)
def events():
    yield [{"event_id": 1, "name": "login"}, {"event_id": 2, "name": "click"}]

Native deduplication on ClickHouse

ClickHouse tables can now use the replacing_merge_tree engine to deduplicate and soft-delete at the storage layer, without dlt's merge write disposition. Select it through clickhouse_adapter and load with append. The dedup_sort column hint picks the version to keep and hard_delete marks rows for removal, and dlt wires both into the engine parameters (#3366).

import dlt
from dlt.destinations.adapters import clickhouse_adapter

@dlt.resource(
    write_disposition="append",
    primary_key="id",
    columns={
        "version": {"dedup_sort": "desc", "nullable": False},
        "is_deleted": {"hard_delete": True, "nullable": False},
    },
)
def events(data):
    yield from data

clickhouse_adapter(
    events,
    table_engine_type="replacing_merge_tree",
    settings={"clean_deleted_rows": "Always"},
)

Parallelize every Airflow source task

PipelineTasksGroup.add_run gains a serialize_first_task argument for the parallel and parallel-isolated decompose modes. Set it to False to fan out every decomposed source component concurrently from a shared start node, rather than waiting for the first component to finish. The default True keeps the old behavior, running the first component first to create the schema. This release also adds basic Airflow 3 support (#3652).

import dlt
from dlt.helpers.airflow_helper import PipelineTasksGroup

@dlt.source
def smoke_source():
    @dlt.resource
    def users():
        yield [{"id": 1, "name": "Alice"}]

    return users

tasks = PipelineTasksGroup("dlt_smoke_tasks", wipe_local_data=True)
pipeline = dlt.pipeline(pipeline_name="dlt_smoke", destination="duckdb")

tasks.add_run(
    pipeline,
    smoke_source(),
    decompose="parallel",
    serialize_first_task=False,
)

Handle Arrow schema mismatches across batches

When a resource yields several Arrow tables or pandas DataFrames, dlt concatenates them and by default requires identical schemas, so any type difference across batches raises an error. Set arrow_concat_promote_options to "default" to fill missing columns with nulls, or "permissive" to also promote across type families such as int64 to double (#3701).

[data_writer]
arrow_concat_promote_options = "permissive"

DATA_WRITER__ARROW_CONCAT_PROMOTE_OPTIONS=permissive

Shout-out to new contributors

Big thanks to our newest contributors:

• @prevostc
• @veeceey
• @JustinSobayo
• @shnhdan
• @YuF-9468

Full release notes

View the 1.24.0 release notes