---
title: "Release highlights: 1.24"
description: Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update.
keywords: [dlt, data-pipelines, etl, release-notes, data-engineering]
---

# Release highlights: 1.24

## Breaking changes

- **Incremental metrics moved in the pipeline trace.** Incremental metrics are now stored in table format in the trace. Counters like `unfiltered_items_count` and `initial_unique_hashes_count` no longer sit as scalar keys under a resource's `custom_metrics`. They now live in an `incremental` child table of the extract `resource_metrics`. Update code that reads them from the trace ([#3718](https://github.com/dlt-hub/dlt/pull/3718)).

## New `insert-only` merge strategy

A fourth merge strategy joins delete-insert, scd2, and upsert. `insert-only` inserts a record when its primary key is not yet in the destination and skips the record when the key already exists, never updating or deleting. It suits append-only data like events, logs, and transactions, and gives idempotent, faster loads by skipping UPDATE operations. Works on all SQL destinations, Delta, Iceberg, and LanceDB ([#3741](https://github.com/dlt-hub/dlt/pull/3741)).

```py
import dlt

@dlt.resource(
    write_disposition={"disposition": "merge", "strategy": "insert-only"},
    primary_key="event_id",
)
def events():
    yield [{"event_id": 1, "name": "login"}, {"event_id": 2, "name": "click"}]
```

## Native deduplication on ClickHouse

ClickHouse tables can now use the `replacing_merge_tree` engine to deduplicate and soft-delete at the storage layer, without dlt's merge write disposition. Select it through `clickhouse_adapter` and load with `append`. The `dedup_sort` column hint picks the version to keep and `hard_delete` marks rows for removal, and dlt wires both into the engine parameters ([#3366](https://github.com/dlt-hub/dlt/pull/3366)).

```py
import dlt
from dlt.destinations.adapters import clickhouse_adapter

@dlt.resource(
    write_disposition="append",
    primary_key="id",
    columns={
        "version": {"dedup_sort": "desc", "nullable": False},
        "is_deleted": {"hard_delete": True, "nullable": False},
    },
)
def events(data):
    yield from data

clickhouse_adapter(
    events,
    table_engine_type="replacing_merge_tree",
    settings={"clean_deleted_rows": "Always"},
)
```

## Parallelize every Airflow source task

`PipelineTasksGroup.add_run` gains a `serialize_first_task` argument for the `parallel` and `parallel-isolated` decompose modes. Set it to `False` to fan out every decomposed source component concurrently from a shared start node, rather than waiting for the first component to finish. The default `True` keeps the old behavior, running the first component first to create the schema. This release also adds basic Airflow 3 support ([#3652](https://github.com/dlt-hub/dlt/pull/3652)).

```py
import dlt
from dlt.helpers.airflow_helper import PipelineTasksGroup

@dlt.source
def smoke_source():
    @dlt.resource
    def users():
        yield [{"id": 1, "name": "Alice"}]

    return users

tasks = PipelineTasksGroup("dlt_smoke_tasks", wipe_local_data=True)
pipeline = dlt.pipeline(pipeline_name="dlt_smoke", destination="duckdb")

tasks.add_run(
    pipeline,
    smoke_source(),
    decompose="parallel",
    serialize_first_task=False,
)
```

## Handle Arrow schema mismatches across batches

When a resource yields several Arrow tables or pandas DataFrames, dlt concatenates them and by default requires identical schemas, so any type difference across batches raises an error. Set `arrow_concat_promote_options` to `"default"` to fill missing columns with nulls, or `"permissive"` to also promote across type families such as int64 to double ([#3701](https://github.com/dlt-hub/dlt/pull/3701)).

```toml
[data_writer]
arrow_concat_promote_options = "permissive"
```

```sh
DATA_WRITER__ARROW_CONCAT_PROMOTE_OPTIONS=permissive
```

## Shout-out to new contributors

Big thanks to our newest contributors:

* [@prevostc](https://github.com/prevostc)
* [@veeceey](https://github.com/veeceey)
* [@JustinSobayo](https://github.com/JustinSobayo)
* [@shnhdan](https://github.com/shnhdan)
* [@YuF-9468](https://github.com/YuF-9468)

**Full release notes**

[View the 1.24.0 release notes](https://github.com/dlt-hub/dlt/releases/tag/1.24.0)