Release highlights: 1.31

Breaking changes

• pendulum 3 required. dlt now requires pendulum>=3.0.0 on every Python version (#4033).
• Datetimes in json columns serialize with +00:00. Datetimes stored inside json columns now end with +00:00 instead of Z (#4033).
• Incremental.last_value is the current cursor position. last_value is the actual cursor value at call time (#4033). Before the first batch it equals start_value, then the running value. lag no longer applies to it, and it keeps advancing when end_value is set. This affects rest_api {incremental.last_value} placeholders on dependent resources.
• Datetime cursor values follow the declared cursor type. start_value, last_value and get_current_range() return pendulum.DateTime only for pendulum cursors (#4406). datetime.datetime cursors now return stdlib datetimes. The lag unit comes from the declared cursor type or the type of initial_value, not from the string format of the values, which used to read lag=3 as 3 seconds for ISO strings.
• Relation.incremental() advances state by default. With a cursor bound to a resource, Relation.incremental() now advances the stored cursor on each run (#4033). Pass advance=False for the old stateless filter. Calling .incremental() twice on a relation combines both filters with AND.
• Filesystem timestamps are stdlib datetime. modification_date and other timestamps from the fsspec layer and the filesystem source are now stdlib datetime objects, tz-aware in UTC (#4388). Code that calls pendulum-only methods on them breaks.
• Snowflake tz-aware timestamps default to TIMESTAMP_LTZ. New Snowflake tables store tz-aware timestamps as TIMESTAMP_LTZ instead of TIMESTAMP_TZ (#4397). This keeps values consistent across loader file formats, since parquet could not carry offsets. Existing columns keep their type. Set use_timestamp_tz=True on the destination to keep TIMESTAMP_TZ when a consumer depends on the stored offset.
• Delta upsert now honors hard_delete. On the Delta table format, upsert now deletes records that the hard_delete column marks as deleted and never inserts them (#4305). With nested tables, this combination raises SchemaCorruptedException before the load starts.

New cdc merge strategy

The new cdc merge strategy treats each load as a complete snapshot of the source system (#4305). It inserts new keys, updates changed keys, and deletes destination records that the snapshot no longer contains. It needs a unique primary_key. Supported on duckdb (1.4+), motherduck, ducklake, Snowflake, Postgres, BigQuery, Databricks, MSSQL, Fabric, Athena with Iceberg, and filesystem with Delta.

import dlt

@dlt.resource(
    write_disposition={"disposition": "merge", "strategy": "cdc"},
    primary_key="id",
)
def customers():
    # each run yields a complete snapshot of the source table
    yield [{"id": 1, "name": "ada"}, {"id": 2, "name": "alan"}]

Skip unchanged records when merging

Set skip_unchanged_rows on an upsert or cdc resource so dlt updates only the records that actually changed (#4305). An unchanged record keeps its _dlt_load_id, so a Snowflake stream or similar consumer sees only real changes. Set row_version_column_name to compare one version or hash column instead of every column.

import dlt

@dlt.resource(
    write_disposition={
        "disposition": "merge",
        "strategy": "upsert",
        "skip_unchanged_rows": True,
    },
    primary_key="id",
)
def customers():
    yield [{"id": 1, "name": "ada"}]

Limit a merge with SQL conditions

Two new write-disposition options limit a merge with a SQL condition (#4305). source_filter chooses which loaded records to merge. destination_scope chooses which destination records a merge may delete or retire. Together they enable partition replace, where BigQuery and Snowflake prune partitions, and keyless delete-insert. They work with delete-insert, scd2 and cdc. upsert takes source_filter only.

import dlt

@dlt.resource(
    write_disposition={
        "disposition": "merge",
        "strategy": "delete-insert",
        "destination_scope": "day = '2026-08-01'",
    },
)
def events():
    # yield only the records for 2026-08-01
    yield [{"id": 1, "day": "2026-08-01"}]

Stateful incremental reads from a dataset

Relation.incremental() now pushes the cursor range down to SQL and keeps state between runs (#4033). When the cursor is bound to a resource, each run locks its range end to MAX(cursor) and advances the stored cursor, so repeated runs cover the full range without gaps or duplicates. It supports date and datetime cursors, range_start / range_end overrides, and dotted cursor paths that auto-join another table. Pass advance=False for the old stateless filter.

from datetime import datetime, timezone

import dlt

pipeline = dlt.pipeline("events_pipeline", destination="duckdb", dataset_name="events")


@dlt.resource(write_disposition="append")
def recent_events(
    created_at=dlt.sources.incremental(
        "created_at",
        initial_value=datetime(2026, 1, 1, tzinfo=timezone.utc),
        # the cursor is unique, so rows at the range end are not deferred
        primary_key="created_at",
    ),
):
    dataset = pipeline.dataset()
    events = dataset.table("events").incremental(created_at)
    yield dlt.mark.with_hints(
        events, dlt.mark.make_hints(columns=dataset.schema.tables["events"]["columns"])
    )


# each run reads only rows past the stored cursor, then advances it
pipeline.run(recent_events())

Context timezone for stored timestamps

dlt can now store timestamps in a single process-wide timezone instead of always UTC (#4033). The context timezone decides how naive and tz-aware timestamps are normalized for the timezone column hint, on both the object and arrow paths. Set it for a block of code with TimezoneContext, and read the active zone with dlt.current.timezone(). The default is still UTC.

import dlt
from dlt.common.configuration.container import Container
from dlt.common.configuration.specs import TimezoneContext

@dlt.resource
def events():
    yield {"id": 1, "created_at": "2026-01-01T12:00:00"}

pipeline = dlt.pipeline("my_pipeline", destination="duckdb")

with Container().injectable_context(TimezoneContext("Europe/Berlin")):
    # timestamps in this run are normalized to Europe/Berlin, not UTC
    pipeline.run(events())

Shout-out to new contributors

Big thanks to our newest contributors:

• @anxkhn
• @sdebruyn
• @Sanjays2402
• @adrienModeo
• @bunnysayzz
• @eeshsaxena
• @chjnett
• @arose26
• @tushardev-365
• @eddietejeda
• @jtcurlin
• @simpleqt
• @rooperuu
• @schrodervictor

Full release notes

View the 1.31.0 release notes