Release highlights: 1.28

Breaking changes

• Python 3.9 support dropped. Python 3.9 reached end of life, so dlt no longer supports it. The minimum is now Python 3.10 (#4074). Install this release on Python 3.10 or newer, and stay on an earlier dlt version if you cannot upgrade the interpreter (since 1.28.1).
• refresh="drop_data" keeps files on Delta and Iceberg. Truncating a Delta or persistent-catalog Iceberg table is now a transactional delete (#4051) that keeps its files for time travel, so refresh="drop_data" no longer frees storage. Pipelines that relied on it to reclaim disk must now run vacuum explicitly.
• replace truncates empty and orphaned tables. Tables of a replace resource that receive no data in a run, including nested, dynamic-name, and variant tables, are now consistently truncated (#4010). Previously they could keep stale rows across reloads, so pipelines relying on that leftover data will now see those tables emptied.

Configurable CSV encoding

You can now set the encoding of the CSV files dlt writes with a new encoding option (default utf-8) under [normalize.data_writer] (#4045). Use utf-8-sig for an Excel BOM or latin-1/cp1252 for legacy importers, and encoding_errors to control how unrepresentable characters are handled.

[normalize.data_writer]
encoding="utf-8-sig"
encoding_errors="replace"

Reliable full refresh with the refresh option

refresh is now the recommended way to fully reload data. Set refresh="drop_data" to truncate the selected resources' tables and reset their incremental state, leaving other tables intact (#4010). The old write_disposition="replace" switch for a full refresh is deprecated. Truncation now reliably covers nested, dynamic-name, and variant tables.

import dlt

@dlt.resource
def users():
    yield {"id": 1, "name": "alice"}

pipeline = dlt.pipeline("app", destination="duckdb")
# full refresh: truncate the resource's tables and reset incremental state
pipeline.run(users(), refresh="drop_data")

Refreshable cloud credentials on long-running loads

Default cloud credential chains (AWS, Azure, GCP) are now handed to the underlying access libraries, fsspec, the object_store crate, pyiceberg, and DuckDB, so each refreshes tokens through its own provider chain instead of a frozen snapshot (#4056, #4021). This stops ExpiredToken failures on long reads and loads that use temporary credentials, while static keys and explicit sessions stay frozen.

NUL characters stripped from insert-values string literals

String values containing a NUL byte (the ASCII null character, 0x00), common in data from MongoDB, previously produced invalid INSERT statements on Postgres, Redshift, and DuckDB. dlt now strips NUL from escaped string literals so those rows load (#4086), and splits multi-row inserts on newlines only so values holding other line separators stay intact.

Correct temporal precision from newer connectorx

With a newer connectorx, the sql_database connectorx backend returns timestamps and times at nanosecond precision through its arrow_stream type instead of the older millisecond date64. dlt now normalizes every connectorx temporal column to the microsecond precision destinations expect, truncating losslessly (#3996).

Shout-out to new contributors

Big thanks to our newest contributors:

• @0ywfe
• @Analect
• @wtfashwin
• @nuetu
• @tderk

Full release notes

View the 1.28.0 release notes