Release highlights: 1.15

⚠️ Breaking changes: .gz extensions now added to compressed files

PR: #2835

Starting with v1.15, all compressed files will include the .gz extension by default. This applies to:

• Filesystem destinations
• Internal working directories
• Staging locations (feeding downstream destinations like Redshift, BigQuery)

Notes

• Does not apply to .parquet files
• Existing filesystem destination will keep the old behavior (no .gz) for backward compatibility
• Compressed files uploaded to staging destinations will now have the .gz extension, also if dlt is configured to keep data in stage.

More details → File Compression in Filesystem Destination

---

Export schemas in DBML format

PR: #2929

You can now export your pipeline schema in DBML format, ready for visualization in DBML frontends.

# Generate a string that can be rendered in a DBML frontend
dbml_str = pipeline.default_schema.to_dbml()

This includes:

• Data and dlt tables
• Table/column metadata
• User-defined/root-child/parent-child references
• Grouping by resources
• etc.

---

REST API Client Enhancements

1. has_more flag in paginators

PR: #2817

Some APIs provide a has_more boolean instead of total or offset limits.

You can now configure paginators to use that flag.

from dlt.sources.helpers.rest_client.paginators import OffsetPaginator, PageNumberPaginator

# Paginator starts at offset=0, increments by `limit`
# Stops automatically when response["has_more"] == False
offset_paginator = OffsetPaginator(
    offset=0,
    limit=2,
    has_more_path="has_more"   # JSONPath to the boolean flag
)

# Paginator starts at page=1, increments by page
# Stops automatically when response["has_more"] == False
pn_paginator = PageNumberPaginator(
    page=1,
    has_more_path="has_more"  # JSONPath to the boolean in response
)

2. JSON body pagination support

PR: #2917

Pagination parameters can now be placed in the request body (e.g., for POST requests).

from dlt.sources.helpers.rest_client.paginators import OffsetPaginator, PageNumberPaginator

# We tell dlt to place pagination values INSIDE the request JSON body
offset_paginator = OffsetPaginator(
    offset=0,
    limit=3,
    offset_body_path="offset",     # put 'offset' in request.json["offset"]
    limit_body_path="limit",       # put 'limit'  in request.json["limit"]
)

pn_paginator = PageNumberPaginator(
    page=1,
    page_body_path="page",           # put 'page'      in request.json["page"]
)

---

Databricks + Unity Catalog enhancement

PR: #2674

New support for comments, tags, and constraints in Databricks Unity Catalog tables.

• Add table comments and tags
• Add column comments and tags
• Apply primary/foreign key constraints

[destination.databricks]
# Add PRIMARY KEY and FOREIGN KEY constraints to tables
create_indexes = true

Use the adapter for column- and table-level hints:

from dlt.destinations.adapters import databricks_adapter

databricks_adapter(
    my_resource,
    table_comment="Event data",
    table_tags=["pii", {"cost_center": "12345"}],
    column_hints={
        "user_id": {
            "column_comment": "User ID",
            "column_tags": ["pii"]
        }
    },
)

---

SQLAlchemy destination: MSSQL + Trino support

PR: #2951

• MSSQL fully supported (including JSON fields)
• Trino partially supported (with limitations: no merge/SCD2, JSON casted to string)
• Fixes for ClickHouse temp tables and BigQuery numeric handling

Example of customizing a type mapper for Trino →

---

Delta Lake: control over streamed execution

PR: #2961

By default, dlt runs Delta Lake upserts in streamed mode.

• Streamed execution = the source table is read row by row, which reduces memory pressure and works well for large datasets.
• But in streamed mode, table statistics and pruning predicates are not used — meaning Delta can’t skip irrelevant partitions efficiently.

New option: deltalake_streamed_exec

Now you can explicitly choose whether to run Delta merge operations in streamed mode or not.

• true (default): safe for very large datasets, lower memory usage.
• false: enables Delta to use table statistics and pruning (better performance when partitions and stats exist).

[destination.filesystem]
deltalake_streamed_exec = false

---

Python 3.14 support

PR: #2789

Python 3.14 is currently in beta4. We now provide experimental support for Python 3.14 to make it easier for projects depending on dlt to prepare for the upgrade.

---

Switched ConnectorX backend from arrow2 to arrow

PR: #2933

ConnectorX dropped the arrow2 backend in v0.4.2, which was dlt’s default.

dlt now defaults to the arrow backend, with minor adjustments for behavioral differences, and relaxed ConnectorX version constraints.

---

AI Command IDE integration: now works with all major IDEs (Cursor, Copilot, Windsurf, etc.)

PR: #2937

The AI command now supports all major IDEs, automatically creating the correct rule files for each AI editor/agent. This ensures seamless integration whether you’re using Cursor, Copilot, Windsurf, Claude, or other supported editors.

---

Shout-out to new contributors

Big thanks to our newest contributors:

• @bayees — #2674
• @7amza79 — #2783
• @michaelconan — #2817
• @Giackgamba — #2917

---

Full release notes

View the complete list of changes →