dlt.destinations.impl.filesystem.configuration
FilesystemDestinationClientConfiguration Objects
@configspec
class FilesystemDestinationClientConfiguration( # type: ignore[misc]
WithAttachableEngine, FilesystemConfigurationWithLocalFiles,
DestinationClientStagingConfiguration)
destination_type
type: ignore[misc]
max_state_files
Maximum number of pipeline state files to keep; 0 or negative value disables cleanup.
always_refresh_views
Always refresh table scanner views by setting the newest table metadata or globbing table files
deltalake_storage_options
Additional storage options passed to deltalake library, overriding credentials-derived values.
deltalake_configuration
Delta table configuration passed to write_deltalake and create_deltalake calls.
deltalake_streamed_exec
When true, delta merge operations use streamed execution to reduce memory usage.
iceberg_table_properties
Default Iceberg table properties applied to all tables; per-table adapter properties take precedence.
iceberg_namespace_properties
Properties passed to the Iceberg catalog when creating the namespace.
iceberg_use_catalog_purge
When true, dropped iceberg tables are purged via the persistent catalog which may leave files in place (e.g. rejected purge, deferred GC). When false, the table is dropped from the catalog and dlt deletes the table files itself.
ATTACHABLE_PROTOCOLS
Protocols a foreign DuckDB engine reads with SQL alone. These protocols mirror the
CREATE SECRET statements that DuckDbSqlClient builds. The other protocols (gs, sftp,
gdrive) access their data through an fsspec filesystem that only the process which
registered it holds.
attach_type
def attach_type() -> Optional[TAttachType]
Returns None for a protocol a foreign DuckDB engine cannot access with SQL.
data_location
def data_location() -> str
Returns scheme://netloc for remote filesystems, or the absolute local path.
can_write_from
def can_write_from(other: DestinationClientConfiguration) -> bool
Filesystem does not have an engine that can write. dlt is that engine,
and setting False here we enforce it's usage
HfFilesystemDestinationClientConfiguration Objects
@configspec
class HfFilesystemDestinationClientConfiguration(
FilesystemDestinationClientConfiguration)
hf_dataset_card
Create and update dataset card (README.md) with subset metadata for the Hugging Face dataset viewer.