Skip to main content
Version: devel

dlt.destinations.impl.filesystem.configuration

FilesystemDestinationClientConfiguration Objects​

@configspec
class FilesystemDestinationClientConfiguration(  # type: ignore[misc]
        FilesystemConfigurationWithLocalFiles,
        DestinationClientStagingConfiguration)

View source on GitHub

destination_type​

type: ignore[misc]

max_state_files​

Maximum number of pipeline state files to keep; 0 or negative value disables cleanup.

always_refresh_views​

Always refresh table scanner views by setting the newest table metadata or globbing table files

deltalake_storage_options​

Additional storage options passed to deltalake library, overriding credentials-derived values.

deltalake_configuration​

Delta table configuration passed to write_deltalake and create_deltalake calls.

deltalake_streamed_exec​

When true, delta merge operations use streamed execution to reduce memory usage.

iceberg_table_properties​

Default Iceberg table properties applied to all tables; per-table adapter properties take precedence.

iceberg_namespace_properties​

Properties passed to the Iceberg catalog when creating the namespace.

iceberg_use_catalog_purge​

When true, dropped iceberg tables are purged via the persistent catalog which may leave files in place (e.g. rejected purge, deferred GC). When false, the table is dropped from the catalog and dlt deletes the table files itself.

physical_location​

def physical_location() -> str

View source on GitHub

Returns scheme://netloc for remote filesystems, or the absolute local path.

can_write_from​

def can_write_from(other: DestinationClientConfiguration) -> bool

View source on GitHub

Filesystem does not have an engine that can write. dlt is that engine, and setting False here we enforce it's usage

HfFilesystemDestinationClientConfiguration Objects​

@configspec
class HfFilesystemDestinationClientConfiguration(
        FilesystemDestinationClientConfiguration)

View source on GitHub

hf_dataset_card​

Create and update dataset card (README.md) with subset metadata for the Hugging Face dataset viewer.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.