Load data into files and object storage with the dlt filesystem destination

Load data into files and object storage with dlt's filesystem destination. One pipeline writes to Amazon S3, Google Cloud Storage, Azure Blob Storage or local files; bucket_url picks which.

dlt's filesystem destination writes your data as files (JSONL, Parquet or CSV, optionally as Delta or Iceberg tables) to cloud object storage or a local folder.

dlt filesystem destination docs →

Choose a destination

Amazon S3bucket_url = "s3://<bucket>/<path>"Write data from any source to Amazon S3 buckets as Parquet, JSONL or CSV files with the dlt filesystem destination.
Google Cloud Storagebucket_url = "gs://<bucket>/<path>"Write data from any source to Google Cloud Storage buckets as Parquet, JSONL or CSV files with the dlt filesystem destination.
Azure Blob Storagebucket_url = "az://<container>/<path>"Write data from any source to Azure Blob Storage containers as Parquet, JSONL or CSV files with the dlt filesystem destination.
Local Filesystembucket_url = "file:///absolute/path/to/files"Write data from any source to local files as Parquet, JSONL or CSV with the dlt filesystem destination.

One pipeline, any destination

Every one of these is dlt's filesystem destination. Write the pipeline once with destination="filesystem", then point bucket_url at the destination you want; switching later is a one-line config change, not a code change.


import dlt

pipeline = dlt.pipeline(
    pipeline_name="my_pipeline",
    destination="filesystem",
    dataset_name="my_data",
)


# .dlt/config.toml
[destination.filesystem]
bucket_url = "s3://<bucket>/<path>"              # Amazon S3
# bucket_url = "gs://<bucket>/<path>"            # Google Cloud Storage
# bucket_url = "az://<container>/<path>"         # Azure Blob Storage
# bucket_url = "file:///absolute/path/to/files"  # Local Filesystem

How do I configure credentials?

Filesystem is one of dlt's most powerful and most complex destinations. For the credential configuration options, refer to the dlt docs for your destination:

dlt's filesystem destination also writes to SFTP servers and Hugging Face datasets, among others; the dlt docs cover how to configure them. dlt filesystem destination docs →

It is one destination in dlt: the same pipeline code loads into Amazon S3, Google Cloud Storage, Azure Blob Storage or local files, and only the bucket_url scheme changes.