Load Local Filesystem data to Amazon S3

Load CSV, Parquet and JSONL files from a local directory or mounted drive with dlt. This page has everything needed to load CSV, Parquet or JSONL files from Local Filesystem into Amazon S3 with dlt's filesystem source: the credentials, the reader to pick for each format, incremental loading by modification date, plus the steps that hand the job to your coding agent.

---

Build your Local Filesystem to Amazon S3 pipeline

Set up a project, install the dltHub AI harness and its filesystem-pipeline toolkit, and let your coding agent build the pipeline:

Copy
uv init
uv add "dlt[hub,s3]" pandas
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex
uv run dlthub ai toolkit install filesystem-pipeline

Then paste this prompt into Claude, Codex, or Cursor:

The toolkit asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline and runs it on one file before loading everything. Credentials are inspected through MCP tools, so your agent never reads secrets.toml itself. About the toolkit →

Prefer to write it yourself? Every fact the agent uses is below.

---

Local Filesystem at a glance

---

How do I connect to Local Filesystem?

Point dlt at the folder in .dlt/config.toml. Nothing here is a secret, so config.toml is the right file:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "file:///absolute/path/to/files"

No credentials: dlt reads the folder directly. A plain absolute path works as well as the file:// form, including Windows paths such as C:\data\files.

Amazon S3 credentials go in the same file; the Amazon S3 destination docs list the fields it needs. dlt reads .dlt/secrets.toml automatically at runtime — never hardcode credentials in your pipeline script. For production, see setting up credentials with dlt.

---

Which Local Filesystem files can I load into Amazon S3?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the Amazon S3 table the rows land in.

---

How do I load only new Local Filesystem files?

Pass an incremental on modification_date and dlt skips every file it has already loaded:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)

On the first run dlt loads every file the glob matches; after that it reads only files modified since the last run and appends their rows to Amazon S3. If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

What does the Local Filesystem to Amazon S3 pipeline look like?

This loads the CSV files below bucket_url into one orders table in Amazon S3, and only new files on every run after the first. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_local_filesystem_to_amazon_s3_filesystem() -> None:
    files = filesystem(
        file_glob="**/*.csv",
        incremental=dlt.sources.incremental("modification_date"),
    )
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="local_filesystem_pipeline",
        destination="filesystem",
        dataset_name="local_filesystem_data",
    )
    load_info = pipeline.run(orders)
    print(load_info)


if __name__ == "__main__":
    load_local_filesystem_to_amazon_s3_filesystem()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml. Run it with uv run python local_filesystem_pipeline.py.

---

How do I configure Amazon S3 as the destination?

dlt writes to Amazon S3 through its filesystem destination; the s3:// scheme in bucket_url selects it. Install the extra with uv add "dlt[s3]", then add to .dlt/config.toml and .dlt/secrets.toml:

Copy
# .dlt/config.toml
[destination.filesystem]
bucket_url = "s3://<bucket>/<path>"

# .dlt/secrets.toml
[destination.filesystem.credentials]
aws_access_key_id = "<access key id>"
aws_secret_access_key = "<secret access key>"

dlt writes JSONL by default; pass loader_file_format="parquet" to pipeline.run() for Parquet. Delta and Iceberg table formats are supported too. See the dlt filesystem destination docs.

Changing bucket_url is all it takes to write the same data to Google Cloud Storage, Azure Blob Storage or local files instead; see dlt's filesystem destination for all four.

---

How do I make it fast?

Reading fewer files beats reading them faster, so the incremental above comes first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query Local Filesystem data in Amazon S3?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it. A filesystem destination holds files rather than tables, so dlt reads them through DuckDB views and the interface is the same as for a SQL destination; for frequent reads load Delta or Iceberg tables, which DuckDB reads only as far as the query needs (see reading data from filesystem).

Copy
import dlt

pipeline = dlt.pipeline("local_filesystem_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against Amazon S3
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

How do I deploy the Local Filesystem to Amazon S3 pipeline in production?

The pipeline runs locally, which is ideal for prototyping and one-off loads. When you need it on a schedule, monitored on every load, and shared with your team, deploy the same dlt code on the dltHub platform — no infrastructure to maintain. The harness's dlthub-platform toolkit takes your agent there:

Copy
uv run dlthub ai toolkit install dlthub-platform

• Deploy & schedule — run the pipeline as a managed job with automatic retries.
• Monitor — observable job queues, alerting, and load metrics for every run.
• Transform — promote raw Local Filesystem loads into governed, documented models.
• Visualize & share — explore data in notebooks and publish live dashboards instead of static screenshots.

Book a demo →

---

What other destinations can I load Local Filesystem data to?

dlt loads into any of these — only the destination argument changes:

Set dlt.pipeline(destination="snowflake") and add that destination's credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

• Ready for production? Book a demo of the dltHub platform →
• Filesystem source reference
• The filesystem-pipeline toolkit
• Join the dlt community on Slack