Amazon S3 Files Python Docs | dltHub

Load CSV, Parquet and JSONL files from Amazon S3 buckets with dlt. dlt's filesystem source lists the files behind a bucket_url, and a reader turns each one into rows: CSV, Parquet and JSONL out of the box, anything else with a few lines of your own. See the dlt filesystem source docs for the full reference.

---

Amazon S3 at a glance

---

How do I connect to Amazon S3?

dlt needs credentials that can list and read the bucket. The bucket_url goes in .dlt/config.toml and the credentials in .dlt/secrets.toml:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "s3://<bucket>/<path>"

# .dlt/secrets.toml
[sources.filesystem.credentials]
aws_access_key_id = "<access key id>"
aws_secret_access_key = "<secret access key>"

Create an access key in the AWS console under IAM → Users → your user → Security credentials → Create access key. The user needs s3:GetObject and s3:ListBucket on the bucket to read, plus s3:PutObject and s3:DeleteObject to write. If your keys are already in ~/.aws/credentials, leave the credentials table out and dlt uses your default profile.

dlt reads these files automatically at runtime — never hardcode credentials in your pipeline script. For production environments, see setting up credentials with dlt for environment variable and vault-based options.

---

Which files can I load from Amazon S3?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the table the rows land in.

---

How do I set up and run the pipeline?

Set up a virtual environment and install dlt:

Copy
uv init
uv add "dlt[hub,s3,duckdb]" pandas

1. Install the dlt AI harness:

Copy
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex

This installs project rules, a secrets management skill, appropriate ignore files, and configures the dlt MCP server for your agent. Learn more →

2. Install the filesystem-pipeline toolkit:

Copy
uv run dlthub ai toolkit install filesystem-pipeline

This loads the skills and context about dlt the agent uses to scaffold, run and tune the pipeline safely. The agent uses MCP tools to inspect credentials — it never needs to read your secrets.toml directly. Learn more →

3. Start LLM-assisted coding:

Use /create-filesystem-pipeline to load the CSV files in my Amazon S3 bucket into DuckDB.

The filesystem-pipeline toolkit takes over from here — it asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline, and runs it on one file before loading everything.

4. Run the pipeline:

Copy
uv run python filesystem_pipeline.py

If everything is configured correctly, you'll see output like this:

Copy
Pipeline amazon_s3_pipeline load step completed in 0.26 seconds
1 load package(s) were loaded to destination duckdb and into dataset amazon_s3_data
The duckdb destination used duckdb:/amazon_s3_pipeline.duckdb location to store data
Load package 1749667187.541553 is LOADED and contains no failed jobs

Inspect your pipeline and data:

Copy
uv run dlthub show

This opens the Pipeline Dashboard where you can verify pipeline state, load metrics, schema (tables, columns, types), and query the loaded data directly.

---

Python pipeline example

This example loads the CSV files below bucket_url into DuckDB as one orders table. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_files() -> None:
    files = filesystem(file_glob="**/*.csv")
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="amazon_s3_pipeline",
        destination="duckdb",
        dataset_name="amazon_s3_data",
    )
    load_info = pipeline.run(orders, write_disposition="replace")
    print(load_info)


if __name__ == "__main__":
    load_files()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml.

Load only new files:

Reading the whole bucket on every run is rarely what you want. Pass an incremental on modification_date and dlt skips every file it has already loaded; the rows of new files are appended:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)
orders = (files | read_csv()).with_name("orders")
load_info = pipeline.run(orders)

If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

How do I make it fast?

Reading fewer files beats reading them faster, so set up the incremental above first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query the loaded data?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it.

Copy
import dlt

pipeline = dlt.pipeline("amazon_s3_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against DuckDB
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

What destinations can I load Amazon S3 data to?

dlt supports loading into any of these destinations — only the destination parameter changes:

Change the destination in dlt.pipeline(destination="snowflake") and add credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

Continue your data engineering journey with the other toolkits of the dltHub AI harness:

• data-exploration — Build custom notebooks, charts, and dashboards for deeper analysis with marimo notebooks.
• dlthub-platform — Deploy, schedule, and monitor your pipeline in production.

Copy
uv run dlthub ai toolkit install data-exploration
uv run dlthub ai toolkit install dlthub-platform

• Filesystem source reference
• The filesystem-pipeline toolkit
• How to deploy a pipeline
• Join the dlt community on Slack