Load Amazon S3 data to Amazon S3

Load CSV, Parquet and JSONL files from Amazon S3 buckets with dlt. This page has everything needed to load CSV, Parquet or JSONL files from Amazon S3 into Amazon S3 with dlt's filesystem source: the credentials, the reader to pick for each format, incremental loading by modification date, plus the steps that hand the job to your coding agent.

---

Build your Amazon S3 to Amazon S3 pipeline

Set up a project, install the dltHub AI harness and its filesystem-pipeline toolkit, and let your coding agent build the pipeline:

Copy
uv init
uv add "dlt[hub,s3]" pandas
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex
uv run dlthub ai toolkit install filesystem-pipeline

Then paste this prompt into Claude, Codex, or Cursor:

The toolkit asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline and runs it on one file before loading everything. Credentials are inspected through MCP tools, so your agent never reads secrets.toml itself. About the toolkit →

Prefer to write it yourself? Every fact the agent uses is below.

---

Amazon S3 at a glance

---

How do I connect to Amazon S3?

dlt needs credentials that can list and read the bucket. The bucket_url goes in .dlt/config.toml and the credentials in .dlt/secrets.toml:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "s3://<bucket>/<path>"

# .dlt/secrets.toml
[sources.filesystem.credentials]
aws_access_key_id = "<access key id>"
aws_secret_access_key = "<secret access key>"

Create an access key in the AWS console under IAM → Users → your user → Security credentials → Create access key. The user needs s3:GetObject and s3:ListBucket on the bucket to read, plus s3:PutObject and s3:DeleteObject to write. If your keys are already in ~/.aws/credentials, leave the credentials table out and dlt uses your default profile.

Amazon S3 credentials go in the same file; the Amazon S3 destination docs list the fields it needs. dlt reads .dlt/secrets.toml automatically at runtime — never hardcode credentials in your pipeline script. For production, see setting up credentials with dlt.

---

Which Amazon S3 files can I load into Amazon S3?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the Amazon S3 table the rows land in.

---

How do I load only new Amazon S3 files?

Pass an incremental on modification_date and dlt skips every file it has already loaded:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)

On the first run dlt loads every file the glob matches; after that it reads only files modified since the last run and appends their rows to Amazon S3. If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

What does the Amazon S3 to Amazon S3 pipeline look like?

This loads the CSV files below bucket_url into one orders table in Amazon S3, and only new files on every run after the first. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_amazon_s3_to_amazon_s3_filesystem() -> None:
    files = filesystem(
        file_glob="**/*.csv",
        incremental=dlt.sources.incremental("modification_date"),
    )
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="amazon_s3_pipeline",
        destination="filesystem",
        dataset_name="amazon_s3_data",
    )
    load_info = pipeline.run(orders)
    print(load_info)


if __name__ == "__main__":
    load_amazon_s3_to_amazon_s3_filesystem()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml. Run it with uv run python amazon_s3_pipeline.py.

---

How do I configure Amazon S3 as the destination?

dlt writes to Amazon S3 through its filesystem destination; the s3:// scheme in bucket_url selects it. Install the extra with uv add "dlt[s3]", then add to .dlt/config.toml and .dlt/secrets.toml:

Copy
# .dlt/config.toml
[destination.filesystem]
bucket_url = "s3://<bucket>/<path>"

# .dlt/secrets.toml
[destination.filesystem.credentials]
aws_access_key_id = "<access key id>"
aws_secret_access_key = "<secret access key>"

dlt writes JSONL by default; pass loader_file_format="parquet" to pipeline.run() for Parquet. Delta and Iceberg table formats are supported too. See the dlt filesystem destination docs.

Changing bucket_url is all it takes to write the same data to Google Cloud Storage, Azure Blob Storage or local files instead; see dlt's filesystem destination for all four.

---

How do I make it fast?

Reading fewer files beats reading them faster, so the incremental above comes first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query Amazon S3 data in Amazon S3?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it. A filesystem destination holds files rather than tables, so dlt reads them through DuckDB views and the interface is the same as for a SQL destination; for frequent reads load Delta or Iceberg tables, which DuckDB reads only as far as the query needs (see reading data from filesystem).

Copy
import dlt

pipeline = dlt.pipeline("amazon_s3_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against Amazon S3
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

How do I deploy the Amazon S3 to Amazon S3 pipeline in production?

The pipeline runs locally, which is ideal for prototyping and one-off loads. When you need it on a schedule, monitored on every load, and shared with your team, deploy the same dlt code on the dltHub platform — no infrastructure to maintain. The harness's dlthub-platform toolkit takes your agent there:

Copy
uv run dlthub ai toolkit install dlthub-platform

• Deploy & schedule — run the pipeline as a managed job with automatic retries.
• Monitor — observable job queues, alerting, and load metrics for every run.
• Transform — promote raw Amazon S3 loads into governed, documented models.
• Visualize & share — explore data in notebooks and publish live dashboards instead of static screenshots.

Book a demo →

---

What other destinations can I load Amazon S3 data to?

dlt loads into any of these — only the destination argument changes:

Set dlt.pipeline(destination="snowflake") and add that destination's credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

• Ready for production? Book a demo of the dltHub platform →
• Filesystem source reference
• The filesystem-pipeline toolkit
• Join the dlt community on Slack