Load Google Cloud Storage data to DuckDB

Load CSV, Parquet and JSONL files from Google Cloud Storage buckets with dlt. This page has everything needed to load CSV, Parquet or JSONL files from Google Cloud Storage into DuckDB with dlt's filesystem source: the credentials, the reader to pick for each format, incremental loading by modification date, plus the steps that hand the job to your coding agent.

---

Build your Google Cloud Storage to DuckDB pipeline

Set up a project, install the dltHub AI harness and its filesystem-pipeline toolkit, and let your coding agent build the pipeline:

Copy
uv init
uv add "dlt[hub,gs,duckdb]" pandas
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex
uv run dlthub ai toolkit install filesystem-pipeline

Then paste this prompt into Claude, Codex, or Cursor:

The toolkit asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline and runs it on one file before loading everything. Credentials are inspected through MCP tools, so your agent never reads secrets.toml itself. About the toolkit →

Prefer to write it yourself? Every fact the agent uses is below.

---

Google Cloud Storage at a glance

---

How do I connect to Google Cloud Storage?

dlt needs credentials that can list and read the bucket. The bucket_url goes in .dlt/config.toml and the credentials in .dlt/secrets.toml:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "gs://<bucket>/<path>"

# .dlt/secrets.toml
[sources.filesystem.credentials]
project_id = "<project id>"
private_key = "<private key>"
client_email = "<service account email>"

Create a service account in the Google Cloud console under IAM & Admin → Service Accounts, grant it a role on the bucket (Storage Object Viewer to read, Storage Object Admin to write), then add a JSON key under Keys → Add key. project_id, client_email and private_key are fields of that JSON file. If default Google Cloud credentials are available in your environment, leave the credentials table out and dlt falls back to them.

DuckDB credentials go in the same file; the DuckDB destination docs list the fields it needs. dlt reads .dlt/secrets.toml automatically at runtime — never hardcode credentials in your pipeline script. For production, see setting up credentials with dlt.

---

Which Google Cloud Storage files can I load into DuckDB?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the DuckDB table the rows land in.

---

How do I load only new Google Cloud Storage files?

Pass an incremental on modification_date and dlt skips every file it has already loaded:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)

On the first run dlt loads every file the glob matches; after that it reads only files modified since the last run and appends their rows to DuckDB. If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

What does the Google Cloud Storage to DuckDB pipeline look like?

This loads the CSV files below bucket_url into one orders table in DuckDB, and only new files on every run after the first. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_google_cloud_storage_to_duckdb() -> None:
    files = filesystem(
        file_glob="**/*.csv",
        incremental=dlt.sources.incremental("modification_date"),
    )
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="google_cloud_storage_pipeline",
        destination="duckdb",
        dataset_name="google_cloud_storage_data",
    )
    load_info = pipeline.run(orders)
    print(load_info)


if __name__ == "__main__":
    load_google_cloud_storage_to_duckdb()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml. Run it with uv run python google_cloud_storage_pipeline.py.

---

How do I make it fast?

Reading fewer files beats reading them faster, so the incremental above comes first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query Google Cloud Storage data in DuckDB?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it.

Copy
import dlt

pipeline = dlt.pipeline("google_cloud_storage_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against DuckDB
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

How do I deploy the Google Cloud Storage to DuckDB pipeline in production?

The pipeline runs locally, which is ideal for prototyping and one-off loads. When you need it on a schedule, monitored on every load, and shared with your team, deploy the same dlt code on the dltHub platform — no infrastructure to maintain. The harness's dlthub-platform toolkit takes your agent there:

Copy
uv run dlthub ai toolkit install dlthub-platform

• Deploy & schedule — run the pipeline as a managed job with automatic retries.
• Monitor — observable job queues, alerting, and load metrics for every run.
• Transform — promote raw Google Cloud Storage loads into governed, documented models.
• Visualize & share — explore data in notebooks and publish live dashboards instead of static screenshots.

Book a demo →

---

What other destinations can I load Google Cloud Storage data to?

dlt loads into any of these — only the destination argument changes:

Set dlt.pipeline(destination="snowflake") and add that destination's credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

• Ready for production? Book a demo of the dltHub platform →
• Filesystem source reference
• The filesystem-pipeline toolkit
• Join the dlt community on Slack