Load Azure Blob Storage data to BigQuery

Load CSV, Parquet and JSONL files from Azure Blob Storage containers with dlt. This page has everything needed to load CSV, Parquet or JSONL files from Azure Blob Storage into BigQuery with dlt's filesystem source: the credentials, the reader to pick for each format, incremental loading by modification date, plus the steps that hand the job to your coding agent.

---

Build your Azure Blob Storage to BigQuery pipeline

Set up a project, install the dltHub AI harness and its filesystem-pipeline toolkit, and let your coding agent build the pipeline:

Copy
uv init
uv add "dlt[hub,az,bigquery]" pandas
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex
uv run dlthub ai toolkit install filesystem-pipeline

Then paste this prompt into Claude, Codex, or Cursor:

The toolkit asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline and runs it on one file before loading everything. Credentials are inspected through MCP tools, so your agent never reads secrets.toml itself. About the toolkit →

Prefer to write it yourself? Every fact the agent uses is below.

---

Azure Blob Storage at a glance

---

How do I connect to Azure Blob Storage?

dlt needs credentials that can list and read the container. The bucket_url goes in .dlt/config.toml and the credentials in .dlt/secrets.toml:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "az://<container>/<path>"

# .dlt/secrets.toml
[sources.filesystem.credentials]
azure_storage_account_name = "<storage account name>"
azure_storage_account_key = "<storage account key>"

In the Azure portal open Storage accounts → your account → Access keys and copy the account name and one of the two keys. dlt also accepts a SAS token (azure_storage_sas_token) or a service principal (azure_client_id, azure_client_secret, azure_tenant_id) in place of the account key; the abfss://<container>@<account>.dfs.core.windows.net/<path> URL form works too.

BigQuery credentials go in the same file; the BigQuery destination docs list the fields it needs. dlt reads .dlt/secrets.toml automatically at runtime — never hardcode credentials in your pipeline script. For production, see setting up credentials with dlt.

---

Which Azure Blob Storage files can I load into BigQuery?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the BigQuery table the rows land in.

---

How do I load only new Azure Blob Storage files?

Pass an incremental on modification_date and dlt skips every file it has already loaded:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)

On the first run dlt loads every file the glob matches; after that it reads only files modified since the last run and appends their rows to BigQuery. If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

What does the Azure Blob Storage to BigQuery pipeline look like?

This loads the CSV files below bucket_url into one orders table in BigQuery, and only new files on every run after the first. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_azure_blob_storage_to_bigquery() -> None:
    files = filesystem(
        file_glob="**/*.csv",
        incremental=dlt.sources.incremental("modification_date"),
    )
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="azure_blob_storage_pipeline",
        destination="bigquery",
        dataset_name="azure_blob_storage_data",
    )
    load_info = pipeline.run(orders)
    print(load_info)


if __name__ == "__main__":
    load_azure_blob_storage_to_bigquery()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml. Run it with uv run python azure_blob_storage_pipeline.py.

---

How do I make it fast?

Reading fewer files beats reading them faster, so the incremental above comes first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query Azure Blob Storage data in BigQuery?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it.

Copy
import dlt

pipeline = dlt.pipeline("azure_blob_storage_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against BigQuery
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

How do I deploy the Azure Blob Storage to BigQuery pipeline in production?

The pipeline runs locally, which is ideal for prototyping and one-off loads. When you need it on a schedule, monitored on every load, and shared with your team, deploy the same dlt code on the dltHub platform — no infrastructure to maintain. The harness's dlthub-platform toolkit takes your agent there:

Copy
uv run dlthub ai toolkit install dlthub-platform

• Deploy & schedule — run the pipeline as a managed job with automatic retries.
• Monitor — observable job queues, alerting, and load metrics for every run.
• Transform — promote raw Azure Blob Storage loads into governed, documented models.
• Visualize & share — explore data in notebooks and publish live dashboards instead of static screenshots.

Book a demo →

---

What other destinations can I load Azure Blob Storage data to?

dlt loads into any of these — only the destination argument changes:

Set dlt.pipeline(destination="snowflake") and add that destination's credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

• Ready for production? Book a demo of the dltHub platform →
• Filesystem source reference
• The filesystem-pipeline toolkit
• Join the dlt community on Slack