Load Local Filesystem data to Azure Synapse Analytics

Load CSV, Parquet and JSONL files from a local directory or mounted drive with dlt. This page has everything needed to load CSV, Parquet or JSONL files from Local Filesystem into Azure Synapse Analytics with dlt's filesystem source: the credentials, the reader to pick for each format, incremental loading by modification date, plus the steps that hand the job to your coding agent.

---

Build your Local Filesystem to Azure Synapse Analytics pipeline

Set up a project, install the dltHub AI harness and its filesystem-pipeline toolkit, and let your coding agent build the pipeline:

Copy
uv init
uv add "dlt[hub,synapse]" pandas
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex
uv run dlthub ai toolkit install filesystem-pipeline

Then paste this prompt into Claude, Codex, or Cursor:

The toolkit asks for the bucket_url, file format and glob, confirms the plan, scaffolds a single-table pipeline and runs it on one file before loading everything. Credentials are inspected through MCP tools, so your agent never reads secrets.toml itself. About the toolkit →

Prefer to write it yourself? Every fact the agent uses is below.

---

Local Filesystem at a glance

---

How do I connect to Local Filesystem?

Point dlt at the folder in .dlt/config.toml. Nothing here is a secret, so config.toml is the right file:

Copy
# .dlt/config.toml
[sources.filesystem]
bucket_url = "file:///absolute/path/to/files"

No credentials: dlt reads the folder directly. A plain absolute path works as well as the file:// form, including Windows paths such as C:\data\files.

Azure Synapse Analytics credentials go in the same file; the Azure Synapse Analytics destination docs list the fields it needs. dlt reads .dlt/secrets.toml automatically at runtime — never hardcode credentials in your pipeline script. For production, see setting up credentials with dlt.

---

Which Local Filesystem files can I load into Azure Synapse Analytics?

filesystem() only lists files: it yields one item per file (name, path, size and modification_date), not rows. To get rows you pipe it into a reader, a dlt transformer resource: filesystem(...) | read_csv(). dlt ships one reader per common format; for anything else you write your own @dlt.transformer.

file_glob selects the files: *.csv for one folder, **/*.csv for every folder below bucket_url. Give the reader a name with .with_name("orders"); that is the Azure Synapse Analytics table the rows land in.

---

How do I load only new Local Filesystem files?

Pass an incremental on modification_date and dlt skips every file it has already loaded:

Copy
files = filesystem(
    file_glob="**/*.csv",
    incremental=dlt.sources.incremental("modification_date"),
)

On the first run dlt loads every file the glob matches; after that it reads only files modified since the last run and appends their rows to Azure Synapse Analytics. If rows inside a file change too, add a primary key and merge: orders.apply_hints(primary_key="id", write_disposition="merge"). The toolkit's /add-incremental-loading skill sets this up for you. See incremental loading.

---

What does the Local Filesystem to Azure Synapse Analytics pipeline look like?

This loads the CSV files below bucket_url into one orders table in Azure Synapse Analytics, and only new files on every run after the first. Replace the glob and the table name with your own:

Copy
import dlt
from dlt.sources.filesystem import filesystem, read_csv


def load_local_filesystem_to_synapse() -> None:
    files = filesystem(
        file_glob="**/*.csv",
        incremental=dlt.sources.incremental("modification_date"),
    )
    orders = (files | read_csv()).with_name("orders")

    pipeline = dlt.pipeline(
        pipeline_name="local_filesystem_pipeline",
        destination="synapse",
        dataset_name="local_filesystem_data",
    )
    load_info = pipeline.run(orders)
    print(load_info)


if __name__ == "__main__":
    load_local_filesystem_to_synapse()

bucket_url and the credentials are not passed in the code: filesystem() reads them from [sources.filesystem] in .dlt/config.toml and .dlt/secrets.toml. Run it with uv run python local_filesystem_pipeline.py.

---

How do I make it fast?

Reading fewer files beats reading them faster, so the incremental above comes first. After that, the levers are the format, the reader and the glob:

The toolkit's /optimize-filesystem-performance skill measures where time goes and applies these one at a time. See the filesystem source reference.

---

How do I query Local Filesystem data in Azure Synapse Analytics?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it.

Copy
import dlt

pipeline = dlt.pipeline("local_filesystem_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against Azure Synapse Analytics
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones in your files; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

How do I deploy the Local Filesystem to Azure Synapse Analytics pipeline in production?

The pipeline runs locally, which is ideal for prototyping and one-off loads. When you need it on a schedule, monitored on every load, and shared with your team, deploy the same dlt code on the dltHub platform — no infrastructure to maintain. The harness's dlthub-platform toolkit takes your agent there:

Copy
uv run dlthub ai toolkit install dlthub-platform

• Deploy & schedule — run the pipeline as a managed job with automatic retries.
• Monitor — observable job queues, alerting, and load metrics for every run.
• Transform — promote raw Local Filesystem loads into governed, documented models.
• Visualize & share — explore data in notebooks and publish live dashboards instead of static screenshots.

Book a demo →

---

What other destinations can I load Local Filesystem data to?

dlt loads into any of these — only the destination argument changes:

Set dlt.pipeline(destination="snowflake") and add that destination's credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

• Ready for production? Book a demo of the dltHub platform →
• Filesystem source reference
• The filesystem-pipeline toolkit
• Join the dlt community on Slack