Microsoft SQL Server Database Python Docs | dltHub

Microsoft SQL Server is a relational database management system developed by Microsoft. dlt's sql_database source connects to it through SQLAlchemy with the mssql+pyodbc driver from the pyodbc package, reads the table schemas from the database, and loads the tables you choose into any destination. See the dlt SQL database source docs for the full reference.

---

How do I choose what to load from Microsoft SQL Server?

The sql_database source reads the schema of each table you name and turns it into a resource, so the pipeline works with whatever tables your database holds. These options narrow a load down to what you need:

Always pass table_names=[...] to sql_database(): without it, dlt reflects every table in the schema before loading anything.

---

How do I connect to the Microsoft SQL Server database?

dlt needs a database user with read access to the tables you want to load. Add its connection details to .dlt/secrets.toml:

Copy
[sources.sql_database.credentials]
drivername = "mssql+pyodbc"
host = "localhost"
port = 1433
username = "my_user"
password = "my_password"
database = "my_database"
query = { "driver" = "ODBC Driver 18 for SQL Server" }

Or as a single connection string:

Copy
[sources.sql_database]
credentials = "mssql+pyodbc://my_user:my_password@localhost:1433/my_database?driver=ODBC+Driver+18+for+SQL+Server"

Use SQLAlchemy's mssql dialect with a supported driver such as pyodbc; the mssql+pyodbc connection requires pyodbc and, for hostname connections, an installed ODBC driver whose URL name is supplied in the lowercase driver query parameter. SQLAlchemy also supports the mssql+mssqlpython dialect in version 2.1 and later, which does not require a separately installed ODBC driver.

Every field of the connection string is documented in the sql_database configuration reference. dlt reads this automatically at runtime — never hardcode credentials in your pipeline script. For production environments, see setting up credentials with dlt for environment variable and vault-based options.

---

How do I set up and run the pipeline?

Set up a virtual environment and install dlt:

Copy
uv init
uv add "dlt[hub,sql_database,duckdb]" pyarrow pandas pyodbc

1. Install the dlt AI harness:

Copy
uv run dlthub ai init --agent <your-agent> # <agent>: claude | cursor | codex

This installs project rules, a secrets management skill, appropriate ignore files, and configures the dlt MCP server for your agent. Learn more →

2. Install the sql-database-pipeline toolkit:

Copy
uv run dlthub ai toolkit install sql-database-pipeline

This loads the skills and context about dlt the agent uses to connect to the database, choose tables, and build the pipeline safely. The agent uses MCP tools to inspect credentials — it never needs to read your secrets.toml directly. Learn more →

3. Start LLM-assisted coding:

Use /find-source to load tables from my Microsoft SQL Server database into DuckDB.

The sql-database-pipeline toolkit takes over from here — it checks whether a specialised source fits better, lists the schemas and tables in your database, asks which tables to load and whether you need normalization, then scaffolds a one-table pipeline and runs it with a load limit before scaling up.

4. Run the pipeline:

Copy
uv run python sql_database_pipeline.py

If everything is configured correctly, you'll see output like this:

Copy
Pipeline microsoft_sql_server_pipeline load step completed in 0.26 seconds
1 load package(s) were loaded to destination duckdb and into dataset microsoft_sql_server_data
The duckdb destination used duckdb:/microsoft_sql_server_pipeline.duckdb location to store data
Load package 1749667187.541553 is LOADED and contains no failed jobs

Inspect your pipeline and data:

Copy
uv run dlthub show

This opens the Pipeline Dashboard where you can verify pipeline state, load metrics, schema (tables, columns, types), and query the loaded data directly.

---

Python pipeline example

This example loads the customers and orders tables from the dbo schema into DuckDB. Replace the table names with your own:

Copy
import dlt
from dlt.sources.sql_database import sql_database


def load_tables() -> None:
    source = sql_database(
        schema="dbo",
        table_names=["customers", "orders"],
    )

    pipeline = dlt.pipeline(
        pipeline_name="microsoft_sql_server_pipeline",
        destination="duckdb",
        dataset_name="microsoft_sql_server_data",
    )
    load_info = pipeline.run(source, write_disposition="replace")
    print(load_info)


if __name__ == "__main__":
    load_tables()

Credentials are not passed in the code: sql_database() reads them from [sources.sql_database] in .dlt/secrets.toml.

Load only new and changed rows:

Reloading a large table on every run is rarely what you want. Give each table a cursor column and a primary key, and dlt loads only rows changed since the last run and merges them into the destination:

Copy
source.orders.apply_hints(
    incremental=dlt.sources.incremental("updated_at"),
    write_disposition="merge",
    primary_key="id",
)

Use a timestamp such as updated_at for tables whose rows change, and an increasing id for append-only tables. The toolkit's /adjust-table skill sets this up for you.

---

Which backend should I use?

The backend decides how rows are read from Microsoft SQL Server. The default, sqlalchemy, works everywhere but is the slowest; switch once the pipeline works:

Copy
source = sql_database(
    schema="dbo",
    table_names=["customers", "orders"],
    backend="pyarrow",
    chunk_size=50000,
)

pyarrow, pandas and connectorx skip dlt's row-by-row normalization. The toolkit's /optimize-sql-performance skill measures where time goes and picks a backend, chunk size and parallelism for you. See configuring the backend.

---

How do I query the loaded data?

After a run, pipeline.dataset() holds the loaded tables. A Relation is a query on that dataset, either a whole table or a SQL string, and reading it gives you the data as a Pandas DataFrame, a PyArrow table or Python tuples. Nothing is read until you ask for it.

Copy
import dlt

pipeline = dlt.pipeline("microsoft_sql_server_pipeline")

# Step 1: Get the dataset from the pipeline
dataset = pipeline.dataset()

# Step 2: Get a Relation, either a whole table...
orders = dataset.table("orders")
# ...or any SQL query against DuckDB
recent = dataset("SELECT * FROM orders ORDER BY updated_at DESC LIMIT 100")

# Step 3: Read the data
df = orders.df()  # or orders.df(chunk_size=50) for a large table
arrow_table = recent.arrow()  # a PyArrow Table
rows = recent.fetchall()  # a list of Python tuples

The columns are the ones of your Microsoft SQL Server tables; dataset.row_counts().df() lists every table with its row count. Your agent can run the same queries through the MCP server's execute_sql_query tool, and uv run dlthub show opens the Pipeline Dashboard with the schema, load metrics and a query box. See access loaded data in Python and exploring it in marimo notebooks.

---

What destinations can I load Microsoft SQL Server data to?

dlt supports loading into any of these destinations — only the destination parameter changes:

Change the destination in dlt.pipeline(destination="snowflake") and add credentials in .dlt/secrets.toml. See the full destinations list.

---

Next steps

Continue your data engineering journey with the other toolkits of the dltHub AI harness:

• data-exploration — Build custom notebooks, charts, and dashboards for deeper analysis with marimo notebooks.
• dlthub-platform — Deploy, schedule, and monitor your pipeline in production.

Copy
uv run dlthub ai toolkit install data-exploration
uv run dlthub ai toolkit install dlthub-platform

• SQL database source reference
• How to deploy a pipeline
• How to explore your data in marimo Notebooks
• Join the dlt community on Slack