# dlt - data load tool > dlt is an open-source Python library that loads data from various sources into well-structured datasets. Built for LLMs with 8000+ source connectors. ## Getting started - [Introduction](/docs/intro.md): Introduction to dlt - [Installation](/docs/reference/installation.md): How to install dlt - [Dashboard: inspect the pipeline](/docs/general-usage/dashboard.md): Open a comprehensive dashboard with information about your pipeline ## Getting started > Quickstart - [Load data from a REST API](/docs/tutorial/rest-api.md): How to extract data from a REST API using dlt's REST API source - [Load data from a SQL database](/docs/tutorial/sql-database.md): How to extract data from a SQL Database using dlt's SQL Database core source - [Load data from a cloud storage or a file system](/docs/tutorial/filesystem.md): Learn how to load data files like JSON, JSONL, CSV, and Parquet from a cloud storage (AWS S3, Google Cloud Storage, Google Drive, Azure Blob Storage) or a local file system using dlt. - [Build advanced dlt pipeline from scratch](/docs/tutorial/load-data-from-an-api.md): Build custom, production grade pipeline just by writing code ## Getting started > Release highlights - [Release highlights: v1.21.2](/docs/release-notes/1.21.2.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.19](/docs/release-notes/1.19.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.18](/docs/release-notes/1.18.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.17](/docs/release-notes/1.17.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.16](/docs/release-notes/1.16.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.15](/docs/release-notes/1.15.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.12.3 - 1.14.1](/docs/release-notes/1.13-1.14.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. - [Release highlights: 1.12.1](/docs/release-notes/1.12.1.md): Release highlights provide a concise overview of the most important new features, improvements, and fixes in a software update, helping users quickly understand what's changed and how it impacts their workflow. ## Build with AI - [REST API Source with dlthub AI Workbench](/docs/dlt-ecosystem/llm-tooling/llm-native-workflow.md): Build any REST API source with dltHub AI Workbench toolkits - workflows, skills, rules, and MCP tools - [Explore and Transform your data with dltHub AI Workbench](/docs/dlt-ecosystem/llm-tooling/explore-and-transform.md): Explore loaded pipeline data, build interactive dashboards, and transform data into a Canonical Data Model using dltHub AI Workbench toolkits ## Core concepts - [How dlt works](/docs/reference/explainers/how-dlt-works.md): How data load tool (dlt) works - [Glossary](/docs/general-usage/glossary.md): Glossary of common dlt terms - [Source](/docs/general-usage/source.md): Explanation of what a dlt source is - [Resource](/docs/general-usage/resource.md): Explanation of what a dlt resource is - [Pipeline](/docs/general-usage/pipeline.md): Explanation of what a dlt pipeline is - [Destination](/docs/general-usage/destination.md): Declare and configure destinations to which to load data - [Access datasets in Python](/docs/general-usage/dataset-access/dataset.md): Conveniently access the data loaded to any destination in Python - [State](/docs/general-usage/state.md): Explanation of what a dlt state is - [Schema](/docs/general-usage/schema.md): Schema ## Sources > REST API - [REST API source](/docs/dlt-ecosystem/verified-sources/rest_api/basic.md): Learn how to set up and configure - [Advanced configuration](/docs/dlt-ecosystem/verified-sources/rest_api/advanced.md): Learn custom response processing, headers configuration and more - [OpenAPI source generator](/docs/dlt-ecosystem/verified-sources/openapi-generator.md): OpenAPI dlt source generator ## Sources > SQL database - [Setup](/docs/dlt-ecosystem/verified-sources/sql_database/setup.md): basic steps for setting up a dlt pipeline for SQL Database - [Configuration](/docs/dlt-ecosystem/verified-sources/sql_database/configuration.md): configuring the pipeline script, connection, and backend settings in the sql_database source - [Usage](/docs/dlt-ecosystem/verified-sources/sql_database/usage.md): basic usage of the sql_database source - [Troubleshooting](/docs/dlt-ecosystem/verified-sources/sql_database/troubleshooting.md): common troubleshooting use-cases for the sql_database source - [Advanced usage](/docs/dlt-ecosystem/verified-sources/sql_database/advanced.md): advance configuration and usage of the sql_database source - [Incremental loading guide](/docs/walkthroughs/sql-incremental-configuration.md): Incremental SQL data loading strategies ## Sources - [Cloud storage and filesystem](/docs/dlt-ecosystem/verified-sources/filesystem.md): dlt-verified source for reading files from cloud storage and local file system - [Arrow Table / Pandas](/docs/dlt-ecosystem/verified-sources/arrow-pandas.md): dlt source for Arrow tables and Pandas dataframes ## Sources > Verified sources - [Airtable](/docs/dlt-ecosystem/verified-sources/airtable.md): dlt verified source for Airtable - [Amazon Kinesis](/docs/dlt-ecosystem/verified-sources/amazon_kinesis.md): dlt verified source for Amazon Kinesis - [Asana](/docs/dlt-ecosystem/verified-sources/asana.md): dlt verified source for Asana API - [Chess.com](/docs/dlt-ecosystem/verified-sources/chess.md): dlt verified source for Chess.com API - [Facebook Ads](/docs/dlt-ecosystem/verified-sources/facebook_ads.md): dlt verified source for Facebook Ads - [Freshdesk](/docs/dlt-ecosystem/verified-sources/freshdesk.md): dlt verified source for Freshdesk API - [GitHub](/docs/dlt-ecosystem/verified-sources/github.md): dlt verified source for GitHub API - [Google Ads](/docs/dlt-ecosystem/verified-sources/google_ads.md): dlt verified source for Google Ads API - [Google Analytics](/docs/dlt-ecosystem/verified-sources/google_analytics.md): dlt verified source for Google Analytics API - [Google Sheets](/docs/dlt-ecosystem/verified-sources/google_sheets.md): dlt verified source for Google Sheets API - [Hubspot](/docs/dlt-ecosystem/verified-sources/hubspot.md): dlt verified source for Hubspot API - [Inbox](/docs/dlt-ecosystem/verified-sources/inbox.md): dlt verified source for Mail Inbox - [Jira](/docs/dlt-ecosystem/verified-sources/jira.md): dlt verified source for Atlassian Jira - [Kafka](/docs/dlt-ecosystem/verified-sources/kafka.md): dlt verified source for Confluent Kafka - [Matomo](/docs/dlt-ecosystem/verified-sources/matomo.md): dlt verified source for Matomo - [MongoDB](/docs/dlt-ecosystem/verified-sources/mongodb.md): dlt verified source for MongoDB - [Mux](/docs/dlt-ecosystem/verified-sources/mux.md): dlt verified source for Mux - [Notion](/docs/dlt-ecosystem/verified-sources/notion.md): dlt pipeline for Notion API - [Personio](/docs/dlt-ecosystem/verified-sources/personio.md): dlt verified source for Personio API - [Postgres replication](/docs/dlt-ecosystem/verified-sources/pg_replication.md): dlt verified source for Postgres replication - [Pipedrive](/docs/dlt-ecosystem/verified-sources/pipedrive.md): dlt verified source for Pipedrive API - [Salesforce](/docs/dlt-ecosystem/verified-sources/salesforce.md): dlt pipeline for Salesforce API - [Scrapy](/docs/dlt-ecosystem/verified-sources/scrapy.md): dlt verified source for Scraping using scrapy - [Shopify](/docs/dlt-ecosystem/verified-sources/shopify.md): dlt pipeline for Shopify API - [Slack](/docs/dlt-ecosystem/verified-sources/slack.md): dlt verified source for Slack API - [Strapi](/docs/dlt-ecosystem/verified-sources/strapi.md): dlt verified source for Strapi API - [Stripe](/docs/dlt-ecosystem/verified-sources/stripe.md): dlt verified source for Stripe API - [Workable](/docs/dlt-ecosystem/verified-sources/workable.md): dlt pipeline for Workable API - [Zendesk](/docs/dlt-ecosystem/verified-sources/zendesk.md): dlt pipeline for Zendesk API ## Destinations - [Destinations](/docs/dlt-ecosystem/destinations.md): List of destinations - [Cloud storage and filesystem](/docs/dlt-ecosystem/destinations/filesystem.md): Store data in remote file systems and cloud storage services like AWS S3, Google Cloud Storage, or Azure Blob Storage - [30+ SQL databases (powered by SQLAlchemy)](/docs/dlt-ecosystem/destinations/sqlalchemy.md): SQLAlchemy destination - [AWS Athena / Glue Catalog](/docs/dlt-ecosystem/destinations/athena.md): AWS Athena `dlt` destination - [Google BigQuery](/docs/dlt-ecosystem/destinations/bigquery.md): Google BigQuery `dlt` destination - [ClickHouse](/docs/dlt-ecosystem/destinations/clickhouse.md): ClickHouse `dlt` destination - [Databricks](/docs/dlt-ecosystem/destinations/databricks.md): Databricks `dlt` destination - [Delta](/docs/dlt-ecosystem/destinations/delta-iceberg.md): Delta dlt destination - [🧪 Dremio](/docs/dlt-ecosystem/destinations/dremio.md): Dremio `dlt` destination - [DuckDB](/docs/dlt-ecosystem/destinations/duckdb.md): DuckDB `dlt` destination - [DuckLake](/docs/dlt-ecosystem/destinations/ducklake.md): DuckLake destination (DuckDB + ducklake extension) - [Hugging Face Datasets](/docs/dlt-ecosystem/destinations/huggingface.md): Load data into Hugging Face Datasets repositories using dlt - [Iceberg](/docs/dlt-ecosystem/destinations/iceberg.md): Iceberg dlt destination - [Lance](/docs/dlt-ecosystem/destinations/lance.md): Lance is an open-source columnar format for AI/ML that can be used as a destination in dlt. - [LanceDB](/docs/dlt-ecosystem/destinations/lancedb.md): LanceDB is a multimodal lakehouse for AI that can be used as a destination in dlt. - [Microsoft Fabric Warehouse](/docs/dlt-ecosystem/destinations/fabric.md): Microsoft Fabric Warehouse `dlt` destination - [Microsoft SQL Server](/docs/dlt-ecosystem/destinations/mssql.md): Microsoft SQL Server `dlt` destination - [MotherDuck / DuckLake](/docs/dlt-ecosystem/destinations/motherduck.md): MotherDuck and DuckLake `dlt` destination - [Postgres](/docs/dlt-ecosystem/destinations/postgres.md): Postgres `dlt` destination - [Amazon Redshift](/docs/dlt-ecosystem/destinations/redshift.md): Amazon Redshift `dlt` destination - [Snowflake](/docs/dlt-ecosystem/destinations/snowflake.md): Snowflake `dlt` destination - [Azure Synapse](/docs/dlt-ecosystem/destinations/synapse.md): Azure Synapse `dlt` destination - [Qdrant](/docs/dlt-ecosystem/destinations/qdrant.md): Qdrant is a high-performance vector search engine/database that can be used as a destination in dlt. - [Weaviate](/docs/dlt-ecosystem/destinations/weaviate.md): Weaviate is an open source vector database that can be used as a destination in dlt. - [Custom destination](/docs/dlt-ecosystem/destinations/destination.md): Custom `dlt` destination function for reverse ETL - [Community Destinations](/docs/dlt-ecosystem/destinations/community-destinations.md): Community-contributed destinations for dlt ## Configuration & secrets - [Overview and examples](/docs/general-usage/credentials/setup.md): Learn where configs are stored and how to write them - [Access to configuration in code](/docs/general-usage/credentials/advanced.md): Access configuration via dlt function arguments or explicitly - [Vault providers](/docs/general-usage/credentials/vaults.md): Learn how to configure Google Secrets and Airflow providers - [Built-in credentials](/docs/general-usage/credentials/complex_types.md): Configure access to AWS, Azure, Google Cloud and other systems - [How to add credentials](/docs/walkthroughs/add_credentials.md): How to add credentials locally and in production ## Load strategy - [Full loading](/docs/general-usage/full-loading.md): Full loading with dlt - [Merge loading](/docs/general-usage/merge-loading.md): Merge loading with dlt - [Staging](/docs/dlt-ecosystem/staging.md): Configure an S3 or GCS bucket for staging before copying into the destination ## Load strategy > Incremental - [Incremental loading](/docs/general-usage/incremental-loading.md): Introduction to incremental loading with dlt - [Cursor-based incremental loading](/docs/general-usage/incremental/cursor.md): Track changes using cursor fields with dlt - [Lag / Attribution window](/docs/general-usage/incremental/lag.md): Use lag to refresh data within a specific time window - [Advanced state management for incremental loading](/docs/general-usage/incremental/advanced-state.md): Custom state tracking and lag/attribution windows - [Troubleshooting incremental loading](/docs/general-usage/incremental/troubleshooting.md): Common issues and how to fix them ## Schema management - [Schema and data contracts](/docs/general-usage/schema-contracts.md): Controlling schema evolution and validating data - [Schema evolution](/docs/general-usage/schema-evolution.md): A small guide to elaborate on how schema evolution works - [Review dlt schema](/docs/general-usage/dataset-access/view-dlt-schema.md): View your dlt schema via files, CLI, static and interactive diagram - [Adjust a schema](/docs/walkthroughs/adjust-a-schema.md): How to adjust a schema - [Naming convention](/docs/general-usage/naming-convention.md): Control how dlt creates table, column and other identifiers ## Transformations - [Transforming your data](/docs/dlt-ecosystem/transformations.md): How to transform your data ## Transformations > Extract, Transform, Load (ETL) - [Transform data with `add_map`](/docs/dlt-ecosystem/transformations/add-map.md): Apply lightweight python transformations to your data inline using `add_map`. - [Renaming columns](/docs/general-usage/customising-pipelines/renaming_columns.md): Renaming columns by replacing the special characters - [Removing columns](/docs/general-usage/customising-pipelines/removing_columns.md): Removing columns by passing a list of column names - [Pseudonymizing columns](/docs/general-usage/customising-pipelines/pseudonymizing_columns.md): Pseudonymizing (or anonymizing) columns by replacing the special characters ## Transformations > Extract, Load, Transform (ELT) - [Transform data in Python with Arrow tables or DataFrames](/docs/dlt-ecosystem/transformations/python.md): Transforming data loaded by a dlt pipeline with pandas dataframes or arrow tables - [Transform data with SQL](/docs/dlt-ecosystem/transformations/sql.md): Transforming the data loaded by a dlt pipeline with the dlt SQL client - [Transform data with dbt](/docs/dlt-ecosystem/transformations/dbt.md): Transforming the data loaded by a dlt pipeline with dbt ## Data quality - [Data quality lifecycle](/docs/general-usage/data-quality-lifecycle.md): End-to-end data quality checks across the dlt pipeline lifecycle - [Ensuring data quality](/docs/general-usage/dataset-access/data-quality-dashboard.md): Monitoring and testing data quality ## Deploy > Snowflake - [Run dlt in Snowflake](/docs/walkthroughs/run-in-snowflake.md): Run dlt in Snowflake Native App - [dlt Connector App](/docs/walkthroughs/run-in-snowflake/database-connector-app.md): How to use the dlt Connector App ## Deploy > Orchestrators - [Deploy with GitHub Actions](/docs/walkthroughs/deploy-a-pipeline/deploy-with-github-actions.md): How to deploy a pipeline with GitHub Actions - [Deploy with Airflow and Google Composer](/docs/walkthroughs/deploy-a-pipeline/deploy-with-airflow-composer.md): How to deploy a pipeline with Airflow and Google Composer - [Deploy with Google Cloud Functions](/docs/walkthroughs/deploy-a-pipeline/deploy-with-google-cloud-functions.md): How to deploy a pipeline with Google Cloud Functions - [Deploy with Google Cloud Run](/docs/walkthroughs/deploy-a-pipeline/deploy-with-google-cloud-run.md): Step-by-step guide on deploying a pipeline with Google Cloud Run. - [Deploy with Kestra](/docs/walkthroughs/deploy-a-pipeline/deploy-with-kestra.md): How to deploy a pipeline with Kestra - [Deploy with Dagster](/docs/walkthroughs/deploy-a-pipeline/deploy-with-dagster.md): How to deploy a pipeline with Dagster - [Deploy with Prefect](/docs/walkthroughs/deploy-a-pipeline/deploy-with-prefect.md): How to deploy a pipeline with Prefect - [Deploy with Modal](/docs/walkthroughs/deploy-a-pipeline/deploy-with-modal.md): How to deploy a pipeline with Modal - [Deploy with Orchestra](/docs/walkthroughs/deploy-a-pipeline/deploy-with-orchestra.md): How to deploy a dlt pipeline on Orchestra ## Performance - [Optimizing dlt](/docs/reference/performance.md): Scale-up, parallelize and finetune dlt pipelines ## Reference - [Command Line Interface](/docs/reference/command-line-interface.md): Command line interface (CLI) full reference of dlt - [Telemetry](/docs/reference/telemetry.md): Anonymous usage information with dlt telemetry - [Frequently asked questions](/docs/reference/frequently-asked-questions.md): Questions asked frequently by users in technical help or github issues