Skip to main content
Version: devel

Getting started

dlt pacman

What is dlt?

dlt is an open-source Python library that loads data from various, often messy data sources into well-structured, live datasets. It offers a lightweight interface for extracting data from REST APIs, SQL databases, cloud storage, Python data structures, and many more.

dlt is designed to be easy to use, flexible, and scalable:

To get started with dlt, install the library using pip:

pip install dlt
tip

We recommend using a clean virtual environment for your experiments! Read the detailed instructions on how to set up one.

Load data with dlt from …

Use dlt's REST API source to extract data from any REST API. Define the API endpoints you’d like to fetch data from, the pagination method, and authentication, and dlt will handle the rest:

import dlt
from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
"client": {
"base_url": "https://api.example.com/",
"auth": {
"token": dlt.secrets["your_api_token"],
},
"paginator": {
"type": "json_response",
"next_url_path": "paging.next",
},
},
"resources": ["posts", "comments"],
})

pipeline = dlt.pipeline(
pipeline_name="rest_api_example",
destination="duckdb",
dataset_name="rest_api_data",
)

load_info = pipeline.run(source)

Follow the REST API source tutorial to learn more about the source configuration and pagination methods.

tip

If you'd like to try out dlt without installing it on your machine, check out the Google Colab demo.

Join the dlt community

  1. Give the library a ⭐ and check out the code on GitHub.
  2. Ask questions and share how you use the library on Slack.
  3. Report problems and make feature requests here.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.