Postgres
Install dlt with PostgreSQLโ
To install the dlt library with PostgreSQL dependencies, run:
pip install "dlt[postgres]"
Setup Guideโ
1. Initialize a project with a pipeline that loads to Postgres by running:
dlt init chess postgres
2. Install the necessary dependencies for Postgres by running:
pip install -r requirements.txt
This will install dlt with the postgres
extra, which contains the psycopg2
client.
3. After setting up a Postgres instance and psql
/ query editor, create a new database by running:
CREATE DATABASE dlt_data;
Add the dlt_data
database to .dlt/secrets.toml
.
4. Create a new user by running:
CREATE USER loader WITH PASSWORD '<password>';
Add the loader
user and <password>
password to .dlt/secrets.toml
.
5. Give the loader
user owner permissions by running:
ALTER DATABASE dlt_data OWNER TO loader;
You can set more restrictive permissions (e.g., give user access to a specific schema).
6. Enter your credentials into .dlt/secrets.toml
.
It should now look like this:
[destination.postgres.credentials]
database = "dlt_data"
username = "loader"
password = "<password>" # replace with your password
host = "localhost" # or the IP address location of your database
port = 5432
connect_timeout = 15
You can also pass a database connection string similar to the one used by the psycopg2
library or SQLAlchemy. The credentials above will look like this:
# keep it at the top of your toml file! before any section starts
destination.postgres.credentials="postgresql://loader:<password>@localhost/dlt_data?connect_timeout=15"
To pass credentials directly, use the explicit instance of the destination
pipeline = dlt.pipeline(
pipeline_name='chess',
destination=dlt.destinations.postgres("postgresql://loader:<password>@localhost/dlt_data"),
dataset_name='chess_data'
)
Write dispositionโ
All write dispositions are supported.
If you set the replace
strategy to staging-optimized
, the destination tables will be dropped and replaced by the staging tables.
Data loadingโ
dlt
will load data using large INSERT VALUES statements by default. Loading is multithreaded (20 threads by default).
Fast loading with arrow tables and csvโ
You can use arrow tables and csv to quickly load tabular data. Pick the csv
loader file format
like below
info = pipeline.run(arrow_table, loader_file_format="csv")
In the example above arrow_table
will be converted to csv with pyarrow and then streamed into postgres with COPY command. This method skips the regular
dlt
normalizer used for Python objects and is several times faster.
Supported file formatsโ
- insert-values is used by default.
- csv is supported
Supported column hintsโ
postgres
will create unique indexes for all columns with unique
hints. This behavior may be disabled.
Additional destination optionsโ
The Postgres destination creates UNIQUE indexes by default on columns with the unique
hint (i.e., _dlt_id
). To disable this behavior:
[destination.postgres]
create_indexes=false
dbt supportโ
This destination integrates with dbt via dbt-postgres.
Syncing of dlt
stateโ
This destination fully supports dlt state sync.
Additional Setup guidesโ
- Load data from Box Platform API to Azure Cosmos DB in python with dlt
- Load data from Shopify to YugabyteDB in python with dlt
- Load data from Chess.com to YugabyteDB in python with dlt
- Load data from Rest API to YugabyteDB in python with dlt
- Load data from PostgreSQL to YugabyteDB in python with dlt
- Load data from Airtable to EDB BigAnimal in python with dlt
- Load data from Google Sheets to Neon Serverless Postgres in python with dlt
- Load data from Spotify to AlloyDB in python with dlt
- Load data from Google Cloud Storage to Timescale in python with dlt
- Load data from IBM Db2 to Timescale in python with dlt
- Load data from Oracle Database to Timescale in python with dlt
- Load data from Apple App-Store Connect to CockroachDB in python with dlt
- Load data from Slack to Timescale in python with dlt
- Load data from HubSpot to Supabase in python with dlt
- Load data from Jira to EDB BigAnimal in python with dlt
- Load data from Notion to YugabyteDB in python with dlt
- Load data from Shopify to Supabase in python with dlt
- Load data from Chess.com to Supabase in python with dlt
- Load data from Rest API to Supabase in python with dlt
- Load data from SAP HANA to Azure Cosmos DB in python with dlt
- Load data from Rest API to Timescale in python with dlt
- Load data from Pipedrive to Azure Cosmos DB in python with dlt
- Load data from MySQL to Neon Serverless Postgres in python with dlt
- Load data from Google Cloud Storage to Azure Cosmos DB in python with dlt
- Load data from Box Platform API to Neon Serverless Postgres in python with dlt
- Load data from PostgreSQL to Supabase in python with dlt
- Load data from PostgreSQL to Neon Serverless Postgres in python with dlt
- Load data from MongoDB to EDB BigAnimal in python with dlt
- Load data from Azure Cloud Storage to EDB BigAnimal in python with dlt
- Load data from HubSpot to Neon Serverless Postgres in python with dlt
- Load data from MySQL to Azure Cosmos DB in python with dlt
- Load data from Chess.com to AlloyDB in python with dlt
- Load data from HubSpot to YugabyteDB in python with dlt
- Load data from Zendesk to Azure Cosmos DB in python with dlt
- Load data from Notion to Supabase in python with dlt
- Load data from GitHub to Neon Serverless Postgres in python with dlt
- Load data from Pipedrive to AlloyDB in python with dlt
- Load data from Box Platform API to EDB BigAnimal in python with dlt
- Load data from SAP HANA to CockroachDB in python with dlt
- Load data from GitHub to YugabyteDB in python with dlt
- Load data from Apple App-Store Connect to YugabyteDB in python with dlt
- Load data from Sentry to CockroachDB in python with dlt
- Load data from MongoDB to Supabase in python with dlt
- Load data from Jira to Supabase in python with dlt
- Load data from PostgreSQL to EDB BigAnimal in python with dlt
- Load data from Shopify to Timescale in python with dlt
- Load data from Google Analytics to PostgreSQL in python with dlt
- Load data from Box Platform API to AlloyDB in python with dlt
- Load data from Google Analytics to CockroachDB in python with dlt
- Load data from Slack to Neon Serverless Postgres in python with dlt
- Load data from The Local Filesystem to YugabyteDB in python with dlt
- Load data from Spotify to Timescale in python with dlt
- Load data from Oracle Database to Supabase in python with dlt
- Load data from Apple App-Store Connect to EDB BigAnimal in python with dlt
- Load data from Apple App-Store Connect to Supabase in python with dlt
- Load data from MongoDB to YugabyteDB in python with dlt
- Load data from Oracle Database to AlloyDB in python with dlt
- Load data from Azure Cloud Storage to PostgreSQL in python with dlt
- Load data from Jira to AlloyDB in python with dlt
- Load data from AWS S3 to PostgreSQL in python with dlt
- Load data from GitHub to Supabase in python with dlt
- Load data from The Local Filesystem to EDB BigAnimal in python with dlt
- Load data from The Local Filesystem to Supabase in python with dlt
- Load data from GitHub to AlloyDB in python with dlt
- Load data from Box Platform API to CockroachDB in python with dlt
- Load data from Jira to YugabyteDB in python with dlt
- Load data from Oracle Database to YugabyteDB in python with dlt
- Load data from Mux to Azure Cosmos DB in python with dlt
- Load data from Chess.com to Neon Serverless Postgres in python with dlt
- Load data from Jira to Neon Serverless Postgres in python with dlt
- Load data from Stripe to CockroachDB in python with dlt
- Load data from Stripe to EDB BigAnimal in python with dlt
- Load data from Mux to YugabyteDB in python with dlt
- Load data from Zendesk to EDB BigAnimal in python with dlt
- Load data from AWS S3 to AlloyDB in python with dlt
- Load data from Oracle Database to Azure Cosmos DB in python with dlt
- Load data from Jira to Azure Cosmos DB in python with dlt
- Load data from Sentry to Timescale in python with dlt
- Load data from Chargebee to Timescale in python with dlt
- Load data from IBM Db2 to CockroachDB in python with dlt
- Load data from Zendesk to Neon Serverless Postgres in python with dlt
- Load data from GitHub to CockroachDB in python with dlt
- Load data from MongoDB to Azure Cosmos DB in python with dlt
- Load data from Spotify to PostgreSQL in python with dlt
- Load data from Chargebee to PostgreSQL in python with dlt
- Load data from MongoDB to CockroachDB in python with dlt
- Load data from The Local Filesystem to CockroachDB in python with dlt
- Load data from Mux to Supabase in python with dlt
- Load data from Mux to CockroachDB in python with dlt
- Load data from Sentry to PostgreSQL in python with dlt
- Load data from The Local Filesystem to Azure Cosmos DB in python with dlt
- Load data from Azure Cloud Storage to AlloyDB in python with dlt
- Load data from GitHub to EDB BigAnimal in python with dlt
- Load data from Shopify to AlloyDB in python with dlt
- Load data from Google Cloud Storage to CockroachDB in python with dlt
- Load data from AWS S3 to CockroachDB in python with dlt
- Load data from Apple App-Store Connect to Azure Cosmos DB in python with dlt
- Load data from Slack to PostgreSQL in python with dlt
- Load data from Shopify to CockroachDB in python with dlt
- Load data from GitHub to Azure Cosmos DB in python with dlt
- Load data from MySQL to Timescale in python with dlt
- Load data from Airtable to CockroachDB in python with dlt
- Load data from Zendesk to YugabyteDB in python with dlt
- Load data from HubSpot to Azure Cosmos DB in python with dlt
- Load data from Salesforce to Timescale in python with dlt
- Load data from Spotify to CockroachDB in python with dlt
- Load data from Stripe to PostgreSQL in python with dlt
- Load data from Microsoft SQL Server to PostgreSQL in python with dlt
- Load data from AWS S3 to Neon Serverless Postgres in python with dlt
- Load data from Mux to Timescale in python with dlt
- Load data from SAP HANA to Timescale in python with dlt
- Load data from Azure Cloud Storage to CockroachDB in python with dlt
- Load data from Microsoft SQL Server to EDB BigAnimal in python with dlt
- Load data from MySQL to YugabyteDB in python with dlt
- Load data from Google Sheets to EDB BigAnimal in python with dlt
- Load data from Google Cloud Storage to YugabyteDB in python with dlt
- Load data from HubSpot to EDB BigAnimal in python with dlt
- Load data from SAP HANA to Neon Serverless Postgres in python with dlt
- Load data from Box Platform API to Supabase in python with dlt
- Load data from Pipedrive to YugabyteDB in python with dlt
- Load data from SAP HANA to YugabyteDB in python with dlt
- Load data from Salesforce to PostgreSQL in python with dlt
- Load data from Pipedrive to Neon Serverless Postgres in python with dlt
- Load data from Google Analytics to EDB BigAnimal in python with dlt
- Load data from Apple App-Store Connect to AlloyDB in python with dlt
- Load data from Zendesk to Supabase in python with dlt
- Load data from Notion to Azure Cosmos DB in python with dlt
- Load data from Google Sheets to PostgreSQL in python with dlt
- Load data from Sentry to Neon Serverless Postgres in python with dlt
- Load data from Pipedrive to Timescale in python with dlt
- Load data from MySQL to Supabase in python with dlt
- Load data from Stripe to AlloyDB in python with dlt
- Load data from IBM Db2 to PostgreSQL in python with dlt
- Load data from MongoDB to Neon Serverless Postgres in python with dlt
- Load data from Oracle Database to CockroachDB in python with dlt
- Load data from Google Cloud Storage to Supabase in python with dlt
- Load data from PostgreSQL to Azure Cosmos DB in python with dlt
- Load data from Jira to CockroachDB in python with dlt
- Load data from Google Analytics to AlloyDB in python with dlt
- Load data from Chess.com to Azure Cosmos DB in python with dlt
- Load data from Rest API to Azure Cosmos DB in python with dlt
- Load data from Chess.com to Timescale in python with dlt
- Load data from Shopify to Azure Cosmos DB in python with dlt
- Load data from AWS S3 to Timescale in python with dlt
- Load data from Notion to EDB BigAnimal in python with dlt
- Load data from SAP HANA to Supabase in python with dlt
- Load data from The Local Filesystem to AlloyDB in python with dlt
- Load data from Box Platform API to YugabyteDB in python with dlt
- Load data from Airtable to PostgreSQL in python with dlt
- Load data from Pipedrive to Supabase in python with dlt
- Load data from Google Sheets to AlloyDB in python with dlt
- Load data from Azure Cloud Storage to Timescale in python with dlt
- Load data from MongoDB to Timescale in python with dlt
- Load data from Apple App-Store Connect to Neon Serverless Postgres in python with dlt
- Load data from PostgreSQL to AlloyDB in python with dlt
- Load data from Chess.com to CockroachDB in python with dlt
- Load data from Salesforce to Azure Cosmos DB in python with dlt
- Load data from Box Platform API to Timescale in python with dlt
- Load data from Shopify to Neon Serverless Postgres in python with dlt
- Load data from Oracle Database to Neon Serverless Postgres in python with dlt
- Load data from Rest API to EDB BigAnimal in python with dlt
- Load data from Stripe to Azure Cosmos DB in python with dlt
- Load data from HubSpot to PostgreSQL in python with dlt
- Load data from Mux to Neon Serverless Postgres in python with dlt
- Load data from Microsoft SQL Server to Azure Cosmos DB in python with dlt
- Load data from Azure Cloud Storage to Neon Serverless Postgres in python with dlt
- Load data from Microsoft SQL Server to CockroachDB in python with dlt
- Load data from Slack to CockroachDB in python with dlt
- Load data from Salesforce to CockroachDB in python with dlt
- Load data from Chargebee to CockroachDB in python with dlt
- Load data from Google Cloud Storage to EDB BigAnimal in python with dlt
- Load data from PostgreSQL to PostgreSQL in python with dlt
- Load data from Rest API to AlloyDB in python with dlt
- Load data from IBM Db2 to EDB BigAnimal in python with dlt
- Load data from Oracle Database to EDB BigAnimal in python with dlt
- Load data from Chess.com to PostgreSQL in python with dlt
- Load data from Rest API to PostgreSQL in python with dlt
- Load data from Slack to EDB BigAnimal in python with dlt
- Load data from Shopify to PostgreSQL in python with dlt
- Load data from IBM Db2 to AlloyDB in python with dlt
- Load data from PostgreSQL to CockroachDB in python with dlt
- Load data from Airtable to Azure Cosmos DB in python with dlt
- Load data from Jira to Timescale in python with dlt
- Load data from Notion to PostgreSQL in python with dlt
- Load data from Salesforce to Neon Serverless Postgres in python with dlt
- Load data from MySQL to CockroachDB in python with dlt
- Load data from Google Sheets to Azure Cosmos DB in python with dlt
- Load data from Google Sheets to CockroachDB in python with dlt
- Load data from Airtable to Timescale in python with dlt
- Load data from IBM Db2 to Azure Cosmos DB in python with dlt
- Load data from Rest API to CockroachDB in python with dlt
- Load data from Notion to Neon Serverless Postgres in python with dlt
- Load data from Jira to PostgreSQL in python with dlt
- Load data from Google Analytics to Supabase in python with dlt
- Load data from AWS S3 to YugabyteDB in python with dlt
- Load data from The Local Filesystem to Neon Serverless Postgres in python with dlt
- Load data from Azure Cloud Storage to YugabyteDB in python with dlt
- Load data from MongoDB to PostgreSQL in python with dlt
- Load data from The Local Filesystem to Timescale in python with dlt
- Load data from Rest API to Neon Serverless Postgres in python with dlt
- Load data from Oracle Database to PostgreSQL in python with dlt
- Load data from MongoDB to AlloyDB in python with dlt
- Load data from The Local Filesystem to PostgreSQL in python with dlt
- Load data from Spotify to EDB BigAnimal in python with dlt
- Load data from Google Analytics to YugabyteDB in python with dlt
- Load data from Sentry to Azure Cosmos DB in python with dlt
- Load data from Microsoft SQL Server to AlloyDB in python with dlt
- Load data from Apple App-Store Connect to PostgreSQL in python with dlt
- Load data from GitHub to PostgreSQL in python with dlt
- Load data from AWS S3 to Supabase in python with dlt
- Load data from Slack to Azure Cosmos DB in python with dlt
- Load data from Apple App-Store Connect to Timescale in python with dlt
- Load data from Azure Cloud Storage to Supabase in python with dlt
- Load data from Spotify to Azure Cosmos DB in python with dlt
- Load data from Google Cloud Storage to Neon Serverless Postgres in python with dlt
- Load data from Chargebee to Azure Cosmos DB in python with dlt
- Load data from Sentry to AlloyDB in python with dlt
- Load data from Shopify to EDB BigAnimal in python with dlt
- Load data from PostgreSQL to Timescale in python with dlt
- Load data from Chargebee to AlloyDB in python with dlt
- Load data from Spotify to Neon Serverless Postgres in python with dlt
- Load data from GitHub to Timescale in python with dlt
- Load data from Chargebee to YugabyteDB in python with dlt
- Load data from Spotify to YugabyteDB in python with dlt
- Load data from Slack to YugabyteDB in python with dlt
- Load data from Zendesk to AlloyDB in python with dlt
- Load data from Sentry to YugabyteDB in python with dlt
- Load data from IBM Db2 to Neon Serverless Postgres in python with dlt
- Load data from Google Analytics to Azure Cosmos DB in python with dlt
- Load data from Chargebee to EDB BigAnimal in python with dlt
- Load data from Mux to PostgreSQL in python with dlt
- Load data from Sentry to EDB BigAnimal in python with dlt
- Load data from Chargebee to Supabase in python with dlt
- Load data from Spotify to Supabase in python with dlt
- Load data from Stripe to Neon Serverless Postgres in python with dlt
- Load data from AWS S3 to Azure Cosmos DB in python with dlt
- Load data from Stripe to Timescale in python with dlt
- Load data from Azure Cloud Storage to Azure Cosmos DB in python with dlt
- Load data from Slack to Supabase in python with dlt
- Load data from Sentry to Supabase in python with dlt
- Load data from Zendesk to Timescale in python with dlt
- Load data from IBM Db2 to YugabyteDB in python with dlt
- Load data from Microsoft SQL Server to Supabase in python with dlt
- Load data from Google Analytics to Neon Serverless Postgres in python with dlt
- Load data from Zendesk to CockroachDB in python with dlt
- Load data from AWS S3 to EDB BigAnimal in python with dlt
- Load data from Chess.com to EDB BigAnimal in python with dlt
- Load data from HubSpot to AlloyDB in python with dlt
- Load data from Google Sheets to YugabyteDB in python with dlt
- Load data from Notion to Timescale in python with dlt
- Load data from Stripe to Supabase in python with dlt
- Load data from Airtable to YugabyteDB in python with dlt
- Load data from Box Platform API to PostgreSQL in python with dlt
- Load data from Salesforce to Supabase in python with dlt
- Load data from Airtable to AlloyDB in python with dlt
- Load data from MySQL to AlloyDB in python with dlt
- Load data from Mux to AlloyDB in python with dlt
- Load data from Pipedrive to EDB BigAnimal in python with dlt
- Load data from Slack to AlloyDB in python with dlt
- Load data from SAP HANA to AlloyDB in python with dlt
- Load data from Google Cloud Storage to AlloyDB in python with dlt
- Load data from Notion to CockroachDB in python with dlt
- Load data from MySQL to PostgreSQL in python with dlt
- Load data from Google Sheets to Timescale in python with dlt
- Load data from HubSpot to CockroachDB in python with dlt
- Load data from IBM Db2 to Supabase in python with dlt
- Load data from HubSpot to Timescale in python with dlt
- Load data from Microsoft SQL Server to YugabyteDB in python with dlt
- Load data from Pipedrive to CockroachDB in python with dlt
- Load data from Stripe to YugabyteDB in python with dlt
- Load data from Airtable to Neon Serverless Postgres in python with dlt
- Load data from Zendesk to PostgreSQL in python with dlt
- Load data from Google Analytics to Timescale in python with dlt
- Load data from Google Sheets to Supabase in python with dlt
- Load data from Salesforce to YugabyteDB in python with dlt
- Load data from SAP HANA to PostgreSQL in python with dlt
- Load data from Microsoft SQL Server to Neon Serverless Postgres in python with dlt
- Load data from MySQL to EDB BigAnimal in python with dlt
- Load data from Salesforce to EDB BigAnimal in python with dlt
- Load data from Salesforce to AlloyDB in python with dlt
- Load data from Airtable to Supabase in python with dlt
- Load data from Pipedrive to PostgreSQL in python with dlt
- Load data from Chargebee to Neon Serverless Postgres in python with dlt
- Load data from Mux to EDB BigAnimal in python with dlt
- Load data from Notion to AlloyDB in python with dlt
- Load data from SAP HANA to EDB BigAnimal in python with dlt
- Load data from Google Cloud Storage to PostgreSQL in python with dlt
- Load data from Microsoft SQL Server to Timescale in python with dlt