Code Examples

A list of comprehensive code examples that teach you how to solve real-world problems.

📄️ Export Arize Phoenix Telemetry

Export traces, spans, and evaluations from Arize Phoenix to your data lake

📄️ Backfilling in chunks

Learn how to backfill in chunks of defined size

📄️ Run chess pipeline in production

Learn how run chess pipeline in production

📄️ Load mysql table with ConnectorX & Arrow

Load data from sql queries fast with connector x and arrow tables

📄️ Use custom yaml file for config and secrets

We show how to keep configuration in yaml file with switchable profiles and simple templates

📄️ Custom destination with BigQuery

Learn how use the custom destination to load to bigquery and use credentials

📄️ Custom destination with LanceDB

Learn how use the custom destination to load to LanceDB.

📄️ Create and use own naming convention

We demonstrate how to create naming conventions that allow UNICODE letters and never generate collisions

📄️ Data masking for SQL database sources

Learn how to mask sensitive columns using closures that work with all sql_database backends

📄️ Google Sheets minimal example

Learn how work with Google services

📄️ Load Zendesk tickets incrementally

Learn how do incremental loading in consecutive runs

📄️ Export Langfuse Observability Data

Export traces, evaluations, datasets, and cost data from Langfuse to your data lake

📄️ Export Logfire Telemetry

Incrementally export traces and metrics from Pydantic Logfire to your data lake

📄️ Control nested MongoDB data

Learn how control nested data

📄️ Backfill to Filesystem with partial replace

Load chess game data from Chess.com into a filesystem destination, while deleting old backfill files.

📄️ Load PDFs to Weaviate

Extract text from PDF and load it into a vector database

📄️ Load from Postgres to Postgres faster

Load data fast from Postgres to Postgres with ConnectorX & Arrow export as Parquet, normalizing and exporting as DuckDB, and attaching it to Postgres for bigger Postgres tables (GBs)

📄️ Propagate primary_key from root to nested tables

Learn how to propagate any column to nested tables

📄️ Similarity searching with Qdrant

Learn how to use the dlt source, Zendesk and dlt destination, Qdrant to conduct a similarity search on your tickets data.

📄️ Pokemon details in parallel using transformers

Learn how to use dlt transformers and how to speed up your loads with parallelism

📄️ Dispatch stream of events to multiple tables in DuckDB

Learn how to efficiently dispatch a stream of GitHub events, categorized by event type, to different tables in DuckDB

📄️ Moving from local to production

Share a local dataset by moving it to BigQuery

📄️ Create new destination

How to create a new dlt destination

📄️ Import ticket data from Zendesk API to Weaviate

How to Import Ticket Data from Zendesk API to Weaviate