Python Data Loading from mux
to motherduck
using dlt
Library
Join our Slack community or book a call with our support engineer Violetta.
This guide provides technical documentation on how to load data from mux
, a platform that simplifies the process of building video applications, to motherduck
, an in-process analytical database known for its fast performance and SQL dialect support. The process leverages dlt
, an open-source Python library, to facilitate the data transfer. For a comprehensive understanding of mux
, visit https://www.mux.com/. The guide will walk you through the steps of setting up and executing this data pipeline effectively.
dlt
Key Features
- Scalable Data Extraction:
dlt
offers scalable data extraction by leveraging iterators, chunking, and parallelization techniques. This approach allows for efficient processing of large datasets by breaking them down into manageable chunks. Learn more - Implicit Extraction DAGs:
dlt
incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. Learn more - Governance Support:
dlt
pipelines offer robust governance support through three key mechanisms: pipeline metadata utilization, schema enforcement and curation, and schema change alerts. Learn more - Schema Enforcement and Curation:
dlt
empowers users to enforce and curate schemas, ensuring data consistency and quality. Learn more - Schema Evolution:
dlt
enables proactive governance by alerting users to schema changes. When modifications occur in the source data’s schema,dlt
notifies stakeholders. Learn more
Getting started with your pipeline locally
0. Prerequisites
dlt
requires Python 3.8 or higher. Additionally, you need to have the pip
package manager installed, and we recommend using a virtual environment to manage your dependencies. You can learn more about preparing your computer for dlt in our installation reference.
1. Install dlt
First you need to install the dlt
library with the correct extras for MotherDuck
:
pip install "dlt[motherduck]"
The dlt
cli has a useful command to get you started with any combination of source and destination. For this example, we want to load data from Mux
to MotherDuck
. You can run the following commands to create a starting point for loading data from Mux
to MotherDuck
:
# create a new directory
mkdir mux_pipeline
cd mux_pipeline
# initialize a new pipeline with your source and destination
dlt init mux motherduck
# install the required dependencies
pip install -r requirements.txt
The last command will install the required dependencies for your pipeline. The dependencies are listed in the requirements.txt
:
dlt[motherduck]>=0.3.5
You now have the following folder structure in your project:
mux_pipeline/
├── .dlt/
│ ├── config.toml # configs for your pipeline
│ └── secrets.toml # secrets for your pipeline
├── mux/ # folder with source specific files
│ └── ...
├── mux_pipeline.py # your main pipeline script
├── requirements.txt # dependencies for your pipeline
└── .gitignore # ignore files for git (not required)