Skip to main content

Version: 1.5.0 (latest)

Transforming data

If you'd like to transform your data after a pipeline load, you have 3 options available to you:

Using dbt - dlt provides a convenient dbt wrapper to make integration easier.
Using the dlt SQL client - dlt exposes an SQL client to transform data on your destination directly using SQL.
Using Python with DataFrames or Arrow tables - you can also transform your data using Arrow tables and DataFrames in Python.

If you need to preprocess some of your data before it is loaded, you can learn about strategies to:

This is particularly useful if you are trying to remove data related to PII or other sensitive data, you want to remove columns that are not needed for your use case or you are using a destination that does not support certain data types in your source data.

Learn more

🗃️ Transforming data with dbt

2 items

📄️ Transforming data in Python with Arrow tables or DataFrames

Transforming data loaded by a dlt pipeline with pandas dataframes or arrow tables

📄️ Transforming data with SQL

Transforming the data loaded by a dlt pipeline with the dlt SQL client

📄️ Renaming columns

Renaming columns by replacing the special characters

📄️ Pseudonymizing columns

Pseudonymizing (or anonymizing) columns by replacing the special characters

📄️ Removing columns

Removing columns by passing a list of column names

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.

The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.