With 91% of dlt pipelines AI-written, learn Agentic Data Engineering in this free 1-hour course.
Adrian Brudaru
Agents don't hallucinate. They navigate without a map. Ontology engineering is how you build one, and why every team pulling humans out of the loop needs it now.
Adrian Brudaru
The dltHub AI Workbench gives Claude Code a structured workflow for building data pipelines. We put it to the test with a real geopolitical question.
Roshni Melwani
dlt handles schema evolution efficiently but silently. Here's how to read dlt's metadata and be informed of what's shifting in your pipeline.
Aman Gupta
A "Success" exit code only tells you the pipeline ran. Use `load_id` to join `_dlt_loads` with your source table and check if the data is actually fresh.
Aman Gupta
We're in an LLM-coding junior bubble. "It runs" isn't the senior bar. Lifecycle rigor and dependency management are.
Adrian Brudaru
The dlt AI Workbench transforms AI-generated "vibe coding" from an unmanaged process full of hidden risks into a mature engineering workflow that prioritizes security, current documentation, and persistent state by default.
Adrian Brudaru
Part of the [dltHub AI Workbench series](https://dlthub.com/blog/ai-workbench)
Adrian Brudaru
TL;DR: Cortex Code helps you work with data already in Snowflake. dltHub Pro gets data into Snowflake from any source, especially the ones no ETL tool covers. They operate at different layers of the stack and they are designed to hand off to each other.
Adrian Brudaru
Call it the MVC problem: minimum viable context. Too little and it hallucinates your domain. Too much and it drifts from your actual goal. The process has to be controlled.
Hiba Jamal
How are LLMs supposed to know the business logic of how you use Hubspot, Luma and Slack together? How are they supposed to know what a customer means to you?
Hiba Jamal
Today we are introducing the dltHub AI Workbench: an infrastructure layer for dltHub that makes AI-generated dlt pipelines trustworthy enough to run and deploy in production.
Matthaus Krzykowski
Stop PII leaks before they hit your warehouse. By using dlt and Pydantic to enforce data contracts, you can sanitize or quarantine sensitive fields the moment they’re ingested.
Aman Gupta
In this blog post, I will describe the actual, hard real world barriers that make your LLM setup collapse, and propose principles for making your systems work.
Adrian Brudaru
Add data quality gates to Microsoft Fabric with dlt. Validate schemas, catch bad records, and mask PII before data reaches your lakehouse and downstream analytics.
Rakesh Gupta
Production traces are scattered across databases, log aggregators, and storage buckets, and most of them aren't clean (input, output) pairs you can hand to a training job. This walkthrough shows how to build a dlt pipeline that extracts traces from any source, transforms them into structured conversation formats, and lands them as versioned Parquet on Hugging Face, ready for Distil Labs to generate synthetic training data and deliver a specialist model that beats the LLM you're running today.
Alena Astrakhantseva +1
From raw data to production ML: load, transform, embed, and publish curated datasets with declarative pipelines powered by dltHub.
Elvis Kahoro +2
Single-gate validation fails to decouple row-level syntax from batch-level semantics. Evolve from WAP to the AWAP protocol with this simple dlt tutorial to stop pipeline corruption at the source.
Roshni Melwani
Trying to force an LLM to reconstruct the 'world' using only a semantic layer is like trying to turn cheese back into milk. The information required to understand the original system was stripped away during the modeling process.
Adrian Brudaru
For the more classic data engineering crowd, here’s an explainer of how unstructured AI memory works, though the lens of what we know from working with structured data.
Adrian Brudaru
By upgrading only the generative model, we achieved a 3x accuracy boost but hit a hard ceiling, proving that not only LLMs are needed for good retrieval.
Aashish Nair
Remus Molnar
I didn't vibe-build a product. I wrote a messy scaffold that runs a pipeline, grabs the schema, and forces an agent to build a star schema. It works shockingly well.
Adrian Brudaru
Analyzing UFC greatness by building a full stack (dlt, dbt, Metabase) to transform raw fight stats into a data-driven search for the true GOAT.
Reshef Sharvit
Moved 5M rows from DuckDB to MySQL 3.7x faster, reducing time from 344s to 92s by switching from SQLAlchemy’s row-by-row path to Arrow + ADBC’s columnar pipeline.
Aman Gupta
We were told that democratization meant 'safety,' but all we got were expensive cages. The era of the SaaS hostage is ending; the era of the sovereign Builder has begun.
Adrian Brudaru
The “data is oil” era is over. With LLMs, data is plutonium: powerful, toxic. Shift left and secure the reactor with 5 quality pillars.
Adrian Brudaru
Our docs RAG was failing quietly. We stopped guessing and built a real-user evaluation: the first baseline we could actually measure and improve.
Aashish Nair
Adrian Brudaru
11 practical, copy-paste data quality recipes for dlt. From schema freezes to alerts, learn how to keep pipelines clean, safe, and production-ready
Aman Gupta
Start local with DuckLake, validate your data, then deploy to MotherDuck in minutes. Same pipeline, same code, just switch the destination.
Aman Gupta
Data contracts keep systems predictable by pairing clear rules with checks that catch bad data before it flows downstream.
Adrian Brudaru
Most LLM runs don’t fail. They converge fast, and the secret isn’t smarter models but better scaffolds that guide the work instead of forcing it.
Adrian Brudaru
Openflow and dltHub represent two distinct but valuable visions for the future of data ingestion.
Adrian Brudaru
This is, we’re told, the great democratization of data engineering. The tedious work is gone. The barrier to entry is gone. Everyone can now be a data engineer.
Adrian Brudaru
MotherDuck lands in Europe with serverless DuckDB warehousing. dlt adds DuckLake support, giving EU teams a fast, modern data stack.
Adrian Brudaru
SAP data is hard to extract. Dominik’s new Python connector replaces pyRFC, enabling faster, chunked ingestion into modern pipelines.
Mateusz Paździor
LLM leaders agree: the true win is "scaled mediocrity." We're empowering teams with good enough tools for massive, real-world impact.
Adrian Brudaru
For quick tasks, df.to_sql() is perfect. But for production pipelines, it quickly shows its limits when data volume, frequency, and schema change.
Adrian Brudaru
Learn how dlt automates SCD2 for nested JSON data without complex SQL headaches. Real BigQuery benchmarks show incremental loading cuts costs by 25-35%.
Aman Gupta
Emmanuel built a slim framework on top of dlt that levels up the vanilla Kafka source into a production-ready setup. Check it out 🚀
Aman Gupta
You want connectors, and you want them to be many, high quality and customisable? A man can dream? here’s our roadmap to making those dreams a reality, and how you can help us today.
Adrian Brudaru
We compared dlt and Sling for data ingestion performance, cost, and flexibility. See how they stack up and which might suit your data needs best.
Adrian Brudaru +2
Ajay Moorjani turned a deceptively simple JSON to Snowflake task into a rock solid pipeline using dlt, dbt, and Airflow, built in less than a coffee break.
Aman Gupta
Leveraging AI to build a dlt extract and load of coldplay data from spotify and visualize it in Visivo.
Jared Jesionek
Built another pipeline just to keep a dashboard alive? Then it broke again? Michael Shoemaker shows how dlt makes API pipelines fix themselves, no drama.
Adrian Brudaru
We’re excited to announce that we’re building dltHub, an LLM-native data engineering platform that enables any Python developer to build, run dlt pipelines, and deliver valuable end-user-ready reports.
Matthaus Krzykowski
LLM-native scaffolds for 1000+ APIs. The IKEA moment in data engineering is here. Build pipelines with LLMs, faster and cleaner.
Adrian Brudaru
Using dlt + Cognee, we take API docs from Slack, PayPal, and TicketMaster and built a knowledge graph.
Hiba Jamal
Dev takes Alena’s dlt course, then uses AI to build a WHOOP sleep-data pipeline, saving the data to Parquet, demonstrating that beginners can master pipelines quickly.
Roshni Melwani
We've been using LanceDB for months at dltHub to build AI systems more quickly. The same setup works locally and in the cloud. Handles structured and vector data in one place.
Adrian Brudaru
Mixing Spark, DuckDB, and Snowflake? Iceberg decouples data, Ibis decouples logic, run your analytics anywhere, without rewrites or vendor lock-in.
Adrian Brudaru
Taktile cut 70% of data loading costs by shifting ingestion to Iceberg via Lambda + dlt, keeping Snowflake for analytics. Smart layers, big savings.
Adrian Brudaru
Singer was Stitch's incomplete competitive response to Fivetran. Meltano completed what Stitch never intended to fully open source. dlt learned from both and built the fitting abstraction for pythonic data teams.
Adrian Brudaru
A side-by-side look at Fivetran and dlt, covering cost models, customization, and how each approach affects team workflows as your data needs evolve.
Adrian Brudaru
REST API integrations come with hidden costs, pagination, schema drift, rate limits. With dlt + Cursor, you skip the boilerplate and build pipelines in minutes, not days. Less code. Less chaos. More time to build.
Aman Gupta
A hands-on guide to combining dlt and Dagster for orchestrating multi-endpoint API ingestion pipelines, with assets materialized into DuckDB. Three patterns. One powerful workflow. Plus, a peek at the new CLI and DuckDB UI.
Jairus Martinez
Data engineering shouldn't require rewriting the same logic multiple times for different environments. dlt's dataset interface gives you one consistent way to work with your data, regardless of where it lives.
Adrian Brudaru
Ingesting to Databricks should be simple. With dlt, it finally is. No config files, no staging, just Python and go.
Aman Gupta
Vibe coding so clean, it will make your old code look bad.
Adrian Brudaru
Julian Alves builds reliable, simple data infrastructure. He partners with dlt to help companies create systems that deliver value, not burden.
Adrian Brudaru
dlt has grown from 1,000 to over 3,000 open-source users in just six months, with monthly downloads surpassing 1.4 million. This momentum reflects a growing demand for Python-native, modular, and AI-ready data tools — and dlt is building exactly that.
Matthaus Krzykowski
dlt started as a tool for handling JSON documents. It was meant for the average Python user that does not want to deal with creating and evolving schemas, SQL models, backends and data engineers that control them.
Marcin Rudolf
Let's stop reinventing connectors in isolation. Use LLMs to transform scattered integrations into shared, reusable solutions.
Adrian Brudaru
As Rakesh was exploring Fabric, dlt kept showing up in Rakesh's stack. Not by design, but because it just worked. Different projects, same ingestion layer, quietly doing its job.
Adrian Brudaru
I tried Vibe-coding a Singer tap (Pipedrive) into dlt and it worked, but it needed some user intervention.
Adrian Brudaru
Explore four ways to run dlt with Apache Airflow, from PythonOperators to KubernetesPods, and learn which setup scales best for clean, reliable pipelines.
Francesco Mucio
Building pipelines with AI isn’t one task, it's many. In this post, we explore how to split and test them individually, so failures are easier to diagnose and fix.
Adrian Brudaru
The Write. Audit. Publish. (WAP) framework brings discipline from software engineering: write in isolation, audit for correctness, quality, and compliance, publish with confidence. But can data engineering really follow suit? Let's discuss.
Aman Gupta
Modernisation at its finest, from trash to cutting edge in seconds. It works amazing, just give it a try, stop paying for tech debt
Adrian Brudaru
In this microblog + video we explore generating python pipelines (dlt REST API) from Airbyte low code yaml spec. tl;dr: it works well.
Adrian Brudaru
