Productionize Python ETL Scripts: Migrate to dlt & dltHub

Your Python ETL scripts were the right call. Here's how to keep it that way.

You did the smart thing. Your team needed data from a few APIs. You wrote an integration.

That’s already a good place to be in - you retain control over your pipelines instead of abandoning the simplest and most foundational part of the data pipeline to a vendor to own.

But from here there is a lot of opportunity for improvement. Namely, what makes the difference between a hobbyist and a professional is the failure modes. How often does it happen? what breaks the pipeline? Does it recover on transient errors? when it breaks, what’s the blast radius? how long until we’re back, and at what cost? And most importantly, who can even do it?

For a DIY pipeline, the answer often is

• Schema changes break your back: It loads data without typing it or it relies on partial depth or non deterministic typing that’s not durable from run to run
• it often doesn’t handle incremental state - resuming where you left off on the last run, or if it does, it’s not reusable across other pipelines due to different requirements or has high cost consequences such as selecting a max() over a big table.
• It can be maintained by the author, and maybe a few other seniors, but not by the vast majority of the team. if the author is on vacation and the pipeline breaks… ugh…
• transient errors such as rate limits or network issues are not handled and cause failure.
• recovery is often ugly and slow. Idempotency and atomicity are often broken leading to duplicates or partial loads, which needs slow, careful manual cleanup before service can resume

From DI-WHY to DIY: That’s why data engineers love dlt.

If you’ve worked in this field, you know this to be true. Myself, i did over 10 years in the data engineering trenches and that’s why I started dlthub: to solve all the above and turn data loading into something anyone on the team can do with joy.

We created dlt, “data load tool”, an open source python library that’s loved by data engineers.

As a commercial offering, we built the data engineering endgame: An infrastructure you can build, run, and maintain your pipelines on completely agentically. This doesn’t replace your data team - human judgment, decision, responsibility are core alongside dlthub’s capabilities. What it does replace is the need to maintain an operating system of capabilities and tools around your pipelines.

What does dltHub add or replace?

1. A place to schedule and run your pipelines: dltHub provides serverless sandboxes billed on compute time as a means to bill for the infrastructure and software. It’s a similar usage model to clouds (AWS, GCP) or some of your favorite products like snowflake or databricks. Read more about how much data this infrastructure moves for $1 here BENCHMARK dlthub
2. Senior data engineer heavy lifting work: dlt upgrades your python scripts to the best possible standard, with features like schema evolution, state, parallelism and hardware management at the flip of a switch. Your data engineer can now reclaim their sanity and time and focus on something better. Case study.
3. The team’s learning curve. The dlthub agent knows how to best use dlt and build pipelines, and ensure it develops the pipeline for best practice implementation while following a best-practice process such as not reading your credentials. Everyone on the team can get it right the first time. Read the eval blog.
4. The deployment operations: Deploying code is a process that needs to be invented, implemented and learned learned by everyone. In many organisations it becomes a complicated bottleneck. For dltHub, you just ask your agent who then handles testing and the actual deployment.
5. The maintenance operations: Reading logs and tracking down breakages is specialist work. It’s tedious and often has to be done under pressure. On dltHub, your agent does the heavy lifting, finds the issues, tests a fix and proposes it before waiting for your confirmation for deployment. Watch 3min maintenance video to see the workflow.
6. An ecosystem that needs to be set up and stitched together: dltHub comes with a transformation framework and a skill for creating agent-native canonical models. It also comes with apps like marimo and streamlit that enable you to create visualisations or interactive apps.
7. Skills, context, and their maintenance: Creating LLM context is the new software engineering, so don’t fall for the same trap. Skills need maintenance and being kept in sync with stack changes. We do this for the dltHub context. A continuous context works across the entire stack, enabling an agent to propagate changes between ingestion and usage. Read more.

dlt is observed to be 99.83 reliable in the wild and a back of envelope calculation points to a TCO per pipeline in the $100/year range. Read the TCO math here.

• Try dltHub free for 2 weeks: Ask the agent to convert and run your most problematic pipeline on.
• Don't want to do it yourself? Book a migration — we'll move it for you.