Storage scales. Compute scales. Development drags. Let's fix that.

1. Why did we move to ELT, and why is that a problem?

ETL was hard to build - so we went ELT

About a decade back, the industry was centered around ETL - Java was fast, while databases were not. However, building these stacks required specialized engineers and getting into it was a very high learning curve. Pipelines took ages to create. Iteration cycles between teams were slow. It was clear: we couldn’t scale ETL fast enough to match data usage demand.

With Redshift launch promising scale in SQL, teams moved transformation into the hands of analysts or analytics engineers. Teams moved faster, worked more independently, and built pipelines without waiting on Java developers.

ELT is hard to maintain, time to fix it.

Switching Java + Mysql for Python + MPP SQL worked. That is, until the volume of data and transformations exploded. At the same time, requirements for its quality also increase, with the industry shifting from reporting + research to more ML and AI cases, requiring more compliance and traceability.

ELT made writing transformations easier, but managing them harder. There’s no local development, no pre-load validation, and no fast feedback loop. Engineers test in production, debug in the warehouse, and rerun expensive queries just to see if something works. Instead of scaling, teams are drowning in technical debt and slow systems. The faster they grow, the more fragile they become.

A bottleneck of pain

We fixed the build phase by shifting the burden after it. Now, scaling isn’t about data, but about how much pain teams can handle.

The solution? We need to rethink how we build, test, and deploy them before the system collapses under its own weight.

2. The problem: ELT management doesn’t scale

ELT was supposed to make data faster, cheaper, and easier to scale. Instead, teams are drowning in repetitive work, debugging costs are skyrocketing, and companies are getting less value from their most skilled people.

Engineers are stuck doing repetitive work instead of getting better at engineering

There’s no local testing, no pre-load validation. Every mistake is a budget hit, every retry is a 20min waiting time.

Engineers spend their days rerunning queries, fixing brittle pipelines, and firefighting broken transformations. There’s no time to improve skills, optimise workflows, or work on higher-value problems. They aren’t learning, they’re maintaining. Instead of growing, they stagnate. Stuck in reactive work that doesn’t advance their careers, their skills fall behind, reducing their value to the market or their employer.

Companies are paying More for Less

The industry scaled faster than its tools. We moved transformations to warehouses because it was the simplest path forward. But what worked at 10 pipelines is breaking at 10,000.

The more pipelines you add, the slower everything gets. Maintenance ties up the workforce and Instead of unlocking value from data teams, companies pay premium cloud costs for engineers to troubleshoot, rerun, and maintain unstable pipelines.

• No isolation—every mistake runs live.
• No visibility—queries fail unpredictably.
• No safety net—debugging is just rerunning and hoping.

ELT won’t scale any further, but the Hype machine yaps on

Every time pipelines slow down, the industry prescribes the same cure: more tools, more orchestration, more dashboards to manage the dashboards. But no one is fixing the root problem: the way we work.

More tests mean more vendor costs. Every validation query, every debug run, every pipeline retry: billable events.

So, either we rethink how we build, test, and deploy transformations, or we keep layering new tools on top of a broken system, hoping this time, it’ll work. Spoiler alert: it won’t.

3. The Fix: A Staging Layer for Data Transformations

We scaled storage. We scaled compute. But we never scaled development.

Software engineers have staging, CI/CD, and instant feedback loops. Data engineers? They push to production and pray.

Josh Wills said it best in his video “Shift yourself left”:

"Do you know how much money it would cost us if every pull request had to load data into Snowflake and run tests? That's insane."

Exactly. The warehouse isn't built for testing. Debugging pipelines at full scale isn’t just bad practice: it’s a revenue stream for vendors, and that’s why you'll find commercial "discourse" on the matter.

We don’t need more dashboards or governance layers. We need a staging layer for data transformations.

Introducing dlt+ Staging: The Missing Development Layer for ELT

dlt+ Staging provides a portable compute engine for development, testing, contracts and validation, without touching the warehouse.

• Run transformations locally → No more waiting for warehouse queries.
• Validate before loading → Catch schema mismatches before they break dashboards.
• Test without burning cloud costs → In-memory execution means no more wasted compute.

ELT made prototyping easy. dlt+ makes it production-ready.

Now, it’s time to stop prototyping and start building for production.

Software engineers don’t deploy untested code. Data engineers shouldn’t deploy untested pipelines.

Stop Debugging in Production! → Join dlt+ Staging early access