From Commoditization to Democratization: Building the Data Platforms of Tomorrow

Hi, I’m Adrian, data engineer and co-founder of dlt-hub. I’ve spent years in the trenches of data engineering designing pipelines, helping teams scale, and wrestling with the so-called “convenience” of commoditised tools. If you’re a data engineer like me, you’ve probably felt it too: moving data still feels way harder than it should (or, at least until dlt came to be).

And I think I know why.

Let’s talk about commoditisation and how it promised simplicity but brought limitations. How it promised to save us time but cost us more in maintenance. How it trapped us into tools that just aren’t designed with engineers in mind.

But here’s the good news: there’s a way out. Let me tell you why I think the data community, along with tools like dlt (and dlt+) can redefine how we work with data.

What are Commoditised tools?

Commoditised tools are those that offer standardised, easy-to-use solutions but often come with hidden costs, rigidity, or long-term limitations. They promise plug-and-play simplicity but can lead to vendor lock-in, high costs as data scales, or lack of flexibility for custom workflows.

Examples in Data Engineering: The "connector catalogs"

• ETL as a Service (e.g., Fivetran, Stitch) – Easy to start but costly at scale, with limited customisability.
• No-code/Low-code Data Integration (e.g., Airbyte Cloud, Matillion) – Great for a fast start but may not fit well in complex data ecosystems.

The Problem: Why does Moving data still feel so hard?

Picture this:

You’ve just finished setting up your pipeline. Maybe it’s an open-source connector, quick to deploy but not quite polished, so you spend hours adding fixes. Or maybe it’s a SaaS tool, effortless at first, until you hit the pricing wall.

Or worse, you went DI(-WH)Y. Hand-coded ETL scripts that worked beautifully on day one but now require constant patching to handle API changes, schema drift, gotchas, or scaling bottlenecks. It started simple. Now it’s a full-time maintenance job.

The reality is, commoditisation, whether OSS, SaaS, or DIY, hides complexity. You pay for it later.

• With OSS connector catalogs, the cost is debugging brittle connectors and adding endless workarounds.
• With SaaS, it’s exponential fees and vendor lock-in as your data scales.
• With custom scripts, you become the maintenance department for your team.

Meanwhile, the knowledge burden grows: incremental loads, schema evolution, PII anonymisation, governance… it’s exhausting.

This isn’t about skill or dedication. It’s about systems that bog us down and chew us up, leading to burnouts and ragequits.

Escaping the Commoditisation Trap

The antidote to commoditisation is democratisation.

So, what’s the difference?

• Commoditisation simplifies the surface but hides complexity beneath. It’s “easy” to start but becomes brittle, expensive, or unwieldy over time.
• Democratisation, on the other hand, redistributes control. It doesn’t remove complexity, it makes it accessible and manageable, empowering teams to build systems their way.

The Evolution: Monopoly → Commoditisation → Democratisation

Industries evolve in three stages:

• Monopoly: A single entity dominates with expensive, exclusive solutions.
• Commoditisation: Competitors emerge, simplifying access but sacrificing flexibility and scalability.
• Democratisation: Open, transparent tools shift power to users, enabling them to fully own and adapt systems to their needs.

In data engineering, we’ve lived through monopoly (Informatica), commoditisation (tools like Fivetran), and now… democratisation. Open-source tools like dlt are putting engineers back in control.

How to spot true democratisation

Democratized tools empower users by offering transparency, flexibility, collaboration, and cost predictability.

1. Transparency means you’re never guessing what’s happening behind the scenes.
2. Flexibility ensures the tools work for you, not the other way around.
3. Collaboration empowers everyone on your team to contribute meaningfully.
4. Cost predictability and easy maintenance allow you to scale without financial or manpower barriers.

The changing role of Data Engineers

Five years ago, data engineering was all about moving data: patching brittle pipelines, writing custom ETL scripts, and troubleshooting vendor tools. Complexity was the norm, and mastering it set you apart.

Today, moving data isn’t enough. Automation and better tools mean we’re no longer “plumbers” of the data world. Instead, we’re becoming platform architects who design scalable, AI-ready systems that empower teams to innovate.

• Junior engineers can now contribute meaningfully from day one, focusing on value rather than grunt work.
• Senior engineers can tackle higher-order challenges: governance, multi-cloud interoperability, and shaping data culture.

The future belongs to the data platform engineers and the teams they enable.

dlt+: Building the Platforms of the Future

Democratisation is already happening, and dlt’s usage numbers are proof of democratisation in action.

Despite dlt being a relatively new tool, thousands (3k+) of engineers have built tens of thousands (20k+) of data pipelines with dlt in the past year, gaining control, transparency, and flexibility for thousands of (2k+) production deployments.

But pipelines are just the beginning. The next chapter is platforms. That’s where dlt+ comes in, to bring the same democratisation principles to the entire team.

• Transparent, flexible and predictable, it's built with developer enablement in mind, to provide fast, safe iteration on data work.
• Designed to make the entire team proficient, it supports end to end collaboration, from the data engineer, to the analytics engineer, ML professional or AI engineer.

Be Part of What’s Next

The first wave of democratisation empowered engineers to own their pipelines. Now, we’re writing the next chapter: empowering platform builders to take control of the entire data lifecycle.

If you’re ready to build scalable, AI-ready platforms that are secure and fully under your control, we’d love for you to join us.

Join the dlt+ Early Access Program today to:

• Get early access to the next-gen platform-building framework.
• Collaborate directly with the dltHub team to shape the future of dlt+.
• Build data platforms that are portable, transparent, and built to scale.

Join the waitlist now!