Why you still need data people in the agentic era - and what for
An agent can write a pipeline now, faster than a contractor and cheaper than a hire. So the question is obvious: What people do we still need on the team?
Adrian Brudaru,
Co-Founder & CDO
An IBM training slide from 1979 read "a computer can never be held accountable," and concluded it must therefore never make a management decision. What a data stack measures and how is also management dependent - every metric, its specific calculation, and whether we look at it or not should be management decision.
Should is a big word, and in practice management often defers metric decisions to the data person, who’s accountable for their selection. Accountable as in, they can explain the reason they decided for that in front of others, and it wasn’t at random but aligned with best interest of the company and their goal.
Your pipeline is full of decisions nobody made
Inactive is 30+ days of inactivity.
Where did that come from? Usually the metric first was just a report. It lived there for a while, then someone asked for it in another report. Suddenly it’s on the weekly meeting slides and now it’s a company metric. Usually somewhere in between the analyst sat down with the stakeholder, one came with data and the other with business reality and they agreed on a definition. Management later asked what this metric was and agreed it sounds reasonable for reporting up or to the team.
But if the agent built the stack? well then probably nobody really decided, it all just kind of happened and maybe whoever was chatting with the agent saw it and maybe they even had the authority over the decision, and maybe they had the understanding of the data sources too.
But each stack has hundreds such decisions. “Maybe” isn’t good enough and what’s already complex and chaotic can only get moreso if we take the accountability out of the loop.
Some simple examples that an agent shouldn’t solve but will: Does a refund reverses revenue in the month of the sale or the month of the refund? What counts as an active user? What a "customer" is and how many do we report when one company signs three contracts through two subsidiaries?
Humans always filled these gaps or found some sensible way to navigate the lack of authority. A human guess took an afternoon, lived in someone's head, and could still be asked about two weeks later. A model makes the same guess in four seconds and then the context window closes.
Review is not accountability, and neither is decision power
| The question | Automatable? | |
|---|---|---|
| Review | Does this match the spec? | Increasingly, yes |
| Decision | Which spec do we adopt? | No — needs authority |
| Accountability | Can someone explain and defend this? | No — needs a person |
Most systems currently still focus on review - make diff more visible, write more tests, read traces. It’s valuable work and increasingly more of the review and extra safety can be automated.
But this extra speed only gets you to the next bottleneck faster: Who can decide how we count customers? In one case, I built a warehouse in 2 weeks (sources and model), but the customer decision took 2 months of weekly meetings with CEO, CFO, CMO to untangle what had been done, what the understanding was, what the measured reality is and how to go forward bridging the gap between historic report, measurement, understanding of the metric and future reporting. That’s the decision bottleneck - because this was an investor metric, the CEO was accountable and had to decide, but the metric was previously reported by CMO to CFO (which changed) and neither was sure what exactly it measured.
Being accountable for the user definition doesn't mean deciding what a user is - if we report paying users, finance should decide that. What accountable means is that the CEO can explain it in front of the investors, is confident it's defensible rather than arbitrary, and when something about it changes he knows what it means and what to do.
The new data job looks a lot like the old, with more code reading and less code writing
Look at the ratio in that story. Two weeks to build the warehouse, two months to get one definition decided. Agents collapse the two weeks into an afternoon. They do nothing to the two months.
And in that project, after building the warehouse, I hired a team, trained the company, got 1/3 to self serve data for operations, scaled down and handed over by month 12 - the kind of work that you cannot just hand over to an agent.
What i’m trying to say is that even for the teams where building is a bottleneck, the majority of the work is not constant coding but rather taking accountability for how the system operates and defending real needs and goals while getting it done. Accountability is an obligation to explain, plus a willingness to carry the consequence when the explanation turns out wrong.
Why it can't be the manager, and why it can't be a sign-off step
Mark Bovens' definition, from public administration: accountability is a relationship between an actor and a forum, in which the actor is obliged to explain and justify their conduct, the forum can question and judge, and the actor faces consequences.
A model can't be the actor: no obligation, no stake, no memory. Accountability is built by duration. Naur, 1985: the real artifact of programming is the theory, the working understanding held by the people who built it. A program starts to die when the team holding its theory disperses. Its failure mode is like a slow legacy decay. It keeps running, but nobody knows how to keep it aligned to new changing reality.
Generated pipelines on the other hand arrive with no theory-holder at all and nobody spent any time considering what was decided and why. This means they fundamentally lack the entire decision heavy, valuable part.
Before starting dlthub I did 5 years of freelancing, of which half build&hire projects. A build and hire project usually has the phases.
- Bootstrap the data warehouse - orchestrator, ingestion, data model. Interview the future hire and owner. (15% of project time)
- Defining metrics over the data model; New hire starts, work with them to build up this knowledge (70% of project time)
- Hand over and exit - hand over knowledge, teach, and run along side them for a short time to catch any questions. (15% of project time)
Again, here we see the same pattern - the “build heavy, decision light” part of the project is very short. The bulk is building up accountability.
What changes with agents? The start is faster, and the technical (not the knowledge) handover can be partly done to an agent. But the rest remains human-speed.
The unbundling of the engineer
A data engineer's salary always bought a bundle: hands and head. The hands wrote the connectors, the pipelines, the orchestration glue — that's the part agents now automate.
The senior brain is what decided how it all composes: what's reliable enough to ship, what a schema change means downstream, how the pieces stay coherent as a system. That part is still needed, and it was always the scarce part.
Data engineers have been in short supply since we started calling them data engineers. Most teams never got one at all. This makes the unbundling cut in your favor - you can now get the hands for your team, and your seniors can leverage them responsibly.
Our honest pitch
dltHub is the agentic data and managed-infra layer on top of any warehouse that gives you the senior engineer judgment.
The six layers encode the decisions a senior engineer makes the same way every time: schema evolution, contracts, incremental state, run traces, secrets that never touch disk.
And the extra blueprints go further - they take existing setups or cases and reproduce the same judgment that created them in the first place. Some examples: