What’s in the Kool-Aid: field notes from data engineers already drinking it

The AI frontier is moving, and the profession with it

Over the past year, coding models have changed the economics of building software and, with it, the possibilities for data engineering. But technological capability and professional practice don't move at the same speed.

Most of us have established ways of working, responsibilities built around existing constraints, and mental models formed when certain things were simply too expensive or difficult to attempt. Even when those constraints disappear, it takes time for practice to catch up.

Meanwhile, some pioneers are already writing the playbook. They're discovering new workflows, taking on problems differently, and producing results that look almost hard to believe for the rest of the practitioners.

A year ago we did a similar exercise and drew similar conclusions

As tool builders, we do our best to learn from the early successes of pioneers and build robust tools that enable anyone to achieve similar results. Since almost all change has been brought about by AI, this means building for that too.

Around this time last year, we looked at what practitioners do and at the limits of technology and realized there are two remaining moats: Complexity where LLMs fail, and “private” ontologies, or rules and decisions about how something would work that’s not yet available to the agent - such as, the exact business processes you run.

The complexity moat is durable but shrinking - in an evolving product, there will always be something that’s a little too complex for some LLMs, even if that’s just the newer released product features. This type of complexity can be turned into rules that can live in the platform though, so “complexity” simply describes a pre-documentation state of rules or ontology.

The data ontology moat is complex and large because it requires defining rules together with business personas, which is a human-speed endeavor.

To put everything through the “rules for agents” lens, you would essentially need these three ontologies:

• Data platform ontology - This contains the rules of how the data platform should operate - the infrastructure, the governance rules.
• Source system ontology - This contains the rules of the operational systems where data is produced, and it is necessary to encode the source’s “meaning” of the data.
• [Business] process ontology - This is the business processes and various rules, goals and metrics that define what we care about to optimise in our process.

To cater to these moats, we created the dltHub AI Workbench that spans across infrastructure operations, ingestion and AI-ready data modeling,

What the pioneers are doing

In September, a lead data engineer at an AdTech company posted ten months of field notes to r/dataengineering: Thoughts on AI in DE After Drinking the KoolAid For 10 months. His team has agents "building enterprise pipelines across our entire data eng stack in hours of iterations," work they used to budget a month for. The backlog is clearing so fast he doesn't know "what we'll do in like a month."

His job, he writes, has been "basically reduced to two tasks":

1. "Define and maintain the system ontology, expected features, guardrails, and appropriate persona docs for agents to assume/use."
2. "Define test scenarios, definition of done, deployment strategies, and verify that system is on track performing correctly and is on-track for long-term stability."

He also names what agents still can't do: a "moat of high context" in tuning Spark jobs, and a "moat of poor decisions" in legacy systems that are "not conceptually consistent even where they are documented."

How they got there

The OP's team didn't start with agents. They "invested a ton of time" in test infrastructure, CI/CD environments, PR reports and agent docs first. Agents follow TDD and "an established protocol of checks before anything gets pushed to PR stage."

What other teams are doing

Other teams are at very different points, but the same patterns repeat: experiencing productivity, building for control.

• Migrations got short. One commenter moved "60+ pipelines in 3-4 months, with full testing and validation" from Redshift to Databricks.
• New hires ship in days. One DE got ready to mentor a hire with no data engineering background. "Just days later she was making PRs for spark jobs that would take me a week or two." His review: "It was ugly and not very optimal but it worked."
• Legacy systems get documented first. For badly documented systems, one commenter's advice is to "start by having AI document the systems and improve the instrumentation."
• Agents get pointed at the data. One engineer runs a script that "just queries the data to figure out the shape" and adds upstream and downstream context. It has found major issues, though "it was absolutely directed on where to look."
• Regulated teams keep a hand on the wheel. The OP calls AdTech "an incredibly fault tolerant industry." A commenter in financial accounting has "lots of context management and guardrails set up" and still steps in often.
• Some still write the code themselves. One engineer uses OpenCode for about $10 a month and writes most of the code by hand. At the other end, a principal bills $4-5k a month in tokens to the client.

The work isn't shrinking. "I've never been busier in my career," one DE writes, and the OP's floodgates are open: "every half-baked idea that we've ever had but didn't have the time to work is going out."

How people feel depends on what they liked about the job. One engineer would "honestly prefer prompting agents than writing Spark code myself." Another says AI "takes away every aspect of the job I truly enjoyed (problem solving and technical details) and I'm left with all the boring stuff." Someone else is "tired of having to integrate the latest new AI thing every other week," and one reply worries about "training my replacement without being told that I am training my replacement."

What it costs

A second thread from the same month, Is critical thinking dead? New hire spends $20k on tokens pushing PRs (1.2K upvotes), turned into an open comparison of what teams spend.

Token budgets and spend per engineer, as reported in the replies. Informational, unverified.

Budgets in the replies run from "100 euro per month per person" to "like $1k/day." The new hire in the title spent "well over $20k on tokens in their 3rd week on the job," and leadership celebrated it "as productivity." Several commenters doubted the story; the upvotes say it rang true anyway. Elsewhere, another team's agents looped Databricks clusters into a $75k night, and "there was no one to blame."

The cost isn't only money:

• The OP's "review fatigue has gotten so bad at times that I feel like I've just straight up forgotten how to read."
• One company "measured performance by number of new AI sessions one quarter," so engineers vibe coded a tool to create sessions.
• Inside a single team, "everyone on the team uses their flavor of harness."
• And more output isn't always better: one reply notices those new features "don't seem to be driving meaningful change."

A principal data engineer summed it up: "burning a lot of tokens is the easy part. Making all that activity add up to something useful is the tricky bit."

Everything they describe is ontology

Read through the three ontologies, the OP's moats and tasks land in one place.

• Platform. Spark tuning, his "moat of high context," depends on "the data going in/out, infra, very verbose logging, and a ton of tacit, tribal knowledge of the system and data itself." That's platform knowledge nobody wrote down, which is why it looks like complexity. His first task, guardrails and persona docs, is writing it down, and so are the deployment strategies in his second.
• Source system. The "moat of poor decisions" is legacy systems whose meaning was never settled. One commenter's script "just queries the data to figure out the shape," and it finds real issues, but "it was absolutely directed on where to look." Someone still holds the map.
• Business process. Expected features, test scenarios and the definition of done say what the business wants. When that's missing, the work ships anyway. An agent asked to minimize memory usage in ingestion wrote empty files, "indeed, optimal in that sense, but pointless regarding our business."

The OP is already "less confident in moat #1." Once the platform knowledge is written down, what's left of the complexity moat is the part nobody could have written down yet.

Build it with us!

Building AI-ready data infrastructure is a brand-new frontier, and there are no true "experts" yet.

We’ve already done the heavy lifting by aggregating the experiences of early pioneers into a platform that already works. Use it to solve your actual business problems, and tell us what you love (and what you don't). Your feedback directly steers our roadmap, so we can keep solving the platform problems while you focus on your unique logic.

Try the dltHub AI Workbench, and tell us what we should build next.

Paste this prompt into Claude, Codex, or Cursor: “Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub”

Or book a demo!