The future of data work, as discussed in a panel at Helsinki data week

This post is a summary of the panel discussion: The future of data teams and their tools hosted by MotherDuck (scroll the website all the way) and moderated by ​Dumky de Wilde,

• ​Adrian Brudaru, co-founder @ dltHub
• ​Amy Raygada, Head of Data & Al Central Europe @ Thoughtworks
• ​Stijn Zanders, Data & IT @ Aiven

all 3 of us now have to work on the topic of how to move forward our teams, our tools and our trade.

Since the talk was not recorded, i'll try to piece it back together from my sleep deprived scatterbrain. Here's the rough message of our exchange, for your benefit.

1. The role is changing, and if you don't adapt, natural selection is coming for you.
2. Learning by doing, not from books needs to be practiced intentionally or you risk falling behind.
3. Creativity is becoming a key human skill for finding novel solutions enabling the application of AI in production.
4. Not everyone has to or is needed to keep working on core reporting problems. There's a rising opportunity to close the action loop on data.
5. Individual productivity may be rising but collaboration is still the main bottleneck that slows down organisations to human speed
6. Governance and safety-first in experimentation become paramount to prevent private data leaks.

Let's expand.

How the role is changing

As individual contributors expand their abilities through LLMs, this creates a rise in productivity and removes human handovers. Core reporting is becoming a codified art that requires less professionals to achieve the same.

Analytics engineers are picking up data ingestion and data analysts are absorbing the data science work. But what will the rest of the professionals do? Not everyone wants to build the next generation data stack and not everyone is needed in this endeavour, so how do you pay the bills without going mad?

There's an emerging opportunity larger than reporting - and that's closing the production operation loop with data driven agents. For example, agents can now process support queries and orders, or draft outreach messages.

But applied to any domain, agents can pick up the work that can be codified - all they need is data and a map of the domain to take data driven decisions and take actions against real-life things.

For example, an ecommerce inventory manager could have an inventory agent that has access to sales feed, warehouse stock and supplier contacts could place restock orders automatically and only surface relevant decisions to the human manager.

This is where a domain expert - data person symbiosis enable us to close the data action loop. we always wanted data to be used to take decisions, this is that, and more.
The huge upside to this change is that while core data reporting and analysis work is growing in scope and shrinking in body count, the application of data for operational [action and decision] automation is growing in both scope and demand.

What to learn? An analytics engineer asked me after the talk. I hope she asked all of us, but I told her,

• learn the principles of data ingestion so you can understand the buttons and options you have and tradeoffs that the machine makes and you're responsible and accountable for.
• Learn about knowledge graphs, specifically virtual knowledge graphs and ontology. Of course, ingestion being our specialty, i recommended our free courses, of which these 3 may take around 5 hours together.
  • elt with dlt fundamentals: This course teaches the basics of ingestion. In roughly 2 hours you will learn how data extraction and loading work while using dlt, the robus open source python library for data loading.
  • elt with dlt advanced: This course teaches how to handle advanced data extraction and loading scenarios with dlt. This course really matters when trade offs of various techniques are expensive and need to be understood and managed. Take this course if you plan to maintain and own production pipelines.
  • The agentic engineering one for applying that knowledge today
• Try the dltHub context on load (lesson 5 in our agentic DE course) auto-model your raw data into a virtual knowledge graph that serves as the foundation for making agents such as analytics agents or productive agents (the ones that take action and close the data-action loop).

Learn by doing not by reading

This theme has a couple of directions: The value of each type of learning, and triggering a paradigm shift.

A change in learning economics

There are two kinds of professional knowledge based on how it is obtained: Codified, textbook knowledge and trade or experiential knowledge that is learned in a complex context and handed down from person to person.
Knowing codified knowledge is starting to be less useful because LLMs can look up and apply rules faster than you. It's of course still valuable to understand and remain in control of trade offs, but human control has a cost which is worth expending sparingly.

Relearning and mental paradigm shifts requires a change in behavior, not reading more books

We all agree that in order to learn the new stuff, a different approach is needed, namely, experimentation.
There are a few reasons to support that. We know from behavioral therapy that changing behavior changes the mind. When we read, information stays in our short term memory from where it's lost, unless we can contextualize it.
By forcing yourself to build, you generate the messy, contextual "trade knowledge" that no LLM can replicate. By creating a failing system and gradually making it work, you also develop a first hand feeling for when to apply the knowledge you have and when to go off the beaten path.

But how do you build off the beaten path? which brings us to the next point

Creativity is becoming a key human skill.

Creativity is a broad topic that can mean many things, so for the sake of clarity let's limit its definition here to the modern creative workflow.

1. Problem framing is a high real-world context, uniquely human skill that allows us to reframe a problem so a different solution space can be accessed.

Example: Website shows wrong information when accessed from a specific browser. Obvious frame: Fix the website. Less obvious frame: Block the traffic from that browser.
Another: You ask your seo agent to audit and propose. It finds website is slow for users in space, so the agent suggests expensive upgrades to your hosting. You know nobody orders from space so you decide to block the traffic instead.

2. Solution ideation is a judgment-free solution search that LLMs are great at

3. Solution selection is again a real-world high context task that humans are best suited at because they understand the tradeoffs, often quickly, emotionally.

So how do you train this creative process? We all agreed one good way is to try and fail because this forces a paradigm shift. To expand your understanding of what exact effort pays.
I added a weird parallel, buddhist koans. Koans are an unanswerable riddle like "what is the sound of one hand clapping" that forces your brain to try to keep reframing until the frame expands. The answer to a koan is the permanent change in you from having experienced the friction.
Your koan can be building a system until it works, sitting in the friction between not knowing, and an answer you currently cannot imagine.

here’s John Cleese on creativity.
https://www.youtube.com/watch?v=nvKeu46jgwo

There's a growing opportunity to do something new

And i don't just mean give up and drive an uber (oh shoot)

There is an expanding opportunity to build domain-specific automation alongside subject matter experts.
Many complex workflows were previously impractical to automate because they relied too heavily on uncodified trade knowledge. The automation engineer builds the pipeline, but the domain specialist provides the ground truth required to handle edge cases and maintain the link between software and reality. This model acts as a force multiplier rather than people replacement, making high-complexity work feasible at scale.

At the same time there s a growing pressure to find the valuable work as demand for work around topics like data science dwindles.

Collaboration remains a human problem.

While individuals sometimes reach significant productivity gains, there is little consensus on how reuse of these setups. Ideas revolved around data products, data product catalogs, knowledge catalogs and ontologies.

Governance and safety first in experimentation

One thing is becoming clear, if you aren't careful, LLMs are a tangible security risks.

In one example given, a team leaked PII and got asked for money as a "beg bounty" because they vibe-coding dashboardds on a popular general purpose web app building tool.
To solve for this, practitioners build safety harnesses with human-like rights system, while MotherDuck and dltHub highlighted that they handle security for you which is one of the benefits of using a managed solution.

What did the vendors present?

• dltHub presented Context on load, or turning a raw data source into a canonical model/ knowledge graph by asking the dlthub agent to make a canonical model for your data. 1. Ask your LLM for our sample pipeline. which will set up the AI toolkit: “Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub” and 2. Ask it to model the data into a canonical model. Data quality checks, deployment and maintenance are also agentically performed. Learn more in this short course.
• MotherDuck presented a complete analytics lifecycle possible with their agentic harnesses that you can try here: https://motherduck.com/