The Last Mile is Solved by Slop

The Last Mile is Solved by Slop

In the LLM age, ingestion is easy and robust when building with dlt, that handles nested JSON, schema inference, incremental loads. But modelling that data for BI? Still a manual slog. Inspect the data, collect the questions the business wants to answer, design canonical and then dimensional layers, write SQL, validate, document, and test assumptions. Multi-day work for each data source. The gap between raw data and a dashboard-ready star schema is where projects stall.

To tackle this problem, I didn't build a platform or a product. Instead, I wrote some slop. A scaffold that runs a pipeline, grabs the schema from DuckDB, and forces an agent to build a star schema. A script and a workflow. Code mixed with examples, prompts, workflows, and autofill hints. No polish, no review, just chatted to it until it worked in a “blind” test.

Call it slop unironically. The real world is organic and dirty. You explore your data sources, draw the schemas on a notebook, scribble some notes, and collect some former reports that document intentions. Working the slop into a technical solution, shaping it question by question, feedback by feedback - this is the messy reality anyway. You don't clean it up first, design the perfect model, and then build. You work the slop into shape.

The “scaffold as skill” proof of concept works shockingly well, producing production-ready executable code. The agent crosses the gap from messy raw tables to a clean star schema in minutes. Not hours. Not days.

This is far from a polished product; it’s just a working prototype I built in a few hours that proves something: with the right workflow, an LLM can completely automate the really difficult skills of dimensional modeling. DDL, normal forms, join logic, the stuff that took years to learn. Once this concept gets refined into a formal product, it's game over for the traditional manual modeling industry. The innovation we’re looking at isn't the code I wrote; it's the revelation that the methodology of wrapping an LLM in a strict scaffold of reality and intent works also in semantic space. This builds on work we've done: generate ingestion, generate dimensional from canonical, autofill the semantic layer. By bridging the ingestion to canonical and synchronizing the layers end to end by prompt, we close the loop.

Skill Democratization: Turning the "Black Box" of Modeling into a Shared Map

You might know me as the data engineer who helped kick off dlt. Our original mission was to bottle the "skill" of a senior data engineer, starting with mine, into a tool that anyone could use. We wanted to democratize ingestion, and we did exactly that: the tool became the curriculum. If you know how to push the buttons on dlt, you are effectively applying Senior-level engineering leverage.

But there was always a "black box" past ingestion: the messy reality of business semantics. This was the part we could never quite automate, until now.

Evidence, Goal, and the Scaffold

LLMs are semantic engines, but without a map, they are just guessing every time. To stabilize this, you need a semantic mapping of your space, a structure that captures truth before it tries to find meaning.

A typical data stack needs the following components to transform raw data into business answers.

• Semantic skill: Semantic classification into fact and dimension tables (canonical data model)
• Coding skill: code that implements it
• Business skill: business rules to customise calculations or logic (usually comes last and iteratively)
• Reporting skill: Building a dim model and dashboards to answer questions.

The LLM can do everything, but struggles with the business part if it doesn't have context. The rest is automatically possible as a semantic modeling of the data without business rules. But those rules can be added later by the user.

Our agent doesn’t rely on one-shot guesswork. Instead, it replicates the work of a senior data team by defining a stack that creates a feedback loop. You can consume, discuss, and customize this stack to first call out the known semantics and then iteratively add any missing context by simply asking the business user. It’s a bridge between raw code and human intent.

To create this end-to-end, fact-driven system, we start with three pillars:

• The Evidence: What we actually have. We anchor the experiment by running a dlt pipeline to discover the schema. This contains its own semantics in its names and relations. We export this into 1_raw.dbml.
• The Goal: This is both a testable end-spec and semantic guidance. We use the "Top 20 Questions" constraint - by coming up with 20 questions we want to answer from the data, we force a collapse of ambiguity into very specific requirements, enabling us to validate an result. In our case, the agent fills TOP_20_QUESTIONS.md first, and you can then edit those questions to steer outcomes to your custom case. All questions should be able to be answered within a single join from fact to dimension (star schema), so this also constrains the final possible outcome.
• The Scaffold: The "how" of the operation. Think of the scaffold as programmable skill. It’s a mix of instructions, templates, and tools where the LLM acts as the "implementer middleware." Whether you call it context, a prompt, or a workflow, the scaffold forces the agent through a disciplined 10-step sequence: inspect, design, implement, and validate.

But this is a general process, so feel free to get creative. Try something sloppy - for example, most data stacks start from a finance report that needs to be automated - that P&L report might be all you need as a starting point - with existing values as targets, the LLM can even test or brute force trying metrics until it finds the desired one.

Running the demo

The exploration contains the following raw materials

• A starting point of evidence - Here we include a pipeline that will load 100 dummy records of HubSpot data in the schema that we originally got from the API.

• A scaffold - a sloppy combination of skill, prompt, and code that I prompted for - good enough for testing purposes but not what I’d call production grade rigor. It contains
  • A protocol of how to map the data semantically from source to question: raw - canonical model - star schema - dashboard
  • A visualization of the layers that we use for inspection; this offers us a high-level schema view with SQL for lineage that enables us to quickly inspect a LLM’s modelling proposal
  • “Micro Prompt” templates - in the visualization that offers us lineage, we have the SQLs to produce the data models - these are literally small prompts that the LLM traverses and fills with context-relevant code.
  • A local DuckDB with data - the pipeline will be used by the agent to load a sample dataset against which it can deploy the SQL layers as views.
  • A dashboard for answering the questions on top of the views - this enables us to validate outcomes and explore what else we could do with the data model (prompt for other questions, styles, formats, or visualizations; get creative; allow yourself to be sloppy and try new things!).

How to run:

This is a Cursor or similar LLM-powered DE demo, so make sure you have something like this, and it will figure out itself what to do from the slop. The easiest way to get started is to just clone the repo in Cursor and ask your agent to run the demo. Or, you could ask the agent to run it with your own dlt pipeline so you can try it with your own data.

The HubSpot Example: How It Runs

The repo ships with a HubSpot dummy pipeline: 100 companies, 100 contacts, 100 deals, 100 tickets, and 10 owners. Nested associations, junction tables, property history. Classic CRM mess. Here's what happens when you run it.

Step 1: Scaffold

The goal of this step is to create the evidence and templates the LLM should fill. By scaffolding, we generate a new folder called {pipeline}_{unix_timestamp} that contains the pipeline’s schema, a DuckDB with data, and serving of slop: a disposable, empty template of code and microprompts that the LLM will traverse to do the job. We can then give this folder to a new chat in Cursor and ask it to “do the thing”.

We can do this by running python scaffold_model.py --pipeline hubspot_dummy_pipeline.py or by asking the agent to do it with our pipeline. This also runs the pipeline, generating the first layer of evidence.

A timestamped workbench folder appears. The pipeline runs, dlt infers the schema and deploys it with the data to DuckDB, the script extracts the schema, infers some references, and writes a first version of 1_raw.dbml. Some relationships will not be known in this step but will be obvious to the LLM later.

Note: The docs are “overfitted” to HubSpot. Because of having created this example with HubSpot, despite asking for generic examples, the model documented specific fact and dimension names in the modelling examples. This will bias towards reproducibility of this example, so I encourage you to try other pipelines and share findings. In my experiments with other pipelines, the process still works despite the “overfitting”. As with anything, you will still need to have back and forth with the LLM later for improvements.

Step 2: The Agent Runs the Workflow

Open the folder in Cursor. Say "Run the workflow" or "Execute docs/WORKFLOW.md". The agent:

• Inspects the raw schema (companies, contacts, deals, junction tables like companies__deals),
• Fills TOP_20_QUESTIONS.md with questions likely to be asked from the entities it sees. If at this step you give it extra evidence, such as existing reports, it will make better questions.
• Reviews the raw model and might fill missing joins
• Designs the dimensional model it needs for the questions and designs a canonical model to bridge to it from raw data
• Populates 2_canonical.dbml and 3_dimensional.dbml for use in the visualization for your review

• Generally stops here and lets you review before continuing - now open the visualization and criticize it or ask it to explain decisions or what doubts it has, if any. Or simply ask it to improve its work and work harder because the boss is watching. Here I’d ask it to give me the SQLs in the visualization too so I can confirm them before it deploys them. In the vis above, you can see the SQLs.
• Implement SQL in build_views.py in DuckDB, fills top20_queries.json with validated SQL, then runs generate_dashboard_html.py and sync_visualization.py.
• You can then prompt for formatting, visualizations, etc. And review the SQLs. If you want follow-up questions, it is able to make the necessary model adjustments.

The Three layers

Layer 1: Raw

Tables as dlt loaded them: source-shaped, junction tables, and property history. No business logic. Main entities (companies, contacts, deals, tickets, products, quotes, owners), pipelines & stages, bridge tables for handling many-to-many relations, and property history.

Layer 2: Canonical

Normalized, business-ready entities. Since modeling is a bit random here, it might stack bridge tables into a megatable and later unpack it or not. If you are using other pipelines, here the LLM will try to find the canonical model and stack similar fact tables (like grouping marketing activities or customer support email and chat channels) and optionally merge information into dimensions to reduce unneeded complexity (for example, a customer_address table might be merged into the customer table).

Canonical layer: normalized entities (companies, contacts, deals, tickets, products, quotes, owners), resolved many-to-many via junction views.

Layer 3: Dimensional

Star schema. This is a denormalization of the canonical layer to enable fact tables to join directly to dimensions, to support tools like Metabase.

Since a deal can link to many companies and vice versa, for the fact table, you’d pick one company per deal - the primary, the first, the last or something, to enable using a dimensional model. And if you want to explore different questions, you can always use the canonical tables.

On top of this, we document the intended usage as a semantic layer.

Usage instructions: Semantic layer

Join patterns, metrics, and how to interpret the model. The contract for BI tools and LLMs. What we see here is functionally a semantic layer, or usage instructions for the dimensional layer.

If you are not using a BI tool to read the data and do not care about query performance, you could use the semantic layer straight on canonical and skip the dimensional bit.

The Dashboard: Visualization of the 20 questions

top20_dashboard.html renders each question with its SQL and the live result from DuckDB. Expand any card to see the SQL. The agent crossed the gap from raw to star in one session. You review. Done.

For example you could create a version in your company colors with visuals and formatting

Or a format that works better for a blog

The Nuclear Future: Automating the Feedback Loop

Right now, the human is the bottleneck. You run the workflow, the agent produces, and you validate. It is collaborative, but it isn't hands-free. We are inches away from removing the human from that loop entirely.

The leap to full autonomy happens the moment we replace human "review" with automated validation. If the agent generates a canonical model and we have end-to-end tests that validate the SQL against the actual data, the agent can self-correct and finish the job without a single prompt from you. We already do this implicitly during the "slop" phase: the agent generates a query, it fails, the agent sees the error, and it tries again until it works. Scaling that means the agent captures a request, proposes a plan, applies changes, and runs a battery of tests. If it passes, it deploys; if it fails, it loops back.

In this scenario, the dashboard itself becomes a live test suite. If a visualization breaks or a metric looks off, the system detects the anomaly and heals the underlying SQL. The business user becomes the final, implicit validator. If we design a sensible system that manages change and versioning, we can stop being the "implementers" in the middle. The agent handles the execution, the tests handle the truth, and we move from being the people who do the work to the people who define the destination.

The Commoditization of Skill

This workflow turns "hard" data engineering into a commodity. DDL, normal forms, and complex join logic, career-worthy skills are now executed by an agent in minutes.

The value shifts to defining the outcome, not writing the code. If you don't learn to scaffold and guide agents, you get commoditized. The things we've already built get automated. The things yet to be built need humans to define the target. That's your edge.

Further, if all formal skill is getting commoditized, this points to a situation where those working in R&D (experimentation, implementation) have a durable future.

How to Run It Recap (The Slop in Action)

One command, the rest is agent stuff.

1. Scaffold: Run the command. It executes the pipeline, extracts schema from DuckDB, creates a timestamped workbench folder.
2. Pull the generated folder into a new cursor chat.
3. Say: "Run the workflow" or "Execute docs/WORKFLOW.md". The agent fills everything.
4. Review: Open visualization.html for the schema, top20_dashboard.html for the dashboard.

Is the code breaking? No worries, this is a research tool not a product. You don't maintain this code. You regenerate it. The "product" is the implicit workflow that enables the agent to assign semantics to data and define reusable definitions. The agent does the rest, deliberately disposable.

Further reading

• Convergence: The Anti-Entropy Engine — scaffolds vs crutches, engineered convergence
• The Builder: Outliving the Modern Data Stack — code-native, builder vs consumer
• Autofilling the Boring Semantic Layer — dlt + LLM + BSL, another convergent approach
• Semantic Layer as the Data Interface for LLMs — dbt's take on the same convergence