Convergence: The Anti-Entropy Engine

Convergence works

We analyzed 500+ human runs of the LLM native workflow

And here is the crazy part: The majority of those runs succeed.

But the real story is how they get there:

• 45% finished in under 15 minutes.
• another 24% finished in 15-30min.
• The rest are done by the 2h mark

And what about those runs that failed? They either never tried, or due to lack of early success, context accumulates leading to higher entropy and increasing divergence from solution.

And this was before Gemini 3 and GPT 5.1, last night, Gemini went 2.5→3 and GPT5→5.1. Preliminary success rates seem up significantly.

Try it now yourself, start vibe coding from 8000+ connector scaffolds!

The earlier version of this workflow was producing much more entropic results, with success rates at around half of what we see now. What changed in between is not the models, but the way we engineer iterative convergence to a goal.

The "Deer in the Headlights" Moment

There is a specific kind of silence that falls over a conference room when an engineering team realizes they’ve just been obsoleted.

I call it the "Deer in the Headlights" moment.

It happens constantly. A team at a massive tech company spends six months building a complex internal tool to patch a flaw in an LLM's reasoning. They build wrappers, prompt chains, and aggressive regex filters to force the model to do something it isn't naturally good at.

Then, OpenAI or Anthropic drops a model update.

Suddenly, the flaw they spent budget and headcount fixing is gone. The model just does it now. The complex wrapper they built isn't just redundant; it’s worse than the raw model.

The team isn't celebrating. They are frantically editing their quarterly review to justify why they should have a job in February

This panic reveals a fundamental mistake in how we build with AI.

When you build to fix a model's current limitations, you are betting against the rate of progress. You are building crutches. And in this industry, crutches have a half-life of about three weeks.

To survive, you have to stop building for what the model can’t do today, and start building for what the workflow must do forever.

Crutches vs. Scaffolds (The Imperative Trap)

When we panic about a model update breaking our tools, we assume the problem is the model. "It changed! It drifted! It stopped following instructions!"

The problem isn't the model. The problem is that we are building crutches when we should be building scaffolds.

This isn't just a metaphor; it is a fundamental engineering distinction between Imperative and Declarative design.

The Imperative Crutch (Fragile)

Most LLM workflows today are Imperative. They try to micromanage how the model behaves.

You write prompts that look like middle-management instructions: “First, plan your steps. Then, use Tool A. Then, check the output. If the output contains X, do Y. Finally, summarize.”

This is a crutch. You are wrapping the model in a rigid procedural script to compensate for its lack of reasoning.

• Best Case: It works, sometimes... for this specific model version, on this specific day for the first similar 20 api calls before the model cache kicks in and degrades performance.
• Worst Case: Failed outcome: The model gets smarter, changes its formatting, or skips a step it no longer "needs." Your script breaks and gets in the way of the model’s success.

Imperative workflows are Fragile. They depend on the model behaving exactly as it did yesterday. When the model improves (a positive change!), your workflow shatters (a negative outcome).

The Declarative Scaffold (Antifragile)

The alternative is Declarative. You stop caring about how the model works and obsess over what the outcome must be.

You don't write a script; you build a scaffold that you put up and send the robot to work on. Rules, blueprint template, test, evidence, and human validation loop tools. This is a Rudder. It doesn't force the model to walk a specific path; it steers the model toward a specific destination.

• Best Case: Works every time: The model is brilliant. It fills the mold instantly with or without the scaffold.
• Worst Case: Works sometimes: The model is dumb, the scaffold helps.

The Distribution of Outcomes

The difference isn't just preference; it’s the distribution of risk.

1. Crutches are concave (Fragile): They provide limited upside when things go right, but catastrophic failure when things change.
2. Scaffolds are convex (Antifragile):
   • If the model is weak, the scaffold saves you (it catches errors).
   • If the model is strong, the scaffold gets out of the way (you get instant shipping).

You win either way - if your work becomes obsolete, it’s because you’ve already succeeded.

When you stop micromanaging the model (Imperative) and start giving it a definition of "Done" (Declarative), you stop fearing model updates. In fact, you start looking forward to them, because a better model just means your scaffold gets filled faster.

Engineered Convergence (The Loop & The Dashboard)

Once you accept that you are building a Scaffold and not a Crutch, your engineering priorities change. You stop trying to prevent the model from ever making a mistake. Instead, you build a system that captures mistakes and increases the context’s specificity.

We call this Engineered Convergence.

It relies on three things: Information (Context), a goal (The Scaffold), and a way to validate and improve instead of degrade (The Loop).

The "Generate running connector" Loop

In a traditional workflow, a runtime error is a failure. In a convergent workflow, a runtime error is a specific piece of information that tells you exactly where the code’s internals disagree with reality.

We operationalised this into the "Generate running connector" loop:

1. Generate: The LLM populates the connector template based on context. It defines a pagination next_link json path based on context.
2. Run: We execute the code against live data. It crashes. The error informs paginator failed and gives an example record to inform the LLM of the response structure.
3. YOLO mode until code runs - Call the agent again without further input. The agent reads the error and fixes it. However, if the errors accumulate without resolution, this may over time dilute the context and cause divergence. So, if you haven’t succeeded by try n (where n depends on your model), consider giving up and starting fresh.

You aren't asking the LLM to guess; you are giving it the verified truth it needs to collapse the search space.

The Dashboard: “code runs” vs. “code is correct and does what i want”

However, a running code only proves we are calling the api correctly. It doesn't prove Semantic Correctness of the pipeline.

The model might successfully paginate through an API, but if it uses the wrong primary key for incremental loading, it’s loading the wrong data. Or maybe you wanted to remove all PII before loading but your data contains a PII column. The code "works," but the outcome is wrong.

This is why you need a Pipeline Dashboard (or a Debug App).

We built a pipeline debug app specifically for this. It lets an engineer (or even a non-technical user) open the hood of the generated pipeline, see the data flowing through, and confirm: "Yes, this is the customer ID" or "No, this is missing the email field."

The Dashboard is the interface for troubleshooting semantics. It allows a human to look at the converged output and answer the question: "Is this correct? is this what we wanted?"

• The Machine handles the syntax, the schema, and the runtime errors.
• The Human handles the semantics and intent.

If it’s wrong, the human provides the feedback, and the "Generate running connector" loop starts again.

The Convergence Engine

This combination, The Scaffold + Generate running connector loop + The Dashboard, is the engine.

The Scaffold constrains the search space. The “Generate running connector” loop automates the technical corrections. The Dashboard validates the business intent.

When you have all three, you aren't just "using AI." You are engineering a process where the model can be wrong, messy, or chaotic, and the system still converges on a useful result.

The Invisible Senior Engineer (Maintenance & Democratization)

There is a misconception that Scaffolding is only for generating new code from scratch.

But a Scaffold is valuable long term. It creates a safety net that extends into maintenance, allowing you to widen the circle of who can contribute. It turns the system into an "Invisible Senior Engineer" that empowers anyone to submit a pull request in minutes.

The Maintenance Bottleneck

In a traditional setup, data engineering is a bottleneck. If a Data Analyst or Analytics Engineer (who know SQL but not Python) need a new field in an API connector, they have to file a ticket. A Senior Data Engineer has to context-switch, read the docs, update the code, clarify anything unclear, test it, and deploy.

It’s slow, expensive, and it turns your best engineers into mechanics.

Empowering the Edge

But one of our users (Hive.app) reported a different reality. They have a large team of domain experts who own the data (SQL analytics engineers) but aren't Python experts. Because they use a good context for their pipelines (schemas, rules, code examples), they don't file tickets, they open PRs.

The SQL folks prompt in Cursor: "Add the shipping_address field to the Hubspot pipeline."

The LLM attempts the change, the analyst tests it and submits a PR when happy.

Are you a non-engineer data professional? Build robust pipelines in minutes with dlt + LLMs and stop waiting for your engineer or budget holder.

DE turned data platform engineer, AE leverages LLM DE.

Their only Data [platform] Engineer, who set up the system, gets the PR notification, opens it. The code is clean and the change is small, It follows the style guide. All new data is automatically and robustly handled by dlt. They just review in a minute and merge - they could even do it while stuck in a less relevant meeting.

Sure, each PR is only 1h of time saved for the senior engineer, but the real win is removing the bottleneck and keeping context switching low.

Using LLM context as skills for the entire team, and their successors to use.

This is the ultimate payoff.

Because the system is robust to variance, you can lower the bar for who (or what) makes the changes.

• Work Wider: You can let non-experts safely modify complex systems because the Scaffold prevents them from breaking the build.
• Work Longer: The system doesn't degrade over time. The Scaffold ensures that year-old pipelines still adhere to today's standards.

You don't need a team of 10x engineers to maintain every single pipeline. You need a system where 1x engineers (or 0x models) can safely contribute, guided by an Invisible Senior Engineer that never sleeps.

The Call to Action

So, when solving with LLMs, stop building fragile wrappers that break every time Sam Altman tweets. Instead build something where your convergence rate goes up.

Try the scaffolds that we built, and get out of the modes’ way

Further Reading

The shift from "Prompting" to "System Engineering" is happening everywhere. If you want to dive deeper into the mechanics of Convergence, here are the foundational texts that validate the approach.

1. Andrew Ng: Agentic Workflows (Sequoia AI Ascent) convergent "loops beat models" Ng presents findings showing that an older model (GPT-3.5) wrapped in a reflection loop outperforms a newer model's zero-shot (GPT-4). He describes a version of our "Generate running connector" Loop.

2. Andrej Karpathy: Software 2.0 The foundational essay on the shift from Imperative to Declarative. Karpathy predicted years ago that we would stop writing explicit code (Crutches) and start defining the goals and datasets for an optimizer to fill (Scaffolds). The "Deer in the Headlights" moment is simply the industry realizing Software 2.0 has arrived. Read the essay

3. Anthropic Research: Building Effective Agents The creators of Claude explicitly advise against complex prompt chains (fragile wrappers). Instead, they advocate for simple, composable patterns where the model can see its own output and correct it. This is the "Generate running connector" loop in practice. Read the guide

4. Stanford NLP: DSPy (Programming—not Prompting) This is the code implementation of the Antifragile Scaffold. DSPy is a framework that abstracts prompts away entirely. You define the signature (input/output) and the metric (success), and it compiles the prompts for you. If the model changes, you don't rewrite prompts; you just recompile. Check out the repo

5. Lilian Weng: LLM Powered Autonomous Agents Weng (OpenAI) breaks down the agent architecture into Planning, Memory, and Tool Use. This is essential reading to understand "Context Entropy." She explains how to structure memory so that long workflows converge on a solution rather than diverging into chaos. Read the deep dive

6. Swyx: The Rise of the AI Engineer The cultural side of our "Invisible Senior Engineer." Swyx argues that the barrier to entry for software engineering is collapsing, allowing domain experts (like the SQL analysts mentioned above) to build complex pipelines by gluing models to tools. Read the manifesto

7. Eugene Yan: Patterns for Building LLM Systems You need a Dashboard because code that runs isn't always code that is right. Yan breaks down the hierarchy of "Evals" - how to move from "vibes-based" testing to rigorous semantic metrics that ensure the business intent is met. Explore the patterns

8. Shreya Shankar: Operationalizing Machine Learning A look at why pipelines break. Shankar’s research validates our view that runtime errors are not failures, but valuable information bits. Her work focuses on observability and how to capture these signals to prevent system degradation. Read the paper