Vibe Coding: Why Building Data Pipelines with LLMs Actually Works

The Secret: Data Pipelines Are a Well-Defined Problem Domain

Most software engineering requires creative problem-solving across unbounded domains. Data pipelines, especially REST API integrations, are fundamentally different: they're structured configurations with clear patterns and limited variability.

Every REST API pipeline needs a limited number of features:

- A base URL (where's the API?)
- Authentication details (how do I get in?)
- A list of endpoints (what data is available?)
- A pagination strategy (how do I get ALL the data?)
- Data selectors (where in this response is the stuff I actually want?)
- Incremental strategy
- Dependencies between endpoints such as list/detail.
That's it. The entire universe of possibilities fits on a napkin. When your problem space is this constrained, LLMs stop hallucinating in python and start doing what they're actually good at: Translating information between formats.

Our Approach: Make the LLM extract Parameters, Not write Open-ended code

Traditional AI coding tools fail because they attempt to replace developers with generative models. This fundamentally misunderstands both the strengths of LLMs and the realities of software engineering.

The marketing of data integration tools has always centered around connector quantity: "our platform has over 500+ connectors!" This misses the point entirely. Organisations need reliable, maintainable solutions for the handful of critical data sources that power their business, not a vast library of mediocre connectors.

With vibe coding, we treat the LLM as an intelligent information processor:
1. Read API docs to identify authentication methods, endpoints, and pagination
2. Extract the relevant parameters
3. Apply them to a proven, battle-tested framework
4. But if the framework is not enough, you are enabled to add full-code customisation.

The output is a clean, declarative configuration that follows consistent patterns, as opposed to an unpredictable codebase that future engineers will struggle to maintain. This approach leverages LLMs for what they excel at while keeping humans in control of the deterministic precision work and architecture.

The Real Advantage: Testability Before Deployment

The most significant flaw in traditional data pipeline tools is the deploy-to-test workflow. This creates unacceptable feedback loops and prevents iterative development.

With our pythonic approach, you can run everything locally in seconds. No waiting for infrastructure provisioning, no mysterious errors from the monolithic software etc.

When issues inevitably arise, you ask the LLM to adjust the configuration, test again in seconds, and continue refining until everything works correctly. This tight feedback loop transforms pipeline development from a painful exercise in patience to an efficient, iterative process.

Why This Matters: Democratizing Data Access

The core challenge in most organisations isn't a lack of data engineers, but that the people who understand the data aren't empowered to access it themselves. They depend on engineers who are busy with other priorities and often don't fully grasp the business context.

With vibe coding, domain experts who can clearly articulate their data needs can build pipelines themselves. This fundamentally shifts the balance of power in organizations, giving data owners direct access to the tools they need, without creating maintenance nightmares.

This isn't another low-code solution that breaks down when faced with real-world complexity. It's a robust approach that handles edge cases, rate limits, and API inconsistencies while remaining accessible to motivated non-engineers.

The Results Are Transformative

We've seen analysts with minimal coding experience successfully build pipelines from complex APIs and put them into production within minutes, not weeks.

We've watched senior engineers redirect their time from "cost center" repetitive connector development to "revenue center", adding value solving broader problems, where their expertise is more useful.

Most importantly, we've observed teams successfully maintain and evolve these pipelines over time with ease and low maintenance. A solution that works today but can't be maintained tomorrow is ultimately a liability, not an asset.

This Fundamentally Changes Data Integration

Data teams have been forced to choose between two inadequate options: inflexible GUI-based tools, or custom code that leads to maintenance burdens. Both approaches ultimately fail to deliver sustainable value.

Vibe coding creates a third path: leveraging LLMs to handle the repetitive pattern extraction while producing clear, maintainable configurations that humans can understand, validate, and modify when needed.

It works because we're not trying to replace engineers. Instead, we're empowering them with tools that handle the tedious aspects of data integration while preserving their ability to reason about and control the system.

Try it.

You'll reclaim countless hours currently wasted on boilerplate configuration and finally focus on delivering actual value with your data.

• Docs for how to vibe code
• Docs for the command to copy our rules into your cursor
• Video (old, new version coming soon)