The data engineering autonomy moment: agents now build and run dlt pipelines, and ingestion becomes a 40x bigger market
Since June 2026, coding agents build, run and fix dlt pipelines largely on their own. Our usage data shows what that does to ingestion: about 28,000 custom source families against about 700 catalog connectors, and 92% of agent-started runs coming from Claude Code.

Matthaus Krzykowski
Co-Founder & CEO
16 min read
On this page
- 1. Two moments
- Moment one, May 2025: Claude Code launches and new private sources jump 7x
- Moment two, since June 2026: agents take over pipelines, Claude Code leads and new private sources are up 4.8x
- 2. Two ways to build a source: a connector catalog with a hard cap, or code that agents can work with
- The connector way: a GUI catalog stops at about 700 connectors
- The code way: agents read, change and run a dlt source, and a GUI setting offers none of that
- 3. The bigger market: about three in four sources dlt users run are custom, in three kinds of segments
- Traditional source segments covered by connectors, but customized
- New source segments unleashed by AI use cases
- New source segments never fully covered by connectors
- 4. Our point of view: ingestion becomes software that agents run, and the connector business gives way to it
In May 2025, the launch of Claude Code marked an inflection point for AI coding agents. It reached data ingestion too, and we measured it in dlt.
Since June 2026 we have seen a second inflection point. We call it the data engineering autonomy moment: agents now build, run and fix dlt pipelines largely on their own. Three things changed at once. Models became capable enough, and cheap enough, to leave alone with a pipeline. Agents moved from assisted coding to autonomous, multi-agent software factories. And the people who need pipelines moved to the edge of the company: GTM and finance teams who ask an agent instead of a data team. This post is our point of view on what that looks like.
Two numbers carry this post. About 28,000 custom source families against about 700 connectors in a managed catalog. And 92%: the share of October's coding-agent pipeline runs that Claude Code started. Claude Code dominates the data engineering autonomy trend.
The four key takeaways are:
- Agents create a much bigger ingestion market, about 40x bigger. A managed connector catalog ships about 700 connectors. dlt users and agents have run about 28,000 custom source families since January 2025, each on two or more different days. This matters because it shows where ingestion is going as it becomes autonomous. A connector is a product that a vendor builds once and sells to many. A custom source is code that an agent writes for one company and runs in a loop until it works. In recent months companies have stopped waiting for one. Ingestion stops being something you buy and becomes something your agents build and run. This is why we think the traditional connector business is coming to an end, and why ingestion ends up inside a code-first data platform such as dltHub.
- Claude Code dominates the data engineering autonomy trend. Claude Code started 92% of the coding-agent pipeline runs we see in October: 7.31M of 7.9M. Codex started 519K and Cursor 82K. About 94% of the growth in agent-started runs since June comes from Claude Code, not from OpenAI's Codex or Cursor. This matters because the agent that writes the code now decides how data moves, and in the dlt ecosystem that agent is mostly Claude Code. As Anthropic and OpenAI productise their models, we see them treat dlt, and the developer experience with dlt, differently. That puts us where mobile vendors once stood when they had to choose between Windows Mobile, Symbian, iPhone and Android first. For us the answer is Claude Code. Ingestion tools have to work for agents first. A source an agent can read, change and run gets used. A GUI setting does not.
- Companies no longer wait for a connector. Agents build the one they need, and change the ones that exist. A connector is a ready-made link between a business tool and your data. When the link you need does not exist, or does not do what you need, an agent now writes it or changes it. In October 2026, dlt users and agents created 258,632 new private sources of their own, +85% on September. Even where a ready-made source exists, people change it: customized variants are 71% of Jira use, 70% of Google Sheets, 63% of GitHub and 52% of Stripe. This is a second signal of the data engineering autonomy moment. Agents need code they can read and change, and since June humans and agents create, read, change and run dlt sources at scale. A GUI connector offers none of that.
- The bigger ingestion market shows up in three kinds of new segments.
- Traditional source segments covered by connectors, but customized. Traditional sources in finance, billing and ERP are 90% custom: NetSuite SuiteQL, SAP Business One, Odoo. Traditional sources in marketing are 69% custom: Meta Ads insights, TikTok Ads and more.
- New source segments unleashed by AI use cases. AI and agent telemetry (Anthropic and OpenAI admin and cost APIs, LangSmith runs, agent memory), internal systems (wrappers around in-house APIs), and industry and IoT (fleet telematics, energy allocation).
- New source segments never fully covered by connectors. Databases and change data capture (Debezium change events, Postgres, MySQL and MSSQL replication), security (vulnerability exports, device inventories) and real estate (property-management systems, booking engines).
1. Two moments
Moment one, May 2025: Claude Code launches and new private sources jump 7x
Claude Code launched in May 2025. Newly seen private dlt sources went from 4,479 in April 2025 to 32,105 in May. That is 7x in one month. The level held: every month from May to November 2025 had between 28,982 and 50,595 new private sources.
Unique devices loading with dlt and DuckDB jump 1.7x
The top of the funnel moved with it. DuckDB devices went from 3,974 in April to 6,696 in May, 13,661 in June and 16,626 in July.
We read this as the Claude Code moment arriving in dlt. Developers stopped searching for a connector and asked an agent to write the pipeline. Claude Code, dlt and DuckDB on a laptop or in a notebook are where that starts.
Moment two, since June 2026: agents take over pipelines, Claude Code leads and new private sources are up 4.8x
In June 2026 the same curves started to bend up again.
Pipeline runs started by coding agents grew from about 0.8M in May to 7.9M in October. One agent drives nearly all of it, so the trend depends heavily on Claude Code adoption. Claude Code started 7.31M of the October runs (92%), Codex 519K and Cursor 82K. About 94% of the growth in agent-started runs since June comes from Claude Code. We do not see a similar inflection point for OpenAI's Codex or for Cursor. Agent detection started in February 2026 and misses older dlt versions, so read these as lower bounds.
New private sources doubled from 54,424 in May 2026 to 121,140 in June. They reached 140,124 in September and 258,632 in October. That is +85% on September and 4.8x since May 2026. As of November 17, the November count already stands at 248,108, so we expect the month to finish above October.
Devices. DuckDB devices rose every month from 71,837 in May to 205,813 in October. That is 2.9x in five months.
Why DuckDB devices matter. DuckDB is an in-process database that runs on a laptop, in a notebook or in a codespace. dlt ships a DuckDB destination. dlt creates the database, infers the schema and loads the rows, with no table definitions to write. That makes it the cheapest place to try a pipeline, with no warehouse or cloud account to set up. DuckDB also gives agents autonomy. It is a local warehouse on a machine the developer controls. An agent can process large amounts of data on this machine, without a cloud warehouse. It needs no credentials request, no permissions process and no account that someone else has to open. An agent writes a dlt source, runs it into DuckDB, reads the rows and the errors, and fixes the code in a loop. When it works, the same pipeline can load into any cloud destination by changing the destination connection string. So the number of devices loading into DuckDB is our top-of-funnel measure of how many people and agents are building pipelines.
Why now: three things changed at once
Models crossed a threshold. Fable 5 launched in June 2026. Opus 5.5 is far more capable than its predecessor, and Opus is already good enough for pipeline code at a lower cost than Fable 5. We do not credit one release. Claude models taken together became capable enough, and cheap enough, to leave alone with a pipeline.
Agents work in factories. During the same months, software moved from assisted coding to autonomous, multi-agent software factories. A factory generates and tests many versions of a source until one settles. Most of those versions are throwaway code. We think this is one reason new sources grow faster than the sources that stay in use.
The builders moved to the edge of the company. We increasingly hear from customers and partners that marketing, sales and finance teams no longer wait for the data team. A GTM person asks an agent for the HubSpot or Meta Ads dlt pipeline they need. This fits the data: marketing sources are 69% custom and sales and CRM sources are 63% custom (section 3). At Legora, the whole GTM team builds dlt pipelines.
2. Two ways to build a source: a connector catalog with a hard cap, or code that agents can work with
The connector way: a GUI catalog stops at about 700 connectors
A managed connector catalog is a good business with a hard cap. The vendor builds each connector once. They host it, support it and expose it as settings in a GUI. That only pays when thousands of customers need the same source. So catalogs have a hard stop. Most of them are limited to about 700 connectors.
The incentive limit is narrower still. A common industry assumption is that the top 25 connectors earn a traditional GUI connector vendor 80% of the profit. The other 675 are the cost of being complete and serving a customer, and many of them lose money.
A source with ten customers will not get a vendor's engineering time. A source with one customer, such as an internal API, never will. The model is also closed by design. Connector logic is closed. Configuration lives in a GUI. An agent cannot operate a GUI. The market is therefore limited to the sources many companies share. Until now, a company that needed a missing source waited for a vendor or built it by hand. Agents remove that wait.
The code way: agents read, change and run a dlt source, and a GUI setting offers none of that
A GUI connector is a setting. A dlt source is a file. Agents work with files.
A source is a Python function. It is a logical grouping of resources, which are the endpoints of a single API. In dlt, it is a Python function decorated with @dlt.source that returns one or more resources. It usually holds the authentication and pagination code for that API.
import dlt
import requests
BASE_URL = "https://api.hubapi.com/crm/v3/objects"
@dlt.source
def hubspot(api_key: str = dlt.secrets.value):
headers = {"Authorization": f"Bearer {api_key}"}
def get_resource(endpoint):
response = requests.get(f"{BASE_URL}/{endpoint}", headers=headers)
yield response.json()["results"]
for endpoint in ["companies", "deals", "products"]:
yield dlt.resource(get_resource(endpoint), name=endpoint)Customizing a source does not mean editing it. You start with dlt init. It copies a verified source, such as HubSpot, into your project as Python code. You do not have to change that code to make it do what you want.
Developers have mixed feelings about this, and so do we. Standardization is important. One maintained HubSpot connector beats a thousand private copies. We do something different from a black box. Each verified source is an SDK: a ready-made toolkit of code that someone maintains for you. You build your own source on top of it. You can deeply change how the connector works without changing the toolkit itself.
This blends two things that are usually separate. Configuration means choosing settings, such as which data to load. Customization means changing what the pipeline does, such as adding your own logic. A GUI connector gives you settings only. Here you use the standard, maintained HubSpot toolkit and still get deep customization.
The example below does exactly that. It does not change the HubSpot source code. It loads only two of the data sets, companies and deals. It keeps only recent deals. It adds a new table with a score from your own model. All of it lives in your own code, on top of the toolkit. This is the part we like most.
import dlt
from hubspot import hubspot # the verified source that `dlt init hubspot duckdb` copied into your project
# load only two of the resources
source = hubspot().with_resources("companies", "deals")
# keep only deals created after a cutoff date
source.deals.add_filter(lambda deal: deal["created_at"] > cutoff)
# add a new table: score each deal with your own model
@dlt.transformer
def deal_scores(deal):
yield {"deal_id": deal["id"], "score": model.predict(featurize(deal))}
source.resources.add(source.deals | deal_scores)
pipeline = dlt.pipeline(pipeline_name="hubspot", destination="duckdb")
pipeline.run(source) # a new `deal_scores` table appears next to companies and dealsA GUI connector offers the fields its vendor exposed. A source in code offers everything Python offers.
The filesystem source is the stronger case. For it, the SDK is the primary way to use it. You attach your own parser: pandas to parse Excel (we do that for SharePoint) or DuckDB to parse CSV and JSON. This is more advanced than the HubSpot changes above.
Why agents prefer this. An agent reads the source, changes it, runs it and reads the error. A human reviews the diff in git. A GUI setting offers none of that. The agent cannot operate it, and nothing exists to diff.
3. The bigger market: about three in four sources dlt users run are custom, in three kinds of segments
The 40x comes from one division. dlt users and agents have run about 28,000 custom source families since January 2025. A family counts only if one device ran it on two or more different days, so one-off tests do not count. A managed connector catalog ships about 700 connectors. 28,000 divided by 700 is 40. The strict version survives the hardest objection: 3,383 of those families run on 10 or more devices. That is still 4.8x the whole catalog. A source family is a normalized source name, so this counts distinct sources, not revenue.
In October 2026, custom sources held 73.8% of durable installations. Verified dlt sources held 14.4%, core dlt sources 8.5% and demo or generic sources 3.3%. Custom was also 78.3% of all source families active that month, whether or not they had durable use.
How to read the chart and tables. Size is a segment's bar length in the chart above, relative to the largest segment, Internal systems & unrecognized, at 100%. "Verified" means dlt ships a verified source for that vendor. It does not mean a GUI connector exists. Customized own sources for those vendors also count as Verified, so the green bars overstate unmodified connector use.
What is new in this section. It goes beyond counting sources. It shows which segments they sit in, how little of each segment a verified source covers, and how much of it is custom. Three results stand out. Segments that connectors cover are still mostly custom: finance, billing and ERP has 8% verified share and is 90% custom. Segments created by AI use cases have close to no verified share: AI and agent telemetry and internal systems are at 0%. Some segments connectors never fully reached: databases and change data capture sit at 2% verified.
Traditional source segments covered by connectors, but customized
Verified sources exist in these segments. Custom is still 60% to 90% of each bar.
Finance, billing & ERP. It has only 8% verified share and is 90% custom, so it is the clearest case of customization on top of known vendors.
| Segment | Size | Verified | Custom | What is inside |
|---|---|---|---|---|
| Marketing, ads & SEO | 60% | 31% | 69% | Meta Ads insights, TikTok Ads by creative, Klaviyo, Instagram |
| Sales, CRM & support | 53% | 37% | 63% | Intercom, ticket exports, CRM sync |
| Finance, billing & ERP | 41% | 8% | 90% | NetSuite SuiteQL, SAP Business One, Odoo, Chargebee, PayPal |
| Commerce, retail & marketplaces | 38% | 35% | 64% | Marketplaces, subscription billing, inventory, checkout reports |
| Knowledge, docs & unstructured | 32% | 11% | 88% | SharePoint, Gmail and Outlook metadata and attachments |
| Engineering, DevOps & observability | 31% | 37% | 60% | GitHub issues and repos, Airflow, cloud usage, hosting |
Customization shows up in vendor-level data too. Customized variants are 71% of all Jira use, 70% of Google Sheets, 63% of GitHub and 52% of Stripe.
New source segments unleashed by AI use cases
These segments have close to no verified share. Nothing existed to buy.
AI, LLM & agent telemetry. It is the clearest case: the one segment whose sources exist only because LLM tools exist.
| Segment | Size | What is inside |
|---|---|---|
| Internal systems & unrecognized | 100% | Bespoke wrappers around in-house APIs with no vendor name |
| Markets, trading & crypto | 16% | Exchange perp feeds, FRED, central-bank FX rates, power-market prices |
| Public, geo & open data | 12% | Weather services, satellite data, customs data |
| Industry, IoT, energy & logistics | 12% | Fleet telematics, energy allocation, oil brokerage, shipping |
| Lakehouse layers & plumbing | 9% | Bronze, silver and gold layer pipelines, dimension and fact loads |
| AI, LLM & agent telemetry | 8% | Anthropic and OpenAI admin and cost APIs, LangSmith runs, agent memory |
| Healthcare & insurance | 7% | FHIR bundles, hospital-system tables, fee schedules |
New source segments never fully covered by connectors
Databases, warehouses & CDC (size 76%, 2% verified) covers Debezium change events, Postgres, MySQL and MSSQL replication and SQL-to-lakehouse loads. 42% of that bar is core dlt and 55% is custom. HR, security and real estate are each dominated by one family (47%, 59% and 71% of the segment), so read them by family count.
4. Our point of view: ingestion becomes software that agents run, and the connector business gives way to it
Ingestion becomes software. Software is how agents work with the world: they read it, change it and run it. GUI connectors and black boxes do not allow that, so they are out. An agent cannot read a setting, change it or review it. It can do all three with code.
Agents need all the data. Only full reach is acceptable. An agent that sees only part of a company works from a partial picture. Full reach needs a way to add the source that nobody else has built. That takes code, not a catalog.
Ingestion is also about context. Context is what the data means, such as which table holds customers or what counts as a deal. Information about the data like this is called metadata. Ingestion is the gateway through which outside information enters a company, and it works in two directions. First, it must know how the company describes its own business, what experts call the business ontology, and not drift away from it. Code is shaped by the business, not the other way round. Second, it must tell the business when the outside world changes. For example, new fields appear in Salesforce or the sales team redefines its deal pipeline. Ingestion has to pick up that change and pass it on.
That makes ingestion a much more important part of the data platform than before. Once data arrives with its context and annotations, agents can build the transformations and reports by themselves. Transformations turn raw data into clean tables. This becomes a solved, automatic task for agents. Humans take care of the context.
So ingestion will not stay a market segment of its own. It merges as code with the context graph (a map of what a company's data means and how it connects), business knowledge, transformation and data modelling. From this point of view, the connector business is a dinosaur: it cannot merge this way.
Underneath all this, what a company gets for ingestion is changing. It used to buy a connector: a finished product that a vendor builds and runs for many customers. Now it owns a source: a piece of code written for its own case, changed by an agent and reviewed by a human. That is a different market, not a faster version of the old one. A catalog can only sell what many companies share. Code can reach everything else: a company's own internal systems, industry-specific systems and data that exists only because a new tool exists. We believe most of the growth in ingestion will come from there, and that the share of custom sources will keep rising.
The first moment showed that agents can write pipelines. The second shows that they can run and maintain them. Writing code is cheap and getting cheaper. What stays hard is running it safely: who can change what, who reviews it, who notices when it breaks, and what the next agent can learn from it. We think the value will sit there. dlt gives agents the building blocks. dltHub is where teams run what the agents build.


