Faster pipelines, fewer bottlenecks: new performance tools in dltHub

Scale your pipelines, not your platform team

Every slow or memory-heavy pipeline ends up on a platform engineer’s desk or as a line on the cloud bill. Now the engineer who owns the pipeline can fix it, the performance toolkit enables the agent to find the bottleneck, and they can run their job on the hardware it needs.

When a production pipeline slows to a crawl or crashes at 2 a.m., do you actually know which dial to turn on the next run? Or, do you just restart and hope?

The problem is that building pipelines with a coding agent has become much easier, but getting them to run reliably and efficiently as workloads grow is harder. Running jobs still requires knowledge of backends, buffers, parallelism, and hardware, and most teams have more urgent work than becoming specialists in all four. Another way to look at is realizing that bottlenecks once put on building have migrated to keeping those pipelines fast, reliable, and cost-efficient once real production volume hits or scales up.

Defaults help you get started and suit a broad range, but they offer limited depth for understanding table size, source response speed, or how much memory a particular load will need.

Today, we’re releasing performance optimization features across dltHub in General Availability to make closing that gap part of everyday pipeline work.

The new performance toolkit in the dltHub AI harness gives your agent the skills to find bottlenecks, apply tuning and measure the improvement. Hardware configuration lets your team choose the CPU, memory and timeout for each job, without provisioning infrastructure or depending on a platform team.

For SQL databases, copying existing high-speed SQL backends, including pyarrow and ConnectorX, can deliver up to 30x speed gains.

For matching appropriate hardware to the job, the agent can profile performance bottlenecks and pick the optimal instance size in the job configuration which helps manage both more complex requirements, but also consuming only as many resources as useful. Most importantly: clever tuning pays for itself.

Problem: Why pipelines hit a wall after day one

Consider a database historical backfill. This job can take hours, and attempts to speed it up by processing more data in parallel can cause it to run out of memory.

Finding the right fix can feel like a game of whack-a-mole. Smaller batches can reduce memory use but may slow the load. More parallelism can improve throughput but increase memory pressure. And a beefier machine won’t help much if the pipeline is waiting on a slow source.

When the job does need more CPU or memory, the next step often means filing a ticket with Platform Engineering and waiting for infrastructure changes. The team loses time, and a larger instance adds to the bill without necessarily addressing inefficient code or configuration. And, RAM is expensive.

Data teams need a way to identify the bottleneck, test a fix and adjust resources themselves, without turning each performance problem into a bigger engineering project.

"We have an absolute mountain of Klaviyo data and due to API rate limits it’s very slow and was prone to failure on extraction. Using the dltHub performance optimization skill, the agent devised a tiered chunking strategy which does 7 days, then if it times out drops to 1 day, then 1 hour and has managed to get through some highly problematic data patterns without missing a beat." - Jim Barrow, Pinter

Profile it, tune it, size it

We built dltHub's performance optimization features so data teams and their agents can fix slow or failing pipelines on their own. Instead of guessing, there are three steps:

1. Profile the bottleneck
2. Let the agent tune the code using our performance tuning skill
3. Size the instance to match the workload

Profile it: Ask your agent to find the bottleneck

Start with the pipeline that is slow, failing, or using too many resources.

The performance toolkit guides your agent through its extraction, normalization and loading stages, using timings and resource usage to identify where the problem sits.

That diagnosis gives the agent a specific part of the pipeline to work on. It also gives teams without specialized engineers an explanation of what is happening before they change the configuration.

Tune it: Put the settings to work

The toolkit then guides the agent through relevant changes to workers, parallelism, buffers and batching. Importantly, it does measure again to check whether those changes helped.

In a scenario where database historical backfill takes hours, reducing the amount of data held in memory may make the job reliable without increasing its instance size. Where several independent tables are being read one after another, fetching them concurrently may shorten extraction.

Source-specific skills add further options. For example:

• Optimizing filesystem performance
• Optimizing REST API performance
• Optimizing SQL performance

For SQL databases, pyarrow and ConnectorX backends skip the schema type inference step but because of each technology's slight differences, they come with various other considerations regarding memory utilization and parallelism

You can review the changes and results, while expertise needed to begin tuning becomes available through the coding agent you already work with.

Size it: Choose the resources the job needs

Once you understand the workload, you can give it the appropriate hardware on dltHub.

Hardware configuration on the dltHub platform is available in General Availability, with four instance sizes from small to x-large:

You can also extend the 120-minute default timeout for longer jobs and slow APIs, including substantial backfills.

These are per-job settings. A demanding load can use a larger runner while smaller jobs keep their existing allocation. dltHub manages the infrastructure, so changing the resources doesn’t require your team to provision and maintain another machine.

Why tuning pays for itself

In testing and benchmarking, a tuned dltHub pipeline ingested a 20 GB database in under 5 minutes when compared to the default settings where the same ingestion took 74.6 minutes.

Here’s a breakdown:

With this kind of improvement in compute run times, tuning pays for itself.

Defaults work to provide a compatible starting point, and then the best configuration for production depends on the source, data types, and available resources.

The dltHub platform uses runtime-based pricing, so pipeline speed maps directly to your bill. Speeding up a pipeline with these tools costs nothing extra, and immediately lowers your compute spend. For a recurring pipeline, that improvement can keep paying off on subsequent runs.

To dig into the benchmarking results, you can read more here.

Try it on your own pipeline

You can start tuning your pipelines and configuring runner hardware today:

Run uvx dlthub-start, then /optimize-performance to find out why the pipelines in this workspace are slow or running out of memory, and fix them.

Not on dltHub yet? Start a free trial, deploy an existing dlt pipeline, and let your agent show you how much faster and more cost-effectively it can run.