Postgres to BigQuery benchmark: dltHub vs. Fivetran, Airbyte, and Stitch

Transferring data from SQL databases to data warehouses like BigQuery, Redshift, and Snowflake is a common part of data platforms. And with various tools available, how do you choose the right one for your needs? We conducted a detailed benchmark test to answer this question, comparing popular tools like dltHub, Fivetran, Airbyte, and Stitch.

Our benchmark approach

We analyzed these tools across several key parameters:

• Speed
• Ease of use
• Customizability

The dataset: TPC-DS benchmark

We used the industry-standard TPC-DS dataset for our benchmark, consisting of:

• Total size: 20 GB in PostgreSQL
• 24 tables
• 191,500,208 rows ( ~ 191.5M )

Environment used

For Airbyte (v2.4), Fivetran, and Stitch, we used each vendor's managed cloud service. For dltHub, every job ran on the large instance size:

• vCPUs: 8
• Memory: 16 GiB RAM

Hosting and destination

• Database Hosting: PostgreSQL 18.6 on GCP Cloud SQL
• Destination: BigQuery

Speed results

Here's how the tools performed in terms of pipeline execution speed:

Key takeaways:

• dltHub with the ConnectorX backend was fastest
• ~3x slower: dltHub with PyArrow, Fivetran
• ~5x slower: dltHub with Pandas
• ~8x slower: Airbyte
• ~16x slower: dltHub with the SQLAlchemy backend
• ~22x slower: Stitch

Performance Tuning for dltHub

The two different types of approaches (infer types vs capture types) scale differently when tuned. Let’s look at ConnectorX and SQLAlchemy backends.

1. ConnectorX

Worker configuration impact

dltHub agents use parallelized extraction while running the pipeline. We experimented with how the number of workers in extract, normalize, and load stages affected the pipeline run times. Since using ConnectorX means we do not have to normalize the data, the normalize time represents only the overhead of handling the chunks. Adding workers sped up the load time because each worker runs its own BigQuery load job, but barely changed extract, which is limited by database and network throughput and only finishes when the largest table is done.

All timings and credits are the average of three runs.

Key takeaways:

• Time Reduction: Total pipeline time decreased significantly from 14.67 minutes to 4.58 minutes by increasing the number of workers from 1 to 20.
• Load Time: dropped from 11.4 minutes to 1.53 minutes, because each load worker runs its own BigQuery load job, so more files load at the same time.
• Extract stays flat: because it ends only when the largest table finishes extracting, more workers hardly make it faster.
• Normalize: is negligible, as ConnectorX returns data that is ready to load, so normalize takes under 2 seconds at any worker count.
• More workers - less credits: credits fell from 0.99 with one worker to 0.32 with 20, and most of that saving came from the first five workers (0.99 to 0.49).

2. SQLAlchemy

Worker configuration impact

Same setup as ConnectorX: parallelized extraction, the same number of workers for extract, normalize and load, on a dltHub large instance.

All timings and credits are the average of three runs.

Key takeaways:

• Workers more than five helped only slightly: 20 workers was the fastest and cheapest, but only about 3 min faster than 5 workers.
• Extract: increased slightly as workers increased. The likely cause is that SQLAlchemy processes rows sequentially in Python, so additional threads introduce overhead.
• Normalize: the biggest gain came from going from 1 to 5 workers (53.6 → 20.1 min). Going further reduced it only a little, to 16.3 min at 20 workers.
• Load: dropped from 6.1 to 2.9 min, since each worker runs its own BigQuery load job.
• One worker run: got cut off after passing the 2-hour default limit.

Conclusion

This thorough benchmark demonstrated a few things:

• The backend matters more than tool and dltHub gives that room to customize. The same dltHub pipeline took 4.58 min with ConnectorX and 74.6 min with SQLAlchemy.
• Tuned dltHub beats the managed services. ConnectorX was about 3x faster than Fivetran, 8x faster than Airbyte and 22x faster than Stitch.
• Network and API bottlenecks limit pipeline performance significantly.
• Python performance in EL is not a bottleneck. Optimizations like faster languages or more hardware have insignificant returns if code is optimized.

Ease of use and ownership

The tools in the benchmark Fivetran, Airbyte, and Stitch are GUI tools and they are quick to setup. dltHub is set up agentically with this one command:

uvx dlthub-start@latest

It creates a workspace, then logs you in, and hands off to a coding agent like Claude Code, Cursor, or Codex. You simply ask the agent something, say:

And that's it. It'll do all the work like writing the pipeline code to deployment, asking you for inputs like secrets along the way. Here’s a snippet of the code written by agent:

for stage in ("EXTRACT", "NORMALIZE", "LOAD"):
    os.environ[f"{stage}__WORKERS"] = "20"

source = sql_database(
    schema="tpcds_sf10",
    table_names=TPCDS_TABLES,  # all 24 TPC-DS tables
    backend="connectorx",
    chunk_size=100_000,
    backend_kwargs={"return_type": "arrow_stream", "batch_size": 100_000},
).parallelize()

pipeline = dlt.pipeline(
    pipeline_name="tpcds_sf10_pg_to_bq_cx_w20",
    destination="bigquery",
    dataset_name="tpcds_sf10_cx_w20",
    progress="log",
)
load_info = pipeline.run(source, write_disposition="replace", loader_file_format="parquet")
print(load_info)
report(pipeline)  # compares BigQuery row counts with extracted rows, prints timings

And to deploy and maintain of course it’s just a matter of asking for it

And in case it breaks?

If you want to make it AI native by adding context on load and modeling for comprehension?

The agent will do so in several steps starting by asking you questions like:

• What should the model cover: store sales, all three sales channels, or the full warehouse? → Store sales and returns.
• What should the model be called? → retail_sales. This becomes the model's folder and, later, its BigQuery dataset name.

And a few more, like which tables represent which business entities, and builds an ontology: the business entities how they relate. It checks the outputs with you, confirms the grain (one item per store receipt). Generates a canonical data model (CDM), a start shema written in DBML:

Then from dltHub UI, you can inspect the run metadata like: status, trigger type, profile, start and end timestamps and more.

The pipeline is plain Python, so you can keep improving it. Like we took ours from 15 minutes to under 5 by telling the agent what to change. That control is what lets you get more out of the same compute.

Need help optimising your pipelines?

Reach out to our solutions engineers to get your team trained on dlt best practices and its ecosystem, completely customized to your data stack.