From compute hours to data moved: a benchmark series

What an hour buys depends on what's in the way

You pay dltHub for compute time. The thing you care about is how much data that time moves. The two aren't the same, and the ratio isn't one number — it's set by whatever is slowest in the path. So we ran five benchmarks on one 2 vCPU / 4 GB worker, one per bottleneck.

Same machine, ~30x spread. The engine isn't the variable. The format and the source are.

A small team usually runs most of these: a database on a schedule, a few APIs, some files. Budget about an hour a day for each and the numbers above are what land.

Tools that bill per row charge you for every nested record that fans out on the way in — and as you'll see, the fan-out is real. dltHub bills the time. The rest of this post is the methodology behind the table.

SQL: the engine ceiling

With nothing external in the way — no rate limit, no parsing, no I/O pressure — this is as fast as the engine goes. It's the ceiling the other three cases fall short of, and the reason they fall short tells you where your money goes.

Setup. 2 vCPU / 4 GB runner. Postgres on GCP (4 vCPU / 16 GB / 200 GB SSD, US) → BigQuery (US). pyarrow on the sql_database source. 8/8/8 workers, chunk size 150,000, full refresh. TPC-H at SF 5, 10, 20, 50.

The rate holds from 8 GB to 82 GB. It doesn't drift as the job grows, which is the property you want when you're billed by the hour: you can estimate a 10x larger job by multiplying, not by re-benchmarking.

1 hour = 65 GB, ~350M rows, Postgres to BigQuery — with source, runtime, and destination in the same region.

REST: you don't set the rate, the API does

SQL is network-bound and nothing else. A REST source moves the bottleneck off your machine entirely. Records per request, requests per window, how long a 429 makes you wait — those are the API's numbers, not yours. No worker size changes them.

So the question isn't how fast dltHub is here. It's whether dltHub adds anything on top of the wait. We ran two sources that bracket the range.

Setup. 2 vCPU / 4 GB (US) → BigQuery (US). HubSpot CRM REST API v3 (free trial), cursor pagination, ~100 records/request, 5 retries to 16s. GitHub GraphQL API, microsoft/vscode issues, 100 items/page, sleep-until-reset under 50 credits.

HubSpot — 4,960 records, total 2m 11s (1m 7s extract, 369ms normalize, 1m 3s load).

Normalize is 369 milliseconds. The run is request round-trips end to end; the worker is idle. That's the normal case for a CRM. Note the fan-out, because it's where per-row billing bites:

• Pulled from the API: 4,960 records.
• Landed in BigQuery: 9,177 — the 4,960 plus 4,217 association rows (companies__contacts, companies__deals, contacts__deals) that normalize into their own tables. 1.85x.
• On dltHub: 2m 11s, ~$0.04 a run, ~$1/month daily. A per-row tool meters the 9,177, not the 2 minutes.

GitHub — 124,875 rows, total 33m 8s (32m 20s extract, 14s normalize, 32s load).

Extract is 98% of the clock. Normalize and load are 46 seconds combined, for 125k nested rows. The other 32 minutes are the API handing back 100 items at a time and making us sleep until the window resets. That time is not ours to optimize, and a bigger worker doesn't touch it — on compute-hour billing you pay for the wait the source imposes, and that's the whole bill.

Both sources landed roughly 225–275k rows an hour — HubSpot ~252k, GitHub ~226k — but read that as the rate each API allowed on the day, not a number dlt sets. Point a faster API at the same worker and it goes faster; point a stricter one and it crawls.

Files: the format decides, not the engine

SQL and Parquet hand you a fixed shape — the types come with the data. JSON and REST don't: both arrive untyped, and the engine has to work out what every field is. The difference between the slow case and the fast one is exactly that — typed or untyped at the source. We ran both ends on the same worker.

JSON is untyped, so the CPU pays. Every field gets inspected, every type inferred, every nested array split into its own table. On messy data the inference is the job. We built the input to be hostile: 6.9M deals where a field is a number in one record and a string in the next, three date formats, values that are empty or null or missing. Normalize ends up being most of the run.

5 GB JSONL, 6.9M deals, GCS → BigQuery. Total 48m 16s (extract 15m 45s, normalize 30m 19s, load 2m 11s).

Normalize is 63% of the run, CPU-bound and pinned on a 2-core box. This is the one case in the series where throwing hardware at it actually moves the number — a bigger machine, or splitting normalize across more workers. The work parallelizes; the 2 cores were the limit, not the engine. Rate: ~4.6 GB/hour landed, ~8.6M source records/hour. The 6.9M deals become ~38M rows once contacts and tags split out — size the job by source records, not by what lands, or you'll be off by 5x.

Parquet is typed at the source, so the CPU does nothing. The schema travels with the file. No inference, no type coercion, the engine reads column batches and writes them. No normalisation needed.

TPC-H SF20, 6.44 GB across 8 Parquet files, GCS → BigQuery via Arrow. Total 9m 24s (extract 7m 9s, normalize 8s, load 1m 58s), 166.8M rows at ~389K rows/sec.

Normalize: 8 seconds, against 30 minutes for JSON. Same engine, same worker. The only thing that changed is the source carried its schema. Rate: ~170 GB/hour, ~1B rows/hour. (6.44 GB of Parquet lands as 26.69 GB in BigQuery — that's compression unpacking on read, not duplication.)

JSON took 48 minutes, Parquet took 9, and Parquet moved 25x more rows doing it. If you control how the data gets written, writing Parquet instead of JSON is the largest single lever on this bill.

MongoDB: The fast way or the normalized way

The JSON case above was 6.9M messy deals read from files. We loaded the same dataset from MongoDB.

dlt supports 2 ways to grab data from Mongo. Pyarrow and BSON object.

• Pyarrow enables moving the data without retyping, but mongo types can have higher entropy than what pyarrow accepts, which sometimes breaks pipelines and requires typing to string first.
• Bson means everything is strongly typed and normalised by dlt engine which is of course much slower than skipping the operation. This method is resilent because dlt ignores the mongo types and re-types the data for the destination.
  

Setup: 2 vCPU / 4 GB worker → BigQuery. MongoDB collection deals_full: 6,902,021 documents, 4.94 GB data size, avg object 716 bytes, one index.

• pymongoarrow (data_item_format="arrow"): total 4m 19s (extract 3m 29s, normalize 19s, load 31s).
• object (data_item_format="object"): total 38m 39s (extract 10m 10s, normalize 25m 13s, load 3m 16s).

Object normalizes the nested arrays into child tables, the same shape as the JSON case. Arrow keeps them columnar and lands one:

Arrow is 9x faster, but not like-for-like: the normalizer has less to infer, and it never does the flattening at all. Object's 25 minutes of normalize buys type inference and three relational tables.

pymongoarrow infers the schema per batch, and on fields that drift between number and string across documents, batch schemas stopped matching and the load failed. The fix was pinning the ten unstable columns to a fixed string type. Manual, and lossy: numbers arrive as text. Object needed no such intervention. The normalizer absorbed the drift on its own, routing conflicting values into typed variant columns instead of failing.

dlt runs every bottleneck at its ceiling

Five cases, same worker, ~30x between fastest and slowest.

• SQL ran at the network's ceiling.
• Parquet ran at I/O's ceiling.
• JSON ran the CPU flat out until the cores ran out.
• REST sat at exactly the API's rate and added nothing on top of the wait.
• MongoDB emulated either parquet or json behavior depending on chosen method

dlt gets out of the way when the data is easy to transfer - no overheads, high resilience. When the data is not easy to transfer, dlt solves that at the cost of compute.

Try it on your own data

dltHub bills time, not rows or GB, so the fan-out that runs up a per-row invoice costs you nothing here. The trial includes 30 hours, no credit card. Overage costs $1 an hour: dlthub.com.