Bad data stops before the warehouse

Expectations, schema contracts and redaction run on the way in. Analytics engineers hear about a broken source at the pipeline that loaded it, while the fix is cheap and the warehouse is still clean.

Checks, contracts and redaction

Quality governs the values, contracts govern the shape, and redaction governs what is allowed to land at all. All three run in the pipeline rather than in a dashboard downstream.

Docs

Data quality

Declare expectations once. Rows that fail are quarantined before they reach the warehouse, per dataset, table or row.

Checked per dataset, table or row

38 rows held back. Load shipped.
Docs

Contracts

Declare what the warehouse is allowed to receive. Schema changes are accepted, blocked or quarantined by rule, on the run that introduced them.

Schema changeDatasetContract says
Column added: discount_pctordersAllowed, added
Type changed: amountordersBlocked, run halted
Table added: refundsfinanceAllowed, added
A type change stopped the run at 06:07.
Docs

Redaction

Mark a column as sensitive and it is redacted before the load, so the raw value never lands in the warehouse.

Redacted before the first write.