Data Quality Management: Where Checks Should Run

When a warehouse test or a dashboard finds bad data, the data has already landed. Someone corrects it, but the source keeps sending the same error, so they have to repeat the fix. Data quality is managed in the pipeline by the people who own it, not in a dashboard after the data lands. Warehouse tests and monitoring still have a job: they check data that only exists after the load.

What is data quality management?

Data quality management means deciding what data may land, checking it as it enters, and holding the pipeline owners responsible for fixing what fails. The UK framework defines the goal: data that is fit for its intended use. The definition alone does not say where each check belongs. That decision comes first.

Where should data quality checks run?

Before the load, if the check decides whether data may land. After the load, if the check needs data that only exists in the destination, such as earlier loads.

A wrong column type or a null in a required field can be judged one record at a time. Stop that record before it lands. Rapido checks each incoming record against its registered schema. Records that fail go to a dead-letter queue with the reason, so a user can correct them and resend them. A warehouse test for the same null finds it only after the data lands.

dltHub schema contracts govern permitted schema changes. They run before the load: during normalization, or already during extraction for Arrow, pandas, and Polars data.

• evolve****: accepts schema changes.
• freeze****: fails the run on a prohibited schema change, so no data from that load reaches the destination.
• discard_row****: drops the row that does not fit the schema.
• discard_value****: drops the value that does not fit and loads the rest of the row.

Contracts define the data shape; value checks currently run after the load, with pre-load checks in progress.

After the load, dltHub data quality checks run on the destination so that they can read the current load and earlier loads through the dataset. Each top-level row carries the ID of the load that produced it so that you can trace a failing row back to its load.

How does data quality management differ from data governance?

Governance decides what good data means. The pipeline owner makes it true. The UK framework asks for accountability at every level, gives strategic direction to data leaders, and asks organizations to understand their users’ quality needs. Governance assigns ownership, and it should go to the pipeline owner. Data quality management measures data against the standards governance sets and corrects what fails.

Each needs the other. A standard that nobody checks has no effect. A check that nobody answers produces numbers and no change.

What should you measure?

Fewer rules than you think. Pick the dimensions that your data users depend on, and enforce those strictly.

The UK framework names six core dimensions and says they are not fixed:

• Completeness: required records and essential values are present.
• Uniqueness: records do not contain unintended duplication.
• Consistency: values do not contradict other values for the same entity.
• Timeliness: data reflects the correct period and is available when needed.
• Validity: data has the expected range and format.
• Accuracy: data matches reality.

For each table, write rules only for the dimensions whose failure would break a decision downstream.

How do teams manage data quality in practice?

They keep checks with the pipeline code. A rule change then gets the same review as a pipeline change.

In dltHub, you write checks as Python objects and attach them to resources with @dq.with_checks(...). They stay in the pipeline repository and go through pull-request review and a staging run before production. The pipeline passes the same check objects to dq.run_checks(...) after the load, which stores one summary row per check in _dlt_checks. The pipeline owner sets the rules and thresholds.

Who owns data quality?

The pipeline owner. They must hear about the failure and coordinate or apply the fix at the source.

At Rapido, producers define their own schemas. Notified users correct and replay failed records.

In dltHub, a freeze contract fails the run, and Slack or email tells the owner that the run failed. The job inspector reads the run record, the logs, and the job definition. It reports the likely cause of the failure, its confidence, the evidence, and a proposed fix. It does not change code or data, so a person reviews and applies the fix.

What tools are needed for data quality management?

Choose by where the check runs and who owns it.

GX Core

Scope: GX Core defines checks as Python Expectations, grouped into suites. Execution: It runs Validations where you invoke it, and you choose the timing. Failures: It returns Validation Results and runs the Actions you configure.

Data Quality within Informatica Intelligent Data Management Cloud

Scope: Data Quality within Informatica Intelligent Data Management Cloud supports profiling, rules, dictionaries, cleansing, deduplication, and exception records. Execution: Rules run in profiling or Data Integration tasks. Failures: The configured rule outputs or exception tasks set the response.

dltHub

Scope: Schema contracts govern permitted schema changes, and data quality checks can access loaded data from the current and earlier loads. Execution: Contracts run before the load, and checks run on the destination after it. Failures: The configured contract mode determines whether dlt accepts a change, fails the run, or removes nonconforming data. dq.run_checks(...) stores one summary row per check in _dlt_checks.

Start with what may land

Write the rules that decide what may enter first. Then name the owner of each failure. Add after-load checks when the questions need destination data, including current or earlier loads.

Frequently asked questions