What Is Data Governance in Data Engineering?

An unexpected column is added to an existing schema with a defined contract. The contract rejects the column before the load begins. No data loads, and the run trace shows the validation failure. When governance lives only in policy documents, a council, and a data catalog, it sets the rules but cannot stop that load. You need to turn each standard into a control, assign ownership of failures, and preserve evidence for each run.

Data governance in data engineering includes ownership, data standards, and the controls pipelines enforce, plus the records each run creates to show what happened. Ownership determines which policies apply and what standards the data must meet. Controls apply those decisions. Atlan makes a similar distinction between governance standards and the daily work that meets them.

What does data governance cover?

Data governance covers these areas:

• Ownership and stewardship assign decisions and daily responsibility.
• Standards define permitted data shape and values.
• Access and security decide who can reach data, credentials, and pipelines.
• Sensitive data needs its own classification and masking controls.
• Lineage and metadata explain source and change.
• Quality, retention, and compliance determine what teams check and which evidence remains available.

The governance team decides, while the data engineer enforces controls, records outcomes, and responds to failures.

This work matters because unapproved changes can cause incidents and weaken trust. They can also complicate audits and increase regulatory exposure. During an audit, engineers must show which rule ran and what happened. During an incident, they must identify the unapproved change and its result. SR 26-2 addresses model validation and exceptions. ALCOA+ requires life-sciences records to remain trustworthy throughout their lifecycle.

Access is a governance decision: someone must approve who can access production data, credentials, and pipelines, and with what rights. In dltHub, each user has an organization role and a role in each workspace, and a workspace API key works in only one workspace. Deployed pipelines use the production secrets in the prod profile, while interactive notebooks and agent jobs use the read-only access profile by default.

Put controls where changes occur

Put a control where the governed event occurs, because later evidence cannot prevent an unapproved change from loading. In a dltHub pipeline, you set the schema contract for tables, columns, or data types. dltHub evaluates incoming data after name normalization and before it loads the data into the destination. The contract applies your decision; it does not decide which changes your organization permits. The contract lives in workspace code that runs locally, in CI, and on the platform. A change to the contract can then go through the same pull-request review as any other code change.

A column set to freeze stops the load when an unexpected change arrives, so no data loads. A documented example permits new tables but freezes columns, and an unexpected column fails the pipeline step before loading.

Other contract modes keep the load running: they accept the change, or drop the offending row or value. Choose the mode from the approved data standard and the consequence of accepting bad data. Checks against data already in the destination serve a different purpose.

Governance defines rules; quality measures data

Governance sets responsibility, which policies apply, and what standards data must meet, while data quality measures values against those rules.

In dltHub, schema contracts control shape before loading. When you call dq.run_checks() with an explicit check mapping, checks assess loaded values and write summaries to _dlt_checks. The data quality article covers check placement and failure handling in detail. Contract failures and check results both leave a record, and each record answers a different question.

Each artifact answers a different question

Each artifact a run leaves answers a different question, because no single artifact explains the full run. A dltHub schema shows the tables, columns, types, and contract settings. A run shows status, timestamps, logs, and row counts. A freeze violation appears in the trace, while a check result appears in _dlt_checks.

The dltHub context catalog records object-level lineage. Lineage carried from source to model is in public preview. Each load package receives a load_id, and top-level destination rows receive _dlt_load_id. Nested rows can also carry generated parent-child identifiers.

Retention must match the period in which your organization needs the evidence. dltHub sets a default maximum of 90 days for pipeline logs. The pipeline working directory keeps only the most recent trace. Schemas and load history follow separate storage rules. Preserve required evidence before later runs or storage changes replace or remove it.

The pipeline owner owns each failure

The pipeline owner owns each failure until the team resolves it, accepts an authorized exception, or escalates it. The policy defines permitted behavior and names the person who can approve exceptions.

If your team sends failed-job alerts to Slack or email, the owner follows the alert to the dltHub run-detail page. The run detail identifies what failed and where the owner should start.

An accepted exception also needs evidence. Preserve who approved it, what failed, why the approver granted it, and the period covered. Otherwise, the pipeline shows a failure without explaining the decision that followed.

Start with one enforceable standard

Choose a violation that matters to the data consumer. Write down who responds, where the control runs, what happens on failure, and where the result appears.

Trigger a known violation before you rely on the control. Confirm that the configured behavior occurs and that the owner can find the result.

Governance starts with one owned rule that the pipeline can enforce and explain.

FAQ