Data Contracts and Data Quality Checks for Reliable Pipelines

Make expectations visible before the pipeline breaks

A pipeline can finish on schedule and still deliver unusable data. A producer changes a field's meaning, a currency column disappears, or a valid-looking timestamp starts arriving in local time. The SQL runs, the dashboard refreshes and the problem surfaces only when someone makes a decision from the wrong result.

This guide is for engineers and technical leaders establishing data contracts and practical quality checks. The goal is to turn informal expectations into versioned agreements with owners, executable validation and an explicit response when incoming data fails those expectations.

Key takeaways

• Define business meaning, ownership and freshness alongside column types.
• Distinguish schema compatibility from semantic compatibility.
• Validate at ingestion and at important transformation boundaries.
• Choose blocking, quarantine and warning behavior deliberately.
• Make recovery and reconciliation part of the contract.

Write a contract that describes a usable dataset

A useful contract begins with the dataset's purpose and grain. Explain what one row represents, how it is identified and when it becomes authoritative. For an order event, the grain might be one source transition rather than one order. That distinction determines whether repeated order identifiers are valid or indicate duplicates.

Specify fields, types, nullability, units, time zones and allowed values. A decimal amount without a currency or unit is incomplete. A timestamp without a definition may mean event time, processing time or the time of the last update. These are different facts and should not share an ambiguous name.

Ownership is part of the interface. Name the producer responsible for the source and the consumer responsible for interpreting it. Include a change-notification path, expected availability, retention and incident escalation. A contract that nobody owns is documentation that will drift when delivery pressure arrives.

The following YAML is an illustrative contract format, not a configuration for a specific library. Adapt it to your organization's tooling while keeping the semantics explicit.

dataset: order_events
version: 1
owner: commerce_data
grain: one source transition per order
key: [order_id, source_sequence]
time:
  event_time: occurred_at
  timezone: UTC
fields:
  order_id: {type: string, nullable: false}
  source_sequence: {type: integer, nullable: false}
  amount_minor: {type: integer, nullable: false}
  currency: {type: string, allowed: [USD, EUR]}
  occurred_at: {type: timestamp, nullable: false}
quality:
  duplicate_keys: reject_batch
  unknown_currency: quarantine_record
availability:
  freshness: agreed_with_consumers
changes:
  breaking: new_version_and_migration

The currencies and policies in this example are illustrative choices. Real contracts should reflect the source's supported currencies and the business consequences of each failure. Do not copy a reject rule merely because it looks strict; rejecting a complete operational feed may be worse than isolating a small invalid subset.

Distinguish promises from observations

A contract records what the producer promises. A profile describes what the data happened to look like in a sample. A field that was always populated last month is not necessarily guaranteed to remain non-null. Use profiling to discover questions, then confirm expectations with owners before turning accidental patterns into blocking tests.

Place checks at the boundaries that matter

Check raw input before it contaminates derived tables. Validate schema, required fields and parseability, then preserve enough source context to investigate failures. A malformed row should retain its source location and ingestion identifier without being silently converted into a plausible default value.

Check transformed output against its intended grain and business rules. An input can be individually valid while a join duplicates every row. Uniqueness, relationships and reconciliation checks after the join detect a different class of error than ingestion validation. Both are needed when the transformation changes meaning.

+------------------+     +------------------+
| Producer contract| --> | Input validation |
+------------------+     +--------+---------+
                                  |
                    +-------------+-------------+
                    |                           |
          +---------v----------+     +----------v---------+
          | Accepted records   |     | Quarantine + reason|
          +---------+----------+     +--------------------+
                    |
          +---------v----------+
          | Transform + checks |
          +---------+----------+
                    |
          +---------v----------+
          | Publish + monitor  |
          +--------------------+

The following runnable Python example validates a small in-memory batch using only the standard library. It deliberately rejects booleans as integer sequences because Python considers bool a subclass of int. Production validation should also cover timestamps, currencies and schema completeness according to the actual contract.

def validate_batch(rows):
    seen, errors = set(), []
    for index, row in enumerate(rows):
        order_id = row.get("order_id")
        sequence = row.get("source_sequence")
        if not isinstance(order_id, str) or not order_id.strip():
            errors.append((index, "invalid order_id"))
            continue
        if type(sequence) is not int or sequence < 0:
            errors.append((index, "invalid source_sequence"))
            continue
        key = (order_id, sequence)
        if key in seen:
            errors.append((index, "duplicate key"))
        seen.add(key)
    return errors

rows = [
    {"order_id": "o1", "source_sequence": 1},
    {"order_id": "o1", "source_sequence": 1},
]
assert validate_batch(rows) == [(1, "duplicate key")]
print(validate_batch(rows))

If you use dbt, its data tests provide a familiar place for uniqueness, nullability, relationship and custom assertions around models. Keep the test's meaning visible in the model documentation. A failing count is more actionable when the reader knows which business promise it violates.

Choose failure behavior by consequence

Not every failure deserves the same response. A missing primary business key may make a record unusable. A delayed optional enrichment might justify publishing the core record with a visible quality status. A sudden collapse in volume may require pausing the whole dataset until an upstream outage is understood.

Define whether a check blocks publication, quarantines individual records or emits a warning. Include who is notified and what evidence accompanies the notification. Avoid the vague policy of log and continue when nobody regularly reviews the logs. A warning is useful only if a process exists to act on it.

Response	Suitable for	Main risk
Block batch	Dataset-wide integrity failure	Valid data is delayed too
Quarantine record	Isolated invalid records	Published totals may be incomplete
Publish with status	Known partial data with consumer support	Consumers may ignore the status
Warn and monitor	Low-impact anomaly needing review	Problems can accumulate unnoticed

Quarantine needs an operational design. Store the rejected record or a protected reference, rule identifier, source version and failure time. Provide a review and replay path. Restrict access if the records contain sensitive information, and apply retention limits so quarantine does not become an unmanaged archive.

Make partial publication honest

When publishing accepted records while quarantining others, expose completeness information. A dashboard total that excludes rejected orders should not appear final without qualification. Consumers need to know the affected interval, approximate scope where measurable and whether a corrected publication is expected.

For aggregates, record both the input population and the included population. This helps distinguish an actual business decline from a validation change that excluded more rows. The contract should explain whether downstream users can operate on partial data or must wait for a complete corrected version.

Manage changes as interface migrations

Schema compatibility is only one part of change management. Adding an optional field may be compatible for many consumers, but changing an amount from major units to minor units without changing its type is a breaking semantic change. Similarly, keeping a timestamp column while changing its time zone can silently alter daily aggregates.

Classify changes by their effect on actual consumers. Record which fields are read, which values are assumed and which downstream calculations depend on them. A consumer inventory is more useful than a blanket claim that all additive changes are safe. Some strict parsers reject unknown fields, while others ignore them.

For breaking changes, publish a new contract version and a migration plan. Consider dual publication during a transition, with reconciliation between versions. Set a retirement date only after consumers have tested their migration. Keep the old semantics documented so historical data remains interpretable.

Contract checks belong in producer and consumer development workflows. A producer can validate sample output against the agreed schema. A consumer can run transformation fixtures covering old and new versions. These checks catch interface drift earlier than a production dashboard alert, but they still need real data monitoring after deployment.

Review an illustrative currency migration

Suppose a source changes amount from a decimal value to an integer count of minor units. Introduce an explicitly named field, document the currency's scale and update consumers to use it. Compare old and new calculations on a controlled dataset that includes refunds, zero values and currencies with different conventions if they are supported.

Do not simply cast the new integer to a decimal and assume the meaning is preserved. The migration must specify conversion and rounding rules. Keep the source version with historical rows so a backfill does not apply today's interpretation to yesterday's representation accidentally.

Recover failures and measure trust

Recovery should begin with a stable input set. Record source files or offsets, contract version, transformation version and validation results for each run. When a rule is fixed, replay the affected inputs into a controlled destination and compare the corrected output before replacing published data.

Distinguish a bad source record from an incorrect validation rule. Relaxing a rule to clear an alert may admit genuinely invalid data. Conversely, keeping a faulty rule can reject valid business activity. Require an explanation and evidence for changes to blocking rules, especially when they alter reported totals.

The Kafka and CDC pipeline guide discusses replay and source ordering in event streams. Contracts complement those mechanics by specifying what an event means, how deletion is represented and which violations should stop or isolate processing.

Measure useful outcomes: repeated rule failures, time to identify ownership, time to repair, unreviewed quarantine volume and consumer incidents. Avoid declaring a dataset trustworthy merely because many tests exist. A large set of redundant checks can miss the one semantic rule that determines whether a metric is correct.

Reconcile after replay

After replaying an affected interval, compare keys, counts and domain totals against the approved input and prior publication. Explain expected differences and investigate unexplained ones. Check that reprocessing did not duplicate accepted rows or resurrect records that had been intentionally deleted.

Notify downstream owners when corrected data changes a published result. Some consumers cache extracts, send reports or train models on earlier versions. A repaired table does not automatically repair those artifacts. Record which downstream outputs need refresh or annotation and who owns that follow-up.

Common mistakes

• Documenting column types while omitting units, time zones and row grain.
• Turning a temporary data profile into an undocumented permanent promise.
• Running only input checks and missing duplication introduced by transformations.
• Quarantining records without a reviewer, replay procedure or retention policy.
• Relaxing a failed rule to restore a green dashboard without understanding the cause.

Checklist

• Define dataset purpose, grain, keys, units and temporal meaning.
• Name producer and consumer owners with an escalation path.
• Version contracts and test both schema and semantic changes.
• Validate ingestion and important transformation boundaries.
• Assign blocking, quarantine or warning behavior to every important rule.
• Preserve reproducible inputs and a controlled replay process.
• Reconcile corrected output and notify affected consumers.

FAQ

Is a data contract just a schema?

No. A schema describes structure, while a useful contract also describes meaning, ownership, availability and change behavior. Those additional expectations determine whether consumers can use structurally valid data correctly.

Should every failed check stop the pipeline?

No. Choose the response based on the consequence of publishing invalid or incomplete data. Some failures require blocking the dataset, while others can be isolated if consumers understand the resulting limitations.

Who should own data quality?

Producers own the correctness of the interface they publish, and consumers own the assumptions and transformations they apply. Platform teams can provide tooling, but they should not become the default owner of every business definition.

How do contracts help with backfills?

They identify the meaning and validation rules associated with each data version. A backfill can then replay historical inputs using an explicit interpretation rather than silently applying incompatible current assumptions.

Final thoughts

Data contracts work when they connect expectations to executable checks and operational decisions. Make meaning explicit, assign ownership and design the repair path before the first important dataset fails.

For help applying these practices, explore my pipeline engineering service and book a conversation.