Modern Lakehouse Architecture with Apache Iceberg, dbt and Airflow

Separate the responsibilities behind the platform

A lakehouse can accumulate object storage, table formats, SQL engines and orchestration without becoming a reliable data platform. The failure usually appears when a team cannot explain which dataset is authoritative, whether a retry will duplicate rows or why two consumers read different versions of the same business event.

This guide is for data engineers and technical decision makers designing a lakehouse with Apache Iceberg, dbt and Airflow. The useful starting point is a division of responsibilities: how data is stored, how table state is committed, how transformations are defined and how scheduled work is recovered.

Key takeaways

• Treat object storage, table metadata, compute and orchestration as separate concerns.
• Use stable data contracts and explicit table ownership across layers.
• Design each load around a reproducible input interval or source version.
• Validate engine and adapter support before relying on a table feature.
• Budget for compaction, retention, access control and operational recovery.

Establish a clear architecture and ownership model

Object storage holds data and metadata files. Iceberg provides a table format and metadata structure that supported engines use to interpret table state. A catalog helps locate and coordinate tables. Compute engines execute reads and writes. dbt defines transformations through the capabilities of its adapter, while Airflow coordinates scheduled dependencies and task execution.

Those boundaries matter. Airflow does not make a non-idempotent SQL statement safe to retry. dbt does not guarantee that every engine supports the same Iceberg operation. Object storage permissions alone may not express all business-level access rules. Assign ownership for each layer and test the combinations you plan to deploy.

+--------------------+
| Source systems     |
+----------+---------+
           |
+----------v---------+       +--------------------+
| Ingestion + checks | <---- | Airflow scheduling |
+----------+---------+       +----------+---------+
           |                            |
+----------v---------+       +----------v---------+
| Raw immutable data | ----> | dbt transformations|
+--------------------+       +----------+---------+
                                        |
                             +----------v---------+
                             | Iceberg tables     |
                             | Catalog + metadata |
                             | Object storage     |
                             +----------+---------+
                                        |
                             +----------v---------+
                             | SQL, BI and ML     |
                             +--------------------+

The Iceberg documentation describes capabilities such as schema evolution and table snapshots. Treat those as table-format capabilities that must still be matched to the chosen engine, catalog and version. Build a compatibility matrix for the operations you need, especially merges, deletes, maintenance and concurrent writers.

Choose a table owner and a consumer contact for every published dataset. The owner is responsible for schema changes, quality expectations, freshness and incident handling. The consumer contact helps establish what a broken or delayed table actually affects. Without ownership, a platform can be technically healthy while serving unusable business data.

Design layers around meaning and replay

Raw, cleaned and published layers are useful when they describe different contracts. A raw layer preserves source events or extracts with provenance. A cleaned layer resolves types, duplicates and agreed business keys. A published layer exposes stable entities and measures for consumers. The names matter less than the promises attached to each layer.

Keep raw data sufficiently faithful to support investigation and replay, while respecting retention and privacy requirements. Do not assume that immutable means permanent. Some data must be deleted or restricted, and your architecture needs a procedure for propagating those requirements through derived tables and retained snapshots.

For batch ingestion, identify inputs by source version, partition or extraction window. Record a manifest containing file references, row counts, checksums where appropriate and the run identifier. An orchestration retry should consume the same approved input set rather than silently reading whatever happens to be newest.

Define the grain before choosing partitions

A table's grain says what one row represents. An orders table might contain one row per order, while an order-events table contains one row per transition. Confusing these grains creates duplicate joins and unreliable aggregates. Define primary business keys and update semantics before optimizing physical layout.

Partitioning should reflect query patterns, data volume and engine behavior. Excessively fine partitions can create many small files and metadata overhead. A partition choice that works for a large event table may be wasteful for a small dimension. Measure representative queries and inspect file distribution before treating a naming convention as a performance strategy.

The following SQL is a runnable PostgreSQL example that illustrates grain and deduplication. It is not Iceberg DDL; the selected lakehouse engine determines the corresponding table and merge syntax. The source sequence provides a deterministic tie-breaker for this illustrative dataset.

WITH source_rows(order_id, source_sequence, amount) AS (
  VALUES
    ('o1', 1, 20.00),
    ('o1', 2, 25.00),
    ('o2', 1, 12.00)
), ranked AS (
  SELECT *, row_number() OVER (
    PARTITION BY order_id
    ORDER BY source_sequence DESC
  ) AS position
  FROM source_rows
)
SELECT order_id, amount
FROM ranked
WHERE position = 1;

Make transformations testable and repeatable

Use dbt models to express transformation dependencies and document assumptions close to the SQL. Keep source normalization, business logic and presentation aggregates distinct enough to review independently. A model that combines ingestion repair, customer identity resolution and executive metrics is difficult to test or change safely.

Tests should describe the dataset contract. Uniqueness checks verify the declared grain, not merely the presence of an arbitrary ID. Relationship checks can reveal missing dimensions. Business checks may compare totals or enforce valid transitions. Explain what a failure means and whether it blocks publication or sends data to a quarantine process.

The dbt data tests documentation explains built-in and custom checks. Use those facilities where they fit, but remember that a test passing today does not prove the model handles tomorrow's schema change. Test transformation behavior on small fixtures containing late updates, missing values and duplicate source records.

For incremental models, define exactly which rows can change. A watermark based only on event time can miss corrections to older events. An update timestamp may be more suitable if the source maintains it reliably. A bounded lookback can reduce risk, but it still requires a reconciliation strategy for changes older than the lookback window.

Load strategy	Useful when	Main trade-off
Full rebuild	Data is small or logic changes broadly	Higher compute and longer publication time
Append	Source events are immutable	Corrections need separate semantics
Keyed merge	Existing business entities change	Requires stable keys and supported write behavior
Partition replacement	Inputs map to complete partitions	Partial inputs can erase valid data

Keep transformation configuration versioned with the model code. A change to an adapter, table materialization or unique-key setting can alter correctness as much as a SQL edit. Test those changes in a representative environment and compare both row-level differences and aggregate expectations before promoting them.

Use Airflow to coordinate bounded units of work

An Airflow task should have a clear input boundary, output artifact and retry policy. Pass references to data rather than moving large datasets through orchestration metadata. Make success mean that the output is ready for the next consumer, not simply that a subprocess exited without an error.

The Airflow best practices emphasize repeatable task outcomes and avoiding incomplete published results. Apply that principle by writing to a staging location, validating the result and publishing it through the table engine's supported commit mechanism. Consumers should not discover half-written outputs by listing a folder.

This YAML is an application-level orchestration sketch, not an Airflow configuration API. It describes the contract to implement using the operators and scheduling interfaces supported by your installed Airflow version.

pipeline: daily_orders
input:
  source_manifest: "manifests/orders/2026-10-30.json"
  interval_start: "2026-10-30T00:00:00Z"
  interval_end: "2026-10-31T00:00:00Z"
steps:
  - name: validate_manifest
    retries: 0
  - name: load_staging
    retries: 2
    idempotency_scope: source_manifest
  - name: transform_and_test
    retries: 1
  - name: publish
    requires: all_checks_passed
failure:
  retain_staging: true
  notify_owner: data_platform

Set timeouts and concurrency based on downstream capacity. A backfill should not starve the daily pipeline or overload the catalog. Separate queues or resource pools may help, but the exact configuration depends on your deployment. Define priorities and cancellation behavior before a large repair job competes with normal production work.

Make backfills reviewable

A backfill plan should identify affected intervals, source versions, transformations and downstream tables. Estimate resource needs from a small representative run rather than guessing from total row count alone. Record whether published results will change and how consumers will learn about corrections.

Test rerunning the same interval twice. Compare logical outputs and operation records. If row counts grow unexpectedly or previously published data disappears, the pipeline is not ready for automatic retries. Resolve those semantics before adding more scheduling sophistication.

Operate metadata, files and consumer expectations

A lakehouse needs maintenance as well as transformation jobs. Small files can increase planning and read overhead. Retained snapshots and orphaned files consume storage. Maintenance procedures can remove information needed by active readers or recovery workflows if retention is configured carelessly. Coordinate policies with the engines and applications that use the tables.

Observe freshness from the consumer's perspective. A successful orchestration run does not prove that the expected source interval arrived or that the published table contains it. Track source completeness, ingestion lag, transformation lag and publication time separately. This makes an upstream delay distinguishable from a slow query or broken task dependency.

Access control should cover catalog operations, underlying storage and query surfaces. A restricted table is not protected if a broad storage credential allows direct file reads. Audit service identities and avoid sharing a single administrative credential among ingestion, transformation and reporting jobs.

For contract design and failure handling, see data contracts and quality checks. The connection is practical: table ownership, freshness definitions and schema compatibility need to be agreed before a quality gate can make a meaningful publication decision.

Rehearse recovery from a bad publication

Choose a staging table and publish an intentionally incorrect transformation version using synthetic data. Practice identifying the producing run, locating the prior approved table state and restoring the consumer view through the engine's supported mechanism. Verify downstream behavior, since reverting a table does not automatically undo exports or cached dashboard results.

Document the retention window needed for this procedure. If the old snapshot or source files have already been removed, recovery may require a rebuild rather than a fast rollback. Treat that difference as an explicit service expectation and test it periodically after maintenance-policy changes.

Finally, review total operating cost across storage, compute, metadata operations and engineering time. A flexible platform can become expensive when every team maintains its own incompatible write path. Standardize the small set of supported patterns that covers actual workloads, and introduce exceptions with a documented owner and reason.

Common mistakes

• Treating the table format, catalog, engine and scheduler as one interchangeable product.
• Publishing tables without a declared grain, owner or freshness expectation.
• Retrying jobs that append duplicate data or read changing input sets.
• Assuming every dbt adapter supports the same Iceberg features.
• Running retention or file cleanup without testing reader and recovery requirements.

Checklist

• Assign ownership across storage, catalog, compute and orchestration.
• Validate the required engine, catalog and adapter combinations.
• Define table grain, business keys and update semantics.
• Record source manifests and reproducible processing intervals.
• Test transformations, retries, incremental corrections and backfills.
• Establish maintenance, retention and access-control procedures.
• Rehearse bad-publication recovery with downstream consumers.

FAQ

Does Iceberg replace a query engine?

No. It defines a table format that compatible engines use to read and write table state. You still need compute, a catalog strategy and operational controls appropriate to your environment.

What does dbt contribute to a lakehouse?

It provides a way to organize SQL transformations, dependencies, documentation and tests through an adapter. The adapter and engine determine which materializations and table operations are available.

Should every pipeline use Airflow?

No. A small event-driven or managed ingestion workflow may have simpler orchestration options. Airflow is useful when scheduled dependencies, backfills and operational visibility justify running or adopting that orchestration layer.

Is a raw layer always permanent?

No. Retention must account for privacy, legal obligations, storage cost and recovery needs. Define a policy that preserves enough evidence for replay without assuming all source data should be kept forever.

Final thoughts

A reliable lakehouse is built from clear contracts between layers. Reproducible inputs, explicit table ownership and tested recovery matter before a platform's diagram becomes impressive.

For architecture or implementation support, explore data engineering or pipeline engineering, then book a conversation.