A common failure mode: a pipeline appends 50 GB of new data and still rewrites a 5 TB table every run.
That is the full rewrite anti-pattern.
It usually looks respectable on the surface. Fetch a delta, union it with history into a replacement table, drop the old table, rename the new one. The job calls itself incremental because the input was incremental.
The output is not incremental. The output photocopies the table.
Why it feels attractive
The pattern is easy to reason about. You can inspect a complete table before replacing the old one. You avoid thinking about deletes, partition replacement, and idempotency. The code is often simple enough to survive for years.
That is exactly why it is dangerous.
The cost model is hidden inside the write path. A small logical change produces a full physical rewrite. Old files pile up until snapshot expiry and cleanup catch up. Snapshot history grows. Compute spends most of its time rewriting rows that did not change.
The number to check
Compare bytes written per run with total table size.
If they are close, your incremental pipeline is probably lying.
The delta size is not the number that matters. The number that matters is what the platform actually writes.
Better patterns
For partitioned data, replace only the partitions that changed where possible. With Spark and Iceberg, dynamic overwrite can do that in one atomic commit.
For row-level changes, Iceberg recommends MERGE INTO because it can rewrite only the affected data files. If your engine does not support those shapes cleanly, delete plus insert can still beat a full rewrite, but only with idempotency and failure handling.
The right design depends on table shape, engine support, and failure semantics. The wrong design is approving a full-table rewrite just because the code path was easy.
Review checklist
Before approving a rewrite, ask:
- What fraction of the table actually changes?
- Does write volume shrink when the delta shrinks?
- How many old files become unreachable after each run?
- Can the job fail between delete and insert?
- Is compaction cleaning up a real problem or masking the bad write pattern?
Full rewrites are sometimes justified. Backfills, schema corrections, and physical layout changes can need them.
But if the pipeline rewrites the lake every night because nobody modeled the update, that is not architecture. That is a recurring invoice.