For years, the answer to “how do we get Kafka data into the lake?” was a job in the middle.
A Kafka Connect cluster. A Spark Structured Streaming job. A Flink pipeline. Something that consumed topics, wrote Parquet, and committed Iceberg metadata.
At Current London this year, every vendor was selling the same idea: delete the job. Confluent Tableflow, StreamNative Ursa, Aiven Iceberg Topics, Streambased zero-copy views. Two weeks later, Snowflake announced Datastream and joined from the warehouse side.
Before you pick one, it is worth understanding what the job in the middle actually did. Because every one of these products still has to do it — they just disagree about where.
What the middle layer actually solved
Every Kafka-to-Iceberg pipeline, no matter the engine, had to solve the same five problems:
- Atomically tie Kafka offsets to Iceberg snapshots, so a restart does not duplicate or lose data.
- Avoid commit contention, because many writers committing to one table bloats metadata and causes retries.
- Evolve the table schema when the events change.
- Control small files, because freshness multiplies file count.
- Maintain the table — compaction, snapshot expiry, orphan cleanup — forever.
The classic tools each solved the first two well and left the rest to you.
The Kafka Connect Iceberg sink elects one worker as a coordinator that performs a single commit per table per interval (five minutes by default), with its commit progress recorded in the Iceberg snapshot itself. Clever design.
But the Apache version is append-only — upsert support was dropped over equality-delete performance concerns — and compaction is explicitly not included.
Spark Structured Streaming gets exactly-once from checkpoints plus idempotent Iceberg commits, as long as nobody touches the checkpoint directory. Change data capture (CDC) means MERGE INTO per micro-batch, which rewrites files on every batch and gets painful when most rows are inserts.
Flink commits on checkpoints, so table freshness is your checkpoint interval. Upsert mode writes an equality delete plus an insert for every update — SmartNews illustrated how, with a 20-minute checkpoint and ten writers, a single partition can emit roughly 90 small files an hour, and shorter checkpoints only make it worse. And because the job’s restore point lives in the snapshot summary, aggressive snapshot expiry can corrupt a running job. Ingestion and maintenance become operationally entangled.
This template is older than Iceberg. LinkedIn’s Camus and Gobblin established it in the Hadoop era: land files fast, commit metadata separately and centrally, compact later. Nobody has escaped it since.
Why the middle layer became the bill
The pattern worked. It also accumulated four standing costs:
- Duplicate bytes. With Kafka tiered storage, the same record sits in the broker tier, the tiered segments, and the Iceberg Parquet. Aiven’s estimate: a 1 GiB/s cluster can pay to write, ship, and store the same byte up to four times before anyone runs a query — about $3.4M a year to move bytes you already had.
- A second distributed system. The Connect or Spark or Flink cluster needs sizing, monitoring, upgrades, and an on-call rotation — and it owns offsets, dedup, flush tuning, and schema mapping while neither Kafka nor the catalog helps it.
- Small files, forever. Every approach commits per interval. Wanting fresher tables means more commits, more files, more snapshots. No ingestion tool shipped compaction; every team bolted on
rewrite_data_filesandexpire_snapshotsand hoped the maintenance job kept up. - CDC pain. Streaming writers cannot know where the old row lives, so updates became equality delete files — markers that say “drop the row with this key” — that readers must reconcile at query time. Fresh-but-mutable tables were only viable with aggressive compaction someone had to own.
That list is the convergence wave’s pitch, inverted. Which is why the honest comparison question is not “which vendor?” It is “which of these costs does each design actually remove, and what does it charge instead?”
The landscape, sorted by where the bytes live
Sorting these by vendor tells you nothing. Sort them by how many copies of your data each one keeps. Every solution falls into one of five shapes:
Zero-copy views. Streambased materializes nothing. A gateway synthesizes Iceberg metadata and Parquet on the fly from Kafka segments at query time. Zero lag by construction, no table to maintain — and the bill moves to query-time CPU and Kafka read load. (Aged-out data can optionally spill to a real Iceberg “coldset,” which is then a genuine second copy.) There are no upserts, because a log view has no row identity.
One copy: the segments are the table files. Aiven Iceberg Topics does this inside the open-source tiered-storage plugin: when a segment rolls, the broker writes Parquet and commits it to the Iceberg catalog, and later serves Kafka replays from those same Parquet files. Bufstream did the same with broker-side Protobuf enforcement — until CoreWeave acquired it in May 2026 and folded it into an internal platform, which is its own lesson, below. StreamNative Ursa rebuilds the broker around it: object-storage-first, with the lakehouse table as the primary storage.
Two copies, managed materialization. Confluent Tableflow (Iceberg generally available (GA) since March 2025, with automated compaction; upsert tables came later, as a paid add-on), WarpStream Tableflow (bring-your-own-cloud (BYOC) agents that materialize from any Kafka into your buckets), Redpanda Iceberg Topics, and AutoMQ Table Topic all keep the log and the table as separate representations. You pay for the bytes twice; in exchange, table maintenance is decoupled from the Kafka read path. How much the vendor automates varies: Confluent and WarpStream run full background compaction, while Redpanda and AutoMQ auto-expire snapshots but leave small-file compaction to external tools.
The warehouse pulls the stream in. Snowflake Datastream — Kafka-compatible topics that land directly as governed Snowflake or Iceberg tables, with role-based access control (RBAC) on topics and Horizon Catalog policies on tables. Announced June 2026, private preview, so treat the architecture claims as claims. The strategic direction matters more than the details: Confluent is pushing the stream toward the lakehouse; Snowflake is pulling the stream into the warehouse.
Replace Kafka entirely. Apache Fluss (incubating, donated by Alibaba) is the solution that gets left out of the vendor lists because it refuses the premise. It is not Kafka-protocol-compatible. It is streaming storage designed for the lakehouse from scratch: columnar log storage (Apache Arrow), primary-key tables that generate changelogs natively, and a tiering service that moves data into Iceberg while “union reads” stitch fresh and historical data into one query result. The most principled answer to CDC-into-the-lake on this list — at the price of leaving the Kafka ecosystem, with Flink as the practical entry point.
The comparison table
| Solution | Conversion runs in | Copies | Upserts | Table maintenance | Status (mid-2026) |
|---|---|---|---|---|---|
| Connect / Spark / Flink job | separate cluster | 2 | engine-dependent, painful | you | mature |
| Streambased | query-time gateway | 1 (hot data) | no | none (Kafka retention) | available, small vendor |
| Aiven Iceberg Topics | tiered-storage plugin | 1 | no | you, constrained | experimental, OSS |
| StreamNative Ursa | broker (rebuilt) | 1 | no | vendor | GA on AWS, BYOC |
| Confluent Tableflow | managed service | 2 | yes (merge-on-read) | vendor | GA |
| WarpStream Tableflow | BYOC agents | 2 | no | vendor | GA Feb 2026 |
| Redpanda Iceberg Topics | broker | 2 | no | snapshot expiry only | GA (enterprise) |
| AutoMQ Table Topic | broker | 2 | yes (Debezium since 1.6) | snapshot expiry only | OSS since 1.5 |
| Snowflake Datastream | managed service | unclear | unclear | vendor | private preview |
| Apache Fluss | tiering service | 1 (tiered) | yes (PK tables) | built in | incubating |
Two readings of this table matter more than any single row.
First: the single-copy designs all share one structural tension. If the Kafka replay path reads the same Parquet files that the Iceberg table references, then nothing else may rewrite those files. Aiven is explicit that the broker is the sole writer and compaction must run carefully outside Kafka. Bufstream warned against managed catalogs that rewrite or repartition data. Your table layout becomes hostage to Kafka’s segment layout — the heart of Jack Vanlightly’s critique of zero-copy, and exactly the argument WarpStream and others make for paying the two-copy tax. There is no free option here. You are choosing which bill to pay: storage twice, or a table you cannot freely optimize.
Second: “open format” is not “open exit.” Every product on this list emits Iceberg. The lock-in concentrates in the write engine, the catalog commit path, and the maintenance automation. Bufstream is the cautionary tale: technically excellent, acquired by CoreWeave in May 2026, folded into an internal platform, and no longer sold on its own. The table survives a vendor’s exit. The pipeline that produces it does not.
What must be true before a stream becomes a table
Picking a product is the second decision. The first is whether your data is even table-shaped yet.
The products differ in architecture. The preconditions do not. A Kafka topic is a log of things that happened. An analytical table is a claim about the current state of the world. Turning one into the other requires answers to eight questions, and no vendor can answer them for you — they are properties of your data, not of the pipe.
Keys. A log has no row identity; a table with updates needs one. The Kafka message key is usually a partitioning choice, not a business key — it may be a session ID when the table needs a user ID, or absent entirely. If you cannot name the column set that makes a row unique, you do not have an upsert problem, you have a modeling problem, and every “CDC-informed table” feature will faithfully materialize it.
Event time. Every event carries at least three timestamps: when it happened, when the broker received it, and when it landed in the table. Dashboards, joins, and partition pruning behave differently depending on which one the table is partitioned and filtered by. Pick the business answer once, write it into the table spec, and stop letting each consumer guess. Most “the numbers don’t match” incidents between a streaming metric and a lake table are two systems disagreeing about which timestamp is “time.”
Late data. Streams are never in order and never complete. The table needs a policy: does an event that arrives three days late rewrite a partition the business already reported on, land in today’s partition with its true event time as a column, or get dropped? Each answer is defensible. Having no answer means closed periods silently change and downstream consumers cache contradictory results. This is also where completeness signals belong — a partition is not “done” because the clock rolled over.
Delete semantics. A tombstone in Kafka and a delete in Iceberg are different mechanisms. Merge-on-read deletes accumulate delete files that readers must reconcile until compaction rewrites them. Old snapshots still serve the deleted rows to time travel. And if the delete is a GDPR erasure, the bytes must actually leave object storage — expired snapshots, cleaned orphans — not just vanish from the current view. A topic-to-table product that “supports deletes” has answered the easy half of that sentence.
Schema contracts. The topic schema is the producer’s contract; the table schema is every downstream consumer’s contract. They evolve at different speeds and break different people. Decide where enforcement happens — produce time, conversion time, or query time — and who approves a breaking change. Automatic schema evolution is a feature until a producer renames a column and forty dashboards inherit the rename overnight. Somebody must be allowed to say no, and the pipeline must have a place to put records that fail the check.
Compaction. Streaming freshness multiplies files; analytical queries want few large ones. Whatever you pick, the table needs a standing maintenance budget — compaction, snapshot expiry, manifest rewrites — sized to the write rate, not configured once and forgotten. If the vendor automates it, verify what happens when their automation falls behind. If it is yours, it is a production service, not a cron job.
Table ownership. Every previous item is a decision, and decisions need a decider. The failure mode is structural: the streaming team owns the topic, the data team owns the table, and the conversion layer — where keys, time, lateness, deletes, and schema all get decided — is owned by whoever wrote the config. Name one owner for the table as a product, with the authority to change the topic side when the table side requires it.
Consumer SLAs. The moment a topic becomes a table, it acquires consumers who never agreed to streaming semantics: BI tools, finance reports, ML training jobs. They assume freshness, completeness, and query latency that nobody promised. Write the SLA down — how fresh, how complete, how fast to query, and what the consumer should do during a backfill — before the first dashboard ships. “Queryable in seconds” is a vendor claim about latency. It is silent about completeness, and completeness is what the CFO’s report actually needs.
If your team cannot fill in those eight answers for a topic, the topic is not ready to be a table — with any vendor, at any price. The convergence products remove the pipeline code. They do not remove the data product work the pipeline code was quietly hiding.
What we learned running our own
At Wix, we built this before the products existed: an EMR-based system that mirrors Kafka topics into data-lake tables with configurable latency. Two design choices did most of the work, and neither was about moving bytes.
The first was making latency a menu, not a default. Teams choose a freshness band per table instead of everything pretending to be real-time. Most analytical consumers are fine with minutes, and the cost difference is large.
The second was ownership: the same team owns the telemetry Kafka topics and the Iceberg tables underneath them. When ingestion and table layout share an owner, small files get fixed because the people who caused them are the people paged for slow queries. Less finger-pointing, more maintenance actually happening.
Because writing the files was the easy half. The system had to own small-file control, deduplication, sorting for query performance, and continuous optimization as data kept arriving through the day. We know what skipping that looks like: when large tables once moved into a new catalog before automated compaction was ready, one day of hourly writes was enough to make Trino struggle to plan queries. Same data, different file layout, dead table.
Every product above will write your topics into Iceberg. The table staying queryable is a separate feature. Check for it explicitly.
The questions that actually pick a solution
The eight above are about your data. These are about the products. Assuming the data is ready, these five are what separate one solution from another — and none of them appears on a vendor spec sheet in this form:
- What freshness does the table consumer actually need? If the honest answer is minutes, a boring Connect sink plus solid compaction is still a legitimate answer, and most of the list is overkill.
- Append-only or upserts? CDC into the lake eliminates more than half the options immediately. If you need row updates, you are choosing between Tableflow, AutoMQ, Fluss, or running the merge yourself.
- Can whoever runs compaction rewrite files without breaking Kafka reads? This is the single-copy versus two-copy decision in operational form. In a single-copy design the answer is “not freely,” and that constraint outlives every demo.
- Which catalog is the source of truth, and who is allowed to commit to it? A table that two systems commit to is an incident schedule. Replay and backfill are the same question in disguise: reprocessing a topic must not corrupt the table, and the vendor’s exact answer reveals how the whole system works.
- If the engine disappears, what survives? Format openness is the baseline now. Price the exit path of the writer, not the files — Bufstream emitted perfectly open Iceberg right up to the morning it disappeared into an acquisition.
The contract still needs an owner
The market is collapsing the pipe. Brokers write Parquet, warehouses speak Kafka, and the connector era is visibly ending.
What no product collapses is the table contract: keys, event time, late data, deletes, schema approval, maintenance, ownership, and the SLAs of consumers who never asked for a stream.
The middle layer was never just plumbing. It was where those decisions lived, unwritten, in some team’s job code. Deleting the job does not delete the decisions.
The hard part was never moving the bytes. It is deciding who owns the table once the bytes arrive.