Skipping the Broker: Streaming Database Changes Straight into Iceberg with Flink CDC

Ask most data platform teams how to stream changes out of a relational database, and you'll hear one answer: run Debezium on Kafka Connect, publish every change to Kafka, and let downstream jobs read the topics. It's the default for good reason, and we ran it ourselves.

But when we set out to bring every operational database behind our products into an Apache Iceberg lakehouse, we landed on two less common choices:

1. Flink as the capture layer. With Flink CDC, Flink reads the database binlog itself, so capture, filtering, and writing happen in one job, and dropping changes nobody needs becomes a one-line rule.

2. Iceberg as the Kafka substrate. Each source table streams into an append-only Iceberg table that downstream jobs read incrementally, the way a consumer reads a topic. The lake is both the log and the destination.

Why the default strained

For multiple security & operational features that Druva offers to its customers, we had a requirement to ingest multiple databases sitting inside dozens plus of RDS & Auroras, some of which were operating at a scale of ~1000 TPS.

Some of these operational databases rewrite the same rows constantly. Picture a backup_jobs table: while a job is RUNNING, the backup agent updates bytes_transferred and progress_pct every few seconds, yet for analytics only two moments matter: when the job was queued and how it ended. Debezium faithfully emits every update. Kafka stores every one. Every consumer reads them all and throws most away.

This is not a rounding error. Across millions of protected devices, backup-job tables alone produce about a billion change events a day, most of them progress updates. Keeping only the moments that matter cuts that to roughly one in ten.

At our scale, that showed up in three ways: Kafka capacity grew with churn rather than value; filtering happened only after the noise was stored; and our scheduled Spark consumers had to run so often to keep up that one run could still be running when the next was due.

The shape of the pipeline

Flink owns getting changes in; Spark owns turning them into tables people query. An append-only changelog table is how both connect.

Idea 1: Flink as the capture layer

In most CDC stacks, Flink shows up late, one of several consumers reading Kafka topics. Flink CDC moves it to the front. Its MySQL source embeds Debezium as the capture engine, so you keep Debezium's mature binlog parsing, but events flow straight into a Flink job instead of a topic (Flink CDC sources). The connector takes a consistent initial snapshot, switches to the binlog without a gap, and resumes from checkpoints with exactly-once semantics. One job replaces Kafka Connect, the brokers, and the first consumer.

Filtering is where it pays off. Debezium makes table and column filters easy, and can skip whole operation types. But filtering on a row's values means either writing your own SMT in Java or using Debezium's Filter SMT, which evaluates a Groovy or JavaScript expression on each message. The Filter SMT isn't shipped with the connectors: you add a scripting engine to your Connect workers, or on a managed service like MSK Connect, package it into your custom plugin. Debezium also warns that once it's installed, anyone who can create a connector can run scripts. Filtering downstream instead avoids that, but by then the noise is already in Kafka.

In Flink CDC, the same rule is SQL-like configuration in the pipeline definition, applied before anything is written (illustrative):

transform:
- source-table: ops.backup_jobs
    projection: "*"
    filter: "__data_event_type__ = 'INSERT' 
OR (__data_event_type__ <> 'UPDATE_BEFORE' 
AND end_time IS NOT NULL)"

The job's creation and its final state land; every progress update in between never leaves the Flink task. Transforms support computed columns, built-in functions and user-defined functions (transform docs), and they live in the same YAML as the routing that maps each source table to its changelog table.

If you want Debezium without a broker, Debezium Server with a community Iceberg consumer is another route; row-level filtering still goes through SMTs, whereas Flink CDC lets us express it as SQL inside the same pipeline.

Idea 2: Iceberg as the Kafka substrate

What does a CDC consumer actually need from Kafka? An ordered, append-only sequence of changes, a position it can save and resume from, and a way to replay. An append-onlyIceberg table provides all three. Every commit creates an immutable snapshot, readers see a commit entirely or not at all, and Iceberg can scan only the rows added between two snapshots.

Our Spark jobs work exactly like consumers. Each run reads the snapshots committed since its stored watermark, keeps the newest change per key (by event time, then file position), MERGEs it into a latest-state table such as backup_jobs_latest, and only then advances the watermark. A failed run simply re-reads the same range.

The read itself is a few lines of PySpark (illustrative). Iceberg's incremental scan returns only rows appended after start-snapshot-id, up to and including end-snapshot-id — valid for append-only tables, which a changelog always is:

# The watermark is a snapshot ID of the SOURCE changelog table,
# tracked separately for each consumer (job/query) that reads it.
def read_new_changes(spark, source_table, last_snapshot_id):
    snaps = spark.sql(f"SELECT snapshot_id FROM {source_table}.snapshots "
                      f"ORDER BY committed_at DESC LIMIT 1").collect()
    if not snaps:
        return None, last_snapshot_id              # nothing written yet
    latest = snaps[0]["snapshot_id"]
    if latest == last_snapshot_id:
        return None, latest                        # no new commits

    reader = spark.read.format("iceberg")
    if last_snapshot_id is None:                   # first run: read everything so far
        reader = reader.option("snapshot-id", latest)
    else:                                          # only (last, latest]
        reader = (reader.option("start-snapshot-id", last_snapshot_id)
                        .option("end-snapshot-id", latest))
    return reader.load(source_table), latest

watermark = load_watermark("backup_jobs_latest_job", "lake.ops.backup_jobs_changelog")
changes, latest = read_new_changes(spark, "lake.ops.backup_jobs_changelog", watermark)
# ... dedupe + MERGE into backup_jobs_latest, then save `latest` as this consumer's watermark

The watermark always points into the source changelog, and each consumer keeps its own, just as each Kafka consumer group keeps its own offsets. Save it only after the MERGE commits; that ordering is what makes a failed run safe to retry.

The same pattern refreshes materialized views and derived tables incrementally. A daily failure count per customer, for example, reads only the new slice of the changelog and folds it into its existing totals instead of recomputing from full history. Every derived table becomes just another consumer with its own watermark on the same log.

Replay, Kafka's strongest argument, carries over. Reset a consumer's watermark to an older snapshot, and it reprocesses everything since, just like rewinding an offset. Because every row carries its event timestamp, you can also replay by time with SQL: "all changes to this customer's backup jobs since Tuesday" is a WHERE clause.

And the history is longer. Kafka's default retention is seven days (log.retention.hours=168,broker configs); keeping more means more broker disk or tiered storage. Iceberg history lives in object storage, so weeks or months is a setting, not a capacity plan; we keep 30 days of snapshots. Expiring snapshots doesn't delete change rows from an append-only table, so time-range replay reaches back as far as you keep the rows, without ever touching the source database.

We store the tables in Amazon S3 Tables, which runs compaction and snapshot cleanup for us, important when a streaming writer commits every checkpoint.

The catch is latency: the Flink sink commits on each checkpoint, so changes appear in minutes, not milliseconds, which is fine for use-cases we serve.

Why keep a changelog at all?

Flink and Iceberg can upsert directly into a latest-state table. We chose not to, mainly because we wanted the changes themselves:

• Event analytics. "How long do backups wait in the queue before they start, and how often do they fail, per customer, this week?" is a question about changes over time, easy on a changelog, impossible on a table that keeps only the current row.

• Soft deletes. A delete in the source can become a "deleted at" marker in the lake, so investigations still see records that no longer exist upstream.

• Many views, one stream. Latest-state tables, aggregates and cross-system views all derive from the same changelog, and any of them can be rebuilt from it.

• A thin streaming layer. Streaming upserts need per-key state for every row in every table, which slows checkpoints and recovery. Keeping Flink close to stateless and doing MERGEs in bounded Spark batches avoided that.

When to use which

This design shines for point-to-point sync: changes flow from a source database to one destination, and everything downstream reads from there. A broker in between mostly adds cost and moving parts.

Kafka is still the better choice when:

• several consumers, especially independent services outside the lake, need the same change stream;

• Latency matters, consumers must react in seconds or less;

• The events are a product in their own right, consumed outside the lake. Flink CDC also has a Kafka sink, so the two aren't mutually exclusive: tables that need a live stream can still be published to Kafka, while the rest go straight to Iceberg.

Without a broker, you also own a few things Kafka used to absorb: running stateful Flink jobs, and keeping enough binlog retention on the source to recover from a long outage. We run both patterns today and choose per source.

The takeaway

Kafka earned its place as the default CDC backbone. But if your changes are headed for a lakehouse anyway, ask whether you need a second copy of them in a broker. With Flink capturing and filtering at the source, and Iceberg holding an ordered, replayable change log, we got the guarantees we relied on Kafka for, kept a longer history, and stopped storing changes nobody reads.