Context
I work on the backend for the Norwegian National Archive (Asta) at Cefalo as Senior Staff Software Engineer, across a Java/Kotlin stack backed by PostgreSQL and MongoDB. Elasticsearch powers archive search for both archivists and members of the public, and the pipeline that keeps those search indexes in sync with the source databases is something I designed personally.
Search freshness matters here in a very direct way: archivists change records, and the public and other archivists need to be able to search on what just changed, not on a stale snapshot.
Problem & constraints
Before this architecture existed, the application wrote to Elasticsearch the same way it wrote to the database: dual writes from application code, in the same code path as the database write.
That produced two recurring symptoms. First, load spikes — indexing work rode on the same request path as the database write, so search indexing load was coupled directly to application write traffic. Second, missed updates — when one of the two writes failed and the other succeeded, the index quietly drifted out of sync with the source of truth, with no atomicity to catch it. The workaround at the time was occasional full reindexes to patch the drift back into alignment, which is a symptom fix, not a structural one.
Options considered
Dual writes from application code (status quo)
Rejected
Write to the database and to Elasticsearch in the same application code path. There's no atomicity between the two writes, so a partial failure leaves the index silently stale with nothing to signal it. It also couples indexing load directly to the request path, which is what caused the load spikes.
Periodic full reindex as the primary mechanism
Rejected
Rebuild the Elasticsearch indexes from the source databases on a schedule. This is heavy and slow to run, and search is stale for the entire gap between runs. It's useful as an occasional repair tool, but it can't be the primary way indexes stay in sync.
Change data capture with Debezium and Kafka Connect
Chosen
Stream committed changes out of the databases through Debezium and Kafka Connect into Kafka, and let consumers apply those changes to Elasticsearch. Indexing is decoupled from the application write path entirely — the index follows what was actually committed to the database, not what application code hoped it wrote twice.
Decision & architecture
The core of the architecture is Debezium running on Kafka Connect, capturing changes from both PostgreSQL and MongoDB — the archive's two source databases. Rather than one big undifferentiated pipeline, the topology is organized per module and per table: connectors are set up per source database, topics are organized per table and per module, and consumers are scoped per module, each writing to its own Elasticsearch index. That topology mirrors the archive's own module and table boundaries. Search is not the only consumer, either: modules also consume the change streams to keep cross-module shared data up to date, so data owned by one module stays current wherever other modules depend on it.
Backfilling or rebuilding an index — for example after a mapping change, or to repair an index — is handled by a separate batch job that runs independently of the streaming path. It's a deliberate, on-demand tool now, not the thing keeping the index in sync day to day.
Failure recovery relies on Kafka's own consumer offset model: if a consumer crashes, it resumes from its last committed offset, and transient failures writing to Elasticsearch are retried rather than treated as terminal.
Outcome
Missed updates are eliminated as a class of problem: the index follows committed database changes instead of depending on application code successfully writing to two places at once. Indexing load is decoupled from the request path, so there are no more search-driven load spikes triggered by application writes.
Full reindexing changed role entirely — from a routine bandage patching over drift, to a deliberate, separately-run repair tool used for mapping changes or index repair, independent of the day-to-day streaming path. And because the change streams carry committed changes rather than search documents, the same pipeline keeps cross-module shared data in sync — one mechanism serving both search freshness and inter-module consistency.