Context
The public Arkivportalen portal at the Norwegian National Archive serves 1M+ users, and its Elasticsearch-backed search needs to reflect which archive items are published on an external digital-media system. That system is external in a specific sense: content reaches it from many different sources, and the archive side does not publish to it — the portal only needs to know, for each item, what its publish status there is.
I built the middleware that answers that question, and it's in production today. The service was started in Python, and I continued in it rather than rewrite it in something else — the language choice was inherited, and it hasn't been a reason to change course. Development was AI-assisted, using Claude.
Problem & constraints
Before this middleware existed, the portal called the media system's single-item status endpoint directly, and stored no status anywhere on the portal side. Every time the portal needed to know whether an item was published, that was a live call out to an external API.
Two constraints made that untenable. First, the media system's only status endpoint is a single-item GET — one item per request, with no batch alternative. Second, the portal operates at 1M+ user scale, and a rate-limited external API sitting in that request path simply doesn't hold at that scale. On top of both: nothing about publish status may be silently missed — the portal's search has to converge on the truth, not on whatever the last successful call happened to catch.
Options considered
Status quo: portal calls the media system directly, no stored status
Rejected
The media system's API limits can't absorb portal-driven traffic at 1M+ user scale, and because nothing was stored, every status need repeated the same rate-limited single-item call. It also meant every part of the portal that cared about publish status carried a live external dependency.
Standalone middleware with its own store
Chosen
A separate Python service that owns the reconciliation problem: it pulls portal data gradually into its own PostgreSQL store, checks publish status against the media system on its own schedule, stores the result, and lets the portal consume it — so the portal never calls the media system at all.
Decision & architecture
The middleware is a Python service backed by its own PostgreSQL database. It gradually pulls portal data into that store, and a scheduler polls the media system's single-item status GET endpoint item by item, updating each record's publish status as responses come back.
On the portal side, a separate scheduler — inside the portal, not the middleware — pulls publish status from the middleware and updates the portal's Elasticsearch index. The portal's only dependency in this whole flow is its own scheduler reading from the middleware; it has no path to the media system at all.
Both directions are pull-only: the middleware pulls from the portal and from the media system; the portal pulls from the middleware. Nothing pushes into the portal, and the portal never calls the media system directly. Resilience is built into both sides — failures and exceptions are handled, calls are retried, and progress is checkpointed via a last-success-sync and last-success-id recorded per item, since the media system's API only exposes status one item at a time. If either scheduler restarts or fails partway through, it resumes from the last successfully processed item rather than starting over or skipping ahead.
Outcome
The portal's Elasticsearch index converges on correct publish status without the portal ever touching the rate-limited external API — that call path is gone entirely, replaced by the portal reading from its own middleware on its own schedule.
The property that nothing gets silently missed doesn't come from hoping the single-item polling never fails. It comes from per-item checkpointing — last-success-sync and last-success-id — combined with retries on both sides, so a restart or a failed call resumes exactly where it left off instead of skipping an item or starting over. The middleware has been running in production since.