Your events live in a store built for aggregation, not for page loads. The dashboard has to be instant and correct at the same time, and those pull in opposite directions. This is how to have both — and the read path that took ours from 20 seconds to 200 milliseconds while doubling what fits on a page.
Every event your system produces gets logged — sent, delivered, read, clicked, converted. That log is the truth, and it lives somewhere built for aggregating across millions of rows: a search cluster, a columnar warehouse, whatever you picked.
Then someone opens a dashboard and wants fifty rows of computed metrics, paginated, sorted, with rates and revenue attribution. Two requirements, in direct opposition:
Aggregate live on every request and the page takes as long as your heaviest query — for us, twenty seconds on a busy account, which is long enough to alt-tab and forget why you opened it. Serve a precomputed copy and the numbers for something still actively running are minutes behind reality, which for a campaign mid-send is worse than slow.
The first move is to stop computing metrics in the request path at all. Keep the event store as the source of truth, and keep a materialised copy of the computed numbers in your transactional database. Pages read the copy. Nothing else.
One decision worth making deliberately: put the numbers on the entity row, not in a separate analytics table. A campaign row carries its own sent, delivered, read, revenue and the timestamp of when those were last computed. No join, no second table to keep in step, no window where the entity exists and its metrics don't.
The alternative — a metrics table keyed by entity id — sounds cleaner and costs you a join on every list query, plus a whole class of bug where the two tables disagree about what exists. Metrics are an attribute of the campaign, not a separate concept. Store them like one.
You now have a fast read and a new problem: the copy goes stale. Everything below is about refreshing it as rarely as you can get away with.
The frontend fires the same request twice, differing in one flag. Not two endpoints — one endpoint with two modes, so there is no chance of the shapes drifting apart.
Two requests, one endpoint, one flag. The dashed block is the only part that touches the event store, and on most page loads it doesn't run at all — everything on screen is older than the freeze threshold.
Two details that matter more than they look:
A single TTL is the obvious implementation and it's wrong in both directions at once. Set it tight and you recompute year-old campaigns that cannot possibly have changed. Set it loose and a campaign that is mid-send shows numbers from ten minutes ago.
The fix: make the tolerance a function of how old the thing is. Recent data is volatile and worth recomputing. Old data is effectively immutable and recomputing it is pure waste.
| Age of the entity | Refresh if older than | Why |
|---|---|---|
| < 7 days | 5 min | still moving — reads, clicks, orders arriving |
| 7 – 14 days | 15 min | slowing down |
| 14 – 30 days | 30 min | occasional late attribution |
| > 30 days | never | frozen — nothing will change again |
Two guards worth adding while you're in there:
Here is the part that most affects your event-store bill. A list page and a detail page need very different amounts of data, and if you compute one payload for both you are computing the expensive one fifty times per page load.
So define two tiers, and make one a strict superset of the other:
| Tier A — the list | Tier B — the detail | |
|---|---|---|
| Computes | headline counts, revenue, rates | everything in A plus engagement detail, failure reasons, funnel steps, opt-outs |
| Query cost | light | heavy |
| Rows per request | up to a page | exactly one |
| Stamps written | A | A and B |
That asymmetry is why you need two freshness timestamps, not one. The list checks the A stamp; the detail page checks the B stamp. Collapse them into one and you get a detail page that believes it's fresh because a light list refresh touched the row — showing stale funnel numbers next to correct headline numbers, which is the kind of bug that erodes trust in the whole dashboard.
Call them what they are — headlineComputedAt and
detailComputedAt beats updatedAt1 and
updatedAt2. Someone will have to reason about which screen a
given stamp gates, at speed, during an incident.
Most systems like this have parent/child structure. A campaign gets retried, so the original send has three follow-up attempts, each its own row with its own numbers. The list wants one line per campaign; the detail page wants a choice.
Resist the urge to store a pre-combined total. Aggregate on read. A stored rollup is a third copy that can disagree with the two you already have, and it needs invalidating every time any child changes. Summing four rows at read time is free by comparison.
Three rules, and the second and third are where the bugs live:
If a parent's displayed total includes its children, then the parent is stale when the parent or any child is stale. Check freshness across the family, not just the row you're rendering — otherwise the combined number sits wrong until something else happens to refresh a child, and you get bug reports about totals that "fix themselves eventually".
Then expose the choice rather than deciding for the user. The list collapses to one row per family with combined numbers and expands to show each attempt separately. The detail page offers combined, original, or a specific attempt. Same aggregation function behind all of it — only the selection differs.
Everything so far removes the event store from the hot path. If a dashboard is opened often enough — a team all watching the same account, or a page that polls — the transactional read becomes the hot path instead, and that is when a cache in front earns its place.
Add it after the rest, not instead of it. A cache over a slow aggregation just serves stale slow data; a cache over a materialised read is genuinely just speed.
The crossed circle is the whole point. If the refresh call is allowed to hit the cache, it returns a cached answer, nothing is ever recomputed, and you ship a dashboard that is fast and permanently behind.
A time-series chart across the same data is worth caching per segment rather than per request — each bucket of the range under its own key, so changing the window reuses most of the work instead of recomputing the whole series.
The design above is what we ended with. Here is what it replaced and what went wrong on the way.
The original read path aggregated from the event store on every load, for every row on the page. On a large account that was 20 seconds — long enough that people opened it, switched tabs, and came back having forgotten why. Moving to a materialised copy with the two-call pattern took it to 200ms, and we doubled the page size at the same time because per-row cost had stopped mattering.
The first aggregation across a retry family added the rates. Read rate came out over 100% for any campaign with two retries, which at least fails loudly. Click-through came out plausible-but-wrong, which doesn't. Rates are now always recomputed from summed counts, and never stored pre-combined.
Reach was summed across the family. Since a retry targets people who didn't engage the first time, the overlap is nearly total — so a campaign with three retries reported roughly three times the audience it actually touched. The fix was a de-duplicated count computed once across the family, stored on the parent, and never derived by addition again.
Freshness was checked on the row being rendered. A parent whose displayed cost included its children would show a stale total whenever a child was stale — then silently correct minutes later when something else refreshed that child. Support tickets described it as numbers that "settle down after a while", which is a horrible thing for a billing figure to do. Staleness is now evaluated across the whole family.
The pattern across all four: the slow version was obvious and the wrong version was quiet. A 20-second page gets fixed because everyone complains. A reach number inflated 3× by a summed cardinality gets quoted in a board deck.
A dashboard is a cache with opinions. The only question is how many layers of it you're willing to admit to.