What 38 nodes were hiding
A Pub/Sub delivery backlog the team had been absorbing with hardware, and what actually fixed it.
Draft — the writeup below is a placeholder. The numbers in the summary box are the real ones; the prose is not written yet.
The symptom
Placeholder. Describe what the backlog looked like from the outside — the delivery delay, the alerting, and the fact that the team had been scaling out to absorb it.
What the 38 nodes were actually doing
Placeholder. Explain why adding consumers appeared to help and why it could not converge.
Ack deadlines
Placeholder. Inline code renders like
ackDeadlineSeconds and
maxOutstandingMessages.
// Placeholder — replace with the real consumer configuration.
const subscription = pubsub.subscription(name, {
flowControl: {
maxOutstandingMessages: 1000,
maxOutstandingBytes: 100 * 1024 * 1024,
allowExcessMessages: false,
},
ackDeadline: 60,
});
The fix
Placeholder. Walk through the three changes:
- Consumer flow control
- Ack-deadline handling
- Backpressure
Result
Placeholder. Close with what changed and what was retired.
Placeholder for a pull-quote or a caveat worth calling out.
Outcome
- 2.4 h → 94 s
- 236K → ~100
- 38 → 0