You split work across queues so the slow stuff can't block the fast stuff. Then you point one consumer service at all of them with identical settings and quietly undo the entire point. Here's how to size each one, and which knob to actually turn.
Say you have four kinds of background work. Forwarding webhooks to customer endpoints. Updating user properties across a search index. Sending campaign messages. Reindexing search documents. Somebody sensibly gives each one its own queue, because a slow reindex shouldn't stall a webhook.
Then one service consumes all four, and because it's the same code path, they all get the same worker count, the same batch size, the same pacing. Congratulations — you've built one queue wearing four hats. The isolation is real at the broker and imaginary in the consumer.
A queue's consumer config should be a function of what processing one message costs. Not of what the other queues are doing, and definitely not of a shared default. Two queues with genuinely different per-message cost that share a config means one of them is wrong — usually both.
A webhook forwarder does one outbound HTTP call and acks. It's almost pure I/O wait, the work is trivially parallel, and the sink is somebody else's server. That queue wants big batches and a fast cadence.
A property updater that writes to a search cluster, invalidates a cache and touches a relational row does three round trips and some CPU, into infrastructure you own and can overwhelm. That queue wants to go slowly on purpose.
Same broker, same service, same library. Configurations that should differ by more than an order of magnitude.
Every pull-based consumer I've built reduces to the same identity. Write it on the wall before you tune anything:
throughput = workers × batch_size ÷ cycle_timeWhere a worker is one independent pull→process→ack loop with its own connection, batch size is how many messages a single pull asks for, and cycle time is how long one loop takes — your deliberate pacing delay plus however long processing actually took.
Five workers pulling 50 messages with a one-second cycle is 250/s. So is one worker pulling 250 with the same cycle. So is ten workers pulling 25. The equation does not care which factor you change.
Your infrastructure bill cares enormously. That's the part nobody tells you, and it's the rest of this article.
The three factors are interchangeable for throughput and wildly different in what they cost you. In rough order of cheapness:
One pull that returns 250 messages costs about the same as one that returns 50. Same round trip, same connection, marginally more bytes. If your work is cheap per message, raising batch size is the highest-leverage change available and it is almost always the one people reach for last.
Halving the delay doubles how many pull requests you issue. The catch is the
rate of those requests is workers ÷ delay, so the cost of
going faster is multiplied by however many workers you have. Cheap with two
workers. Expensive with thirty.
Each worker is a connection, and a connection is not free while it's idle. Keepalives, TLS state, load-balancer bookkeeping, a slot in the client's internal machinery — all charged per worker, all the time, whether messages are arriving or not.
Hold throughput constant and rearrange. Total pull rate is
workers ÷ delay, so the variable cost — the
requests, the decoding, the actual processing — is identical either way.
The only thing that changes is fixed per-worker overhead.
Fewer workers pulling faster beats more workers pulling slower, at the same throughput. Every time.
That result implies a tuning order, and getting it backwards is how you make
things worse while "optimising". Lowering the delay on its own
raises CPU. You've increased workers ÷ delay
without reducing the worker count, so you're paying for more pulls and the
same overhead.
The saving only materialises when you lower the delay and cut workers together. Do the first half alone and you'll watch the service scale out and conclude, wrongly, that it needed more capacity.
One reason, and it isn't throughput: when processing blocks and you need overlap. A single worker is a serial loop — while it's waiting on an HTTP call it isn't pulling. A second worker can be pulling during the first one's dead time.
Which gives a rule you can apply without a profiler:
Before you set a single number, sort every queue into one of three shapes. What matters is the cost of handling one message and who absorbs it.
| Shape | What one message does | Batch | Workers | Cycle |
|---|---|---|---|---|
| Forwarder | one outbound call, ack | large — 250+ | 2 | fast — ~1s |
| Writer | 1–2 writes you own | medium — ~50 | 2 | moderate |
| Fan-out | several systems, CPU | small — 10–50 | 1–2 | slow, deliberately |
Forwarders are the ones people under-configure. Processing is a single HTTP call — mostly waiting, no CPU worth mentioning, and the thing you might overwhelm belongs to someone else and has its own limits and its own retry policy. Pull 250 at a time and keep the cadence tight. A forwarder throttled to 50 messages a second is just a queue that grows.
Fan-outs are the ones people over-configure. Each message means a search write, a cache invalidation, a row update. Ten workers pulling 200 apiece will find the write ceiling of your own cluster in about four seconds, and then everything you're doing turns into timeouts and redeliveries. Go slowly here on purpose. It is not underperformance, it is the queue doing its job.
Work out what each queue's real busy-hour rate looks like and size for that with headroom. Not for the theoretical maximum your current config could sustain — that number is an artefact of settings you're about to change, and treating it as a requirement is how you end up permanently over-provisioned.
One piece of background before the tuning makes sense, because every capacity mistake eventually shows up here.
When a message is handed to you, the broker starts a clock. Ack before it expires and the message is done. Miss it and the broker assumes you died and gives the message to somebody else — while you are still working on it. It is a lease on exclusive ownership, not a request timeout.
This is why holding more messages than you can process is actively harmful rather than merely wasteful. The clock is running on every message you're holding, not just the one you're working on. Pull 250 messages into a worker that handles two per second and the tail of that batch is guaranteed to be redelivered before you reach it.
Which gives the real constraint on batch size:
batch_size × time_per_message < ack_deadlineA forwarder at 20ms per message can safely hold hundreds. A fan-out at 400ms per message cannot hold more than a couple of dozen. Same broker, same deadline — and the answer differs by 10×. That inequality is the actual reason the config table above looks the way it does.
If you genuinely need longer, extend the lease while you work rather than setting a huge global deadline. A long deadline is the wrong fix: it also means a message from a worker that really did crash sits undelivered for that entire window.
A fixed delay is wrong in both directions. Tight enough for a backlog, and you're hammering an empty queue all night. Relaxed enough to be polite when idle, and you're crawling when it matters.
So make it adaptive, with two rules:
The floor is the part to be careful with, because it's easy to write a ramp-down that silently does nothing — clamping against the current value instead of a constant, so the delay can only ever go up. Everything looks correct in code review and in tests, because the ramp-up works fine.
The failure mode is nasty: after one quiet spell the consumer pins at its maximum delay and stays there through the next backlog. The queue is full, the consumer is idling on purpose, and there's no error anywhere. A quick sanity check — log the delay per subscription and confirm it comes back down under load — costs nothing and catches it.
There is a point past which pull capacity buys you nothing, and it isn't in your consumer. It's in whatever the consumer writes to.
If a queue's handler writes to a search cluster, that cluster's sustainable write rate is the ceiling on the whole pipeline. Configure the consumer to pull three times that and you have not tripled throughput — you've tripled concurrency against a saturated sink. Latency rises, handler time rises with it, and handler time is exactly what the ack deadline is racing.
Now the thing worth internalising, because it is the opposite of the intuition
that got you here. Past saturation the numerator in our equation stops being
workers × batch and becomes the sink's fixed capacity. Extra
workers cannot raise it. What they can do is push messages past their
deadline, so the broker redelivers them, and those redeliveries are served out
of the same fixed capacity:
useful_throughput = C ÷ (1 + redelivery_rate) — C fixed, r risingThat falls as you add workers. Not diminishing returns — actual regression. Every worker past saturation costs you first-pass throughput, and add enough and useful throughput drops below the arrival rate. The backlog is now growing on more hardware than it was draining on.
This is why "the queue is backed up, add consumers" is a coin flip rather than a fix. If you are genuinely under-provisioned it works. If you are at a downstream ceiling it makes things worse, and the dashboard everyone watches — messages processed per minute — goes up either way, because it counts the repeats.
Acked messages per worker per minute, plotted against worker count. Flat means you're genuinely short of capacity and scaling out will work. Sloping down means your workers are interfering through something they share — stop adding them and go find it.
Queue depth is the obvious signal and the wrong one. Redeliveries inflate depth, so a consumer that's failing to ack in time produces exactly the signal that provokes the autoscaler into adding more consumers — which produces more redeliveries. The feedback loop runs in the wrong direction and it does so confidently.
Better signals, in order of preference:
And cap the ceiling deliberately. An unbounded maximum on a consumer that hits a downstream limit is a bill with no upper bound and no matching increase in work completed.
One service, seven subscriptions plus a handful of staging ones, all configured the same because they'd been added one at a time and nobody had revisited the shape.
Five workers per subscription, nine subscriptions, one pod: 36 independent connections on a container sized at 616 millicores. The per-connection overhead alone was starving the networking layer, and almost none of those workers were busy — several of those queues handled a few dozen messages an hour.
The adaptive delay clamped against its own current value rather than a constant, so the ramp-down was a no-op. The ramp-up worked perfectly. Any subscription that saw one quiet spell climbed to the 10-second ceiling and stayed there — including through backlogs. Two subscriptions had been fixed at some point; the other seven hadn't, which is why they were the ones falling behind.
Correcting the floor took those subscriptions from a 10-second cycle to one second — a 10× increase in pull rate across 36 connections, with no reduction in connections. CPU went up and the deployment scaled out. The fix was right and incomplete: it was the first half of a change that only pays off when you also cut workers.
So we sized each subscription to its actual shape instead of a shared default. The webhook forwarder and the property updater — both cheap-per-message, both already on a real one-second floor — went to large batches and two workers each. The busy message paths kept two workers for pull/process overlap. The genuinely quiet ones went to one.
| Subscription shape | Workers before | Workers after |
|---|---|---|
| busy message paths (×3) | 5 each | 2 each |
| forwarder + property writer | 4 each | 2 each |
| low-volume (×2) | 4–5 each | 1 each |
| staging (×4) | 1 each | 1 each |
| total connections | 36 | ~16 |
The measured outcome on the two worst subscriptions: combined backlog from 1.26M messages to under 4K, average delivery delay from 2.4 hours to 94 seconds, peaks from 8 hours. No capacity was added. The capacity had been there the whole time — it was being spent on redelivering work we'd already done, from connections that were mostly asleep.
Not universal values — there are none. A starting point that is wrong in defensible directions, to be replaced by measurement.
| Setting | Forwarder | Writer | Fan-out |
|---|---|---|---|
| Workers | 2 | 2 | 1–2 |
| Batch size | 250 | 50 | 10–25 |
| Idle delay ceiling | 5–10s | 5–10s | 5–10s |
| Busy delay floor | ~1s | ~1s | 1–2s |
| In-flight cap | ≈ batch | ≈ batch | < batch |
| Handler concurrency | bounded | bounded | bounded, low |
| Autoscale on | oldest unacked age | oldest unacked age | downstream saturation |
batch × time_per_message < ack_deadline.Doing these out of order is what produces the classic outcome: a fleet that costs four times as much and moves twice the work, with everyone convinced the queue is the bottleneck.
If every queue in your system has the same consumer config, you don't have multiple queues. You have one queue and four names for it.