Repository navigation
Queues, in standalone workerd and in a cluster - #8328
Conversation
The bundled module /workerd/queues.js is a Queues Worker: the broker between workerd's `queue` binding, which already speaks the producer protocol, and a consumer's queue() handler, reached through a service binding. Like KV and cron, it is a plain config documented in its header: the QueueShard class and the bindings SHARD and CONSUMERS. A queue is up to 1024 shards, each a SQLite-backed Durable Object, so in a cluster shards replicate and fail over as any object. A shard stores one row per message. A delivery leases its rows; an ack deletes them, a retry or an expired lease makes them visible again (a retry after its delay, without a second batch timeout), and messages out of attempts go to the dead letter queue in two phases. One alarm covers batch deadlines, lease expiry, dead letters and 4-day retention. Delivery is at least once, with production's limits, defaults and retry rules; a full shard answers 429 and its producer moves on. wrangler.json's `queues` maps onto it, with --experimental: each producer a `queue` binding to `<name>-queues`, each consumer delivering to the Worker. Tests cover formats, batching, retries, dead letters, lease expiry, metrics, limits, and a SIGKILL and restart. None sleeps: a TEST_CLOCK binding moves the broker's clock.
There was a problem hiding this comment.
Superseded by the review of
906390dd.
This review, of 7bcfafd1
💬 Approved with comments — No blocking issues or warnings. One INFO: in wrangler.rs a second consumer for the same queue silently replaces the first.
Findings: 1 info (1 is an inline comment on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📋 Code Quality: 💬 1 info
One minor issue: duplicate consumers for a queue are silently collapsed in the wrangler translation.
Recommendation: 💬 Optional Nits
- [INFO] A second consumer for the same queue silently replaces the first in the generated CONSUMERS config. (
src/workerd/server/cli/wrangler.rs:633)
View full review
A second consumer for a queue silently replaces the first (src/workerd/server/cli/wrangler.rs:633)
queues_services builds the CONSUMERS JSON with consumers.insert(c.queue.clone(), settings). If a wrangler config lists two consumers for the same queue, only the last one is kept, with no message. The module says any other field is an error rather than silently ignored, so this goes against its own convention.
The same applies to max_concurrency: any u32 is passed through. shard() in queues.js uses it directly as the shard count, so a value above 1024 makes more shards than the "up to 1024 shards" the header documents.
The rest holds up:
- The KV and cron setup moved into
bundled_workeranddisk_servicekeeps its service offsets and binding order. - The CONSUMERS key order asserted in
wrangler-test.rsis stable. - A message gets
maxRetries + 1attempts, as the tests expect. - The lease-generation fencing, the alarm re-arming and the dead-letter flow are consistent.
✅ All other checks passed
🔒 Security: ✅ LGTM
The Queues Worker is reachable only through queue bindings and its own SHARD binding. Queue names and consumer settings come from trusted config.
📖 AGENTS.md: ✅ LGTM
The src/workerd/server/AGENTS.md update is accurate.
⚡ Performance: ✅ LGTM
No performance concerns.
📝 Documentation: ✅ LGTM
The README and the module header match the behaviour.
Reviewed commit: 7bcfafd1
A wrangler config that lists two consumers for one queue is an error, rather than the second silently replacing the first. maxConcurrency picks the shard count, so it is capped at the 1024 shards a queue has.
There was a problem hiding this comment.
Superseded by the review of
e834b4d7.
This review, of 906390dd
✅ Approved — No findings. The earlier note about a second consumer replacing the first, and about maxConcurrency going over 1024, is fixed in 906390d.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
No security concerns in this revision.
📖 AGENTS.md: ✅ LGTM
The src/workerd/server/AGENTS.md update is accurate.
📋 Code Quality: ✅ LGTM
Two things are fixed now:
- A second consumer for the same queue is now an error (
wrangler.rs:635), andwrangler-test.rstests it. shard()caps the shard count atSHARDS(queues.js:129).
These still match the code:
- The bundled service offsets.
- The binding order for KV, cron and queues.
- The
CONSUMERSJSON key order in the test. - The
--experimentalgating.
⚡ Performance: ✅ LGTM
Each hot query in the broker has a matching partial index.
These stay bounded:
- batch size, capped at 100;
- the number of batches in flight;
- the
#leasesmap, which is pruned on every pump.
The constructor's count scan runs once each time a shard wakes up.
📝 Documentation: ✅ LGTM
The README and the module header match the behaviour.
Reviewed commit: 906390dd
danlapid
left a comment
There was a problem hiding this comment.
This is exactly the shape we want: a bundled JS worker, the queue name in the binding's props, the service's setup as JSON bindings, and the only native change is one entry in the findBundled table. Nice!
A few things to fix:
1. A consumer slower than ~10 s gets cancelled and redelivered
#pump() starts each delivery as ctx.waitUntil(this.#deliver(...)), and both alarm() and call() return right away. So no request is open while the consumer runs. workerd shuts a Durable Object down 10 s after its last request ends (IDLE_SHUTDOWN_DELAY in server/actor.rs), and a pending waitUntil doesn't count as a request: activity is counted through Worker::Actor::addRef(), and the server installs no WaitUntilTaskTracker. Shutting down cancels the delivery. When the shard is re-created, its constructor's #fail(clock(), 'lease IS NOT NULL') counts the batch as a failed attempt, even if the consumer finished, and delivers it again.
So any consumer slower than ~10 s gets duplicates and uses up its retries, and can land in the dead letter queue having succeeded every time. Production allows 15 minutes. #bury's dead-letter sends run the same way. I got this from reading the code rather than reproducing it, so please double-check.
cron.js already does this right: it awaits scheduled() inside alarm(). Could Queues do the same, with alarm() awaiting the deliveries it starts and call() setting an immediate alarm instead of delivering inline? A test that holds a consumer on a signal, evicts the shard, then releases the consumer and checks the batch is acked once would cover it.
2. wrangler configs with valid queue fields fail to load
Producer and Consumer use deny_unknown_fields, but wrangler's schema also allows:
- producers:
delivery_delay, andqueueis optional - consumers:
type: "worker"andvisibility_timeout_ms
delivery_delay is worth supporting rather than ignoring, since it's the producer's default delay. It could travel in the binding's props.
3. Producers get generic error messages
fetch returns new Response(r.body, { status }) with no statusText and no CF-Queues-Error-Code / CF-Queues-Error-Cause headers, which is what buildQueueErrorMessage in queue.c++ reads. So a 413 or a 429 reaches the producer as "Unknown Internal Error (15000)", or "Queue send failed: " without queue_expose_error_codes, and the useful text in the body is lost.
4. Cluster mode isn't tested yet
The title and description cover cluster behaviour (sends going to a shard on the producer's own instance, shards failing over with leased messages), but cluster-test.mjs isn't touched. A test like KV's, killing a shard's owner with messages leased and checking a survivor redelivers them, would cover it.
5. Delays aren't validated (minor)
Production accepts delays from 0 to 43200 s and rejects others. Here any value is accepted, and a non-numeric X-Msg-Delay-Secs makes visible_at NaN, so the insert fails and the producer gets a 500 rather than a 400.
Besides these, LGTM to land. Really nice work 🙂
A delivery ran in waitUntil after the call that started it returned, so nothing kept the shard in memory: evicting it mid-delivery cancelled the delivery, and the next instance counted the batch as a failed attempt and delivered it again. Production evicts such an object too, after 70 to 140 s idle, unless the durable_object_io_tasks_prevent_eviction date applies. Deliveries now run inside alarm(), which waits for them: a send wakes a running alarm or sets one for now. An alarm starts deliveries for 5 minutes, within the 15-minute alarm limit, waits for batches due within 2 s rather than end, then sets itself again. The alarm is not moved while deliveries run, so a survivor adopting a dead instance's shard runs it at once, rather than at the end of the lease 16 minutes later. The alarm is no longer set again when it is already set no later. Re-setting a due alarm on every send kept it from firing in a cluster, where an alarm fires about 450 ms after it is set. On a three-node cluster, throughput and delivery latency match the previous code, and a killed owner's leased messages were redelivered by a survivor 24 s after the kill.
There was a problem hiding this comment.
✅ Approved — Delivering inside the alarm fixes the eviction-cancellation problem, and the changed code and tests have no correctness, security or performance issues.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
/deadreturns 404 on the HTTP path, so only shard-to-shard RPC can insert dead-letter batches.- The queue name comes from the binding's props, not from the request.
- The producer paths enforce size limits.
wrangler.rswrites the generated JSON withserde_json.- The feature is gated behind
--experimental.
📖 AGENTS.md: ✅ LGTM
The src/workerd/server/AGENTS.md update is accurate.
📋 Code Quality: ✅ LGTM
Alarm-driven delivery (queues.js:312-337):
- The loop doesn't spin, because
#nextleaves out work that is at the concurrency limit or in flight. - Leases a crashed instance held are failed in the constructor.
- A settle from an expired lease is fenced by its generation, and
leaseExpirytests that path.
The test bindings in the .wd-test match what the tests use.
⚡ Performance: ✅ LGTM
Nothing new since the last review. Hot queries are indexed, batch size is capped at 100, and the number of batches in flight is bounded.
📝 Documentation: ✅ LGTM
The module header's description of alarm-driven delivery matches the code.
Reviewed commit: e834b4d7
Adds Queues to workerd as a bundled Worker,
/workerd/queues.js, alongside KV and cron. It sits between workerd's existingqueueproducer binding and a consumer Worker'squeue()handler, which it calls through a service binding (service_binding_extra_handlers).Each queue is up to 1024 shards, each a SQLite-backed Durable Object that stores one row per message, so in a cluster the shards replicate and fail over like any other object. A delivery leases its rows: an ack deletes them, a retry or an expired lease makes them visible again, and messages out of attempts go to the dead letter queue. Delivery is at least once, with production's limits, defaults and retry rules.
With
--experimental,queuesin wrangler.json maps onto it: each producer becomes aqueuebinding, and each consumer delivers to the Worker.Tests cover formats, batching, retries, dead letters, lease expiry, metrics, limits, and a SIGKILL and restart. None of them sleeps: a
TEST_CLOCKbinding moves the broker's clock. I also ran a wrangler.jsonc project by hand (batch timeout, retries into a dead letter queue, a full batch, restart with messages pending).