Skip to content

Queues, in standalone workerd and in a cluster - #8328

Merged
ry merged 3 commits into
mainfrom
ry/cluster-queues
Nov 11, 2026
Merged

ry merged 3 commits into
mainfrom
ry/cluster-queues

Conversation

@ry

@ry ry commented Nov 10, 2026

Copy link
Copy Markdown
Member

Adds Queues to workerd as a bundled Worker, /workerd/queues.js, alongside KV and cron. It sits between workerd's existing queue producer binding and a consumer Worker's queue() handler, which it calls through a service binding (service_binding_extra_handlers).

Each queue is up to 1024 shards, each a SQLite-backed Durable Object that stores one row per message, so in a cluster the shards replicate and fail over like any other object. A delivery leases its rows: an ack deletes them, a retry or an expired lease makes them visible again, and messages out of attempts go to the dead letter queue. Delivery is at least once, with production's limits, defaults and retry rules.

With --experimental, queues in wrangler.json maps onto it: each producer becomes a queue binding, and each consumer delivers to the Worker.

Tests cover formats, batching, retries, dead letters, lease expiry, metrics, limits, and a SIGKILL and restart. None of them sleeps: a TEST_CLOCK binding moves the broker's clock. I also ran a wrangler.jsonc project by hand (batch timeout, retries into a dead letter queue, a full batch, restart with messages pending).

The bundled module /workerd/queues.js is a Queues Worker: the broker
between workerd's `queue` binding, which already speaks the producer
protocol, and a consumer's queue() handler, reached through a service
binding. Like KV and cron, it is a plain config documented in its
header: the QueueShard class and the bindings SHARD and CONSUMERS.

A queue is up to 1024 shards, each a SQLite-backed Durable Object, so
in a cluster shards replicate and fail over as any object. A shard
stores one row per message. A delivery leases its rows; an ack
deletes them, a retry or an expired lease makes them visible again
(a retry after its delay, without a second batch timeout), and
messages out of attempts go to the dead letter queue in two phases. One alarm covers batch deadlines, lease expiry, dead letters
and 4-day retention. Delivery is at least once, with production's
limits, defaults and retry rules; a full shard answers 429 and its
producer moves on.

wrangler.json's `queues` maps onto it, with --experimental: each
producer a `queue` binding to `<name>-queues`, each consumer
delivering to the Worker.

Tests cover formats, batching, retries, dead letters, lease expiry,
metrics, limits, and a SIGKILL and restart. None sleeps: a TEST_CLOCK
binding moves the broker's clock.
@ry
ry requested review from a team as code owners November 10, 2026 04:49

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 906390dd.

This review, of 7bcfafd1

💬 Approved with comments — No blocking issues or warnings. One INFO: in wrangler.rs a second consumer for the same queue silently replaces the first.

Findings: 1 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

One minor issue: duplicate consumers for a queue are silently collapsed in the wrangler translation.

Recommendation: 💬 Optional Nits

  • [INFO] A second consumer for the same queue silently replaces the first in the generated CONSUMERS config. (src/workerd/server/cli/wrangler.rs:633)
View full review

A second consumer for a queue silently replaces the first (src/workerd/server/cli/wrangler.rs:633)

queues_services builds the CONSUMERS JSON with consumers.insert(c.queue.clone(), settings). If a wrangler config lists two consumers for the same queue, only the last one is kept, with no message. The module says any other field is an error rather than silently ignored, so this goes against its own convention.

The same applies to max_concurrency: any u32 is passed through. shard() in queues.js uses it directly as the shard count, so a value above 1024 makes more shards than the "up to 1024 shards" the header documents.

The rest holds up:

  • The KV and cron setup moved into bundled_worker and disk_service keeps its service offsets and binding order.
  • The CONSUMERS key order asserted in wrangler-test.rs is stable.
  • A message gets maxRetries + 1 attempts, as the tests expect.
  • The lease-generation fencing, the alarm re-arming and the dead-letter flow are consistent.

✅ All other checks passed

🔒 Security: ✅ LGTM

The Queues Worker is reachable only through queue bindings and its own SHARD binding. Queue names and consumer settings come from trusted config.


📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md update is accurate.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

The README and the module header match the behaviour.


Reviewed commit: 7bcfafd1

Comment thread src/workerd/server/cli/wrangler.rs Resolved
A wrangler config that lists two consumers for one queue is an error,
rather than the second silently replacing the first. maxConcurrency
picks the shard count, so it is capped at the 1024 shards a queue has.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of e834b4d7.

This review, of 906390dd

✅ Approved — No findings. The earlier note about a second consumer replacing the first, and about maxConcurrency going over 1024, is fixed in 906390d.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

No security concerns in this revision.


📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md update is accurate.


📋 Code Quality: ✅ LGTM

Two things are fixed now:

  • A second consumer for the same queue is now an error (wrangler.rs:635), and wrangler-test.rs tests it.
  • shard() caps the shard count at SHARDS (queues.js:129).

These still match the code:

  • The bundled service offsets.
  • The binding order for KV, cron and queues.
  • The CONSUMERS JSON key order in the test.
  • The --experimental gating.

⚡ Performance: ✅ LGTM

Each hot query in the broker has a matching partial index.

These stay bounded:

  • batch size, capped at 100;
  • the number of batches in flight;
  • the #leases map, which is pruned on every pump.

The constructor's count scan runs once each time a shard wakes up.


📝 Documentation: ✅ LGTM

The README and the module header match the behaviour.


Reviewed commit: 906390dd

@danlapid danlapid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly the shape we want: a bundled JS worker, the queue name in the binding's props, the service's setup as JSON bindings, and the only native change is one entry in the findBundled table. Nice!

A few things to fix:

1. A consumer slower than ~10 s gets cancelled and redelivered

#pump() starts each delivery as ctx.waitUntil(this.#deliver(...)), and both alarm() and call() return right away. So no request is open while the consumer runs. workerd shuts a Durable Object down 10 s after its last request ends (IDLE_SHUTDOWN_DELAY in server/actor.rs), and a pending waitUntil doesn't count as a request: activity is counted through Worker::Actor::addRef(), and the server installs no WaitUntilTaskTracker. Shutting down cancels the delivery. When the shard is re-created, its constructor's #fail(clock(), 'lease IS NOT NULL') counts the batch as a failed attempt, even if the consumer finished, and delivers it again.

So any consumer slower than ~10 s gets duplicates and uses up its retries, and can land in the dead letter queue having succeeded every time. Production allows 15 minutes. #bury's dead-letter sends run the same way. I got this from reading the code rather than reproducing it, so please double-check.

cron.js already does this right: it awaits scheduled() inside alarm(). Could Queues do the same, with alarm() awaiting the deliveries it starts and call() setting an immediate alarm instead of delivering inline? A test that holds a consumer on a signal, evicts the shard, then releases the consumer and checks the batch is acked once would cover it.

2. wrangler configs with valid queue fields fail to load

Producer and Consumer use deny_unknown_fields, but wrangler's schema also allows:

  • producers: delivery_delay, and queue is optional
  • consumers: type: "worker" and visibility_timeout_ms

delivery_delay is worth supporting rather than ignoring, since it's the producer's default delay. It could travel in the binding's props.

3. Producers get generic error messages

fetch returns new Response(r.body, { status }) with no statusText and no CF-Queues-Error-Code / CF-Queues-Error-Cause headers, which is what buildQueueErrorMessage in queue.c++ reads. So a 413 or a 429 reaches the producer as "Unknown Internal Error (15000)", or "Queue send failed: " without queue_expose_error_codes, and the useful text in the body is lost.

4. Cluster mode isn't tested yet

The title and description cover cluster behaviour (sends going to a shard on the producer's own instance, shards failing over with leased messages), but cluster-test.mjs isn't touched. A test like KV's, killing a shard's owner with messages leased and checking a survivor redelivers them, would cover it.

5. Delays aren't validated (minor)

Production accepts delays from 0 to 43200 s and rejects others. Here any value is accepted, and a non-numeric X-Msg-Delay-Secs makes visible_at NaN, so the insert fails and the producer gets a 500 rather than a 400.

Besides these, LGTM to land. Really nice work 🙂

A delivery ran in waitUntil after the call that started it returned, so
nothing kept the shard in memory: evicting it mid-delivery cancelled the
delivery, and the next instance counted the batch as a failed attempt
and delivered it again. Production evicts such an object too, after 70
to 140 s idle, unless the durable_object_io_tasks_prevent_eviction date
applies.

Deliveries now run inside alarm(), which waits for them: a send wakes a
running alarm or sets one for now. An alarm starts deliveries for 5
minutes, within the 15-minute alarm limit, waits for batches due within
2 s rather than end, then sets itself again. The alarm is not moved
while deliveries run, so a survivor adopting a dead instance's shard
runs it at once, rather than at the end of the lease 16 minutes later.

The alarm is no longer set again when it is already set no later.
Re-setting a due alarm on every send kept it from firing in a cluster,
where an alarm fires about 450 ms after it is set.

On a three-node cluster, throughput and delivery latency match the
previous code, and a killed owner's leased messages were redelivered by
a survivor 24 s after the kill.

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — Delivering inside the alarm fixes the eviction-cancellation problem, and the changed code and tests have no correctness, security or performance issues.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

  • /dead returns 404 on the HTTP path, so only shard-to-shard RPC can insert dead-letter batches.
  • The queue name comes from the binding's props, not from the request.
  • The producer paths enforce size limits.
  • wrangler.rs writes the generated JSON with serde_json.
  • The feature is gated behind --experimental.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md update is accurate.


📋 Code Quality: ✅ LGTM

Alarm-driven delivery (queues.js:312-337):

  • The loop doesn't spin, because #next leaves out work that is at the concurrency limit or in flight.
  • Leases a crashed instance held are failed in the constructor.
  • A settle from an expired lease is fenced by its generation, and leaseExpiry tests that path.

The test bindings in the .wd-test match what the tests use.


⚡ Performance: ✅ LGTM

Nothing new since the last review. Hot queries are indexed, batch size is capped at 100, and the number of batches in flight is bounded.


📝 Documentation: ✅ LGTM

The module header's description of alarm-driven delivery matches the code.


Reviewed commit: e834b4d7

@ry
ry merged commit 48c9ab1 into main Nov 11, 2026
47 of 51 checks passed
@ry
ry deleted the ry/cluster-queues branch November 11, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants