Skip to content

Cut the cluster test's waits - #8299

Merged
danlapid merged 2 commits into
mainfrom
dlapid/cluster-test-lease
Nov 10, 2026
Merged

danlapid merged 2 commits into
mainfrom
dlapid/cluster-test-lease

Conversation

@danlapid

@danlapid danlapid commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

cluster-test spent most of its time waiting, and its slowest shard took 120-146 s.

Lease and accept times. Each takeover waited out a dead owner's 20 s lease, and each drain waited out the 5 s a stopping node accepts requests for. ClusterConfig.unsafeTimeDivisor, for tests only, divides those times: the lease, the rounds that renew it and the bounds on them, the rounds that adopt dead nodes' alarms, and the time a stopping node accepts requests for at least. stopTimeoutMs is not divided.

  • The lease's times are now lease::Timing, whose divided keeps their proportions. A divisor of 0, or one that rounds a time down to 0, fails the config. The lease's tests run at divisors 1, 2, 4 and 10, and check that a round renews before the lease needs to fence.
  • handoff::Budget gains accept_ms, GRACE_MS by default.
  • cluster-test.wd-test divides by 4, for a 5 s lease. The test's waits that are measured against the lease or the accept time derive from the same divisor (TIME_DIVISOR in cluster-test.mjs). cluster-cron.wd-test reuses its cluster, so cluster-cron-test runs at the same divisor.

A larger divisor doesn't fit. A stopping node's accept time has to cover its handoff (about 0.3 s under TSan) and one round of its peers' listings. At 4 that leaves about 0.2 s; at 10 it is negative.

The fake S3. It ran in a workerd of its own, one per test, and served every key through one Durable Object. Under TSan, while a test loaded the cluster, it answered a listing of the nodes in 90 ms at the median and in up to 600 ms. With the shortened lease, the nodes' listings outran their 500 ms bound, so a peer still routed to a node whose accept time had ended: the rolling restart test lost a request in 4 of 20 TSan runs. fake-s3.mjs now serves the same API from the test's own process, one for every test, emptied for each; a request that a fault delayed past the reset answers 503, and lists, counts and changes nothing. With it, that test passed 20 of 20 TSan runs, and no listing timed out.

The rest.

  • A test's nodes start at once rather than one by one. Each lease record is matched to its node by the port it advertises.
  • The default config's stop timeout is now 10 s, and shortStop's 4 s. The holds measured against them or against the lease are shorter, and the bounds the tests put on drains are tightened to match. No assertion is loosened.
  • cluster-test runs in 5 shards rather than 3.

Timings

bazel test --nocache_test_results on remote execution, rebased on main: 5 runs each regular and ASan, 10 under TSan. Every run passed, cluster-cron-test (8-9 s) included.

Per shard, max / average, in seconds:

config before (3 shards) after (5 shards)
regular 66/62, 84/82, 122/120 22/21, 25/24, 24/22, 27/27, 27/26
ASan 64/63, 86/83, 133/128 21/21, 23/23, 23/22, 27/26, 26/25
TSan 89/82, 114/104, 146/142 28/27, 31/29, 31/28, 34/32, 33/30

The sum over every case, which sharding does not change: regular 263 s → 102 s, TSan 328 s → 129 s. The divisor alone, with the old fake S3 and stop timeouts, got to 158 s and 227 s.

Per case, average seconds
case regular before regular after TSan before TSan after
a killed owner's alarm runs once on a survivor 26.2 6.7 27.5 7.2
a rolling restart under load loses and replays no request 20.4 8.2 25.4 9.6
killing the owner allows another node to take over 22.0 6.5 23.0 7.5
a KV namespace survives its owner 21.0 5.7 21.9 6.1
a peer takes over a draining owner's DO before its lease ends 13.1 3.9 14.4 4.4
a stopping node's DOs finish calls to themselves and across nodes 12.4 6.7 14.2 7.3
a DO whose node drains while restoring it does not start there 11.7 3.5 13.6 3.9
a DO whose handoff outlasts its deadline moves once its stopping node left 10.9 3.9 12.2 4.4
a node stopping under load hands its DOs to its peers 8.9 5.2 11.2 5.6
requests to a DO of a stopping node wait for its handoff and follow it 9.2 3.0 10.5 3.7
two nodes stopping at once under load hand their DOs to the third 7.6 2.6 9.9 4.0
a request that reached a draining owner is not replayed 6.9 2.5 8.2 3.2
a held stub follows its DO once the owner drains 6.5 4.5 7.8 5.0
a stopping node hands off a DO it evicted before 6.6 2.2 7.7 2.6
other DOs' calls to a DO of a stopping node wait for its handoff 6.6 5.0 7.7 5.7
a draining node's own JS RPC call that waits for a handoff follows the DO 6.5 2.1 7.6 2.5
a JS RPC call that waits for a handoff follows the DO 6.4 2.1 7.5 2.8
draining node still routes in-flight requests to a remote owner 6.5 2.1 7.5 2.7
a draining node sends a request for an unowned DO to a peer 6.4 2.3 7.5 2.7
a held stub follows its DO once the drained owner exited 6.2 1.9 7.5 2.5
draining cancels a running onHibernate handler 5.7 1.7 6.5 2.1
an alarm runs once on its object's owner 4.1 3.6 5.2 4.0
DO state is consistent across instances (single writer) 2.2 0.9 4.2 1.4
a node caches KV blobs on its disk 1.5 0.7 4.2 2.3
a KV namespace is an object, seen from every node 1.4 0.7 3.0 1.4
a request a forwarding peer could not route is retried 1.4 0.7 2.9 1.1
concurrent first requests to a DO read its owner record once per node 1.3 0.7 2.7 1.4
a node serves beside deployments it cannot serve 1.8 1.4 2.6 1.7
aborting a DO with a request in flight lets it be re-instantiated immediately 1.4 0.9 2.6 1.3
a freshly minted RpcTarget stub passed via props reaches its DO after it breaks 1.3 0.5 2.6 1.1
requests to non-owner are transparently forwarded 1.8 0.7 2.6 1.0
a stub resolved to a remote DO keeps failing after that DO breaks 1.3 0.5 2.5 1.0
a DO whose constructor fails releases its ownership lock 1.3 0.6 2.5 1.0
storing and loading a stub claims nothing 1.2 0.4 2.5 0.9
KV reads are stale for at most cacheTtl 1.2 0.6 2.4 1.0
a broken DO pinned by a stub does not keep its ownership lock 1.1 0.5 2.4 0.9
workerd deploy publishes a wrangler project 1.0 0.4 2.4 0.8
a restored RpcTarget stub is redeemed on the root DO owner 1.1 0.6 2.4 0.9
a fresh cluster serves the deployment in its bucket 1.1 0.4 2.3 0.8
an instance bound to a wildcard advertises --cluster-advertise 1.0 0.5 2.2 1.0
each running instance holds a lease record 1.1 0.4 2.2 0.8
workerd deploy publishes a Cap'n Proto config, and refuses one that conflicts 1.1 0.7 2.2 1.2
a KV read racing a write keeps the write cached 0.8 0.6 2.1 1.3
Durable Objects from before cluster mode are refused 1.0 0.8 2.0 1.5
a DO reopened where it was closed resumes its epoch 1.3 1.0 2.0 1.3
a CONNECT is refused when replayed, stale, misdirected or unsigned 0.6 0.3 1.4 0.8
workerd deploy refuses a deployment that holds more than services 0.7 0.5 1.3 0.8
the command line's overrides are for the node's own services 0.6 0.3 1.0 0.5
a public cluster address is refused 0.2 0.2 0.2 0.2
total 263 102 328 129

What is left is mostly one workerd start per test (about 1 s under TSan), the three takeovers' 5 s lease, and the drains' 1.25 s accept time. One drain waits longer than it should: in "a stopping node's DOs finish calls to themselves and across nodes", the objects that served calls back to themselves are handed off only when the requests in flight are cancelled, at half the stop timeout, though every call returned seconds before. That looks like a handoff issue rather than a test one, so it is not changed here.

The cluster crate's tests (cluster_test, cluster-logic_test) pass, and so does clippy.

🤖 Generated with Claude Code

@danlapid
danlapid requested review from a team as code owners November 9, 2026 05:42

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 2d352a2a.

This review, of a0d02f6f

💬 Approved with comments — A clean, test-only timing change. The only finding is an INFO: unsafeTimeDivisor has no upper bound, so a very large value can round some times down to zero.

Findings: 1 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

The divisor is passed through consistently; one minor robustness note.

Recommendation: 💬 Optional Nits

  • [INFO] unsafeTimeDivisor has no upper bound, so a large value rounds the lease, retry or accept times down to zero. (src/workerd/server/server/cluster.rs:319)
View full review
  • unsafeTimeDivisor has no upper bound (src/workerd/server/server/cluster.rs:319). The divisor is only clamped to at least 1, and every division is integer division. A divisor of about 1000 rounds retry_ms and the skew bound down to 0 or 1 ms. A divisor of about 5000 makes accept_ms 0, and one of about 20000 makes the lease itself 0, which would fence every node on its first round. The field is marked unsafe and documented as test-only, so the risk is low. Rejecting values that make any divided time zero when the config loads would turn a mistake into a startup error. The .max(1) here and the divisor == 0 check in Timing::divided also repeat each other.

✅ All other checks passed

🔒 Security: ✅ LGTM

No security concerns.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md updates needed.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

No documentation issues.


Reviewed commit: a0d02f6f

Comment thread src/workerd/server/server/cluster.rs Resolved
@jasnell
jasnell requested review from piscisaureus and ry November 9, 2026 15:14
@danlapid danlapid changed the title Let the cluster test divide the times it waits out Cut the cluster test's waits Nov 9, 2026

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 8658b35e.

This review, of 2d352a2a

💬 Approved with comments — A test-only timing change with no defects in what changed since the last review. One INFO from that review is still open: unsafeTimeDivisor has no upper bound.

Findings: 1 info

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

No new defects. The changes since the last review are the in-process fake S3, nodes starting at the same time, and tighter test bounds. One earlier INFO about the divisor's missing upper bound is still open.

Recommendation: 💬 Optional Nits

  • [INFO] unsafeTimeDivisor has no upper bound, so a large value rounds the divided lease, retry, skew and accept times down to zero. (src/workerd/server/server/cluster.rs:319)
View full review

No new defects since the last review. Here is what was checked:

  • Every place that builds a Setup, a Budget or a Lease uses the new Timing and accept_ms.
  • Each lease record is matched to its node by clusterPort.
  • FakeS3Server.reset() bumps a generation counter. A request held up by a fault across a reset therefore cannot write into the next test's bucket.
  • Nothing still refers to the removed fake-s3.js or fake-s3.wd-test.

The earlier note about the divisor having no upper bound still applies.


✅ All other checks passed

🔒 Security: ✅ LGTM

No security concerns. unsafeTimeDivisor comes from the operator's config and is clamped to at least 1. The fake S3 is test-only and listens only on 127.0.0.1.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md updates needed.


⚡ Performance: ✅ LGTM

At the default divisor of 1, production timings are unchanged. The other changes affect only how long the tests take.


📝 Documentation: ✅ LGTM

No documentation issues.


Reviewed commit: 2d352a2a

@codecov-commenter

codecov-commenter commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.09302% with 72 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.26%. Comparing base (a40bbf9) to head (8658b35).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
src/workerd/server/cluster/logic/lease.rs 0.00% 37 Missing ⚠️
src/workerd/server/tests/cluster/fake-s3.mjs 93.46% 8 Missing and 5 partials ⚠️
src/workerd/server/cluster/cluster.rs 0.00% 11 Missing ⚠️
src/workerd/server/server/cluster.rs 0.00% 8 Missing ⚠️
src/workerd/server/cluster/logic/handoff.rs 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8299      +/-   ##
==========================================
+ Coverage   65.19%   65.26%   +0.07%     
==========================================
  Files         758      758              
  Lines      190916   190990      +74     
  Branches    30114    30163      +49     
==========================================
+ Hits       124471   124657     +186     
+ Misses      48916    48800     -116     
- Partials    17529    17533       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@piscisaureus piscisaureus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Speeding up the clock" to test timing-dependent behavior faster makes sense to me.

My main concern would be is that the drain timing margin is thin. At divisor=4, the accept time leaves about 0.2s over a handoff under TSan (as the PR itself notes). It could easily become its own source of flakes in the future.

danlapid and others added 2 commits November 9, 2026 21:35
cluster-test spent most of its time waiting: each takeover waited
out a dead owner's 20 s lease, and each drain the 5 s a stopping
node accepts requests for. Its slowest shard took 120-146 s.

`ClusterConfig.unsafeTimeDivisor`, for tests only, divides those
times: the lease, the rounds that renew it and the bounds on them,
the rounds that adopt dead nodes' alarms, and the time a stopping
node accepts requests for at least. `stopTimeoutMs` is not divided.

* The lease's times are now `lease::Timing`, whose `divided` keeps
  their proportions. A divisor of 0, or one that rounds a time down
  to 0, fails the config. The lease's tests run at several divisors,
  and check that a round renews before the lease needs to fence.
* `handoff::Budget` gains `accept_ms`, `GRACE_MS` by default.
* cluster-test.wd-test divides by 4: a 5 s lease. The test's waits
  that are measured against the lease or the accept time derive
  from the same divisor. cluster-cron.wd-test reuses its cluster.
* Each test starts a cluster of its own, 2-3 s under TSan, so the
  test runs in 6 shards rather than 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit left the test in 6 shards, most of it still
waiting, and with a flake: under TSan the rolling restart test lost
a request in 4 of 20 runs.

The fake S3 ran in a workerd of its own, one per test, and served
every key through one Durable Object. Under TSan, while a test loaded
the cluster, it answered a listing of the nodes in 90 ms at the
median and in up to 600 ms. The nodes' listings outran their 500 ms
bound, so a peer still routed to a node whose time to accept
requests had ended.

* fake-s3.mjs serves the same API from the test's own process. One
  serves every test, emptied for each; a request that a fault delayed
  past the reset answers 503, and lists, counts and changes nothing.
* A test's nodes start at once rather than one by one. Each lease
  record is matched to its node by the port it advertises.
* The default config's stop timeout is now 10 s, and `shortStop`'s
  4 s. The holds measured against them or against the lease are
  shorter, and the bounds the tests put on drains are tightened to
  match.
* The test runs in 5 shards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danlapid
danlapid force-pushed the dlapid/cluster-test-lease branch from 2d352a2 to 8658b35 Compare November 10, 2026 02:43

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — A test-only timing change. No issues found, and the earlier note about the divisor having no upper bound is fixed at this commit.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

The production changes depend on a config setting whose default keeps the old timings. The fake S3 is test-only and listens only on 127.0.0.1.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md updates needed.


📋 Code Quality: ✅ LGTM

  • Timing::divided (lease.rs) refuses a divisor of 0, and any divisor that rounds a time down to 0. ClusterNode::new fails the config with that, before the GRACE_MS and ALARMS_INTERVAL divisions run.
  • Lease::before_renew/renewed, Cluster::maintain and announce_draining all read the lease's own Timing, so the retry, listing and renew bounds are divided together.
  • Lease records are matched by :<port> suffix. The leading colon stops one port matching another that ends in the same digits.
  • fake-s3.mjs checks the reset generation after a fault's delay and after reading a PUT body.

⚡ Performance: ✅ LGTM

Production timings are unchanged at the default divisor. The new code runs once per node start or once per lease round.


📝 Documentation: ✅ LGTM

No documentation issues.


Reviewed commit: 8658b35e

@danlapid

Copy link
Copy Markdown
Collaborator Author

Thanks. On the drain margin: agreed, it is the thinnest part, so here is what sets it. A stopping node accepts requests until accept_ms after its drain starts. Peers route to it until one listing round after it deregisters: the 750 ms interval here, plus the listing itself. The deregistration follows the handoff. At a divisor of 4 that leaves 1250 ms − ~300 ms (the handoff under TSan) − 750 ms − the listing, so about 0.2 s.

The part that varied was the listing. The workerd-hosted fake S3 took up to 600 ms to answer one under TSan, and that is what made the rolling restart test lose requests. The in-process fake answers in milliseconds. Since then the test has passed every run: rebased on main, 10 of 10 under TSan and 5 of 5 each regular and ASan, on top of the earlier rounds.

If it does flake, the knob is to divide the accept time by less than the lease. Dividing it by 2 rather than 4 gives about 1.45 s of margin, at about 1.25 s more per drain, or roughly 4 s more per shard. I left it at 4 while nothing fails on it.

@danlapid
danlapid merged commit af679fd into main Nov 10, 2026
36 of 39 checks passed
@danlapid
danlapid deleted the dlapid/cluster-test-lease branch November 10, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants