Repository navigation
Cut the cluster test's waits - #8299
Conversation
There was a problem hiding this comment.
Superseded by the review of
2d352a2a.
This review, of a0d02f6f
💬 Approved with comments — A clean, test-only timing change. The only finding is an INFO: unsafeTimeDivisor has no upper bound, so a very large value can round some times down to zero.
Findings: 1 info (1 is an inline comment on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📋 Code Quality: 💬 1 info
The divisor is passed through consistently; one minor robustness note.
Recommendation: 💬 Optional Nits
- [INFO]
unsafeTimeDivisorhas no upper bound, so a large value rounds the lease, retry or accept times down to zero. (src/workerd/server/server/cluster.rs:319)
View full review
unsafeTimeDivisorhas no upper bound (src/workerd/server/server/cluster.rs:319). The divisor is only clamped to at least 1, and every division is integer division. A divisor of about 1000 roundsretry_msand the skew bound down to 0 or 1 ms. A divisor of about 5000 makesaccept_ms0, and one of about 20000 makes the lease itself 0, which would fence every node on its first round. The field is marked unsafe and documented as test-only, so the risk is low. Rejecting values that make any divided time zero when the config loads would turn a mistake into a startup error. The.max(1)here and thedivisor == 0check inTiming::dividedalso repeat each other.
✅ All other checks passed
🔒 Security: ✅ LGTM
No security concerns.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md updates needed.
⚡ Performance: ✅ LGTM
No performance concerns.
📝 Documentation: ✅ LGTM
No documentation issues.
Reviewed commit: a0d02f6f
There was a problem hiding this comment.
Superseded by the review of
8658b35e.
This review, of 2d352a2a
💬 Approved with comments — A test-only timing change with no defects in what changed since the last review. One INFO from that review is still open: unsafeTimeDivisor has no upper bound.
Findings: 1 info
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📋 Code Quality: 💬 1 info
No new defects. The changes since the last review are the in-process fake S3, nodes starting at the same time, and tighter test bounds. One earlier INFO about the divisor's missing upper bound is still open.
Recommendation: 💬 Optional Nits
- [INFO]
unsafeTimeDivisorhas no upper bound, so a large value rounds the divided lease, retry, skew and accept times down to zero. (src/workerd/server/server/cluster.rs:319)
View full review
No new defects since the last review. Here is what was checked:
- Every place that builds a
Setup, aBudgetor aLeaseuses the newTimingandaccept_ms. - Each lease record is matched to its node by
clusterPort. FakeS3Server.reset()bumps a generation counter. A request held up by a fault across a reset therefore cannot write into the next test's bucket.- Nothing still refers to the removed
fake-s3.jsorfake-s3.wd-test.
The earlier note about the divisor having no upper bound still applies.
✅ All other checks passed
🔒 Security: ✅ LGTM
No security concerns. unsafeTimeDivisor comes from the operator's config and is clamped to at least 1. The fake S3 is test-only and listens only on 127.0.0.1.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md updates needed.
⚡ Performance: ✅ LGTM
At the default divisor of 1, production timings are unchanged. The other changes affect only how long the tests take.
📝 Documentation: ✅ LGTM
No documentation issues.
Reviewed commit: 2d352a2a
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8299 +/- ##
==========================================
+ Coverage 65.19% 65.26% +0.07%
==========================================
Files 758 758
Lines 190916 190990 +74
Branches 30114 30163 +49
==========================================
+ Hits 124471 124657 +186
+ Misses 48916 48800 -116
- Partials 17529 17533 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
piscisaureus
left a comment
There was a problem hiding this comment.
"Speeding up the clock" to test timing-dependent behavior faster makes sense to me.
My main concern would be is that the drain timing margin is thin. At divisor=4, the accept time leaves about 0.2s over a handoff under TSan (as the PR itself notes). It could easily become its own source of flakes in the future.
cluster-test spent most of its time waiting: each takeover waited out a dead owner's 20 s lease, and each drain the 5 s a stopping node accepts requests for. Its slowest shard took 120-146 s. `ClusterConfig.unsafeTimeDivisor`, for tests only, divides those times: the lease, the rounds that renew it and the bounds on them, the rounds that adopt dead nodes' alarms, and the time a stopping node accepts requests for at least. `stopTimeoutMs` is not divided. * The lease's times are now `lease::Timing`, whose `divided` keeps their proportions. A divisor of 0, or one that rounds a time down to 0, fails the config. The lease's tests run at several divisors, and check that a round renews before the lease needs to fence. * `handoff::Budget` gains `accept_ms`, `GRACE_MS` by default. * cluster-test.wd-test divides by 4: a 5 s lease. The test's waits that are measured against the lease or the accept time derive from the same divisor. cluster-cron.wd-test reuses its cluster. * Each test starts a cluster of its own, 2-3 s under TSan, so the test runs in 6 shards rather than 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit left the test in 6 shards, most of it still waiting, and with a flake: under TSan the rolling restart test lost a request in 4 of 20 runs. The fake S3 ran in a workerd of its own, one per test, and served every key through one Durable Object. Under TSan, while a test loaded the cluster, it answered a listing of the nodes in 90 ms at the median and in up to 600 ms. The nodes' listings outran their 500 ms bound, so a peer still routed to a node whose time to accept requests had ended. * fake-s3.mjs serves the same API from the test's own process. One serves every test, emptied for each; a request that a fault delayed past the reset answers 503, and lists, counts and changes nothing. * A test's nodes start at once rather than one by one. Each lease record is matched to its node by the port it advertises. * The default config's stop timeout is now 10 s, and `shortStop`'s 4 s. The holds measured against them or against the lease are shorter, and the bounds the tests put on drains are tightened to match. * The test runs in 5 shards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2d352a2 to
8658b35
Compare
There was a problem hiding this comment.
✅ Approved — A test-only timing change. No issues found, and the earlier note about the divisor having no upper bound is fixed at this commit.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
The production changes depend on a config setting whose default keeps the old timings. The fake S3 is test-only and listens only on 127.0.0.1.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md updates needed.
📋 Code Quality: ✅ LGTM
Timing::divided(lease.rs) refuses a divisor of 0, and any divisor that rounds a time down to 0.ClusterNode::newfails the config with that, before theGRACE_MSandALARMS_INTERVALdivisions run.Lease::before_renew/renewed,Cluster::maintainandannounce_drainingall read the lease's ownTiming, so the retry, listing and renew bounds are divided together.- Lease records are matched by
:<port>suffix. The leading colon stops one port matching another that ends in the same digits. fake-s3.mjschecks the reset generation after a fault's delay and after reading a PUT body.
⚡ Performance: ✅ LGTM
Production timings are unchanged at the default divisor. The new code runs once per node start or once per lease round.
📝 Documentation: ✅ LGTM
No documentation issues.
Reviewed commit: 8658b35e
|
Thanks. On the drain margin: agreed, it is the thinnest part, so here is what sets it. A stopping node accepts requests until The part that varied was the listing. The workerd-hosted fake S3 took up to 600 ms to answer one under TSan, and that is what made the rolling restart test lose requests. The in-process fake answers in milliseconds. Since then the test has passed every run: rebased on main, 10 of 10 under TSan and 5 of 5 each regular and ASan, on top of the earlier rounds. If it does flake, the knob is to divide the accept time by less than the lease. Dividing it by 2 rather than 4 gives about 1.45 s of margin, at about 1.25 s more per drain, or roughly 4 s more per shard. I left it at 4 while nothing fails on it. |
cluster-testspent most of its time waiting, and its slowest shard took 120-146 s.Lease and accept times. Each takeover waited out a dead owner's 20 s lease, and each drain waited out the 5 s a stopping node accepts requests for.
ClusterConfig.unsafeTimeDivisor, for tests only, divides those times: the lease, the rounds that renew it and the bounds on them, the rounds that adopt dead nodes' alarms, and the time a stopping node accepts requests for at least.stopTimeoutMsis not divided.lease::Timing, whosedividedkeeps their proportions. A divisor of 0, or one that rounds a time down to 0, fails the config. The lease's tests run at divisors 1, 2, 4 and 10, and check that a round renews before the lease needs to fence.handoff::Budgetgainsaccept_ms,GRACE_MSby default.cluster-test.wd-testdivides by 4, for a 5 s lease. The test's waits that are measured against the lease or the accept time derive from the same divisor (TIME_DIVISORincluster-test.mjs).cluster-cron.wd-testreuses its cluster, socluster-cron-testruns at the same divisor.A larger divisor doesn't fit. A stopping node's accept time has to cover its handoff (about 0.3 s under TSan) and one round of its peers' listings. At 4 that leaves about 0.2 s; at 10 it is negative.
The fake S3. It ran in a workerd of its own, one per test, and served every key through one Durable Object. Under TSan, while a test loaded the cluster, it answered a listing of the nodes in 90 ms at the median and in up to 600 ms. With the shortened lease, the nodes' listings outran their 500 ms bound, so a peer still routed to a node whose accept time had ended: the rolling restart test lost a request in 4 of 20 TSan runs.
fake-s3.mjsnow serves the same API from the test's own process, one for every test, emptied for each; a request that a fault delayed past the reset answers 503, and lists, counts and changes nothing. With it, that test passed 20 of 20 TSan runs, and no listing timed out.The rest.
shortStop's 4 s. The holds measured against them or against the lease are shorter, and the bounds the tests put on drains are tightened to match. No assertion is loosened.cluster-testruns in 5 shards rather than 3.Timings
bazel test --nocache_test_resultson remote execution, rebased on main: 5 runs each regular and ASan, 10 under TSan. Every run passed,cluster-cron-test(8-9 s) included.Per shard, max / average, in seconds:
The sum over every case, which sharding does not change: regular 263 s → 102 s, TSan 328 s → 129 s. The divisor alone, with the old fake S3 and stop timeouts, got to 158 s and 227 s.
Per case, average seconds
workerd deploypublishes a wrangler projectworkerd deploypublishes a Cap'n Proto config, and refuses one that conflictsworkerd deployrefuses a deployment that holds more than servicesWhat is left is mostly one workerd start per test (about 1 s under TSan), the three takeovers' 5 s lease, and the drains' 1.25 s accept time. One drain waits longer than it should: in "a stopping node's DOs finish calls to themselves and across nodes", the objects that served calls back to themselves are handed off only when the requests in flight are cancelled, at half the stop timeout, though every call returned seconds before. That looks like a handoff issue rather than a test one, so it is not changed here.
The cluster crate's tests (
cluster_test,cluster-logic_test) pass, and so does clippy.🤖 Generated with Claude Code