Repository navigation
Fence a cluster node whose event loop makes no progress - #8637
piscisaureus wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Superseded by the review of
a15ea2b9.
This review, of 0d0fa671
💬 Approved with comments — No blocking or warning-level issues; one suggestion (log when the watchdog cannot be armed) and two informational notes on test timing and documentation wording.
Findings: 2 info · 1 suggestion (3 are inline comments on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📋 Code Quality: 💬 1 info · 1 suggestion
Sound design; the watchdog fails to arm silently, and one test has tight timing slack.
Recommendation: 💬 Optional Nits
- [INFO] 2 ms timing slack in the keeper fence-timing assertion may be flaky under load (
src/workerd/server/cluster/keeper-test.rs:364) - [SUGGESTION] If the watchdog cannot be armed,
serving()runs the node unwatched and logs nothing (src/workerd/server/server/cluster.rs:497)
View full review
src/workerd/server/server/cluster.rs:497: ifkj_rs_tokio::watch_progress()returnsNone, the node runs without a watchdog and logs nothing. That happens with no port orLocalSet, or whenstart_progress_probe()fails, and that error is dropped by.ok()?.watchedis still set totrue, so it never retries.src/workerd/server/cluster/keeper-test.rs:364(and:396): the assertion allows only 2 ms between the watchdog'ssince_msand the test'slast_msstamp. A descheduled thread on a loaded CI machine could fail it.
📝 Documentation: 💬 1 info
The new AGENTS.md notes are worded more broadly than the documented gaps.
Recommendation: 💬 Optional Nits
- [INFO] AGENTS.md note says progress is counted "busy or asleep" without the documented busy-loop gaps (
src/rust/cxx/AGENTS.md:41)
View full review
src/rust/cxx/AGENTS.md:40-41: the note says the counter tracks progress "busy or asleep", butprogress.rsdocuments busy stretches it does not count.
✅ All other checks passed
🔒 Security: ✅ LGTM
The watchdog adds no new untrusted input, authentication change or secret handling. Fencing goes through the existing fail-closed path, and the /wedge endpoint exists only in test config.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md issues found.
⚡ Performance: ✅ LGTM
No performance issues found.
Reviewed commit: 0d0fa671
0d0fa67 to
a15ea2b
Compare
There was a problem hiding this comment.
Superseded by the review of
c4c4571d.
This review, of a15ea2b9
✅ Approved — No issues found. The suggestion and two informational notes from the previous review are all fixed at this commit.
Not opened by the correctness review: 1 changed file it was meant to read
src/rust/cxx/kj-rs-tokio/lib.rs
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
The watchdog adds no untrusted input, no authentication change and no secret handling. Fencing goes through the existing fail-closed Armed::fence path. The /wedge endpoint exists only in the test config.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md issues found.
📋 Code Quality: ✅ LGTM
Both earlier notes are fixed:
serving()now logs atracing::warn!with theWatchErrorreason (NoPortorProbe(String)) and keeps serving unwatched.- The keeper fence-timing assertions now allow a quarter of
stall_ms.
Watchdog::observe starts timing when it first sees a count value and fences only after that count has stood still for stall_ms. The keeper checks it after the lease step, so a whole-process pause is still reported as a lost lease.
⚡ Performance: ✅ LGTM
The probe adds one KJ turn per pass through a busy run queue, and only in cluster mode. The lease thread reads the counter once per round, which costs very little.
📝 Documentation: ✅ LGTM
Both AGENTS.md notes now say the count misses the busy stretches that progress.rs lists.
Reviewed commit: a15ea2b9
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## piscisaureus/cluster-lease-thread #8637 +/- ##
====================================================================
Coverage ? 66.98%
====================================================================
Files ? 825
Lines ? 212554
Branches ? 32467
====================================================================
Hits ? 142386
Misses ? 51139
Partials ? 19029 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ce76f07 to
1728f15
Compare
a15ea2b to
c4c4571
Compare
There was a problem hiding this comment.
Superseded by the review of
a216de7a.
This review, of c4c4571d
✅ Approved — No findings. The code is unchanged since the previous approval, and kj-rs-tokio/lib.rs, which that review did not open, only adds re-exports and a module declaration.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
No security defects found. Fencing uses the existing fail-closed Armed::fence path, and the /wedge endpoint exists only in the test config.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md issues found.
📋 Code Quality: ✅ LGTM
src/rust/cxx/kj-rs-tokio/lib.rs only adds the mod progress; declaration, a doc line and re-exports of watch_progress, Progress and WatchError. current_local_set in port.rs is now pub, but its module is private and the function is not re-exported, so the crate's public API stays the same.
The progress probe code checks out:
- Destruction order:
progressProbeis declared afterloopandrustPort, so the probe is destroyed before both. - Probe start: starting the probe more than once is safe. It is created once and only re-armed after that.
- Idle beat task: it stops once the last
Progresshandle is dropped.
⚡ Performance: ✅ LGTM
No performance issues found. The probe adds one KJ turn per pass through a busy run queue, and only in cluster mode.
📝 Documentation: ✅ LGTM
No documentation issues found.
Reviewed commit: c4c4571d
A cluster node keeps its lease on a thread of its own, so a turn of
its event loop of any length keeps it. A loop that is stuck for good
(a deadlock, native code that does not return, a turn that never
ends) therefore kept the lease, and its Durable Objects unreachable,
for as long as the process lived.
Once the node serves, the lease thread now watches the loop. A loop
that makes no progress fences the node through the same path and
fatal line as a lease that cannot be kept:
* `kj_rs_tokio::watch_progress()` gives a count that the loop advances
while it makes progress, readable from any thread. A busy KJ loop
polls its port only once its run queue is empty, so between its
turns no tokio task runs, no KJ timer fires (`TimerImpl` fires only
after the port's `wait()` or `poll()`) and no `kj::Executor` ping is
taken in (drained only after the port wakes). The count's busy half
is therefore a KJ event, the port's progress probe: it counts and
re-arms itself breadth-first, so it fires once per pass through the
run queue, and leaves the queue once nothing but itself is left, so
that the loop still sleeps and runs its `evalLast()` events. The
port re-arms it after every `wait()`, and after a `poll()` that
leaves events to turn. While the loop sleeps, a task on its
`LocalSet` counts once a beat.
* The count stands still while one event holds the thread, and also
through three kinds of busy stretch that begin while the probe is
out of the queue:
* one that an `evalLast()` / `yieldUntilQueueEmpty()` event starts:
`armLast()` queues each in front of the ones before, so the probe
cannot follow them;
* one that a `yieldUntilWouldSleep()` waiter starts: KJ arms the
waiter after the port's `poll()` returns, and only if that poll
left no event;
* one that a cross-thread event starts when KJ takes it in after
the port's `poll()` rather than its `wait()`. Arming the probe for
the wake would let a stale wake, whose event KJ cancelled without
consuming it, keep `WaitScope::poll()` from returning and starve
would-sleep waiters.
Such a stretch is counted again once the run queue empties. Only
work that is ready without the port keeps it from emptying, and one
that lasts a whole `stall_ms` fences the node. Closing all three
needs a per-turn hook in KJ's `EventLoop::turn()`.
* `cluster_logic::watchdog` decides: a count that stands still for
`Timing::stall_ms` fences the node. The keeper reads it once a
round, after the round's lease step, so that a process stopped as a
whole reports its lapsed lease. Its reason ends the rounds as a
lapsed lease does, through `Armed::fence`, while the lease holds.
* The watching starts with `ClusterNode::serving`, which sends the
count to the lease thread (`Cluster::watch`). Linking, after `join`,
runs in turns as long as itself, and the lease thread keeps the
lease through it as before. A loop that cannot be watched
(`watch_progress()` answers why) serves on unwatched, with a
warning that says so.
* `stall_ms` is three leases, 60 s, divided by `unsafeTimeDivisor` as
the other times are. That is twice the turn of one and a half
leases that the long-turn test rides out. The beat is the renewal
interval.
* The fatal line now reads "leaving the cluster: {reason}", where a
lease's reasons begin "its lease cannot be kept: " and the
watchdog's reads "its event loop made no progress for N ms".
The probe costs a KJ turn per pass through a busy run queue and one
per wake of the loop, in cluster mode only. A chain of events that
arm each other depth-first runs ahead of it, as it runs ahead of
every other event, and counts as one event until it ends.
The cluster test's Worker gains a `/wedge` endpoint that loops
forever, and a test that its node exits with status 1 and the reason
on stderr, under a `shortLease` config with a tenth of the default
times. The long-turn test now aims at one and a half leases, at most
four times its last guess, rather than doubling past a lease.
1728f15 to
088fe06
Compare
c4c4571 to
a216de7
Compare
There was a problem hiding this comment.
✅ Approved — No findings. The changes since the last approval to the keeper, serving() and the test config check out, and the earlier points raised in discussion are fixed.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
🔒 Security: ✅ LGTM
No security issues found. Fencing uses the existing fail-closed path, and the /wedge endpoint exists only in the test config.
📖 AGENTS.md: ✅ LGTM
No AGENTS.md issues found.
📋 Code Quality: ✅ LGTM
No issues found.
keeper.rs:Watchgoes through the same command channel asMaintain. The watchdog checks progress after the lease step, and its reason goes throughArmed::fence.server/cluster.rs:serving()starts the watchdog only once, guarded bywatched.replace(true). Ifwatch_progressfails, it logs a warning and serves on unwatched.cluster-test.wd-test:shortLeasehasunsafeTimeDivisor = 10, which matchesSHORT_LEASE_STALL_MS.
⚡ Performance: ✅ LGTM
No performance issues found. The keeper reads one atomic counter per lease round, and in cluster mode the probe adds one KJ turn per pass through a busy run queue.
📝 Documentation: ✅ LGTM
No documentation issues found.
Reviewed commit: a216de7a
This PR is stacked on #8631, which moves lease renewal off the
server event loop. That keeps healthy busy nodes alive, but also keeps
a permanently stuck node alive forever. This PR adds a watchdog so
peers can eventually take over that node's Durable Objects.
How it works
A normal heartbeat is not enough. KJ only polls timers and its event
port after its run queue becomes empty. A healthy loop processing many
short turns, such as a fast upload, might never become empty. A timer,
tokio interval, or executor ping would not run, so the watchdog would
mistake useful work for a wedged loop.
Instead, this PR counts progress in two ways:
TokioEventPort::ProgressProbeevent movesto the back of the KJ queue after each run through the queue. Each
trip increments a counter.
sleep and lets
evalLast()events run normally. The event portrestores the probe after every wait, and after a poll that leaves
events to run.
LocalSetincrements the samecounter once per lease-renewal interval.
kj_rs_tokio::watch_progress()returns a thread-safe handle to thatcounter.
has changed. If it has not changed for three leases, normally 60
seconds, the node fences itself with exit status 1. Like the other
cluster times, this threshold is divided by
unsafeTimeDivisor.SIGSTOPor a VM pause, is still reported as a lost lease.ClusterNode::serving()is called.Linking workers after joining is one long synchronous turn and must
not look like a stall.
reason and serves on unwatched.
Armed::fencepath as a lost lease, so ithappens only once and while the lease still holds. The supervisor
can restart the node, and peers can take over its objects after the
lease expires.
leaving the cluster: {reason}. Watchdogfailures say that the event loop made no progress for the measured
number of milliseconds.
A depth-first chain counts as one event until the chain finishes.
Known gaps and follow-up
There are three cases where a busy loop can make progress without
moving the counter. They happen when a busy stretch starts while the
probe is outside the queue, and something other than the port's wait
starts it:
evalLast()oryieldUntilQueueEmpty()event, which KJ queueswhere the probe cannot follow;
yieldUntilWouldSleep()waiter, which KJ arms only after theport's poll found nothing to do;
poll returns. Re-arming the probe on every such wake is not safe: a
cancelled cross-thread call leaves its wake behind, and the probe
would then keep
WaitScope::poll()from returning.The counter starts moving again when the run queue empties. The
watchdog only fences if that does not happen for the full stall
threshold. During such a stretch the loop also accepts no new
connections, timers, or cross-thread events, so the node is
unresponsive.
The complete fix is a per-turn counter in KJ's
EventLoop::turn(),in capnproto. With that, the probe can be removed. That would be a
follow-up. The current gaps are documented in
progress.rs.Cost
In cluster mode, the probe adds one KJ turn for each pass through a
busy run queue and one turn each time the loop wakes. There is no
watchdog cost outside cluster mode.
Tests
that never sleeps. They also verify that the probe lets the loop
sleep, does not delay
evalLast()orWaitScope::poll(), standsstill while one event holds the thread, and exposes the three
documented gaps.
loop fences while time remains, and rounds begin watching only after
the
Watchcommand.not trigger the watchdog.
cluster-test.mjsadds a/wedgeendpoint whose worker loopsforever. With the new
shortLeaseconfiguration, the node exitswith status 1 and the watchdog reason on stderr after about 6.6
seconds. The threshold is 6 seconds.
the three-lease watchdog threshold.
port tests, the cluster Rust and logic tests, the server tests and
cluster-cron-test, andcluster-testpassed all 200 of its shardruns. An earlier version of this commit had one hang in 200 shard
runs, in a drain test that is already on main. It did not recur in
another 2,200 targeted shard runs, on either that version or Keep a cluster node's lease on a thread of its own #8631's
head.
on Keep a cluster node's lease on a thread of its own #8631), and time out now and then on a slow machine either way.
Make the cluster crate's tests fast again #8560 takes them to about 8 s.
Lines changed