Skip to content

Fence a cluster node whose event loop makes no progress - #8637

Open
piscisaureus wants to merge 1 commit into
piscisaureus/cluster-lease-threadfrom
piscisaureus/cluster-loop-watchdog
Open

piscisaureus wants to merge 1 commit into
piscisaureus/cluster-lease-threadfrom
piscisaureus/cluster-loop-watchdog

Conversation

@piscisaureus

@piscisaureus piscisaureus commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

This PR is stacked on #8631, which moves lease renewal off the
server event loop. That keeps healthy busy nodes alive, but also keeps
a permanently stuck node alive forever. This PR adds a watchdog so
peers can eventually take over that node's Durable Objects.

How it works

A normal heartbeat is not enough. KJ only polls timers and its event
port after its run queue becomes empty. A healthy loop processing many
short turns, such as a fast upload, might never become empty. A timer,
tokio interval, or executor ping would not run, so the watchdog would
mistake useful work for a wedged loop.

Instead, this PR counts progress in two ways:

  • While the loop is busy, a TokioEventPort::ProgressProbe event moves
    to the back of the KJ queue after each run through the queue. Each
    trip increments a counter.
  • When only the probe remains, it leaves the queue. This lets the loop
    sleep and lets evalLast() events run normally. The event port
    restores the probe after every wait, and after a poll that leaves
    events to run.
  • While the loop is idle, a task on its LocalSet increments the same
    counter once per lease-renewal interval.
  • kj_rs_tokio::watch_progress() returns a thread-safe handle to that
    counter.
  • After each lease step, the keeper thread checks whether the counter
    has changed. If it has not changed for three leases, normally 60
    seconds, the node fences itself with exit status 1. Like the other
    cluster times, this threshold is divided by
    unsafeTimeDivisor.
  • Checking after the lease step means a whole-process pause, such as
    SIGSTOP or a VM pause, is still reported as a lost lease.
  • Watching starts only when ClusterNode::serving() is called.
    Linking workers after joining is one long synchronous turn and must
    not look like a stall.
  • If the watchdog cannot start, the node logs a warning with the
    reason and serves on unwatched.
  • Fencing uses the same Armed::fence path as a lost lease, so it
    happens only once and while the lease still holds. The supervisor
    can restart the node, and peers can take over its objects after the
    lease expires.
  • Fatal messages now say leaving the cluster: {reason}. Watchdog
    failures say that the event loop made no progress for the measured
    number of milliseconds.

A depth-first chain counts as one event until the chain finishes.

Known gaps and follow-up

There are three cases where a busy loop can make progress without
moving the counter. They happen when a busy stretch starts while the
probe is outside the queue, and something other than the port's wait
starts it:

  • an evalLast() or yieldUntilQueueEmpty() event, which KJ queues
    where the probe cannot follow;
  • a yieldUntilWouldSleep() waiter, which KJ arms only after the
    port's poll found nothing to do;
  • events from another thread, which KJ takes in only after the port's
    poll returns. Re-arming the probe on every such wake is not safe: a
    cancelled cross-thread call leaves its wake behind, and the probe
    would then keep WaitScope::poll() from returning.

The counter starts moving again when the run queue empties. The
watchdog only fences if that does not happen for the full stall
threshold. During such a stretch the loop also accepts no new
connections, timers, or cross-thread events, so the node is
unresponsive.

The complete fix is a per-turn counter in KJ's EventLoop::turn(),
in capnproto. With that, the probe can be removed. That would be a
follow-up. The current gaps are documented in progress.rs.

Cost

In cluster mode, the probe adds one KJ turn for each pass through a
busy run queue and one turn each time the loop wakes. There is no
watchdog cost outside cluster mode.

Tests

  • Watchdog unit tests cover its decisions.
  • Progress tests cover the Rust handle and counter.
  • C++ event-port tests verify that the count advances in a busy loop
    that never sleeps. They also verify that the probe lets the loop
    sleep, does not delay evalLast() or WaitScope::poll(), stands
    still while one event holds the thread, and exposes the three
    documented gaps.
  • Keeper tests verify that a progressing loop keeps its lease, a stuck
    loop fences while time remains, and rounds begin watching only after
    the Watch command.
  • A startup test verifies that a long synchronous turn after join does
    not trigger the watchdog.
  • cluster-test.mjs adds a /wedge endpoint whose worker loops
    forever. With the new shortLease configuration, the node exits
    with status 1 and the watchdog reason on stderr after about 6.6
    seconds. The threshold is 6 seconds.
  • The long-turn integration test now runs for up to 1.5 leases, below
    the three-lease watchdog threshold.
  • Forty uncached runs each passed the kj-rs-tokio tests, the event
    port tests, the cluster Rust and logic tests, the server tests and
    cluster-cron-test, and cluster-test passed all 200 of its shard
    runs. An earlier version of this commit had one hang in 200 shard
    runs, in a drain test that is already on main. It did not recur in
    another 2,200 targeted shard runs, on either that version or Keep a cluster node's lease on a thread of its own #8631's
    head.
  • The cluster Rust tests average 41.5 s against a 60 s limit (40.4 s
    on Keep a cluster node's lease on a thread of its own #8631), and time out now and then on a slow machine either way.
    Make the cluster crate's tests fast again #8560 takes them to about 8 s.
  • Clang-tidy and clippy are clean.

Lines changed

Kind Added Removed
Implementation 434 21
Tests 571 9
Documentation 7 2

@piscisaureus
piscisaureus requested review from a team as code owners November 18, 2026 23:57
@piscisaureus
piscisaureus requested a review from ry November 18, 2026 23:57

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of a15ea2b9.

This review, of 0d0fa671

💬 Approved with comments — No blocking or warning-level issues; one suggestion (log when the watchdog cannot be armed) and two informational notes on test timing and documentation wording.

Findings: 2 info · 1 suggestion (3 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info · 1 suggestion

Sound design; the watchdog fails to arm silently, and one test has tight timing slack.

Recommendation: 💬 Optional Nits

  • [INFO] 2 ms timing slack in the keeper fence-timing assertion may be flaky under load (src/workerd/server/cluster/keeper-test.rs:364)
  • [SUGGESTION] If the watchdog cannot be armed, serving() runs the node unwatched and logs nothing (src/workerd/server/server/cluster.rs:497)
View full review
  • src/workerd/server/server/cluster.rs:497: if kj_rs_tokio::watch_progress() returns None, the node runs without a watchdog and logs nothing. That happens with no port or LocalSet, or when start_progress_probe() fails, and that error is dropped by .ok()?. watched is still set to true, so it never retries.
  • src/workerd/server/cluster/keeper-test.rs:364 (and :396): the assertion allows only 2 ms between the watchdog's since_ms and the test's last_ms stamp. A descheduled thread on a loaded CI machine could fail it.

📝 Documentation: 💬 1 info

The new AGENTS.md notes are worded more broadly than the documented gaps.

Recommendation: 💬 Optional Nits

  • [INFO] AGENTS.md note says progress is counted "busy or asleep" without the documented busy-loop gaps (src/rust/cxx/AGENTS.md:41)
View full review
  • src/rust/cxx/AGENTS.md:40-41: the note says the counter tracks progress "busy or asleep", but progress.rs documents busy stretches it does not count.

✅ All other checks passed

🔒 Security: ✅ LGTM

The watchdog adds no new untrusted input, authentication change or secret handling. Fencing goes through the existing fail-closed path, and the /wedge endpoint exists only in test config.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md issues found.


⚡ Performance: ✅ LGTM

No performance issues found.


Reviewed commit: 0d0fa671

Comment thread src/rust/cxx/AGENTS.md Resolved
Comment thread src/workerd/server/server/cluster.rs Resolved
@piscisaureus
piscisaureus force-pushed the piscisaureus/cluster-loop-watchdog branch from 0d0fa67 to a15ea2b Compare November 19, 2026 00:12

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of c4c4571d.

This review, of a15ea2b9

✅ Approved — No issues found. The suggestion and two informational notes from the previous review are all fixed at this commit.

Not opened by the correctness review: 1 changed file it was meant to read
  • src/rust/cxx/kj-rs-tokio/lib.rs

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

The watchdog adds no untrusted input, no authentication change and no secret handling. Fencing goes through the existing fail-closed Armed::fence path. The /wedge endpoint exists only in the test config.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md issues found.


📋 Code Quality: ✅ LGTM

Both earlier notes are fixed:

  • serving() now logs a tracing::warn! with the WatchError reason (NoPort or Probe(String)) and keeps serving unwatched.
  • The keeper fence-timing assertions now allow a quarter of stall_ms.

Watchdog::observe starts timing when it first sees a count value and fences only after that count has stood still for stall_ms. The keeper checks it after the lease step, so a whole-process pause is still reported as a lost lease.


⚡ Performance: ✅ LGTM

The probe adds one KJ turn per pass through a busy run queue, and only in cluster mode. The lease thread reads the counter once per round, which costs very little.


📝 Documentation: ✅ LGTM

Both AGENTS.md notes now say the count misses the busy stretches that progress.rs lists.


Reviewed commit: a15ea2b9

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 12.40876% with 120 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (piscisaureus/cluster-lease-thread@088fe06). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/workerd/server/cluster/keeper.rs 0.00% 28 Missing ⚠️
src/rust/cxx/kj-rs-tokio/progress.rs 0.00% 25 Missing ⚠️
src/rust/cxx/kj-rs-tokio/tokio-event-port.c++ 29.41% 23 Missing and 1 partial ⚠️
src/workerd/server/cluster/logic/watchdog.rs 0.00% 15 Missing ⚠️
src/workerd/server/server/cluster.rs 0.00% 8 Missing ⚠️
src/workerd/server/tests/cluster/cluster-worker.js 0.00% 8 Missing ⚠️
src/rust/cxx/kj-rs-tokio/port.rs 53.84% 6 Missing ⚠️
src/workerd/server/cluster/cluster.rs 0.00% 3 Missing ⚠️
src/workerd/server/cluster/logic/lease.rs 0.00% 3 Missing ⚠️
Additional details and impacted files
@@                         Coverage Diff                          @@
##             piscisaureus/cluster-lease-thread    #8637   +/-   ##
====================================================================
  Coverage                                     ?   66.98%           
====================================================================
  Files                                        ?      825           
  Lines                                        ?   212554           
  Branches                                     ?    32467           
====================================================================
  Hits                                         ?   142386           
  Misses                                       ?    51139           
  Partials                                     ?    19029           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@piscisaureus
piscisaureus force-pushed the piscisaureus/cluster-lease-thread branch from ce76f07 to 1728f15 Compare November 19, 2026 23:35
@piscisaureus
piscisaureus force-pushed the piscisaureus/cluster-loop-watchdog branch from a15ea2b to c4c4571 Compare November 19, 2026 23:35
@piscisaureus
piscisaureus requested a review from jasnell November 19, 2026 23:35

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of a216de7a.

This review, of c4c4571d

✅ Approved — No findings. The code is unchanged since the previous approval, and kj-rs-tokio/lib.rs, which that review did not open, only adds re-exports and a module declaration.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

No security defects found. Fencing uses the existing fail-closed Armed::fence path, and the /wedge endpoint exists only in the test config.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md issues found.


📋 Code Quality: ✅ LGTM

src/rust/cxx/kj-rs-tokio/lib.rs only adds the mod progress; declaration, a doc line and re-exports of watch_progress, Progress and WatchError. current_local_set in port.rs is now pub, but its module is private and the function is not re-exported, so the crate's public API stays the same.

The progress probe code checks out:

  • Destruction order: progressProbe is declared after loop and rustPort, so the probe is destroyed before both.
  • Probe start: starting the probe more than once is safe. It is created once and only re-armed after that.
  • Idle beat task: it stops once the last Progress handle is dropped.

⚡ Performance: ✅ LGTM

No performance issues found. The probe adds one KJ turn per pass through a busy run queue, and only in cluster mode.


📝 Documentation: ✅ LGTM

No documentation issues found.


Reviewed commit: c4c4571d

A cluster node keeps its lease on a thread of its own, so a turn of
its event loop of any length keeps it. A loop that is stuck for good
(a deadlock, native code that does not return, a turn that never
ends) therefore kept the lease, and its Durable Objects unreachable,
for as long as the process lived.

Once the node serves, the lease thread now watches the loop. A loop
that makes no progress fences the node through the same path and
fatal line as a lease that cannot be kept:

* `kj_rs_tokio::watch_progress()` gives a count that the loop advances
  while it makes progress, readable from any thread. A busy KJ loop
  polls its port only once its run queue is empty, so between its
  turns no tokio task runs, no KJ timer fires (`TimerImpl` fires only
  after the port's `wait()` or `poll()`) and no `kj::Executor` ping is
  taken in (drained only after the port wakes). The count's busy half
  is therefore a KJ event, the port's progress probe: it counts and
  re-arms itself breadth-first, so it fires once per pass through the
  run queue, and leaves the queue once nothing but itself is left, so
  that the loop still sleeps and runs its `evalLast()` events. The
  port re-arms it after every `wait()`, and after a `poll()` that
  leaves events to turn. While the loop sleeps, a task on its
  `LocalSet` counts once a beat.
* The count stands still while one event holds the thread, and also
  through three kinds of busy stretch that begin while the probe is
  out of the queue:
  * one that an `evalLast()` / `yieldUntilQueueEmpty()` event starts:
    `armLast()` queues each in front of the ones before, so the probe
    cannot follow them;
  * one that a `yieldUntilWouldSleep()` waiter starts: KJ arms the
    waiter after the port's `poll()` returns, and only if that poll
    left no event;
  * one that a cross-thread event starts when KJ takes it in after
    the port's `poll()` rather than its `wait()`. Arming the probe for
    the wake would let a stale wake, whose event KJ cancelled without
    consuming it, keep `WaitScope::poll()` from returning and starve
    would-sleep waiters.
  Such a stretch is counted again once the run queue empties. Only
  work that is ready without the port keeps it from emptying, and one
  that lasts a whole `stall_ms` fences the node. Closing all three
  needs a per-turn hook in KJ's `EventLoop::turn()`.
* `cluster_logic::watchdog` decides: a count that stands still for
  `Timing::stall_ms` fences the node. The keeper reads it once a
  round, after the round's lease step, so that a process stopped as a
  whole reports its lapsed lease. Its reason ends the rounds as a
  lapsed lease does, through `Armed::fence`, while the lease holds.
* The watching starts with `ClusterNode::serving`, which sends the
  count to the lease thread (`Cluster::watch`). Linking, after `join`,
  runs in turns as long as itself, and the lease thread keeps the
  lease through it as before. A loop that cannot be watched
  (`watch_progress()` answers why) serves on unwatched, with a
  warning that says so.
* `stall_ms` is three leases, 60 s, divided by `unsafeTimeDivisor` as
  the other times are. That is twice the turn of one and a half
  leases that the long-turn test rides out. The beat is the renewal
  interval.
* The fatal line now reads "leaving the cluster: {reason}", where a
  lease's reasons begin "its lease cannot be kept: " and the
  watchdog's reads "its event loop made no progress for N ms".

The probe costs a KJ turn per pass through a busy run queue and one
per wake of the loop, in cluster mode only. A chain of events that
arm each other depth-first runs ahead of it, as it runs ahead of
every other event, and counts as one event until it ends.

The cluster test's Worker gains a `/wedge` endpoint that loops
forever, and a test that its node exits with status 1 and the reason
on stderr, under a `shortLease` config with a tenth of the default
times. The long-turn test now aims at one and a half leases, at most
four times its last guess, rather than doubling past a lease.
@piscisaureus
piscisaureus force-pushed the piscisaureus/cluster-lease-thread branch from 1728f15 to 088fe06 Compare November 20, 2026 06:51
@piscisaureus
piscisaureus force-pushed the piscisaureus/cluster-loop-watchdog branch from c4c4571 to a216de7 Compare November 20, 2026 06:51

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — No findings. The changes since the last approval to the keeper, serving() and the test config check out, and the earlier points raised in discussion are fixed.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

No security issues found. Fencing uses the existing fail-closed path, and the /wedge endpoint exists only in the test config.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md issues found.


📋 Code Quality: ✅ LGTM

No issues found.

  • keeper.rs: Watch goes through the same command channel as Maintain. The watchdog checks progress after the lease step, and its reason goes through Armed::fence.
  • server/cluster.rs: serving() starts the watchdog only once, guarded by watched.replace(true). If watch_progress fails, it logs a warning and serves on unwatched.
  • cluster-test.wd-test: shortLease has unsafeTimeDivisor = 10, which matches SHORT_LEASE_STALL_MS.

⚡ Performance: ✅ LGTM

No performance issues found. The keeper reads one atomic counter per lease round, and in cluster mode the probe adds one KJ turn per pass through a busy run queue.


📝 Documentation: ✅ LGTM

No documentation issues found.


Reviewed commit: a216de7a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants