Skip to content

fix(util): timed-out shell commands no longer leave child processes running - #62815

Open
only21mil wants to merge 2 commits into
anomalyco:devfrom
only21mil:kill-whole-group
Open

only21mil wants to merge 2 commits into
anomalyco:devfrom
only21mil:kill-whole-group

Conversation

@only21mil

@only21mil only21mil commented Nov 9, 2026 •

Copy link
Copy Markdown

Issue for this PR

Related to #59006, which asks for a way to stop an agent's whole process tree. No issue covers this exact case, so this PR fixes one focused part of it: what a shell timeout or abort leaves behind.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

On a shell timeout or abort, stop sends SIGTERM to the process group and escalates to SIGKILL only if the leader (sh) or its output is still open when forceKillAfter ends. The leader usually exits at once, so a child that ignores SIGTERM keeps running with no parent. One that keeps starting children can use up the user's process limit.

With forceKillAfter set, once the leader has exited and its output has closed, stop now sends one SIGKILL to the group. It doesn't wait for the group to empty, since zombies keep it alive where nothing reaps orphans (opencode as PID 1 in the Docker image, #53564). The grace-period path is unchanged.

Trade-off: a child that redirected its output gets SIGKILL as soon as the leader exits, with no grace period. A child holding the pipes gets the same wait as before.

This covers every stop with forceKillAfter: shell timeout and abort, MCP stdio servers and scope release. Windows, detached: false and stops without forceKillAfter are unchanged, and a command that exited 0 and closed its output still leaves background children running. Separate from #61876.

How did you verify your code works?

  • New test in cross-spawn-spawner.test.ts: the leader exits on SIGTERM and its child ignores SIGTERM with its output redirected. It fails on dev and passes with this change on Linux x86_64 (host, and PID namespaces whose PID 1 reaps or never reaps) and macOS 27 arm64.
  • Suites from packages/core: 33 pass, 1 skip, 0 fail; with the shell, MCP and process suites, 206 pass, 24 skip, 0 fail. Type checks, Prettier and oxlint pass.
  • Runtime: on dev, two group members are still starting processes 4 s after kill. With this change the group is empty. Under a PID 1 that never reaps, kill takes 2 to 4 ms over six runs, against 3 to 10 ms on dev.

Not checked: Windows, whose code path is unchanged.

Screenshots / recordings

No UI change, so the evidence is test and runtime output, unedited. Linux x86_64, bun 1.4.4. dev is 1017760240 with only this PR's test file added, and this PR is ad0abe3.

Commands and unedited test and runtime output

Commands, from packages/core unless named: bun run script/test.ts test/effect/cross-spawn-spawner.test.ts, then with the shell, MCP and process suites added. bun check --no-script and tsc --noEmit in packages/util, bun check --no-script -b -p tsconfig.tests.json in packages/core, and Prettier and oxlint on the two changed files.

The new test waits after kill until the child is gone or a zombie.

Runtime case: a child that ignores SIGTERM and keeps starting sleep, stopped with handle.kill({ forceKillAfter: "3 seconds" }) after 1 s. The PID 1 timing case is kill on sh -c 'sleep 30 & echo $!; wait'.

dev: the new test times out (child still running)

The new test on dev: the child is still running after kill, so the wait times out.

$ bun run script/test.ts test/effect/cross-spawn-spawner.test.ts -t "process group after its leader exits"
bun test v1.4.4 (38afee6c8)

test/effect/cross-spawn-spawner.test.ts:
[07:23:20.856] ERROR (#1):  9 |   Effect.gen(function* () {
10 |     const deadline = Date.now() + 10_000
11 |     while (true) {
12 |       const value = yield* effect
13 |       if (predicate(value)) return value
14 |       if (Date.now() > deadline) return yield* Effect.fail(new Error("Timed out waiting for value"))
                                                                    ^
error: Timed out waiting for value
      at /work/opencode/packages/core/test/lib/eventually.ts:14:64

 9 |   Effect.gen(function* () {
10 |     const deadline = Date.now() + 10_000
11 |     while (true) {
12 |       const value = yield* effect
13 |       if (predicate(value)) return value
14 |       if (Date.now() > deadline) return yield* Effect.fail(new Error("Timed out waiting for value"))
                                                                    ^
error: Timed out waiting for value
      at /work/opencode/packages/core/test/lib/eventually.ts:14:64
      at ~effect/Effect/successCont (/work/opencode/node_modules/.bun/effect@4.1.0/node_modules/effect/dist/internal/effect.js:1017:28)
      at runLoop (/work/opencode/node_modules/.bun/effect@4.1.0/node_modules/effect/dist/internal/effect.js:476:88)
      at evaluate (/work/opencode/node_modules/.bun/effect@4.1.0/node_modules/effect/dist/internal/effect.js:425:23)
      at runInAsyncScope (node:async_hooks:207:23)
      at evaluate (/work/opencode/node_modules/.bun/effect@4.1.0/node_modules/effect/dist/internal/effect.js:414:29)
      at /work/opencode/node_modules/.bun/effect@4.1.0/node_modules/effect/dist/internal/effect.js:836:17
(fail) cross-spawn spawner > process control > forceKillAfter kills the rest of the process group after its leader exits [10067.53ms]

 0 pass
 33 filtered out
 1 fail
 1 expect() calls
Ran 1 test across 1 file. [10.30s]

ad0abe3: the new test passes

The new test at ad0abe3: it passes.

$ bun run script/test.ts test/effect/cross-spawn-spawner.test.ts -t "process group after its leader exits"
bun test v1.4.4 (38afee6c8)

 1 pass
 33 filtered out
 0 fail
 1 expect() calls
Ran 1 test across 1 file. [290.00ms]

dev: two group members still running 4 s after kill

A timeout through the package's spawner: the leader starts a child that ignores SIGTERM and keeps starting sleep, and handle.kill({ forceKillAfter: "3 seconds" }) runs at 1 s. On dev, kill returns at once and two group members are still running 4 s later.

$ bun run timeout-repro.ts
[0.0s] started leader sh, pid 38
[1.0s] before kill: 3 member(s) in process group 38
  pid 38  ppid 2  state S  sh -c (trap "" TERM; while :; do sleep 0.5; done) >/dev/null 2>&1 & echo ready; wait
  pid 39  ppid 38  state S  sh -c (trap "" TERM; while :; do sleep 0.5; done) >/dev/null 2>&1 & echo ready; wait
  pid 48  ppid 39  state S  sleep 0.5
[1.0s] kill({ forceKillAfter: "3 seconds" }) returned after 2 ms
[5.0s] 4 s after kill: 2 member(s) in process group 38
  pid 39  ppid 1  state S  sh -c (trap "" TERM; while :; do sleep 0.5; done) >/dev/null 2>&1 & echo ready; wait
  pid 56  ppid 39  state S  sleep 0.5
[5.0s] cleanup: sent SIGKILL to 39, 56

ad0abe3: the group is empty 4 s after kill

At ad0abe3: kill returns at once and the group is empty.

$ bun run timeout-repro.ts
[0.0s] started leader sh, pid 26
[1.0s] before kill: 3 member(s) in process group 26
  pid 26  ppid 2  state S  sh -c (trap "" TERM; while :; do sleep 0.5; done) >/dev/null 2>&1 & echo ready; wait
  pid 27  ppid 26  state S  sh -c (trap "" TERM; while :; do sleep 0.5; done) >/dev/null 2>&1 & echo ready; wait
  pid 36  ppid 27  state S  sleep 0.5
[1.0s] kill({ forceKillAfter: "3 seconds" }) returned after 2 ms
[5.0s] 4 s after kill: 0 member(s) in process group 26

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Generated with Claude Opus 5.5 and GPT-6.1 Sol in T3 Code Nightly


Added by the opencode bot (AI-generated).

Related: #59006, #53564

A stop with forceKillAfter sent SIGTERM to the process group and escalated
to SIGKILL only while the leader was still running. When the leader exited
on SIGTERM, any child that outlived the signal kept running, and could keep
starting more processes, with nothing tracking the group.

After the first signal, wait for the whole group to empty. When the grace
period ends, send SIGKILL to the group until it has no members, for at most
one second.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the fix works on a normal Linux host, but it adds a delay of about 4 s to every shell timeout or abort when opencode runs as PID 1, which is what the official Docker image does.

How I checked

  • In a PID namespace whose init reaps processes, the new test fails on the base and passes with this change. The whole spawner test file also passes there.
  • In this sandbox, PID 1 doesn't reap orphaned processes, so dead group members stay as zombies. kill(-pgid, 0) keeps succeeding on zombies, so the group never looks empty.
  • In that setup, I timed handle.kill({ forceKillAfter: "3 seconds" }) on sh -c 'sleep 30 & echo $!; wait', where sleep exits on SIGTERM. It takes 2 ms on dev and 4002 ms with this change: the full grace period, then the full 1 s SIGKILL loop. Under a reaping init it takes 2 ms.
  • The new test also fails in that setup.

Who is affected: packages/cli/Dockerfile uses ENTRYPOINT ["opencode"] with no init, so opencode is PID 1 and never reaps orphans (#53564). In that image, any timed-out or aborted command with a background child now costs forceKillAfter + 1 s, as does any scope release that goes through stop. Details and options are in the inline comment.

Public API: unchanged.

Related: #59006, #53564

Waiting for kill(-pgid, 0) to fail never ends where nothing reaps orphans,
such as opencode as PID 1 in its Docker image: zombies keep the group
alive, so every stop with forceKillAfter took the grace period plus the
1 s SIGKILL loop.

Once the leader has closed, send one SIGKILL to its group instead and
return. Members that outlived SIGTERM still die, and a stop where every
member exited on SIGTERM returns in milliseconds again. After the grace
period, the SIGKILL path is the same as before this PR.

The test now waits until the child is gone or a zombie (state Z in
/proc/<pid>/stat), so it also passes under a PID 1 that never reaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@only21mil

Copy link
Copy Markdown
Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

Both review threads are answered in ad0abe3. The check, check-nix, check-test and check-bot-scope runs on that head are waiting for a maintainer to approve workflows from this fork (action_required); could someone approve them? check-standards and check-compliance already pass.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The zombie concern from my last review is resolved, and I found nothing else that should block this.

How I checked

  • With PID 1 never reaping orphans, stopping sh -c 'sleep 30 & wait' takes 2 ms, the same as dev.
  • A SIGTERM-ignoring child that keeps starting sleep is still running after kill on dev. With this change, only zombies are left.
  • The new test fails on dev, with and without a reaping init, and passes with this change in both.
  • The spawner test file and tsc in packages/util pass.

Trade-off: a child that redirected its output now gets SIGKILL with no grace period once the leader exits. It was already sent SIGTERM, and this only happens when forceKillAfter is set, so that seems right for a timeout or abort.

Public API: unchanged.

Related: #59006, #53564

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant