Skip to content

pi-phase: measured concurrency, hang-timeout and fast-admission trials #93

Open

Description

@woksin

Concurrency and hang settings need controlled trials before enforcement. Ledger and source review: 5 heavy and 3 fast slots, MSBuild cap 3, pressure and budget only observed, hang timeout off, and no timeout record shows an applied hang timeout.

Scope

  • Concurrency trial: compare 5 heavy x 3 fast with 4 x 2, with matched workload batches and test parallelism varied separately. Measure completed phases per hour, p95 queue plus run time, CPU time, RSS, swap growth and timeouts.
  • Hang trial: enable hang detection for eligible VSTest phases at 240 s on a subset, recording hang signatures, false terminations, and timeout plus retry hours.
  • Fast-admission shadow mode: compare predicted with actual duration, including warm and cold runs and failure history, without changing admission.
  • Decide enforcement from the measured results and record the decision on this issue.

Status

State: Paused — no session holds this. Phase 1 (hang-240, Stage only) has been live since 2026-11-02 and is running on its own; the accept-or-roll-back decision follows the daily readouts.
Branch: (#98)
Scope: Measured concurrency, hang-timeout and fast-admission trials (user: do all recommended).
Plan: Instrument first, default-off: explicit-path dotnet records worker source bypass; fast-admission shadow (predicted vs actual, warm/cold) without changing admission; hang_timeout observe mode + label scope + hangKill; trial marker on ledger records with pi-phase report --trial; childCpuS. Then trials: baseline, hang 240s subset, 5x3 vs 4x2 A/B, fast shadow read after 7 days.
Delivered: Phase 1 hang-240 trial (Stage only) is live.
Remaining: Read the trial results, decide whether to accept or roll back, then run the 5x3 vs 4x2 A/B and the fast-admission shadow read.
Follow-ups: None.

Activity

  1. self-assigned this
    on Nov 1, 2026
  2. woksin commented on Nov 4, 2026

    @woksin
    OwnerAuthor

    Day-2 readout of phase 1 (177 applied phases): 0 hang kills, 0 false kills, no rollback trigger, and all 11 timeouts were caller timeouts. The go-global accept rule is not met: the treated Stage subset spends 0.89 timeout-hours per 100 phases, against 0.23 for the control group, with zero kills.
    Decision: keep hang-240 on for the Stage subset only, do not roll it out globally, and start phase 2 with the hang configuration frozen. The config is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions