BuildTestEmbedded constrained-test harness can orphan CPU spinners after launcher termination.

Reported (UTC): 2026-07-15 21:16

What was being attempted: reproduce and thread-dump e2eCancelResumedBuildProducesCanceled with scripts/constrained-test.bash on the shared /code host.

What went wrong: ps showed multiple groups of 'bash -c while :; do :; done' processes still RUNNING on CPU 0 with PPID 1, including PIDs 571453-571462 at elapsed 22+ minutes and PIDs 574646-574676 at elapsed 15+ minutes. These are the exact spinner command launched by scripts/constrained-java/java, but their parent launcher was gone, so its EXIT trap could no longer reap them. A fresh constrained run then added another 32 spinners to the same CPU. The wrapper tracks spinner PIDs only in the parent shell and has no parent-death signal or independent ownership cleanup.

Impact: later tests inherit uncontrolled CPU saturation, thread-dump attach takes several seconds, the nominal per-run pressure is no longer reproducible, and unrelated agents/builds on the shared host can be slowed indefinitely. This makes before/after flake rates untrustworthy because pressure accumulates across aborted/timed-out harness invocations.

Workaround used: I identified the current run by process ancestry and did not kill PPID-1 spinners because they may belong to other active work; evidence collection explicitly accounts for the pre-existing pressure.

Suggested durable fix: launch spinners in a kill-on-parent-death wrapper (for example a tiny helper using prctl(PR_SET_PDEATHSIG, SIGTERM)), place every constrained run in its own cgroup/process scope that is killed as a unit, or add startup reconciliation that safely reaps only stale spinners carrying a unique run token. Also add an end-to-end harness test that SIGKILLs the launcher and proves no spinner survives.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.