Shared build worker saturation made a single targeted BuildTestEmbedded test compile for many

Reported (UTC): 2026-07-16 07:43

Shared build worker saturation made a single targeted BuildTestEmbedded test compile for many minutes without output.

What was being attempted: Run the hermetic dynamic-dispatch real-runner E2E after first running its deterministic command-generation regression test.

What went wrong: The 15 GiB worker had load average 47.10, 45.39, 40.45, only 994 MiB free memory, and no swap. Process inspection showed several unrelated kompile full-suite and targeted test JVMs compiling concurrently. The targeted command remained healthy and CPU-active but emitted no progress for more than ten minutes.

Impact: Strict red-green validation and later full-suite verification are greatly delayed, and memory pressure risks unrelated OOM failures that could be mistaken for code regressions.

Workaround used: Kept the existing test constraints intact, monitored process liveness and resource state, and waited rather than increasing timeouts or reducing coverage.

Suggested durable fix: Add worker-level admission control for heavyweight local kompile builds, cap concurrent compiler JVMs according to memory/CPU capacity, and expose queue/progress telemetry so queued compilation is distinguishable from a hung build.

---

Production verification — 2026-07-17

Status: STILL EXISTS. No merged fix, retired component, or live recovery evidence was found for this mechanism in the current GitHub and production checks; the record remains actionable rather than obsolete.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.