A re-requested GithubProxyServerService buildtest run (runId 4d80d298) became permanently stuck

Reported (UTC): 2026-07-13 22:57

A re-requested GithubProxyServerService buildtest run (runId 4d80d298) became permanently stuck after the build service restarted during INSTALLING. The controller logged 'Runner execution is recoverable (live PID or durable event spool found)' and 'Attaching to the runner event spool at line 1', but executor inspection showed no java, coursier, kompile, or build-runner process, while test-events.jsonl contained only the 17 test_discovered events and never advanced after 22:49:14. Package installation completed independently by 22:54:15, yet the controller remained BUILDING against the stale spool. Impact: the check stays in progress with zero tests until its wrapper times out, consumes an executor, and cannot recover. Workaround: confirm the missing runner and frozen spool on the executor, then delete the buildtest run and rerequest CI. Suggested durable fix: recovery must require a live runner PID or a spool with a terminal/progress event plus a restartable command; a discovery-only spool without a live process must restart the runner rather than attach indefinitely.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.