While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770,

Reported (UTC): 2026-07-13 23:15

While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770, buildtest run 866d3874 was falsely recovered twice after ContainerNursery/service restarts. /root/buildtest-data/runs/866d3874/build.log said Runner execution is recoverable (live PID or durable event spool found) and Attaching to the runner event spool at line 1 at 22:52:35 UTC, but a host-side process listing on assigned droplet 134.199.225.219 showed no Java, kompile, build, or runner process, docker ps -a was empty, and a filesystem scan found no event/jsonl/spool file. The SSH attachment session remained idle with no child process. Central test-events.jsonl never advanced beyond 1102 discovery events at 22:41:40. The server later removed the entire run directory/droplet, and GitHub check 86952286037 failed with Status poll for build run 866d3874 ... failed continuously for 300231ms ... Remote build run 866d3874 was not found ... most likely cleaned up after a server-side timeout. Impact: ~38 minutes of false BUILDING state, an infrastructure-red required check, and operator time needed to distinguish a compile phase from a dead runner. Workaround: preserve the run until the watch-build host checks established the contradiction, warm github-webhooks.wasmserver.com, rerequest only suite 79267702146, and verify replacement check run 86958679318. Suggested durable fix: make restart recovery validate the recorded PID is alive and the durable spool path exists/is advancing before logging recoverable or attaching; if neither is true, mark the run infrastructure-failed immediately while retaining its run directory/logs for diagnosis instead of deleting it into five minutes of not-found polling.

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/770. A live ProductionHealth connection also emitted repeated NothingToCompleteException gossip failures, while the stopped HardwareControlFabric daemon log ends with Netty ByteBuf leak reports.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.