While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770

Reported (UTC): 2026-07-14 14:54

While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770 at exact head 2acea3500047c26fafd97fccfc100b7d761e9330, buildtest run b4f5dd79 reached 92/100 test scripts but shard 5 then emitted the exact diagnostic Control socket idle for 60000ms with no RPC from the child; child process is still alive, so continuing to wait (the per-test timeout still bounds the run) once per minute for more than 35 minutes. test-events.jsonl last advanced at 2026-07-14 14:17:22 UTC while build.log kept advancing only because of these idle diagnostics through at least 14:53:28 UTC. This is a contradiction: the runner claims the per-test timeout bounds the live child, but no per-test timeout fires, so the strict wedge detector cannot recover because build.log remains fresh and the run can consume its full maxBuildDurationMs=10800000. Impact: a single lost child RPC blocks eight remaining scripts and keeps ten CI shards occupied for up to three hours; operators cannot apply the documented dual-file wedge remediation because synthetic idle logging defeats the frozen-build.log predicate. Workaround used: leave the run alone and wait for its overall three-hour cap, per watch-build policy. Suggested durable fix: make the control-socket watchdog compare the current test monotonic deadline and forcibly terminate and report the child when exceeded; classify periodic watchdog diagnostics separately from substantive build progress so build-watchman can recognize an event-stalled child without those lines resetting build.log freshness.

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/770. A live ProductionHealth connection also emitted repeated NothingToCompleteException gossip failures, while the stopped HardwareControlFabric daemon log ends with Netty ByteBuf leak reports.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.