UrlResolver remote build discovered every test but wedged before executing any of them.

Reported (UTC): 2026-07-14 03:59

What was being attempted: shepherding https://github.com/CodexCoder21Organization/UrlResolver/pull/770 head 2acea3500047c26fafd97fccfc100b7d761e9330 through the kotlin.build remote check with build-watchman 0.0.12.

What went wrong: buildtest run d8b073f0 discovered all 100 tests but remained at 0/100. Host ground truth showed /root/buildtest-data/runs/d8b073f0/test-events.jsonl frozen at 2026-07-14 03:26:35 UTC (181749 bytes, ending in test_discovered events) and build.log frozen at 2026-07-14 03:35:13 UTC (2577270 bytes, ending in compiler warnings). A second probe at 2026-07-14 03:56:29 UTC showed both files byte-for-byte unchanged while GitHub still reported the check pending, satisfying the >20-minute dual-file wedge rule. HardwareControlFabric ping also returned exactly FAILED - Daemon did not respond, requiring the documented read-only SSH fallback for host evidence.

Impact: the PR remote CI could never complete despite no test failure, and GitHub would have remained pending indefinitely without active three-layer monitoring and recovery.

Workaround used: as the single recovery owner, force-failed d8b073f0 with timeout 90 java -jar /root/ContainerNursery/apps/buildtest-cli.jar delete d8b073f0, verified the run disappeared from /api/runs and its test-results became empty, warmed github-webhooks.wasmserver.com three times, rerequested only check suite 79303773200, and re-armed build-watchman. Replacement run 87a1ee7d appeared in the check URL and materialized on the host at 2026-07-14 03:58:42 UTC with fresh run.json and run files.

Suggested durable fix: make buildtest detect the invariant all tests discovered, zero test_started/test_completed events, executor/build log inactive and terminally fail or automatically re-drive the run after a bounded inactivity period. Persist and expose the executor PID/exit state in run metadata so the service can distinguish a dead post-discovery executor from a legitimate compile phase without requiring SSH forensic checks.

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/770. A live ProductionHealth connection also emitted repeated NothingToCompleteException gossip failures, while the stopped HardwareControlFabric daemon log ends with Netty ByteBuf leak reports.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.