While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770

Reported (UTC): 2026-07-14 04:38

While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/770 at head 2acea3500047c26fafd97fccfc100b7d761e9330, buildtest replacement run 87a1ee7d remained PENDING for over 40 minutes despite normal five-minute admission retries. A live GET https://buildtest.kotlin.build/api/runs showed eight TESTING and three PENDING entries; four TESTING entries were 169-171 minutes old with testsPassed=0/testsFailed=0 (ae34b898, 8b2db307, 47a67ab5, 1819fdd1). Run 47a67ab5 had previously been force-failed/deleted after a confirmed provisioning wedge, yet it was again counted as TESTING. The apparent stale/ghost TESTING records may be consuming capacity accounting and starving the queue; 87a1ee7d was the oldest pending run, while later pending runs accumulated. Impact: a required PR check cannot obtain capacity even though its GitHub Actions check is green. Workaround used: preserve queue position and monitor; do not delete unrelated runs. Suggested durable fix: make admission count only leases backed by live droplets/runner heartbeats, expire or reconcile TESTING records whose lease/process/droplets are absent, and ensure delete cannot resurrect a run as TESTING after service restart.

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/770. A live ProductionHealth connection also emitted repeated NothingToCompleteException gossip failures, while the stopped HardwareControlFabric daemon log ends with Netty ByteBuf leak reports.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.