While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/680,

Reported (UTC): 2026-07-13 22:43

While shepherding UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/680, replacement buildtest run 689ff28d executed successfully on all 10 shards, but central /root/buildtest-data/runs/689ff28d/test-events.jsonl remained frozen at its discovery events from 21:47 UTC and run.json reported TESTING with 0/100 while shard spools grew to roughly 290-342 KB. Direct inspection proved all 10 /root/btr-events-689ff28d.spool files ended with ##BTR## {"event":"done"}, all 10 /root/kompile-exit-code files contained 0, and sampled events reported passing tests. The pre-restart buildtest-server stderr repeatedly emitted, for every shard, "[BuildTestService] WARNING: Failed to renew droplet lease for run '689ff28d', droplet <id>: previous renewal is still running." Consumer calls occasionally advanced the timestamp, but the previous renewal operations never completed and the coordinator never consumed test spools. ContainerNursery replaced buildtest-server-service at 22:41 UTC; its recovery loaded run 689ff28d, retained all 10 droplets, then ingested the already-complete spools and immediately projected 1108/1108 passed, 0 failed, buildSucceeded=true. Impact: GitHub displayed 0/100 and build-watchman simultaneously advised force-failing the run and called it progressing, which would have discarded a completely successful 10-shard execution. Durable fix: in buildtest-server, bound each droplet lease-renew operation with an injectable-clock deadline/cancellation, clear in-flight renewal bookkeeping in finally on every completion/failure, ensure spool/result ingestion is independent of lease-renew completion, add an end-to-end test where droplet-service renewal never returns while shard spools finish and verify the run still finalizes 1108/1108, and expose distinct metrics for shard spool freshness versus central projection freshness.

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/680. A live ProductionHealth connection also emitted repeated NothingToCompleteException gossip failures, while the stopped HardwareControlFabric daemon log ends with Netty ByteBuf leak reports.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.