buildtest-cli delete succeeds but hangs during noisy UrlProtocol shutdown.

Reported (UTC): 2026-07-16 13:30

buildtest-cli delete succeeds but hangs during noisy UrlProtocol shutdown.

What was being attempted: Following the watch-build skill remediation for a BuildTest merge-queue run (84c1cbcb) frozen at 0/100 tests, I ran ssh -p 23 root@198.199.106.165 'timeout 90 java -jar /root/ContainerNursery/apps/buildtest-cli.jar delete 84c1cbcb'.\n\nWhat went wrong: The delete mutation succeeded, but the CLI emitted roughly 13,260 lines / 592 KB of UrlProtocol peer discovery and gossip output, including repeated Task ... rejected from java.util.concurrent.ThreadPoolExecutor...[Terminated] messages, and never exited. The outer timeout ended it with exit code 124.\n\nImpact: The command appeared failed even though it had deleted the run, produced enough output to overwhelm normal agent/tool output limits, and made it unsafe to know whether re-requesting the GitHub check suite would leave an active orphaned run. This delayed merge-queue recovery and required host-side forensic verification.\n\nWorkaround used: After the timeout, I checked /root/buildtest-data/runs/84c1cbcb/ on the host; the directory and run.json were gone, proving the mutation completed. I then warmed the webhook receiver and re-requested the GitHub check suite once.\n\nSuggested durable fix: Make buildtest-cli delete close its UrlResolver/UrlProtocol resources deterministically after the RPC returns, suppress routine peer-registry/gossip logs in CLI mode, and return exit 0 once deletion completes. Add an end-to-end CLI test that deletes a run, asserts the run is absent, bounds stdout/stderr, and asserts the process exits promptly.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.