BuildTest GitHub check remained in_progress for orphaned run e38ce218, while the BuildTest API

Reported (UTC): 2026-07-16 08:59

BuildTest GitHub check remained in_progress for orphaned run e38ce218, while the BuildTest API returned zero tests and the server-side buildtest-cli delete command failed with: Cannot delete build run: no run found with id 'e38ce218'. The GitHub check at https://buildtest.kotlin.build/run?id=e38ce218 had been frozen at 0/0 for over 13 minutes, so the documented watch-build remediation could not force-fail it because backend bookkeeping did not contain the advertised run id. Impact: the PR could never become green without creating another check, and the prescribed delete command spent almost 90 seconds emitting extremely high-volume mesh gossip before surfacing the actual RPC error. Workaround: warm github-webhooks.wasmserver.com and use the task-authorized single GitHub check-suite rerequest, which created replacement run a8c3b1b7. Suggested durable fix: reconcile GitHub in_progress checks against BuildTest run storage, make the dispatcher atomically create/register a run before publishing its check URL, and have delete fail fast with quiet/log-level-controlled output when the run is absent.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.