buildtest resume bookkeeping attaches to a nonexistent runner spool and leaves a fresh droplet

Reported (UTC): 2026-07-13 21:46

buildtest resume bookkeeping attaches to a nonexistent runner spool and leaves a fresh droplet permanently idle.

What was being attempted: shepherd UrlResolver PR 680 remote CI run 0f284ca7 to green after build-watchman reported 0/100 tests for fifty minutes.

What went wrong: buildtest provisioned droplet 584376508 at 64.23.140.13 and central build.log recorded Runner execution is recoverable (live PID or durable event spool found) followed by Attaching to the runner event spool at line 1 at 21:23:47 UTC. Central test-events.jsonl and build.log then froze for over twenty-one minutes. Direct read-only inspection of the assigned droplet showed a healthy OS but NO runner process of any kind and NO jsonl/build/runner spool file anywhere on the root filesystem; the complete process list contained only system services and inspection shells. The run remained status BUILDING with zero completed tests. The server therefore treated stale/recovery bookkeeping as proof of a recoverable execution on a newly provisioned droplet and skipped the runner-launch path, then waited forever on a spool that did not exist.

Impact: every such run consumes a paid droplet while doing zero work, GitHub checks remain pending indefinitely, and generic monitoring initially misdiagnoses the symptom as slow provisioning or frozen tests. No amount of waiting can recover because no executor process exists.

Workaround used: after proving the droplet has neither runner nor spool, force-fail run 0f284ca7 with buildtest-cli delete and re-request the GitHub check suite so a fresh run takes the normal launch path.

Suggested durable fix: make the recoverability predicate require evidence on the CURRENT assigned droplet (validated live runner PID or an existing, nonempty durable spool with the expected run identity), not stale central metadata. Before attaching, verify the spool exists and is advancing; if neither runner nor spool exists, launch the runner. Persist and compare droplet identity/generation across service restarts, and add an end-to-end test covering service restart before droplet assignment followed by fresh provisioning, asserting the new droplet actually starts a runner.

---

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.