Buildtest simultaneously watchdog-failed fresh CLI and UrlResolver reruns during droplet

Reported (UTC): 2026-07-14 16:47

Buildtest simultaneously watchdog-failed fresh CLI and UrlResolver reruns during droplet provisioning.

What was being attempted: After two confirmed remote-run wedges were deleted under the build-watchman single-owner procedure, fresh kotlin.build suites were requested for CodexCoder21Organization/kompile-cli PR 127 and CodexCoder21Organization/UrlResolver PR 775. Replacement run IDs were 955fac54 and 963b1dfc.

What went wrong: Both fresh runs failed before any test executed. Run 955fac54 reported exactly: Build watchdog detected that run '955fac54' has made no provisioning progress for 8 minutes, which exceeds the maximum allowed provisioning stall duration of 5 minutes. Run 963b1dfc reported the same simultaneous eight-minute stall. Their final build-log lines were respectively Sharding 110 discovered tests across 4 droplets and Sharding 1102 discovered tests across 10 droplets. Both checks failed after about nine minutes with zero test execution.

Impact: Two unrelated PRs could not obtain CI signal and had to be re-requested again; repeated infrastructure failures are delaying an already fully validated six-repository upstream runner fix and consume fresh droplet fleets without running tests.

Workaround used: As the sole rerun owner, preserve the failed-run evidence, warm github-webhooks.wasmserver.com three times, rerequest only the failed check suites, verify changed details_url run IDs and fresh host directories, and re-arm one build-watchman --to-merged per PR.

Suggested durable fix: Investigate buildtest droplet creation/boot coordination and capacity in sfo3, especially correlated multi-run provisioning stalls. Record per-droplet create/boot failures in run.json and surface the exact missing shard/droplet rather than only the aggregate watchdog message; consider admission control that does not start a run until its requested shard fleet can actually be provisioned.

---

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.