Buildtest service restart invalidated all ten shards and terminally failed a three-attempt

Reported (UTC): 2026-07-14 05:41

Buildtest service restart invalidated all ten shards and terminally failed a three-attempt reprovision path after attempt one.

What was being attempted: completing the kotlin.build remote check for https://github.com/CodexCoder21Organization/UrlResolver/pull/770 head 2acea3500047c26fafd97fccfc100b7d761e9330 on replacement run 87a1ee7d after an earlier wedged run was recovered.

What went wrong: after buildtest resumed, nine existing shard droplets each failed Failed to connect to root@<ip>:22 after 12 attempts. Last error: timeout: socket is not established. Shard 1s original droplet was unavailable and replacement provisioning logged provisioning_failure ... attempt=1 maxAttempts=3 ... action=exhausted, followed by ERROR: Provisioning failed ... attempts=1/3. The run then terminally failed with Build failed after service restart: 10 of 10 shard(s) failed during resume, even though the shard-1 message advertised three attempts but stopped after one. The run stayed at 0/100 tests; this was entirely infrastructure-side.

Impact: a healthy PR head lost a second remote CI run after waiting through capacity admission, compilation, and shard startup, adding over an hour to validation and requiring another full remote run.

Workaround used: preserved the run while build.log and run.json advanced, allowed its own recovery accounting to finish, and let the CI integration supersede the terminally failed check. Build-watchman detected the check-run change and attached to new run 9173ad87; host verification showed its directory materialized and fresh build.log activity with status TESTING.

Suggested durable fix: make restart recovery honor maxAttempts for every failed shard before terminalizing the run, and distinguish droplets that genuinely no longer exist from temporarily unreachable SSH endpoints. Persist the per-attempt state so a service restart cannot collapse attempt=1 maxAttempts=3 into an exhausted terminal result; add an end-to-end restart test covering ten shards where existing droplets become temporarily unreachable and one replacement creation stalls.

---

Production verification — 2026-07-17

Status: STILL EXISTS. Live GitHub verification found 1 referenced tracker(s) still open: https://github.com/CodexCoder21Organization/UrlResolver/pull/770. The production host simultaneously showed load average 60.59, CPU pressure near 90%, ContainerNursery at 238% CPU with 543 threads, and the BuildTest server at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.