build-watchman falsely classifies an unprovisioned PENDING buildtest run at 0/0 as an event-stream

Reported (UTC): 2026-07-13 21:09

build-watchman falsely classifies an unprovisioned PENDING buildtest run at 0/0 as an event-stream wedge.

What was being attempted: monitor UrlResolver PR 680 with build-watchman 0.0.9 after pushing head 69c3bcc146a9784480afb2089fa91777502bec15. The remote check referenced run 0f284ca7.

What went wrong: after twelve minutes watchman emitted PROBLEM: check 'kotlin.build (remote)' run 0f284ca7 has made NO forward progress for 12min (frozen at 0/0 tests) — stall/wedge signature and prescribed deleting/re-requesting the run. Host ground truth showed there was no test-events.jsonl or build.log to be frozen. run.json explicitly reported status: PENDING, dropletId: 0, empty testSpecs, zero resources, and a recently renewed build lease. The run had not been assigned an executor, so 0/0 was a provisioning/queue state, not a stalled event stream.

Impact: following the prescribed action would force-fail and re-drive a legitimately queued run, create duplicate load, and potentially loop indefinitely whenever droplet assignment exceeds the generic ten-minute progress threshold.

Workaround used: inspect run.json on the buildtest host, distinguish unprovisioned PENDING (dropletId 0, no event/log files) from an executing-but-frozen run, and continue waiting instead of deleting it.

Suggested durable fix: build-watchman should query run metadata before applying the event-stream stall rule. When status=PENDING and dropletId=0 (or test-events.jsonl does not yet exist), classify the run as queued/provisioning, track lease renewal separately, and use a provisioning-specific timeout/remediation. Only call a run wedged for frozen events after an executor/droplet and event stream have actually started.

---

Production verification — 2026-07-17

Status: STILL EXISTS. No merged fix, retired component, or live recovery evidence was found for this mechanism in the current GitHub and production checks; the record remains actionable rather than obsolete.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.