Handoff: Separate run verdict publication from shard collection ownership

Written 2026-10-05. RE-VERIFY: This is a write-time snapshot. Recheck the branch heads with git ls-remote and main with git fetch. No product fix or PR exists. Checked branch PR and Actions run lists were empty; no branch CI verdict is available. The first baseline attempt failed on the intended COMPLETED versus TESTING assertion in65141ms; see the linked findings and console report.

The user requested: “publish a run's verdict when its test evidence is complete, not when every shard's collection work has returned.” Production had90 backstop-finalized runs since22:02 UTC4Oct, with machines held for roughly two minutes or more after results; a merge-queue entry was evicted while in that state. The goal is machine utilization near100%. This lane was instructed to stop if the change required an ownership redesign; that stop condition was reached before any product edit.

The mechanism is that both fresh and resumed completion paths join whole shard workers before publishing the verdict, while those workers still own optional remote collection and distribution/cache cleanup. Complete authoritative test evidence does not release that join. A deterministic real-SSH probe holds both optional heap probes after durable PASSED results. This lane reran it once on c6e73aa5 main:0passed/1failed in65141ms, with the expected COMPLETED versus TESTING assertion. The console also shows the unchanged120000ms backstop publication and both droplets being deleted. The command accidentally omitted --local and defaulted to a local-and-remote race; the local fork executed the reported scenario, so do not describe this as an explicit local-only invocation. Diagnosis branch has its valid failing attempt and controlled stacks; the product branch here cherry-picks the same test and its fixture correction onto newer main.

Current architecture makes terminal status both a verdict and an ownership boundary: ordinary worker logs and runner events are refused, dynamic writers close, admission state is released, terminal-run droplets become eligible for deletion, and startup resumes only nonterminal runs. A fresh worker also assigns its in-memory artifact and clears requiredBuildPending after optional heap collection; the parent persists the artifact after the joins. An earlier call to the evidence backstop helper therefore cannot satisfy all of the user's requirements: it can lose requested artifacts or XML/output enrichment, suppress optional collection warnings, and make remote inputs disappear during collection. Merely detaching the probe to another in-memory thread does not create restart recovery.

A separate production case must remain distinct. Run7cbce8ad had FAILED XML at00:08:31.252 while authoritative retry resolution did not arrive until00:11:45.781 (aggregate authority00:11:46.852,451ms before verdict). Its alleged two-minute complete-input interval was not complete authoritative evidence. Do not publish solely from terminal projected XML counts. This readiness mismatch is a second cause; no fix for it was attempted here.

Relevant PRs / refs

Nothing was deployed, published, merged or enqueued in this lane. This is established by the action history: only source reads, test runs, branch pushes and handoff writes were done; no production operations were invoked. No production version is claimed.

Remaining work

1. Re-verify current state first. Obtain a new brief authorizing the necessary ownership redesign; this lane's brief explicitly stopped before it.
2. Define a durable per-shard mandatory-work completion boundary using exact test identities, selected retry authority, requested build outcome and artifact obligations. Preserve missing-result TESTING behavior and actual build failures.
3. Separate that boundary from post-verdict collection ownership. Promised data must be staged durably before its droplet is released; remaining attachments and diagnostic uploads must have an explicit owner, visible failure recording and restart recovery independent of run status. Update deletion/reconciliation and log/event publication contracts coherently, rather than bypassing all terminal guards.
4. Use the existing terminalization/disk boundary to publish exactly one verdict from complete authoritative evidence. Force normal/backstop orders, cancellation orders, restart mid-collection and retried-shard ordering through public API.
5. Prove V1–V5 with hostile public-API tests: held optional probe flips before release; missing mandatory result keeps TESTING; normal/backstop races yield one verdict; artifacts/logs attach and optional failure is visible after verdict; each shard releases independently; cancellation/restart/retry preserve durable counts. Existing diagnostic test includes a backstop-only assertion that must be changed explicitly for the repaired contract while preserving its held-probe scenario.
6. Five consecutive first-attempt targeted passes and exact-head full local suite once (the user's direct brief overrides common.md's older no-full-local instruction), then ready PR with the production reason first. No merge, enqueue or deploy.

Commands and constraints

Fresh clone was made under this lane's workspace. Baseline test selector is e2eTerminalVerdictIsNotGatedOnHeapDumpProbe. Use explicit --local or --remote: the script defaults to racing both when omitted. Every build/test goes through scratchpad/locks/run-slot.sh; at most two lane invocations, with a full suite additionally globally exclusive. Never cancel a running gate or count lock wait against the90-minute box. Keep120000ms backstop grace and60s sweep unchanged; no timeout/iteration/assertion weakening. Source-only PR branch; investigation files belong on the linked WIP branch. Findings must be incremental, OBSERVED versus INFER, at least every5minutes, and end FINAL when stopping. Clean own caches/build outputs and verify no owned processes remain using ps -eo pid,args -ww.

Read before resuming: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/main/docs/UTILIZATION_ARCHITECTURE.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md, plus https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/STATE_MANAGEMENT.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md. The findings carry precise current source locations and the invariant table; source main may have moved since this snapshot.