Finish round 2 of the queued-retry status correction

Written 2026-10-11 12:17 UTC during stuckrun3.

RE-VERIFY: This is a write-time snapshot. Check https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1427 with gh pr view --json state,isDraft,headRefOid,mergeStateStatus. Compare all branch heads with git ls-remote. Latest round-2 findings and test XMLs are on the linked evidence branch.

Mission summary

Finish A–E of the round-2 brief: the generated reference-model mismatch, late-authority work bound, migrated-name publication, warm-row clock-offset replay, and terminal queued-retry failure retention including saved-cursor feed. The owner authorized only the original five status-literal edits plus precisely scoped generated MODEL data updates for the requested RUNNING to PENDING correction. No other existing assertions, counts, seed ranges or work bounds may change. Fresh-before-retry scheduling stays. No enqueue, merge, deploy or publish. Mark this PR ready only after all required gates pass.

What was found and done

• The provided CI failure list contains 12 distinct names, despite its stated total of 13. The union baseline test was included as a thirteenth gate. All 13 passed first attempt in stuckrun3-fixed-1.xml and fixed-2.xml, with no earlier history for those names.
• A: The frozen generated model is SHA-based, not a separately checked-in reference implementation. Captured all 8000 seeded observations plus six reduced traces against original main projection and the corrected product. Comparison found 7732 identical captures, 274 changed, zero unclassified. All differences are allowed RUNNING to PENDING transitions and linked first-authority row publications; histories/messages/counts remain identical. The comparator rejects reverse status, history/count/message edits, extra observations/publications, and observed live same-test siblings. Optional running-history evidence exists for 868 changes; 2043 observations omit it, so that subset additionally relies on the explicit source predicate and dedicated live-sibling cases. Never present the optional trace as proof it does not contain. Updated only 274 digest rows in eight fixture TSVs, without seed/count/bound changes. Model statement for the PR, verbatim: “When a recorded accepted FAILED attempt has explicit deferred authority and no accepted attempt remains live, its canonical lifecycle status is PENDING rather than RUNNING. Applying the first such authority returns the newly PENDING row. No other observation changes.”
• B: false authority previously normalized the selected PASS and rebuilt its growing failure warning every time. Gate publication to newly deferred accepted FAILED rows only, leaving PASS warning materialization lazy. The original <=3x work bound passes untouched.
• C: false authority on an already-PENDING or passed migrated row incorrectly returned that row. The same gated publication fixes it; original null assertion passes untouched.
• D: replay eagerly normalized an incomplete clock-stream boundary before its offset arrived. Replay now skips the unused live publication, leaving normalization after the complete event prefix. Original warm-fold assertions pass untouched.
• E: terminal finalization mistook normalized queued PENDING for never-run and wrote a new synthetic completion, replacing the first FAILED attempt. It now appends true authority selecting the first eligible completed failure, preserving message/output/duration/boundaries/history and failure-budget metadata; donor failures with failureBudgetCounted=false are skipped. Started retry cancellation keeps its original CANCELED behavior. Existing provisioning not-run disposition remains preserved. Six deterministic terminal/control probes passed first attempt; saved-cursor initially had no FAILED publication, exposing the need for a durable terminal authority callback. That callback now publishes FAILED after append/apply/exact fingerprint. The feed then saw RUNNING,PENDING,FAILED because initial queued RUNNING publication lagged the initial saved cursor read; the new harness now explicitly observes that initial RUNNING before releasing the failed test. The initial feed also contains TEST_STATUSES_REPLACE baselines, which the harness now consumes according to the public contract. Buffered initial RUNNING rows can still follow that baseline, so the newly authored probe now checks the exact requested terminal sequence from first PENDING onward, rejecting any subsequent RUNNING or duplicate PENDING/FAILED. Full terminal-feed error/duration and final/history output/timing equality assertions were added after independent test review. This last revision passed first attempt in family batch4, including exact terminal feed error/duration and canonical/history output/timing.
• Independent product review found a second startup issue: preserveTestsNotRun gates only the later result loop, while the earlier outstanding-attempt loop still synthesizes a counted FAILED completion. The precise new public recovery test persists INFRASTRUCTURE/DYNAMIC_COMPLETION_INGESTION + discovered + started and no completion; expects preserved RUNNING, zero failed/passed counts, empty completed-attempt history, startup plus reopen. The public regression failed as expected (RUNNING versus FAILED), then adding !preserveTestsNotRun to that outstanding loop made it pass first attempt in fixed-3.xml. The original not-run recovery contract is retained.
• All three local build slots were occupied during the new recovery/feed gate. That targeted task eventually acquired slot3 after roughly nine minutes. A corrected-source 21-selector batch then passed20/21 first attempts; the only remaining failure was the saved-cursor harness boundary. No remote submission has yet been used. Do not remove another lane’s slots.

Relevant PRs / refs

This round deployed or published nothing. No production host was accessed.

Next steps

1. Fresh clone into workspace/<lane>/BuildTestEmbedded. Checkout PR branch, fetch/rebase origin/main. Apply evidence round2.patch; copy added-tests/*.kts into tests/ (cancellation file already exists). Saved helper scripts have scratchpad absolute paths: adapt their root or issue equivalent commands from your clone.
2. Preserve the proven recovery guard and corrected feed protocol boundary; retain the proven saved-cursor revision and run the 13 original gate names plus eight new terminal/control tests, first attempts only. Full earlier-attempt histories are siblings of test rows in XML.
3. Source-only mutation must make the targeted terminal tests red again. Complete common.md 3/3 first-attempt requirement; do not infer it from final retry rows.
4. Latest checkpoint: batch4 passed81/81 first-attempt rows across80 selected files; batch3 stopped before XML; other batches unrun. No remote submitted. All owned family process-tree tasks stopped. Required local family manifest has 343 original file selectors plus seven additional untracked test files (cancellation already part of main manifest), totaling 350. Includes all 90 round-1 tests. Run ALL as four batches of at most 100 selectors with distinct XMLs, under build slots and bounded tasks. Sum the first-attempt XML counts; unfinished batch is not a pass.
5. Use ONE remote submission of <=12 selectors (eight new terminal/control cases plus B,C,D,SharedAlias). Keep its ID and watch via build-watchman; no CI watch after final push. Cancel any submitted run that has not started TESTING before stopping.
6. Independent final test and product review; all review findings done or evidence-refuted. Rebase, run exact-head required gates, verify PR still OPEN, commit/push source+tests+README+authorized fixture data only. All investigation files stay on evidence branch.
7. Update PR body with Round 2 A–E paragraphs and quote model edit verbatim; all existing-test edits explicitly listed. Mark ready only if every gate passed. No enqueue/merge/deploy/publish. Stop owned tasks, confirm ps -eo pid,args -ww, and give the brief’s exact READY or NOT READY line.

Original round-1 history

Handoff: Finish the corrected queued-retry status PR's final remote gate

Written 2026-10-11 10:11 UTC.

RE-VERIFY: This is a write-time snapshot. Re-check the PR with gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1427 --json state,isDraft,headRefOid,mergeStateStatus,mergedAt; compare branch heads with git ls-remote. Read the 09:16 ruling in the linked findings before changing scheduling.

Mission summary

The owner asked why run https://buildtest.kotlin.build/run?id=11261b57 showed 19 RUNNING tests with only four surviving lanes, including 17 whose only attempt had already failed and whose retry top-up had waited through fifty-plus fresh leases. The SSH owner-loss issue is owned elsewhere and was excluded. The later 09:16 orchestrator ruling explicitly authorized correcting queued status and related old RUNNING assertions, but preserved fresh-before-retry scheduling. This lane's request was one final remote submission, no merge/enqueue/deployment/publishing, and a 70-minute limit starting around 09:08 UTC.

What was found and done

1. Proven status mechanism: a dispatch-gated failed completion retains the raw RUNNING row from test_started; false authority defers its final verdict, and the old normalization adds endedAt without correcting status. The fix uses existing PENDING when a completed failure has explicitly deferred authority and no accepted live sibling. It preserves RUNNING before authority arrives and while a sibling is live. Late false authority returns the normalized row so the existing service feed enqueue guard can publish it. Completed physical intervals still contribute utilization even though the row is PENDING.
2. Proven queue mechanism and refutations: retry demand is appended to the run-wide queue behind the first-attempt pool. That explains the production wait and is existing policy. Preference checks only defer when an admissible alternative exists; the failed-on sole survivor is already permitted once fresh work drains. Top-ups are shared queue entries, not owned by the later-lost droplets. Nothing in the inspected mechanisms would permanently strand these seventeen completed failures after the finite fresh pool drains. No LeaseScheduler change remains.
3. Deterministic final fixtures: two public scheduler/service fixtures fail A, lose A or its alternative, assert PENDING, complete all twelve fresh specs in exact lexical order, then lease the retry to the sole survivor and verify failed-plus-passed history. The status fixture uses a real DynamicDispatchOrchestrator plus FakeDropletService/ManualClock, deterministic resident-memory admission, actual queued decision and authority records, service result/history/feed snapshots and cold replay, then fresh completion and retry. A public RunEventProjection fixture verifies the authority return row with and without a live sibling. No sleeps, reflection, mocks, timeout increases or count reductions.
4. Evidence: final source-only mutation reverts just RunEventProjection and RunUtilization while retaining final fixtures: all four fail first attempts at PENDING-vs-RUNNING or missing publication, and fail again on runner retries. Restoring the fix gives all four 3/3 first-attempt passes. Eight focused neighbors also pass 3/3. The entire affected family passes 90/90 first attempts, zero attempt-history entries. All timestamps/utilization arithmetic in the five old tests remain unchanged.
5. Remote limitation: the sole permitted submission https://buildtest.kotlin.build/run?id=fe4d0c7b contains the pre-correction numeric fresh-spec order. It finished failed, CLI summary 10/12; public results confirm the two survivor fixtures expected fresh-2 but received fresh-10, and ten passed. Final corrected lexical expectations are green locally. The failed remote invocation produced no requested XML; public rows do not prove first-attempt provenance. It is not a final-head gate. An additional submission was requested at 10:10 UTC but no answer was available at writing. No extra submission was made and no remote runs remain active.
6. An earlier retry-first experiment was rejected by the 09:16 ruling. Its source patch and held-priority fixture are not on the product branch; the rejected fixture is preserved only as evidence. The initial failing checkpoint contains that earlier assumption and is superseded. Do not restore retry-first ordering or modify reLeaseIsQueuedBehindFreshWork.

Relevant PRs / refs

Initial reproducer checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/6ffb833cf2d7f52a6170539c261a26624286a424. Final fix: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/31a0dabada3af2ab22d01007251bde8211c4deb2.

Deployed or published but not merged: none. This lane issued no deploy, restart, publish, enqueue or merge operation. Product code exists only on the remote branch and draft PR; mergedAt is null. The clone is clean, has one worktree, no stash, and no unpushed local commits. Build outputs and dependency caches are excluded; logs with full process arguments are intentionally not uploaded.

Next steps

1. Re-verify current state first; fetch the product and evidence branches and read README, required Testing Architecture and Utilization Architecture dispatch/retry sections, common.md, and findings' 09:16 ruling.
2. Resolve the pending question: authorize a new twelve-selector remote gate for the final corrected head, or have the orchestrator take it in a new round. The original lane's one submission is already spent; do not silently add a second.
3. If authorized, run LC_ALL=C.UTF-8 scripts/test.bash --remote with the twelve selectors in evidence/stuckrun1-final-selectors.txt and --log XML. Confirm first attempts only; a completed run without rows or XML is not a pass. Use watch-build/build-watchman 0.0.21 for the run once its ID is known. Keep at most two remote runs in flight, at most six submissions per round, and the task's stricter authorization applies.
4. Reconcile the new gate into the PR body using gh api -X PATCH ... -F body=@/absolute/file.md, then re-read it. The PR already ends with the required Claude Code attribution; every commit ends with the required Co-Authored-By footer.
5. The local 3/3 and family gates are met; do not repeat additional verification batches without a new change or concern. The orchestrator owns CI watching after push. No merge, enqueue, marking ready, deployment or publishing is authorized by this lane.

Reusable / operational knowledge

• Evidence files: stuckrun1-corrected-round2/3/4.xml each 12/12, stuckrun1-corrected-family.xml 90/90, stuckrun1-corrected-mutation.xml 0/4, test-review.md and code-review.md. The corrected source patch is final-status-fix.patch; product commit is authoritative.
• The 90 family selectors are in stuckrun1-run-corrected-family.sh: dynamicDispatch11, failureBudget13, reLease15, runUtilization43, testFailedCompletion1, testRetryPreference1, heavyHead4, inRunReLease2.
• Five existing status-only updates: runUtilizationEndsFailedAttemptWhileRetryIsPending, runUtilizationPendingRetrySurvivesRestart, runUtilizationClosesPendingReplacementOnAnotherMachine, runUtilizationClosesTwoSupersededAttemptsInSequence, runUtilizationSupersededSequenceAcrossCoordinatorRestart. Every assertion change is RUNNING -> PENDING; all numeric assertions remain.
• Service feed snapshots and cold replay are tested; a saved-RUNNING-cursor incremental service delta is not directly covered. The public projection returned row is tested, and the service's non-null enqueue guard was independently inspected. The fixture uses public processRunnerEvent and journals orchestrator decisions; it does not use the full SSH runner.
• Local builds require one of the three shared build slots, bounded ten-minute acquisition, immediate release on exit, LC_ALL=C.UTF-8 and at least six GB free. Never alter another lane's slot. A slot wait ended without a test verdict and was not counted as a pass.
• Working checkout was workspace/stuckrun1/BuildTestEmbedded; use a fresh workspace clone on pickup. Temporary fixtures clean up resources; cached build outputs remain outside the source diff. The final ps -eo pid,args -ww audit found no owned test/build workers remaining.