Handoff: Final-review and merge the verified C8 candidate

Written 2026-10-02 21:17 UTC. RE-VERIFY: This is a write-time snapshot. Check the current PR head/state/checks and current main before acting. The fresh October 2 verification was OPEN/CLEAN at d2f274f841d45b6779c16c67281216f6b3582f39, all five checks SUCCESS, and current main f3efa847c6e78233cd006f42528bd6534a578c26.

Delegate work complete — 2026-10-02 21:17 UTC

Remaining work: supervisor final review and merge decision for https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041, then supervisor handoff completion. Do not replay the obsolete September 28 candidate. This delegate never merged, enqueued, deployed, published an artifact or completed the handoff.

OBSERVED: The newer September 30 candidate was rebased onto current main, preserving CPU reservations and logical-attempt identity together, XML execution-ID normalization with typed failure metadata, and retry-ledger eviction through centralized run cleanup. Published PR head is d2f274f841d45b6779c16c67281216f6b3582f39; git ls-remote matches. Fresh PR state is OPEN/CLEAN, mergedAt=null, with all four shard checks and required bld-build COMPLETED/SUCCESS.

OBSERVED: Local tests passed 7/7, 7/7 and 2/2 across the initial proof, required pre-product-push gate, and final evidence pre-push gate: 16 passed, 0 failed. Every local invocation used the shared slot helper. The last two-proof batch followed fresh fetch/rebase with unchanged main and source, and passed F2 in 20,440 ms and F5 in 6,213 ms. No test assertion, iteration count, timeout or deadline changed. Test-comprehensiveness and code-review passes found no remaining integration issue.

OBSERVED: Exactly one new-head CI run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37061521724 completed SUCCESS. Build-watchman 0.0.21 returned GREEN exit 0 without merge mode. XML counts: shard 1 836/0, shard 2 839/0, shard 3 838/0, shard 4 844/0, for 3,357 passed, 0 failed. F2's combined three-order source/PASS/restart proof passed in shard 4 in 28,524 ms; F5's runner-loss budget proof passed in shard 1 in 9,729 ms. The workflow verified every test owned by each shard executed and verified the complete four-shard partition. No rerequest, rerun or additional remote run occurred.

Evidence: candidate proof, current CI proof with artifact links, findings, final report. The PR proof table and validation section were updated to exact current-head results while preserving historical evidence.

OBSERVED: Both local tests and the owned watcher exited 0. No remaining owned test/watch processes, subagents, or dirty working tree were found. Raw XML/zip/build/process outputs are outside source control. A first-batch non-failing terminal-memory output-order warning is preserved in tooling-warning.md; later batches did not repeat it. Report-challenge was not invoked because its automatic enqueue/merge conflicts with this lane's hard no-merge rule; supervisor can file the preserved draft under its authority.

INFER: The delegate mission is complete. The initially obsolete handoff premises were corrected; old green CI alone did not satisfy current-main integration, so the lane completed that integration and proved the rebased candidate locally and in its single current-head CI run. All earlier pending-work and initial completion sections below are historical and superseded by this section.

Prior handoff history — retained in full

Handoff: Watch the rebased C8 candidate shards, then final-review the existing PR

Written 2026-10-02 20:18 UTC. RE-VERIFY: This is a write-time snapshot. Check the current PR head/state/checks and current main before acting. The September 28 candidate and provisioning claims below are historical; the newer September 30 PR head already passed both proof tests and all four shards.

Pushed current-main candidate — 2026-10-02 20:37 UTC

RE-VERIFY: Check https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 head/state/checks and https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37061521724 before acting. This section supersedes earlier pending local/push and delegate-complete statements while preserving their history.

OBSERVED: Published PR head is now d2f274f841d45b6779c16c67281216f6b3582f39, rebased onto main f3efa847c6e78233cd006f42528bd6534a578c26. Immediate pre-push PR read was OPEN with no unaccounted recent commit/comment. git ls-remote confirms the PR branch SHA. Both seven-selector local batches exited 0, for 14 passed / 0 failed. F2 times: 22220 and 22180 ms; F5: 7398 and 7763 ms. The required pre-push repetition followed a fresh fetch/rebase that left this exact candidate unchanged.

OBSERVED: Durable checkpoint https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte-c8f2-2026-10-02 is 37fcfcae8f4eae63cde95cf379457f74672bf610, also verified with git ls-remote. It carries the rebased product commits plus curated notes. Its source/test/docs/script trees were checked identical to the product candidate. Candidate proof records selectors, durations and review passes. Tooling warning preserves the one non-failing terminal-memory output-order warning and full trace. The report-challenge CLI was not used because its automatic enqueue/merge conflicts with this lane's explicit no-merge rule.

OBSERVED: Exactly one new-head CI run was launched by the one PR branch push: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37061521724, IN_PROGRESS. One build-watchman 0.0.21 plain --pr watcher is active on this head; four shards are pending/running and bld-build has not reported yet. No rerequest, rerun, enqueue, merge, deploy, restart, artifact publication or handoff completion occurred. The PR C8-F2/F5 table and current-main validation now link the local proof and explicitly keep new-head CI pending.

Current next steps: watch this single CI run to its verdict. If green, refresh the body/table with exact new-head results and leave supervisor final review/merge handling. If red, reproduce the named failures locally before any code changes; do not use CI as diagnosis or repeat the run blindly. At the lane cutoff, record the run/check URL and stop the owned watcher so no orphan remains. Never merge, enqueue, deploy or complete this handoff.

Current-main rebase correction — 2026-10-02 20:24 UTC

OBSERVED: The final PR read reports mergeStateStatus=DIRTY. The earlier claim that only supervisor review/merge remained was too broad: this lane explicitly owns rebasing onto current main. September 30 proof gates are green, but that evidence does not complete the current-main gate. These observations supersede the initial completion wording below.

OBSERVED: Rebasing the newer ccdbf781 PR head onto main f3efa847 required conflicts in BuildDriver.kt, BuildTestEmbeddedService.kt, and LeaseScheduler.kt. Resolved by retaining current-main XML execution identity while carrying the typed failure fields, adding retry-ledger eviction to main's centralized clearDeletedRunCaches helper, and retaining both CPU reservations and logical-attempt identity plus both scheduler helper families. Rebased candidate is locally d2f274f841d45b6779c16c67281216f6b3582f39. No product branch has been pushed yet; no test assertion, timeout, iteration count or test selection contract was changed.

OBSERVED: One local seven-selector proof batch started at 20:23 UTC through the shared slot helper and acquired slot 2 immediately. Selectors: workspaceSourceFailureOverridesTailAccelerationPass, e2eLiveRunnerLossSpendsNoBuildBudget, xmlBuildRuleMergeAfterRestartKeepsOneRowPerExecution, cpuAdmissionBoundsTailAccelerationDuplicates, failureBudgetRecoveryBuildCopiesShareLogicalAttempt, e2eDeleteBuildRunCleansUpDroplet, e2eRegistryBuildFailureRetriesThenSucceeds. No results yet. This is the lane's first test execution. No new CI/remote run was submitted.

Current next steps: verify this local candidate; re-check for another active session; push a passing candidate to the existing PR and the lane WIP checkpoint; refresh the C8 proof table with exact-head local results; watch one new-head CI run. If the local compile hits its unchanged 20-minute deadline, preserve the exact error/module and stop that step as the lane requires. Supervisor still owns merge/enqueue/deployment and handoff completion.

Initial exact-head verification — bte-c8f2 (2026-10-02)

OBSERVED: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 remains OPEN, not merged, at ccdbf781af9a7f9c7b3527c2fb460bf703f9d391. Its newest commit is dated 2026-09-30 06:40:37 UTC and its newest comment 2026-09-23, so no unaccounted two-hour PR activity was present. The current PR body and commit history show the corrected c25 candidate fixtures were incorporated in the September 30 squash, followed by projection-feed compaction and dispatch-scan fixes. The three copied single-order tests were superseded by workspaceSourceFailureOverridesTailAccelerationPass, which source inspection confirmed runs every ordering and collects any scenario failures.

OBSERVED: The four XML test-result artifacts from https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/36679496907 record shard pass/fail counts of 780/0, 783/0, 782/0, and 788/0: 3133 passed, 0 failed. F2's combined proof PASSED in 43165 ms in shard 3 XML. F5 PASSED in 9401 ms in shard 4 XML. All four shard checks and bld-build are SUCCESS. One read-only build-watchman 0.0.21 watch returned RESULT: GREEN exit=0 for that exact head. It classified kotlin-build-ci-test as installed but non-dispatching on both current and sampled main commits; no check suite was rerequested.

INFER: The handoff's candidate-not-pushed and shards-unverified premises are obsolete. A fresh run or code push of historical candidate 490686308 would repeat completed proof work and risk discarding subsequent fixes, so this lane did neither. The original fail-first proof links and F5 refutation are carried forward below and in the PR. The PR's C8-F2 and C8-F5 table cells now link the passing current-head artifacts, and its validation text carries the full shard counts and this handoff link.

OBSERVED: Main is f3efa847c6e78233cd006f42528bd6534a578c26, 40 commits after the common base 04301a1b81371f20f45a8b301927a0bee6511191; the PR has 3 commits after that base. Existing green CI proves the recorded PR head, not a future rebase. The supervisor must assess current-main integration during final review and merge handling. No local test was run in this lane: 0 passed, 0 failed, 0 executed. No new remote run or CI rerun was submitted. No product/test code changed. No merge, enqueue, deploy, restart, handoff completion, or Maven publication occurred.

OBSERVED: A fresh historical-run read at /api/runs/8a0d1ea7 returned HTTP 404, so this lane cannot freshly confirm that run's old infrastructure outcome. Its prior log/evidence stays on the existing c25 proof-evidence branch. This does not affect the newer passing GitHub artifacts.

Evidence checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte-c8f2-2026-10-02/handoff-artifacts/bte-c8f2 at 1569dd85959c44e0fc11975b3eb8e913e34b8411, verified with git ls-remote. It contains a curated verification note only. The code PR remains untouched.

The following September 28 sections preserve the investigation chain and historical refs; the current verification above supersedes their pending-gate claims.

Mission

The user requested: “finish the C8-F2 and C8-F5 proofs” on https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041, then push the candidate, update the C8 table with proof links, and watch all shards green. Lane c25 has a 120-minute limit ending 17:29 UTC. Never merge, enqueue, deploy, raise waits, reduce iterations, bypass CI, or re-run CI to diagnose a failure. The remaining gates are candidate confirmation, PR branch push/body update, and all new-head shards green.

Findings and completed work

1. F2's missing first lease was a fixture startup error, not product evidence. Named diagnostic https://buildtest.kotlin.build/run?id=080af8f6 failed at the first-lease assertion with getBuildLog included. The two-shard fixture deleted the source archive while a follower still needed to fetch it; curl returned 52. Its broad compiler-command interceptor also treated the shared-cache build-only child as a dynamic runner and required a work-queue argument it never has. The known-working e2eSourceDefectFailsOnlyDependentProjectTests fixture has one shard and bypasses both operations. Earlier c18 runs c29ec935, d2f333a3, and 2fa20e65 are setup evidence only; do not repeat their investigation.
2. The corrected fixture lets the real shell retain/extract the archive and execute preparation with an executable successful fixture compiler. A later real queue run https://buildtest.kotlin.build/run?id=a769ade2 exposed inconsistent path matching: BuildDriver prefixes file selectors with the remote projects directory. All matching now normalizes transported paths. Public RUNNING plus four helper PASS rows gate clock advance; the existing durable workspace_unbuildable decision orders source ingestion on both baseline and candidate. The duplicate PASS must reach public attempt history before assertions execute. Both live orders assert FAILED and the exact source error before finalization can hide a wrong PASS.
3. Clean F2 fail-first proof is COMPLETE on the F2-product-free branch at 5419e95cbaa8e402d3f159bd96e852777666d3bb: https://buildtest.kotlin.build/run?id=dcf55132. sourceFailureBeforeTailPass failed in 34,660 ms (expected FAILED, captured pre-PASS row RUNNING, duplicate PASS already observed); sourceFailureAfterTailPass failed in 33,540 ms (expected FAILED, live row PASSED); sourceFailureAfterTailPassRestart failed in 33,637 ms (expected FAILED, recovered row PASSED before dispatch). These are post-seam assertion failures, not timeouts. The product mechanism is missing durable per-identity source precedence and missing reconstruction before restart dispatch. The supplied candidate already contains that product fix; c25 changed only tests.
4. F5's missing-product-behavior premise is REFUTED. e2eLiveRunnerLossSpendsNoBuildBudget PASSED in 24,821 ms in dcf55132. The baseline already carries the PR ledger behavior: public running attempt → PID loss without completion/exit → re-lease → exactly three real failed build attempts → exhaustion naming those three, excluding the loss from failed ledger entries. The old test wrongly expected a dynamic_row_attempt_authority failure record for the loss; production instead writes a CANCELED completion with dispatchAttemptAbandoned=true. The corrected pin checks this public canceled history, no counted-failure authority for the loss, exact exhaustion text/IDs, and the failed-ledger set. Issued-attempt attribution remains in the ledger by design; only the failed-attempt set excludes the loss.
5. Candidate 490686308c4d3cc70106f039bbc9414a3a576c71 contains identical proof files and the supplied product fix. Its proof run is https://buildtest.kotlin.build/run?id=8a0d1ea7, selecting the three named F2 scenarios, F5, and the original aggregate workspaceSourceFailureOverridesTailAccelerationPass. Re-check this run first. It is now terminal FAILED before tests: cleanup-before-requeue could not be verified because IllegalStateException: Job was cancelled. The run API confirms zero passed and zero failed tests. It reached provisioning at 17:04:00, but at 17:04:51 the persistent RPC response for getDropletCreationStatus on digitalocean-droplets ended with EOF while pending. The coordinator requeued the same run; no test executed. The second provisioning attempt began at 17:13:36 and failed at 17:14:28 with the same status-RPC EOF; the coordinator again requeued the same run. A third provisioning attempt began at 17:17:32 and failed at 17:18:03 because createDropletAsync stalled for 30,000 ms. It was requeued again. No candidate test has executed. This is infrastructure evidence, not a product/test result. The exact error is preserved as provisioning-challenge.md and candidate-run-log.txt on the evidence branch. Do not duplicate it.
6. Read-only code review found no additional scoped product blocker. Independent test review requested a live public-result check after duplicate PASS; it is implemented in both branches. Review and full failure evidence are preserved in the evidence branch linked below.
7. The PR body was updated at 17:08 UTC using the required PATCH API. It now contains the clean baseline links, F5 refutation, and explicit pending candidate/new-head gates; the candidate code has not been pushed to the PR branch. The current published body is preserved as c25-pr-body-published.md on the evidence branch.
8. The old PR head 0b2940fb7f249371014b6a81d7ef3488000184e4 remained OPEN at 16:43. Its old CI https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/36441759271 had three passing shards and shard 1 at 734/736. Its only failures were the two fixture defects above (F2 aggregate startup and F5 missing private authority row). Do not re-run that old CI; the fixes must go on the PR branch after candidate confirmation.

Relevant PRs / refs

| BuildTestEmbedded | wip/bte-c8f2-2026-10-02 | 1569dd85959c44e0fc11975b3eb8e913e34b8411 | no PR | Current proof/shard verification note; evidence only | Pushed; remote SHA verified | | BuildTestEmbedded | fix/unbuildable-workspace-classification (September 30 snapshot) | ccdbf781af9a7f9c7b3527c2fb460bf703f9d391 | target PR | September 30 squash of product changes/corrected candidate fixtures plus later fixes | OPEN; F2/F5 and all four shards passed; final review/merge remain |

| BuildTestEmbedded | fix/unbuildable-workspace-classification (October 2 current) | d2f274f841d45b6779c16c67281216f6b3582f39 | target PR | Current-main rebase with both conflict contracts retained | Local 14/14 PASS; CI 37061521724 IN_PROGRESS | | BuildTestEmbedded | wip/bte-c8f2-2026-10-02 (candidate checkpoint) | 37fcfcae8f4eae63cde95cf379457f74672bf610 | no PR | Rebased candidate plus proof/findings/tooling evidence; do not merge evidence files into PR | Pushed and remote SHA verified; source trees identical to candidate |

No production service was deployed/restarted and no Maven artifact was published in this lane. Operations were source inspection, test submissions, GitHub read operations, and WIP branch pushes. No PR was merged or enqueued. The challenge CLI was not used because it automatically merges; the provisioning challenge is preserved on the evidence branch for later authorized filing.

Historical next steps (superseded by current verification)

1. Re-verify PR state/head and run 8a0d1ea7. Use gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 --json state,headRefOid,statusCheckRollup,mergedAt. Use the run API/raw log and the saved c25-poll-proofs.py collector. Raw logs are /log/raw?id=<id>; result API supports page=1&limit=500 and returns totalPages. The default unpaginated results omit later alphabetical names; do not mistake that for no results.
2. The existing candidate run failed before tests. Diagnose or verify resolution of the provisioning/cleanup condition before intentionally submitting one new candidate proof run with the same five selectors. Do not blindly repeat it. If that candidate passes all five selectors, do not launch another verification batch. Fresh-fetch/rebase origin/main, confirm the PR remains OPEN, and push the candidate to fix/unbuildable-workspace-classification (normal fast-forward expected). The code checkout was fresh under c25/workspace/BuildTestEmbedded; clone fresh in your own workspace if resuming elsewhere.
3. Complete the draft C8 table/body from the evidence branch with the candidate link/results and current-head CI link. Body changes MUST use gh api -X PATCH repos/CodexCoder21Organization/BuildTestEmbedded/pulls/1041 -F body=@/absolute/body.md. Preserve the two opening motivation sentences and end with 🤖 Generated with [Claude Code](https://claude.com/claude-code).
4. Watch checks at 120-second cadence, up to 45 minutes, with build-watchman 0.0.18 and no --to-merged. Never merge or enqueue. Investigate failures locally from evidence; never use CI reruns as a debugging loop. Old CI failures were exactly the corrected fixture cases, but new-head results must still be observed.
5. If candidate fails, classify the exact seam before editing. Do not count setup errors or timeouts as product evidence. Fix the mechanism, keep waits and iteration counts unchanged, preserve parity between final baseline and candidate proof tests, and checkpoint every milestone.
6. Update the c25 findings/this handoff with final outcomes. Definition of done: F2 fail-first+green links for both orders and restart, F5 refutation+candidate green, new-head shards green, C8 table updated. Stop without merging/deploying.

Operational details

Export PATH=/home/u/.local/bin:$PATH; its coursier 2.1.25 works, while /usr/local/bin/coursier is the wrong architecture. Prefer scripts/test.bash --remote --test <selector>; at most one local JVM for this lane, and local runs must use env -u JAVA_TOOL_OPTIONS. Never run a whole suite locally on the shared host. Toolchain pin 0.0.112 supports actual local source edits. Both final branches were fetched/rebased before their submissions; origin/main was adcfc309c then. The c25 candidate wrapper was deliberately detached by stopping only its local PID330343 with SIGKILL, because its RemoteBuildWorkspace shutdown hook sends remote cancellation. The durable run survived and continued provisioning; no cancellation request or duplicate was sent. Session1209 is no longer active. Observe the existing remote run through build-watchman or the HTTP APIs.

Required reading: build-failure invariants, utilization contract, Testing Architecture, Engineering Philosophy. No repository AGENTS.md existed. Read README after cloning.

Commits end with Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>. Never message the supervisor; append state to the lane findings. Do not introduce a production deploy or artifact publish. No timeout was raised, iteration reduced, test disabled/deleted, mock/reflection added, or branch protection bypassed.

Historical September 28 final status: baseline proofs and C8 body update are complete; candidate green, PR code push, and new-head shards remain blocked. The final status report and published body are preserved in the evidence branch.

Initial next steps (superseded by current-main rebase below)

1. Re-verify https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 state/head/checks and current main. The delegate proof and shard work is already complete at the head recorded above; do not replay the obsolete candidate.
2. Supervisor: perform final review, including integration with current main and any resulting rebase/merge-queue verification, then make the merge decision. This delegate did not merge or enqueue.
3. Supervisor: update or complete this handoff after its own review and merge handling. No production deployment is authorized by this handoff.

Reading in this lane: handoff-triage README, repository README context, docs/BUILD_FAILURE_INVARIANTS.md, docs/UTILIZATION_ARCHITECTURE.md, Testing Architecture, Engineering Philosophy, and the exact F2/F5 test source and result artifacts. Testing Architecture's rule that a red kotlin.build (remote) check cannot be accepted remains intact: no such red check was observed on the current PR; the app has not dispatched checks on sampled main heads either. The lane's explicit no-rerun/no-merge limits govern this verification.