Handoff: Finish CI review of the held-replay readiness probe

Written 2026-10-09 09:34 UTC.

RE-VERIFY: This is a write-time snapshot. Before acting, run gh pr view 1367 --repo CodexCoder21Organization/BuildTestEmbedded --json state,isDraft,headRefOid,statusCheckRollup,mergeStateStatus,mergedAt, inspect raw first-attempt XML from the CI run, and compare remote branch heads. A fresh query wins.

Mission

The user asked to carry out bteflake2: derive the readiness contract from test history, fix the false fixture premise in tests/vTJProbeAwaitReadinessWhileReplayHeld.kts with public observers, prove unchanged-product 10/10 first attempts and one product-only contract mutation failure, and open a draft PR. This is a small part of the utilization/startup-compaction campaign tracked by url://handoff/handoffs/hf-2026-08-24-fix-startup-compaction-delta-heap-exhaustion-then-revalidate-the-extreme-suite. Do not modify that larger handoff. No merge, enqueue, review-ready conversion, production deploy, pin change, timeout increase, workload decrease, sleeps, mocks, or reflection is authorized.

What was found and done

1. Verified the previous investigation rather than repeating its diagnostic: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-bteflake1-probe/handoff-artifacts/bteflake1/HANDOFF.md. The two original CI runs are https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37882967093 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37884015361. Aggregate success hid original first-attempt assertion failures. Raw first trace is preserved in evidence. The prior controlled premise probe failed 10/10 even while product full/page reads returned correct results.
2. Read test history and both introducing/correcting PRs: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/974 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/988. Contract written before editing: while startup reconstruction is held, construction and run/list metadata remain available, but a public result read with no committed snapshot waits for reconstruction publication and then both full and paginated reads return the journal's pending test.
3. Mechanism: old observer holds only reason startup recovery; resumed-run evidence rebuild uses after service restart and publishes the same snapshot before deciding that the test remains pending. The product is correct; the old test's snapshot-absence premise is false. No product fix was made.
4. Corrected fixture holds every same-run rebuild and separately observes both writers. It asserts snapshot absence before public read and after observed miss, and checks the full read and reader thread remain pending before release. Only one original assertion text changes: exact awaitStartupRecovery(250L) worker name from buildtest-startup-recovery to buildtest-startup-run-resume. This API joins run-resume first; holding its evidence child keeps it pending. Every other original assertion and every original workload/time budget remains unchanged. The page call stays sequential and verifies the published value; it does not independently observe a missing snapshot.
5. Initial broad gate failed all samples at the old expected worker name; initial mutation failed at the same unrelated diagnostic. These earn zero gate/control credit and are preserved honestly. Corrected fixture then passed 10/10; final reviewed candidate passed a separate 10/10. Raw XML, logs and exit codes are on evidence.
6. Generic product-only mutation: full getTestResults emits existing miss observation and returns [] if snapshot is absent, before reconstruction publication. Corrected test fails on pendingTest equality; final reviewed test fails directly on pre-release full-read completion. No fixture id is embedded in mutation product code. It exists only on the evidence/control branch.
7. Independent test-comprehensiveness and adversarial source review complete. Final review found no further gap in writer hold, exact diagnostic or pending-read checks; inherited page sequencing is stated accurately. README already describes the public contract, so no README edit was needed.
8. Main advanced from c8024fa363a4c250559680dbd8adcdca715fbd59 to 82d1c3376caa2e9edf7dc08eebebc6f36560ef24. Rebase clean. Incoming classification/storage changes do not touch the result handoff or resume/rebuild/await code, but startup setup traverses adjacent storage, so one healthy/control pair was checked on moved main. The ten-run gate retains credit under common-flake's amended rebase rule. Incoming hunks are saved in rebase-hunks.patch.
9. Draft PR opened, body re-read, all target proof complete. Four Actions shards still running at the latest snapshot in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37909662882. Full-suite gate not complete at this snapshot. No retry or CI rerun was requested.

Gates on actual kompile-cli 0.0.112

Relevant PRs / refs

All branches below are in CodexCoder21Organization/BuildTestEmbedded and verified by git ls-remote.

All supporting artifacts: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-bteflake2-evidence/handoff-artifacts/bteflake2. Older local tips are superseded versions of the saved candidate/mutation patches; no unique unfinished source remains local. Build outputs/caches are intentionally excluded. Nothing was deployed or published by this effort: only gh/git, read-only APIs, test Actions and handoff tooling were invoked; no production or publish command was run.

Next steps

1. Re-verify current state first with the command in the banner and inspect the current PR run. Keep it draft.
2. Watch plain PR CI with PATH=$HOME/bin:$PATH timeout 5400 coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/BuildTestEmbedded --pr 1367 --all-checks, as a tracked finite background task. Never use --to-merged here. Follow the shared API rule: no polling faster than 120 seconds.
3. Download the four bld-test-shard-*-test-results artifacts. Count first-attempt rows, not aggregate successes. If a first failure is in the changed test, reproduce the named mechanism before editing. Outside this change, catalogue full test name, stage and stack head with full XML; do not loosen tests or rerun hoping for green.
4. If CI is green, report READY FOR ORCHESTRATOR REVIEW with PR URL, exact head SHA and run URL. Leave the PR draft, unmerged, unenqueued. If a required gate is still pending/failing, report NOT READY with exact reason.
5. Do not restart a ten-run batch merely because main moves. Apply the amended rebase-credit rule with hunk evidence; intersecting paths require only the affected pair once. No launcher pin changes: 0.0.121 is not a gate while its upstream issue is being fixed.

Operational knowledge

• bld-test-shard jobs execute --local on Actions machines. They are independent of the buildtest coordinator. Multi-test gates used the finite wrapper preserved as gha-gate.sh, derived from supplied tools/gha-run.sh.
• The watchman's first poll identified the known kotlin-build-ci-test app non-dispatch: current and sampled main suites queued with zero runs, including mature main evidence. It reports INFO and does not rerequest; do not investigate the production service in this lane.
• Public observer beforeTestResultProjectionRebuild fires before publication lock. Startup run-resume coordinator waits its child resume boundary; that boundary completes after held evidence rebuilding. awaitStartupRecovery joins that coordinator first. These source facts justify the changed exact diagnostic.
• Full/page result APIs share prepareTestResultsForPublicRead and TestResultsReadHandoff; the current test still calls page only after full returns.
• PR bodies should be changed through gh api PATCH with -F body=@file, then re-read. gh pr edit is known unreliable here.
• Coursier: use ~/bin/coursier. No local builds were run, so no lane-specific ~/.aibuildcaches entry was created.
• The full Testing Architecture and Engineering Philosophy documents were read; findings records links and invariant table. Handoff README's legacy merge exception is superseded by current central-service skill and the explicit no-merge instruction.

Central handoff created successfully: url://handoff/handoffs/hf-2026-10-09-finish-full-suite-ci-review-for-the-held-replay-readiness-probe (https://www.handoff.wasmserver.com/handoffs/hf-2026-10-09-finish-full-suite-ci-review-for-the-held-replay-readiness-probe). The service is authoritative; this branch copy also preserves the full continuation if its mirror lags.

Final snapshot 09:34 UTC: OPEN/draft at the same head, mergeStateStatus UNKNOWN; four Actions shards IN_PROGRESS. NOT READY until full-suite first-attempt results are reviewed. The follow-up claim command failed before service-bytecode fetch completed, exit 124; full trace is in handoff-claim.log. No claim success was observed. The finite watcher is stopped, own checkouts are removed after the final remote checkpoint, and no local background task is left to continue the work. CI on GitHub remains running normally.