Verify helper bytecode and qualify coordinator-to-site gates

RE-VERIFY — written 2026-10-09 16:26 UTC. This is a snapshot. Re-fetch live handoff/claims, query each linked PR state/head/checks, fetch main and verify remote branches before acting. Fresh evidence wins. Do not treat retry summaries or old helper output as proof of a correction.

Mission summary

The owner asked for a hermetic end-to-end reproduction of the Out of date notice on https://buildtest.kotlin.build using the real coordinator and site, then a fix of the underlying cause. Preserve that boundary: real coordinator journal, real RPC and served client, private transport, durable site SQLite and HTTP. The lane owns this handoff only; the supervisor owns every merge/enqueue/deploy/publish/close/completion decision.

What was found and done

1. Claim fable-hq-20261004 is retained. Live get succeeded; accepted report14:44:04.419UTC and accepted update14:32 replaced the stale October5 body. Original supplied snapshot path is absent; full live JSON and run1 evidence are preserved. Never release the claim at the end of the current continuation.

2. Old engine snapshot-hold remedy is superseded, not an accepted fix: its PR is CLOSED, not merged. Current store and fragments landed, but fresh engine main 6213c7ae133ebec99bcf4cd36748e42aad6ed906 still does not select the current store in its root service. The opt-in module README explicitly states that. Root wiring and site reader remain other-owner OPEN/red PRs; do not edit/copy their product work. A manually driven store facade is not the requested wired coordinator/site proof.

3. WUI cost repair and durable replica landed on main. The owned real-service draft uniquely supplies the coordinator/RPC/SQLite/HTTP scenarios and must be retained/salvaged rather than discarded with the retired repair. Its head remains daf3d4098dd51c4b6effd1df16db8045c26e81be, OPEN draft and bld-build FAILURE. Metadata alone was corrected14:54 to remove stale green-gate and retired-repair claims; no source/check action occurred.

4. Historical corrected-four run https://github.com/CodexCoder21Organization/BuildTestWui/actions/runs/37326538290 has2PASS/2FAIL summary. Numbered attempts: retention3/3FAIL; child cost3/3FAIL. Original draft full CI https://github.com/CodexCoder21Organization/BuildTestWui/actions/runs/37327031208 has1001/1004PASS,3FAIL: retention, missing second coordinator-restart HTTP row, and fixture producer lease expiry. These are owned integration failures, not proven main/infra failures. Automatic retries do not qualify first attempts; full logs/XML/stacks are preserved. No failing check was rerun.

5. Run1 controlled restart-page guard fails3/3 against the unchanged helper. It holds a real completed page, appends through the source API and requires HTTP plus an acknowledged ordered RUN_UPSERT for the later row. Evidence shows awaited t:2 versus HTTP t:1/one row. Proposed confirmedSourceHead correction remains unqualified. Two initial-page guard versions passed1/1 before the correction; they are not missing-row reproducers. Original coordinator restart later failed its first attempt then passed automatically; that is not a green gate.

6. Three diagnostic runs loaded old helper bytes after source edits:46,827bytes, sha25656731671a4c00c7165fb825be1ffa6519db1c546718738c30c6966a6c2a08d58, lacking confirmedSourceHead/diagnostics. Cache record inputFileHashes={} and parameterless descriptor. Six scenario-owned support rules now expose defaulted File inputs as a proposed input-tracking correction; ordinary local/warm invalidation applicability must be proved. No cache deletion workaround shipped and warm cache was restored.

7. Fresh run2 clone restored candidate tree e1588e2fc2a804c075b6165cc4d94d1406541278 exactly, committed locally as df47a4106a33db9acf21e33130a50c370b3d3a5d, fetched/rebased onto unchanged WUI main62bffb6b0a5bf5f9cfbe3fd50a248a82644b60f9. Later README-only correction produces local head dd37bf973c0d5566bad145e001a25637955cc81b/tree a617154bc0e8d5a0bfcc8521be375089e8111259 with unchanged helper/product bytes. Independent temporary-index patch recovery matches that tree and whitespace checks pass. Source commit is unpublished because exact-head full-suite gate has not passed; the full patch/ref record is durable on evidence branch. Restart source sha256fd9f03fac13f6001f58af6f1ae6c592cf0b8b3de8675c0184df16cfa7ee98e49; other five helpers b14cd3b8e3f3e353e9601c0ffe71192dafbf66fcebbf232995a4dbdc4db102fa. Fresh clone cache absent before first test.

8. Run2 has ZERO executed tests:0PASS/0FAIL. Every bounded admission ended before launch, last exit75 at16:25UTC. No ADMISSION_ATTEMPT, selected output/cache record or jar exists in the fresh clone. Total memory.current never met the explicit4831838208-byte admission threshold. The host wrapper now uses anonymous memory, but that does not override the lane's rule. All recorded owned background processes ended. These are admission outcomes, not test failures or a historical root cause. Accepted final progress report16:18:06.761 keeps the claim live; final CHECKPOINT report follows.

9. Separate test/code review records and invariant table identify unqualified cold/null-head equivalence, wake-subscription startup readiness, constructor-failure cleanup, and native/custom work-count equivalence. New work callback occurs after SQL commit BEFORE advanceRunsGeneration; existing native callback occurs after it. A re-entrant HTTP read or thrown callback needs a deterministic public-path proof before retaining this placement. Full wake-contract inventory finds398source/testfiles and six explicit new overrides, all scenario helpers; full remote suite remains mandatory for this shared contract.

10. Read-only other-owner source verifies current route uses existing getProjectionChanges with p3 cursors, not a new getCurrentProjection API. Root wiring adds readCurrentProjection/reconciler and routes before legacy journal; fresh engine main lacks that branch. Earlier no-getCurrentProjection marker inference is withdrawn: it does not test protocol support. Verify existing server/client forwarding before assuming a bump. Current-route qualification needs the real wired root producer and site reader; no artifact published. Retention adaptation design preserves held-page stale HTTP/frozen confirmation, exact membership, real retention and60one-minute ongoing source-write iterations.

11. Pool reads14:24:54/14:29:49 had advancing counts but no qualifying COMPLETED started after14:00. Read14:58 has no qualifying completed run and zero TESTING entries in the returned100rows. Remote permission unused. A final-head full suite is allowed only after its gates qualify and the current pool rule is satisfied; do not submit for discovery.

Relevant PRs / refs

No source branch except the existing draft was pushed by this continuation. Local wip/L50-run2-coordinator-site is preserved as the full candidate patch/ref file; never represent its local commit as a published source SHA. No open PR is recommended for closure.

Next steps

1. Re-verify current state first. Retain claim, check all refs/main and other-owner overlap. Recover current candidate from L50-evidence/run2/current-candidate.patch and current-candidate-refs.json; exact pre-rebase tree must be a617154bc0e8d5a0bfcc8521be375089e8111259 at recorded base62bffb6b. The earlier patch preserves the original e1588e2f tree. Read the full required docs/READMEs and current findings.
2. If total-memory admission continues to be refused, supervisor must explicitly choose to retain that rule or align the brief with the host wrapper anonymous-memory rule; this lane did not relax it. Under the authorized explicit memory.current threshold, run the existing controlled restart selector through one bounded jvm-slot acquisition. Confirm selected output digest, confirmedSourceHead/diagnostic class markers, console markers and observed support-source inputs BEFORE accepting a verdict. Then verify a source-only edit under the same warm cache invalidates the selected artifact; a cold successful compile alone is insufficient. If incorrect reuse persists, isolate/fix the upstream cache with fail-first tests, never delete cache entries as a correction.
3. Establish unchanged-helper failure versus candidate first-pass success on the same controlled condition. Then qualify initial guard, both original restart scenarios and unchanged child cost gate, one selector/acquisition. Reject retry-only passes. Address each review/invariant row with a public-path fail-first proof; run2/proof-plan.md spells out the cold-page and callback ordering experiments.
4. Adapt retention using the verified real current coordinator/RPC/client/site route once other-owner dependencies and published inputs are available. Preserve legacy fail-first evidence, stale/frozen/current HTTP checks and all60writes; do not substitute a facade-only test or resurrect the closed hold remedy. See run2/retention-adaptation.md for deterministic sequence.
5. Exactly ONE final-candidate remote full suite may be submitted when /api/runs shows COMPLETED started after14:00UTC today and TESTING counts advance across two reads at least5minutes apart. Ignore outOfDate. Run it before source push touching this shared contract; record exact-head pass/fail counts/run ID and use bounded build-watchman plain mode. If gate is unmet, do not submit or pretend the suite passed.
6. Before source update, rebase main, verify owned PR stillOPEN and no duplicate, force-with-lease and WHY-first body with first-attempt counts. Qualify all required checks. READY_FOR_REVIEW only when every gate is green; otherwise CHECKPOINT at2.5hours, hard stop3hours. Supervisor owns final review/landing/completion. Never release claim on this continuation.

Operational knowledge

PATH=/code/ws/bin:$PATH every shell. Fresh clones remain inside lane directory; root READMEs read and no AGENTS found. No local full suite/large fatjar. Local selector JVMs only through jvm-slot, one selector,1800s inside; total-memory gate unchanged despite wrapper's anonymous-memory admission. No background pattern kills; track/kill only owned PIDs and every wait has a terminal deadline.

Current WUI scripts pin verified JVM Coursier; global /usr/local/bin/coursier is wrong architecture. Read-only cache audit is run2/provenance.py; no output jars/class files committed. New helper source records and SHA markers are preserved, not inferred from the test's final summary. Historical XML attempt histories override summary retries.

Handoff report/update permits three bounded serial attempts; preserve exact body/report if transport fails. gh pr edit failed with Projects(classic) GraphQL deprecation; REST PATCH via gh api succeeded and returned exact intended body with unchanged head. The report-challenge CLI auto-enqueues/merges, so lane prohibition prevents invoking it; cache/tool friction records are preserved for supervisor. Required-doc audit notes conflicting Embedded spinner guidance; no spinner run. These are documentation/tool findings, not historical test root-cause claims.

Durable evidence

https://github.com/CodexCoder21Organization/BuildTestWui/tree/wip/L50-coordinator-site-evidence/L50-evidence holds full run1attempt evidence and run2 recovery, provenance, invariant/review records, contract inventory, chosen-route source/artifact proof, complete docs audit and admission logs. All meaningful work is pushed there, with no generated jars or credentials. Re-query remote head because checkpoint commits advance the evidence branch. Source PR is still red and this handoff is unfinished.

Current terminal state

CHECKPOINT after the final bounded admission window. Runtime qualification remains unfinished; draft stays OPEN/red. All source work is recoverable from the evidence patch. Keep the claim; do not complete/release the handoff. No merge, enqueue, deploy, publish, close or source push occurred. Latest full live PR/ref snapshot is run2/final-prs.json and final-remote-refs.json; re-fetch before acting.

Final live PR verification