Handoff: Restore test admission and finish stage 1 canonical-result gates

Written 2026-10-04 14:26 UTC.

RE-VERIFY: This is a write-time snapshot. Re-check PRs with gh pr view --json state,headRefOid,statusCheckRollup,mergedAt; verify every remote branch with git ls-remote; read live coordinator run.json/test-results.json and WUI SQLite identities before acting. A fresh query wins. The supervisor owns every merge, enqueue, artifact release, production change and handoff-completion decision.

Mission summary

The original request was to reconcile completed remote run d0cb2b45: coordinator2240 passes versus2239 detailed rows. Required behavior is a detailed terminal projection containing every authoritative canonical identity while preserving separate failed-attempt history. This lane may change BuildTest repositories, but does not own merge, publication or deployment. The same projection defect blocks https://www.handoff.wasmserver.com/handoffs/hf-2026-10-04-finish-the-urlprotocol-binding-clock-fix-pr-after-its-full-suite-verdict-is-reconciled and https://www.handoff.wasmserver.com/handoffs/hf-2026-10-04-resolve-failed-attempts-and-incomplete-results-before-final-review-of-urlprotocol (636ce0de). Neither dependent handoff was modified.

Findings and completed work

1. Read-only durable coordinator evidence: d0cb2b45 contains2240 unique PASSED rows, agreeing with summary and final log. WUI SQLite contains2238, missing stressTestConcurrentInMemorySnapshotsUseExplicitHandlerBoundary and testTcpServerStopCancelsMixedQueuedAndRunningRequestHandlers. Both have accepted authoritative success events; the formerly missing relay row is now present. WUI fingerprint equals compacted replacement marker04b98a48424f2558c0286d6d4476e52b93a04c1fc5036c196054b33a24374ab2; no walk is pending. For636ce0de, canonical count2255 versus WUI2042 (213 missing,0extra); the earlier2176/79 observation is stale.
2. Refuted premise: complete=false on a100-row API response does not mean the whole projection is incomplete. TestResultsApiServlet defines complete for that response covering the entire set. Count/identity disagreement is the actual wrong behavior.
3. Proven source mechanism: getTestResultsPaginated recalculates severity order from mutable global duration averages on every offset call. Existing tests deliberately require live reordering. WUI walks offsets, then commits by run/project/test identity; an identity repeated across a moved boundary collapses, omitting another row while retaining the source fingerprint.
4. Deterministic baseline: producer failed2/2 against published0.0.69274748 and1/1 against main remotely, https://buildtest.kotlin.build/run?id=9a51ddf6 (0passed/1failed): both pages return Suite.test0000 although totalCount remains2. A real embedded coordinator + actual Jetty/SQLite/HTTP WUI baseline returns349 unique identities from350 after the exact inter-page history change. The first remote consumer attempt returned2 feed records to limit1; corrected fixture honors the limit, and that earlier fixture failure is not bug proof. Full failure stacks are saved on the pushed proof branches.
5. Unverified implementation: local producerea3b6e32d89411886d50e55d95799489901c97bf adds a concrete stored-order page and tests for history changes, mutation/restart, oversized concurrent reads and same-length/timestamp atomic replacement, callback re-entry and close during a read. Local API13c69ec2a7479ea54c3048a99456e0041650f9a7 adds a standalone abstract BuildTestCanonicalResultsApi with old interfaces unchanged. Actual contract branches have not been pushed ahead of their full gates; complete source patches are pushed separately atc62a49c0265ea0118ea1373df9287a262cf7b48a so they survive filesystem loss.
6. Validation: targeted producer2245f78e, full APIb14fb717 (232 discovered), and exact committed producer full6a55e49e (3512 discovered) each exited1 at the original30-minute deadline while backend PENDING.At12:47UTC all three backend runs were CANCELED with0passed/0failed;0tests executed in these gates and no implementation pass is available. The small targeted local API compatibility tests passed2/2 against13c69ec; this does not replace its required full suite. Build-watchman is tracked with a hard deadline; read-only coordinator logs confirm admission wait. Published API latest0.0.93 lacks the new interface, so supervisor-approved immutable releases are needed before normal downstream dependency pins/served wiring can compile. The producer method does not yet implement the new interface.
7. Consumer production code is unchanged. CORRECTION: updateTestStatusCountsFromRows writes only derived test_status_counts_json/status and observed_at; retained run_summaries.tests_total/tests_passed/tests_failed remain authoritative (live2240/2238 and2255/2042). Compare those totals for narrow requested terminal-run repair; blanket proof invalidation on the earlier false premise is unnecessary. Stable replacement reads, staged distinct-row/count validation, old in-flight offset migration and bounded repair remain required. A real published old-path stop/reopen experiment still returns349/350 with pending=false and complete=true; expected pending assertion fails0/1. Explicit RPC/client/README patch and direct-handler test are preserved inactive; client now rejects malformed/discovery responses rather than inventing empty results. The committed direct-handler test is pushed at98691a561333db928a4b99d016c67cf0dd3a7c6d, explicitly unrun; no server production source change is activated.
8. Adversarial review found a remaining contract gap: TestResultPage exposes no generation and the new method accepts no expected generation. Stored order alone guarantees unchanged-source duration independence, not one generation across actual canonical source changes. Settle enforceable generation/marker pairing using durable TestResultsSnapshotMarker folded-source proof before releasing the replacement contract. A separate public baseline now proves it: readingA,B then replacingA,B,C,D withA,X,C,Y returnsC,Y next;4rows/4distinct belong to neither generation (0passed/1failed). Distinct/count checks cannot detect this. See checkpoint review-resume.md and generation-repair-invariants.md.
9. Batch baseline experiments with2255unchanged source rows return2176unique after79tail-history updates and2042unique after213updates; each fails0/1 at the exact boundary. No host-dependent tuning.
10. This mechanism explains the demonstrated loss class for both completed runs. Exact historical per-page duration maps/interleavings were not retained, so the historical trigger is not claimed exclusively reconstructed. Complementary terminal-publication work and separate failed-attempt concerns remain with their owners. No tests were weakened and no service restarted, deployed or published.

Stage 1 progress at this checkpoint

The supervisor narrowed this invocation to additive API, coordinator stored-order pages, generation/proof pairing, and served RPC/client. Stage 2 WUI work has not started. Source is unverified, not ready for merge: inactive patches are preserved remotely because full exact-head shared-contract gates have not passed. The new invariant table and README audit are on the stage1 checkpoint branch.

A page sequence is consistent only within one admitted immutable generation. Continuations carry that generation; a changed generation returns no rows and signals a restart from offset0. The optional event-prefix fingerprint must come from the marker paired with those admitted bytes. The writer records its generation before publication observations and verifies it before marker publication. Canonical admission briefly shares the projection ownership lock, releases it before any observation, and preserves post-close per-run reads. Change time and precise modification time distinguish same-size restored-mtime rewrites; a content digest supplies the discriminator on filesystems without change time. Duration-oriented legacy routes remain unchanged.

API full run 0ffb0828 failed before execution: provisioning deadline exceeded, no created droplet IDs, createDropletAsync transport closed after request write began and reported an ambiguous outcome (EOF reading0of4bytes). Preserved baseline e292c718 remained PENDING at the pinned manager's inner1800000ms deadline despite outer timeout5400. Neither supplied a source verdict:0passed/0failed tests for both. Per the supervisor's explicit two-admission-failure rule this lane ends CHECKPOINT. The later source-composed targeted client was terminated while provisioning; no run ID was printed and no tests ran. All owned remote JVMs/watchers were stopped or reached terminal exit. BuildTestCli0.0.3's POM is absent, and build-watchman --once could not find the known runs in its100-row feed; PENDING exit0 is no green gate.

The coordinator/API/server-client/isolated served test source is complete enough to preserve, but uncompiled and unrun. API source reserves buildtest.api:buildtest-api:0.0.94 (metadata ends0.0.93), with generation up to128chars and lowercase64charSHA256 proof. Explicit handler argument type/length checks and strict client response checks are drafted. Root review fixed a missing argument declaration, coarse classification byte-index cache key, writer/marker gap, unrelated legacy diagnostics, immutable API coordinate reuse, public-only test driving, silent cleanup exceptions and map-order-dependent error text. No ready PR exists.

Open proof gates: establish the writer-marker boundary scenario's fail-first behavior against the unguarded draft; force the no-change-time digest branch deterministically via a public seam; run the served invalid-field/boundary/unsupported-version review matrix, all targeted/new/neighboring cases and full exact-head suites. Checkpoint resume.md carries exact commands and separates source-composed validation from final published-coordinate gates. Temporary validation overrides are never publication capability.

Relevant PRs / refs

Next steps

1. Reverify claim, refs, PRs and current remote verdicts. Apply the latest inactive api.patch/producer.patch/rpc.patch from stage1 checkpoint to fresh main branches, not older incomplete patches. Validate the new marker/generation boundary scenarios and actual served-client test before relying on source.
2. Complete targeted remote coordinator/served tests against source-composed unpublished API. Run the full remote suite once from each exact shared-contract head before activating/pushing it. No increased timeouts, reduced iteration counts, skipped tests, added retries, merge/enqueue/deploy/publication or handoff completion by a lane.
3. Finish separate test-comprehensiveness and adversarial passes. Rebase immediately before main-target PRs; descriptions begin with coordinator2240 versus projected2238 rows on d0cb2b45 and the two blocked UrlProtocol handoffs. Gate required checks green. If admission fails twice, checkpoint with exact resume command as instructed; no unexecuted run is a test verdict.
4. Supervisor owns API release and dependent pins. API source reserves0.0.94, currently unpublished; existing coordinator/server pins remain0.0.93 and served-client pin0.0.91. Embedded source coordinate0.0.69274749 is unpublished (metadata ends0.0.69274748), client source0.0.47 is unpublished (metadata ends0.0.44), and server-service source0.0.26709273972 is already published and must not be reused for this change. Reverify before any authorized release; no source validation override is a published capability.
5. Stage2 next invocation after publication: stable replacement walks, migration of saved legacy offsets, bounded requested terminal count-mismatch repair, preservation of visible rows/sparse overlays/feed/backfill barriers and distinct staged-row validation. Read consumer-resume.md. Reconcile both d0cb2b45 and636ce0de identity sets.

Reusable operational knowledge

Every shell needs PATH=/code/ws/bin:$PATH. cs is the working JVM coursier; global coursier is the wrong architecture. Modern BTE/API launchers download their verified JVM coursier; JAVA_OPTS is rewritten by some scripts, so a cap can use JAVA_TOOL_OPTIONS. This lane clones under/code/ws/L4/workspace. WUI DB:/root/ContainerNursery/buildtest-wui-data/projections-rebuild-20260821.sqlite; SQLite URI mode=ro was used. HardwareControlFabric certificates are absent here; supplied diagnostic SSH was used only for read-only files, never management. A complete source snapshot was submitted in6a55e49e with headCommitd39f6e3ba5c28bdbabd50dbcf290cdad8741030a. Failure proof/source and resume patches are durable remotely; full generated logs remain local and are not committed as build outputs. Unified resume patches intentionally contain blank context prefixes; source whitespace checks are clean.

CHECKPOINT outcome and salvage

The lane reached its approximately2.5-hour checkpoint with proven source-level duration-order loss and public consumer/restart/generation counterexamples, but no ready PR. Preserve every listed branch. Nothing is recommended for closing: the related PR content is not proven already on main, and the publication/clock/failed-attempt work remains useful with its owners. No source implementation pass, merge, enqueue, release, production change or handoff completion occurred. Resume the unfinished implementation and open reviewed main-target PRs only after their required gates pass. The supervisor owns every landing/release/deploy/completion decision.

Local producer ea3b6e32d89411886d50e55d95799489901c97bf and API13c69ec2a7479ea54c3048a99456e0041650f9a7 are represented byte-for-byte by inactive patches on the checkpoint remote branch because the explicit full-suite-before-shared-contract-push gate was not met. These local commits are not claimed as remote heads. No partial source exists only locally.

Full README contract audit is preserved on the checkpoint branch as readme-contract-audit.md. Both BTE/WUI READMEs were read in full; the audit confirms post-close addressed reads, current display ordering, equivalent-fingerprint saved-offset behavior and sparse authority barriers. Existing README retry behavior was not changed; its old artificial-load testing guidance conflicts with this brief and was not used.

Current CHECKPOINT outcome

This invocation stopped under the explicit two-admission-failure rule, not because tests passed. The new remote branches are the stage1 baseline and inactive checkpoint only. Active source branches were not pushed without the full shared-contract gate; exact local content heads and bases are saved in source-heads.md and full patches. Source-only helper syntax/export and three patch checks passed; no Kotlin compilation or behavioral verdict is claimed. Supervisor must restore test admission, resume source gates, decide the staged release/pin/landing sequence, and assign stage2 afterward. No merge, enqueue, publication, production deploy, PR closure, other-handoff edit or completion occurred. Retain every satellite; the old proofs and complementary work are useful and none is proven superseded.