Handoff: Finish changed-only BuildTest run-store recovery and open the unmerged PR

Written 2026-10-20 04:50 UTC. RE-VERIFY: This is a write-time snapshot. Check df -h /, remote branch heads, current main, native attempt histories, service health and open PRs before acting. No PR or production change has been made by r88. The current candidate is not qualified.

Design decisions in force

Owner direction, verbatim, 2026-10-19:

"startup (and incremental recovery) should always be fast. If there is any situation where startup or incremental recovery are slow, that's a bug which should have been tested and exercised by https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos and which should be fixed."

The owner chose "Trust the verified index". Restart loads a checksummed index and opens only changed/unfinished runs. An unchanged unreadable source is reported when addressed. Unknown markers use verified indexed absence; known markers needing that source retain the exact read error; restoring reads makes both succeed. Torn/corrupt selected index state must produce a notification and a full correct rebuild. This resolves the previous eager-validation question; do not reopen it.

Additional owner approval, verbatim, 2026-10-19:

Answer: "Approve both (Recommended)" (1) "Test fixtures that edit a finished run's files on disk while the service is stopped now explicitly invalidate that run's index entry before restarting; every error and recovery assertion in them is kept." (2) "The stable-owner guard captures its startup work counts before the addressed known-upload lookups, so it measures startup only; all limits stay." "Both changes follow directly from 'trust the verified index'. No error or recovery assertion is dropped: edits made through the service are still caught, and the work-count limits stay the same."

The invalidation hook must remain a genuine public operator operation. Both approvals and the trust decision belong verbatim in the eventual PR and comments in changed tests. The 76-candidate classification is complete: reuse it, do not replace it with the later physical-boundary inventory.

Journal remains0.0.18; the0.0.21 conflict is separate. Never increase timeouts, cut iterations, drop assertions, disable tests, merge, enqueue, publish or deploy. Read ~/.claude/CLAUDE.md first on resumption. Clone only under workspace. Check df -h / before every clone/build and STOP under10GB. Local JVM cap768MiB–2GiB, one heavy job, prefer remote. Kill only recorded PID/start identities; delete only aibuildcache entries whose names contain this exact r88 path. Commit trailer: Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. PR footer: 🤖 Generated with [Claude Code](https://claude.com/claude-code). Checkpoint out/r88-findings.md every15minutes and push milestones. Every wait has a deadline and is reaped before stopping.

Mission and current block

Finish the run-store part of the incident: after the04:46UTC deploy, recovery took69minutes and builds/webUI stalled. Restart must stop reading retained history. Finish the source-authoritative persistence design, public fail-first crash/reentry/read guards, targeted qualification, exact-head full remote suite, final independent reviews and a clean unmerged PR from main. The PR opens with owner direction and incident, then mechanism/change/before-after counts.

At04:46UTC the shared disk had8.2GiB free. The native64 runner stopped automatically below10GiB before emitting XML; all5recorded identities were absent. Root checked live Java commands, then removed only its inactive r88 cache (754473438bytes). Free space remained8.7GiB, so builds/clones/tests remain stopped. The user was asked to free shared space;15GiB is suggested so compilation can proceed above the mandatory10GiB stop. No unrelated cache/process was touched. This is a resource interruption, not a verdict about the candidate. Metadata-only capped CLI calls remain permitted to preserve the handoff; no compilation is being run.

Work and evidence chain

1. Continue from the prior r81/r76 handoffs. Full original README, TESTING, PHILOSOPHY, linked r81 materials and publication inventory were read. No repository AGENTS.md exists. The root cause on main was direct enumeration/decoding of all run sources, eager owner snapshot loading and historical recovery queues, including Java File reads invisible to injected Okio counters. Main baseline https://buildtest.kotlin.build/run?id=f29c9ca7 had4936rows:4931filtered,4executed first failures,1preparation failure,0passes. At6/48history unchanged sources produced6/48decodes and9438/75676bytes; terminal baseline19164/153656bytes. Real child restarts reread4historical sources each. These are native body failures, not infrastructure failures.
2. The branch implements checked bounded scalar catalog/owner pages, source-work UUIDs, selection redo and conditional publication. Source atomic rename is authoritative. Pending work precedes source publication; source force precedes committed indexed facts; immutable carrying cuts become durable before exact-token retirement. Reentrant updates share a transaction frame and cannot mutate a frozen published cut. A bounded redo selects at most64changes/32768rawpagebytes; packed pages contain at most8aliases/16KiB. Selected-page decoding and actual Java reads are counted. Corrupt selected state falls back to diagnosed correct full rebuild. Full design/invariant and proof records are in the branch findings and linked plans; do not rewrite the design from scratch.
3. All76offline candidates were individually classified in out/r81-test-classification.md and out/r81-offline-test-facts.md. Approved fixture changes and every error/recovery assertion are recorded in out/r88-fixture-findings.md plus the restoration/graceful fixture ledgers. Public invalidateBuildRunIndex is a real operator action. The unreadable-owner test now follows the approved verified-index contract. The separate98physical-boundary inventory covers installation assumptions, not a new classification of76candidates.
4. Physical page/root installation was moved from every selected redo to bounded checkpointing, while first-complete generations checkpoint immediately. Physical installation still forces all replacement inodes and page directory before the selected physical root and its directory, then removes redo. Close error guards showed checkpoint failure could skip later ownership release; checkpoint-only cleanup was corrected. Native59 qualified many of these changes:35selectors,28first-attempt passes,7first failures,0preparation failures and14extra failed attempts (49body attempts). First passes include the physical work guard, page/root errors, all3real child index/page/root crash boundaries, torn-index recovery, graceful archived/terminal restart, bounded selection/reentry/cache/close and approved unreadable-owner behavior. Full traces are retained. Its combined-close and64-run full-catalog failures were fixture prerequisites, later corrected without assertion changes. Its3original150cycle timing guards,1222run timing guard and stable-owner exact-one-row guard remain red.
5. Native60 proved more paths against old candidate behavior:4actual first bodies,1pass/3fail,1separate generation preparation error,6extra failed attempts. Corrected64-run settlement passed first9393ms. Low-storage constructor decoded all518untouched sources on all3attempts; combined checkpoint-plus-row-cleanup failure preserved exact primary/suppressed errors but saw unregister0/1 on all3attempts. The150cycle timing case timed out all3attempts. A9second partial JFR recording on only its first attempt showed2001FileForce events,395redo tmp forces,113source forces; sums overlap across threads and are not wall percentages. Original JFR and21hash-verified compressed records remain; large derived JSON exports were removed after verification and can be regenerated. See out/r88-profile60-profile-report.md and cleanup note.
6. The generation guard initially failed preparation because assertContains used a positional message as ignoreCase. Corrected native61 executed3actual attempts and failed3/3: a reentered complete new generation remained in redo while the physical root stayed at the prior complete generation after the outer public call returned. Earlier exact warning chain, source/public facts and bounded redo assertions passed. This is distinct from two renderings of the same unhandled notification. Native60/61 full archives and root verification are durable.
7. Source checkpoint8652fc9 contains the narrow generation comparison correction, mandatory control/live-registration cleanup after broad row cleanup, and indexed low-storage selection. A new terminal-retention-order table (catalog format3) preserves positive completedAt else startedAt, then ID. Automatic reclaim checks capacity before choosing/opening one candidate, then validates its authoritative source/order and existing exclusions before deleting. Explicit operator reclaim retains raw offline discovery. Format1/2 rebuild guards and full5-table packing coverage retain original density/work bounds. A public4-source guard covers fallback completion time, ID ties, offline invalidation and repeated restart with exact source/tombstone/owner/read assertions. None of these latest corrections is natively qualified yet.
8. Remote62 failed BEFORE submission: health returned STARTING: recovering local state (elapsedMs=3259486), expected exactly OK. No run ID/link/XML exists; counts unknown. The prebuild rebase was separately refused due two unstaged report corrections; launching anyway was a process error, recorded, and corrected before subsequent local attempts. Native63 stopped after28.299s below10GiB; native64 stopped after387.446s below10GiB. Neither emitted XML, so neither is a red/green test verdict. Both selected source/test inputadf9a89 (same source as8652fc9). All processes are reaped; root independently verified3remote62,5native63 and3native64gzip records. Reports and exact console/error traces are under out/r88-evidence.

Unresolved mechanisms and next proof designs

Do not implement the proposed post-source grouping as written. out/r88-post-source-cut-design-review.md found that a new index.read outer frame around owner.committed reverses public owner-query lock order (owner→index versus index→owner), admitting a two-thread cycle. Nested same-thread index.read also returns before outer drain, so the plan's claimed post-drain identity check is false for nested writers. Current source does not contain this grouping. Root verified projectionLockFor uses canonical-path weak-map locks, not stripes. A common owner/index publication lock is only a prospective idea: not decided, implemented or tested. Root must finalize lock invariants and a public deterministic child-JVM overlap reproducer, including bounded cleanup, before a change. A callback-driven owner query can pause in complete.json while an index observer calls sibling create; ThreadMXBean can observe actual lock ownership without reflection/private API. Do not blindly turn this sketch into code.

The existing physical guard was strengthened with a final maximum2redo/create oracle AFTER all its prior source/public/error/physical assertions. Current path still performs3(pending before source, committed owner after force, then catalog); native64 did not reach a verdict, so the stronger fail-first proof is still outstanding. Pending-owner source ordering must stay intact. Review out/r88-post-source-cut-proof-plan.md and revise its lock/nested-frame claims.

Indexed storage race is static, not reproduced. out/r88-indexed-storage-delta-review.md: helper can see complete, another public query resets to incomplete and starts full source rebuild, then helper's withVerifiedRunIndex tries to claim a second rebuild and throws IllegalStateException. The accepted design required incomplete-state source fallback without recursive rebuild. Root must force that exact ordering through public filesystem callbacks before fixing. A canonicalize callback before index-lock acquisition might provide a gate; this is only a draft experiment. Do not ship an unproven catch/retry/timeout.

1222run guard and bulk invalidation. Latest native59 stacks time out in per-run invalidation BEFORE the second service or paused-source overlap. out/r88-bulk-offline-invalidation-proof-plan.md proposes a genuine public bulk operator API, first implemented naively, then a fail-first force-count/error/immutable-input/restart guard before sharing only final parent-directory barriers. Existing single-source UUID protocol, exact failure/recovery assertions,611midpoint,4×1000callers,1222IDs,30s limit remain. No API/test is implemented. Follow the complete plan and its durability invariants, not this summary alone.

Stable-owner Journal0.0.18 guard. Startup counts now capture before addressed reads per approval, but a NEW guard expects exactly one reset feed row while0.0.18 supplies the old PENDING baseline plus the CANCELED tail. Native59fails3/3 after startup/public/source checks pass. An optional owner question was sent asking whether to assert the final ordered RUN_UPSERT rather than exactly one row. No reply was received. Approval2 does not cover row count. Assertion unchanged; do not drop the guard or bump Journal. Resolve with evidence/owner guidance while independent work proceeds.

Relevant PRs and remote refs

Main at last check: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/451ba11b10b5c86dab70d6425750e355e9abfc4f . Open/all-state PR query for this WIP head returned[]; there is no PR.

Deployed/published-but-unmerged: NONE from this effort. No production mutation/publish/merge/enqueue command was issued; no statement about current production version is made. Both local checkouts have no stash or extra worktree. Historical branches above were checked with git ls-remote; their local heads match. The primary WIP checkpoint is pushed and independently compared with its remote after this document is committed.

Durable starting points on the r88 branch (all under out/): r88-findings.md; r88-pr-description-draft.md; r81-handoff.md and everything it links; r81-test-classification.md; r88-fixture-findings.md; r88-additional-restoration-fixture-ledger.md; r88-additional-graceful-fixture-ledger.md; r88-indexed-storage-reclaim-proposal.md; r88-retention-order-inventory.md; r88-indexed-storage-reclaim-implementation-findings.md; r88-indexed-storage-order-test-findings.md; r88-generation-and-close-correction-plan.md; r88-physical-install-implementation-brief.md; both new independent review reports; native59/60/61/63/64 and remote62 reports/manifests; profile60report/originalJFR. These are WIP evidence, not final PR content. Original central handoff is also saved as r88-original-central-handoff.json before replacement.

Ordered next steps

• [ ] Re-verify disk/main/branches/service state and read required linked documentation first. Restore disk above10GiB; ideally15GiB before compiling. No blind remote retries: classify failures before tests start separately.
• [ ] Re-run the14native64selectors against unchanged current source to qualify generation/combined-close/storage and obtain the max2redo old-code proof. Read native top-level attempt-history; report first passes/fails, automatic attempts and preparation separately. Preserve new failures in full.
• [ ] Force and fix the indexed-storage race using public operations; finalize post-source lock/nested-token invariants, then fail-first and implement grouping without moving source authority or losing reentry receipts.
• [ ] Execute the bulk invalidation plan test-first if still justified by measured1222guard; keep all existing workload/assertions/limits.
• [ ] Pass all original150/1222timing cases, source/read/index-page bounds, selected corruption, source/owner supersession, close/reentry and graceful/real-child/torn/repeated restarts. Resolve the stable-owner row contract without an unapproved weakened assertion.
• [ ] Prepare a clean feature branch from latest main. Exclude ALLout and5WIP-only tests: testR88RuntimeOwnerCostProfileCurrentIndex; testRunIndexCloseDrainsQueuedFeedBeforeRecovery (keep InstallationCloseDrainsQueuedProjection); testUploadOwnershipCheckpointCleanupDuringFullRebuildRetainsOriginalFailure (keep modified original); testR88InterruptAfterRenameExceptionTrace; testR88RecoveredSourceWorkLockTrace. Retain all genuine new guards and public API/docs.
• [ ] Fetch/rebase, run targeted tests then exact-head FULL REMOTE suite; if service fails before tests, record returned link or explicitly no link and counts unknown. Final independent test-comprehensiveness and adversarial source reviews on that exact head. Open unmerged PR with full owner wording, incident, mechanism and native before/after counts, monitor required CI with bounded build-watchman. No merge/enqueue/publish/deploy.

Operational knowledge and deliberately omitted files

The primary checkout's Git metadata was safely reduced from1.4GiB to47MiB using206commits of only this WIP plus main. Fetch refspec intentionally covers those2branches; do not fetch every unrelated branch and refill disk. Main remains an ancestor. Working sources/README/evidence were unchanged. Local historical control checkout metadata is31MiB and contains no unsaved work.

r88-local-native-disk-guard-runner.py records exact process identities, uses768MiB/2CPU/SerialGC and a live10GiB stop. Its native64 selector command is in the archived result. r88-remote-native-runner.py preserves native logs. r88-metadata-cli-runner.py is a bounded384MiB metadata-only adaptation for Handoff CLI; it is not a build runner. Scripts are checkpointed underout. Existing tests retain128MiB JVM defaults. Full suite must be remote per brief. Native result rows with time0/noattempt-history are preparation, not bodies; filtered-out rows are not passes.

Large derived profile JSON was removed only after verifying compressed copies and original JFR; manifests, parser commands and exact hashes allow regeneration. Dependency caches/build jars are deliberately omitted from Git. All source/test work and needed small scripts/reports/archives are remote; no build output is committed. The report-challenge skill's automatic merge conflicts with the explicit no-merge instruction, so infrastructure friction is preserved in this WIP instead. The central Handoff service health and main-task get returnedOK via published0.0.8CLI, with no server change. A separate owner-decision get failed with a read0of4 EOF after transport write; its complete error is archived, and no mutation was attempted by that get. The existing dependency is resolved by the explicit owner quote, not by a new interpretation. The canonical handoff README still describes old PR merging, which conflicts with the current skill and the user prohibition; only the current central CLI process is used.

Remote continuation document: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/5fc521d25143a174f43d33097aced293b3c19de3/out/r88-handoff.md