Finish final-head startup-recovery CI and reconcile prerequisites

RE-VERIFY: snapshot at 2026-10-09T01:05:57.222096+00:00. Read this current section and the linked PR body, then query the draft head, four shards, and first-attempt XML. This lane stops before enqueue; no merge or production deployment is authorized.

Current draft PR: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1342 at 1761d7da2607a2e86c3b855c02e3a11df8706592. Source checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1i . Evidence and complete gate/review files: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1i-evidence . Diagnostic source: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1i-diagnostic . This round adds fixture ordering and public coverage; it does not change production source.

The two cache fixtures requested replacement compaction without enough public positions before the protected suffix. After cold measurement, create/cancel/delete control writes now advance those positions. Every original measured-source assertion remains, with exact extra tombstone identity/kind and total row count. Both capacity cases pass 3/3 first attempts and cleanup 1/1.

Saved rec1h XML actually has 11/12 first attempts for the header/retention/warning/close set, with one suffix retry-only pass; that retry receives no credit. The suffix fixture queued five asynchronous renewals into a possible shared batch while requiring fingerprints on both retained rows. Only a batch’s final same-source row has a completed-source marker. A deterministic held-writer reproducer failed all three local and all three Actions runner attempts, showing the product preserved complete rows correctly. The original fixture now publicly drains the cursor after each renewal, preserving all counts, bounds, waits and assertions; it passes 3/3 first attempts. Existing batching control passes 1/1. A new deterministic grouped-suffix case checks complete retained-row equality through checkpoint rollback, suffix-only replay, restart and compaction, including absent marker; it passes 3/3. Total current targeted credit 14/14 first attempts, zero histories. Independent test and code reviews are complete; review notes and the refuted startup-count concern are in the evidence branch.

Sole final-head full-suite gate: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37867112322 . All four builds succeeded and all four test steps are in progress. Plain build-watchman --all-checks tracks the final head with a deadline. No suite verdict or retry credit is claimed. Earlier capacity-only run 37865394656 was superseded and canceled without suite credit.

Prepared inherited source retains O1 investigation-file removal, S2 checkpoint cadence/fixed-cut force ordering, honest overlay comparison and U2 non-reproduction, Clock declarations, header/warning/double-close fixes, and main disk-backed cache contract. The draft body has the complete continuation ledger and historical gate links.

Both prerequisites were open on the last check: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1340 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1343 . Latest orchestrator instruction assigns their rebase/reconciliation to a later round. Keep the lock-free per-reader descriptor pools and public I1–I5 contracts when that round proceeds, with the large-delta heap proof required by the prior brief.

Next: collect all four final-head XML artifacts and require first-attempt-clean counts, write final CI verdict and PR body via REST PATCH with body=@file, checkpoint evidence, stop owned watcher, remove remaining owned checkouts/caches and release claim. Keep draft. The diagnostic checkout/cache are already removed and no local slot is held. Product/evidence checkouts remain while this lane watches CI. The lane has a 90-minute hard time box and begins cleanup before it.

Historical handoff entries

Resolve the remaining startup-recovery findings and finish PR CI

RE-VERIFY: snapshot at 2026-10-08 22:44:57 UTC. Query the PR, branch head, four bld shards and saved XML before proceeding. This continuation must stop before enqueue; no merge or production deploy is authorized.

The user's remaining request is to finish startup-recovery checkpoint and lazy reset restoration, fix first-attempt failures at their cause, and make the PR's own four bld shards green. The filename decision is implemented and proven. Three findings from the prescribed nine-check run remain unresolved, so the PR is deliberately draft. CI is running; no suite verdict exists yet.

Current source commit: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/8fafa4a5d4fab697e543ee2e7ce17bd6e0f3e9b1 . Latest rebase main is d9d410b3aa919a02f0b09569e67b056e97979710, conflict-free; dedicated projection paths have no incoming hunks. The extracted lease-interruption path was rechecked by both maintenance tests in the final 30-case gate. The tree was clean and remote head matched immediately before opening the PR.

Completed behavior and evidence

Durable files use checkpoint- plus their existing generation basename. Transient journal-pages-* files remain transient. Restore requires checkpoint- before stripping it and validating safe name, order, checksum and row count; undeployed bare checkpoint names are rejected. Cleanup compares retained names exactly, then strips checkpoint- only for family recognition and preserves existing live-file and OS-lock checks. README names the durable family.

The existing main tests/projectionMaintenanceStopAtBaselineFileLockIsNormal.kts is byte-for-byte unchanged. The corruption prefix assertion is branch-only, first introduced at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/a9c9a3cc484966dfa9c4ab59c472537b2aa74f19 ; changing it to checkpoint-journal-pages-data- is the brief's permitted correction.

Two new public tests assert surviving exact checkpoint bytes plus no transient spills after maintenance stop, and cleanup that retains exact current names/bytes while deleting stale prefixed and bare files. Old names fail both tests 3/3. Naming-only old cleanup fails the corrected cleanup test 3/3 at prefixed stale deletion. Final 30-case XML has 30 direct first passes and zero attempt histories. The fixture uses latestProjectionCursor to drain the accepted public write before reading its suffix: recovered getProjectionChanges may intentionally return prior committed state while reconciliation is incomplete. The rejected earlier fixture run has no cleanup credit.

Inherited evidence retains six embedded regressions 3/3 first passes, all 20 touched new tests passing, the warning-loss reproducer failing 3/3 before drain-then-propagate and passing after, and prior public checkpoint/lazy-reset gates and mutation controls. Deferred notification scope now delivers the whole warning batch before propagating the original handler failure. Independent test-completeness and source review found no concrete naming/restore/cleanup defect. Two inherited coverage gaps remain listed honestly: combined valid-checkpoint over-retention appended suffix/cursor, and oversized/incomplete optional-header boundary. Additional low-confidence review notes concern stale index/classification examples and partial-file deletion; no proven bug was found.

Historical 13-test experiment: completed once, hypothesis unconfirmed

Historical source ccc3afcbb4eebe6786bd5190a781cbb0a68cd220 and helper ad9dc7643bc345a56466245d339273472c9696b0, identical 966-file physical manifest SHA c127c091c9cb35bb11177e619e3480df4f451be4ed83120689027217210d6b1e, pin 0.0.112. The single bounded experiment passed 13/13 first attempts with zero histories. Original failure: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37824718278 . The experiment selects 13 names rather than running all 974, uses local Java 21 rather than Ubuntu Temurin 21, and differs in workspace/cache preparation; original CI POM/runtime are unavailable. API POM bytes match the prior unchanged local baseline and declare Clock. Captured three WorkingSetWorker classpaths contain Clock, but the 13 bootstrap classpaths were not fully captured. Passing constructors do not prove identical runtime closure. No consumer dependency was changed. Full comparison: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-rec1g/docs/investigations/coordinator-startup-readiness/rec1g-partition-experiment.md . The PR's four shards adjudicate the remaining hypothesis.

Nine-check run: completed once, no retry credit

The single 40-minute-bounded command ran 21:39:23–22:01:57 and ended normally exit1 on d371ac8b over then-current main b9fe7a62. XML records nine tests, thirteen attempts: six first passes, two retry-only final passes and one final failure. All eight pre-existing scripts are unchanged relative to current main; overlay fixture is branch-added.

1. testMixedReleaseModuleCacheByteBoundEvictsAcrossReleases: first attempt times out at 120000ms in TailingInputStream.awaitNextPoll line164, transferTo at test321; retry passes106998ms. First thread dump parks in Condition.await. Saved producer dump shows shard0 awaiting SharedPreparationPhase.exec in prepareSharedPreparationPublicationCopy, shard1 awaiting seed-cache publication, projection writer idle. Public log reader is normally caught up while runbe94403a is BUILDING. The external copy/check shell child is not captured; exact outstanding operation remains unknown. This does not support the same checkpoint cause as the overlay finding. Capture owned phase process/input byte counts rather than extending timeout.
2. tailAccelerationSurvivesDeathOfOriginalRunnerE2E: first assertion at test677 says the run must reach terminalization after the original runner dies; last recorded state TESTING with replacement shards. Retry passes. Diagnose the runner-death/replacement terminalization sequence; do not treat retry as proof.
3. testProjectionInitialOverlayHeapIsBounded: all three attempts fail before BEFORE_RELEASE and before heap measurement. Helper holds collected baseline240s while 32768 real renewals must drain through latestProjectionCursor, then release it. Recorded renewals11856/24598/23098 before the parent300s kill; held baseline throws at OverlayHeapMain.kt:54. This is neither an OOM nor an observed heap-bound breach. Durable checkpoint copying/forcing/hash conversion is a candidate extra renewal cost, but not a demonstrated mechanism. No guessed fix, iteration reduction or timeout increase was made.

The single measurement-only overlay follow-up on final head8fafa produced a valid90-second child JFR22:21:54–22:23:24 and thread dumps61s apart. It stopped deliberately22:23:51 after capture, exit143 with no XML verdict or gate credit; every owned process is gone and slot3 released. Both samples show main waiting for projection writes and writer forcing checkpoint-generation/manifest bytes. Writer samples2150 total,1817 in force0/sync0(84.5%); checkpoint-specific force groups1364(63.4%). Main recorded parking76.937s;91 GC events total0.155s. Source line1524 persists checkpoint before signalling each committed append batch; each serial renewal drains that path. This strongly identifies repeated durability work and refutes a GC-storm explanation. It does not prove a safe cadence change: any new design must preserve bounded startup suffix, force-before-advertisement, crash recovery and public cursor ordering. No code was changed from these samples.

CI and next steps

Draft PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1342 is OPEN on8fafa. Plain build-watchman --all-checks is tracked until22:44. All four bld shards entered Run tests22:21–22:22 and remain pending. Actions run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37852579416 . No buildtest run URL yet. At22:40 watcher compares21min elapsed with~41min typical recent successful runs and reports this within range; no infrastructure-stall diagnosis is supported. The installed kotlin-build-ci-test app has queued zero-run suites on every sampled main commit, including suites older than an hour. Watchman correctly treats that as base-wide nondispatch rather than rerequesting this PR. Do not rerun red CI to seek a green retry. Capture first attempts from each shard's test_log_file.xml; a final green check alone is not first-pass credit.

Continue from saved evidence, resolve each of the three findings with a reliable public failing-first reproducer and source fix, preserve all current test bounds, fetch/rebase before builds and pushes, run the appropriate gate once, and update this draft PR. Never merge, enqueue or deploy. Pin remains0.0.112. Use tracked bounded commands and shared build slots; stop all owned descendants before removing checkouts. No artifacts were published and no production deployment was performed in this invocation.

Current local evidence is scratchpad/out/rec1g-findings.md, rec1g-review.md, rec1g-retry-only-stacks.md, rec1g-retry-failure-summaries.md, rec1g-retry-findings.md, rec1g-final-gate.xml, rec1g-ci-watch.log, and measurement artifacts when available. Continuation evidence is now preserved at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1g-evidence/docs/investigations/coordinator-startup-readiness/rec1g-evidence . Verified evidence head is4e342bd9a1a9e78efa07564eaaadbd4f58d41ef0; it contains findings, full sanitized failure stacks/console, first-attempt accounting with original XML hashes, the JFR and thread samples, review, historical experiment and PR body. It has no source changes after8fafa and does not change the source PR CI head. Historical gate ledger and production motivation are in the PR body and inherited committed rec1f files.

Explicit continuation experiments

Read https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-rec1g-evidence/docs/investigations/coordinator-startup-readiness/rec1g-evidence/rec1g-continuation-experiments.md before a new source edit. It separates a temporary causality mutation from a product fix, requires a checkpoint/generation/manifest/suffix invariant table with public crash/close/restart proofs, and names the external shared-cache and resumed-runner processes whose state was missing from current evidence. The profile does not justify removing required force operations or changing test bounds.

Terminal CI snapshot

Freshly verified2026-10-08 22:44:57 UTC: draft PR OPEN, unmerged on8fafa; all four bld shards remain in_progress, zero artifacts/XML, no suite verdict. Local plain --all-checks watcher hit its existing deadline and exited124; its tracked timeout/Java identities are gone. Remote Actions CI continues at https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37852579416 . Watchman described elapsed time as within the~41min normal range; no PROBLEM line. No rerun/cancellation/merge/enqueue.

This bounded invocation ends with three unresolved first-attempt findings. Source/fixture/code bounds are preserved. Finished experiment worktrees and owned helper temporary data are removed; remaining source/evidence checkouts were verified deleted after the final push. The terminal status report records final deletion and claim release. No production change or publication.

Verified cleanup2026-10-08T22:45:12.524458+00:00: all four owned clone/worktree paths are absent; local build/test/profile/watch processes are stopped, source/evidence work was committed and pushed, and other lanes were untouched. Final evidence head4e342bd9a1a9e78efa07564eaaadbd4f58d41ef0 at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1g-evidence . Remote PR CI remains running; no first-attempt suite credit. Handoff claim release is the final service operation.

rec1h continuation checkpoint — 2026-10-08 23:18 UTC

Claimed as sol-rec1h. Source is preserved at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1h ; evidence, including all43 original investigation files, is preserved at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1h-evidence . O1 is removed from the product diff. The14 missing Clock callers and51 equivalent callers now declare their actual runtime dependency. New checkpoint cadence source compiled locally; remote correctness gates remain pending.

Main and unchanged candidate overlay both pass3/3 remotely without fixture changes. Setup-inclusive mean durations156.203s/main and150.135s/candidate do not support a20% candidate slowdown; precise loop stdout was omitted by successful-run reporting. The original3failures are not yet attributed. All3main partitions pass both U2tests first attempt;3old-candidate U2partitions are active, so these are not classified baseline.

Source review S2 removes perappend checkpoint IO, uses4096row/4MiB cadence and captures owned fixed cuts outside journal monitor; force ordering is preserved. Approved test trigger moves and both coverage cases are adopted; independent reviews and remote red proof are active. Red proof https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37858162837 . Inherited full CI: shards1/3/4green, shard2red14missingClock, buildgatedbyshardfailure. Draft remains unchanged; no enqueue, merge or deploy. Prerequisites https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1340 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1343 remain open; reconciliation waits for both to merge. Hard stop00:17UTC.

rec1h continuation checkpoint — 2026-10-08 23:53:37 UTC

Source f392571e915b311acb28132639a6aacc1cde73b1 is verified at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1h; evidence aef66e885a6b50f97231ed66bfd0c672e3803fca at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-rec1h-evidence. Both prerequisite PRs remain open and unmerged; this checkpoint is over main c91873ddab, preserving its disk-backed baseline descriptors. Draft PR source remains8fafa4a5; its full CI finished with shards1/3/4 green and shard2 red14missingClock declarations. Corrected source has18Clock/neighbor first-attempt passes; its full suite is not yet verified.

Original six overlay runs all passed. Setup-inclusive main durations160.956/163.614/144.038s versus original candidate125.347/193.601/131.458s imply209.78/218.26 renewals/s proxies, not exact loop rates. Successful logs omit child timing output; this evidence does not establish20%regression or attribute prior240s failures. Independent work-count reproducer failed3/3 on old perappend checkpoint code. New cadence copies/forces a leased fixed cut outside journal monitor at4096rows/4MiB, compaction, maintenance and orderly close; zero-append-window work count, force ordering, suffix and cleanup tests pass. Unchanged overlay after change passes3/3 at94.220/126.253/119.022s.

Each U2test passes3/3 on main and3/3 oldcandidate, plus latest-source gate firstattempt; neither historical failure is currently reproduced. Do not label them baseline or branch-caused without additional evidence.

Independent review found finally masking a primary checkpoint failure. Public doublefailure reproducer is red3/3 then green; cleanup now attaches the generation deletion error to the original sentinel. Optional header test found FilterReader bulkread bypassing the4096character counter; fixed bothsingle/bulkread and preservedexact4096authorityboundary. New retention fixture must publicly start lazy projection reconciliation before awaiting its latch or work count. Header and retention cases have fresh3runs pending.

N1worker directprints removed after3/3red, but newgate foundsecond printin DeferredUtilizationNotifications beforetoss, also3/3red. Latestsource postpones onlyoptional notification printing toexisting unhandled fallback. Handled warning test andclose proof are included in fresh3coveragegates: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37861583766 https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37861581603 https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37861581487

Cache decision23:43 in orchestrator findings supersedes the pending user clarification: preserve main disk-backed contract, replace3branch-only obsolete16entry/64KiB tests with actual resident-row/disk-reader/cleanup contracts and remove deaddecodedcache. Mechanical implementation is active in own worktree; no source contract will be changed to fabricate cache observer values. Need independent review andremoteproof beforeadoptioncredit.

Next: collectfreshcoveragegates, adopt/review/test approvedcache replacements, runfullcurrenthead unit suite. Once prerequisite https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1340 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1343 are inmain, rebasepreserving lockfreeperreader descriptors andprivatecompactionstage; keepallI1-I5tests unchanged andlarge-deltaheap3/3. ThenpushdraftPRbranch, updatebody via API-Fbody=@file andreread; tracked plainwatch-build allchecks withdeadline. No merge,enqueuing,deploying orpublisher actions havebeen taken. Absolute session stop00:17UTC; remote runs may continue, owned local wrappers/checkouts must be removed.