Handoff: Reconcile write-contract rulings and finish recovery verification

RE-VERIFY — 2026-10-04 14:05 UTC. This is a checkpoint, not approval to merge. Query the PR head/checks and every named ref again. Handoff findings are hypotheses until supported by current public tests. No merge, enqueue, deployment, Maven publication, or handoff completion is authorized in this lane.

Mission

Finish the public-test and adversarial review of startup recovery for the operator's report of 41 old builds remaining active, including run c5c3e418 whose source directory was gone. Keep the selected source-loss outcome consistent during one service lifetime; adopt returned canonical metadata only on the next start. Preserve complete ordered projections, reader ownership, numeric values, and ordinary never-lost behavior. The supervisor owns final review, merge, and handoff completion.

Chain of findings

1. The original investigation reproduced interrupted-history ownership and unavailable-journal admission failures, but its sole combined 55-selector submission was 1 PASS/54 compiler failures. Its draft capacity changes were not proven. Original evidence and failing tests remain on the two historical review branches below; do not discard them.

2. The completed decision handoff hf-2026-10-03-decide-the-corrected-review1207-gate-and-journal-paging-merge-order and evidence at wip/L9-review1207-decisions-20261004 resolved the stale block: paging landed first, keep one PagedJournalRecords store with ProjectionRecordRows facade, and do not repeat the historical 55-selector gate. A later historical 76-selector gate was clean on its first attempts. The stale blockedReason/dependency was cleared after claim.

3. Paging merged at 068b5607bc15a53346d3ee6ad0bebd935aeba061 through https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208. Current main 1de45a68db4c34d27e12494b2df00877d012e527 is already an ancestor of the recovery PR head a05b6cfc21b7e894feb84bc2548189d4afbd6d83. Its description opens with the operator's request and describes restart-only adoption.

4. Source split commit 975f0e3b3 removed projection-unavailable append-queue implementation from this recovery PR and preserved it unchanged on feat/projection-unavailable-append-queue-20261003. Eleven selectors from the old gate are now absent from the PR. That refutes the premise that current recovery source still contains that queue; it does not prove the original full-capacity admission bug fixed. The queue code, tests, and insights remain useful salvage and need a separate disposition.

5. Current Actions https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37197426471 finished red: shards 2/4 each 895/896, and bld-build FAILURE. Existing public ownership tests fail deterministically on all three automatic attempts: removed-source validation throws NoSuchFileException because raw file identity bypasses IOException conversion; the merged/sorted owner list is immutable for singleton results and caller clear throws UnsupportedOperationException. Four other tests passed only after automatic retries; those are unresolved failures, not clean proof.

6. Own unchanged-source baseline executions produced 1 PASS/8 FAIL across five invocations, including deterministic ownership errors, archive/archived deletion/retention overwrites and completion-gated archived/deleted restart feed differences. Candidate fixes pair selected payload with its identity, convert ownership IO errors through the ordinary reader, return mutable owner copies, preserve selected execution-plan fields, and recognize earlier advisory fingerprints when reconciling returned archived/deleted records. Seven fixed executions passed and one failed on the initial candidate; the failure is the newly added unreadable control-feed equality, not a returned-valid-record failure. Valid final/active/deleted/archived restart variants pass.

7. Adversarial neighboring probes are 1 PASS/1 FAIL before an additional fix: repeated archive calls after refusal preserve bytes; a canonical record returning during the public advisory source-close boundary is overwritten by archive because startup stores that returned identity beside the accepted advisory payload. The startup identity correction flips both probes to 2/2 PASS on exact source head 1e027d60800c8eb1a984d32f297a790eff667eb3. Current watchdog deletion completion and idle-heartbeat gates also pass 2/2 first attempts on that source. Earlier/current head differences are explicit in the XML; no repeated-CI success is accepted as proof.

8. The new unreadable comparison reaches identical ordinary direct/error/owner/index behavior but compares different previously committed feed rows: ordinary control PENDING versus recovered FAILED. README's committed-feed contract says unreadable authoritative sources retain committed projection history. No assertion was weakened. The supervisor must decide whether to retain those distinct histories and correct this new fixture, or define a new public reset/retraction contract. Recommended: preserve committed history and compare ordinary error behavior plus each store's committed rows. Exact failing rows and stack are durable.

9. All candidate code, tests, and README work is recoverable as ordered patches and an exact Git bundle on wip/L10-recovery-checkpoint-20261004. The candidate-source.bundle requires durable base d6f60e6002ae16ce944de50d38a35e3dca24404d and restores exact source head 1e027d60800c8eb1a984d32f297a790eff667eb3. Do not apply the entire candidate: the concurrent V1 writing ruling below conflicts with its preservation/refusal expectations. A separate maintenance-diagnostics.bundle preserves diagnostic-only commits on that candidate. Product PR remains unchanged at a05b6cfc because the full remote gate has no verdict. Full remote run 05cafde8 was submitted from a clean worktree at 1e027d608, reported PENDING/PROVISIONING through direct public RPC, reached the existing 1,800,000 ms client deadline, sent cancellation, and is confirmed CANCELED. No test results were returned and no full-suite verdict exists. The watcher could not find it among stale HTTP projection rows; service cause remains unknown. No deployment/publication drift exists.

10. Separate own test-comprehensiveness and adversarial passes are recorded in the findings. They found and drove the fixes above but cannot yet approve the entire lifetime/concurrency surface, current retry-dependent CI cases, or removed queue admission work. Historical family retry stacks and old/current test diffs were recovered completely; the two tests now coordinate actual deletion completion and recovery-row release without changed limits.

11. Final candidate targeted invocations are 7 PASS/1 FAIL: neighboring return/repeated-write probes 2/2, watchdog/heartbeat 2/2, real started-test progress/deadline 2/2, dynamic recovery/continuous maintenance 1/2. Earlier candidate invocations were 7 PASS/1 FAIL; the retained unreadable feed equality fails on that earlier candidate. These are first attempts, not automatic retries. Three separate maintenance diagnostic executions fail the unchanged test (0 PASS/3 FAIL); one reaches a reader waiting in awaitProjectionChangeWrites, another never reaches its tail-refresh callback. The last pre-staging failure shows maintenance in baseline encoding and the writer idle, with zero completed writes and both release latches still closed. Capture the blocked writer during the separate page-budget failure and name its mechanism before a fix. Do not conflate these paths or call host load a cause.

12. The original later 74/76 family artifact and full old stacks are recovered. Watchdog now gates on completed deletion rather than provider-call entry; orphan recovery now gates startup completion and live rows rather than temporary ownership. Current orphan passed twice and watchdog once; that is not sufficient amplification proof that those old intermittent failures are eliminated. The removed idle-only lease probe does not exercise actual test progress, while current started-test and heartbeat-only contracts pass.

13. COURSE-CHANGING LIVE FINDING: satellite wip/fix1207i-2026-10-04 advanced to f37cdb310fbe795d95121120021969cb5e0b00cc with candidate implementation. Its committed repair ledger says “V1 replaces the reviewer's preservation rule only for actual record writes”; V2 requires canonical startup publication after an advisory last row and V3 ordinary-path equivalence. Its README/source allows a public record write to replace returned canonical bytes with the pinned FAILED/archive/deletion result; non-writing calls preserve them. That contradicts this lane's preservation/refusal candidate. These are newly observed instructions in another brief, not a ruling supplied to this lane: the supervisor must reconcile them before code adoption. Do not apply the whole L10 candidate. Salvage ordinary ownership IO/list fixes, selected malformed-plan behavior, startup feed/selection probes and fail-first evidence as appropriate. The concurrent branch records baseline 2 PASS/3 FAIL, not final verification. Its brief calls maintenance failure nonblocking, whereas this lane explicitly requires green CI; reconcile that instruction too. Existing production-sized startup, dynamic first-attempt ingestion, stalled-create and maintenance failures remain unresolved here.

Relevant PRs / refs

| wip/fix1207i-2026-10-04 | f37cdb310fbe795d95121120021969cb5e0b00cc | No open PR at inventory | Public probes, V1 canonical-materialization implementation and baseline 2 PASS/3 FAIL; reconcile differing write ruling before consolidation |

| Paging PR | b6fb5bce51f70be70131055e84184cf9199ea6d8 | MERGED; squash 068b5607bc15a53346d3ee6ad0bebd935aeba061 is on main | Integration decision resolved | | Crash-fixture PR | 1596e8d76412f46c1ade93c687faa8b311dc777c | OPEN; bld-build and shards 2/3 FAILURE | Read-only overlap audit; touches two crash fixtures, not continuous-maintenance test; keep under its owner |

Current incremental findings: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/L10-recovery-checkpoint-20261004/review-evidence/L10/findings.md . Original full archive: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/review1207-handoff-2026-10-02b/incident-notes/review1207/evidence.tar.gz . No deployment/publication drift was created; no production state was changed.

Next steps

1. Re-verify current state first. Confirm the live V1/V2/V3 write-contract and maintenance-gate instructions with the supervisor, then consolidate with the active fix1207i branch rather than landing conflicting implementations. Do not apply the whole L10 preservation/refusal candidate. Verify claims before any update; another drain session must not own the same handoff. Read latest current evidence rather than assuming the historical head still matches.
2. Recover exact candidate commits from review-evidence/L10/candidate-source.bundle on wip/L10-recovery-checkpoint-20261004 after fetching prerequisite d6f60e600. Verify the bundle then git fetch <bundle-path> HEAD:wip/resume-recovery-review. Archived candidate head is 1e027d60800c8eb1a984d32f297a790eff667eb3; use it for evidence/salvage and compare with the now-observed V1 ruling before selecting code. For diagnostics, verify/fetch maintenance-diagnostics.bundle after restoring that prerequisite. Read compressed XML/logs and patches; these files are evidence, not product changes to include in the PR. Settle the retained unreadable equality question before editing its assertion. Keep committed history unless the operator explicitly chooses a new public feed reset/retraction contract. Run all fixed returned-record probes on the final exact source. Full run 05cafde8 is CANCELED, not resumable proof; obtain a new full remote gate only after diagnosing the remaining failures and confirming service execution can progress.
3. Complete row-by-row lifecycle, receipt/history, numeric-render/decode, orphan/live-reader and real progress proofs. The historical progress probe updated only idle ingestion activity; distinguish actual test progress before changing the enforcer. The currently retained heartbeat-only test forbids extending a never-started lease indefinitely.
4. Diagnose both first-attempt failures from the older 76 family (watchdogAuthoritative404FailsRunWithDestroyedFleet and testProjectionRowOrphansAreRemovedWithoutAdoption) and current CI retry-dependent tests. No timeout increase, iteration reduction, skipped test, assertion weakening, or blind rerun. Record honest amplification counts if a failure cannot be reproduced; never declare retries acceptable.
5. Resolve the original full-queue admission item's disposition on its preserved feature branch. It is not a current recovery-PR implementation; no good test or root-cause insight may be silently dropped. Do not include the removed queue in a recovery readiness claim without proof or an explicit scope decision.
6. Perform two separate own review passes: test comprehensiveness and adversarial code review. Address findings. Run the full suite remotely from the exact source head before a shared-contract push. Rebase immediately before updating the still-open PR, retain the description's original motivation, and monitor all required checks plus kotlin.build (remote); absence is not green. Do not merge, enqueue, deploy, or complete the handoff.

Reusable / operational knowledge

• Host PATH must start /code/ws/bin. Fresh own clone is /code/ws/L10/BuildTestEmbedded. Shared container ceiling6GiB; never full local suite or large fat-jar build. Above4.5GiB, wait before another heavy command. Only one own small targeted local invocation at a time; at most two small selectors, no changed test limits. Guard memory numerically before launching.
• scripts/test.bash pins/verifies its own JVM Coursier and avoids the wrong-architecture system binary. It overwrites JAVA_OPTS at line239, so an externally supplied heap cap is ignored. The exact kompile-executionenvironment0.0.37 also copies JAVA_OPTS into compiler children. A client-only -Xmx512m environment cap caused a compilation-only OOM; restored baseline compiler settings passed. Cap the client directly via java arguments, leave the compiler environment equal to the baseline, and preserve BUILDTEST_TEST_WORKSPACE and its normal own-clone cache path. Use set-e so a failed rebase cannot continue into a build.
• The remote ownership baseline reached health=OK, then spent22min in submitted-run polling without a transition/verdict and was terminated through its own shutdown. Both direct listBuildRuns and listBuildRunsPaginated RPC probes timed out at30000ms; HTTP /api/runs explicitly says outOfDate=true, refreshFailing=true. These are symptoms; the shared service's root cause is unknown. No restart or production change was attempted. Remote full-suite proof remains outstanding.
• Existing orphan test now releases its callback before waiting for the completion observer and closes it in finally. One current-head first attempt passed; that does not establish the old flake eliminated. Public completed-feed reads can return before background startup finishes; comparison fixtures must observe completion on both independent stores.
• Current CI artifact XML records first attempts separately: testStartupRecoveryProductionSizedHeap fails360000ms then passes; projectionChangeFeedReadRemainsAvailableDuringContinuousMaintenance fails twice then passes; testSingleDropletDynamicRecoveryReplacesLostRunnerWithQueueRunner fails twice with FileLockInterruptionException then passes; e2eStalledCreateReleasesAdmissionAndRelaunches times out60000ms then passes. Full stacks are compressed on the evidence branch. A host-capacity claim is not a diagnosis.
• All long commands have tracked sessions and hard limits. Kill only owned exact PIDs, drain terminal notifications, and audit processes before final reporting. No generic Java kill. Build-watchman plain mode only, never --to-merged. Installed kotlin-build-ci-test app was never dispatching across mature main suites; no rerequest was issued. Report-challenge auto-enqueues/merges and is forbidden by this lane; tooling friction is preserved for the supervisor instead.

Checkpoint and remaining gate

CHECKPOINT, not READY_FOR_MERGE. First reconcile the concurrent V1/V2/V3 and maintenance-gate rulings. Consolidate useful independent fixes, reproduce and fix remaining CI cases, decide the unreadable comparison contract, finish lifecycle/receipt/numeric/concurrency proofs, run the full remote suite at the exact final source, consolidate useful satellite probes, then rebase and update the still-open recovery PR. After required checks including kotlin.build (remote) are green, only the supervisor may review and merge. The removed full-capacity queue must receive an explicit separate disposition; it is not silently considered fixed. No satellite was closed or deleted. No merge, enqueue, deployment, Maven publication or handoff completion occurred.

The complete concurrent repair ledger observed at f37cdb310 is preserved in review-evidence/L10/concurrent-fix1207i-findings.md.gz on the checkpoint branch. It is evidence of the competing brief, not approval of its unfinished implementation.

Final diagnostic-only head is 18d526653535c2c44e66bf949f054bd319b1621f, available through maintenance-diagnostics.bundle after restoring candidate 1e027d608. Candidate SHA256 f0c1de0c2ad79e5c8bd8266aab2cbbb8159622c931887458eaee19363fd625d9; diagnostic SHA256 209c4b44a1a4660e2f35954df1a25c0bf98be04ac61f127e5be5a68b2da4f24c. Verify bundles before recovery. Do not include bundle/log/evidence artifacts in a product PR. All tracked own test/RPC/watch processes ended; full remote run cancellation is verified.