← Handoffs

Repository · handoffs

Handoff: Resolve the closed-read contract and finish recovery verification before updating the PR

View on GitHub ↗

id: hf-2026-10-02-finish-lostterminal-recovery-verification-and-open-its-pr url: url://handoff/handoffs/hf-2026-10-02-finish-lostterminal-recovery-verification-and-open-its-pr title: Resolve the closed-read contract and finish recovery verification before updating the PR summary: Resolve closed-read and selector scope, finish lifecycle and full remote gates, then update the open recovery PR. Preserved checkpoint has seven heap and four neighbor first passes; the PR remains red and unpublished source is saved as an exact patch. created: 2026-10-02T11:35:32.820Z completed: null blocked-reason: Local family remains 74/76 first-attempt and both startup cases need CI retries; storage integration follows https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208. dependencies:

Handoff: Resolve the closed-read contract and finish recovery verification before updating the PR

Written: 2026-10-05T14:53:43.311220+00:00.

RE-VERIFY: This is a write-time snapshot. Recheck handoff-cli claims and handoff-cli get, gh pr view <URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, git ls-remote, and the actual buildtest run state before acting. A live query wins over this document.

Mission summary

The original operator reported 41 old builds still displayed as active, including c5c3e418 after its authoritative records were missing. This handoff asks to finish recovery verification and open the PR. The lane brief additionally requires integrating the landed storage work, naming the red CI selectors, completing the original 76-selector first-attempt proof, and leaving final review/merge to the supervisor. No merge, enqueue, production deployment, Maven publication, or handoff completion is authorized for this lane.

What was found and done

  1. Claimed only this handoff as fable-hq-20261004. Storage integration in PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 is merged at 068b5607bc15a53346d3ee6ad0bebd935aeba061. PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1188 is merged. Recovery PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1207 remains open at a05b6cfc21b7e894feb84bc2548189d4afbd6d83, conflicted with main, with red bld-test-shards 2 and 4 and their aggregate. Latest GitHub merge state is UNKNOWN; no PR/source update by this lane.
  2. Consolidated the current recovery branch's final tree to integrate with main 441a2fcb531323894e6c3130f7ceec3413d4dbaa. Resolved startup load and reset-snapshot conflicts, retaining main's validation, digest and seek-offset contracts and the recovery branch's leased encoded rows. The first targeted build exposed a duplicate declaration from that resolution; it was corrected before the actual unchanged selector ran. The current source tree is preserved as rebased-source.patch on the remote evidence branch; it has not replaced the PR branch because the shared-contract full remote suite has not run.
  3. Downloaded all four CI XML artifacts from https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37197426471. Counts: 3,580/3,590 first-attempt passes; 3,588/3,590 final passes. The final failures are uploadOwnershipQueryRepairsOnlyItsRun and uploadOwnershipRemovalDuringValidationTerminates. All complete traces and attempt histories are preserved on the evidence branch.
  4. Proved immutable owner-list behavior through the unchanged rebased selector (first failure, 4,386 ms) and a broader public empty/singleton/multiple mutation selector (first failure, 4,584 ms). Applied ArrayList to the final owner return. The broader unchanged selector now passed its first fixed attempt: 1 passed / 0 failed, 3,268 ms. The removal mechanism is also in this branch: it replaced main's I/O-classifying currentRunSnapshotIdentity with a raw canonical-file identity read. The prior hypothesis of a main defect is refuted by that content comparison and by five unrelated branch runs passing this selector. Restored readRunRecord around both canonical-only identity reads without broadening catches or adding retries. The unchanged removal selector passed its first fixed attempt, 1/0 in 2,985 ms. Both ownership guards passed FIRST on the later disk-store implementation, as did accepted-return ownership and archived-adoption/deletion indexing: four current-head passes / zero failures. Read the preserved XML and head records.
  5. The old verification scope changed before this lane: commit https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/975f0e3b3dd02ba7ff02d781ee8ed143dcf7ff21 separated unavailable append-queue work into feat/projection-unavailable-append-queue-20261003. Of the original 76 names, 62 exist in this recovery tree; 14 do not. The 62 recorded CI results are 60/62 first-attempt passes and 62/62 final passes. Missing selectors and exact attempt counts are preserved. This is not a completed 76-selector proof. Do not silently adopt the current PR description's narrower 48-selector claim as satisfying the original brief.
  6. The original family’s startup heap failure is independently observed on five unrelated branches; the module-cache failure is independently observed on four. Continuous-maintenance fixture fixes are already merged in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1240 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1248 and included in this rebase. Independent tail-acceleration and torn-manifest first failures are now recorded on unrelated branches. Cancellation-before-droplet, single-droplet recovery and stalled-create failed on this PR but passed all 11 examined independent first attempts; their mechanisms remain unclassified. The missing coordinator backstop selector was explicitly removed by branch commit 4266e62e0bae535077011079728b4c0c32153fc0 and is not equivalent by assertion content to current main tests. Full first-attempt proof remains incomplete.
  7. Adversarial review proved that accepted advisory outcomes retain full decoded runs and raw bytes for all archived history. The new public-restart probe seeds 2,048 archived outcomes with 64 KiB notes under -Xmx128m and fails 3/3 recorded attempts with OutOfMemoryError in readSourceLossOutcome during startup. Implemented service-owned paged disk rows and row-index snapshots using the merged storage API, replacing lifetime payload caches and eager all-history feed JSON. The first fixed build failed compilation at the synchronized block’s final if-expression; explicit Unit corrected it. The unchanged heap probe then passed its first executed fixed attempt, 1/0 in 42,782 ms at source 88e1504a on main d4d7d718. Scalar owner metadata and the ordinary empty-selection feed fast path are preserved. Six additional bounded heap proofs finished with FIRST passes in 40,984/28,747/30,019/31,302/31,193/30,947 ms. Total fixed proof: 7 FIRST passes / 0 failures against 3/3 baseline heap failures, with unchanged count/payload/heap/retained/deadline bounds. Static eight-row source-loss contract coverage and lifecycle plan are preserved; runtime review is incomplete.
  8. The main-control remote client exited 1 after getBuildRun timed out and its automatic close-time cancelBuildRun lost a response. The final /api/runs query lists 194360f0 PROVISIONING/scheduledAt=1791206292654, with outOfDate=true and runMembershipReconciled=false. The bounded watchman ended at its outer deadline with RESULT: INTERRUPTED, without a terminal verdict. Its later 100/100-completed message is refuted: API page 1 contains only FILTERED_OUT rows, which its status counter incorrectly counts as completed. The selected page returns 503 and a read-only SSH durable-record probe was refused; Fabric certificates are absent. Do not count any of this as a test verdict or submit another remote run until the prior one is resolved.
  9. New public sourceLossArchivedSelectionSurvivesClose failed 3/3 in 2,941/3,054/2,885 ms: after peer deletion, an archived selected FAILED record becomes CANCELED when the older service closes. Pre-store 7f5d4ea75 already clears the same maps at close, refuting attribution to the new store. This is a saved contract probe, not a completed fix. Resolve whether closed per-run reads must retain the selected generation or close ends instance selection; the recommendation is to retain existing public read guarantees, but no storage change is justified before settling that reading.
  10. Historical source and evidence are preserved. The fresh lane evidence branch contains the rebase patch, claim-time handoff, live PR JSON, full CI XML, exact selector audit, and failing unchanged ownership proof. It now includes all 20 carried-forward raw independent XML shards, additional first-failure traces, fixed heap XML and the failing closed-read probe. The incremental findings file was initially ignored by *-findings.md; that omission is corrected and explicitly tracked. Source patch was independently compared byte-for-byte with its named base/head. No old branches/PRs were closed. The unreachable inherited repair experiment and its notes are preserved under inherited-repair-evidence on the evidence branch and removed from the product diff; no executable test was removed. The reason-first PR description draft is also preserved; it has not been posted.

Relevant PRs / refs

Ref Remote head / merge commit Write-time state
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1207 a05b6cfc21b7e894feb84bc2548189d4afbd6d83 OPEN; conflicted; shard 2/4 and aggregate red; untouched by this lane
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 068b5607bc15a53346d3ee6ad0bebd935aeba061 MERGED; storage integration now on main
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1188 982407992a49ae35f0b9859e94fbf4c617a768ff MERGED
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/feat/projection-unavailable-append-queue-20261003 73eb2dc2e3d605bce83679daec6e68f6a85fb090 Separated queue source/tests; keep pending supervisor scope decision
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/journal-unavailable-appends-and-lost-source-terminal-outcome a29597087d2d3c15cac08989ec21186b6fad29e1 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/lostterminal7-recovery a05b6cfc21b7e894feb84bc2548189d4afbd6d83 Current recovery PR source; keep
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L49-lostterminal-evidence 001109e3f243204429d54ed3732120366845eb91 Current lane evidence and resume patch; keep
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte-lostterminal-2026-10-02 5c0414b769fe3f01123fa14788118ca013fcb0da Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal-evidence-2026-10-02 804865d6d1cb07a39ec2092d4a6cd3897ecc7787 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal-handoff-2026-10-02b e475b8a6d8ce94856ba7ac81019d6aa651025cb9 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal2-evidence-2026-10-02 aafda37db09d186aa4a65c784a38e34f11f50c9b Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal2-handoff-2026-10-02b d8d973bfac3ced0d632b0f18840ffcb86005c46b Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal3-append-green-2026-10-02 79e09af0c503d489058604066181430bca673465 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal3-evidence-2026-10-02 309274620bbd0e3c454a0b3d193842d362558ca5 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal3-handoff-2026-10-02b a29597087d2d3c15cac08989ec21186b6fad29e1 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal5-handoff-2026-10-02b 46301b75fa97a06407dfb545e3cbdef6fe1bd23e Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal5-source-2026-10-02b e717b07ad06f5d7848ff8b4b4a4c54f505e380d3 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal6-handoff-2026-10-02b 1a61c77f8ba617988a448338daad96a1c8b32692 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal6-source-2026-10-02b 6ce16b20b0beef75ade39f6d188465626c1b4bbe Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal6-source-2026-10-02c 0a12bbcbf03758d8319d8b608eeee0ac556918bf Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal7-handoff-2026-10-02b 4f98f180991ce2a5dccaf3d38aa366804c28c370 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lostterminal7-source-2026-10-02b e2e3fc5b477b997ae0315db0e9fa6526cbeb3458 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lt1207fix-handoff-2026-10-03 3dc71e6d9dd80313b00b7368f100de390b48d04c Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lt1207fix2-handoff-2026-10-02b 23cd35f3da4e37f7dc0a8c396c2d3177bb266531 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/lt1207fix2-handoff-2026-10-03 3b0c19707ec1a7814b9465270fc8e1d768ee4c08 Historical source/evidence; preserved, not current verification
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1240 77137a504e15b44894759f5e84e7736c75c1ae56 MERGED; head 60e950ae84f627d854f90ec115f918179d5e3c4b; included on current main
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1248 fd3efca04da36d7aceb2a30748064613254655ae MERGED; head 4dd5f2731a4f4fff6a106b39a4ff6cbebc219295; included on current main
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1244 873257e520eb0e17b254f87b8f7412986df697c4 MERGED; head 9dc242d84207aa3e53a5aeba198d138e38760acb; included on current main
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1258 35563d4f998b72b05503f2fe380a37bf2b138c2c MERGED; head 5fd63f2adbd6eb768181826ccfc94af5e8188b71; included on current main
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1259 d4d7d7185798086bb8fb24103caa84411bfeffc2 MERGED; head 8b444953cf0ee4d543a5e410a968f06f505e4413; included on current main

Next steps

  1. Re-verify current state first, including ownership claim. Another live claimant other than fable-hq-20261004 means finish/push the current checkpoint and stop with ALREADY_CLAIMED.
  2. Resume from incident-notes/L49/rebased-source.patch on main d4d7d7185798086bb8fb24103caa84411bfeffc2, then fetch/rebase current main. Preserved local source head 56df4b3222250f511e53bd6e281fffb653ee32b7 includes both ownership fixes, the fixed heap probe/store, and the 3/3-red closed-read contract probe. The first fixed heap pass was at parent 88e1504a with identical production code. Source is preserved as a complete binary diff on the evidence branch; the actual source branch/PR is unpublished because the full remote contract gate has not run.
  3. Heap confidence results are terminal 7/7 FIRST passes. Four current-head neighbors (accepted-return ownership, archived adoption/deletion index, ownership query and removal during validation) passed FIRST, 4/0. Continue the remaining ownership/lifecycle/error selectors against the disk-row implementation. Earlier broader owner-list mutation passed 1/1 before the store change; current query/removal and two peer-selection/index guards now cover the later store, with remaining hostile-caller lifecycle/failure gates open. Review owner-query metadata paging, ordinary feed fast paths, storage-error classification, borrowed feed reads through close, peer retirement/deletion and restart cleanup; retain original count/heap/time bounds.
  4. Resolve the closed-read contract first: must archived source-loss selections remain stable after close even after a peer removes the advisory, or does close end that instance selection? The inherited implementation does the latter; the new saved probe asserts the former. Do not remove/weaken it or change the implementation by guess. Then resolve the original 76-selector scope explicitly with the supervisor: restore/integrate the queue family in this handoff, or treat the preserved queue branch as a separate pending item and define a source-loss family covering all current changes. Missing selectors are not passes. Preserve independent pre-existing failure evidence and classify each other first-attempt CI failure by name.
  5. Finish separate test-comprehensiveness and adversarial code reviews. Resolve the existing remote run 194360f0 through authoritative state and watchman; its API row is stale. Run a full remote suite from the exact source head before a shared-contract source push. Record first-attempt and final counts separately, check actual buildtest terminal state, then check the PR is OPEN and rebase immediately before force-with-lease update. Update the reason-first description honestly and retain its required footer. Watch required checks in plain watchman mode. Supervisor alone reviews and merges; never pass --to-merged.

Operational knowledge

  • Lane files: /tmp/claude-1000/-code/4127b3f9-6050-446f-a28d-b0511dacd396/scratchpad/hq/out/L49-findings.md and L49-report.md. Append findings as work advances; report RUNNING every ~20 minutes to keep the claim alive.
  • Host workspace is /code/ws/L49. Export PATH=/code/ws/bin:$PATH every shell. Current main's scripts already bootstrap the verified JVM coursier jar; no shim was needed.
  • Installed jvm-slot has no intrinsic admission timeout or after-admission 4.5-GiB guard, contrary to the brief. Use a tracked outer deadline and check memory before each heavy start; do not edit the shared helper.
  • Shared host ceiling is 6 GiB. Local selectors only through /code/ws/bin/jvm-slot scripts/test.bash --local --test ONE; no full local suite or large local fat jar. At most one remote submission. All long waits need tracked PIDs and hard limits. Never pattern-kill sibling jobs.
  • Evidence is at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L49-lostterminal-evidence/incident-notes/L49. Historical pickup body is preserved there for earlier conclusions and refs; treat those conclusions as hypotheses under PlanRepository handoffs README, Triaging handoffs.
  • Required policy reading: PlanRepository handoffs README (format, triage and salvage), PHILOSOPHY.md (reproduce before fixing and refuted hypotheses), TESTING.md (public API, exact failure conditions and first-attempt flake policy), CODE_REVIEW.md (host pressure is a symptom; never relax tests). No timeout, iteration, assertion, retry or fallback changes are authorized.

Resume commands and unresolved gates

Clone this repository fresh into the next lane workspace and retrieve incident-notes/L49/rebased-source.patch and resume.txt from the evidence branch. Reconstruct the committed source tree on the named base, then rebase current main before each build:

export PATH=/code/ws/bin:$PATH
git switch -c wip/recovery-resume d4d7d7185798086bb8fb24103caa84411bfeffc2
git apply --index /path/to/rebased-source.patch
git commit -m 'Resume recovery verification'
git fetch origin && git rebase origin/main
# one selector per local invocation, tracked outer deadline and memory check
timeout 600 /code/ws/bin/jvm-slot scripts/test.bash --local --test review1207gAcceptedReturnRefreshesUploadOwner --log /tmp/accepted-return.xml
timeout 600 /code/ws/bin/jvm-slot scripts/test.bash --local --test review1207gArchivedAdoptionUpdatesDeletedIndex --log /tmp/archived-index.xml
timeout 600 /code/ws/bin/jvm-slot scripts/test.bash --local --test uploadOwnershipQueryRepairsOnlyItsRun --log /tmp/owner-query.xml
timeout 600 /code/ws/bin/jvm-slot scripts/test.bash --local --test uploadOwnershipRemovalDuringValidationTerminates --log /tmp/owner-removal.xml

Read the latest neighboring-selector ledger first and avoid repeating a proven exact-head pass without a new reason. Resolve the closed-read contract and missing-selector scope before interpreting those probes as acceptance gates. Source-loss borrowed-feed close/re-entry, returned-record mutation, startup/storage-failure cleanup and generation cleanup/restart coverage remain required. Three original CI mechanisms remain unclassified. After run 194360f0 is authoritatively terminal, run the full exact-head suite via scripts/test.bash --remote --log /tmp/recovery-full.xml, track its run with bounded build-watchman --run ID, and validate the actual terminal API/durable state before trusting the count. Only after all failures are fixed at cause, reviews approved and the full gate passes may the source be pushed to the still-OPEN recovery PR branch with force-with-lease, following an immediate main rebase. Do not enqueue, merge, deploy or publish.

Checkpoint state

CHECKPOINT: partial source and all fail/pass evidence are preserved on the lane evidence branch. Actual source/PR is unpublished because its full remote shared-contract gate has not run. Both separate review passes are written up and remain unapproved for the named lifecycle, metadata-scaling, contract and scope gaps. Keep all satellites; none is proven redundant by content. No process from this lane will be left waiting when it stops.