Repository · handoffs
← Handoffs
View on GitHub ↗
id: hf-2026-10-02-reproduce-and-fix-interrupted-buildtest-projection-snapshot-readers url: url://handoff/handoffs/hf-2026-10-02-reproduce-and-fix-interrupted-buildtest-projection-snapshot-readers title: Finish reader cancellation ownership checks and implement the completion fix summary: Implement complete cancellation and ownership handling, then obtain passing gates and a green source PR. CHECKPOINT: six public bodies executed; five proved bugs, one close-contract question; tests and full evidence pushed, no production edits. created: 2026-10-02T19:10:22.869Z completed: null dependencies:
Handoff: Finish reader cancellation ownership checks and implement the completion fix
RE-VERIFY — 2026-10-05T14:06:18.183326+00:00: This is a snapshot. Recheck claims, fetch main, verify branch heads with git ls-remote and PR state/checks with gh pr view before acting. The supervisor alone owns merge, enqueue, deploy, publish and handoff-completion decisions. None occurred in this lane.
Mission
Reproduce and fix interrupted reset-snapshot readers following the owner's 2026-10-02 projection-feed outage request. B1/B2 are already on main through https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1234. B3/B4 still need a complete independently cancellable publication/read path and retained recovery causes. The supervisor ruled that uninterrupted detecting readers keep receiving the recovered reset; preserve that public behavior and the README, with no new API.
What was found and done
- Required handoff, PHILOSOPHY, TESTING and CODE_REVIEW documents, full repository README and linked lifecycle documents were read. Ownership and completion hypotheses were re-verified against a fresh own clone. Initial publication and recovery ownership are already merged through https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1187 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1189.
- Three baselines executed against production main 441a2fcb531323894e6c3130f7ceec3413d4dbaa: recovery cancellation, retained recovery cause, and initial publication cancellation. Each failed deterministically (0 passed / 3 failed). Both cancellation stacks show ConditionObject.await reacquiring the journal lock held by the publisher. The retained-cause assertion expected one initiating suppressed failure and found zero.
- A fourth body executed after rebase to main 35563d4f998b72b05503f2fe380a37bf2b138c2c: the late reader failed prompt cancellation while in noninterruptible ReentrantLock.lock inside requireNotAdvertisedResetRecoveryOwner, before either completion wait. Four cancellation/cause bodies failed; the two pre-start experiments below have also executed. Main's intervening change was only the tests from https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1258; src and build.kts match the first three baselines exactly.
- The saved latch/cause proposal applies by content, but is now proved incomplete for the earlier public lock acquisition. It has not been applied as production source. Other public availability/startup/page acquisitions after the guard also use the journal lock. A diagnostic prechecked-reader scenario pauses a named public reader after its guard, lets another reader hold recovery commit, then resumes and interrupts it. Root review added a callback-exited signal so it cannot pass by interrupting its setup gate. This diagnostic draft is unexecuted and is not proof.
- Seven public scenarios are pushed on the test-only branch: the four executed baselines, pre-start reentry and close experiments, and the saved close-drain scenario. The recovery baseline was strengthened using the existing public before-recovery-staging hook to force Condition wait entry before owner commit. Every previous assertion and bound remains. Pre-start reentry executed and failed: its requesting callback waits for the registered publisher it has not yet started. Pre-start close executed and failed its expectation of immediate refusal: close completed, then the newly started worker aborted for shutdown. All six bodies ran (0 passed / 6 failed); five demonstrate named bugs, while the close-refusal expectation remains unsettled. The README's close refusal covers a callback running on the publisher, while the pre-start hook runs on the requesting reader; assess the experiment before extending the contract.
- Earlier runs executed zero bodies: one local acquisition/compilation deadline, one local compiler OOM under the lane-chosen 512 MiB cap, and two remote pre-submission failures (full-clone tar construction OOM including .git/jars; tracked-HEAD export health RPC TIMEOUT before upload). No remote runId was issued, no checks were rerun and no remote resubmission followed. The corrected 1 GiB local launcher compiled and executed all four failures under a stricter memory-below-3.5-GiB admission rule. Test deadlines and counts were unchanged. These preparation outcomes are not behavioral results or a host-overload diagnosis.
- Separate root test-comprehensiveness and adversarial review sections are in the full progress record. Both remain open: current-head passing proof, later acquisition cancellation, cursor reads and lifecycle/ordinary-reader neighbors are not verified. No source PR exists and nothing is ready for merge. The lane stops at CHECKPOINT within its time box. All 49 captured runner/tracker PIDs were absent during cleanup; only conversation harness processes and the cleanup command matched L28.
- Salvage is preserved: old failing/passing logs, public tests, the complete source proposal, current deterministic failure stacks, the corrected fixture and the lifecycle inventory. Historical proposal results (2/2 passes on old bases) are not current-head gates. No satellite was closed or deleted; keep all older references until the useful material is landed. Full run2 written findings are tracked as diagnostics/L28-run2/progress-record.md because this repository ignores *-findings.md.
Relevant PRs / refs
PR states below were refreshed 2026-10-05T14:02Z; refresh again before acting. WIP run2 heads were verified with git ls-remote at this checkpoint.
| Repo | Remote branch/ref | Head SHA | PR | Contents and write-time state |
|---|---|---|---|---|
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L28-interrupted-snapshot-reader | 3c735db47ac74ca20b3df2692c1b30899bad79ef | none | Nine public scenarios: seven migrated drafts, proved initial cancellation test, unexecuted late-reader experiment. Test-only WIP, production source unchanged; never open all drafts as a ready PR. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L28-reader-analysis | a31830ec4895f6d918169cbbb146f43530d1610d | none | Full traces, passing logs, exact remote CLI bytecode, findings, source patch and resume state. Intentional diagnostics, not a source PR. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/interrupted-reader-not-snapshot-damage-2026-10-02 | 47634515c286379b74e68358790cbdf911ba7de9 | none | Historical seven drafts, superseded by observer-migrated WIP; preserve until salvage is landed. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/f-bt5b-defect-b-handoff-2026-10-02 | b3d2f4a4941e603701a88de1c6bc863fd322310b | none | Historical improved seven drafts with obsolete constructor. Preserve until salvage is landed. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L28-run2-public-baselines | c49ef726b9ebe28d02bc8672ba1cf3a3169db81d | none | Current-main seven public tests only; deterministic recovery-staging coordination added; Six public bodies ran (0 passed / 6 failed): five proved bugs plus one unresolved close-contract experiment. Production source unchanged. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L28-run2-reader-analysis | 6aeea72b203eb17a2a01e11cb78695fd69b20752 | none | Full current-main failure logs, complete progress record, source comparison, reviewed prechecked draft, inventories and unchanged source proposal; diagnostics only. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/resetpub/journal-owned-reset-publication | ce3dd64f56cb1b68573ee65aa3530326babccb8d | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1187 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/journal-recovery-publication-owned-by-journal | 9d3ce7b6557c1f7de0556072f032aaf620d81592 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1189 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/projection-repair-missing-child-sources-2026-10-02 | 3a7293e1d73b8a9be56a48d91ec06e8aa84c3042 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1198 | OPEN; checked 2026-10-05T14:02Z: 0 green / 5 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/feat/shared-helper-buildmaven-20261004 | 2f89862024bf38c0d2840bbe9b5470007833df39 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1228 | OPEN; checked 2026-10-05T14:02Z: 1 green / 4 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/startup-single-record-pass-20261004 | 7d1ef210f42f4f378fdcc6543e000013f3dc9a26 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1231 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/feed-authority-stability-20261004 | 63ce7a14394685147b693d5e7e52bfc35fe933f8 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1234 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/test/deterministic-runner-policy-storage | 4db77d73cd2c9f76fa0e8d0d244e6fd31a9349ae | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1235 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/RSV9-admission-reservations | ec1f67f0e2295e8342117101e30cc90118a40155 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1237 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/FPCF2-continuous-maintenance-event-gates | 4dd5f2731a4f4fff6a106b39a4ff6cbebc219295 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1248 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/SSH1-before-exec-channel-open | 321ced9e6ab6712585fd563ed5451852c5f6bb9c | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1249 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/SSH2-preserve-session-on-channel-refusal | 4c55b6f794b1dc48fa71a37c8d976f9a82663396 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1250 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/l052-reconnect-publication-ownership | 4cfe2b2e761125aa7eb4932efa44cf3f09aae525 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1197 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/snapshot-reader-keeps-its-continuation | 781068845ce0ad39a326e0603b09b0bf27308213 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1254 | OPEN; checked 2026-10-05T14:02Z: 2 green / 0 red / 2 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/interrupted-reset-requester-caller-local-page | 5fd63f2adbd6eb768181826ccfc94af5e8188b71 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1258 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/FLK3-reset-interrupted-callers | e47118fc102e31a6e7bb1d93172f907af93a1548 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1252 | MERGED; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
| BuildTestEmbedded | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/FLK8-publication-arrival | 8b444953cf0ee4d543a5e410a968f06f505e4413 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1259 | OPEN; checked 2026-10-05T14:02Z: 5 green / 0 red / 0 pending. Keep owner's PR; refresh before action. |
Historical artifacts: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/handoff-artifacts-fable-7dd6a506-2026-10-02 (not refetched; preserve). Local experiment SHAs are evidence, not remote source branches. No source PR exists for B3/B4.
Next steps
- Recheck live ownership before each update. A foreign live claim means finish current step, push and stop ALREADY_CLAIMED. Do not push to another lane's queued branch or dequeue it.
- All six requested bodies are now executed. First resolve pre-start-close behavior: should the preparing caller be refused immediately, or should close complete and abort the pending request without starting a worker after return? The current draft expects refusal, while the README names only running-publisher callbacks. Prefer ownership refusal if the preparer cannot drain itself, but obtain the supervisor ruling before adopting that expectation. Keep all assertions and record the terminal-close public evidence.
- Run the reviewed prechecked-reader draft through the public API and add equivalent cursor-read cancellation coverage. Capture the exact later lock boundary, then extend the invariant table and implementation plan. The saved latch proposal alone does not cover the proved early guard wait.
- Implement the complete reader/publication ownership fix only after the named conditions have deterministic public failing proof. Keep owner commits independent of caller interruption, retain the initiating snapshot cause, preserve close/drain/reentry behavior and uninterrupted recovered-reset pages. No new public seam, retry, timeout increase or reduced test bound.
- Flip the same current-main failing bodies to passing, then run touched tests and the listed close/staging/commit/restart/ordinary-reader neighbors from the exact source head. Complete both independent review passes, fix findings and re-verify. Use remote only under the supervisor's stated health/one-gate rules; otherwise bounded local one selector per invocation. Do not treat old logs or zero-executed runs as gates.
- Rebase immediately before opening/updating a main-targeting source PR, check current open overlaps and PR state, and lead its description with the owner's interrupted-reader symptoms and preserved-reset ruling. Footer: "🤖 Generated with Claude Code". Watch every required check with plain build-watchman, never --to-merged. Supervisor review and merge remain the final steps after green checks.
Operational knowledge
- Every shell starts with export PATH=/code/ws/bin:$PATH. Use a fresh own clone under /code/ws/L28; do not reuse sibling checkouts. cs is the working JVM Coursier wrapper; the repository scripts already use their pinned digest-checked launcher jar. rg is absent here; use the next available search tool.
- Remote tar construction includes .git and jars; if remote is authorized, export tracked HEAD bytes before invoking it. The two earlier launches never submitted a remote run. Do not retry a zero-executed failure as a debugging strategy.
- Local runner is run-selector.py in diagnostics/L28-run2. It checks safe memory and slot availability before the existing blocking jvm-slot helper; one selector per invocation, total guard at most 1800 s. Final lane selectors have shorter operation guards to respect the job time box, with all test timeouts unchanged. Kill only recorded PIDs. No full local suite or fat-jar build.
- Findings: /tmp/claude-1000/-code/4127b3f9-6050-446f-a28d-b0511dacd396/scratchpad/hq/out/L28-findings.md. Append around five minutes, report around twenty, and checkpoint every milestone. Diagnostics are intentional evidence and must stay out of the source PR.
- Do not merge, enqueue, deploy, publish or complete this handoff. Report-challenge auto-merges and conflicts with this lane's rule; tool friction is preserved in the evidence for supervisor action.