Handoff: Diagnose the partial-write service recovery issue and finish the draft CI audit

Snapshot time: 2026-10-09 12:10 UTC. At this snapshot, the pull request is still a draft at head d8a8e516e7df9e684557d8ba03bcc4c76939d5d7, rebased on main at 44842fcf0cd380c7e27d5a4381f0a7ac45cfc04e. Full CI is in progress in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37927817157. Update this snapshot after final CI and cleanup.

Re-verify before continuing: Check the pull request with gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1370 --json state,isDraft,headRefOid,statusCheckRollup, inspect raw Actions run XML, and run git ls-remote for every branch. Everything below is a snapshot from the time it was written.

The user requested the sshdisp3 brief: finish all four first-attempt shard audits, compare each failure with recent main, reproduce and fix each failure attributable to this change, and pass a 10/10 gate plus neighboring tests before rebasing, pushing, and updating the existing draft. Do not mark the PR ready, enqueue it, merge it, or deploy it. Do not increase timeouts, reduce iterations, weaken or skip assertions, use artificial load, mocks, or reflection. Handle dependency fixes as separate work; stop with evidence instead of adding a downstream workaround. Use only the actual kompile-cli 0.0.112. The hard local deadline was 12:50 UTC. Before finishing, clean up this lane's clone, cache, and owned processes.

The original production dispatch and owner-marker motivation, along with the historical gates, are in the preserved draft description and predecessor handoff: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-sshdisp2-evidence/handoff-artifacts/sshdisp2/sshdisp2-HANDOFF.md. The marker remains owned by SshExecutor under its lifecycle lock. Driver release and reconnect preserve it, a confirmed reset reconciles it, and terminal owner close clears it and prevents late reinsertion. This continuation changed only five test fixtures; it made no runtime product changes.

Verified findings

The original CI run, https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37919682950, is terminal RED. Shards 1, 2, 3, and 4 had first-attempt pass counts of 992/994, 991/994, 997/1000, and 993/994. Their final pass counts were 993, 993, 999, and 993. Overall, 3973/3982 passed on the first attempt and 3978 passed in the final results. There were nine first-attempt failures, five retry-only passes, and four final failures. The evidence branch contains the full messages, stacks, and classification.

Three failures also appear on main:

• Concurrent finalization has the same caller-completion assertion: main run 37904488977 failed on caller 2; the candidate failed on caller 3.
• Projection startup has the same 180-second rewrite-child assertion in main runs 37912082379, 37910377334, 37910376946, and 37919473930. The captured child output differs.
• Snapshot verification interruption has the same exact message and stack in main run 37907737564.

Ten raw main runs were compared, with four XML files per run. These failures count as cross-PR under the brief's landing rule.

Five attributable fixture issues have been repaired. Two interrupt tests named old SshExecutor wait lines. The unchanged targeted pair first failed 0/2; changing only 1301 to 1333 and {751,776} to {772,797} then passed 10/10 for each test, plus three neighboring tests (50/50 first attempts), in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37925835708.

Three fake runner handlers read the queue immediately at launch while registration starts asynchronous writers. The PR's added wc -c round trip exposes this assumption. The forced ordering makes the first queue-size read wait for runner entry. All three original handlers failed on their first attempt, 0/3, in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37925681380. The repaired fixtures run the actual provided Bash printf/dd command, wait for its successful exit, and signal a write-completion latch before runner consumption. Replacement latches are keyed by droplet working root. All old assertions, timeouts, and iterations remain unchanged.

Ten runs each of the three repaired queue fixtures, plus the partial-write, late first-write, and session drop fixtures, passed 60/60 on their first attempts in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37926692506. An independent test and code review found no must-fix issue. Its precision note: the interceptor callback launches real Bash and returns its actual output over real SSH; it does not use the provider's default shell branch.

The partial-write service failure remains unresolved. The new e2eDispatchPartialQueueWriteKeepsLiveDroplet test first failed its original terminal-state assertion, then passed on retry. An unchanged local run passed 1/1, the unchanged remote baseline passed 10/10 actual test rows, and the neighbor gate passed 10/10. These passes do not fix the failure. The auxiliary baseline job was RED because root requested two nonexistent neighbor selectors; those selectors receive no gate credit. The correct neighbors ran in the 60/60 gate.

Two separate real-SSH diagnostics prove a pending-channel issue. The first withholds exactly one real OPEN_CONFIRMATION, observes the public read caller waiting in Channel.sendChannelOpen, retires the old session, and reconnects successfully. The SshExecutor diagnostic failed 0/1 after 10 seconds while its reader remained parked. A raw-JSch-only diagnostic repeats the same condition without SshExecutor and failed 0/1 (readerFailure=null). At pinned JSch 0.2.18, the source parks once for the configured 30 seconds at Channel.java:823, sets connected=true only after confirmation at line 840, and disconnect returns at lines 632–634 without notifying when connected=false. This proves a dependency defect under that forced ordering. No CI reader stack proves that the original service failure used this path.

A separate full-service pending-spool-open diagnostic is in progress. It is intended to test that exact consumer contract while retaining the original partial-write queue and recovery assertions. Keep it on a separate WIP branch. The diagnostic is not complete, and the original CI failure is not attributed to this issue.

Branches and artifacts

The draft PR is OPEN at d8a8e516e on wip/sol-sshdisp2. It is rebased on main 44842fcf0. There are 30 incoming files covering projection setup, readers, cache, build-rule folding, and tests. The diffs for SshExecutor, BuildDriver, and DynamicDispatch are empty. The reviewed service hunks do not intersect the queue-owner, reconnect, dispatch, or result-collection paths, so historical gate credit was retained.

The five accepted fixture blobs exactly match the tested 0a8e3516d head. The accepted clean branch excludes the intentionally failing newly added diagnostic modules; their WIP branches retain them. No existing test was removed.

Nothing was deployed or published, and nothing was enqueued or merged. No production state needs reconciliation. The draft body was updated with gh api -F body=@file and its exact readback was verified. Full CI watchman 0.0.21 was started with --all-checks, a 240-second cadence, and a bounded 2480-second run. It reports that the installed kotlin-build-ci-test app is still not dispatching. No coordinator, WUI, or production-host changes are authorized.

Next steps

1. Re-verify the current draft head and the new full-CI run. Download all four raw XML artifacts and audit first attempts separately from retry results. Fix any newly attributable failure only after a deterministic public-API RED. Record exact main matches separately.
2. Continue the separate full-service pending-spool-open diagnostic and inspect its actual RED result when available. Keep forced-contract proof distinct from uncaptured CI ordering. Capture the exact public recovery-reader state instead of inferring it from a terminal timeout.
3. Take the raw-JSch public-API reproducer into an authorized upstream-dependency task. Do not add an SshExecutor or BuildDriver timeout or another downstream workaround. After an upstream fix and release, reproduce the original service failure with the same public reproducer, then verify RED-to-GREEN and 10/10 plus neighbors.
4. Keep the draft NOT READY until the original partial-service failure is repaired or refuted with evidence. This brief grants no authority to mark it ready, enqueue, merge, or deploy.

Commands and infrastructure notes

Make a fresh clone under scratchpad/workspace/sshdisp3-bte. Read README, AGENTS, and the linked TESTING and PHILOSOPHY documents first. Put "$HOME/bin" first on PATH for Coursier; the system launcher is broken. Run git fetch origin and git rebase origin/main before every build.

Run the selected diagnostic from its WIP branch with:

scripts/test.bash --local --test e2eJschDisconnectReleasesPendingChannelOpen --log jsch-red.xml

Use the same pattern for e2eSshReconnectReleasesPendingChannelOpen. A single short local run requires a bounded 10-minute acquisition of one of the three build slots. Multi-run gates use the repository Actions-machine helper, not shared coordinator capacity. Never hand-roll an unbounded wait or blindly rerun CI. GitHub polling cadence must be at least 120 seconds. Remove only this lane's checkout and cache entries; do not touch other workspaces or slots.

The central handoff claim and RUNNING report uploads both failed or timed out during the urlresolver bytecode fetch (ConnectionClosedException / ReadTimeoutException); acceptance is not assumed. Raw logs are local; sanitized challenge and evidence are checkpointed remotely. The report-challenge publication was not invoked because it automatically enqueues and merges, both explicitly prohibited by this brief.

The existing central handoff ID is hf-2026-10-09-finish-ssh-queue-owner-lifetime-gates-and-review-the-draft-change. Its authoritative URL is url://handoff/handoffs/hf-2026-10-09-finish-ssh-queue-owner-lifetime-gates-and-review-the-draft-change, and its WUI page is https://www.handoff.wasmserver.com/handoffs/hf-2026-10-09-finish-ssh-queue-owner-lifetime-gates-and-review-the-draft-change. This GitHub record is the latest durable continuation if the central update is not confirmed.