Handoff: Finish hosted verification and classify unmatched failures for BuildTestEmbedded SSH recovery

Written 2026-10-21 22:22 UTC.

RE-VERIFY: This is a write-time snapshot. Re-read https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1399 with gh pr view (state, headRefOid, mergeStateStatus, statusCheckRollup, mergedAt), and inspect native XML for the linked workflow runs before acting. The source was open/unmerged at f920cafd4d0c47a99e5e9257bd412bd74e1b4396 at 22:22 UTC.

Design decisions in force

Owner's 2026-10-21 RF5 brief:

• “Bring https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1399 up to date with current main and carry it to a landable, verified head (or show it is superseded).”
• “Keep the PR's existing WHY paragraph at the top of its description, unchanged in substance.”
• “Gates — hosted shards only”; “40-minute deadline”; “one failed shard never cancels siblings”.
• “Report first-attempt counts from the runner's own records (JUnit/XML)”; classify each failure as PR-caused (fix/re-gate), proven on the same main commit (leave main-owned), or infrastructure with zero tests (one redrive permitted). Unknown XML counts are not zero.
• “Never raise a timeout, reduce an iteration count, weaken/skip/delete a test, add a sleep or poll”; “no lane notes in the PR tree.”
• “No local builds or test runs (hosted shards only).”
• “Push the PR branch first (git push --force-with-lease), then mirror the same commit to wip/laneRF5.”
• “PROHIBITED ... merging, enqueueing into the merge queue, gh pr merge, publishing artifacts ... deploying anything, restarting any service, changing any timeout, re-running CI to see whether a failure goes away.”
• Throwaway hosted branches must use wip/gha-run/rf5-<n>, unique workflow names, Git installed before checkout, full fetch, and be deleted after terminal collection. Do not fetch DocumentationRepository architecture/philosophy documents. No real thread/heap dumps or dump tools.
• Dependency changes must probe the published POM. Main-owned files may only change as required by this fix; preserve historical main notes. No changed test timeout or iteration limit.

Mission and current result

RF5 has rebased and reconciled the old SSH recovery PR, but has NOT delivered a landable verified head. All focused scenarios pass first attempt. Seven full-suite shards and all four family shards exceeded the mandatory 40-minute job limit before XML publication. Two first-attempt failures on completed full shards remain unclassified. Do not call cancelled workers infrastructure-with-zero-tests or count automatic retries as clean first attempts.

The PR body was PATCHed at 22:11 UTC with the original first WHY paragraph preserved verbatim, current conflicts/pins, exact gate counts and honest incomplete verdict. Its requested Claude footer is preserved. One gh pr checks read was made about two minutes after final push; no second such read was made. Do not merge.

What was found and done

1. Main 932e63a0228d70ea6397fead0a9badbaf44ff0e9 had 109 newer commits, 66 touching the same service/SSH/observer behavior. Read matching messages and relevant patches. The old writer-close premise WAS superseded by https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/e24a21cfe8f2f798bf58ecd916ed0446330a1bb3. Main now retains uncertain dispatched work and the live owner. All three callbacks retain main's behavior; obsolete PR close/retirement fields and duplicate observer API were removed. Main's registered-opener interruption, spool recovery, dynamic physical-worker replacement, diagnostics and droplet IDs remain.
2. The whole PR is NOT superseded: main's SshExecutor.requireSession directly calls availableSession and cannot restore a previously connected but missing/disconnected generation. Three public real-SSH scenarios fail all nine native attempts on main: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39676181706. Two throw the exact not-connected error; observed peer closure loses the recovered line. Final focus passes all three on all workers.
3. Final code restores through the existing shared reconnect flight, preserving explicit close as terminal. Reconnect failure retains the entire original diagnostic cause and adds run/shard/droplet/endpoint/operation context. Main's pending-reader-open no-send diagnostic stays unchanged; two complete-message controls were updated.
4. Main's ordinary aggregate RuntimeException is classified by allocation text from any cause and can incorrectly requeue a mixed terminal failure. A terminal aggregate marker plus early classifier guard preserves FAILED. Public RunnerAttachPolicyException deliberately excludes main's newer physical replacement so the test actually reaches the final classifier. Corrected main baseline fails all three attempts and explicitly logs the requeue: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39678931735. Both cause orders pass in all three final focused workers.
5. Four PR-only old close/reassignment fixtures were reconciled to retained ownership: real queue commands write a lease before shell failure; assert retained owner, exact one uncertain write, accepted PASS, no premature deletion/terminal state, eventual explicit cancellation or real spool completion, and terminal policy aggregate. All ten new scenario scripts pin the actually published fake-droplet-service 0.0.20 (POM HTTP 200); production dependencies stay at main versions. No timeout/iteration loosening, added sleep/poll, removed/disabled/skipped scenario, main-record deletion, local build/test, publication/deployment/restart or merge occurred.
6. Independent test and source review passes are recorded in durable FINDINGS. Nineteen net files, ten new scenario scripts, no lane notes/workflows in delivered source. Clean worktree, no stash. All local source commits are remote in the verified PR and mirror; local origin tracking refs only fetched main, so git log --branches --not --remotes alone misleadingly lists remote PR commits.

Exact gate state

• Focused final: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39678916607 — 14 scenarios × 3 workers = 42/42 first attempts, 42/42 primary, 3/3 auxiliary build rules.
• Full final, standard sixteen-way partition: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39678964739 — all 5,013 files are present once in native manifests. Nine XML-bearing shards: 2,725/2,730 first attempts, 2,730/2,730 primary, 52/52 auxiliary. Shards 3,4,5,6,9,15,16 hit 40 minutes, no XML; counts unknown. GitHub annotations prove the job deadline. No re-drive was dispatched.
• Final related family: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39678983011 — 1,090 files / four workers; all four hit 40 minutes, manifest only, no XML. Executed count unknown; not a pass.
• Five non-green first attempts from full XML: three proven same native failure on exact main by https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39660254456: currentProjectionMaintenanceVacuumFailureKeepsCommittedPages (SQLite native step, 30,000ms), testBuildScriptsRefuseChangedResolutionBytes and testReviewBootstrapConnectionResetIsBounded (30,000ms). Leave these main-owned.
• Unclassified: testBuildScriptsRecheckAdoptedImageBytes (30,000ms at test line 15), and e2eWatchdogTimeoutCapturesThreadDumpBeforeHeapDump (marker index 2198 comes after first heap marker index 1513). Both pass primary after native automatic retry, which is NOT first-attempt success. Pure-main comparison https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39683333765 passes both first attempt on all three workers (6/6). Additional main control https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39682645742 passes 2/2 first attempt. Passing main comparisons do not establish category (a) or refute an intermittent main issue. No root cause is claimed for either.
• Preliminary family source e7bf698 was 623/629 first/primary across 628 files. Five failures were fixture/message corrections now covered green by final focus. Sixth ciShardExecutedTestCheckRejectsAVacuousShardRun rejects three unchanged main scripts all declaring main; pure main https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39682620569 fails all three native attempts. Category (b), leave unchanged. Do not confuse this preliminary receipt with the incomplete final family.
• Automatic repository PR workflow https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39678856486 completed SUCCESS by 22:21: 5,034/5,038 first attempts, 5,038/5,038 primary, 96/96 auxiliary. Its existing repository-owned 60-minute limit was unchanged. It is separate from the mandatory 40-minute RF5 gate; do not substitute it silently.

Automatic workflow update at 22:17 UTC: 14 XML-bearing workers are collected. Four additional first-attempt failures there remain unclassified: dynamicRecoveryAcrossThreeGenerationsKeepsLateReplacementExclusive (duplicate accepted counts), testBuildScriptsAvoidPreparationOptimizingCompilation (30,000ms timeout), testDispatchFailureHeldCallbackCannotKeepShutdownWaiting (assertion), and testRecoverableStreamingAttemptThrowsRetirementWithStorageFailure (stderr spill expected 0, actual 1). All pass native automatic subsequent attempts; none is called a clean first attempt. Existing exact-main full receipts did not reproduce these errors. These are separate from the two unmatched failures in the RF5 mandatory full gate. Do not report the head landable solely because primary records are green.

Automatic workflow completed SUCCESS by 22:21 UTC: native first attempts 5,034/5,038, primary 5,038/5,038, auxiliary 96/96. The four automatic first-attempt failures remain unclassified. The separate mandatory RF5 broad gates remain incomplete.

Relevant PRs / refs

All twelve terminal wip/gha-run/rf5-1 through wip/gha-run/rf5-12 branches were deleted and verified absent after their workflow snapshots and records were checkpointed. Their exact SHA/source/run metadata are preserved in evidence rf5-gates.json, so none carries unique source work. Deletion absence was rechecked at 22:22 UTC. No other checkout/worktree/stash was used. No code was deployed or artifact published; verified from the commands/actions performed in this lane, no production mutation was attempted.

Next steps

1. Re-verify PR, main and automatic workflow state. Read durable FINDINGS and runner XML. Current main was still 932e63a at 22:11. Do not assume the source remains current on pickup.
2. Diagnose the two unclassified first-attempt failures using the exact original native attempt evidence; establish deterministic reproduction before a code change. A native first-attempt comparison failure on exact main is required to call them main-owned. No blind rerun of PR workflows.
3. Determine why the forty-minute hosted jobs did not reach XML publication and meet the original broad-gate requirement without increasing any timeout or reducing test population. Cancellation logs show compilation/preparation consuming most of the deadline; that is an observation, not a root-cause explanation. Unknown counts do not permit a zero-test-infrastructure re-drive. Any new gate needs the same hosted-only rules and explicit failure classification discipline.
4. Reconcile only proven PR-caused behavior, preserve main's ownership policy. Push PR first and exact mirror second; refresh body preserving original WHY and footer; re-gate only as the brief allows. Do not merge/queue/publish/deploy/restart.
5. Update this handoff and final evidence as progress changes. Only complete it once all required verification is truly satisfied or the work is proven superseded.

Operational knowledge

Evidence tools/ includes hosted workflow generator, gates metadata, collector, XML first-attempt summarizer, and checkpoint tool. No tokens are stored: collector obtains a token from gh only at runtime. Download redirects deliberately omit Authorization. XML <test> is the primary/final record; <attempt-history>/<attempt number="1"> determines first-attempt success. Manifests are NUL-separated, not line-separated. Count test scripts and actual scenario records separately; several scripts contain additional scenarios. Full automatic suite partitions with scripts/ci-test-shards.bash; gates verify owned scenarios ran.

Shared REST quota was exhausted 21:28–21:41:46; reset header was obeyed, no alternate API quota workaround. Later gh GET calls hung for 120 seconds; bounded Python HTTP/1 metadata/artifact reads worked. Do not tight-loop on quota. One failure of a raw gh run watch can look like successful exit, so XML/run metadata remains authoritative. Source branch pushes must precede checkpoint pushes or the CI app may omit the required check. All throwaway names require the wip prefix or the app submits costly shared remote work.

Only this lane's own watcher PIDs may be stopped. Never pattern-kill shared processes. No full raw hosted console files or diagnostic dumps are committed; native XML, safe metadata, manifests and workflow snapshots carry the evidence. Current source checkout is /code/workspace/rf5/workspace/BuildTestEmbedded; local paths are disposable, use remote evidence as the handoff record.