Handoff: resolve the additional validation failures and finish client-close verification

Written 2026-10-21 12:09:26 UTC.

RE-VERIFY: Query https://github.com/CodexCoder21Organization/UrlProtocol/pull/748 with gh pr view ... --json state,headRefOid,mergedAt,statusCheckRollup. Verify every ref with git ls-remote; native XML, not process exit or retries, determines first-attempt results. This is an active snapshot; final cleanup and exact-head CI counts will be added if available within the task limit.

Design decisions in force

Original request: “We need to investigate why https://my.externalbrain.wasmserver.com/ is failing to load and fix the underlying problem.” Assigned p107 task authorizes resolving the new client-close completion test and the prescribed23×10+18validation. It supersedes r102’s host-only edit scope for this new test. No edits to tests on main; no new production/test timeouts, sleeps or iteration reductions; no merge/enqueue/publish/deploy or remote buildtest submission; no user questions. Commits carry the prescribed coauthor footer. One heavy local invocation at a time, memory.current below8.5GB and disk above8GB. This lane hard-stops12:30UTC, with owned work shut down beforehand.

What was found and done

1. The client-close terminal predicate was too narrow. A deterministic real-loopback scenario holds public pipeline cleanup before the child close future is published. All3/3old native attempts fail the intended assertion with writeDone=false, childOpen=false, childCloseDone=false, rootOpen=true/rootCloseDone=false and both event loops live. AbstractChildChannel sets CLOSED and destroys its pipeline inside doClose before Netty sets closeFuture; removing the output observer wakes the writer on cancellation.
2. Candidate451dcaf0b adds only !child.isOpen to the new client-close assertion. It changes no production behavior or main test and retains every original check. The corrected controlled scenario passes first attempt with explicit state assertions requiring the exact closed-child/pending-future/live-root ordering.
3. Deliberately broken mutation8df0ef534 fires completion at controller.close before either output channel closes. The exact corrected original test rejects3/3native attempts at the intended terminal assertion. Root and child remain open before and after every callback. Mutation and diagnostic fixtures are excluded from the PR. Scoped test/code reviews approved this correction; they do not approve overall readiness.
4. After r102 PID1975545 exited, verified PR OPEN/expected6432a18b, pushed all WIP refs and independently read back their SHAs. Following the first clean23-test round, pushed451 onto the existing PR and started final-head CI while remaining local rounds continued as an independent gate. Why-first description preserves all r102 disclosures and now explicitly says validation is not clean.
5. Mandatory rebase before round2 picked up mainc64fee18d and produced62b0cba75. Exact diff451..62 contains only main’s unrelated tests/testPeerRegistryCapacityBatchRemovalCallbackRejectsLaterVictimReadd.kts update, matching main. All production/build files and all41scoped tests are byte-identical. Rebased verification head62 is separately pushed; the PR remains451 with its own exact-head CI. Never label the mixed source receipts one commit.
6. Local first-clean gate FAILED. Round1:23/23firstPASS. Round2:16/23firstPASS and16/23finalPASS, seven failures through all three attempts. Total39/46firstPASS, seven firstFAIL,60native attempts. Failures span setup/class-loading, socket reads and readiness/test deadlines. Client-close timed out creating the server before reaching its corrected completion assertion. All21failed-attempt stacks/output are in new-02.xml and new-02-failed-attempts.json; first-attempt full details are separately retained.
7. The additional failure class is unexplained. Captured runnable main threads spent~3CPU seconds over~50wall seconds in class definition/Kotlin reflection; another JVM barely initialized over60wall seconds. Post-failure host CPU/memory/IO pressure and load were high, while this cgroup stayed below3.7GB with no CPU quota/throttle events. Cumulative oom_kill=9 has no captured timeline, so it is not attributed to this run. Three later ordinary java-version probes start in0.10–0.18s. These observations do not identify the responsible process or exact failure mechanism, and do not prove a product fix. No timeout/resource increase or guessed code change was made.
8. Remaining fixed samples are explicitly incomplete: rounds3..10 and18regressions. A bounded pre-launch wait is active for foreign runners before regressions; no regression body/result exists yet. Retain the failed round2; collect remaining planned samples without replacing failures, then diagnose the shared startup/execution condition before any fresh clean-validation claim. Every completed milestone is remote.

Current CI and source state

PR OPEN/unmerged at451dcaf0b. Required build passed. Required full suite is running; native first-attempt counts are unobserved at this write. Owned build-watchman0.0.21 plain--pr PID2138523/session77172 gates exactly bld-build+bld-all-tests and ends by12:28UTC. It recovers explicit GitHub connect timeouts. Non-required buildtest zero-dispatch warning is preserved without warming/rerequest/submission. No CI run was requeued.

Relevant refs

PR: https://github.com/CodexCoder21Organization/UrlProtocol/pull/748. Native evidence is under verification/r107 on the evidence branch. Source451, old fixture9ad5ebe13, positive fixturef577f6140, mutation8df0ef534 and rebased verification62b0cba75 are all remote. Nothing was deployed or published in this lane; no production write, merge or enqueue was invoked.

Next steps

Re-verify first. Obtain the existing exact-head CI artifact and audit its first attempts. Preserve all local failures and complete remaining fixed samples if the current brief/time permits; otherwise their precise incomplete state must remain explicit. Investigate the captured bootstrap/class-loading/socket/deadline failure class through real public JVM/transport execution and OS/process measurements; identify the exact mechanism before changing code, and do not loosen test limits or dismiss it as a busy host. A future clean campaign needs a separately identified cohort after a proven fix; never overwrite this lane’s native records. Refresh the PR with actual results, preserve every branch and full stack, and clean all owned clone/cache/watchers. Original first-clean sample has failed and cannot be reported as clean by replacement rounds.

Retained r102 history (historical scope/state)

(scope and state below are historical)

Handoff: resolve the client-close completion assertion and finish clean forwarding validation

Written 2026-10-21 10:00:03 UTC. This scoped session has stopped; the latest terminal observation below supersedes the earlier in-progress snapshot.

RE-VERIFY: State below is a snapshot. Query https://github.com/CodexCoder21Organization/UrlProtocol/pull/748 with gh pr view ... --json state,headRefOid,mergedAt,statusCheckRollup. Verify current main and every branch below with git ls-remote. Native XML determines first-attempt counts.

Design decisions in force

Original service request: “We need to investigate why https://my.externalbrain.wasmserver.com/ is failing to load and fix the underlying problem.” This task handles the host-stop completion-test follow-up for UrlProtocol's completion and typed-forwarding APIs, which ContainerNursery needs for bytecode forwarding.

The assigned p102 brief permits correcting a newly introduced test when evidence shows it asserts an unsupported API ordering, with mutation proof. Its edit scope says “change only that new test's terminal predicate,” referring to the host-stop test. No edits to tests on main, new timeouts, sleeps, or iteration reductions. No merge, enqueue, artifact publication, deployment or remote buildtest submission. No user questions. Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.

Validation requires 23 new tests in ten sequential local invocations plus 18 affected regressions, native first-attempt counts, one heavy local invocation at a time, cgroup memory below 8.5 GB, and free disk above 8 GB. The brief requires waiting for r98 PID 1918882 to exit before pushing, then observing bld-build and bld-all-tests with build-watchman 0.0.21 in plain --pr mode. It gives no merge authority.

Mission and remaining work

Resolve the separate new client-close assertion under appropriate scope, prove the corrected test rejects completion with live output, and obtain clean prescribed validation. The host-stop correction is implemented, reviewed and verified. The complete local campaign is not first-attempt clean; never turn its retry-assisted final result into a clean verdict.

Findings and work completed

1. Fresh-cloned f045e2108 under workspace/r102-UrlProtocol and rebased on main dd51daccd. Read p98 and its full findings, README/KDoc, current Testing and Engineering Philosophy documents, and relevant prior and consumer handoffs. There is no repository AGENTS file. All 23 selected tests are new in this PR; the 18 regressions match main.

2. The API delivers each completion registration once after writer exit, including failure and cancellation. Reading the backend's final byte is insufficient. Root or child termination settles a pending logical write; the API does not require the child's close-future publication after its connection socket has closed.

3. Independently repeated the controlled host-stop experiment, retaining the final physical promise and child-close operation. The old predicate failed 3/3 native attempts. Review found that following every parent to null could identify the listener instead of the connection socket. Repeated the old predicate with traversal stopping before ServerChannel: all 3/3 attempts failed with MuxChannel open and close future pending, actual NioSocketChannel closed and close future done, and listening NioServerSocketChannel closed. This confirms the actual output-root mechanism.

4. Candidate 6432a18b changes only tests/testForwardedPendingWriteCompletionOnHostStop.kts: identify the connection root while excluding its listener, and accept physical promise done, child close future done, root close future done, or root not open. Every other assertion is retained. The corrected controlled scenario passed first attempt. Ten additional separate forced-root invocations also passed on their first attempts, each requiring the physical promise and child close future to remain pending at callback time; only root termination can satisfy the predicate. Their peak observed memory was 5,450,223,616 bytes and minimum free disk was 56,381,399,040 bytes. No product, README, dependency, timeout or iteration change was made.

5. Deliberately broken build bfde0141 records claimed responses and releases their writer synchronously at the beginning of host stop, before closing any transport. The exact corrected original test is unchanged. It fails all 3/3 native attempts at the intended terminal assertion while child and actual root remain open, with both close futures pending before and after completion. The test had already asserted that the physical promise was pending. These are assertion failures, not timeouts or cleanup errors. The mutation is absent from the candidate.

6. Exactly ten invocations of all 23 new tests on 6432a18b recorded 228/230 first-attempt passes, 230/230 final passes and 232 native attempts. The corrected host-stop test passed 10/10 first attempts. The unchanged client-close case failed first in rounds 6 and 7, then passed its built-in second attempt. No replacement rounds were run. All 18 regressions passed first attempt. Peak observed memory was 3,374,485,504 bytes; minimum free disk was 47,100,628,992 bytes. No resource guard stopped a run. Dedicated test and code reviews approve the host-only correction and explicitly do not approve a clean full-campaign verdict.

7. The two client-close failures have the same complete assertion at callback line 70, through writer release. These natural callbacks did not record channel state, so their exact state must not be invented. r98's separate diagnostic captured childOpen=false, childCloseDone=false, rootOpen=true, rootCloseDone=false at failing client-close assertions. This is evidence of a separate publication-order predicate issue. p102 only authorized the host-stop correction, so the client-close test remains byte-identical. The all-first-green requirement remains unmet. The previous head's required Actions passed, but native history recorded 2,865/2,867 first passes, 2,867/2,867 final passes and 18/18 passing build rules. The old host-stop assertion and unchanged testWriteStuckConnectionRecoveredJustBeforeGraceIsNotRetired each failed first attempt. The latter read 786,390 of 920,073 bytes before stream closure; its timeout did not fire. Its loss record reports zero completed frame bytes, requiring a consistent reproduction and a trace of physical write, transport progress and close initiator before attributing the closure. Existing related work is https://github.com/CodexCoder21Organization/UrlProtocol/pull/729; no causal equivalence or fix is established. Full native evidence remains on the r98 evidence branch. This is prior-head provenance, not the corrected head's result.

8. r98 exited at 09:43:39 UTC. A fresh ps check returned exit 1 before any push. Fetch/rebase left the exact tested source unchanged. The PR was freshly OPEN at its expected prior head before an atomic explicit-lease push at 09:44:25 UTC. Source, mutation and evidence refs were independently verified with ls-remote. The why-first PR body readback matches and prominently retains the failed local clean gate.. The corrected-head build check passed: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39579294756 . Its full-suite run is still in progress: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39579294763 . Run Tests began at 09:46:40 UTC. Native exact-head first-attempt counts are not yet available. Owned build-watchman 0.0.21 plain --pr PID2063134 is active; it gates only bld-build and bld-all-tests. The watcher is bounded at 10:50:30 UTC, allowing cleanup before the task wall limit at approximately 10:51 UTC. Do not cancel or resubmit the existing workflow. Prior-head counts are provenance only.. All source JVMs ended and the temporary diagnostic, generated JAR directory and owned build cache were removed. The source checkout is clean and retained only for final evidence preservation while the full-suite watcher runs. The process waiter ended naturally. Final watcher/checkout cleanup remains to be recorded..

No merge, enqueue, Maven publication, deployment, production restart or downstream dependency update was performed by this effort. No subagents or active goal were used.

PR and branches

The earlier evidence is at https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r98-forwarding-evidence-20261021/verification/r98. The consumer motivation is recorded at https://github.com/CodexCoder21Organization/ContainerNursery/blob/474aa8070c15bcc38f98d3d356f8562d1ad834d3/investigations/r81/HANDOFF.md.

Next steps

1. Re-verify PR, main, refs, required CI and native artifacts. Read every failed first attempt. Do not rerun unchanged CI hoping it passes.
2. Obtain scope for the separate new client-close correction. Independently confirm the documented contract and the closed-child/close-future transition through the public transport path. Fix the product only if evidence proves a product defect. Do not add a delay merely to wait for close-future publication.
3. Retain every other client-close assertion. Prove that its corrected predicate fails while the physical promise is pending and child/root remain open, using a deliberately broken build. Preserve native failures and source identity.
4. Rebase current main before builds. Run the prescribed 23 × 10 sequential local invocations and 18 regressions with resource guards and native first-attempt counts. Update the same PR after checking it is open and still at the expected head. Observe the required Actions with coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlProtocol --pr 748. Use plain mode; no merge or enqueue. Download the test-attempt-history artifact and count first attempts.
5. Preserve all work remotely, then clean owned checkouts, caches and watchers. Do not publish or deploy under this brief.

Reproduction and operating notes

Native <test> records and matching <attempt-history> are keyed by file and name. History attempt 1 governs the initial verdict. Exit 0 can contain a failed first attempt followed by a successful retry. Diagnostic and mutation runs are separate from the 230 + 18 campaign denominator.

The evidence head above is a verified snapshot before this document was added; the branch advances with later receipts.

The evidence branch contains old-output-root-fixture.kts and corrected-output-root-fixture.kts. Each defines testForwardedHostStopTerminalChannelEvidence. Copy one temporarily to tests/testForwardedHostStopTerminalChannelEvidence.kts, run scripts/test.bash --local --test testForwardedHostStopTerminalChannelEvidence --log <absolute.xml>, preserve the native record, then remove the diagnostic from the candidate. apply-mutation.py and live-output-mutation.patch reproduce bfde0141 from 6432a18b; its selector is the exact original testForwardedPendingWriteCompletionOnHostStop.

required-tests.json and affected-existing-tests.json are the exact selectors. run-validation.py runs supported local invocations sequentially, rebases before each, guards resources, audits first attempts and checkpoints each milestone. To use it unchanged, restore evidence under out/r102-evidence and clone under workspace/r102-UrlProtocol. Never overwrite native records or replace failed campaign rounds.

build-watchman may recommend rerequesting a non-required buildtest suite with no dispatched runs while required Actions progress. The explicit no-buildtest instruction overrides that advice; do not warm, rerequest or delete that suite. The r98 process wait uses a bounded pidfd notification and requires a fresh ps absence receipt before pushing.

Documentation conflicts: p102's explicit local-plus-Actions instruction supersedes Testing.md's remote-gate policy for this task. The current handoff skill retires the legacy README's merge exception. No merge is authorized. No production latency improvement has been measured or claimed.

Terminal observation — 2026-10-21 10:51:49 UTC

2026-10-21 10:51:49 UTC — BLOCKED. Assigned task: resolve the new host-stop completion test on https://github.com/CodexCoder21Organization/UrlProtocol/pull/748 for the original externalbrain loading investigation.

The corrected test accepts actual root termination, excluding the listening parent, while keeping every other check. Host-stop passes10/10 prescribed rounds; ten forced-root controls pass first attempt; the exact corrected original rejects3/3 premature live-output mutations. Source and proofs are remotely preserved.

Fixed local campaign:228/230 new-test first passes,230/230 final passes across232attempts;18/18 regression first passes. Both first failures are the unchanged client-close test. The brief permits only the host-stop predicate change, so the clean gate remains unmet. No replacement rounds or readiness claim.

Exact-head native first-attempt counts remain UNOBSERVED. The owned watcher ended at its task deadline after GitHub API quota and timeout errors. The existing full-suite workflow is not cancelled or resubmitted: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39579294763 . Prior-head counts are not substituted.

The corrected-head build passed. Last verified PR state was OPEN/unmerged at the corrected source. No merge, enqueue, publication, deployment or CI resubmission occurred. No delegated work, user question or new owner direction. No unnecessary holds; no idle landing gates. The owned watcher has ended.

Next: authorize the separate client-close correction, continue observing the existing full-suite run and obtain its native artifact, then repeat the required clean validation without changing main-branch tests or loosening limits.

Lessons: count first attempts rather than retry-assisted finals; distinguish sockets from listeners; pair a corrected predicate with a live-output mutation.

Actual cleanup verified: owned checkout, build caches and watcher are removed; all source, mutation and evidence refs were remotely verified before deletion.