Handoff: Fix the backend-close test completion check after cancellation

Written 2026-10-22 UTC.

RE-VERIFY: This is a write-time snapshot. Recheck main and branch heads with git ls-remote and the original PR with gh pr view https://github.com/CodexCoder21Organization/ContainerNursery/pull/702 --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt before acting.

Design decisions in force

Owner direction 2026-10-21: "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

Lane constraints 2026-10-22: "Absolute rules: never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test." "You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs."

Use local build/tests only, no remote runs or new JVM tuning flags. Count a pass only on attempt 1; retries remain visible. Public behavior assertions must retain the complete framed error, no client-facing prefix EOF/ambiguity, and exactly one side effect. Do not resolve this by relaxing the test's existing 20/10/30-second waits or 120-second watchdog.

Mission summary and observed failure

While completing the review of https://github.com/CodexCoder21Organization/ContainerNursery/pull/702, required bld-build finished green but its XML exposed a first-attempt failure in the existing main test testUrlFacadeRpcBackendConnectionEndsAfterSideEffectReturnsFramedError. Attempt1 failed in 13480 ms at line 259 with java.lang.AssertionError: The backend handler must finish after recording its side effect; attempt 2 passed in 2644 ms. There were 901 final successes and 900 first-attempt successes. Both changed Direct tests passed first attempt 2844/2879ms. No CI rerun was performed. The original review is addressed and the PR remains open/unmerged, not waiting for a merge in this handoff.

The failed test file is byte-identical to main in the original review PR. Its prior recorded source commit is https://github.com/CodexCoder21Organization/ContainerNursery/commit/9f2a5235125ede3981e271fc551ad943fd997298 . The original PR changes only two different Direct test files.

Mechanism and deterministic proof

The backend callback increments the side-effect counter, signals entered, awaits a CompletableDeferred, then signals handlerCompleted. Main closes the backend TCP connection before completing the deferred. If connection closure cancels the suspended callback first, its await throws and execution skips that completion signal.

The diagnostic branch keeps the original completion assertion and all original parameters. It adds a separate handlerExited latch in finally and captures the callback failure. After cutting TCP, it waits for callback exit and asserts that failure is CancellationException BEFORE completing the release deferred. It then performs the unchanged original completion assertion. The callback is observed to have terminated by cancellation, yet handlerCompleted remains unset.

One targeted invocation built successfully and failed the original assertion on all three automatic attempts: 11680, 12395, 12058 ms. Both new cancellation/exit witnesses passed in every attempt; failure was the original assertion at shifted line 272. This is a deterministic event ordering, not CPU load or a machine-tuned concurrency count. The full stacks and console are in the remote XML/log below. No fix has been made.

The experiment establishes why this assertion flakes. It does not by itself settle the intended protocol contract: if callback cancellation is allowed after disconnect, completion observation must cover that terminal state without weakening the framed-error/side-effect assertions. If the contract requires the handler to continue despite connection loss, the fix belongs in the upstream protocol, with both consumer and upstream reproductions. Read that contract before choosing the repair; do not blindly move the signal to finally and declare every behavior fixed.

Relevant PRs and remote branches

All partial work is remote. Latest diagnostic base is main 57a4e8000738a069f55d51be09e6b8f19a39c0b3, which advanced during the diagnostic fetch. A rebase recording pause due to missing local committer identity was corrected; source SHA256 remained 5d65ecfb2a6dc1ab938bfe92cac56f13c5adac78b081aefc8aa088fc693a265c during the build, then the rebased commit was pushed with an explicit verified lease. No test source changed mid-run. See CN702.md for the complete command-state correction. Original PR source stayed unchanged.

Essential files in lane-evidence: ci/test_log_file.xml, CN702-ci-first-attempt-error.txt, CN702-ci-first-attempt-console.txt, CN702-backend-cancel-control.xml, CN702-backend-cancel-control.log, CN702-final-pr-state.json, CN702.md. XML <attempt-history> elements are global siblings of <test>, so search the entire tree.

No production deployment, artifact publication, service restart, merge, enqueue or CI rerequest was performed. Generated build jars/dependency caches are deliberately not committed. Read-only UrlResolver/UrlProtocol source clones had no changes or stashes. The other open finding from this lane is the independent history period-bounds timeout; it has no proven common mechanism and is not a dependency here.

Next steps

1. Reverify current state, clone fresh into workspace, read AGENTS.md/README and the complete Testing Architecture and Engineering Philosophy. Recover the control branch and evidence before repeating work.
2. Re-run the control with scripts/test.bash --local --test testUrlFacadeRpcBackendConnectionEndsAfterSideEffectReturnsFramedError --log <xml>, after fetch/rebase. Expected existing-bug result: successful build, cancellation witnesses pass, original completion assertion fails. Preserve its binding/dependency setup for the comparison. No public/private visibility changes or mocks.
3. Read the upstream UrlProtocol request/callback cancellation contract and the test's motivation. Identify the owning layer before repair. If only terminal completion observation is wrong, cover cancellation and ordinary completion explicitly; retain all framed-response/no-EOF/exactly-once assertions. If continued handler execution is promised, reproduce/fix upstream rather than making the downstream test accept a violation.
4. Use the same deterministic ordering to demonstrate failing-to-passing behavior, add relevant neighboring cases, and run independent test-comprehensiveness and code reviews. For lifecycle/concurrency changes, write the resource invariant table first and test through public APIs only.
5. Rebase, run the required local tests/full suite under the shared lock as appropriate, create a separate PR whose WHY names the observed CI failure and links its run, then watch CI without rerunning unchanged failures. Leave it open unless the owner explicitly changes merge authorization.

Operational knowledge

One shared 10 GiB cgroup applies to every lane. Full suites require flock /tmp/claude-501/complete-suite.lock scripts/test.bash --local --test . --log <xml> and memory.current sampling. No suite killed for memory should be retried blindly. The original review suite was executed only once; this control is a targeted diagnostic, not a CI or full-suite rerun.

GitHub polling is at most once per two minutes. Shared core quota exhausted during original CI observation and reset at 09:42:24; watcher was stopped during that hold. A stale checkpoint-branch kotlin-build-ci-test suite reported zero dispatched runs, while both actual PR GHA workflows dispatched and passed; never warm/rerequest due to that unrelated alert. All watcher processes were stopped; stop any new background work before final reporting and confirm with ps.

Handoff writes use handoff-cli/service only. The legacy format README's handoff-PR merge instructions are retired by current skill and conflict with this lane's no-merge rule. A context update once failed during bytecode fetch (read 0 of 4 bytes), then a bounded second metadata update and exact readback succeeded; full tool error is preserved in evidence. No service restart was attempted.