Handoff: Complete exact-head ContainerNursery suite verification

Written 2026-10-21 08:59:42 UTC.

RE-VERIFY: This is a write-time snapshot. Check the source draft with gh pr view 686 --repo CodexCoder21Organization/ContainerNursery --json state,headRefOid,baseRefName; check the base draft similarly; inspect the machine-wide process table and each runner cwd before executing tests. A changed source head invalidates this exact-head review.

Design decisions currently in use

The lane brief dated 2026-10-21 requires: "read-only on the PR; do not edit it"; "Run with --local."; "No merge, enqueue, deployment, publication, or tools that merge automatically."; "Use NO outer wrapper cap shorter than 120 minutes, and never change a test's own timeout." These remain binding. Do not raise timeout values, reduce iterations or disable tests. The one full-suite invocation allowed in laneR686 has been used; arrange a fresh execution window before another invocation. No source edits or production action are authorized by this handoff.

Mission summary

The owner asked that https://droplets.wasmserver.com/ stop showing errors and that the root causes be fixed. This lane was assigned a read-only correctness and test review of https://github.com/CodexCoder21Organization/ContainerNursery/pull/686 at exact head 86072cd87e51f3fdc55e01f66bc7a48765e882c0, base wip/laneN1-replacement-gap, plus one complete local suite. Source and test review are complete with CHANGES REQUESTED. The full-suite invocation ended with status 137 before XML, leaving suite verification unfinished.

What was found and done

1. Verified the exact source head in a fresh owned checkout, read README/AGENTS and the requested deployment skill, and read the required Testing Architecture sections and official systemd socket/service/execution/notification documentation. The initial clone was still checking out when the PR checkout ran; its later reference change was caught before any test. Source/index matched the requested commit throughout that interval. The clone finished, the correct branch was restored, and all actual CN tests/probes ran at the requested hash. Both checkouts are now clean.
2. R3, direct public-request evidence: a fully read HTTP/1.1 keep-alive response leaves its connection handler reading the next request. HttpsFacadeProvider.kt:2635 continues while running; :5986 joins all handlers before :5999 makes running false. An idle completed connection therefore keeps the predecessor's writer lock, while the successor waits and new TLS connections remain queued. The final standalone probe releases successor main independently. With idle client open, its explicit new-request requirement is red 1/1: new request reaches its unchanged five-second limit at 5.01 seconds, predecessor remains running, successor waits. Closing only the idle client lets stop complete and the same application serve a response. A paired condition control closes that client before stop: green 1/1, new request completes in 1.34 seconds. The initial probe had a dependent successor gate and is excluded from final contract counts; that correction is documented. No PR source fix was written.
3. R1, direct lifecycle evidence: cn-generation.py starts a separate 30-second writer-lock wait while predecessor HTTPS draining may itself consume the full existing 30 seconds before provider/nursery cleanup and process exit release the lock. A real partial backend response held through that drain makes successor helper exit 1 after 30.03 seconds while predecessor remains alive. Repeating with the exact stop helper makes that helper also exit 1 at its own existing limit. No timeout changed. An actual manager's final synchronous start may submit another job; that behavior was not executed here, so this timing case alone is not claimed always to leave the whole shell procedure without a successor.
4. R2, source-sequence finding: cn-replace.sh:33 submits successor start, :35 stops the serving predecessor, and :38 waits/starts the successor. It has no restoration action for unsuccessful successor startup; both services use Restart=no and the active marker does not start a CN generation. A missing generation JAR or startup error can leave a listener with no serving process. Actual system-manager execution and recovery behavior remain unverified.
5. Tests use real TCP, TLS, CN JVMs and JAR children. Response test compares complete first|second bytes and Content-Length 12, but its entry point is a small public-facade Java fixture rather than production main. N1 setup changed to a parent-owned inherited listener and an asynchronous queued request; original result predicates remain. This lane's queue and N1 tests each pass on attempt 1, 2/2 total. Original N1 on fresh base 0a7b9d264e26fa012c3e1c4d2187a8640d24d6d9 is red on all three attempts; first-attempt pass count 0/1. This is the expected earlier availability behavior.
6. Units retain the listener, use Accept=no, named fd stdin, NonBlocking=yes on both recipients, a 128 MB heap and Restart=no. Lease close stops the selector without closing the inherited channel. No descriptor-growth measurement was made. N1 predecessor uses CN_RESTART_HANDOFF=1, while the supplied service template does not; its same-application result is not proof of application survival under that template. Other public endpoints are not persisted by the one-fd bridge. Socket activation starts only the marker, so installation/boot needs an explicit current-generation startup rule.
7. Waited for the machine-wide full-suite slot behind two other lanes. Slot opened 08:44:07 UTC; one invocation launched at 08:44:51 from exact head with --local --test . and no short outer wrapper limit. It exited 137 at 08:52:43 after about 7m52s. full.xml was not produced. Counts and attempt histories are UNAVAILABLE, not zero. Cause is unconfirmed; no during-run resource snapshot exists. Do not label this as a particular resource event without new evidence. The recorder tracked 315 PID/start-time identities; final checks found zero tracked survivors or CN runners. No rerun occurred.

Relevant PRs / refs

Evidence: findings, probe notes, full-suite summary, cleanup. All needed partial work is on that branch; build JARs, dependency caches, transient full process-table captures and compiled probe classes are deliberately omitted.

Deployed or published but unmerged: none by this lane. Its command/action record contains no deployment, restart, publication, merge or enqueue call. Production state was not changed or re-verified. The handoff-service health probe is only a read operation.

Next steps

1. Re-verify current state first. If the draft head changed, obtain a new exact-head review brief rather than treating these results as current.
2. Read the saved full log and structured result. Determine why the launcher ended with 137, using a targeted local experiment or direct process/resource evidence. No cause is established by the exit code alone. Do not change tests, time limits or runner size as a substitute for identifying the mechanism.
3. Obtain a fresh sufficiently long execution window and the one machine-wide slot; run the complete suite once from the agreed exact head, using --local and XML logging, without a short wrapper cap. Preserve process ownership and settle all descendants at exit.
4. Read attempt 1 from each XML history, treating test records without histories as attempt 1. Compare every history with the exact base; do not count buildrule records as tests or a later pass as a first-attempt pass. Report gate verdicts and counts.
5. R1/R2/R3 remain authoring work for a separately scoped task; this handoff does not authorize changing the reviewed PR. Real systemd execution, initial listener migration, other endpoints and application transfer still need the explicit checks in the findings before any production use.

Operational knowledge

The review evidence branch is separate from the source PR. It was created with a separate index/tree and did not change the tested checkout HEAD. Copy evidence inside .git before selecting the exact source commit; probe-notes.md explains preparation with the locally built thin source JAR first on the resolved dependency classpath. Idle-window.py defaults to the red condition and accepts closed as its fifth argument for the passing control. Draining helpers capture JSON observations rather than returning replacement success.

Canonical handoff-format README still describes a legacy merge path. The current create-handoff skill retires it, and this lane prohibits all merge actions; this handoff uses only the central service.

Final process-table capture at 09:02:03 UTC found zero owned runtime processes. Review evidence remote head was verified as 993f61d8af395e10d956a1fcd8000a59b1c51d93.