Handoff: Resolve the shared-client ContainerNursery soak failure without changing its limits

Written 2026-10-23, CNSOAK lane. RE-VERIFY: This is a write-time snapshot. Recheck pull requests with gh pr view <full URL> --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid, branch heads with git ls-remote, and current shared-machine activity before running tests. There is no completed implementation fix in this handoff.

Design decisions in force

Owner direction, 2026-10-23 (CNSOAK2 brief), verbatim:

Decision (orchestrator, final): the soak keeps ONE shared client — that is how ContainerNursery runs in production. Do not switch the test to independent clients and do not change any limit, round count or assertion.

This resolves the previous topology question. CNSOAK2 is tracing the shared-client execution path. No asynchronous implementation or fix is yet verified.

Owner direction, 2026-10-21, verbatim:

"Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" "If there are gaps, have sol write the missing tests." "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

Lane requirements, 2026-10-23, verbatim:

"Never raise the soak's watchdog, its thread limit, or cut its cycle count." "If the defect is in UrlResolver or UrlProtocol, the fix belongs upstream there (new PR in that repo), not as a workaround in ContainerNursery." "Confirm it fails 3/3 on the first attempt." "Fix the cause; the new test and the original soak must pass first attempt." "Mutation check; full suite of the changed repo under the machine lock per the common 'Full-suite time budget' rule; open a PR (why first: this lane's background above), watch required checks (watch only)." "You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs." "Absolute rules: never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test. A failing test is a defect report: fix the cause. Error messages state what was wrong and the actual value." "Build/test: scripts/build.bash --local and scripts/test.bash --local --test <name> --log <file.xml>. Never --remote (killed remote clients leave orphaned runs). The runner retries a failed test up to three times; count a pass only when attempt 1 passed (no attempt-history in the XML)." "Memory: every lane on this machine shares ONE 10 GiB memory limit (/sys/fs/cgroup/memory.max). A full suite was killed (exit 137) when other lanes compiled at the same time. Run full suites only under the machine-wide lock: flock /tmp/claude-501/complete-suite.lock scripts/test.bash --local --test . --log <xml>; record /sys/fs/cgroup/memory.current before and every few minutes during it. If a suite is killed, report the numbers; do not loop retrying. Do not add JVM tuning flags." "Full-suite time budget: a full suite takes 1–2 hours of execution AFTER it acquires the lock, and the lock wait itself can be 1–2 hours. Two suites today were killed by the lane's own outer deadline mid-run, wasting the shared slot. Never wrap the suite in a deadline shorter than 3 hours of execution measured from lock acquisition, never kill a suite that is executing, and size your lane plan so the suite can finish." "CI: required checks are the GitHub Actions jobs (bld-build, bld-all-tests, bld-build-release, as the repo defines). kotlin.build (remote) is informational in these repos; if it is red, read its log once and report the cause (code vs infrastructure) in findings — do not rerun it. Stop watching CI after 90 minutes and report." "Credentials: JVM recordings, profiles, heap/thread dumps, environment dumps and system-property dumps can contain tokens (GH_TOKEN and others). Never commit or push them, even to wip/ evidence branches; push only derived summaries you have read."

GitHub calls are limited to once per two minutes. PR bodies must be changed with a PATCH API JSON file and read back. Commits require Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; PR descriptions must end with the specified Claude Code footer. No merge or deployment is authorized. Required reading before writing tests remains TESTING.md and PHILOSOPHY.md.

Mission and current decision

The CNSOAK brief asks to explain and fix thread growth/timeouts in the 100-round resource soak associated with https://github.com/CodexCoder21Organization/ContainerNursery/pull/714. It originally suggested the high TcpRpcStream/TcpServer creation rate might cause both issues. The thread-retention part of that premise is refuted by the actual failed sample; the remaining first-attempt watchdog failure has no newly established implementation fix.

A scope question has been presented to the owner and has no answer: should the resource soak keep its current single shared typed client (recommended, preserving current coverage), or model eight independent typed clients? A complete separate eight-client diagnostic passed on attempt one with every original numeric bound and assertion retained. It is not a fix or substitute for the original scenario. Changing shared-VM RPC execution instead would require a concurrency design and effect/cancellation/close/re-entry tests; do not treat serialized execution alone as an undocumented implementation mistake.

What was found and done

1. High TCP thread creation is explained, not shown to be retained. ContainerNursery at d84bf644: src/main/kotlin/org/example/facade/UrlFacadeProvider.kt:1627 enables a persistent backend bridge only for a live outside connection advertising pushSupported=true. Otherwise :1943–1945 opens a backend stream per request and :2068–2069 closes it for non-streamed results. A response stream owns the lifetime for streamed results. The legacy one-request lifetime is explicit in docs/FacadeProviders.md:264–271. Pinned UrlProtocol1429 source0956eaf7566a2418cf55742eb3917e8cd461ac38: UrlProtocol.kt:556/:658, TcpRpcStream.kt:104 starts its reader, TcpServer.kt:1179/:1457/:1468 accepts and starts its reader/writer. Close paths are TcpRpcStream.kt:686–698, TcpServer.kt:1495–1502/:2064–2087. Request-handler workers are a reusable server-owned pool. These pointers were rechecked against exact commits; earlier notes incorrectly put the facade gate at1640 and TCP route/connect at558/660.

2. The original cycle80 excess contains no newly added per-connection TCP readers/writers. The saved unchanged-main1328 result first failed with baseline accounted286/raw262, limit310, samples304/302/307/313. Its added65IDs include17 unnamed cached idle workers,16Relay race workers,8Noise workers,8coroutine workers,5effect-RPC workers and other lazy pools. There are zero added TcpRpcStream-Reader/TcpServer-Reader/TcpServer-Writer workers. A small real TCP public-path control passed three first attempts (1987/1326/1821ms) and checked one-shot worker retirement before stopping the server. It is a passing diagnostic, not a fail-first bug reproducer.

3. Open-only async initialization misses later cached completion workers. JDK ThreadPool.java:105–116 creates a cached completion executor distinct from17 pollers on this host; AsynchronousChannelGroupImpl.java:186–190 submits completion tasks. Real connect controls add3/4 cached workers after the open-only baseline and return to17pollers after65seconds idle. Actual soak parent recordings link5 extra unnamed workers to a known EPollPort poller, and a later snapshot showed3cached idle workers with the same parent. A passing1333worker late snapshot had17pollers+12cached idle workers. This is evidence for late completion capacity, not retroactive proof of ownership of the exact17IDs in the historical failed sample. That attribution remains unresolved. No arbitrary pool cap or allowance change was made.

4. A real1328 allocation defect is already fixed upstream. Small stream pulls offered1MiB arrays for <=4KiB data on both backend and facade layers; a bounded recording saw626outside-TLABarrays of1,048,592bytes and repeated humongous-allocation GC. Existing upstream regression was copied unchanged except version/function names: old1328 failed first attempt three independent invocations (1751/1909/2693ms), assertion Stream 0 offered a 1048576-byte array to read(b, off, len); the limit is 8192 bytes;1333passed first attempt2017ms. This fix was already merged in https://github.com/CodexCoder21Organization/UrlResolver/pull/1431 and published as1333. CN714D owns adoption; no duplicate change was made here. The1333watchdogs cannot be explained by the old bug merely because it existed.

5. Recorded original1328 run failed, without a thread-bound failure. The original test source remained byte-identical, SHA2560d741bf650bfc8dae6b70fddd40044ca2ac118bc516795661bd9316d867ce9cc. Three automatic attempts: first FutureTask.get TimeoutException at test446 after703006ms; next watchdogs1804916/1804572ms. Later attempts reached cycle20 at294/292threads under accounted287+24. This run used an external bounded ThreadStart observer; timing overhead is explicit. CN714D separately recorded unchanged1328 and1333 watchdogs; do not repeat its classification run.

6. Shared1333 first attempt failed while every publisher continued to advance. The diagnostic retains100cycles,128files,8publishers,32senders,128MiB,1800second watchdog, every original assertion/bound; it adds only direct1333pin, identifiers and five-second numeric telemetry. Actual runtime providers are resolver1333SHA07af6af306dbdbf78f1347bb1f2a947efe87df43304003a1ee22a6a9ed206526 and protocol1431SHA4259816acfa12671d50ddebd7c8d04158624b1f8099f676e92303c3f1c030ff8 (not1429). Attempt1 failed watchdog1806090ms at diagnostic551/original446, lastcycle30/publishers4016–4019completed; each advanced126–128files in the last55.044seconds. Longest sampled flat count interval15.065seconds includes startup/housekeeping. This is no proof of absence of a stall between samples, but no lasting stopped publisher is observed. The first worker also had15seconds of bounded exception/socket recording, so it is not an unprofiled timing baseline.

7. The unchanged automatic shared retry passed; it does not count as first-pass success. Attempt2passed1756618ms,103424files/all100cycles, heap checkpoints59504528/60948264/60507200/60775960/61757624, descriptors335/334/334/335/337, threads303/304/306/300/305 under accounted284+24=308, openhandles0. Roughly2.045million total platform starts occurred while retained bounds passed. The XML primary test is this green retry; attempt-history records the red first attempt. Never classify its primary success as attempt-one success.

8. Independent-client scenario passed first attempt. Small batch controls all passed first attempt: shared69.40seconds publish, independent9.09seconds, repeated shared17.35seconds. Similar23150threadstarts; the fourfold shared variance prevents attributing the initial timing difference entirely to topology. A separate full100cycle eight-client diagnostic then passed first attempt813028ms, no attempt-history,103424files/all original resource checks. Its numeric trace records2,040,501total-start delta; sampled live threads287–372 during active work are not the post-cycle checkpoint values. Its successful checkpoint output was suppressed, so exact numerical baseline/checkpoints are unavailable. The test XML plus unchanged assertions establish those checks passed. It overlapped the later shared automatic attempt, so timings are not isolated benchmark comparisons. A read-only independent review verified all49original assertions and27timed guards remain, full normalized source differences, provider hashes and numeric trace schemas.

9. Shared sandbox serialization is part of current behavior. Pinned resolver1333 sandbox/SandboxedConnection.kt:858/:908/:913 has a suspend intrinsic that invokes synchronous RPC handlers on its VM lane. Each typed connection gets a separate VM. SandboxJvm README435–446 documents strict serialized execution within oneVM; UrlResolver testInterruptedCallerDoesNotCancelUnrelatedSharedSandboxCall.kts explicitly describes a second caller waiting for the same sandbox. This explains why separate VM owners can overlap more host/network work, but does not itself prove a violated public contract. No speculative asynchronous-intrinsic patch was made. Existing typed client headers do not automatically advertise pushSupported; no requirement that they should was found, so no capability flag was added to alter connection counts.

10. Other investigated ideas did not produce a fix. Exact libSJVM55 execute bytecode parses line numbers before the interpreter loop (not per instruction). The saved CPU null-path samples were not proof of repeated NullPointerException; a bounded exception recording found zeroNPE. Ten-second socket trace found1273slow4-byte-prefixreads and only2slowbodyreads1–5ms; normal responses aggregate one write, so repeated partial-body ACK delay was not demonstrated. Sampled allocation weights were unreliable under extreme thread churn and must not be treated as exact bytes. CPU/scheduler pressure observations are not a hardware fix or code cause. No timeout, cap, pooling change or tuning flag was guessed.

Brief completion audit

The lane is incomplete. Do not report DONE or merge anything based on the diagnostic passes.

Relevant PRs and remote work

Related overall effort: https://www.handoff.wasmserver.com/handoffs/hf-2026-10-23-resolve-bridge-sizing-and-soak-failures-then-finish-containernursery-verification . CNSOAK did not claim or rewrite that other lane's handoff.

There were no deployments, restarts, uploads of executable artifacts, Maven publications, merges or queue entries in this lane. All product repositories remain unchanged. Source clones UrlProtocol, UrlResolver, SandboxJvm and KompileCli were read-only, clean, no stashes or unpushed commits; only the listed CN WIP branch carries work. Every local test source is copied under lane-evidence with .txt suffix. Do not copy that evidence directory into a PR branch.

Next steps

1. Re-verify current PRs, branches and other lane notes. Read lane findings and final independent review. The findings retain the detailed experiment chronology, error stacks for the known regression, and rejected interpretations.
2. Preserve the original single-client test: the 2026-10-23 CNSOAK2 brief resolves the topology question. The eight-client passing diagnostic is comparative evidence only.
3. For the shared-client case, design a deterministic public-path experiment that distinguishes blocked synchronous host RPC work from guest execution and return marshaling. A proposed async intrinsic must preserve serialized guest execution, effect routing, cancellation isolation, close terminal state and re-entry. Existing raw-RPC concurrency tests do not establish typed-SJVM overlap. Do not add a time-calibrated throughput assertion or an arbitrary pool cap.
4. Only when a real defect and contract are identified, add a failing end-to-end regression and prove first-attempt failure three independent invocations. Fix in its owning upstream repository. New artifact publication is prohibited here; use a local source dependency/build seam for downstream verification if needed.
5. Re-run the original exact100cycle shared-client soak with unchanged watchdog/thread/heap/descriptor assertions; require attempt-one success after the fix. Mutation must restore the demonstrated failure. Run the changed repo's full suite under the shared lock with the required execution budget; never kill an executing suite. Open an unmerged PR and watch required Actions, at most90minutes. Do not use CI to diagnose.
6. Update this handoff with evidence-backed results; if another lane owns a dependency bump or general PR, coordinate rather than duplicating it.

Recovery and operational details

Scratch root was /tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad, findings out/CNSOAK.md, fresh clones workspace/CNSOAK/<repo>. All of those paths may vanish. Recover from the WIP branch: copy only desired lane-evidence/*.kts.txt back to tests/*.kts, with original filenames recorded in their contents. Original test hash above is the identity check. Main at investigation time was CNd84bf644622f542e94c788ced3f40fe2853b846a; pinned protocol1429source0956eaf7566a2418cf55742eb3917e8cd461ac38; published1333fixture actually resolvedprotocol1431.

Use git fetch origin then git rebase origin/main before each build/test. Targeted command: scripts/test.bash --local --test <functionName> --log <absoluteXML>. Result wrappers are preserved in lane-evidence; they use private process groups, PID start ticks and pidfd cleanup, bounded outer/inner deadlines, provider verification from currently open JAR descriptors, and numeric-only outputs. Review absolute diagnostic trace paths before recovery. The JAR classpath is delivered over runner control rather than visible in argv. Verify both resolver and protocol providers; do not mark a provider census complete after finding only resolver. Successful output may be suppressed; a retry's output can be in attempt-history. Counter samples are not completion boundaries and total-started is not live-thread count.

Direct in-test JFR failed in test preparation with NoClassDefFoundError for jdk.jfr.consumer.RecordedEvent from its URLClassLoader before the body. An external jcmd observer worked; this classloader challenge is preserved on WIP. report-challenge CLI auto-merges its record, which is prohibited by this lane, so it was not invoked. Raw JFR recordings, CPU profiles, heap/thread/environment/system-property dumps and full consoles stay local-only and are deliberately omitted. Reviewed ordinary exception stacks, generated thread-role/parent summaries and fixed numeric traces are preserved. No credentials or raw recordings were pushed.

All owned observer/runner/soak/control processes were reaped and absent in the final process audit; no own full suite was ever queued. Do not signal other lanes' jobs. Shared cgroup memory was10,415,378,432bytes in one late audit with no CNSOAK worker alive, so recheck current memory and suite-lock ownership rather than reusing that snapshot. The next run must honor the shared10GiB limit and no-JVM-tuning rule.