Resolve ContainerNursery bridge, callback and remaining test failures before merge

Written 2026-10-23 10:46:50 UTC.

RE-VERIFY: This is a write-time snapshot. Read https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 and its current check runs, compare remote refs with git ls-remote, and read the latest service report before starting. Local disk is disposable; source and derived evidence are on the linked branches. The PR source is 9ffc64c14be01928eba8f0dce12b33b2cda2f893 and main is d84bf644622f542e94c788ced3f40fe2853b846a at the latest ref check. This is not a green-suite or merge-readiness claim.

Design decisions in force

Owner direction, verbatim, 2026-10-21:

Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos

If there are gaps, have sol write the missing tests.

Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation.

Lane constraints, 2026-10-23:

Absolute rules: never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test.

You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs.

The original 1 MiB first-pull bridge guard must remain exactly unchanged, including its failure on resolver 0.0.1333, pending the owner's contract decision. The soak thread-count issue is assigned to a separate lane; this continuation does not change that test. Builds and tests use --local, never --remote, with no added JVM tuning. Count attempt 1 only; a runner retry pass does not clear a failure. Full suites require the machine-wide flock and at least three hours of execution budget after lock acquisition; never stop an executing suite. Watch required Actions once for at most 90 minutes; informational remote checks never authorize a rerun. Never push recordings, profiles, heap/thread dumps, environment dumps, system-property dumps or other runtime data that may contain credentials. This lane pushes only reviewed derived summaries and failure stacks.

Mission and verified current state

The requested continuation recovered the surviving full-suite run, classified all first failures against main, and moved the already reviewed source and the independent 58 MiB test into the existing PR. The explicit-head force-with-lease push succeeded after a fresh fetch/rebase left the tested head unchanged. The PR description was PATCHed with a structured JSON file and exactly read back. It opens with the owner's request and retains Do not merge, the bridge question, soak observations, callback issue, and every first-failure/main comparison.

The local full suite completed naturally: 952 tests; 943 first-attempt passes and 9 first failures; 946 final passes and 6 final failures. Three tests passed only on retry. All three build rules succeeded. The new 58 MiB test passed attempt 1 in 8,157 ms. CN714D's preserved affected-target proof has 13 first-attempt passes. The original bridge and soak files match the original PR SHA-256s byte-for-byte. No source or test was edited in CN714F; it carried forward exact head 9ffc64c1. No branch failure was manually rerun and no second full suite was started.

The suite began 06:54:49 UTC under the shared lock with a four-hour execution cap. A previous session interruption lost its wrapper lock while its coordinator survived. CN714E restored a lock, and another lane held the lock by CN714F pickup; this overlap is recorded, not presented as uninterrupted isolation. CN714F waited for the surviving run to end around 08:47. Shared memory was sampled every minute and oom=16, oom_kill=1 were unchanged from pickup through local-suite completion; the main controls also recorded unchanged counters. XML hash: 01276d84dcc81f487656fb1977ec64cfdacccee23e884556459063850cbf784a, 27,184,719 bytes. CN714F does not add raw XML/logs to its evidence branch; it retains derived counts, hashes and reviewed complete first stacks, with no dump blocks.

The single required-check watch ended at 2026-10-23T10:41:08.063799+00:00 with exit 124, after 5400.57 seconds. Read the final CI summary and report for the exact verdict. It did not enqueue, merge, rerequest or mutate buildtest runs.

Final CI observation: The single required-check watch reached its 90-minute cap at 10:41:08 UTC. bld-build-release passed; bld-build remained in progress, with no terminal test verdict. The informational remote check also remained in progress at https://buildtest.kotlin.build/run?id=126c7c6c. No check or run was rerequested, rerun, deleted or cancelled by this lane.

The separate process guard finished with: Requested watcher stop at original cap. It only managed the exact watcher PID, not any CI run.

What the tests established

Readiness now requires the launched process or a descendant to own the responding TCP listener. Survivor adoption also checks the saved process start identity, when present. Both HTTPS accept loops reject sockets delivered after provider drain starts. Prior fail-first proofs, fixed runs and independent reviews are on CN714C/D evidence branches; the new callback finding below qualifies the earlier no-findings conclusion.

The bridge guard remains open. Resolver 1328 returned 1 MiB, 1 MiB, 1 MiB, 123 bytes, then EOF for 3 MiB + 123 bytes: five outer pulls. Resolver 1333 returned 8,192, 1,040,384, 32,768, 1,015,808, 131,072, 917,504, 123 bytes, then EOF: eight pulls, with complete content. The new resolver's adaptive read behavior permits a short first result, while the unchanged guard requires 1 MiB. The owner must decide the exact contract; the scale test does not replace this requirement.

The separately committed 58 MiB test uses a real local backend RPC connection and the actual facade handler, checks every byte and exact length, and requires EOF within 117 outer handler pulls. The bound is twice 58 measured 1 MiB data quanta plus EOF, independent of machine speed; 256 KiB outer chunks require 233 calls and fail it. It bypasses outer wire framing, so it makes no network-latency claim. The first draft had an EOF decoder cast error; that rejected draft is preserved separately and is not a throughput result. Final test commit 9ffc64c1 accepts binary or base64 replies without changing content, count or EOF assertions.

The unchanged public test testHttpsAcceptCallbackChangesTlsMode failed all three branch attempts and passed attempt 1 on main. Its real listener updates the mapping from its accept callback, switching from passthrough to TLS termination. That cancels the old accept job; the added !isActive check then closes the returned socket while the provider remains running. INFER: this exact source ordering explains the new callback failure. A fix must distinguish cancellation during route replacement from provider drain and preserve both public behaviors. See admission-review-finding.md and the complete first stack. No code fix or branch rerun was made in this continuation.

The soak has distinct observations. CN714C saw 313 threads against a 310 bound on unchanged main/resolver 1328. CN714D's fresh main control instead hit the existing 30-minute watchdog in all three automatic attempts, at the publisher-future wait on line 446. CN714F's branch full suite hit the same watchdog. Do not infer a shared mechanism from those different failure signatures; another lane owns the thread-count diagnosis. No timeout, iteration count, thread limit or assertion changed.

Every first failure and its main control

Seven new main controls used unmodified main d84bf644 and byte-identical test files, each invoked once. Six passed attempt 1; inherited idle failed first through the same TLS read-timeout path and passed an automatic retry. The bridge and soak comparisons are preserved CN714D controls, not new reruns. A main pass only means the failure did not reproduce in that run; attribution for the other failures remains unsettled. Complete first stacks and main summaries are in CN714F lane-evidence.

Relevant PRs and refs

PR: https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 . It is open and unmerged at the last verified read. The table preserves recovery sources, controls and rejected drafts; do not replay experimental branches wholesale.

The 10:41:26 UTC process audit found zero owned live observers/wrappers and zero processes in either source checkout. CN714F source and baseline checkouts are clean; no source stash or unpushed code remains at the latest audit. Recheck the final process audit and Git state before relying on that statement. No code was deployed, published or restarted by CN714F; its commands were local verification, branch/body/handoff updates and read-only CI observation. Published resolver 1333 was pre-existing. There is no deployed-but-unmerged code introduced by this continuation.

Next steps

1. Read the final CI summary, PR body, classification and first stacks before any new run. The bounded watch has ended. Read the existing run outcome before any later verification; do not restart a run or watcher unchanged merely to obtain a green result. No merge or enqueue is authorized.
2. Resolve the bridge contract with the owner. Keep the original guard unchanged until that ruling; complete content plus bounded work does not answer the first-pull requirement. An upstream contract change would need fail-first tests at that layer and separate authorization for publication, which this lane does not have.
3. Fix the callback behavior using the existing public reproducer and both late-socket regressions together, retaining assertions and limits. The 3/3 branch failures and main first pass are preserved; no machine-tuned reproducer is needed.
4. Continue the separately owned soak diagnosis and investigate the other first failures at their actual layer. Do not relabel retry passes as success or main single passes as proof of attribution. Preserve the nine-row failure register until each issue is resolved with evidence.
5. For any subsequent authorized source change, fetch/rebase, run targeted tests, then one correctly budgeted full suite under the shared lock. No timeout increase, iteration reduction, skipped test or weaker assertion is permitted. Update the same still-open PR with an explicit expected-head lease and structured body PATCH/readback; do not merge.

Operational and evidence knowledge

Scratchpad was /tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad. Current source: workspace/CN714F; main controls: workspace/CN714F-main; evidence checkout: workspace/CN714F-evidence; findings: out/CN714F.md. Recover from remote branches after a wipe, not those paths. The complete local suite XML/log path is out/CN714D/full-suite.xml and full-suite.log; derived output is under CN714F lane-evidence.

Important CN714F files: CN714F.md, full-suite-summary.json, classification.json, all -first-stack.txt, main--summary.json, main-control-results.json, source-verification.json, admission-review-finding.md, memory-before.txt, memory.jsonl, memory-after.txt, watch-start.json, watch-result.json, watchman.log, ci-summary.json, status-final.md and process-audit.json (the final files exist once the lane ends). The streaming summarize.py discards console bodies and keeps only metadata/stacks; attempt-history is a sibling of test, not nested inside it. CI/raw XML may contain runtime dumps; never push it without deriving and reviewing safe summaries.

The handoff client first failed to resolve any peer after an update returned success; a later get recovered and exactly matched the 09:16 submitted body. A separate 09:39 report found one peer but all direct/relay bytecode fetch candidates failed. The 09:59, 10:19 and 10:39 reports succeeded. These are distinct observations, with no shared cause asserted; exact bounded diagnostics are in handoff-readback-challenge.md. Do not assume a write was read back until the comparison succeeds. report-challenge CLI was not run because it automatically enqueues/merges, prohibited here.

Required reading: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md, then project README/AGENTS and the source/evidence named above. Common lane rules override generic remote-build preference and instructions to treat the informational remote check as required.