Handoff: Finish required CI for the ContainerNursery resolver upgrade and its reaper-test fix

Written 2026-10-26 08:50 UTC — RE-VERIFY before acting. This is an active L133 execution snapshot. Check the live PR with gh pr view https://github.com/CodexCoder21Organization/ContainerNursery/pull/723 --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, inspect the tracked local tasks if they still exist, and use git ls-remote for each branch below. Do not treat the local PID numbers as current after a restart.

Design decisions in force

Owner, 2026-10-26, exact constraints:

• “No reflection, no mocks, no timeout increases, no iteration cuts, no weakening or deleting assertions. Use the injected Clock.”
• “Do the work yourself; no other model sessions.”
• “No notes files in repos.”
• “Full suites only under flock /tmp/claude-501/hd22-suite.lock (one machine-wide); otherwise targeted --local runs; read first-attempt counts from the XML (the runner retries 3x; any retry is a finding with its stack).”
• “Never merge or enqueue.”
• “kotlin.build (remote) is informational (the shared build host is unhealthy); the required check is bld-build.”
• “Push after the first commit and each milestone and verify with git ls-remote (the machine's /tmp is wiped without warning).”
• “Wait loops need a hard deadline and must be killed before you finish. Kill processes only by PID and only ones your own runs started.”

The owner explicitly requested resolver 0.0.1337, protocol 0.0.1434 and jvm-libp2p snapshot-37 and delegated contract-preserving short-pull expectation decisions. No merge, enqueue, deployment or publication was authorized. Applicable session instructions require skill use despite the user's separate request not to read skill files.

Mission and current work

Finish https://github.com/CodexCoder21Organization/ContainerNursery/pull/723 on the requested stack for a published upstream memory-leak fix, prove the large-reply fix with snapshot-32 versus snapshot-37, run affected and full local tests, update the existing PR branch/title/description, and follow required bld-build to completion. The full suite exposed one existing reaper-fixture flake; it is owned in a separate main-based branch and needs its own full validation and PR. Work continues in the current session; this handoff preserves the state because the broader source handoff was archived by its owner and now rejects reports.

What was found and done

1. Fresh clone under workspace/ContainerNursery; root README/AGENTS and the linked Testing Architecture/Engineering Philosophy read. The continuation wip/hd22-L115-resolver-1337 was rebased onto main after the inherited HTTPS startup work (base https://github.com/CodexCoder21Organization/ContainerNursery/commit/1ce1e02d8de17428053281e82c23105904687688). Prior replay-admission tests and response-reader capability/ownership forwarding through both test network wrappers are retained.
2. Resolver 0.0.1337 POM selects foundation.url:protocol:0.0.1434 and community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-32. An initially empty Coursier cache fetched resolver SHA256 d50db4be730e8b2372c525d92e4400c86b1de75f4eb44052d393921585f83132 and snapshot-37 SHA256 196307da543f4b93ace04dab45539556b0801c0038f25e3f606827317d5ba55e, matching the owner's supplied values. Snapshot-32 and 37 POM dependency declarations are identical, so the comparison changes the libp2p artifact version only.
3. Added testUrlFacadeRpc_largeReplyWithConcurrentSmallCalls: real URL facade and local TCP backend, 16 sequential 1,333,487-byte payloads while two small callers run on sibling streams. It checks exact bodies, caller ids, execution counts and continued connection use. Initial fixture counted the startup probe as a small request; it was corrected to count only the named operation, retaining exact assertions. All subsequent comparisons use that same corrected body.
4. Snapshot-32: 3/5 first invocations failed, 2/5 passed. All nine attempts in failing invocations cut the reply short. First-failure received/expected bytes were501347/1333556,501443/1333557 and455667/1333556. The parser reports Stream closed while reading message data through PersistentRpcResponseReader and AmbiguousRpcRequestException; the PR description retains the complete representative stack and all nine byte-count rows. Snapshot-37: 10/10 first-attempt passes, zero retries. JVM class-load logging explicitly names prepared-test-classpath6539770460967621736/classpath/jvm-libp2p-1.3.0-codexcoder21-snapshot-37.jar as YamuxHandler's source; hashing that exact prepared JAR matches196307da543f4b93ace04dab45539556b0801c0038f25e3f606827317d5ba55e.
5. Continuation/main already selected snapshot-38. Both existing paused-stream close tests passed first-attempt against the byte-checked37 artifact, so the old README claim that 38 was required was not supported by those tests. The requested exact 37 was selected in kompile/Gradle and all 18 explicit fixture pins, including exact-version guards. The18 fixture edits were checked to be version-string replacements only. The README now states the verified behavior.
6. Kept the short-pull bridge contract: all 3,145,851 bytes byte-exact, each data pull 1..1,048,576 bytes with its byte count, no premature EOF/extra bytes, at most 7 data pulls, then terminal EOF. Existing framing code already loops until full/EOF; no speculative production read workaround was added.
7. Affected run:106/106 first-attempt passes, zero retries. Core full run from exact5b2ce5f:1160/1161 first-attempt passes, one test retried twice, final1161/1161, exit0. No137 and no full rerun. The failed test is testReaperCleanupAfterStartSettlesBeforeFirstKillReturns: attempts 1 and 2 expected FAILED but observed QUARANTINED; attempt 3 passed. Five advisory TerminalMemoryWarningEffect messages rejected memory summaries followed by later stdout. Those are recorded separately from test outcomes.
8. The requested PR branch wip/BUMPCN-resolver-1334 was force-pushed with explicit lease against old004ec484512ec784ea6e54dbbd83420a7d9b2023 and verified. Title/body are updated with motivation, exact pins, artifact proof, short-pull rationale, local counts and the flake's full stack. Old gh pr edit failed because it queried deprecated Projects classic; REST PATCH with a JSON body-file succeeded. No CI run was blindly retried.
9. Required bld-build completed successfully at the exact PR head. Downloaded artifact XML: 1,161/1,161 first-attempt passes, zero retries. Build-watchman exited GREEN/0 and is stopped. Job: https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40288490555/job/121457741472 . Non-required remote checks are informational under the owner instruction; do not requeue, fix or wait for those gates.
10. Separate flake diagnosis: the fixture paused Clock.unschedule on the startup coordinator, while expireStartupDeadline independently queued cleanup before notifying waiters. That worker could publish quarantine before the old FAILED checkpoint. This is a fixture ordering issue, not a changed production lifecycle implementation. Unchanged main passed 5/5 isolated runs; genuine four-way concurrency of32 independent scenarios exposed 3/3 first failures, with 29/32,32/32 and31/32 failing scenarios; all 9 attempts including retries failed. Fail-first code is preserved at https://github.com/CodexCoder21Organization/ContainerNursery/commit/5ce6a58d4b41cf2b783aaca15900e6a8b1998088 .
11. Fixture fix: hold the first two background Clock readers after provider.start enters, exempt the driver, and let later reaper/provider readers proceed once both slots are held. Both startup/deadline cleanup actors are confirmed paused before the periodic reaper advances. The original single control plus32 concurrent scenarios is consolidated into the existing test. Every original provider step and scenario assertion was compared byte-for-byte after indentation/arming normalization. No assertion, timeout or iteration was weakened. Fixed 10/10 first passes,330 scenarios, zero retries. Neighboring timer-cancellation fixtures were reviewed: other FAILED checkpoints already gate independent expiry reads; management cases intentionally allow expiry and assert QUARANTINED.
12. The separate fix is pushed at f13517f77. Its full validation is queued under the shared lock, tracked session 60017, timeout PID 7888 / flock 7889. The request began about08:22:52UTC; lock wait 1 hour, suite 2 hours, one 137 retry after dmesg/free/cgroup capture, total 5-hour hard bound. It has not acquired the lock or produced full XML yet at write time. Another task owns the lock; do not kill or interfere with it.

Relevant PRs / refs

All rows were verified with git ls-remote at 08:39UTC. Both touched worktrees are clean, git stash list is empty, and git log --branches --not --remotes found no unpushed commits.

Deployment/publication during L133: none. The session only changed GitHub branches/PR metadata and ran tests/readonly checks. The production deployment state was not inspected, and no code here is claimed to be live. The published resolver/protocol/libp2p versions predate this effort; their JARs were consumed and byte-verified, not republished.

The archived background handoff is url://handoff/handoffs/hf-2026-10-25-stop-large-url-replies-being-cut-short-pin-jvm-libp2p-snapshot-37-wherever-urlresolver-0-0-1337-is-used-publish-a-resolver-that-selects-it-and-find-which-hop-truncated-aiclisupervisormanager-s-history-on-2026-10-22 . An upload initially failed fetching bytecode; a bounded retry reached the service and explicitly reported that this object was already completed and archived. It was not reopened.

Next steps

1. Re-verify PR/branch/task state first. The requested PR is green and its downloaded required-CI XML is clean (1,161 first passes, zero retries). Its watcher exited. Never merge or enqueue it. Continue only the separate fixture fix below.
2. Let the separate fixture full task acquire its slot and finish. Local files, if retained: /tmp/l133-run-reaper-full.sh, /tmp/l133-reaper-full-1.log, eventual /tmp/l133-reaper-full-1.xml and /tmp/l133-reaper-full-1-counts.txt. It runs from workspace/ContainerNursery-reaper and fetches/rebases on main immediately before the suite. Recognize success/failure/cancelled/zero-results/deadline states; do not create a silent wait loop.
3. If its full first attempts are clean, fetch/rebase and confirm the tested head, check for any existing PR on that branch, then create a separate PR with why-first description. /tmp/l133-reaper-pr.template.md contains the draft if retained; otherwise reconstruct from this body. Use REST API JSON if gh's deprecated Projects classic query blocks the usual PR commands. Watch required bld-build with build-watchman 0.0.21 under a hard deadline; no merges.
4. The core required CI passed cleanly. Keep the fixture fix separately reviewable; update the core PR description with the separate fixture PR URL once it exists.
5. Before terminal reporting, confirm every watcher/local wait started by this effort has exited; only signal owned PIDs after verifying their identity/start time. Report PR heads, loaded37 proof, s32/s37 first counts, local suite first counts and required CI result. Close this L133 handoff only once this scoped work is genuinely complete, leaving code PRs open as explicitly requested.

Reusable / operational knowledge

• Tests are run with scripts/test.bash --local --test NAME --log /tmp/result.xml. The XML has sibling test and attempt-history nodes; use attempt number1, not final test success, when history exists. The full core XML had 1161 test outcomes although there are fewer test files.
• Full suites require flock --close on /tmp/claude-501/hd22-suite.lock. --close prevents child JVMs retaining the lock descriptor after the parent task exits.
• Root-host free reports 64 GiB, but the process cgroup limit was 10 GiB. Baseline memory.events had no oom_kill; no L133 run exited 137. If it does, follow the owner's one-rerun rule with kernel/memory evidence, not a heap or timeout increase.
• Core local full result file /tmp/l133-full-1.xml is ephemeral; authoritative counts and complete representative stacks are already retained in the PR description. The code needed to repeat every result is on the linked remote branches/commits. Raw duplicated logs, build outputs, caches and scratch notes were not committed under the owner's no-notes/build-artifact rules; necessary findings are in this body and the PR rather than depending on /tmp.
• The challenge-reporting CLI automatically enqueues and merges; do not use that workflow under this task's no-merge rule. Handoff service writes have no PR/merge step. Ignore the obsolete legacy git-handoff merge exception in the mirror README.

08:50 UTC milestone

The requested PR is green at 5b2ce5f268cb626860028b1e5569a615a27d5f65. Required bld-build finished successfully: https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40288490555/job/121457741472 . Downloaded bld-test-results/test_log_file.xml has 1,161/1,161 first-attempt passes and zero retries. Build-watchman exited GREEN/0; its timeout and watcher have finished. The core PR description records both this clean CI result and the earlier local first-attempt retry. No merge/enqueue/deployment. The remaining work is the separate reaper fixture fix: full task session 60017 remains blocked on the shared suite lock, own timeout PID 7888 and flock PID 7889. Its one-hour lock deadline is approximately 09:22 UTC. After full validation, open its separate PR, watch required bld-build with a hard deadline, update both descriptions and complete this scoped handoff.

09:14 UTC milestone

The fixture task acquired the shared lock at09:11:34UTC and full invocation1 started09:11:35UTC. Fetch/rebase left f13517f77f2879f08f2f1fe0cb218ed1482e888a unchanged; head recorded in /tmp/l133-reaper-full-head.txt. Session60017 is now running the suite, not waiting. Its2h run deadline and once-only137 recovery remain in force. No XML result is claimed yet.