Handoff: Finish first-attempt validation of the UrlResolver and ContainerNursery PRs

Written 2026-10-22 11:26:28 UTC. RE-VERIFY: This is a write-time snapshot. Recheck both PRs with gh pr view <full-URL> --json state,mergeStateStatus,statusCheckRollup,mergedAt, and remote branch heads with git ls-remote. Fresh results override this document.

Design decisions in force

The owner’s direction for this effort, verbatim, dated 2026-10-21:

Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos

If there are gaps, have sol write the missing tests.

Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation.

The lane’s explicit constraints, dated 2026-10-22, remain in force:

Build/test: scripts/build.bash --local and scripts/test.bash --local --test <name> --log <file.xml>. Never --remote (killed remote clients leave orphaned runs). The runner retries a failed test up to three times; count a pass only when attempt 1 passed (no attempt-history in the XML).

Absolute rules: never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test. A failing test is a defect report: fix the cause. Error messages state what was wrong and the actual value.

You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs. Update a PR body with gh api --method PATCH repos/<owner>/<repo>/pulls/<n> --input body.json and read it back. GitHub API calls are rate-limited across many lanes: poll at most every 2 minutes.

CI: required checks are the GitHub Actions jobs (bld-build, bld-all-tests, bld-build-release, as the repo defines). kotlin.build (remote) is informational in these repos; if it is red, read its log once and report the cause (code vs infrastructure) in findings — do not rerun it. Stop watching CI after 90 minutes and report.

Do not touch branch wip/c1-injected-framing-reproducer (separate bug). Do not merge.

Full suites must hold /tmp/claude-501/complete-suite.lock, sample /sys/fs/cgroup/memory.current every few minutes, and use no added JVM tuning. A killed suite is reported, not blindly retried. All lanes share a 10 GiB cgroup. Long processes need tracked tasks and hard deadlines; terminate owned background work before final and verify ps -eo pid,args -ww. Never identify a process to signal by test name alone: verify its ancestry and process group. Commit trailers and PR suffixes remain exactly those in the original lane brief (Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; 🤖 Generated with [Claude Code](https://claude.com/claude-code)). Findings belong only on a separate WIP evidence branch.

Read before tests or implementation: Testing architecture, including extreme conditions and deterministic contract-based reproduction, and Engineering philosophy. These were read in full during this session. The explicit lane rule makes remote checks informational even where older general documentation says otherwise. The Handoff service is authoritative; the older handoff README’s merge exception is superseded. The report-challenge CLI was not run because it would automatically merge a record, forbidden in this lane; operational findings are preserved in the evidence branch instead.

Mission summary and current gates

Open two non-draft PRs, prove the requested fail-first and final repetitions, run the full suites under the lock, and obtain green required Actions checks without merging. Both PRs are now open:

• https://github.com/CodexCoder21Organization/UrlResolver/pull/1431 — head c36784aca7878abbf6835a685803218d59151ca9, main base 71499899e547fb0cd62ff6e7fc04b25c0a97efc1. Open and unmerged; required bld-build passed at 10:45:50 UTC. Actions job.
• https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 — head 1bafd427a292b717628f0d94df9253d9eb8f7192, main base 57a4e8000738a069f55d51be09e6b8f19a39c0b3. Open and unmerged, with a first-line “do not merge” hold. At 11:25 UTC bld-build-release had passed and bld-build was running. Release-build job, build-and-test job.

The request is not complete merely because both PRs exist. The item audit is:

What was found and done

1. The buffer allocation defect is independently established. The brief’s JFR evidence showed 3,711 GC events and 66.7 seconds of pause in 180 seconds. handleStreamPull allocated up to 1 MiB before every read, even tiny headers, short pieces and EOF. The fix starts at 8 KiB, uses bounded source availability and grows after full reads to the existing 1 MiB ceiling. It performs one source read per pull and returns an independently owned chunk. Availability exceptions preserve ownership, and re-entrant close cannot restore registration. The invariant table and independent test/code review are in C1F.md and C1F-review.md.

2. Fail-first and final focused evidence are durable. Test-only commit 51eb9c4915abfd08a9309899f0a800d6115e2292 has production/build/script source identical to parent 9f282dfc67af6e05aa9e2fbb358fdeb4ff67231f. testSmallPullStreamsAllocateForTheirData failed attempt 1 in 952 ms: Stream 0 offered a 1048576-byte array to read(b, off, len); the limit is 8192 bytes; both automatic retries failed the same assertion. Final-head XML is C1F-newbase-ur-{1..5}.xml, seven clean tests each. The seventh existing relay fixture was corrected to report its known remaining payload from available(), preserving all four simultaneous full-size buffers, 128 MiB heap and original assertions. No timeout or iteration changed.

3. Resolver full-suite limitation remains visible. testBytecodeDiscoveryDeadlineExpiresDuringSharedFlight failed attempt 1 with The worker must still be held when the caller returns. Expected <1>, actual <0>. and passed attempt 2. The callback waits on workerRelease.await(10, SECONDS) while production starts a wall-clock 10,000 ms discovery budget before worker admission. The equal budgets can release the callback before caller post-processing/assertions. This explains the fixture’s race mechanism, but no deterministic new reproducer or fix was made. Do not call the full suite first-attempt clean, and do not increase its timeout. Earlier recovery-test retries are also preserved; later isolated first passes do not establish their causes.

4. Consumer fixture port ownership was a real separate defect and is corrected. Restarting each sender at the same released listener port can fail if another caller acquires that port. A public-socket control holds former endpoints; the old fixed-port setup failed on attempt 1 in all ten invocations, including all 30 runner attempts. Each sender generation now requests port 0 and advertises its actual address while keeping the same 32 identities. Original workload, counts, resource ceilings and assertions remain. The finite 100-round workload can merge up to 3,200 listen addresses under those same IDs, which is part of the measured process budget, not a claim of a flat address catalog.

5. The warmed I/O control is limited, and no remaining thread cause is claimed. Opening/closing one AsynchronousSocketChannel showed 17 JVM-owned persistent workers on this 16-CPU host (six threads before, 23 after). The final soak initializes that shared group before baseline. It still fails. Current full-suite added-thread dumps include 16 RelayService-race-dial workers, eight noise-handshake-crypto-*, several JVM asynchronous-I/O completion workers and other runtime workers. Current read-only protocol source shows eight lazy fixed race-dial workers per RelayService. Some measured growth is therefore late pool startup; the evidence does not yet identify whether the remaining bound failure is incomplete warming, unexpected ownership, or another implementation defect. Thread-creation JFR events are historical events, not a list of live retained workers. Never infer a leak from their count alone.

6. A maintenance-warmup prototype was tested and rejected. It tried to require a stopped setup peer in both owners’ registries and call public pruneOfflinePeers(maxAge=0) before baseline. Two attempts found that the consumer correctly knew only its configured relay, refuting that precondition; another still exceeded the thread bound even after the race-dial cohort was initialized. The 19-line prototype was reverted in current head. Preserve branches wip/c1f-soak-public-maintenance-baseline and wip/c1f-maintenance-newbase as rejected controls, not fixes. No 100 ms ping timeout was adopted: nested connection calls have their own longer budget, so a short outer number would not establish that budget.

7. Capacity is five-clean; publish has an unresolved bytecode failure. Capacity’s exact-head first-pass durations were 36,144 / 52,205 / 75,287 / 67,624 / 62,795 ms. A resource-admission cancellation is excluded. Publish’s fourth repetition failed its first two attempts reading 1,834,966 and 786,390 of 2,337,686 bytes, then passed attempt 3. These lengths are 7 and 3 times 262,144 minus 42; that arithmetic is a clue, not an identified framing cause. The excluded framing branch was never accessed or changed. Full stacks are C1F-cn-maintenance-publish-4-failures.txt and corresponding XML. Released resolver 0.0.1332 failed all three soak attempts; one separate publish comparison passed once and was not adopted. Its relation to the bytecode failure is unproved.

8. The exact bug-6 implementation is insufficient for the soak. The local thin resolver build used source identical to the PR fix and protocol 0.0.1429, matching all 80 generated Maven dependencies of released 0.0.1332. Synthetic coordinate foundation.url:resolver:0.0.1333-c1f-local was only a local-file diagnostic label, never a release. JAR SHA-256 was 712db63593917ba84f07df3ab6868f23c8e9c740a0f3827e6de06056ac599a0e; POM SHA-256 2acbf18775e97043b8378b06a5f0b15e4219c25fb7cc8554333d47307e7e2b59. Each actual target worker selected that exact resolver and protocol 0.0.1429 (8d5c86431a98db3823174a9cab9fb698d7008f04164eca9bf4fdde88bee3d138), with no alternate resolver. All three attempts failed at round 20: baseline/current 266/295 (307,555 ms), 263/304 (504,864 ms), 267/300 (295,860 ms). Thus publishing bug 6 alone is not demonstrated to clear this failure. Source/dependency identity was controlled; do not claim byte-for-byte equivalence to released 0.0.1332, because compiled Kotlin module metadata differs (foundation_url_resolver versus foundation_url_resolver_sandbox).

9. Preparation failures and a tracking-reference correction are not test results. Calling the Java getter name getDefaultMavenRepositories() from Kotlin failed compilation; the correct property is defaultMavenRepositories. Next, a per-coordinate MavenPrebuilt2 repository resolved compilation but was lost during generated-POM runtime resolution. File-level WithRepository annotations are the supported fix for that diagnostic setup; see the exact recipe below. One preparation was canceled while checking a stale origin/main ref. Fetching actual main then showed HEAD already contained the new main; the claim that a stale tracking ref proved old HEAD source was wrong and is withdrawn. The actual primary and diagnostic production source trees were identical, 207 paths, manifest SHA-256 b0d14f074a153f299f7cea2cdf3c960ef9150370a946c7b6f241f4cacc6195bb. The canceled preparation launched no target worker and produced no XML. It is not a failed runtime test.

10. The one full consumer suite has two genuine failures plus one agent interruption. Final XML has 908 tests: 907 final passes, 906 first-attempt passes; all three build rules passed. Publish and capacity passed first attempt in 29,797 and 60,723 ms. Soak attempt 1 failed at 268→300 threads in 550,374 ms; attempt 3 at 265→292 in 547,961 ms. Attempt 2 exited 143 after a diagnostic probe selected PID 91247 by name and sent SIGTERM without checking its parent/group. That PID belonged to the primary full suite, not the diagnostic. Root disclosed the error; all later checks were scoped by parent and group. Treat that attempt as agent interference, not a library failure, OOM or watchdog timeout. The single suite is not uncontaminated validation and was not manually rerun.

11. An existing subprocess fixture also retried. testZombieCleanup_dropletE2eMatchesProductionTopology failed attempt 1 when subprocess cn1 could not bind port 36985 (Address already in use) and exited 1 before the exact ready signal. The second automatic attempt passed. The warning about lowering oom_score_adj was not the startup cause; the emitted BindException was. Full stack and recent subprocess output are preserved. The suite’s queued first memory sample was 9,947,783,168 bytes; actual start logged 8,806,301,696 bytes; sampled peak 10,737,385,472 under limit 10,737,418,240. All sampled oom, oom_kill and oom_group_kill counters were zero. Do not describe exit 1 or the signaled worker as an OOM.

Relevant PRs / refs

The branch table below comes from live git ls-remote at 11:18 UTC. All PR checkouts and the isolated diagnostic were clean; all stashes were empty and each had only its listed worktree. Local git log --branches --not --remotes can misleadingly show pushed commits because the narrow fetch refspec does not track every WIP branch; remote SHA verification below is the authority. The read-only UrlProtocol clone was clean on main 50a481caa2b27d3883c3c9f6808d81ac3acb04b9; it supplied ownership clues, not an adopted change.

Deployed or published but not merged

None from this session. The action ledger consists of local builds/tests, Git branch pushes, PR creation/body PATCH requests, read-only CI watchers, and Handoff service records. No publishing/deploy/restart/merge/enqueue command was run. Current consumer dependency pins remain resolver 0.0.1328/protocol 0.0.1429. The synthetic artifact only lived in a scratchpad file repository. The release-build PR job does not publish its main-push-only release steps on a PR event.

Evidence and reproducible commands

The evidence branch contains the complete findings, independent review, raw XML, full stacks, JFR recordings/derived summaries, profiling scripts and an artifact index. The index records uncompressed hashes and remote filenames; large logs/XML and the partial recording are gzip-compressed. Download/unzip before parsing. Maven JAR/POM outputs and caches are deliberately omitted: regenerate them from the source branches, with historical hashes preserved in identity JSON.

Key files (all under lane-evidence/):

• C1F.md, C1F-review.md: incremental record, current plan and dedicated independent reviews.
• C1F-baseline-small.xml, C1F-newbase-ur-{1..5}.xml: resolver baseline and 35 clean final focused results.
• C1F-ur-full-newbase.xml / .xml.gz, C1F-ur-newbase-retry-failures.txt: resolver final full suite and exact retry stack.
• C1F-cn-reviewed-capacity-1.xml, C1F-cn-reviewed-capacity-resume-{1..4}.xml, C1F-cn-reviewed-capacity.md: five exact-head clean capacity runs; canceled run kept separately.
• C1F-cn-maintenance-publish-{1..4}.xml, C1F-cn-maintenance-publish-4-failures.txt: publish repetitions and bytecode failures.
• C1F-cn-full-reviewed.xml.gz, C1F-cn-full-reviewed-histories.txt.gz, C1F-cn-full-zombie-attempt1-stack.txt, C1F-cn-full-reviewed-memory.jsonl: complete final consumer suite, including the interference disclosure.
• C1F-cn-local-bug6-repository-resumed-soak.xml.gz, corresponding .log.gz and -stacks.txt, C1F-cn-local-bug6-repository-resumed-classpath-identity.txt, C1F-local-bug6-runtime-root-verification.json: artifact-verified comparison and all three failures.
• C1F-local-bug6-consumer.md, C1F-local-bug6-repository-research.md, C1F-local-bug6-artifact-identity.json, C1F-local-bug6-pom-comparison.json, C1F-local-bug6-class-comparison.json: full setup/corrections and identity controls.
• C1F-cn-full-soak-thread-groups.json, added-thread stack files, raw/derived JFR files: ownership evidence. Full-suite attempt 1 had 48 newly observed IDs but net growth 32; attempt3 had 41 IDs but net growth 27, demonstrating why new-ID count is not net retained growth.
• C1F-urlresolver-pr.md, C1F-containernursery-pr.md, PR JSON readbacks and CI watcher logs: exact reviewable descriptions and status snapshots.

Clone fresh under the supplied scratchpad’s workspace/C1F/ (or a new lane subdirectory). Do not reuse another lane’s repo. Read AGENTS.md and README after cloning. A single-branch clone must add the main fetch refspec before the guard:

git config --add remote.origin.fetch '+refs/heads/main:refs/remotes/origin/main'
git fetch origin
git rebase origin/main

Reassess rather than continuing if the tested head changes. Target commands are:

scripts/test.bash --local --test testSmallPullStreamsAllocateForTheirData --log /absolute/out/buffer.xml
scripts/test.bash --local --test testRelayGossipDoesNotStarveMultiFilePublish --log /absolute/out/publish.xml
scripts/test.bash --local --test testRelayCapacityEvictionChurnPreservesRpcProgress --log /absolute/out/capacity.xml
scripts/test.bash --local --test testRelayGossipPublishSoakReleasesResources --log /absolute/out/soak.xml
flock /tmp/claude-501/complete-suite.lock scripts/test.bash --local --test . --log /absolute/out/full.xml

The seven resolver names are in C1F-newbase-resolver.py. Do not run these again merely to obtain a green retry. Reproduce a named mechanism deterministically, fix it, then rerun the same test; count a clean result only if its <test success="true"> has no corresponding top-level <attempt-history name="...">. Attempt history is a sibling, not a child of <test>.

To reconstruct the local-artifact comparison, clone the two WIP diagnostic branches, verify controlled source/dependency differences, build a thin Maven output locally, stage its generated JAR/POM under a standard Maven root, and set C1F_LOCAL_MAVEN_REPO to that root:

scripts/build.bash --local foundation.url.resolver.buildMaven /absolute/out/local-resolver-maven

The consumer WIP’s build declaration uses MavenPrebuilt2(..., repositories = listOf(File(envRoot).toURI().toURL()) + defaultMavenRepositories). The existing soak additionally needs these file-level annotations (update the file URI if the root path changes), with import build.kotlin.withartifact.WithRepository:

@file:WithRepository("file:///absolute/out/local-maven/")
@file:WithRepository("https://kotlin.directory/")
@file:WithRepository("https://repo1.maven.org/maven2/")

These annotations feed follow-on generated-POM transitive resolution. A nonempty list replaces defaults, so preserve both public repositories. The pinned CLI has no --repository or --include option; --workspace discovers build scripts and is not that repository mechanism. Do not edit live caches or publish the synthetic coordinate. Check actual target-worker ancestry, expand its @argfile with a shell-aware parser, and verify the flattened classpath filename and staged JAR hash; the bootstrap CLI itself legitimately uses an unrelated old resolver and is not target runtime evidence.

Next steps

1. Re-verify both PRs, actual required Actions, main and source heads first. Keep the consumer hold. Do not merge or enqueue either PR.
2. Finish the existing bounded consumer CI watch without rerunning jobs or treating informational remote status as a gate. If a remote result becomes red, read its log once and classify the cause. If required CI fails, diagnose locally; do not requeue it as a debugging strategy.
3. Build the missing per-resource ownership/lifecycle table for the soak: fixed per-owner race workers, crypto workers, shared JVM asynchronous-I/O workers, RPC workers, gossip and caller executors. Distinguish a legitimate late startup cohort from repeated growth. Use only public APIs to force the exact condition. The one-channel warmup and the rejected maintenance prototype do not establish a complete warmed baseline; do not simply enlarge the thread allowance or sleep until it happens to pass.
4. Explain and reliably reproduce the remaining thread-bound condition, then fix the responsible implementation or a proven fixture error while preserving every existing bound and workload. Bug 6 alone is insufficient. If the defect is upstream, fix it upstream with a failing consumer test and a failing upstream test; publishing remains forbidden in this lane, so record any actual release dependency accurately rather than inventing a version.
5. Resolve or coordinate the incomplete bytecode response through the permitted workflow without touching the excluded framing branch. Keep the exact read counts and full stacks. A four-byte-frame-looking arithmetic pattern is not proof of the source.
6. Investigate the existing deadline-fixture race and the independent subprocess port-allocation gap as separate changes if needed; do not hide retries in a green suite total or mix unrelated changes into the current test PR without reassessing scope.
7. After proven corrections, establish five clean exact-head repetitions for all three consumer scenarios, then an uncontaminated locked full suite and required green CI. Rebase first; stop on source drift. Update each PR body via PATCH and readback, retain WHY first, and preserve the requested attribution/footer. No assertion, timeout, iteration, heap flag or enabled state may be relaxed.
8. Push each milestone and refreshed findings to a WIP evidence branch. Refresh this handoff’s branch table and status after live verification. On stopping, terminate only owned processes after ancestry/group checks, verify ps -eo pid,args -ww, release the handoff claim and leave it incomplete until every remaining gate is actually satisfied.

Reusable operational knowledge

• The prior disk wipe makes local-only evidence unacceptable. All touched clones had no stashes or uncommitted source; the relevant remote SHAs are in the table. Generated Maven outputs/caches are intentionally not committed.
• High memory.current includes reclaimable file cache. Record anon/file/kernel and actual OOM counters; do not infer an OOM from near-limit memory alone. TerminalMemoryWarningEffect about a record not being the final stdout line is output ordering, not an OOM.
• A process-name match is insufficient when identical tests run in separate checkouts. The full-suite interference was an operational mistake, not a code result. Verify parent chain and process group before attaching a result or sending any signal.
• The consumer full-suite lock is machine-wide. Additional compilers were held during its preparation phase to avoid the previously observed shared-limit kill; targeted runs resumed only after preparation completed.
• Build-watchman is read-only here: never use --to-merged, obey the lane’s informational-remote rule, and ignore its automatic suggestion to rerequest a queued informational suite. Use interval 240 seconds or slower; status polling must remain at least two minutes apart.
• The source of any global thread must be established from live stacks and its creating public path. Historical thread-start events cannot establish live retained growth, and a green retry cannot establish a fix.