← Priority list

Resolve ContainerNursery pull contract and full-suite failures before clearing the hold

Published resolver 0.0.1333 returns 8192 bytes where the existing bridge test requires 1048576; the full suite also has a soak thread-bound failure and two other first failures. A scope decision is needed before changing the contract or published coordinate; publication and weakened assertions remain prohibited.

Handoff document

Markdown

Handoff: Resolve ContainerNursery released-resolver pull contract and full-suite failures

Written 2026-10-23 00:19 UTC by CN714B. This is an in-flight snapshot; required Actions observation is continuing. PR state/head were directly checked at 23:27 UTC, with the bounded watch still active at 23:33 UTC.

RE-VERIFY: Read https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 before acting. Recheck its state, head, checks and body with gh pr view <URL> --json state,headRefOid,statusCheckRollup,body. Poll GitHub at most once every two minutes. The PR remains held; do not merge or enqueue.

Design decisions in force

Owner direction, verbatim, 2026-10-21: "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

This lane's constraints, 2026-10-22: select published foundation.url:resolver:0.0.1333, SHA-256 07af6af306dbdbf78f1347bb1f2a947efe87df43304003a1ee22a6a9ed206526; three first-attempt passes per new test; retain all 100 soak rounds and existing 1800-second/128-MiB limits; one full local suite under the machine-wide lock. "Never --remote"; "never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test." No merging, enqueueing, publishing, deploying, restarting services, deleting or rerequesting buildtest runs. Keep the do-not-merge line unless all validation passes.

Mission and current state

Finish released-dependency validation for the existing relay-publication extreme-condition PR. Both resolver declarations are updated and pushed. Nine standalone targeted runs pass on attempt 1, but the full suite is not clean. The published adaptive pull behavior conflicts with an existing ContainerNursery first-pull contract. That conflict cannot be resolved by replacing the specified immutable artifact or weakening a test. A scope question is pending about an upstream correction/new release; no additional scope or publication is assumed authorized.

Brief items

Item	State and evidence
Inspect main pins; select published resolver	Done. Main explicitly pins resolver 1328 and protocol 1429. Candidate changes only resolver to 1333 in build.kts and build.gradle.kts, honoring their existing alignment comments. Runtime mediates protocol 1431 from the resolver POM; no direct protocol edit needed.
Prove actual loaded release	Done. Each of nine targeted workers and both full soak workers has exactly one StreamAwareServiceHandler provider with the expected resolver JAR hash; javap shows adaptive pull state/read sizing. No synthetic coordinate or repository.
Three new tests three times each	Done, all nine first-attempt passes, no attempt-history. Publish 8602/13515/29598ms; capacity 38489/18389/30389ms; soak 867734/1188105/959089ms. Each soak has unchanged 100 rounds and final 103424-file assertion.
One locked full suite	Executed once at exact candidate head, failed:906/910 first attempts,909/910 final, three build rules pass. Not a clean validation gate. See failures below.
Rebase, push, update body	Done. Rebase was a no-op retaining tested head a450e2c6f7914414e68581067b2b865a73c0edf1. Existing OPEN PR updated with the two resolver-pin commits. Body PATCHed using JSON and exact GET readback verified, WHY first and hold retained.
Required checks	In progress. build-watchman tracks bld-build plus bld-build-release. Release-package Action https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867886881 passed; build/test https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867887364 is in progress at 22:59 UTC. No rerun.
Remove hold; complete handoff	Not permitted: full first-attempt gate failed. Keep handoff open and report the precise blocker.

What was found

Published first-pull contract conflict

The existing testUrlFacadeRpc_bridgeFillsRequestedOuterPull failed all three full-suite attempts: Outside pull 1 must fill its requested 1 MiB from the backend stream.. Expected <1048576>, actual <8192>.

Exact published resolver source: https://github.com/CodexCoder21Organization/UrlResolver/blob/b31968f5740d720aaaf1d2e394276ce31d0862a5/src/foundation/url/resolver/StreamAwareServiceHandler.kt#L526 . The first pull buffer starts 8192 at line 1110, sizing is min(requested maximum,max(available,next size)), one source read returns that chunk, and growth affects the next pull. CN's StreamBridgeInputStream reports zero currently buffered bytes before its first refill. Therefore the first requested 1 MiB outer pull returns 8 KiB even when the backend has 3 MiB. The published-commit README explicitly documents one read and the 8 KiB initial buffer. Upstream testShortPullReturnsOneReadWithoutFillingBuffer also asserts exactly one read and 8192 initial array. Simply restoring an unconditional fill loop would violate neighboring upstream coverage. This is a demonstrated contract conflict, not a timeout or host-load explanation. No workaround or assertion change was written.

Soak round40 thread-bound failure

First full soak: 592935 ms, baseline raw 267/accounted 291, race 0/16 and Noise 0/8. Round 20 has 315 threads, round 40 has 319, exceeding 315. Full stack is CN714B-failure-4-attempt-1-stack.txt. Diagnostic list contains 68 added IDs: 16 race, 8 Noise and 44 others, all parked. The ID dump is a second snapshot; IDs can include replacements. It does not prove a leak or identify the net increase's cause. Independent census is in CN714B-review.md.

Automatic second attempt completes 100 rounds in 1233457 ms: heap samples 60106456/61336968/59297552/60293672/59798416, FDs 334/334/334/336/337, threads 297/307/301/307/306, final 103424files and zero handles. This retry does not count as first-attempt success.

Standalone and full main soak class bytes match SHA 22551c58ceb87ef117acafbc8a1af6e21b1a839aefcfb93dd70d55bf43e738ea. All 213 non-generated-test-JAR classpath basenames/order and JVM flags match. Generated JAR packaging differs, main class does not. Do not claim all helper class bytes were compared.

Additional read-only inventory: CN714B-thread-factory-inventory.md identifies candidate executor factories in cached artifacts and the JDK. The JVM default asynchronous-channel group can create unnamed daemon cached workers with a 60-second keepalive; the effect handler uses named workers with a one-second keepalive. The scan recovered199/214 classpath basenames, leaving15 artifacts unavailable after temporary classpath cleanup. These are useful source leads, not proof of the captured threads’ ownership or a cause. No new test or fix was written on that basis.

Other first failures

testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer sees a successful TCP connect after the owned listener reports closed; second attempt passes. testDangerJarFileBackedContainer_startResumesReadinessWithoutRespawn unexpectedly returns from first start while its child is gated before bind; second attempt passes. Each full stack and console is preserved. An unrelated listener on a released port is a testable ownership hypothesis, not established from these logs. Production readiness currently accepts any successful connection to its selected port. Do not label these failures host overload or guess fixes.

Runner warnings and shared memory

Five TerminalMemoryWarningEffect warnings reject a supposedly final memory record because later stdout arrived. Their full stacks/raw log are retained; the same warning class was present in the prior lane. They are distinct from assertion verdicts. The report-challenge skill's CLI auto-enqueues/merges, prohibited here, so a detailed WIP record was saved instead of invoking it.

Full invocation waited roughly 38 minutes for the lock, then ran from 22:10:54 to 22:48:23. Memory before request 8734896128 bytes; sampled maximum 10735669248 bytes against 10737418240 cap; OOM counters remain 16/1. The suite was not killed. The lock was direct flock→test only. Do not rerun a full suite blindly.

Handoff client transport observations

Two body-refresh attempts and one status-report upload using CLI0.0.8 reached their unchanged90-second caps. The captured report client waits in UrlResolver.fetchBytecodeResponseCached before submission. Published Maven metadata identifies CLI0.0.9 as latest; the same23:28 report completed successfully through that client at23:33UTC. Both versions still use resolver0.0.862, so this is an observed recovery, not proof of a version-specific transport fix. Full client traces, exact error text and the successful result are on the WIP evidence branch. Use published CLI0.0.9 for subsequent operations; do not increase a cap or restart a service.

Relevant PRs and remote branches

Repo	Branch	Verified remote head	PR	Contents/state
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/tests/c1-relay-publish-extremes	a450e2c6f7914414e68581067b2b865a73c0edf1	https://github.com/CodexCoder21Organization/ContainerNursery/pull/714	Existing3 test additions plus two resolver-pin lines; held; local full gate fails
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/CN714B-resolver1333-20261022	a450e2c6f7914414e68581067b2b865a73c0edf1	No separate PR	Durable source checkpoint, same candidate
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/CN714B-evidence-20261022/lane-evidence	d7315d0a60b1035e44857efca2ffa7cf20d7159e (later checkpoints may follow)	No PR	Findings, targeted XMLs, exact worker/hash proof, full XML/history/stacks, independent reviews and PR-body verification

This lane published or deployed nothing. Resolver1333 was already published from merged upstream commit b31968f5740d720aaaf1d2e394276ce31d0862a5 before pickup. No production process was touched. Source and evidence worktrees are clean; no stash or unpushed code is intended to remain. Fresh upstream read-only clone is clean, with no changes to recover.

Predecessor evidence to retain

The prior lane's detailed notes are copied to https://github.com/CodexCoder21Organization/ContainerNursery/blob/wip/CN714B-evidence-20261022/lane-evidence/CN714B-previous-FIX714.md . Its original handoff snapshot is https://github.com/CodexCoder21Organization/ContainerNursery/blob/wip/CN714B-evidence-20261022/lane-evidence/CN714B-handoff.txt . These preserve the six existing fixture corrections and earlier close/idle thread experiments; they are historical observations, not substitutes for fresh verification. The pre-existing corrections were already in candidate ancestor dccb783900b7f6adf599229fd1f8041fa4a6f9b6 and were not redone in this lane.

Earlier branch evidence remains at https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/FIX714-fixture-corrections-20261022 , https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/FIX714-standard-local-resolver-20261022 , https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/FIX714-standard-runner-evidence-20261022 and https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/FIX714-thread-diagnostics-20261022 . The predecessor's final thread summary is https://github.com/CodexCoder21Organization/ContainerNursery/blob/wip/FIX714-thread-diagnostics-20261022/lane-evidence/FIX714-FINAL.md . No historical branch was modified by this lane.

Next steps and operating details

Reverify PR/checks and read all full-suite attempt histories. Count first attempts, not runner-final passes.
Resolve the pull-contract direction with the owner before broadening scope. Fix an upstream implementation defect upstream with a failing public-API test and corrected release, if authorized. Publishing remains prohibited until explicitly changed. Do not change the immutable1333 artifact or weaken the CN regression.
Diagnose the soak's actual thread-ownership condition with reliable public-API reproduction. Do not raise the raw limit or timeout. Read the existing census and prior FIX714 corrections first.
Investigate the HTTPS/readiness failures with listener-identity evidence; port reuse is only a hypothesis. Never use CI requeues as diagnosis.
Validate a real fix locally, preserve milestones remotely, then confirm required Actions and their uploaded first-attempt XML. Keep the existing PR held until all gates actually pass. No merge is authorized.

Root scratchpad: /tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad; lane source workspace/CN714B; findings out/CN714B.md. These are conveniences only; use the remote evidence branch if disk vanishes. Required reading: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md . Both were read fully by root/reviewer. Fresh UrlResolver README was read fully in bounded sections, exact published source/README fetched, and the upstream short-read regression was read.

Terminal lane snapshot — 2026-10-23 00:19 UTC

BLOCKED. The source remains a450e2c6f7914414e68581067b2b865a73c0edf1 in the OPEN/unmerged PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/714. Validation update at 2026-10-23 00:19 UTC: bld-build: IN_PROGRESS / pending; bld-build-release: COMPLETED / SUCCESS. No completed CI test artifact was available at the final observation; no CI first-attempt pass count is claimed. The local full suite remains 906/910 on attempt 1 and 909/910 after retries; the merge hold is retained. No check was rerun.

The local watch has ended within the lane budget; its terminal record is preserved. No remote run was cancelled, deleted or rerequested. Local source/evidence/upstream checkouts were clean, and final process cleanup is recorded in CN714B-final-process-audit.json when completed. The original fixed-artifact all-green expectation is refuted by the repeated 8192-versus1048576 bridge failure; this is not a retry pass. The three new tests have nine first-attempt standalone passes, including three complete 100-round soaks.

Remaining work: settle the pull contract/new-release scope, diagnose and reliably reproduce the thread-bound and other two first failures, then validate a real correction. Keep all existing assertions, rounds and deadlines. Keep the PR held and the handoff open. No merge, enqueue, artifact publication, deployment or restart is authorized. Latest evidence: https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/CN714B-evidence-20261022/lane-evidence.

# Handoff: Resolve ContainerNursery released-resolver pull contract and full-suite failures

**RE-VERIFY:** Read https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 before acting. Recheck its state, head, checks and body with `gh pr view <URL> --json state,headRefOid,statusCheckRollup,body`. Poll GitHub at most once every two minutes. The PR remains held; do not merge or enqueue.

## Design decisions in force

This lane's constraints, 2026-10-22: select published `foundation.url:resolver:0.0.1333`, SHA-256 `07af6af306dbdbf78f1347bb1f2a947efe87df43304003a1ee22a6a9ed206526`; three first-attempt passes per new test; retain all 100 soak rounds and existing 1800-second/128-MiB limits; one full local suite under the machine-wide lock. "Never --remote"; "never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test." No merging, enqueueing, publishing, deploying, restarting services, deleting or rerequesting buildtest runs. Keep the do-not-merge line unless all validation passes.

## Mission and current state

## Brief items

| Item | State and evidence |
| --- | --- |
| Inspect main pins; select published resolver | Done. Main explicitly pins resolver 1328 and protocol 1429. Candidate changes only resolver to 1333 in build.kts and build.gradle.kts, honoring their existing alignment comments. Runtime mediates protocol 1431 from the resolver POM; no direct protocol edit needed. |
| Prove actual loaded release | Done. Each of nine targeted workers and both full soak workers has exactly one StreamAwareServiceHandler provider with the expected resolver JAR hash; javap shows adaptive pull state/read sizing. No synthetic coordinate or repository. |
| Three new tests three times each | Done, all nine first-attempt passes, no attempt-history. Publish 8602/13515/29598ms; capacity 38489/18389/30389ms; soak 867734/1188105/959089ms. Each soak has unchanged 100 rounds and final 103424-file assertion. |
| One locked full suite | Executed once at exact candidate head, failed:906/910 first attempts,909/910 final, three build rules pass. Not a clean validation gate. See failures below. |
| Rebase, push, update body | Done. Rebase was a no-op retaining tested head a450e2c6f7914414e68581067b2b865a73c0edf1. Existing OPEN PR updated with the two resolver-pin commits. Body PATCHed using JSON and exact GET readback verified, WHY first and hold retained. |
| Required checks | In progress. build-watchman tracks bld-build plus bld-build-release. Release-package Action https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867886881 passed; build/test https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867887364 is in progress at 22:59 UTC. No rerun. |
| Remove hold; complete handoff | Not permitted: full first-attempt gate failed. Keep handoff open and report the precise blocker. |

## What was found

### Published first-pull contract conflict

The existing `testUrlFacadeRpc_bridgeFillsRequestedOuterPull` failed all three full-suite attempts: `Outside pull 1 must fill its requested 1 MiB from the backend stream.. Expected <1048576>, actual <8192>.`

Exact published resolver source: https://github.com/CodexCoder21Organization/UrlResolver/blob/b31968f5740d720aaaf1d2e394276ce31d0862a5/src/foundation/url/resolver/StreamAwareServiceHandler.kt#L526 . The first pull buffer starts 8192 at line 1110, sizing is min(requested maximum,max(available,next size)), one source read returns that chunk, and growth affects the next pull. CN's StreamBridgeInputStream reports zero currently buffered bytes before its first refill. Therefore the first requested 1 MiB outer pull returns 8 KiB even when the backend has 3 MiB. The published-commit README explicitly documents one read and the 8 KiB initial buffer. Upstream `testShortPullReturnsOneReadWithoutFillingBuffer` also asserts exactly one read and 8192 initial array. Simply restoring an unconditional fill loop would violate neighboring upstream coverage. This is a demonstrated contract conflict, not a timeout or host-load explanation. No workaround or assertion change was written.

### Soak round40 thread-bound failure

First full soak: 592935 ms, baseline raw 267/accounted 291, race 0/16 and Noise 0/8. Round 20 has 315 threads, round 40 has 319, exceeding 315. Full stack is `CN714B-failure-4-attempt-1-stack.txt`. Diagnostic list contains 68 added IDs: 16 race, 8 Noise and 44 others, all parked. The ID dump is a second snapshot; IDs can include replacements. It does not prove a leak or identify the net increase's cause. Independent census is in CN714B-review.md.

Additional read-only inventory: `CN714B-thread-factory-inventory.md` identifies candidate executor factories in cached artifacts and the JDK. The JVM default asynchronous-channel group can create unnamed daemon cached workers with a 60-second keepalive; the effect handler uses named workers with a one-second keepalive. The scan recovered199/214 classpath basenames, leaving15 artifacts unavailable after temporary classpath cleanup. These are useful source leads, not proof of the captured threads’ ownership or a cause. No new test or fix was written on that basis.

### Other first failures

`testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer` sees a successful TCP connect after the owned listener reports closed; second attempt passes. `testDangerJarFileBackedContainer_startResumesReadinessWithoutRespawn` unexpectedly returns from first start while its child is gated before bind; second attempt passes. Each full stack and console is preserved. An unrelated listener on a released port is a testable ownership hypothesis, not established from these logs. Production readiness currently accepts any successful connection to its selected port. Do not label these failures host overload or guess fixes.

### Runner warnings and shared memory

### Handoff client transport observations

## Relevant PRs and remote branches

| Repo | Branch | Verified remote head | PR | Contents/state |
| --- | --- | --- | --- | --- |
| ContainerNursery | https://github.com/CodexCoder21Organization/ContainerNursery/tree/tests/c1-relay-publish-extremes | a450e2c6f7914414e68581067b2b865a73c0edf1 | https://github.com/CodexCoder21Organization/ContainerNursery/pull/714 | Existing3 test additions plus two resolver-pin lines; held; local full gate fails |
| ContainerNursery | https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/CN714B-resolver1333-20261022 | a450e2c6f7914414e68581067b2b865a73c0edf1 | No separate PR | Durable source checkpoint, same candidate |
| ContainerNursery | https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/CN714B-evidence-20261022/lane-evidence | d7315d0a60b1035e44857efca2ffa7cf20d7159e (later checkpoints may follow) | No PR | Findings, targeted XMLs, exact worker/hash proof, full XML/history/stacks, independent reviews and PR-body verification |

## Predecessor evidence to retain

## Next steps and operating details

1. Reverify PR/checks and read all full-suite attempt histories. Count first attempts, not runner-final passes.
2. Resolve the pull-contract direction with the owner before broadening scope. Fix an upstream implementation defect upstream with a failing public-API test and corrected release, if authorized. Publishing remains prohibited until explicitly changed. Do not change the immutable1333 artifact or weaken the CN regression.
3. Diagnose the soak's actual thread-ownership condition with reliable public-API reproduction. Do not raise the raw limit or timeout. Read the existing census and prior FIX714 corrections first.
4. Investigate the HTTPS/readiness failures with listener-identity evidence; port reuse is only a hypothesis. Never use CI requeues as diagnosis.
5. Validate a real fix locally, preserve milestones remotely, then confirm required Actions and their uploaded first-attempt XML. Keep the existing PR held until all gates actually pass. No merge is authorized.

Root scratchpad: `/tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad`; lane source `workspace/CN714B`; findings `out/CN714B.md`. These are conveniences only; use the remote evidence branch if disk vanishes. Required reading: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md . Both were read fully by root/reviewer. Fresh UrlResolver README was read fully in bounded sections, exact published source/README fetched, and the upstream short-read regression was read.

## Terminal lane snapshot — 2026-10-23 00:19 UTC

BLOCKED. The source remains a450e2c6f7914414e68581067b2b865a73c0edf1 in the OPEN/unmerged PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/714. Validation update at 2026-10-23 00:19 UTC: [bld-build](https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867887364): IN_PROGRESS / pending; [bld-build-release](https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39867886881): COMPLETED / SUCCESS. No completed CI test artifact was available at the final observation; no CI first-attempt pass count is claimed. The local full suite remains 906/910 on attempt 1 and 909/910 after retries; the merge hold is retained. No check was rerun.

Model usage

Recorded agent work, including concurrent agents

21.5 agent-minutes · 28,165,135 tokens

Usage timeline

15-minute columns · empty columns are idle gaps · UTC

claude-opus-5-521.5 min · 28,165,135 tokens

Recorded usage totals by series
Series	Minutes	% minutes	Total tokens	% tokens	Output tokens	First active	Last active
claude-opus-5-5	21.5	100.0%	28,165,135	100.0%	48,264	14 hours ago 2026-10-22 23:30:00 UTC	14 hours ago 2026-10-23 00:00:00 UTC

Agent-minutes add across parallel agents. Tokens include input, cached input, cache writes and output. Percentages are rounded to one decimal place in text and may sum to slightly more or less than 100%.

No status reports yet.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.