AiCliSupervisorManager large conversation-history reply cut short: UrlResolver 0.0.1337 does not fix it; the cause is in jvm-libp2p snapshot-32

Written 2026-10-25 ~20:40 UTC. RE-VERIFY: everything below is a write-time snapshot; re-check branch heads with git ls-remote, published POMs on kotlin.directory, and deployed JARs with the ContainerNursery CLI before acting.

Design decisions in force

• Owner, 2026-10-25 (verbatim): "You should test it, if migrating those to the new url resolver doesn't solve the problem, we need to understand the underlying issues and writeup a detailed handoff."
• Owner, 2026-10-23: bumping a consumer to an already-published UrlResolver/UrlProtocol release that fixes a bug is approved; no deep work inside UrlResolver/UrlProtocol in this effort.
• Production deploys only on the owner's explicit ask.

Why this exists

On 2026-10-22 AiCliSupervisorManager's conversation timeline rolled back because a readConversationHistory url:// reply ended after 1,310,678 of 1,333,487 bytes. The owner asked (verbatim, 2026-10-25): "You should test it, if migrating those to the new url resolver doesn't solve the problem, we need to understand the underlying issues and writeup a detailed handoff."

Result: moving to resolver 0.0.1337 / protocol 0.0.1434 does not fix it. As published, it causes the exact failure. 0.0.1337's POM pulls in jvm-libp2p:1.3.0-codexcoder21-snapshot-32, which has the bug. Keeping resolver 0.0.1337 / protocol 0.0.1434 and changing only jvm-libp2p to snapshot-37 makes every scenario pass on the first attempt (20/20).

All work was local. Nothing was sent to deployed services, and nothing was deployed, merged, enqueued, or changed in UrlResolver or UrlProtocol.

Versions involved

Mechanism (verified against source at the exact versions)

This is one deadlock involving the connection's read pause and Yamux flow-control credit:

1. The reply reader holds a parse lease for the whole body while it waits for bytes. In UrlProtocol 0.0.1434 (commit 2e9acd78b), Libp2pRpcProtocol.readJsonFrame reserves 7 × the frame length (acquireParseBudget(length, memoryBudget), src/foundation/url/protocol/Libp2pRpcProtocol.kt:4143) and only then calls readFully(input, bodyBytes) (:4158). The process-wide pool is maxMemory/16, with a minimum of 8 MiB (:172). At a 128 MiB heap (the test JVM measured maxMemory=134217728, parsePool=8388608) one 1,333,531-byte reply needs 9,334,717 permits. That takes the whole pool for as long as the body is still arriving.
2. Any other inbound request frame in that process pauses its substream. AttachmentRpcFrameDecoder.pauseAndAdmitJson (src/foundation/url/protocol/Libp2pHostFactory.kt:9911) calls ctx.setRpcServerAutoRead(false, admissionReadPauseOwner) (:9916, applied at :8562-8567) and waits on the url-json-admission executor for a lease. The 0.0.489 and 0.0.551 code does the same in pauseAndAdmit.
3. In jvm-libp2p snapshot-32, pausing one Yamux child stops reads for the whole TCP connection. In AbstractMuxHandler (jvm-libp2p commit 0f8adf04 = snapshot-32), childRead calls pauseChild (libp2p/src/main/kotlin/io/libp2p/etc/util/netty/mux/AbstractMuxHandler.kt:158), and pauseChild sets parentChannel.config().isAutoRead = false (:207-214). That stops reading Yamux WINDOW_UPDATE frames for every sibling stream. YamuxHandler only returns credit on delivery (onBytesDelivered, libp2p/src/main/kotlin/io/libp2p/mux/yamux/YamuxHandler.kt:166-172), and the window is 256 KiB (:33). This came from https://github.com/CodexCoder21Organization/jvm-libp2p/pull/40 (merged 2026-09-27; first in snapshot-28/30). Snapshots 25, 27 and 29 do not have it.
4. The reply sender then stops at an exact 256 KiB window boundary. Its credit never arrives, so the reader waits for bytes while still holding the parse lease, and the paused frame waits for that lease. Nothing makes progress.
5. After 10 s the sender retires the root, which cuts the reply short. AwaitedResponseOutbox.retireIfStillStalled (Libp2pHostFactory.kt:2068, grace AWAITED_RESPONSE_OUTBOX_STALLED_CONNECTION_GRACE_MS = 10,000 ms at :167) closes the root (closeProtocolRoot, :2109; the logs show closedBy=foundation.url.protocol.AwaitedResponseOutbox.closeFromPipeline). The reader then throws Stream closed while reading message data (read N of M bytes) (Libp2pRpcProtocol.kt:4283). In resolver 0.0.1337 this surfaces through PersistentRpcResponseReader.readResponse (UrlResolver.kt:30719) and createReaderThread (:31911).

The numbers match production. Every cut-short count is a multiple of 256 KiB minus 42 bytes (4-byte length prefix plus 38 bytes earlier on the stream): 524,246; 786,390; 1,048,534; and in run p1337-5, 1,310,678, the exact count production reported. That count is 5 × 262,144 − 42. Thread dump at 6.9 s (probe-2): persistent-rpc-reader is parked in NettyInputStream.read inside readJsonFrame, url-json-admission-1 is parked in SizeAwareReservationScheduler.acquire from AttachmentRpcFrameDecoder.pauseAndAdmitJson, and a small getStatus on a separate stream is sent at 3.8 s but its handler never runs. Retirement is logged at about 13.9 s: [RPC] Retiring root … after 10000ms without a completed response on child ….

The upstream fix already exists: https://github.com/CodexCoder21Organization/jvm-libp2p/pull/63, "Keep Yamux reads per stream while request delivery is paused" (merged 2026-10-22, source version snapshot-37). With it, a paused Yamux child holds only its own delivery and credit, and the parent keeps reading. Its own description records the same signature: "262,084 of 1,675,062 reply bytes received".

Tests and first-attempt results

All results are from scripts/test.bash --local --test <name> --log <xml>, read from the XML (attempt number="1"). The JVM heap is 128 MiB. Logs are under /tmp/claude-501/-code/b18cd447-2305-423b-ac30-50b4949d9abc/scratchpad/lanes/L122/logs/.

Changing only jvm-libp2p (snapshot-32 to snapshot-37) turns 0/5 into 10/10 in both scenarios, so the cause is the jvm-libp2p version.

Putting the daemon's 0.0.966 server and the manager's client in one JVM was not feasible. Both versions use the same foundation.url.* class names, so they cannot share one classpath without separate class loaders, and driving those would need reflection, which is not allowed. The 0.0.966 run above uses that version on both sides.

What the old-pin OUTBOX_FULL means

With nine 1.33 MB replies queued on one connection, 0.0.489/0.0.551/0.0.431 exceed the 4 MiB per-connection awaited-response outbox (Libp2pHostFactory.kt:345 at protocol 0.0.489 = commit c11773964). They return an explicit, documented OUTBOX_FULL. That is a bounded refusal, not a reply cut short, so it is not the production symptom. The manager does not appear to queue several large history reads on one connection (ManagerPollingSupervisorConnector opens one sandbox per operation), so that workload is probably not what production does. With snapshot-37 the same workload passes on 0.0.1337/0.0.1434.

Remaining unknown, and the read-only check that resolves it

The production cut-short count has the snapshot-28/30–36 signature. But the deployed manager (s25) and daemon main (s19) do not contain PR 40. Some hop on the 2026-10-22 path must have run an affected jvm-libp2p (snapshot-28 or 30–36). The likely candidates are:

• the deployed host-supervisor daemon (route url:vpn.2f4acf98240f.aiclisupervisor:, unversioned service-server-launcher.jar; its contents were never established);
• ContainerNursery or any relay that forwards that url:// stream (CN main is now on snapshot-38; it was on snapshot-32 in mid-October lane workspaces).

To find out: read-only, list the jvm-libp2p jar version inside the deployed daemon launcher jar and the deployed ContainerNursery jar (as of 2026-10-22), and confirm whether readConversationHistory goes directly or through a CN relay. Any hop on snapshot-28/30–36 explains the incident. If none does, the remaining candidate is the relay-stream credit deferral in RelayService ("Defers Yamux receive-window credit for one relay stream while its application reader is behind", RelayService.kt:2751 at 0.0.1434). The next experiment would then be the one-large-reply scenario through a real relay hop.

Ruled out

• UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1185 (one reply reader per persistent stream) as the fix: 0.0.1337 contains it and still cuts replies short. The thread dump shows a single reader per stream.
• Slow loopback or an undersized machine: nothing moves for about 10 s (3.6 s to 13.9 s), and a 3-byte request on a separate stream is never even dispatched. This is a deadlock, not slowness.
• The awaited-response outbox limit: the one-large-reply scenario keeps at most one reply queued and still fails on s32.
• ContainerNursery's URL facade on current code: https://github.com/CodexCoder21Organization/ContainerNursery/pull/739 delivers 1,333,300 bytes complete (earlier lane).

Branches

Next steps

1. Run the read-only check above to identify the affected hop on 2026-10-22. Record the libp2p snapshot of each deployed hop.
2. Any consumer moving to resolver 0.0.1337 (e.g. https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/64) must also pin jvm-libp2p:1.3.0-codexcoder21-snapshot-37 or later. On 0.0.1337's default snapshot-32, a single large reply alongside concurrent requests in the same process fails 5/5.
3. Publish a UrlResolver release whose POM selects snapshot-37+ (resolver 0.0.1338's POM still selects snapshot-32). That is a dependency bump, not a code change. Until then, consumers pin snapshot-37 explicitly.
4. Manager bump PR: AiCliSupervisorManagerServiceServer and -Wui to 0.0.1337/0.0.1434 + snapshot-37 (netty and other transitive pins as resolved), with testL122Resolver1337Libp2p37OneLargeReplyWithConcurrentSmallCalls and the concurrent test kept as regression tests. Update testGeneratedServiceServerPomUsesResolver1302 and testPublishedDependencyClasspathIsVersionCoherent as needed, run the full suite once under flock /tmp/claude-501/hd22-suite.lock, and do not merge without review. Main's s27 is not affected by this deadlock, so this bump is about getting the 0.0.1337 fixes safely, not about repairing main.
5. If the affected hop is the daemon or CN, bump that service to a snapshot-37+ stack, add the one-large-reply-with-concurrent-calls test there as a fail-first regression, and redeploy once the owner asks.
6. Separate design question for the UrlProtocol owners (not done here; no deep changes allowed): a reader holds a whole-pool parse lease while waiting on the network, and another stream's admission waits on that lease. With per-stream pause (s37) that wait is local to one stream. With any connection-wide pause it is a deadlock.