Handoff: Stop ContainerNursery's network threads stalling in PeerRegistry ranking so TinySite publishes complete

Written 2026-10-20 05:15 UTC. RE-VERIFY: every PR state, version and host observation below is a write-time snapshot. Re-check PRs with gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt, the live ContainerNursery build with the ContainerNursery CLI (/home/u/bin/cs launch containernurserycli:container-nursery-cli:0.0.20 -r https://kotlin.directory -- --url https://api.nursery.wasmserver.com routes) and the host coordination log /root/cn-watchdog.audit on 198.199.106.165 before acting.

Filed via the legacy git flow because url://handoff/ could not be reached (see Reusable knowledge); the transitional sync should import it. Related handoffs it depends on: hf-2026-10-19-finish-peerregistry-eviction-validation-and-open-its-pr, hf-2026-10-19-finish-urlprotocol-s-bounded-event-loop-handoff-and-prove-every-callback-path.

Summary: Remaining: agree the fix line with the overlapping UrlProtocol handoffs, land UrlProtocol PRs 724/725/694 and UrlResolver PR 1318 (all open; kotlin.build (remote) red on 724, 694, 1318; 694 has conflicts), publish protocol/resolver, bump and (with owner approval) redeploy ContainerNursery and the storage service, then prove a 91-file TinySite publish succeeds. Cause, reproduced 2026-10-20 03:07 UTC with thread dumps: on ContainerNursery (protocol 0.0.1420) each peer announcement over the 1,000-peer registry cap runs an O(n^2) re-ranking under an exclusive lock on a network thread, parking the other network threads, so filedrop's ~30 sequential storage calls per commit exceed the CLI's 30 s deadline; departed peers are never pruned, so the registry stays over the cap. Nothing deployed.

Design decisions in force

• 2026-10-20, user, on the TinySite failures: "retry the tinysite upload. If it fails, you should investigate the underlying root cause of the failure." The retry failed; the investigation below is the result.
• 2026-10-20, user: "Create a handoff for this ContainerNursery issue" (this document).
• Production deploys (ContainerNursery, filedrop, storage) wait for the owner to ask for them.

Mission summary

TinySite publishes of larger sites fail nondeterministically with SJVM virtual method invocation timed out after 30s: filedrop/server/FiledropClientImpl.commitFileUpload (or uploadFileChunk), and occasionally with All 1 peers for service 'filedrop' failed bytecode fetch ... /ip4/198.199.106.165/tcp/35000: TotalTimeoutException. The cause is ContainerNursery's libp2p network threads stalling inside UrlProtocol's PeerRegistry re-ranking, which delays every url:// call relayed through ContainerNursery on :35000. The remaining work is to land the fix in UrlProtocol/UrlResolver, publish, rebuild and redeploy ContainerNursery (owner approval), and prove a large TinySite publish succeeds.

This was found while publishing a Good-UI-Designs capture (GQ x Audemars Piguet, branch add-dark-3d-watch-lab-gq, 91 files / 25.8 MB) for the owner's review; that publish is the end-to-end proof case.

What was found and done (the chain)

Failures observed (all tinysite-cli 0.0.1 publish <dir> --title dark-3d-watch-lab --expires 604800 from the /code box)

1. 2026-10-19 ~08:29 UTC, 149 files / 87.8 MB: commitFileUpload timed out on _payload.json (5 KB).
2. 2026-10-19 ~08:43 UTC, same site, after 728 s: uploadFileChunk timed out on a 60 KB mp3.
3. 2026-10-19 ~15:00-16:10 UTC, 91 files / 25.8 MB: one attempt left no site (error text lost); the next failed with the filedrop bytecode-fetch TotalTimeoutException against :35000 before uploading.
4. 2026-10-20 02:33-02:44 UTC, same 91 files: commitFileUpload timed out on webgl/ground/groundho.png. Earlier on 10-19 a 115-file 41 MB publish succeeded but took ~12 minutes; small publishes succeed.

Mechanism (investigated read-only 02:45-03:15 UTC 10-20; high confidence for failure 4 and the probe, inferred for 1-3)

• The CLI downloads FiledropClientImpl and runs it locally in the SJVM sandbox; each call is one remote call (FiledropServiceServer/src-client/filedrop/server/FiledropClientImpl.kt:123-130) with a 30 s whole-round-trip deadline (DEFAULT_TIMEOUT_SECONDS = 30L, UrlResolver 0.0.1321 SandboxedProxyGenerator.kt:1523, enforced at :1765). Path: CLI -> ContainerNursery :35000 -> filedrop-server -> reply.
• A commit (commitFileUploadExclusively, FiledropEmbedded 0.1.27) makes ~25-35 sequential calls to url://simple-filesystem-vnext/, each also relayed through ContainerNursery (filedrop -> :35000 -> filesystem service on :37141).
• ContainerNursery has 16 network threads. When the peer registry is over MAX_REGISTRY_PEERS = 1000, every incoming peer announcement runs PeerRegistry.publishCompletedAdd -> bootstrapEvictionVictims -> bootstrapRanking (O(n^2) per the code's own comment, UrlProtocol 0.0.1420, the version in the live ContainerNursery build) on a network thread while holding an exclusive lock, called synchronously from Libp2pHostFactory.kt:7235/7362. The other network threads park on the registry locks and read nothing from their connections.
• The registry stays over capacity because departed peers are not pruned: filedrop's log shows dead droplets with "855 failed attempts" at reachability checks, and its sockets show SYN-SENT to dead 137.184.x hosts. Actual registry size was not measured.

Evidence

• Probe reproduction: a 40-file publish of 28-byte files failed on f30.txt at 03:07:40 UTC with the identical commitFileUpload ... timed out after 30s, while dumps of all three JVMs were being taken.
• filedrop dump: the commit thread waits on the filesystem-service reply throughout (03:07:09-03:07:24 in publishListingRequest -> writeStreamedBytes:3746/3752).
• Filesystem service: received nothing 03:07:09.169 -> 03:07:17.648 -> 03:07:26.408 -> 03:07:35.165 (three ~8.7 s gaps), all request threads idle.
• ContainerNursery dumps at 03:07:14-03:07:35: 13-16 of 16 network threads parked on registry locks (read lock at line 1655, another at 1919), one thread computing the ranking, entered from RpcServerMessageHandler.channelRead0. Across 70 dumps over ~3.5 min, every dump had at least one network thread in PeerRegistry; average 8.5/16, >=12/16 in 23 dumps, 16/16 in 5.
• The reply was late, not lost: the filesystem service kept receiving this commit's writes at 03:07:35-37, after the CLI deadline, then the drop deletion.
• Failure 4 shows the same total-silence pattern in the filesystem-service log (12.85 s with no requests, 02:44:16 -> 02:44:29); no ContainerNursery dumps exist for it.

Hypotheses ruled out

• Commit cost growing with the number of files in the site: refuted. Splitting the filesystem-service log into 19 publish runs shows no growth in commit-to-next-file time (the 160-commit run at 12:26 had its slowest commits first); the probe failed on a 28-byte file at index 30.
• filedrop GC: refuted (6.3 s total GC in 428 s, 3 full GCs of ~0.5 s; heap 128 MB, 94% used).
• filedrop restarts or kernel OOM kills during failures 1, 2, 4: none (container_events.db, dmesg).
• Filesystem-service overload: no (a request was active in 25 of 101 samples; after each move the next request followed in a median 0.10 s, p90 1.7 s, over 579 moves).

Failure 3 (filedrop unreachable ~16:00 UTC 10-19)

filedrop was running (started 15:14:15, stopped 17:19:35). The TotalTimeoutException came from opening a connection to ContainerNursery on :35000; a probe at 02:59 UTC 10-20 failed the same way. Consistent with the frozen network threads, but inferred (no ContainerNursery dumps during a connection failure). ContainerNursery did stop filedrop for memory pressure at 15:13:52 (PROACTIVE_MEMORY_PRESSURE), which may explain the attempt that left no site.

Secondary contributor

In the filesystem service every atomicMove re-walks the whole filesystem (calculateUsedBytes, OkioSimpleFileSystem.kt:573) under the tree write lock. Fixed on main by https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/39 (merged 2026-10-19 20:39 UTC) but, per the investigation, not published (still 0.0.32) and not deployed.

Host context (not shown to be part of the mechanism)

Load average 55-78 on 8 cores, CPU pressure ~91%. Largest consumer buildtest-server (pid 685277: ~2.5 cores average, ~1.2 TB read / 1.15 TB written in 22 h); this slows the ranking but its own cause needs separate investigation. Another session had a buildtest-server deploy window 02:41-04:39Z. The TinySite thumbnails release (route updated 22:58Z 10-19) raised filedrop's storage reads 4-8x, adding traffic through ContainerNursery.

Relevant PRs / refs

Fix PRs (snapshot 2026-10-20 05:13 UTC; none merged)

kotlin.build (remote) must be green before any of these merges; the red runs have not been investigated in this effort.

Related handoffs (overlapping work; this handoff depends on them)

• url://handoff/handoffs/hf-2026-10-19-finish-peerregistry-eviction-validation-and-open-its-pr (RED: pending decisions on reviewed test proposals; source at 11eded65c, no PR yet)
• url://handoff/handoffs/hf-2026-10-19-finish-urlprotocol-s-bounded-event-loop-handoff-and-prove-every-callback-path (ORANGE: callback inventory and one UrlProtocol PR outstanding)
• url://handoff/handoffs/hf-2026-10-17-land-the-tinysite-speed-and-parallel-upload-fixes-finish-the-ci-capacity-fixes-and-redeploy-tinysite-filedrop-and-storage (ORANGE: protocol publication and filedrop/storage redeploys) Before starting, decide with their owners which of these lines (PRs 724/725/694 vs. the eviction-validation and bounded-event-loop work) is the one to land, so the same fix is not driven twice.

Branches pushed by this effort

Source clones used for reading (ContainerNursery, UrlProtocol, UrlResolver, Filedrop*, SimpleFileSystemService*, TinySiteCli, sandboxjvm) carried no local commits; the ContainerNursery clone showed 62 deleted test files as a checkout artifact, not work. Raw uncompressed dumps and a 20 MB pickled log index were left out (the gzipped dumps and logs carry the same data).

Deployed or published by this effort

Nothing. No restart, deploy, route change, upload or publication was made; all host access was read-only (SSH reads, thread dumps). Verified by the investigation's own command log; the test sites it created were removed (tinysite list shows no invprobe site).

Next steps

1. Re-verify the PR table and the related handoffs; agree which fix line lands.
2. Build a deterministic reproducer in UrlProtocol if the chosen PR lacks one: a real host with the registry filled above 1,000 peers with varied addresses; hold the registry publication step on a latch (blocking scorer); send an announcement on one connection, then an unrelated echo call on another connection sharing the same network thread; assert the echo completes and no network thread waits on a PeerRegistry lock. It must fail on 0.0.1420 without depending on host speed.
3. Land the fix PRs with kotlin.build (remote) green and reviews done; land UrlResolver pruning (https://github.com/CodexCoder21Organization/UrlResolver/pull/1318) so the registry is not permanently over capacity.
4. Publish the new foundation.url:protocol (latest published at write time: 0.0.1425; check whether 0.0.1421-0.0.1425 already contain any of this) and UrlResolver; bump ContainerNursery; publish SimpleFileSystemServiceEmbedded with PR 39 and bump the storage service.
5. With the owner's go-ahead, redeploy ContainerNursery (read /root/cn-watchdog.audit first; another session operates on this host) and the storage service.
6. Prove it: take ~20 ContainerNursery thread dumps under normal traffic and confirm no network thread is inside PeerRegistry; publish the 91-file capture from the Good-UI-Designs branch above and the 40-file probe (from the evidence branch) with tinysite-cli, both succeeding; give the owner the TinySite review link.

Reusable / operational knowledge

• tinysite-cli: /home/u/bin/cs launch tinysite.cli:tinysite-cli:0.0.1 -r https://kotlin.directory -M tinysite.cli.MainKt -- publish|list|delete|health. Publish is all-or-nothing and removes a partial site on failure. Its final Error: line is buried in hundreds of libp2p log lines on stderr; redirect stdout/stderr to separate files and grep ^Error.
• CLI JVM start-up costs ~30 s on the /code box (1 CPU); measure server-side timings from server logs.
• Fabric read-file refuses files over 10 MB (the audit log is 10.5 MB) and launch tail timed out after 60 s; read-only SSH (ssh -i ~/.ssh/id_ed25519_20260706_035316 -p 23 root@198.199.106.165) was used as the fallback.
• Container logs: /root/ContainerNursery/data/container-logs; container lifecycle events: ContainerNursery container_events.db.
• At write time handoff-cli list failed once with ConnectionClosedException on :35000 (succeeded on retry), and handoff-cli create failed twice with All 1 peers for service 'handoff' failed bytecode fetch against :35000, so this handoff was filed through the legacy git flow in PlanRepository/handoffs/ instead of url://handoff/. Any url:// service relayed through ContainerNursery shows this defect, not only filedrop.
• Challenge records: PlanRepository challenges/2026-10-19-0829-..., 2026-10-19-0843-..., 2026-10-19-1622-tinysite-publish-of-a-26-mb-capture-failed-before-upload.md, https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-20-0316-root-cause-of-tinysite-publish-timeouts-containernursery.md

Open questions

• Registry size on the live ContainerNursery (only known to exceed 1,000).
• ContainerNursery state during failures 1-3 (no dumps then).
• Whether the PROACTIVE_MEMORY_PRESSURE stop of filedrop at 15:13:52 caused the failure-3 attempt that left no site.