Bound UrlResolver's targeted new-peer replay so ContainerNursery stops exhausting its heap

RE-VERIFY: everything below is a snapshot written 2026-10-22 13:45 UTC. Re-check the live ContainerNursery process (ssh -i ~/.ssh/cn_host_diag -p 23 root@198.199.106.165 'ps -eo pid,lstart,args | grep container-nursery') and the preserved dumps under /root/ContainerNursery/heapdumps/preserved/ before acting.

Design decisions in force

(none stated by the owner for this work yet)

Mission

ContainerNursery on 198.199.106.165 fronts kotlin.directory and every wasmserver service. On 2026-10-22 its process (pid 4045645, -Xmx768m, started 02:27 UTC) threw OutOfMemoryError: Java heap space at 12:15:46 and stayed up in a broken state for 48m45s. During that time TLS was rejected and queues were long while listeners stayed bound. A new process was started at 13:04:31; the actor is unrecorded. That was the third ContainerNursery process that day (01:06, 02:27, 13:04). Fix the retention at its owning layer, UrlResolver, and close the ContainerNursery supervision gaps.

What was found (evidence branch: https://github.com/CodexCoder21Organization/PlanRepository/tree/artifacts/cn-heap-exhaustion-2026-10-22/handoffs/artifacts/2026-10-22-cn-heap-exhaustion at 5bc0f7fe6acc84d32d855f2f9df5917074413755)

• Mechanism (named from a 1.17 GB heap dump, Eclipse MAT dominator and leak-suspect reports). UrlResolver builds complete announcement catalogs, in 50-announcement JSON batches, for a newly seen peer before first-contact admission. Suspended replay jobs waiting on UrlProtocol2's first-contact semaphore then hold those batches outside UrlResolver's bounded per-peer gossip queue. The dump holds 10,674 distinct batches targeting 1,029 peers; one peer has 520 retained replays. The traffic that starts them is the received-peers path (onPeersReceived).
• Top retainers (677,152,328 reachable bytes):

• Deployed versions, confirmed by class hashes: UrlResolver 0.0.1328, UrlProtocol 0.0.1429. ContainerNursery jar container-nursery-d8d1f4ef-20261022T021335Z.jar.
• Not a single large request. The allocating relay request retained only 7.5 MB; the dominant condition is accumulated replay work.
• ContainerNursery supervision gaps, owned by ContainerNursery and its deploy scripts: the live command line has -XX:+HeapDumpOnOutOfMemoryError but no exit-on-OOM flag (child containers do have it), the systemd unit has Restart=no, and the installed watchdog was not scheduled. The repository's watchdog gained exit-on-OOM earlier, but the live copy does not auto-sync.
• Correlation, unproven. A 04:50-05:05 UTC kotlin.directory stall (28-39 s per POM, 34-46 s TLS queue wait) and a 09:31 UTC Filedrop storage-discovery failure both happened in this process generation during high-heap periods. Causation is unknown because early GC records were not retained.
• Dump-to-process map. ContainerNursery: 4045645. kotlin-build-ci children: 2964410, 3108965, 3533956, 2712413. Handoff children: 710452, 1866317, 3701132, 3706765, 3715310.

Relevant PRs / refs

• No code changes yet; nothing deployed or published by this investigation. Production was left unchanged and copied heap dumps were deleted from the analysis machine.
• Branch table:

Next steps

1. In UrlResolver: write a deterministic reproducer through public paths. Hold first-contact completion, drive repeated peer and relay events (all four observed callers), and assert a bound on pending targeted-replay jobs and their payload bytes. It must fail on 0.0.1328.
2. Fix at the cause: build replay batches lazily at send time, after admission, and coalesce or replace pending replays per peer instead of stacking them. Cover eventual discovery, withdrawal ordering and close. Publish, then bump UrlProtocol/UrlResolver in ContainerNursery and the consumers that share the pairing.
3. In ContainerNursery's launcher and deploy scripts: add exit-on-OOM to the parent JVM, make the unit restart on failure, schedule the watchdog, and record the restart actor in the audit log. Bring the live host in line with the repository.
4. Add the missing observability: targeted-replay counts, bytes and ages; timestamped full-GC records retained across restarts; generation-tagged metrics.

Reusable knowledge

• Preserved dumps live under /root/ContainerNursery/heapdumps/preserved/gen-*/ with RESTART_RECORD.txt (issue 502 preservation). Analyze copies off-host; MAT batch mode needs about 3 GB for the 1.17 GB dump.