← Priority list
Unclaimed

Bound UrlResolver replay catalog retention that exhausted the ContainerNursery heap (2026-10-22 outage)

Remaining: fix the UrlResolver defect where targeted replay jobs retain eagerly built JSON catalogs outside the bounded gossip queue. On 2026-10-22 it exhausted ContainerNursery's 768 MB heap and took down all HTTPS on 198.199.106.165 from ~12:19 to before 14:34 UTC. Write a failing-first test, bound the work before the catalog is built, publish, bump ContainerNursery and deploy. Evidence is on PlanRepository branch wip/hd22-cn-heap-outage-20261022.

Handoff document

Markdown

Restore production and fix ContainerNursery heap exhaustion (2026-10-22 outage)

What happened (OBSERVED)

  • From about 12:19 UTC, HTTPS on 198.199.106.165 (kotlin.directory, api.nursery.wasmserver.com, www.handoff.wasmserver.com, tinysite.wasmserver.com, ...) failed: TCP connects, but the TLS handshake fails (SSL_ERROR_SYSCALL) or the request times out after 15 s. It was still failing at 12:38. url://handoff/ failed from about 12:05, and kotlin.directory downloads stalled from about 11:55.
  • The ContainerNursery JVM (PID 4045645, jar container-nursery-d8d1f4ef-20261022T021335Z, -Xmx768m, running since about 02:13 UTC) has an exhausted heap and is in back-to-back full GC: 25 full collections across 30 one-second jstat samples, CPU around 212%, RSS about 1.47 GB. TLS handler threads stalled and filled all 4,096 slots; new sockets were closed before TLS, and accepted sockets waited up to 314 s. File-descriptor exhaustion was ruled out. The host has about 16 GB RAM, with about 4.4 GB available.
  • Strongly supported underlying defect: thousands of targeted UrlResolver replay jobs inside ContainerNursery each keep an eagerly built JSON catalog, outside the bounded gossip queue. A class histogram and binary metadata match this; the exact retaining paths and the originating peer catalog are not yet verified.

Evidence

Thread dumps, class histogram, jstat, journal and kernel snapshots were captured read-only. They are in the investigating session's notes (OUT2 findings and lane directory). Not yet pushed to a remote branch.

Remaining

  1. (Done externally by 14:34 UTC) Restore service.
  2. Root-cause fix in UrlResolver: bound replay work BEFORE the JSON catalog is built (make the work lazy and bounded) so retained catalogs cannot grow without limit. Write a failing-first test, publish, bump ContainerNursery, then deploy.
  3. Separately, review whether ContainerNursery's -Xmx768m is intended (the standing rule says 128 MB; an earlier session left 768 MB).

Update 14:34 UTC

Production HTTPS recovered by 14:34 UTC (restored by someone outside this session). The underlying UrlResolver replay-catalog retention defect remains unfixed. Full evidence: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/hd22-cn-heap-outage-20261022

Model usage

Recorded agent work, including concurrent agents

No model usage has been recorded for this handoff yet.

Record work with handoff-cli usage report hf-2026-10-22-bound-urlresolver-replay-catalog-retention-that-exhausted-the-containernursery-heap-2026-10-22-outage.

No status reports yet.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.