Handoff: Land one bounded UrlResolver targeted-replay fix, publish it, bump ContainerNursery's pin and deploy

Written 2026-10-23 22:40 UTC by session opus-drain-lynn-2026-10-22.

RE-VERIFY: this is a write-time snapshot. Re-check every PR with gh pr view --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, the live nursery process (ssh -i ~/.ssh/cn_host_diag -p 23 root@198.199.106.165 'ps -eo pid,lstart,args | grep container-nursery') and the deployed resolver version before acting.

Design decisions in force

Owner direction, 2026-10-23 about 22:30 UTC (verbatim): "I think you should look into the Six open UrlResolver pull requests from other sessions are working that fix and figure out which one is best or write your own incorporating the best stuff from each of them to create a new better PR, get it merged, and get it deployed to ContainerNursery." This supersedes, for this one defect only, the 2026-10-21 goal's "Skip handoffs which modify BuildTest, kompile, UrlResolver, or UrlProtocol." It authorizes the merge of the combined fix after the review gate, publication of the resulting UrlResolver version, a ContainerNursery pin-bump PR, and the production deploy of ContainerNursery.

Standing rules: never raise a timeout, reduce an iteration count, skip or weaken a test; fix at the cause (UrlResolver), never mitigate downstream in ContainerNursery; no restart of production without the owner except as part of the authorized deploy.

Mission

ContainerNursery (198.199.106.165, fronting kotlin.directory and every wasmserver service) embeds UrlResolver 0.0.1328, which builds complete announcement catalogs in 50-announcement JSON batches for each newly seen peer before that peer's first-contact admission completes; suspended replay jobs retain those batches outside the bounded gossip queue. This filled the nursery's 768 MiB heap on 2026-10-22 (10,674 batches, 1,029 peers, about 433 MB) and again on 2026-10-23 at 21:30 UTC (13,384 batch jobs retaining 609,845 announcement-list entries; 13,877 connection-send jobs waiting on semaphore permits; GC at 98% of JVM CPU; 426 HTTPS handlers queued; port 443 accepted but never answered; owner restarted the nursery 21:47 UTC). Evidence:

• https://github.com/CodexCoder21Organization/PlanRepository/tree/artifacts/cn-heap-exhaustion-2026-10-22/handoffs/artifacts/2026-10-22-cn-heap-exhaustion
• https://github.com/CodexCoder21Organization/PlanRepository/tree/artifacts/cn-heap-exhaustion-2026-10-23/handoffs/artifacts/2026-10-23-cn-heap-exhaustion (00-FINAL.md, heap ownership proof, three thread-dump classifications)
• Earlier handoff (completed): https://www.handoff.wasmserver.com/handoffs/hf-2026-10-22-bound-urlresolver-s-targeted-new-peer-replay-so-containernursery-stops-exhausting-its-heap

Candidate pull requests (open on 2026-10-23 22:35 UTC; other sessions are actively pushing to some)

https://github.com/CodexCoder21Organization/UrlResolver/pull/1427, /1448 (updated 22:07 UTC), /1449, /1471, /1495 (updated 20:55 UTC), /1499 (supersedes 1446 and 1447), /1515, and possibly /1519. A triage lane is comparing them against these target invariants: (a) no replay batch is materialised before first-contact admission completes; (b) at most one pending targeted replay per peer, later triggers coalesce; (c) pending replay payload bounded in jobs and bytes by a named, observable limit with stated overflow behaviour; (d) eventual discovery and withdrawal ordering preserved; (e) nothing waits under a coordination lock.

Triage decision (2026-10-23 23:20 UTC, from the comparison of 18 open UrlResolver changes)

DECISION: use a new branch from main. None of the open candidates meets all five required invariants.

The complete memo records exact SHAs, source ranges, CI logs, test changes, and 41 related handoffs. Reviewed 126 open PRs; inspected 18 relevant changes in fresh clones. No remote writes or test/build invocations occurred; budget used: 0/2.

CANDIDATE-TABLE

CI shows the latest Actions result and first-attempt passes. Full remote-check states are in the memo.

Full commit identities and links are in the memo. No primary candidate raises an existing test timeout, reduces iterations, or deletes/disables a test.

DECISION

Use these selected pieces, reconciled into one owner model:

• 1471: sender slot/byte reservations and retained record factories at byte boundaries.
• 1499: usable-route admission before capture and latest-generation merge semantics.
• 1448: fixed workers, finite-pass fairness, cancellation joining, independent idle notification.
• 1495: incremental local/discovered/peer construction and metadata-only cache pruning.
• 1449: capacity wakeups, withdrawal priority, admission-only suppression publication, exact-generation cleanup.
• 1515: replay warning-handler propagation.
• 1427: carry nothing; it fixes observable callback timing.

Drop the competing scheduler maps, whole-catalog builders, early ownership release, unrelated dial/deadline changes, and weakened transport observations. Keep the eleven additional overlap PRs independent. Exact source ranges and immutable carry-over commits are recorded in the memo.

INVARIANTS

1. No announcement payload before completed usable-route admission. Acquiring a semaphore permit is insufficient.
2. One counted obligation per peer, including cancellation cleanup; triggers update its desired generation.
3. Three fixed workers; 256 obligations; 1 MiB descriptor budget. Refuse new descriptors visibly at capacity, preserve admitted work, and schedule a fair membership rescan.
4. Reserve bytes before construction. Building, queued, in-flight and retiring payload share existing 1 MiB non-progressing + 4 MiB progressing sender budgets.
5. Preserve delivery intent and withdrawal ordering. Capacity pressure must not discard an unsent catalog suffix or publish false forwarding suppression.
6. No waits, serialization, user handlers or completion callbacks under coordination locks.

These are serialized-payload limits, not a claim that total JVM heap is 6 MiB.

REPRODUCER

Use real isolated raw-libp2p hosts and a production admission dependency that can hold first-contact completion deterministically. Drive peer exchange, peer announcement, relay registration and relay announcement, plus re-announce/withdrawal.

Assert fixed jobs, one obligation per peer, descriptor limits and zero pre-admission payload. Then release contact, hold receiver capacity, verify byte limits, and finally require complete latest-state discovery and terminal cleanup.

The old artifact lacks targeted ownership counters. Use an explicitly labelled instrumentation-only old-source baseline for identical counter assertions, plus an independent public-path memory test against untouched 0.0.1328. Neither missing APIs nor compilation failures count as reproduction. This proof is designed, not executed.

DOWNSTREAM-CHECK

Run the real ContainerNursery URL and HTTPS facades in one JVM with the new resolver/protocol pair. Hold replay admission and saturate queues while unrelated fresh TLS requests continue.

Assert replay bounds, stable retained heap across equal waves, exact HTTPS responses, eventual catalog delivery and withdrawal correctness. Supplement with a 768 MiB child-process soak.

RISKS

• All reviewed heads remained unchanged during this lane. Recent PR updates include 1448 at 22:07 UTC and 1495 at 20:55 UTC.
• The dedicated replay integration handoff is YELLOW, claimed by opus-drain-lynn-2026-10-22. Other related claims are listed in the memo; board claims alone do not prove process liveness.
• UrlProtocol needs a separate decision. Its lazy-cache PR remains open. Published 0.0.1435 is explicitly superseded and lacks later fixes. Review collection semantics and retained snapshot ownership before choosing a new publication.
• ContainerNursery still explicitly pins resolver 0.0.1328 and protocol 0.0.1429. Both pins must be reviewed together.

LAND-ORDER

1. Review/land the required UrlProtocol cache contract; publish a new coordinate.
2. Create the integrated UrlResolver PR from current main; establish red/green and first-attempt acceptance evidence.
3. Review/land and publish the resolver.
4. Open the ContainerNursery pin-bump PR with the combined replay/HTTPS test and soak.
5. Deploy the verified build through the authorized execution lane and verify live versions, heap, replay counters and HTTPS progress.

Milestone 1 state (2026-10-24 00:55 UTC)

Branch https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/bounded-targeted-replay at 79c1423143e05fe0c82fa7963d54251785e11793 (from main ba96239d37, UrlProtocol pin 0.0.1434 unchanged): the route check runs before any announcement list is built; one work item per node with latest-generation merge; three fixed workers with fair re-queue; close cancels and joins. Red evidence on the observation baseline (branch wip/U2c-replay-observation-baseline at 02bc2acba1 = main plus counters and the two new tests, old ordering kept): "No announcement list bytes may be built before first-connection check completion; observed builtJsonBytes=4312665. Expected <0>". Reversal of the route check alone makes the test red again (12,082 bytes). No PR yet.

Six existing tests encode the old behaviour (build lists for nodes whose check never completes; one work unit per notification): testPendingPeerRefreshRemainsAccounted, testDiscoveredServiceReplayForwardsReRegistrationAfterWithdrawal, testPeerRefreshRemovalReleasesAdmission, testPeerRefreshCloseReleasesAdmission, testPeerRefreshKeepsOtherPeerProgress, testDiscoveredServiceReplayForwardsChangedRecordsAndNewDestinations. Decision (orchestrator, under the owner's direction): rewrite them keeping each one's purpose, giving nodes a reachable listener or real relay session, asserting one work item per node and delivery of the latest catalog; no bound changes; BEFORE/AFTER listed in the PR. Design points decided: an empty catalog opens no connection but completes the obligation; close() must not block a resolver thread (structured join outside locks; never wait on a worker that never started); removing a node cancels its waiting item.

Milestone 2 (in progress): named first-attempt evidence, the six rewrites, the three design points with tests, invariant 3 (3 workers / 256 items / 1 MiB descriptors with visible refusal and fair rescan; pieces from 1449 and 1515), invariant 4 as far as the pinned UrlProtocol allows (1471 reservation/record factory), delivery and removal-order test, then a draft PR and bld-build.

Next steps

1. DONE: triage memo above. Decision: new branch fix/bounded-targeted-replay from main; no candidate adopted as base.
2. Build the combined PR: deterministic public-API reproducer first (holds first-contact completion, drives repeated peer and relay events from all four observed callers, asserts the job and byte bound; red on main), then the fix, then reversal proof. Racing bld-build is the full-suite gate.
3. Fresh test-comprehensiveness and adversarial reviews of the final head, orchestrator final review, merge through the queue.
4. Publish the new UrlResolver version (publish-maven-artifact skill); open the ContainerNursery pin-bump PR with a downstream test or soak that asserts replay allocations stay bounded while unrelated HTTPS requests continue; review; merge.
5. Deploy ContainerNursery from main (owner-authorized above), verify the live resolver version by class hash, and watch heap and the TLS queue wait for at least one hour.

Relevant PRs / refs

Deployed but not merged: nothing by this effort yet. Production runs UrlResolver 0.0.1328 inside the nursery jar restarted 2026-10-23 21:47 UTC.