Land the two remaining UrlProtocol network-thread PRs and the two fixture PRs, then hand the ContainerNursery bump and deploy to the owner session

Written 2026-10-22 03:55 UTC by the session fable-code-peerregistry-20261019 (Claude Code on the /code box). RE-VERIFY: everything below is a write-time snapshot. Before acting, run gh pr view <url> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt on every PR, git fetch origin && git ls-remote --heads origin for every branch, and read the per-attempt test history artifact of each required bld-all-tests run rather than its green badge. Five delegate lanes were still running when this was written (see "Lanes in flight"); their branches and findings files move after this snapshot.

Design decisions in force

All quotes are the owner's words. Nothing here is a tactical choice; each is a project requirement.

1. 2026-10-19, the original request: "You said 'That list is full (1,000 entries), and each new entry triggers a slow re-sort while a lock is held.' - this sounds like a disaster and sounds suspiciously like something that a code review https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/CODE_REVIEW.md#further-lock-and-deadlock-considerations would complain about. Seems that we should fix it, delegate the work to gpt 6.1-sol, drive it to completion including code review and merge and deploy."
2. 2026-10-19, design approval: "Sure, 1+2+3 in parallel sol delegates probably #3 is most important" (1 = single owner thread for registry changes, 2 = immutable snapshots for readers, 3 = incremental eviction instead of a per-add re-rank), and "obviously the tests need to be updated". Existing tests that asserted the old blocking contract may be replaced with same-strength assertions of the new contract; each replacement is listed in its PR.
3. 2026-10-20 ~21:50, snapshots: "Freeze membership only": snapshots freeze membership and order but return the caller's original peer objects; no identity-assertion or README contract change. Ten legacy getter-gate tests may be rewritten to "unrelated peers are never read" assertions of the same strength.
4. 2026-10-20, fixture approvals: B3's two legacy fixture setup changes (a real disconnect promise and an explicit pending release, assertions unchanged); B5's parse-budget contract (release the parse budget before application code runs, keep the callback payload charged against the callback bounds, three parse-lease tests updated to match); the eight legacy eviction-test migrations for the eviction PR; the correction of two host-dependent address checks in UrlResolver tests.
5. 2026-10-20, merge gate: the gate is the branch ruleset's required checks (bld-build and bld-all-tests for UrlProtocol, bld-build for UrlResolver). kotlin.build (remote) is informational on these repos. A green check that needed a retry is not a pass: audit the root-level attempt history for first-attempt failures and compare each with main.
6. Standing rules restated by the owner during this effort: never raise a timeout, lower an iteration count, weaken or skip an assertion without explicit approval; CPU load or machine contention is never accepted as the cause of a flake (2026-09-01 directive, reaffirmed); every flake met during other work is root-caused and fixed, never re-run past; questions go to the owner in status reports while all other work continues; never bypass CI; never --delete-branch; never put the TinySite review link in PRs, commits or files; never commit lane evidence to a PR branch; the production deploy and the ContainerNursery bump are done only by the owner-authorised deploy session.
7. 2026-10-22 03:14, delegation of the open questions: "use your best judgement regarding the open questions." Rulings taken under it: (a) part 1 stops after https://github.com/CodexCoder21Organization/UrlProtocol/pull/755, no further single-owner slices; (b) testTcpCloseWhileConcurrentSilenceSchedulesArePending may count only the timer thread's schedules and must join the reader thread before its final check (full repair in the FL4 proposal review linked below); (c) testForwardedOutputRefusalPreservesAtomicBoundary may force its ordering and must require the cancelled tail write to fail with SocketException (patch linked below). (d) 2026-10-22 03:30: a dependency-pin-only edit to a test file by another PR is not an overlapping change; the fixture PR proceeds.
8. 2026-10-22 03:36: "do not launch any more tasks, let these wind down naturally and then save their status to the handoff." No new lanes after this point. The five running lanes finish on their own; their end state is recorded here when they do.

Mission summary

On 2026-10-19 TinySite uploads failed all day with 30-second timeouts. ContainerNursery (CN), which relays every url:// call on 198.199.106.165, had its Netty event-loop threads parked in PeerRegistry.addPeer behind bootstrapRanking, an O(n²) diversity ranking that runs under the registry lock on every add once the registry holds 1,000 peers. A blocked event loop freezes every connection pinned to it, so the storage server saw 40-second gaps and every WUI request hit its 30-second limit. The owner asked for the three-part redesign above, delegated to codex gpt-6.1-sol lanes, driven through review, merge and deploy.

State of the three parts:

• Part 3, incremental eviction: merged as https://github.com/CodexCoder21Organization/UrlProtocol/pull/739 (2026-10-21 23:55, landed by the sibling session fable-cn-peerregistry-20261020).
• Part 2, immutable snapshots: merged as https://github.com/CodexCoder21Organization/UrlProtocol/pull/750 (2026-10-21 14:28).
• Part 1, single owner: the full 287-test-file single-owner branch was abandoned as too big. Lane A1c then traced main and found that since https://github.com/CodexCoder21Organization/UrlProtocol/pull/725 merged, announcements reach addPeer on url-one-shot-gossip-* workers, not Netty threads. The one proven remaining network-thread path was authenticated-interaction evidence recording (PeerObservationLog upserts under scoreShapeLock on the Netty event loop, Libp2pHostFactory.kt around line 8041 at the time). That is fixed by https://github.com/CodexCoder21Organization/UrlProtocol/pull/755, the last part-1 slice (decision 7a). Guard tests that no network thread enters the registry landed as https://github.com/CodexCoder21Organization/UrlProtocol/pull/781.
• Companion work: https://github.com/CodexCoder21Organization/UrlProtocol/pull/742 moves the remaining network callbacks and PeerExchange work onto bounded worker pools (lane B). UrlResolver 1341, 1342, 1345 and 1346 fixed the resolver-side address tests and two main flakes.

Production: the sibling session deployed CN main a79de48b8 at 2026-10-21 04:00 (its record: CNDEPLOY END result=DEPLOYED_PROVED); its proof showed the network-registry wait at zero 20/20 and a real TinySite publish succeeding on attempt 1 in 189 s. The original incident is therefore fixed in production with parts 2 and 3. PRs 742 and 755 remove the remaining network-thread blocking paths and are the reason a further CN bump and deploy are still wanted. That bump and deploy belong to the sibling session, per decision 6.

What was found and done (the chain)

1. Diagnosis (2026-10-19). Twelve CN thread dumps 1.8 s apart showed 1 to 9 of 16 libp2p-nio-worker-6-* threads BLOCKED in RpcServerMessageHandler.channelRead0 → RelayDiscovery.onRelayAnnouncement → PeerRegistry.addPeer → publishCompletedAdd, with one thread holding the lock inside bootstrapRanking. Raw dumps were in the session scratchpad (inv/evidence/tds/), which is gone with the box; the mechanism is reproduced by the eviction PR's tests.
2. Eviction (part 3). Lanes A3 to A3f built O(log n) incremental eviction with an evidence-before-diversity rule; eight legacy eviction tests were migrated under decision 4. The sibling session took over PRs 725, 737, 738, 739 on 2026-10-20 20:52 and landed them.
3. Snapshots (part 2). Lanes A2 to A2j: atomic swap over persistent collections, "freeze membership only" (decision 3). The 100,000-operation churn test moved from A3 to A2 as its acceptance test because 69% of its time was PeerSnapshot.updated, an O(n) copy. Merged as PR 750.
4. Single owner (part 1). Abandoned in its full form (branch fix/peer-registry-single-owner, 8bb70e6bc, a possible deadlock in awaitOutcome inside runIfActive, 287 test files touched). Replaced by the evidence-off-network-thread slice (PR 755) and the guard tests (PR 781). Owner's decision 7a: no further slices.
5. PR 755 review rounds (lanes A1e to A1x, reviews RV755 to RV755g): fixed the coordinator catching only Exception; owner threads starting on the Netty thread one per connection (now lazy, shared per gossip handler); close hanging; a SecurityManager test hook; a reconnect during owner retirement throwing "still stopping" on both the fallback and shared-owner paths (now a generation scheme); unbounded retiring generations (now capped at 2 per handler); README thread-safety statement. Lane A1t found a wrong baseline in a brief (the control commit already contained the fix); the lesson is recorded under reusable knowledge. The current head 2b5a5dea9 is rebased onto main 6a9a85aa6, resolving a non-trivial conflict with main's host-close and evidence-drain changes from https://github.com/CodexCoder21Organization/UrlProtocol/pull/687 and https://github.com/CodexCoder21Organization/UrlProtocol/pull/749; an independent integration review of that merge found nothing. On that head: red 18/18 for the intended reasons on the pre-fix code, green 6/6, 43-test register 43/43, all first attempts.
6. PR 742 review rounds (lanes B2 to B28, reviews RV742 to RV742f). Fixed: unbounded callback drain at host stop (now capped at 5 s); SEVERE recorded for local-capacity refusals and CLEAN for sender-caused ones (now a typed RejectionKind(senderCaused)); lost warning on refused oversized requests (NetworkCallbackDrainWarningEffect); a thread leak; a dead terminal path; heap-size-dependent tests; a throw in the deadline task that could hang stop; decoded requests queuing on the serial ordinary dispatcher before request-scope admit (B12's reproducer, fixed by B13); and (B27) each stop() constructing two threads on the caller. Two lanes were killed by the codex content filter while building byte-limit probes (B4, B12) or failure-injection tests (B25); the work was relaunched with plain-language briefs.
7. PR 742's one open defect. stressTestRelayLargePayloadLifecycleStartsIndependentHostStopsTogether (500 ms limit) first-fails about 1 in 5 on the PR head and passes on main. B27's matched 259-host profiles: main 0 callback-owned platform threads, all stops 74 ms, caller CPU about 14 ms; PR 1,036 callback-owned threads (four per host, all exiting), stops 313 ms, caller CPU about 93 ms; no GC pauses on either side. Lane B28 (running) has already proved the shape red through public API: four unused host lifecycles create 16 callback worker threads on the PR head (3/3 native attempts fail; out/b28/red-unused-1.xml on wip/lane-b28-evidence), and its JFR pair shows no synchronous caller wait inside stop(), so the cost is asynchronous worker start and exit, not a join.
8. Two main flakes found by the CI first-attempt audits, each with an owner-approved fixture repair (decision 7b, 7c):
   • testTcpCloseWhileConcurrentSilenceSchedulesArePending failed first attempt in main run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39523023281. Mechanism (lane FL4, confirmed by an independent proposal review): the test demands an exact total of Clock.schedule calls, but TcpRpcStream refreshes the silence timer outside its lock and cancels obsolete handles (TcpRpcStream.kt:608-665), so the reader thread can legitimately schedule while a timer replacement is unpublished; the README (README.md:76-84) promises the silence window and the 64-ID threshold, not a schedule count. The injected clock also threw on interruption before returning its handle, so the product never got the handle to cancel. Repair spec: investigation/fl4/proposal-review.md and investigation/fl4/pr-draft.md on fix/tcp-silence-schedule-evidence-fl4. Lane FL13 (running) is applying it on fix/tcp-silence-schedule-count.
   • testForwardedOutputRefusalPreservesAtomicBoundary failed first attempt with "Broken pipe" in PR 742's run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39691632300 and in main run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39547098945. Mechanism (lane FL11): the test closes a 600,000-byte response early and then asserts the backend saw no error, but the README says closing a response cancels an incomplete body and a failed output write closes the affected stream, so a backend still writing legitimately gets a reset. Evidence: 1/257 first-attempt failures with the unchanged scenario on main edc4c63729; 22/22 with the ordering forced on ff2fc35967; a transport control 20/20. Question and exact patch: out/rFL11-question.md and out/fl11-proposed.patch on wip/lane-fl11-evidence. Lane FL12b (running) is applying it on fix/forwarded-refusal-provider-close (already at dda6497b6, one commit beyond main).
9. Other main flakes fixed and merged on the way: UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1345 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1346; UrlProtocol https://github.com/CodexCoder21Organization/UrlProtocol/pull/758, https://github.com/CodexCoder21Organization/UrlProtocol/pull/766, https://github.com/CodexCoder21Organization/UrlProtocol/pull/753.

Relevant PRs / refs

Pull requests

Branches (remote heads verified with git ls-remote at 03:50 UTC)

Session scratchpad files that are NOT on any branch, by decision: the per-lane briefs (h*.md) and codex logs. They are tactical and reproducible from this handoff. Everything with engineering content (findings, probes, patches, XML registers) is on the branches above.

Deployed or published but not merged

This session deployed nothing and published no artifact (checked: no publish-maven or ContainerNursery commands were run by it; the deploy lane belongs to the sibling session). The sibling session deployed CN from main a79de48b8 on 2026-10-21 04:00 (its audit line on the host: /root/cn-watchdog.audit, CNDEPLOY END result=DEPLOYED_PROVED), which contains parts 2 and 3 and nothing unmerged. Re-check with coursier launch containernurserycli:container-nursery-cli:0.0.20 -r https://kotlin.directory -- --url https://api.nursery.wasmserver.com list-routes and the live commit in the CN jar before any further deploy.

Related handoffs created by lanes of this effort

• https://www.handoff.wasmserver.com/handoffs/hf-2026-10-19-finish-peerregistry-eviction-validation-and-open-its-pr (lane A3b): its work is complete, PR 739 merged; completed at wind-down.
• https://www.handoff.wasmserver.com/handoffs/hf-2026-10-19-finish-urlprotocol-s-bounded-event-loop-handoff-and-prove-every-callback-path (lane B, 2026-10-19): its deliverable (one UrlProtocol PR) is PR 742; the remaining work is in this handoff; completed at wind-down as superseded.
• The sibling session's handoff hf-2026-10-19-containernursery-peerregistry-stall-tinysite-publish-timeouts owns the CN bump and deploy.

Lanes in flight at write time (03:55 UTC)

Each runs as a detached codex exec on the /code box in a session scratchpad checkout; if the box is gone, their branches above are the record. Their end states are appended under "Lane end states" when they finish.

Next steps

1. Finish PR 755. When its bld-all-tests run completes, audit the root-level attempt history (first-attempt failures) against main's exact-base artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39706687041 (3089/3089, zero histories). Read RV755h's verdict (out/rRV755h-findings.md, or its evidence branch if the box is gone). If CI is first-attempt clean and the review has no findings, do a final review of the diff yourself and enqueue with /home/u/bin/cs launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlProtocol --pr 755 --to-merged --interval-seconds 180 (background). Verify on main afterwards. Any finding goes back to a delegate, not fixed by hand.
2. Finish PR 742. Take B28's result: if it pushed a fix, run a fresh review of that head, audit CI first attempts, and confirm the stress stop test passed 10/10 first attempts in B28's XML before enqueueing. If B28 ran out of time, the next lane starts from its red test (tests/testB28UnusedHostLifecycleStartsNoCallbackWorkers.kts, name to be made descriptive) and the mechanism it names; allowed fix shapes are in the B28 brief summary: share one bounded pool across hosts, keep workers out of the stop path while preserving the 5-second drain cap, or stop creating threads with nothing to do. Never raise the 500 ms limit.
3. Land the two fixture PRs when FL12b and FL13 open them: mutation check recorded, 90 first-attempt passes from runner XML, forced variants green, CI first-attempt audit, fresh review, enqueue. If FL13's local build keeps hitting the 20-minute build-rule cap, run its proof with scripts/test.bash --remote once the shared buildtest service is admitting; do not shorten any deadline.
4. Then tell the sibling session (or the owner) that 742 and 755 are on main, so it can bump ContainerNursery's foundation.url:protocol pin and deploy. Do not deploy from this effort.
5. Open residual, low priority: the UrlResolver forced-preemption probe that once picked a lower-tier relay (branch wip/rf2-review-evidence); lane F3 on this question was killed by the codex content filter, so a plain-language brief is needed if it is picked up.

Reusable / operational knowledge

• Audit first attempts, not badges. The required bld-all-tests check uploads a test-attempt-history artifact; failed-then-passed tests appear only in its root-level attempt-history entries while every final row is green. Two of 2,954 tests on PR 742 failed first attempt behind a green badge, and one pointed at a real regression. Compare every first-attempt failure with main's exact-base run before enqueueing.
• Before telling a lane to prove a test red against a control commit, confirm the control lacks the fix. Lane A1t lost an hour because the brief's baseline 1c9d40b66^ already contained the fix.
• Codex content filter. Lanes designing deliberate failure-injection tests, byte-limit probes (64 requests of 100k characters) or buffer-accounting code were killed mid-run (B4, B12, B25, F3). Keep such work out of lane briefs or accept a structural fix without a failing-first test; relaunch with plain-language wording. A supervisor note dropped into a running lane's out/ directory is usually not read; put rulings in the successor's brief.
• Write stop conditions by intent. "Stop if another PR changes this test" was read literally against a one-line dependency-pin edit (FL12). Name the purpose of the rule.
• Local builds on the /code box. A full local UrlProtocol suite is OOM-killed by the 10 GiB cgroup when other lanes build (exit 137); use the required CI run as the full gate. kompile keys its cache on the workspace path, so copying sources to a fresh directory costs a cold 0.5 GB cache and 20+ minutes; the kompile build rule has a fixed 20-minute internal cap that cannot be extended from the CLI, and under load average 58 to 72 compiles hit it. One kompile per lane; --remote for streaks when buildtest is admitting (it was recovering local state for about 157 minutes after a restart on 2026-10-21; nobody should restart it).
• Lane liveness is ls -l /proc/*/fd | grep -c out/r<L>.log (4 means alive); ps greps misreport multi-line prompt argv. Kill by process group.
• Thread churn as a performance signal. When two sides of a performance gap show no GC difference, count distinct thread starts and ends per operation (JFR ThreadStart/ThreadEnd; raw ThreadEnd records repeat IDs, so count unique IDs). Four short-lived threads per host lifecycle explained a 4x stop cost here.
• Flaky fixtures that assert unpromised outcomes: make the race deterministic and assert the documented outcome (timer-only schedule count with a reader join; cancelled tail write must fail with SocketException), never accept either outcome and never loosen a limit.

Lane end states (appended as lanes finish; newest last)

• A1x (PR 755 implementer), stopped 04:05 UTC at its cutoff. Pushed 2b5a5dea9; red 18/18, green 6/6, 43/43 first attempts; CI audit left to the reviewer (done below). Records: wip/a1x-review-evidence-20261022.
• B27 (PR 742 implementer), stopped 03:30 UTC at its limit. Pushed 5045c47c6 as a partial publication (17/17 targeted gates; stress stop test 4 pass / 1 fail; thread-churn profile). Records: wip/lane-b27-evidence. Its required run on 5045c47c6 later finished SUCCESS (both checks) but its first-attempt artifact was NOT audited by anyone; audit it before trusting it.
• RV755h (fresh review of PR 755 at 2b5a5dea9), finished 04:48 UTC: REQUEST CHANGES, three P2 findings. Exact-head required run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39717170015 audited: 3,128/3,128 first attempts, zero histories; exact base 3,089/3,089. No existing test weakened. Findings (full detail, state-transition table and a public-API probe that reproduces F1 are on https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/rv755h-review-evidence-20261022, head a3f919a8d, file out/rRV755h-findings.md):
  • F1, AuthenticatedInteractionEvidence.kt:501/:514/:534. A partial host stop publishes its successful shared-delivery result before timer.cancel(); a cancel that throws escapes uncaught while the evidence snapshot still reports failures=0. Reproduced with two loopback hosts and a ManualClock wrapper whose Scheduled.cancel throws (out/RV755hPartialCancelProbe.java). Fix: record cancellation failure before publishing the result, as the standalone close path already does; add the partial-host regression.
  • F2, AuthenticatedInteractionEvidence.kt:723. Repeated sibling lease closes can accumulate lifecycle actions before the checkpoint admission bound and the partial-stop deadline apply. Fix: bound lifecycle admission before retaining work, fail non-waiting when full, start the partial-stop deadline at close admission; add receiver-churn coverage with a held dispatch dependency.
  • F3, the reconnect regressions (DirectRpcSharedReconnectReplacesRetiringEvidenceOwner.kt:449/:465 and the fallback/generation-cap fixtures). They require the old owner's close to stay pending and then succeed; a correct replacement that takes over five seconds makes the old close time out legally and the test fails. Also untested: an expired-but-still-running generation must keep consuming its slot. Fix: gate replacement initialisation through the public ProtocolBinding until the old close has reported its deadline, then assert progress, the cap refusal and recovery. These are the first next step for PR 755. CI is clean; the PR is not mergeable until a fix round and a fresh review.
• FL13 (timer-test fixture), still running at 04:50. Opened DRAFT https://github.com/CodexCoder21Organization/UrlProtocol/pull/821 from fix/tcp-silence-schedule-count at b76a29330 to start the required full suite early (bld-build SUCCESS, bld-all-tests running); the PR body marks every unfinished count PENDING. Initial repair green 2/2; the 103-entry proof register and the mutation control were still pending. Wall limit 05:48.
• FL12b (forwarded-refusal fixture), still running at 04:50. Rebased onto main b3057c064 and pushed fix head 240fa0658 on fix/forwarded-refusal-provider-close (no PR yet; PR creation is gated on the proof counts). Mutation control is a valid negative: 3/3 attempts fail at the source-before-tail barrier when the product keeps draining. The proof register (10+10+90+3) was running. Wall limit 06:00.
• B28 (PR 742 stop slowdown), still running at 04:50. Red test proven (16 worker threads for 4 unused lifecycles, 3/3). Two local product builds hit the fixed 20-minute build-rule cap under machine load and one remote run failed in droplet provisioning, so no native green yet. It rebased onto b3057c064 (source head 794e561ed) and was pushing an explicitly unqualified checkpoint to the PR to start exact-head CI; the PR body lists every unmet gate. Treat that head as unverified until the stress gate (10/10 first attempts) and the new test's green run exist in runner XML.