Handoff: url:// N² gossip/discovery resolution — protocol fixes unified & published, resolver consumers repinned, three consumer merges remain

Written 2026-07-21 ~23:36 UTC

RE-VERIFY BEFORE ACTING. Every state claim below is a write-time snapshot and goes stale fast (PRs merge, CI re-runs, branches get force-pushed, the buildtest host oscillates between load ~5 and ~140). A fresh query always wins over this document.

• PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Published protocol artifact: curl -s -o /dev/null -w "%{http_code}" https://kotlin.directory/foundation/url/protocol/<ver>/protocol-<ver>.pom (200 = published; 401 = absent/free)
• buildtest host load (the dominant blocker all along): ssh -p 23 -i ~/.ssh/id_ed25519_20260706_035316 root@198.199.106.165 uptime

Mission summary

Original user request (verbatim intent): "Drive ALL O(N²) growth issues in the CodexCoder21Organization url:// gossip/discovery stack to final resolution (resolve/eliminate, not mitigate)," delegating the heavy lifting (implementation, review, verification) to codex gpt-5.6-sol, parallelized across agents.

The named N² items were: (a) O(N²) gossip service-announcement fan-out; (b) O(n²) yamux ByteBufQueue/Base58 (already fixed upstream in jvm-libp2p #23); (c) the O(N²)-by-construction forwardedDiscoveredServices table → redesign to O(#services) per node; (d) roll the bounded resolver to all consumers.

Directive evolution during the session:

1. "Merge the n² fixes as soon as green, then upgrade consumers, publish, deploy, merge" (granted merge + deploy authority).
2. "#387 merged, deploy it everywhere to stop the bleed."
3. "The emergency looks over — let's not do any more deploys. Switch to our methodical style of TDD + adversarial reviews + merging before deploying." This CANCELED all prod deploys; completion criterion became "all fix PRs merged," with prod deploy a separate, future, user-authorized step. Publishing Maven artifacts is allowed (it is not a prod deploy).

What was found and done (the chain) — verified unless flagged

1. Attachment-memory / slow-loris hardening (UrlProtocol). A review of the resolver's RPC v1.1 binary-attachment path found it admitted up to 1000 attachments × 100 MB with no aggregate byte budget and no attachment-count-vs-envelope-reference validation — unbounded allocation from a peer. Fixed at the owning layer (UrlProtocol): attachments now draw from the same shared parse-memory pool as JSON, with a whole-pool FrameAttachmentLease, envelope/count bijection validation, and (after an adversarial review found a slow-loris: a peer dribbling 1 byte/~10 s pins the whole pool forever) an absolute minimum-throughput deadline (~4 KiB/s floor, ≥60 s) driven by an injected Clock, plus a fix to NettyInputStream.close() which was a no-op (parked readers never woke on stream teardown). Fail-first tests proven. → UrlProtocol #390 MERGED, published as foundation.url:protocol:0.0.392.

2. Withdrawal-delivery regression (UrlResolver #804). UrlResolver #804 (bounded per-peer gossip send queue, to cap queued heap) was found — by an empirical A/B control, not by mechanism guesswork — to introduce a real regression: testRelayBroadcastsWithdrawalWhenPeerDisconnects PASSES on main, FAILS 2/2 on #804's head c0da095b, even under low host load. Mechanism: an authoritative service withdrawal is broadcast by the relay but never reaches the observer (onServiceWithdrawal never runs) — a pure delivery loss in #804's rewritten gossip forward/coalesce path. Two from-scratch fix attempts (codex, then an Opus agent) failed/wedged. A decisive experiment then showed UrlResolver #737 already fixes it (same reproducer PASSES 2/2 on #737's branch), fixing the root cause upstream (a failed forward of one withdrawal no longer aborts the provider's remaining withdrawals; the client listener stays alive until the relay acks the unregister). Verdict: land #737, close #804 as superseded. An adversarial review of #737 confirmed the resolver code is correct and complete but surfaced a protocol-version-coherence BLOCKER (next item).

3. Protocol version divergence — the integration blocker. #737 pinned protocol 0.0.390 (the relay-delivery fix, published from the unmerged UrlProtocol #389 branch). #813 pinned 0.0.392 (the attachment fix). These two published versions diverged — 0.0.390 lacked the attachment fix; 0.0.392 lacked the delivery fix — so landing both resolver PRs would collide on the single protocol-pin line and silently drop one upstream fix. Resolution: a unify agent applied #389's delivery-fix net diff onto current UrlProtocol main (which already had #390's attachment fix), composing cleanly, as UrlProtocol #393 — which MERGED. Then 0.0.393 was published and verified (pom+jar 200), with a grep confirming main contains BOTH forwardAuthoritativeWithdrawalToRegisteredPeers (delivery) and FrameAttachmentLease (attachment). #389's old red check was root-caused as a pre-existing LAN-discovery load-flake (testLanDiscoveryConcurrentCallbacksRespectExactDialCap CME, byte-identical to main), not a real failure.

4. Repin both resolver consumers to the unified 0.0.393. UrlResolver #737 (0.0.390→0.0.393, resolver coordinates bumped to 0.0.651) and UrlResolver #813 (0.0.392→0.0.393) were both repinned, rebased, and locally verified green (withdrawal reproducer + siblings on #737; both bytecode tests on #813) before pushing.

5. The dominant, recurring blocker throughout: the shared buildtest host (198.199.106.165) oscillated between load ~5 and ~140 for hours, repeatedly flaking / failing to schedule CI. This is legitimate multi-tenant ContainerNursery + CI load (I verified: ~20 CN service JVMs, no OOM, no reapable zombie fault — do NOT reap it). UrlProtocol #390 and UrlProtocol #393 each landed only by opportunistically enqueuing during a load dip. This is why the three remaining resolver merges did not land in-session.

6. codex gpt-5.6-sol died mid-session (shared-account usage-limit exhaustion) and could not be revived; all subsequent implementation/review/repin work was done by Opus subagents (per the delegate-implementation-to-opus and codex-account-usage-limit operating notes).

Relevant PRs / refs (write-time state — RE-VERIFY)

| PR | State @ write | Notes | |----|---------------|-------| | UrlProtocol #390 | MERGED 2026-07-20 21:14 | Attachment memory + slow-loris fix → published 0.0.392 | | UrlProtocol #393 | MERGED 2026-07-20 23:37 | Unified 0.0.393 (attachment + relay-delivery); head 35fec20d | | UrlProtocol #392 | MERGED 2026-07-21 03:56 | "Serialize relay registration recovery" — merged after 0.0.393 was published, by another actor/queue. RECONCILE: confirm main is still coherent and whether a post-#392 protocol republish is needed (see Next steps). | | UrlProtocol #389 | OPEN, DIRTY | Original delivery fix; its net diff is already on main via #393. Likely closeable as superseded — verify first. | | UrlResolver #737 | OPEN, BLOCKED — head 4eb6c1e4 (re-pushed since my repin cc045081 — likely a parallel session); latest CI: bld-build CANCELLED, kotlin.build (remote) FAILURE | Supersedes #804. Repinned to 0.0.393, coordinates 0.0.651. Needs CI re-driven (host now idle → likely the run failed under the earlier load spike; re-run and check the failing test). | | UrlResolver #813 | OPEN, BLOCKED — head 5433a7c2; latest CI: bld-build CANCELLED, kotlin.build (remote) FAILURE | Bytecode-fetch via RPC v1.1 attachments, repinned to 0.0.393. Needs CI re-driven. | | UrlResolver #819 | OPEN, CLEAN — both checks SUCCESS (head 97313abc) | Gossip-test de-flake (deterministic causal oracles). GREEN and ready to merge right now. | | UrlResolver #804 | OPEN, BLOCKED | Close as superseded by #737 once #737 lands. | | UrlResolver #811 | OPEN, DIRTY | The deeper O(#services) forwarded-discovery redesign — rebase onto the fixed base once #737 lands. | | GithubProxyServerService #26 | OPEN, BLOCKED | Downstream consumer; unblocks after #813 lands (may need its own 0.0.393 bump — verify). |

Also merged earlier this session: ten consumer/protocol version-bump PRs that bounded the leak fleet-wide. Published artifacts: foundation.url:protocol:0.0.392 and 0.0.393 (both 200 at write time).

Next steps (ordered — re-verify current state first)

1. RE-VERIFY everything with the commands in the banner. State has drifted materially within hours before (e.g. #392 merged, #819 went green, #737 got re-pushed) — assume it has again.
2. Merge UrlResolver #819 — it is green + CLEAN. Enqueue via GraphQL (see Reusable knowledge); it is a reviewed test-only de-flake.
3. Re-drive UrlResolver #737 and UrlResolver #813. Both are locally-verified green but their last remote CI FAILED/CANCELLED, most likely under the load-~140 spike. The host is now idle (load ~5), so: identify the failing test on each (gh api repos/CodexCoder21Organization/UrlResolver/commits/<head>/check-runs), and if it's a load flake, re-request the check (or push an empty commit) and let it run on the quiet host. If it's a real failure, root-cause it (do not blind-retry — "CI is not a debugger"). NOTE #737's head is now 4eb6c1e4, not my cc045081 — a parallel session may be driving it; check before force-pushing (avoid clobbering their work; rebase the live tip, per restack-pr-tip-not-handoff-commit).
4. Merge #737 and #813 once green (enqueue on a load dip). Then close UrlResolver #804 as superseded by #737, and verify UrlProtocol #389 is closeable (its diff is on main via #393).
5. Reconcile UrlProtocol #392: it merged to UrlProtocol main after 0.0.393 was published, so 0.0.393 does not include any net-new #392 code. Diff main vs the 0.0.393 release commit; if #392 added behavior beyond what #393 carried, publish a new unified protocol version (probe the next free number first) and repin the resolver consumers to it before merging them. If #392 was a no-op superset (my unify agent assessed #389 ⊇ #392), no republish is needed — just confirm.
6. Unblock GithubProxyServerService #26 after #813 lands (bump its protocol pin if needed).
7. Rebase UrlResolver #811 (the O(#services) forwarded-discovery redesign) onto the fixed base and drive it — this is the last core N² item (redesign the forwardedDiscoveredServices table to O(#services) per node). Grep-verify absence of the old per-recipient forwardedDiscoveredServices symbols.
8. NO prod deploys without explicit user authorization (methodical mode). Publishing Maven artifacts is allowed.
9. When all of the above land, delete this handoff (its own PR, standing pre-approved) per the completion procedure in handoffs/README.md.

Reusable / operational knowledge

• Publishing a kompile artifact to kotlin.directory is TWO steps and CI does not auto-publish on merge. buildMaven alone only builds locally (a re-probe still shows 401). The remote upload requires the publisher CLI:

  JAVA_OPTS="-Xmx1536m" ~/bin/cs launch \
    community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.5 \
    -r https://kotlin.directory -- \
    --workspace <checkout> --buildrule foundation.url.protocol.buildMaven \
    --api-url https://api.kotlin.directory/upload --force --verbose
  

  Success = Response code: 200 + {"installed":[...],"success":true}. "Could not install workspace dependency" warnings are benign. Always probe the pom URL before publishing (401 = free/safe; 200 = taken, do not overwrite) to avoid version collisions with parallel sessions.
• Merging in this org: gh pr merge is rejected org-wide. Enqueue via GraphQL:

  PRID=$(gh pr view <n> --repo <owner>/<repo> --json id --jq '.id')
  gh api graphql -f query='mutation($id:ID!){ enqueuePullRequest(input:{pullRequestId:$id}){ mergeQueueEntry { position state } } }' -f id="$PRID"
  

  The merge queue re-runs the full suite merged with main, so it can (and did, repeatedly) evict on the load flake. Only enqueue on a load dip (uptime on the host <~30) or it will thrash.
• The buildtest host oscillates load ~5↔~140. Big fixes (#390, #393) each landed only during a dip. A remote kotlin.build check stuck IN_PROGRESS while the buildtest run shows COMPLETED-green is a lost completion webhook, not a hang — but re-requesting it re-runs the whole suite (I did this to #813 once and it re-flaked; prefer waiting for the webhook or a targeted re-request). Ground-truth via the buildtest API: curl -s "https://buildtest.kotlin.build/api/runs?limit=40" (fields status, testsPassed/testsTotal, dropletId; dropletId==0 + PENDING = droplet-provisioning backlog, back off don't hammer).
• codex gpt-5.6-sol was the intended implementer but the shared account exhausted mid-session and hard-blocked; fall back to Opus subagents. Also: codex's cybersecurity content filter kills sessions that WRITE malformed-input/limit-abuse decoder-robustness tests — hand those to Opus, don't retry codex.
• Opus subagents parked/wedged repeatedly (ended their turn "waiting for a monitor," or marathon-ran one slow local test for 90+ min). Brief them with hard anti-wedge rules: run one test at a time in the foreground, never end your turn to wait, if a test exceeds its ~590 s self-timeout twice it's the host not your test — push and let CI verify.
• Root-cause discipline that paid off: the #804 regression was nearly dismissed as a "load flake" on a mechanism argument ("the bounded queue can't drop announcements"); the empirical A/B control (same test, main vs the PR commit, under low load) overrode the argument and exposed a real delivery-loss defect. Always run the control before trusting a mechanism story.
• Local checkouts from this session (on the /code box, may be reused): /code/workspace/triage-804 (#804 reproducer + diagnostics), /code/workspace/exp-737 and /code/workspace/review-737 (#737 verification), /code/workspace/unify-protocol (#393 build), /code/workspace/repin-737 & /code/workspace/repin-813 (the repins), plus per-effort notes under /code/workspace/results/*.md.