← Priority list

UrlResolver pluggable network layer �� Phases 1-3b done, Phase 4 steps 1-2 merged; 3 open PRs code-complete but gated on CI infrastructure

Six PRs merged (the NetworkProvider seam, exact-peer APIs, and Phase 4 steps 1-2 including the three 1024-node deterministic gossip scenarios). Three open PRs are code-complete and verified but cannot land: PR 422 (replay-determinism fix, all checks green) sits behind unrelated merge-queue entries; PR 416 (peer score records plus an eviction-performance and a health-check fix) was evicted twice by a known load-sensitive keepalive test tracked in UrlProtocol issue 423; resolver PR 852 (a one-line pin fix that unblocks ALL resolver PRs) has burned three CI runs on three distinct infrastructure failures. Remaining to graduation: land those three, wire peer scores into allocation and publish a protocol version above 0.0.409, then add resolver-side scenarios and an in-process storm reproduction.

hf-2026-07-23-urlresolver-pluggable-network-layer-phase3

Edit handoff

Title Summary Markdown body

# UrlResolver pluggable network layer — Phases 1–3b COMPLETE + Phase 4 steps 1–3 landed or green; three PRs open, all gated on CI infrastructure

**Written 2026-07-26 06:20 UTC. RE-VERIFY EVERYTHING BEFORE ACTING.** All PR/CI state below is a
write-time snapshot. Authoritative re-checks:
`gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,headRefOid,statusCheckRollup,mergedAt`;
for buildtest runs the page `https://buildtest.kotlin.build/run?id=<id>` plus host ground truth
`ssh -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<id>/{test-events.jsonl,build.log}'`.

## Mission and graduation criteria

Make UrlProtocol's P2P network layer a swappable constructor parameter (NetworkProvider seam:
Libp2pNetworkProvider default, InMemoryNetworkProvider deterministic test fabric), route
UrlProtocol2 (in UrlResolver) through it, then build a ≥1000-node deterministic in-process gossip
suite gating BOTH repos' CI. Workstream doc:
https://github.com/CodexCoder21Organization/PlanRepository/blob/main/workstreams/UrlResolverPluggableNetworkLayer.md

Graduation (from that doc):
1. Seam merged with libp2p default, zero behaviour change — **DONE**.
2. InMemoryNetworkProvider with bounded buffers + harness controls (stall/slow/kill/partition/heal)
   — **DONE** (all six controls exist).
3. Deterministic ≥1000-node suite, including ≥1 reproduced historical storm, gating **both** repos'
   CI — **UrlProtocol side largely done; resolver side NOT started; storm reproduction NOT done.**

Merges for this workstream are pre-authorised. Production deploys are NOT.

## Merged (this effort)

- https://github.com/CodexCoder21Organization/UrlProtocol/pull/411 — 0.0.402 publish support.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/412 — two flaky-test root-cause fixes.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/413 — exact-peer APIs; **0.0.405**
  published (sha256 `7caf1a1352ff930173f5f9324f747b54831280a5679c1e22bd733c81d29cfae6`).
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/415 — Phase 4 step 1: scale-safe
  fabric observation (immutable snapshots, `drainReadyWork()`, ordered queues).
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/417 — Phase 4 step 2: three
  1024-node scenarios (deterministic replay, amplification linearity, partition/heal) + virtual
  threads for in-flight handler stacks (peak platform threads 3521 → 19).
- https://github.com/CodexCoder21Organization/UrlResolver/pull/847 — Phase 3b: UrlProtocol2 routed
  through the seam.

Note: **0.0.409** is the latest published protocol version (published by other work); UrlProtocol
`build.kts` still declares `0.0.405`. Any new publish must exceed 0.0.409.

## Open PRs — all three are code-complete and gated on CI infrastructure

### 1. https://github.com/CodexCoder21Organization/UrlProtocol/pull/422 — replay determinism fix
Branch `fix-replay-peak-counter-determinism`, head `25ad7f5e`. **All three head checks GREEN.**
In the merge queue at position 3, entry state UNMERGEABLE; the entry ahead (an unrelated PR) is
also UNMERGEABLE while position 1 awaits checks — i.e. queue-group state, not this change.

Why it exists: the 1024-node replay scenario compared the *entire* fabric snapshot between two
same-seed runs, including `peakActiveStreamCount` — a high-water mark of momentarily-overlapping
streams. One handler runs at a time, but a finishing handler's stream can still be closing while
the next opens, so the peak reads 1 or 2 by interleaving. Measured: **1 of 3 runs failed on main
alone; 3 of 3 with PR 416 merged in**; the merge-queue diff was exactly
`peakActiveStreamCount expected 1 | actual 2`, every other field identical. Fix normalises the two
scheduling-derived peaks out of the comparison and asserts the stream peak against an explicit
bound instead. Verified 6/6 consecutive passes; three sibling 1024-node scenarios green.

**This must land before PR 416 can clear the queue.**

### 2. https://github.com/CodexCoder21Organization/UrlProtocol/pull/416 — Phase 4 step 3, peer score records
Branch `phase4-peer-score-records`, head `3b1fcf22`, mergeStateStatus CLEAN, out of the queue.
Implements the PEER_MANAGEMENT.md bounded-series model, plus all 14 findings from two reviews,
plus two fixes made while shepherding:
- eviction-path efficiency: scoring is lazy and only computed when a candidate actually has
  evidence; the evidence check reads the record in place instead of snapshotting every bounded
  series. Measured on a 1000-peer registry: 250 extra adds 1722→122 ms, 500 2607→117 ms,
  1000 4396→275 ms, 2000 9092→479 ms.
- health-check due test: a never-contacted peer is due immediately (backoff has no previous check
  to measure from), removing an ~8 %-per-run flake.
Evicted from the merge queue twice, both times on known causes: the determinism flake above and
the keepalive test in the issue below.

### 3. https://github.com/CodexCoder21Organization/UrlResolver/pull/852 — stray dependency pin
Branch `fix-protocol-pin-conflict`, head `41230477`. One line:
`tests/stressTestRelayFailoverDebugCapture.kts` pinned `foundation.url:protocol` at 0.0.407 while
the build script and 1192 other references pin 0.0.409. The pin scanner fails the whole suite on
that mismatch **before any test runs**, so this blocks every resolver PR. The affected stress test
passes on 0.0.409 (86 s).

**Three consecutive CI runs, three distinct infrastructure failures, zero code signal:**
1. netlab `uploadFile` RPC timeout to `netlab-hosted` (30 s × 5 attempts);
2. netlab route-severing command returned exit code 2 (`testNetlabLinkFlapAndProviderMigration`);
3. the buildtest service restarted mid-run, orphaning the dynamic-dispatch lease scheduler —
   the run failed itself with "…no further work can be handed to this runner… resubmit it".
A fourth run was resubmitted at 06:20 UTC.

## Known blockers that will bite the next person

- **https://github.com/CodexCoder21Organization/UrlProtocol/issues/423** — the relay keepalive
  scenario `testRelayFrozenClockKeepaliveStrictlyExtendsRegistrationExpiry` fails on loaded 2-vCPU
  executors (3 occurrences). It has **evicted PRs from the UrlProtocol merge queue twice**. Passes
  15/15 locally including under `taskset -c 0,1`. Its second latch allows only 3 s for two more
  keepalive round-trips. **Do not pad the timeout** — the fix is to await later keepalives on the
  same deterministic signal used for the first.
- **`url://netlab-hosted/` is unhealthy**, and
  https://github.com/CodexCoder21Organization/NetLabManagerServer/pull/111 (a reaper fix) is
  **merged but never deployed**. Deploying it requires explicit owner authorisation. Its symptoms
  (leaked/missing environments, timeouts) match what is failing resolver CI.

## Remaining work to graduation

1. Land PRs 422 → 416 → 852 (all code-complete).
2. Phase 4 step 4: wire peer scores into allocation/bootstrap ranking, then publish a protocol
   version > 0.0.409 carrying all provider + peer-management APIs.
3. Resolver-side deterministic scenarios so the suite gates UrlResolver CI too (needs the publish).
4. In-process historical storm reproduction (graduation explicitly requires ≥1).
5. Update the workstream doc checkboxes and complete this handoff.

## Two investigations parked with evidence (scratchpad of session c1421455)

- `parked-storm-finding.md` — a 562-node stalled-peer scenario wedges the fabric in
  `awaitConditionLocked`. A 3-node probe **refuted** the obvious cause: `announceService` does NOT
  block on a stalled peer (returns normally). Take a thread dump at the wedge and identify who
  holds the inbound-handler turn before theorising further.
- `parked-resolver-finding.md` — a seeded-shuffle change for gossip fanout selection was **dropped,
  not merely paused**: with the seeding removed the selected peer sets were byte-identical
  (hashes `1115653836, 1115653836, 1544631395` both ways), because only ~16 peers are eligible and
  the fanout cap is 16, so `take(fanout)` consumes the whole list and order cannot matter. Anyone
  reviving it must first find a configuration where eligible peers genuinely exceed the cap.

## Operational gotcha that will waste an hour if missed

`scripts/test.bash` in **UrlResolver** served `foundation.url.resolver.buildMaven` as a **2 ms cache
hit** and silently ignored source edits — appending deliberately invalid Kotlin still produced a
pass. `rm -rf ~/.aibuildcaches` fixes it; after clearing, the same syntax error correctly failed
with 11 compile errors in a real 21 s build. **Clear that cache before trusting any resolver
verification.**

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

UrlResolver pluggable network layer — Phases 1–3b COMPLETE + Phase 4 steps 1–3 landed or green; three PRs open, all gated on CI infrastructure

Written 2026-07-26 06:20 UTC. RE-VERIFY EVERYTHING BEFORE ACTING. All PR/CI state below is a write-time snapshot. Authoritative re-checks: gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,headRefOid,statusCheckRollup,mergedAt; for buildtest runs the page https://buildtest.kotlin.build/run?id=<id> plus host ground truth ssh -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<id>/{test-events.jsonl,build.log}'.

Mission and graduation criteria

Make UrlProtocol's P2P network layer a swappable constructor parameter (NetworkProvider seam: Libp2pNetworkProvider default, InMemoryNetworkProvider deterministic test fabric), route UrlProtocol2 (in UrlResolver) through it, then build a ≥1000-node deterministic in-process gossip suite gating BOTH repos' CI. Workstream doc: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/workstreams/UrlResolverPluggableNetworkLayer.md

Graduation (from that doc):

Seam merged with libp2p default, zero behaviour change — DONE.
InMemoryNetworkProvider with bounded buffers + harness controls (stall/slow/kill/partition/heal) — DONE (all six controls exist).
Deterministic ≥1000-node suite, including ≥1 reproduced historical storm, gating both repos' CI — UrlProtocol side largely done; resolver side NOT started; storm reproduction NOT done.

Merges for this workstream are pre-authorised. Production deploys are NOT.

Merged (this effort)

https://github.com/CodexCoder21Organization/UrlProtocol/pull/411 — 0.0.402 publish support.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/412 — two flaky-test root-cause fixes.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/413 — exact-peer APIs; 0.0.405 published (sha256 7caf1a1352ff930173f5f9324f747b54831280a5679c1e22bd733c81d29cfae6).
https://github.com/CodexCoder21Organization/UrlProtocol/pull/415 — Phase 4 step 1: scale-safe fabric observation (immutable snapshots, drainReadyWork(), ordered queues).
https://github.com/CodexCoder21Organization/UrlProtocol/pull/417 — Phase 4 step 2: three 1024-node scenarios (deterministic replay, amplification linearity, partition/heal) + virtual threads for in-flight handler stacks (peak platform threads 3521 → 19).
https://github.com/CodexCoder21Organization/UrlResolver/pull/847 — Phase 3b: UrlProtocol2 routed through the seam.

Note: 0.0.409 is the latest published protocol version (published by other work); UrlProtocol build.kts still declares 0.0.405. Any new publish must exceed 0.0.409.

Open PRs — all three are code-complete and gated on CI infrastructure

1. https://github.com/CodexCoder21Organization/UrlProtocol/pull/422 — replay determinism fix

Branch fix-replay-peak-counter-determinism, head 25ad7f5e. All three head checks GREEN. In the merge queue at position 3, entry state UNMERGEABLE; the entry ahead (an unrelated PR) is also UNMERGEABLE while position 1 awaits checks — i.e. queue-group state, not this change.

Why it exists: the 1024-node replay scenario compared the entire fabric snapshot between two same-seed runs, including peakActiveStreamCount — a high-water mark of momentarily-overlapping streams. One handler runs at a time, but a finishing handler's stream can still be closing while the next opens, so the peak reads 1 or 2 by interleaving. Measured: 1 of 3 runs failed on main alone; 3 of 3 with PR 416 merged in; the merge-queue diff was exactly peakActiveStreamCount expected 1 | actual 2, every other field identical. Fix normalises the two scheduling-derived peaks out of the comparison and asserts the stream peak against an explicit bound instead. Verified 6/6 consecutive passes; three sibling 1024-node scenarios green.

This must land before PR 416 can clear the queue.

2. https://github.com/CodexCoder21Organization/UrlProtocol/pull/416 — Phase 4 step 3, peer score records

Branch phase4-peer-score-records, head 3b1fcf22, mergeStateStatus CLEAN, out of the queue. Implements the PEER_MANAGEMENT.md bounded-series model, plus all 14 findings from two reviews, plus two fixes made while shepherding:

eviction-path efficiency: scoring is lazy and only computed when a candidate actually has evidence; the evidence check reads the record in place instead of snapshotting every bounded series. Measured on a 1000-peer registry: 250 extra adds 1722→122 ms, 500 2607→117 ms, 1000 4396→275 ms, 2000 9092→479 ms.
health-check due test: a never-contacted peer is due immediately (backoff has no previous check to measure from), removing an ~8 %-per-run flake. Evicted from the merge queue twice, both times on known causes: the determinism flake above and the keepalive test in the issue below.

3. https://github.com/CodexCoder21Organization/UrlResolver/pull/852 — stray dependency pin

Branch fix-protocol-pin-conflict, head 41230477. One line: tests/stressTestRelayFailoverDebugCapture.kts pinned foundation.url:protocol at 0.0.407 while the build script and 1192 other references pin 0.0.409. The pin scanner fails the whole suite on that mismatch before any test runs, so this blocks every resolver PR. The affected stress test passes on 0.0.409 (86 s).

Three consecutive CI runs, three distinct infrastructure failures, zero code signal:

netlab uploadFile RPC timeout to netlab-hosted (30 s × 5 attempts);
netlab route-severing command returned exit code 2 (testNetlabLinkFlapAndProviderMigration);
the buildtest service restarted mid-run, orphaning the dynamic-dispatch lease scheduler — the run failed itself with "…no further work can be handed to this runner… resubmit it". A fourth run was resubmitted at 06:20 UTC.

Known blockers that will bite the next person

https://github.com/CodexCoder21Organization/UrlProtocol/issues/423 — the relay keepalive scenario testRelayFrozenClockKeepaliveStrictlyExtendsRegistrationExpiry fails on loaded 2-vCPU executors (3 occurrences). It has evicted PRs from the UrlProtocol merge queue twice. Passes 15/15 locally including under taskset -c 0,1. Its second latch allows only 3 s for two more keepalive round-trips. Do not pad the timeout — the fix is to await later keepalives on the same deterministic signal used for the first.
url://netlab-hosted/ is unhealthy, and https://github.com/CodexCoder21Organization/NetLabManagerServer/pull/111 (a reaper fix) is merged but never deployed. Deploying it requires explicit owner authorisation. Its symptoms (leaked/missing environments, timeouts) match what is failing resolver CI.

Remaining work to graduation

Land PRs 422 → 416 → 852 (all code-complete).
Phase 4 step 4: wire peer scores into allocation/bootstrap ranking, then publish a protocol version > 0.0.409 carrying all provider + peer-management APIs.
Resolver-side deterministic scenarios so the suite gates UrlResolver CI too (needs the publish).
In-process historical storm reproduction (graduation explicitly requires ≥1).
Update the workstream doc checkboxes and complete this handoff.

Two investigations parked with evidence (scratchpad of session c1421455)

parked-storm-finding.md — a 562-node stalled-peer scenario wedges the fabric in awaitConditionLocked. A 3-node probe refuted the obvious cause: announceService does NOT block on a stalled peer (returns normally). Take a thread dump at the wedge and identify who holds the inbound-handler turn before theorising further.
parked-resolver-finding.md — a seeded-shuffle change for gossip fanout selection was dropped, not merely paused: with the seeding removed the selected peer sets were byte-identical (hashes 1115653836, 1115653836, 1544631395 both ways), because only ~16 peers are eligible and the fanout cap is 16, so take(fanout) consumes the whole list and order cannot matter. Anyone reviving it must first find a configuration where eligible peers genuinely exceed the cap.

Operational gotcha that will waste an hour if missed

scripts/test.bash in UrlResolver served foundation.url.resolver.buildMaven as a 2 ms cache hit and silently ignored source edits — appending deliberately invalid Kotlin still produced a pass. rm -rf ~/.aibuildcaches fixes it; after clearing, the same syntax error correctly failed with 11 compile errors in a real 21 s build. Clear that cache before trusting any resolver verification.