Workstream: UrlResolver Pluggable Network Layer

Status: In progress — UrlProtocol complete; UrlResolver scale scenarios remain · Component: Maximize developer productivity

Goal

Make the network layer beneath UrlProtocol/UrlResolver a constructor parameter with a libp2p default, so the real gossip, peer-management, and routing code can run over a lightweight in-process fake — and gossip behavior at hundreds to thousands of peers becomes an ordinary, deterministic, sandbox-runnable test instead of an untested production risk.

The motivating gap: UrlResolver/UrlProtocol tests that exercised real protocol behavior once had to run the full libp2p stack (TCP + Noise + Yamux per node), so in-process tests topped out at a handful of peers and anything larger needed NetLab with full Docker images. The full e2e tests remain valuable and are explicitly retained and encouraged, but making them the only option left gossip dynamics in large meshes (fanout amplification, slow-reader buffer growth, storm feedback loops) essentially untested. That blind spot produced real production gossip storms (see the gossip flush-completion fix and the stalled-peer fanout OOM reproductions in UrlResolver's testing/ sources).

Current state

• UrlProtocol is complete. The NetworkProvider seam and libp2p default are merged, and InMemoryNetworkProvider provides deterministic bounded buffers plus stall, slow, kill, partition, and heal controls. The protocol changes through peer-score ranking are published as foundation.url:protocol:0.0.416.
• UrlProtocol's scale gates are complete. Its ordinary CI suite includes deterministic 1,024-node amplification, partition/heal, platform-thread, and replay scenarios; bounded peer-score behavior and score-aware allocation/bootstrap ranking; and a 1,024-node historical duplicate-announcement regression that measures 1,079 bounded stream opens against an 8,184-open deliberately weakened control.
• UrlResolver scenarios remain. Equivalent resolver-facing ≥1,000-node scenarios still need to gate UrlResolver CI. This is the sole remaining graduation item; the tier-2 NetLab provider remains a separate, non-gating follow-on.

The plan

• [x] Define the NetworkProvider seam. One interface captures exactly what UrlProtocol consumes from libp2p: derive identity from a key seed; listen and report addresses; dial (peerId, address, protocolId) → RpcStream; register an inbound-stream handler per protocol ID; and emit connection lifecycle events. Everything above the seam — gossip queues and fanout, TTL/dedup, PeerExchange, and the peer-management design — runs unmodified over any provider.
• [x] Land the libp2p default with zero behavior change. Libp2pNetworkProvider wraps Libp2pHostFactory + Noise + muxers behind the seam. Existing tests and production wiring retain the full-stack default.
• [x] Tier 1: InMemoryNetworkProvider — with mandatory backpressure. Streams use paired bounded in-memory pipes and injected-clock scheduling. The harness provides deterministic stall, slow (bytesPerTick), kill, partition, and heal controls while the real UrlProtocol handlers process the bytes.
• [ ] Gossip-at-scale test suite. Deterministic scenarios at ≥1000 nodes must gate both UrlProtocol and UrlResolver CI.
  • [x] UrlProtocol CI covers deterministic large-mesh amplification, partition/heal convergence, handler thread bounds, replay, peer-score dynamics and ranking, and a reproduced historical duplicate-announcement storm.
  • [ ] UrlResolver CI needs the equivalent resolver-facing large-mesh scenarios. This is the remaining graduation work.
• [ ] Tier 2: NetLabFabricNetworkProvider (cross-linked, non-gating). A second test provider backed by NetLab's in-process datapath — the clock-driven userspace TCP/IP stack exposed via the manual-node API — giving real IP/TCP framing, wire-level impairment and firewall rules, deterministic timers, and cross-regime topologies (some peers in-process, some in Docker behind real NATs, one topology). UrlProtocol becomes the flagship consumer NetLab's pluggable-node-types plan is looking for. Because that stack is NetLab's own critical-path, highest-risk component, this item is tracked here but owned by the NetLab workstream's timeline and does not gate graduation.
• [x] Keep the fidelity ladder explicit. The three rungs remain tier 1 (in-memory, maximum scale and speed), tier 2 (NetLab fabric, wire-faithful), and full stack (libp2p in-process and NetLab Docker topologies for NAT traversal, relay, and hole-punching). The full-stack rung is not deprecated by this workstream — it is the only place Noise, muxers, and real NAT behavior are exercised, and the existing canonical scenarios stay mandatory.

Why it accelerates developers

• It closes the specific blind spot that produced production storms. Gossip dynamics at realistic mesh sizes become a five-line deterministic test instead of a production incident followed by heroic forensics.
• It makes the peer-management design testable. Reputation, ostracism, diversity selection, and Sybil-resistance claims in PEER_MANAGEMENT.md are population-scale behaviors; this is the only realistic way to verify them before deployment.
• Fast, sandbox-native, deterministic. Thousand-node tests with a ManualClock run in seconds, in the ordinary sandbox, with no flake surface from Docker, droplets, or wall-clock timing.
• It is where new production transports plug in. The seam that admits the in-process test providers also admits the HTTPS transport: hosting a url:// service behind plain HTTPS is a second link type beneath NetworkProvider, and the gossip, routing, and peer-management protocols run over it unmodified. What that plan does have to change above the seam is the assumption that every address is a libp2p multiaddress.
• It strengthens NetLab rather than competing with it. Tier 2 gives NetLab's userspace stack its first heavyweight consumer, and the full-Docker e2e rung remains the top of the ladder.

Decisions

• NetworkProvider and its providers live in the existing UrlProtocol module.
• Tier 1 models reachability directly; any further RelayService abstraction belongs to the non-gating tier-2 NetLab follow-on.
• The deterministic ≥1,000-node scenarios gate ordinary CI. Their populations and matrices remain sized to the buildtest budget and are adjusted only when measured CI data supports a change.

Graduation

This workstream graduates when:

1. [x] The NetworkProvider seam is merged with Libp2pNetworkProvider as the default and zero observable behavior change.
2. [x] InMemoryNetworkProvider ships with mandatory bounded-buffer/backpressure semantics and harness controls (stall/slow/kill/partition/heal).
3. [ ] A deterministic ≥1000-node gossip suite — including at least one reproduced historical storm as a regression guard and peer-management scale scenarios — gates both CI suites.
   • [x] UrlProtocol
   • [ ] UrlResolver

The tier-2 NetLab-fabric provider is explicitly not a graduation criterion; it graduates with the NetLab workstream's in-process datapath.