UrlProtocol full suite intermittently loses loopback LAN peer discovery under parallel load

Reported (UTC): 2026-07-15 20:57

What was being attempted: Run the complete UrlProtocol test suite with ./scripts/test.bash --test . after adding a backward-compatible bounded PeerExchange response projection.

What went wrong: The suite completed 1006/1007 tests and failed foundation.url.protocol.testLanDiscoveryLoopbackRpcStream after 2.4 seconds with: java.lang.AssertionError: Node A should learn node B's service through UDP-only loopback LAN discovery and source-address peer exchange. Expected value to be not null. The changed feature is absent from this test's legacy three-argument PeerExchange requests, whose serialized wire form and full-response handling remain unchanged. The complete suite log is local at /tmp/urlprotocol-full-suite.log.

Impact: A nearly complete 15-minute local suite reports red for a failure outside the changed path, preventing a clean all-tests signal and risking the same intermittent failure in unrelated PR CI.

Workaround used: Reran the exact unchanged test ten times sequentially with ./scripts/test.bash --test tests/testLanDiscoveryLoopbackRpcStream.kts; all 10/10 passed. No timeout, assertion, provider count, or implementation was changed.

Suggested durable fix: Investigate and amplify the LAN discovery test under the same parallel child-JVM load as the full suite until the UDP beacon/source-address peer-exchange loss reproduces at least 50%. Capture socket ownership, multicast/loopback datagrams, and peer-exchange traces to identify whether parallel tests contend on a shared UDP discovery port or global PublicAddressDiscovery state, then fix that owner and add a reliable stress regression.

Production verification — 2026-07-17

Status: STILL EXISTS. A live ProductionHealth URL connection succeeded but emitted repeated NothingToCompleteException failures while forwarding gossip; the last HardwareControlFabric daemon log also contains Netty ByteBuf leak reports in libp2p negotiation paths.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.