jvm-libp2p full test suite is non-hermetic and order-sensitive.

Reported (UTC): 2026-07-13 13:09

What was being attempted: Verify the Noise-handshake event-loop offload with the required ./gradlew :libp2p:test full-suite run in CodexCoder21Organization/jvm-libp2p.

What went wrong: Both clean full-suite executions ran 902 tests and failed QuicKuboTestJava.pingKubo with java.net.ConnectException: Connection refused from getKuboPeerId(QuicKuboTestJava.java:65), because the test calls an unavailable external Kubo HTTP endpoint. The suite also intermittently failed YamuxOutboundBufferBudgetRefCountTest (expected refCnt 0 but was 2 at line 96) and GossipTwoHostTest.test message larger than mplex frame (Timeout waiting for condition at TwoGossipHostTestBase.kt:123); each of those tests passed immediately when run alone in a fresh Gradle test worker.

Impact: A correct local change cannot produce a green full :libp2p:test result, and order/load-sensitive unrelated failures obscure real regressions. The Kubo test also violates the ecosystem requirement that tests start isolated real dependencies and never depend on external services.

Workaround used: Run the full suite once for classification, then rerun each non-Kubo failure alone without changing assertions, timeouts, or iteration counts. Report the full suite honestly as non-green and retain the isolated green evidence.

Suggested durable fix: Make QuicKuboTestJava start its own isolated in-process/local Kubo dependency or route it to an execution environment that provisions one, and investigate the Yamux ref-count and Gossip ordering/load flakes with reliable stress reproducers rather than retries or timeout changes.

Production verification — 2026-07-17

Status: STILL EXISTS. A live ProductionHealth URL connection succeeded but emitted repeated NothingToCompleteException failures while forwarding gossip; the last HardwareControlFabric daemon log also contains Netty ByteBuf leak reports in libp2p negotiation paths.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.