UrlResolver PR remote buildtest run ea9f6ee1 failed testNetlabDynamicFailureRecovery for an

Reported (UTC): 2026-07-13 09:13

UrlResolver PR remote buildtest run ea9f6ee1 failed testNetlabDynamicFailureRecovery for an infrastructure/classpath-packaging reason unrelated to the branch: every provisioned NetLab serve process crashed before readiness with 'java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolMessageEnum', caused by ClassNotFoundException, while constructing io.libp2p.crypto.keys.Ed25519PrivateKey. The test uploaded 78 classpath JARs (12 split into <=1MB parts), but protobuf runtime was missing inside the applied environment. Exact user-facing failure: Host 'bootstrap' did not report '[harness] serve ready' within 360000ms; last marker was joining-network. Impact: kotlin.build reported 1098 passed / 1 failed after 676055ms even though the same post-rebase cold local suite passed 1099/1099, including this NetLab test. Workaround: re-request the failed remote suite after recording the failure. Durable fix: make the NetLab test classpath manifest include and verify the protobuf runtime transitively required by jvm-libp2p before applying the topology, and add a preflight that reports missing runtime classes immediately instead of waiting six minutes for readiness.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.