Root-caused as shared hosted-NetLab admission/capacity loss: unbounded overlapping suites made manager-to-worker streams unwritable past delivery gate; exact EOF chain verified. Needs user approval for one cross-CI suite lease before measured finite environment cap.
OBSERVED: The handoff snapshot reports two failed attempts of the same GitHub Actions run, with rotating NetLab victims and clean EOFs before the four-byte persistent-RPC response header. Live GitHub state confirms the UrlResolver PRs remain open. The latest UrlResolver PR 885 GitHub Actions run passed, while its referenced remote build run failed. The production handoff lookup itself emitted a NoSuchElementException while jvm-libp2p tried to remove io.libp2p.multistream.Negotiator$ResponderHandler, then completed successfully.
INFER: The handoff lookup stack trace is evidence of a live multistream pipeline race, but it does not yet establish that this race caused the historical NetLab failures. The rotating victims make a test-specific assertion problem unlikely.
OBSERVED: Full GitHub Actions logs were downloaded for UrlResolver runs 30449647043 (both attempts), 30448540760, and 30451287672. The following failures were extracted:
| Run / attempt | Test and isolated environment | Phase | Recorded transport state | Result |
| --- | --- | --- | --- | --- |
| 30449647043 / 1 | testNetlabLinkFlapAndProviderMigration, url-r4dyn-1991892323104, worker e380f8cc… | Worker classpath upload | Pending persistent RPCs closed; one request write raised java.io.IOException: Stream closed; later manager upload calls timed out | Failed during upload |
| 30449647043 / 1 | testNetlabDynamicFailureRecovery, url-dyn-2531026033526, worker 3027e281… | Worker classpath upload | Pending persistent RPCs closed; subsequent manager upload calls timed out | Failed during upload |
| 30449647043 / 2 | testNetlabP2PIntegration, url-p2p-767534416134, worker 39a5290a… | Manager applyAsync | Manager connection read clean EOF before its four-byte response header: Stream closed while reading message data (read 0 of 4 bytes) | Failed before topology creation |
| 30449647043 / 2 | testNetlabLinkFlapAndProviderMigration, url-r4dyn-1729991308965, worker a813a493… | Manager get/applyStatus, then exec | Repeated clean EOF before response header. After an ambiguous exec result, repeating the route-add command returned exit code 2 | Failed after topology changes |
| 30448540760 | testNetlabLinkFlapAndProviderMigration, url-r4dyn-2000981223107, worker 74b418c5… | Worker upload, later first link-sever command | Upload RPCs closed but upload completed; route command later returned exit code 2 | Failed during link-sever phase |
| 30451287672 | testNetlabSevereImpairmentScenarios, url-sev-1244105740294, worker e4224bc8… | Worker classpath upload | Worker upload RPC closed; later manager upload attempts timed out | Failed during upload |
OBSERVED: The runs overlapped in wall-clock time and used distinct test environments and fresh workers. GitHub logs do not record TCP FIN/RST state; the strongest socket-level statement supported by them is that the application reader received EOF before any response-length bytes. The repeated route command returning exit code 2 after an EOF is consistent with the first command having executed before its response was lost, but the logs do not prove that ordering. The current main branch pins foundation.url:protocol:0.0.409, community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-19, and Netty 4.2.10.Final.
INFER: A GitHub runner port collision is not consistent with distinct workers all reaching the shared service before failure. The shared boundary is url://netlab-hosted/ and its manager-to-worker P2P paths. The evidence does not yet distinguish a manager/service resource event from a product race shared by those paths.
OBSERVED: The host currently has 8 cores, load averages above 52, 1.8 GiB available memory, and 4.2 GiB swap in use. This exceeds the lane’s local-build limit, so no local JVM has been started. Read-only artifact and source analysis continues while the host is busy.
INFER: Starting a local concurrency stress run now would add contention and would not produce trustworthy timing evidence.
OBSERVED: Buildtest run 3dc1cacd reported one failure out of 1,232 tests. The failed test was testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork, which deadlocked in InMemoryNetworkFabric.runUntilIdle after a deliberately partitioned in-memory RPC stream and hit its 30-second test limit. Querying all three bounded test-result pages found no hosted NetLab EOF; the only “stream already closed” text belonged to that deliberate in-memory partition.
INFER: The handoff’s description of 3dc1cacd as corroborating a broad hosted-NetLab transport-loss period is not supported by the retained artifact. It is a separate product-test deadlock and is outside this handoff’s question.
OBSERVED: Fresh checkouts of NetLabManagerServer, NetLabWorkerServer, and the CodexCoder21 jvm-libp2p fork were read. NetLab manager main uses one cached raw PersistentRpcConnection per trusted worker and pins resolver 0.0.670, protocol 0.0.401, jvm-libp2p snapshot-19, and Netty 4.2.10.Final. Snapshot-19 includes the snapshot-15 idempotent multistream handler-removal change. Its current Negotiator.kt checks that each handler remains in the pipeline before removing it. The NoSuchElementException observed from the handoff service referenced an older line and attempted unconditional removal, so it demonstrates an older deployed service but not the NetLab manager’s historical stream termination.
INFER: The live exception from url://handoff/ cannot be used as the mechanism for the NetLab failures without evidence that the NetLab manager or affected workers ran the older negotiator implementation.
OBSERVED: The HardwareControlFabric certificate directory is absent, so its read client cannot authenticate. The ContainerNursery read API shows url://netlab-hosted/ is a running danger_jarfile container at port 33959 with a 384 MiB memory limit, SerialGC, 128 MiB direct-memory cap, 30-minute idle environment lease, and 24-hour keep-warm. Its route uses /root/ContainerNursery/apps/netlab-manager-server.jar and /root/ContainerNursery/apps/netlab-worker-server.jar.
OBSERVED: ContainerNursery’s CLI ignored historical log filters. A request with --since 1785324600000 --until 1785335400000 --lines 20 returned the latest 1,000 lines beginning at 16:14 UTC and labeled the result 1000/332851 lines. The current latest window shows the same class of manager-to-worker persistent-RPC close still occurring under active traffic, with NETLAB-FORWARD-DIAG stacks rooted in PersistentRpcConnection.doSendRequest, but it does not include the target 11:30–14:30 window.
INFER: The fault remains observable after the earlier CI runs, and therefore was not limited to one GitHub runner. The ContainerNursery CLI filtering problem requires the lane’s permitted read-only SSH fallback to inspect the retained historical log. The report-challenge tool was not invoked because it automatically merges a PR, which this lane explicitly prohibits.
OBSERVED: Read-only SSH inspection found the target worker IDs in the retained 61 MiB manager stdout log /root/ContainerNursery/data/container-logs/netlab-manager-server-43788c5e.stdout.20260729153338-20260729153339.log. The manager recorded repeated Persistent RPC connection ... closed while requests were still pending events for all six workers in the cross-run table. For workers e380f8cc… and 3027e281…, the immediately preceding owning-side exception was RPC write dropped: stream/connection unwritable after 200ms backpressure wait (peer not reading); failing fast rather than hanging the RPC indefinitely. The current UrlProtocol implementation uses a five-second awaited-write gate on this path even though that error text still interpolates the separate 200 ms best-effort constant, so the historical message proves an unwritable-stream gate fired but does not establish a 200 ms duration.
OBSERVED: In the historical line interval containing the target runs, the singleton manager log names 206 distinct netlab-worker-… peer IDs and 196 distinct environment names over time. Requests for many environments are interleaved, including concurrent bulk classpath uploads. Replaying the log's create and successful-delete events gives a conservative lower bound of 45 simultaneously live environments in the target interval (43 still live at its end), and a lower-bound peak of 70 over the retained process log. It is a lower bound because environments restored at process start have no create event in this file. At the explicit unwritable-write records for target workers e380f8cc… and 3027e281…, the derived live counts were 29 and 21. The target interval contains 779 request-handler errors, eight log occurrences of the explicit unwritable-write text, and 2,661 records that a persistent RPC connection closed while requests were still pending. The deployed ContainerNursery route does not set MAX_LIVE_ENVIRONMENTS; manager main therefore admits an unbounded number. Git history shows that commit 12d352c1bd978bb321677cdc3ea523625e28bb49 deliberately changed the default from six to unbounded on 2026-07-23. Its removed comment records that a 2026-07-04 four-build CI storm had exhausted the manager at roughly eight or more live environments under the previous sandboxed-worker implementation. The replacement assumed one direct RPC connection per worker was cheap enough for unbounded admission.
OBSERVED: The manager has a 384 MiB Java heap, SerialGC, a 128 MiB direct-memory cap, 16 shared libp2p NIO event-loop threads, and 16 provisioning threads. Kernel logs for the target interval contain no OOM kill, conntrack exhaustion, or hung-task record. The retained manager stderr does not contain an OOM, multistream handler-removal exception, or Yamux buffer exception in the target window. A current read-only jstack during recurring stream-close traffic found all 16 libp2p event-loop threads alive in epoll rather than a stable deadlock; it also showed heavy connection churn, with JVM thread sequence IDs exceeding 21,000 in roughly 48 minutes. Historical CPU/run-queue telemetry is unavailable because sysstat was not installed.
INFER: The mechanism supported by the converging evidence is an admission/capacity defect at the shared NetLab service boundary, not a test-specific UrlResolver assertion or GitHub-runner port collision. Unbounded admission put at least 45 simultaneously live environments, and 196 distinct environments over the target interval, through one small manager. Under that burst, manager-to-worker streams stopped accepting writes for longer than UrlProtocol's awaited-write gate; the gate failed the write and retired the persistent connection, while the other endpoint observed clean EOF before the four-byte response header. Which worker was not draining at the instant of the gate rotated, so the failed test rotated. This states the causation at the observed layer; the retained data cannot further divide the transient stall between the manager's shared event-loop/heap budget and individual workers' receive loops.
INFER: The route-add exit code 2 is secondary evidence, not the root mechanism: the first exec can reach the worker and lose its response when the stream is retired, after which a repeated non-idempotent route add reports that the route already exists. The logs do not prove that ordering for every occurrence, so the exit-code result is not used to classify the primary fault.
OBSERVED: The local host remains above the lane's build threshold (load 43.77 on eight cores, with 3.9 GiB swap used). No local JVM or artificial load was started. A faithful local reproduction would require many genuine manager clients and real libp2p worker endpoints on the exact upload path; running that stress experiment on the already-contended shared host would violate the resource rule and confound the timing result.
INFER: This investigation therefore relies on the question's allowed alternative of converging forensic evidence, not a claimed local reproducer. It is not enough evidence to change UrlProtocol or UrlResolver transport code test-first, and no such product change should be guessed.
The mechanism is: the shared hosted NetLab manager admits concurrent test environments without a capacity boundary; overlapping full UrlResolver suites drove at least 45 live environments and simultaneous bulk uploads through one 384 MiB manager, making individual manager-to-worker libp2p streams remain unwritable past the awaited-delivery window. UrlProtocol then fails that write and retires the persistent RPC transport, so the peer's pending reader receives EOF before the response header.
The evidence that demonstrates the mechanism is: (1) every rotating victim resolves to a different worker behind the same url://netlab-hosted/ manager; (2) the manager's owning-side logs for the exact worker IDs show explicit stream/connection unwritable ... peer not reading write failures immediately associated with pending-connection retirement; (3) the client-side logs show the matching EOF before 0 of 4 response-header bytes; (4) a create/delete replay establishes at least 45 live environments in the target interval and 70 elsewhere in the same process, while the route and source prove admission was unbounded; and (5) neither the manager/kernel logs nor the current dependency source show an OOM kill, runner port collision, fixed multistream handler race, or stable event-loop deadlock that better explains the rotating failures.
Classification: this is a shared NetLab admission/capacity problem, expressed at the manager-to-worker persistent-RPC write boundary. It is not evidence for a new UrlResolver or UrlProtocol recovery defect: the transport detects the failed delivery, refuses unsafe replay, and closes pending calls as designed. Confidence is high for the write-failure-to-EOF chain and moderate-to-high for shared admission as the cause of the unwritable interval. The retained evidence cannot identify whether heap/GC scheduling, direct-memory pressure, or event-loop scheduling contributed the largest share because historical resource telemetry and worker-side logs are absent.
3dc1cacd did not show a broad hosted-NetLab transport-loss period. Its only failure was an in-memory partition test deadlock.Immediate containment should admit only one hosted NetLab full-suite workload at a time across GitHub Actions and kotlin.build. This must be a shared lease understood by both CI systems; a GitHub-only concurrency group would leave remote builds competing with it. The latest GitHub Actions run for https://github.com/CodexCoder21Organization/UrlResolver/pull/885 completed 1,232/1,232 while the failing runs were concentrated in overlapping suites, so one suite is the smallest evidence-backed starting point.
After containment, run one isolated full suite while recording live-environment count, concurrent upload count, heap/GC pauses, direct-memory use, and libp2p event-loop delay. Use that measurement to set the already-supported MAX_LIVE_ENVIRONMENTS route value with margin. Do not guess the number from the old value of six: https://github.com/CodexCoder21Organization/NetLabManagerServer/pull/112 proved six could leave the seventh test queued for 14 minutes and consume its readiness budget. A finite per-environment cap without CI-level admission would exchange stream loss for predictable readiness failures.
Longer term, the manager should expose those capacity signals and reject or queue an entire build lease before its tests start. This avoids partially admitting a suite whose later environments cannot become ready within their caller budget.
The Testing Architecture guidance was read before considering a reproducer. No test or implementation was changed. A compliant local reproduction would need genuine concurrent callers driving the public hosted-manager upload path through real libp2p worker endpoints; the shared host stayed above 4x its eight-core count and another lane already owned the local build JVM, so starting that experiment would have violated the lane resource rules and made its timing result unreliable. The cross-run table covers every retained target failure, both sides of the stream where available, distinct phases, and negative checks for kernel OOM, conntrack, multistream, and stable event-loop failure. The remaining evidence gap is the absence of historical resource metrics and worker-side logs. Therefore the report uses the permitted converging-forensics result and does not claim the statistical confidence of a reproducer.
I challenged the shared-capacity conclusion against four alternatives. A runner-local collision does not fit distinct fresh workers and continued service-side failures. The older multistream exception does not fit the manager's pinned guarded implementation and does not appear in its logs. A persistent-connection recovery defect does not fit the owning-side explicit unwritable-write error followed by deliberate ambiguous-outcome handling. A single bad worker does not fit rotating worker IDs. The strongest remaining alternative is a transient receive-loop stall inside each affected worker rather than shared manager scheduling; that could be the immediate source of peer not reading, but the correlated overlapping workload and common unbounded manager remain the shared condition. The review also found that the logged 200ms duration is inaccurate for awaited RPC writes, whose source uses 5,000 ms, so no duration claim relies on that text.
No code PR was created. Writing a transport change without a reliable product reproducer would violate the fail-first rule, while the immediate remediation changes shared CI scheduling and production route policy and requires user approval. No production state was modified.
The handoff's investigation question is answered with a six-row cross-run table, owning-side production evidence, a conservative live-environment reconstruction, source/version checks, and explicit confidence limits. The evidence checkpoint was rebased onto current PlanRepository main and pushed at https://github.com/CodexCoder21Organization/PlanRepository/commit/966ce8c53a3254b23304fa35a7f648ccd45b6099 on branch https://github.com/CodexCoder21Organization/PlanRepository/tree/evidence/hf-persistent-rpc-stream-loss-w19. The branch has no required CI because it is an unmerged lane artifact, and its worktree is clean.
The defect is ready for an operational decision; there is no code branch requiring CI and no PR to merge from this lane. Existing https://github.com/CodexCoder21Organization/UrlResolver/pull/873 remains open with its historical GitHub Actions failure, and https://github.com/CodexCoder21Organization/UrlResolver/pull/885 has a green 1,232/1,232 GitHub Actions run but an unrelated red remote run caused by the in-memory partition-test deadlock.
NEEDS-USER: Should the shared CI scheduler enforce one hosted NetLab full-suite lease at a time across GitHub Actions and kotlin.build? / EVIDENCE: overlapping suites drove at least 45 simultaneous environments through the unbounded singleton manager, whose logs show exact worker streams becoming unwritable and being retired; a later GitHub Actions run completed 1,232/1,232. / PROPOSED ANSWER: yes; enable the cross-system build-level lease first, then measure one isolated suite and set a finite MAX_LIVE_ENVIRONMENTS value with margin. Do not deploy or reroute until that policy is approved.
The supervisor must review the mechanism/classification, the inference boundary around the exact resource that caused peer not reading, and the proposed cross-CI lease. There is nothing to merge from this investigation. If the operational proposal is approved, the supervisor should dispatch a separate implementation lane for the cross-system lease and a separately approved ContainerNursery route update after measurement.