Handoff: UrlResolver main red → netlab resume-on-restart fix (#69) → kotlin-build-ci upload outage → #664 green → #651 blocked by netlab-under-load

Written: doc dated 2026-06-28 (the thread's first message). Actually authored 2026-07-05 after re-verifying live state. Every state snapshot below was verified at authoring time — RE-VERIFY before acting. gh pr view <num> --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative; never trust this doc's snapshot over a fresh query. Host/service forensics go stale in minutes.

Heavy overlap with the 2026-07-05 handoffs — read those; they SUPERSEDE parts of this one. This thread independently walked into the same org-wide disease (the whole control plane on one 2-core/3.9GB droplet 198.199.106.165 swapping under load; SJVM RPC handlers stalling; buildtest OOM-crash-looping). The netlab piece this thread diagnosed as "a slow leak degrading load-tolerance" was later root-caused and fixed by parallel work — see especially:

• 2026-07-05-buildtest-server-service-heap-leak-oom-crashloop-and-netlab-sharing-verified.md (the current #651 blocker),
• 2026-07-05-netlab-manager-oom-uploadfile-materialization-root-cause.md / 2026-07-05-netlab-manager-oom-sjvm-sandbox-leak.md (the netlab OOM root cause + fix),
• 2026-07-05-ci-stabilization-and-fleet-redeploy.md (the host-wide fix).

---

Mission summary

Original request (verbatim): "I don't understand why https://github.com/CodexCoder21Organization/UrlResolver/ main branch is red, we need to fix this."

It expanded, in order, into:

1. Diagnose + fix UrlResolver main being red.
2. "Ensure UrlResolver #664 goes green."
3. "Rebase UrlResolver #651, ensure it goes green, and merge it" (merge of #651 was explicitly user-approved).

---

What was found and done (the chain)

1. Why main was red → netlab flake → NetLabManagerServer #69 (MERGED + deployed + verified)

main (then 6ab127cab0, the #656 merge) failed kotlin.build (remote) 911 passed / 1 failed — the one failure was testNetlabP2PIntegration, which acquires a live env from url://netlab-hosted/ (15-min budget) and timed out with "could not acquire a READY netlab environment." Not a UrlResolver code bug.

Root cause was upstream in NetLabManagerServer: the on-demand manager runs under ContainerNursery, which restarts it (idle eviction / redeploy / OOM). If a restart lands mid-provision, the provisioning coroutine dies and the env is pinned forever at e.g. UPLOADING_WORKER (reconcile keeps it because its droplet still exists) → the test's 15-min acquire budget expires → red.

Fix: NetLabManagerServer #69 — manager 0.0.47, ProvisioningService.resumeInterruptedProvisions() on startup (re-drives recent interrupted provisions on a fresh droplet; fails+cleans ones older than the 15-min window). MERGED (2026-06-29, merge commit dd620540). Deployed + verified live via upload-jar --route url:netlab-hosted: (writes to the route's image path /root/ContainerNursery/apps/netlab-manager-server.jar, restarts) — boot log showed "Recovering environments interrupted mid-provision by a restart… resuming interrupted provision for 'url-dyn-…' (WAITING_FOR_IP) re-driving on a fresh droplet."

2. Deeper blocker: kotlin-build-ci upload outage (org-wide CI down) — fixed by the user redeploying kotlin-build-ci

While executing #1, all CI was wedged: every repo's builds stuck PENDING with no droplet for hours. Root cause (traced to the client side, https:kotlin.build:443, not the buildtest server whose logs looked clean):

java.lang.NoClassDefFoundError: foundation/url/protocol/Libp2pRpcProtocol$RpcError
  at Libp2pRpcProtocol$RpcResponse$Companion.fromJson(Libp2pRpcProtocol.kt:150)

The deployed kotlin-build-ci.jar had an incomplete/version-skewed protocol library — RpcResponse.fromJson referenced sibling class $RpcError which was missing from the jar, so every build's first uploadChunk response parse threw → uploads abandoned after chunk 0 → builds never provisioned. The user redeployed kotlin-build-ci (~01:56), clearing it; new builds provisioned again. (Diagnostic detail saved in auto-memory kotlin_build_ci_classnotfound_rpcerror_upload_outage.md.)

3. #664 — MERGED green. After buildtest recovered, re-triggered kotlin.build on main (6ab127cab0) via the check-run rerequest API → 912/912 green, resolving the original red. UrlResolver #664 (the "sandbox client sends a Map-nested raw ByteArray rpc param" regression test) was then rebased onto that green main and MERGED (2026-06-30, 917/917). Its netlab tests passed once it caught a window with a fresh netlab-hosted.

4. #651 — still NOT green/merged. This is the open task. UrlResolver #651 — "Add regression test: persistent connection recovers from in-flight stream close" — a test-only PR (tests/testPersistentConnectionRecoversFromInFlightStreamClose.kts). Its own test passes every run; it keeps failing on unrelated flaky netlab + resolver-stress tests.

Attempts this thread made (all failed, none merged):

• Confirmed the branch was already 0-behind main (rebase = no-op); its red was a transient "Build infrastructure error during resume" (buildtest orchestrator), not a test result.
• /retry (twice), a --force-with-lease re-created commit (a5a81845 → bbcd43af, author preserved), and a quiet-window retry (fired when only 1 other build was active). 4 runs, all failed on shifting flaky tests (testNetlabP2PIntegration, testNetlabDynamicFailureRecovery, testNetlabLargeSwarmGossip, testNetlabResolutionUnderLatencyAndPacketLoss, testNetlabDiscoveryScenarios, plus resolver stress tests stressTestHighConcurrencyRegistrationStormDoesNotLeakConnections, stressTestWaitForRelayRegistrationReturnsImmediately).
• Live diagnosis at the time: during #651's runs, url://netlab-hosted/ filled with WorkerClient.forwardRequest errors (~650 errors/250 log lines at peak; ~11/500 when idle) — a single build's ~5–6 concurrent netlab environments saturated the one -Xmx128m manager. main/#664 passed the same tests on a fresher manager, so this thread hypothesized "a slow leak lowering concurrency tolerance over uptime."
• Tried restart --route-key url:netlab-hosted: → CLI returned "Unknown response format", and the manager was never idle enough to safely restart (continuous CI netlab traffic).

This hypothesis was CORRECT in spirit and has since been root-caused + fixed by parallel work. The netlab-manager OOM/SJVM-sandbox-leak was fixed by NetLabManagerServer #84 (MAX_LIVE_ENVIRONMENTS admission cap — user-approved stopgap), #85 (lightweight SJVM sandbox), and UrlResolver #669 (free the SJVM sandbox on close, resolver 0.0.591) — all MERGED, deployed to netlab-hosted ~07-04. The 07-05 handoff verified "the netlab OOM is resolved." #651's remaining blocker is a DIFFERENT service: the buildtest-server-service heap-leak OOM-crash-loop (a BuildTestEmbedded leak) — the getBuildRun / "infrastructure error during resume" failures. See 2026-07-05-buildtest-server-service-heap-leak-oom-crashloop-and-netlab-sharing-verified.md.

---

Relevant PRs / refs (verified at authoring time — RE-VERIFY)

| PR / ref | State | Notes | |---|---|---| | NetLabManagerServer #69 | MERGED (06-29, dd620540) | manager 0.0.47 resumeInterruptedProvisions() — fix for the original red. Deployed + verified. | | UrlResolver #664 | MERGED (06-30, 917/917) | Map-nested ByteArray marshaling regression test. The "ensure it goes green" ask — done. | | UrlResolver #651 | OPEN, BLOCKED | The open task. Branch test-persistent-conn-recovers-inflight-stream-close, head a5720db1 (re-pushed by parallel work; my bbcd43af superseded — identical test content). Test-only. kotlin.build (remote) = FAILURE ("infrastructure error during resume" = buildtest-server-service OOM). Merge is user-approved. | | NetLabManagerServer #84 | MERGED + deployed | MAX_LIVE_ENVIRONMENTS cap — netlab OOM stopgap (fixes the load-tolerance issue this thread diagnosed). | | NetLabManagerServer #85 | MERGED | lightweight SJVM sandbox (further per-env memory reduction). | | UrlResolver #669 | MERGED (07-03) | free SJVM sandbox on close (resolver 0.0.591) — the durable netlab-manager leak fix. | | UrlResolver main | HEAD afefcb13 (#690) | kotlin.build (remote) RED at authoring time — the buildtest-server-service infra issue affects main too (not #651-specific). |

No uncommitted/unsaved local changes exist to upload. /code/UrlResolver (#651 branch) was 0-behind and 0-unpushed; my re-created commit bbcd43af was later superseded on origin by a5720db1 (same test content, no lost work). /code/NetLabManagerServer (#69, merged) had only build artifacts (fatjar-build.log, build-fats/), which were deleted, not committed. All meaningful work is already on the linked branches/PRs.

---

Next steps (to drive #651 to green + merged)

#651's code is fine — do NOT change the test or paper over failures. The blocker is infrastructure. Concretely:

1. RE-VERIFY first. gh pr view 651 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,headRefOid,mergedAt and the kotlin.build (remote) conclusion on its head. It may already be green/merged via parallel work.
2. Unblock the real dependency: the buildtest-server-service heap-leak OOM-crash-loop. Per 2026-07-05-buildtest-server-service-heap-leak-oom-crashloop-and-netlab-sharing-verified.md, buildtest-server-service OOM-crash-loops every ~15–40 min even with host RAM free (an application-level leak in BuildTestEmbedded retaining build/test data at -Xmx128m). Until that's fixed/mitigated, kotlin.build (remote) intermittently fails with "getBuildRun … infrastructure error during resume" for all PRs including #651 and main. Follow that handoff's diagnosis/fix plan (it is the highest-leverage blocker).
3. Confirm the netlab OOM fix is holding (it was verified resolved 07-05 via #84/#85/#669). If netlab tests still flake specifically, re-open the netlab angle — but the 07-05 evidence says netlab is no longer the cause.
4. Once buildtest CI is stable, re-trigger #651 (gh pr comment 651 --body "/retry" — the app handles /retry on PR issue_comments; a plain branch push does NOT trigger the remote check, only /retry or the merge-queue does). Watch kotlin.build (remote) to a genuine SUCCESS.
5. Merge #651 (user-approved): repo uses a merge queue with repo-level auto-merge OFF → enqueue via GraphQL enqueuePullRequest(input:{pullRequestId}) (get node id from gh pr view 651 --json id); gh pr merge --auto is rejected ("Auto merge is not allowed"). Monitor state until MERGED. Do NOT use --delete-branch.

Operational knowledge learned this thread (reusable)

• CN CLI route keys are facade-prefixed (from containers): url:netlab-hosted:, url:buildtest:, https:kotlin.build:443. container --route-key returns "Endpoint not found" for an idle container even with the right key — use container-logs / containers.
• upload-jar --route <key> writes to the route's configured image path AND restarts — used for the in-place netlab deploy (zero route-config change).
• Re-trigger kotlin.build on a MAIN commit (not a PR): gh api --method POST repos/<o>/<r>/check-runs/<id>/rerequest → the kotlin-build-ci-test app handles check_run.rerequested.
• HardwareControlFabric daemon was DOWN through this thread (ping → "Daemon did not respond") — all ops went CN-CLI-only.
• Server management order: CN CLI (preferred) → HCF → SSH (ssh -p 23 root@198.199.106.165, last resort).

🤖 Generated with Claude Code