Handoff: UrlResolver #656 cleanup (PR #659) + the netlab-hosted OOM it uncovered

Written: 2026-07-05. UPDATED 2026-07-12 ~16:45 UTC (§2/§3 rewritten — the code is done and pushed; the ONLY blocker is buildtest infrastructure, not the PR). Every state snapshot below was verified at update time — RE-VERIFY before acting. PRs merge, queues shuffle, prod restarts. gh pr view <num> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative; never trust this doc over a fresh query.

TL;DR for the taker (2026-07-12). UrlResolver #659 is rebased onto main, verified locally, and pushed at head 48585009. bld-build (full GitHub Actions suite) is GREEN. The code is proven good. The only thing left is one clean pass of the kotlin.build (remote) lane, which was blocked all night 2026-07-11→12 by buildtest infrastructure on 198.199.106.165 (three buildtest-server backend deaths + a DigitalOcean droplet rate-limit exhaustion), never by this PR's diff. Merge is PRE-APPROVED. Jump to §3 for the exact next steps. Infra failures are documented in challenge 2026-07-12-0248 and /root/cn-watchdog.audit.

Overlap / cross-reference. The deep netlab-hosted OOM investigation that grew out of this task has since been driven much further by other sessions and is documented in handoffs/NetLab-workstream-handoff-2026-07-05.md (and the auto-memory netlab-hosted-oom-root-cause-workerclient-sjvm-proxy-leak.md). That handoff is the authority on the OOM fix and its deploy state. This doc's own scope is the small original cleanup — UrlResolver PR #659 — which is still OPEN and is the one clearly-unfinished item. The OOM material here is context so you understand why #659's CI was blocked and what unblocked it. As of 2026-07-12 the netlab OOM fix is on main and the netlab tests pass on the remote lane (see §2).

---

1. Original request (verbatim intent)

"Consider UrlResolver PR #656. I got reports of some problems that we should probably clean up: (1) System.err.println debug diagnostics left in production source — [PEX-OK], [PEX-EMPTY] (×2), which violates the convention of routing warnings through algebraic effects (toss) / debugLog, not direct printing. (2) A JVM-global mutable counter added to production source for a test to read: val servicelessPeerVerificationCount: AtomicLong (public, top-level), whose KDoc openly says 'a regression test asserts that' — production API shaped around a test."

Later, after CI on that cleanup went red on the testNetlab* tests, the user directed: "You need to understand and fix the underlying problem" (the netlab failures), then "Ensure both go green … ensure they get merged." That expanded the task into the netlab-hosted OOM investigation (§4).

---

2. The cleanup itself — UrlResolver PR #659 (OPEN; code DONE, only remote-CI green outstanding)

UrlResolver #659 — branch cleanup-pex-debug-prints-and-test-shaped-counter, head 48585009 (2026-07-12: rebased onto current main, verified, force-pushed; the old 6a64393 in the original write-up is superseded). Title: "Clean up #656: gate [PEX-] debug prints; drop test-shaped global counter."*

What it does (both original issues, code-complete + verified):

1. [PEX-OK] / [PEX-EMPTY] (×2) System.err.println → gated debugLogErr {} in src/foundation/url/resolver/UrlResolver.kt. debugLogErr is the file's pre-existing convention, gated on -Dfoundation.url.resolver.debug=true. String-building moved inside the gated lambda; stale "we do NOT silently swallow" comments corrected. On the 2026-07-12 rebase, main had already independently gated the [PEX-OK] site and refactored the exchange path into ensureExchangeConnection(...); the rebase kept main's newer code and re-applied only the one still-relevant change — gating the remaining unconditional [PEX-EMPTY] System.err.println in the if (isConfiguredBootstrap) block after the do/while loop. Verified afterward: no ungated [PEX-*] stderr print remains anywhere in the peer-exchange region (grep -n 'System.err.println' src/foundation/url/resolver/UrlResolver.kt in ~lines 1300–1700 is clean).
   • The user once challenged using debugLogErr instead of algebraic effects; it was empirically settled (throwaway experiment, since deleted): these PEX sites run on detached background coroutines (coroutineScope.launch, the periodic peer-exchange job) where (a) an effect tossed from such a coroutine is NOT caught by an outer Effective { handler {…} }, and (b) an unhandled toss(NotificationEffect) prints to stderr unconditionally — reintroducing #656's problem. debugLogErr (gated, silent by default) is correct for these sites. If a reviewer re-raises this, that's the answer — don't re-litigate.
2. Removed the JVM-global val servicelessPeerVerificationCount: AtomicLong and its increment. The regression it guarded (a SERVICE-LESS peer must be routed to direct registration, never the bounded/blocking verification queue) is re-grounded on observable public behaviour via new test tests/testOnPeersReceivedRegistersServicelessPeerWithoutVerification.kts (feeds a service-less peer at an unreachable address; asserts it becomes known via public getKnownPeers() without a reachability dial). On the rebase its @WithArtifact pins were bumped to main's current versions (e.g. libSJVM 0.0.40, observable-core 0.3.13). TDD both-ways re-confirmed on the rebased head: removing the routing guard (&& filteredServices.isNotEmpty()) makes it fail; restoring passes. The old counter-only stressTestServicelessPeerVerificationDoesNotStarveDiscovery.kts is deleted.

Verification done on head 48585009 (2026-07-12): cache-purged scripts/build.bash foundation.url.resolver.buildMaven ✅ (244.5 s); the new test passes; TDD both-ways confirmed (purge ~/.aibuildcaches/_code_workspace_UrlResolver + ~/.cache/coursier/v1/bldbinary between runs — the stale-cache trap gives false PASSes otherwise).

CI state — freshly verified 2026-07-12 ~16:45 UTC (RE-VERIFY):

• bld-build (GitHub Actions, full suite incl. netlab): PASS / COMPLETED SUCCESS on head 48585009. This is the strong signal the code is good — the whole suite, including the netlab tests that originally blocked this PR, passes.
• kotlin.build (remote): not yet green, entirely due to buildtest infrastructure, not the diff. The one remote run that actually ran to completion on this head (run a039164c, ~02:11 UTC) was 1082 passed / 2 failed / 1084 total — the 2 failures being testReconnectSucceedsViaGossipPropagatedServiceInfo (known gossip-propagation flake; its own error text shows the service was discovered — a racing assertion) and testNetlabDynamicFailureRecovery (netlab bootstrap container never reached serve-ready in 360 s). Every other remote attempt died to infra before finishing (see below). The current check-run (31de760a, id 86612631227) has been stuck IN_PROGRESS since 03:48 UTC and is orphaned — a buildtest-server death left it hung ~13 h; its test-events.jsonl is truncated/corrupt. It must be re-requested (§3).

The night's infrastructure saga (why the remote lane never went green) — all VERIFIED, all infra, none is the PR

• ~00:06 UTC buildtest-server backend died (disk-100% incident) → orphaned run a607aabf silently (check stuck IN_PROGRESS, run page renders only a header).
• ~02:11 UTC run a039164c completed 1082/1084 (the 2 known flakes above).
• ~02:37 UTC buildtest-server died again mid-poll → re-fire 6b043a57 concluded as "Build infrastructure error: SandboxException: SJVM virtual method invocation timed out after 120s: BuildTestServiceClientImpl.getTestResults".
• ~02:47 UTC re-fire 0c526413 insta-failed: all 10 shard createBuildDroplet calls exhausted the DigitalOcean account write rate-limit queue (thundering herd after the restart).
• ~03:06 UTC re-fire 140f3161 insta-failed: DO account quota fully exhausted, ratelimit-remaining=0, provider reset 03:46:41Z.
• ~03:48 UTC re-fire 31de760a launched after the reset; provisioned and ran to ~70%+, then a third apparent backend death during the multi-hour session gap left it orphaned IN_PROGRESS (current state).
• Root cause of the deaths is an undiagnosed OOM-looping ContainerNursery child regrowing heapdumps (5.8 G) + accumulated buildtest run dirs (5.1 G) filling the host disk — owned by other lanes (CN issue 498 / claude-observables-cascade-session per /root/cn-watchdog.audit). Full write-up: challenge 2026-07-12-0248.

---

3. Next steps to drive #659 to completion (this is the actionable part)

The code is done. This is now purely a get-one-clean-remote-CI-pass-then-merge task.

1. Re-request the kotlin.build (remote) check-run (the orphaned 31de760a will never conclude):

   CR_ID=$(gh api repos/CodexCoder21Organization/UrlResolver/commits/48585009b8da7442bd1180d2ddb55da9efab5dce/check-runs \
     --jq '.check_runs[] | select(.name=="kotlin.build (remote)") | .id')
   gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-runs/$CR_ID/rerequest
   

   bld-build is already green — do not re-run it.
2. First, sanity-check the host is healthy before re-firing, or you'll just orphan another run. Ground-truth via HardwareControlFabric (do NOT trust the flapping buildtest.kotlin.build frontend, which 503s/time-outs under load): confirm buildtest-server-service.jar has a recent, stable PID (… processes --top 10), and that the DO account isn't rate-limited. Read the run's progress from /root/buildtest-data/runs/<id>/test-events.jsonl and build.log server-side, not the dashboard. HCF CLI + certs: see §6 additions.
3. Watch to green with build-watchman: cs launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 659 (run in background, act on its PROBLEM:/CHANGE: lines; its 503 alarms are degraded observability, not run death). If the remote run fails on only the known infra flakes — testReconnectSucceedsViaGossipPropagatedServiceInfo, testNetlabDynamicFailureRecovery, or the fleet-wide gossip-storm flake issue 754 — re-request the check; do NOT blame the diff. If the host is mid-outage, DO NOT blindly re-fire in a loop — back off, coordinate via /root/cn-watchdog.audit, and let the DO rate-limit window / backend restart settle first.
4. Rebase if needed. Branch is currently 1 commit behind origin/main (was BEHIND at write time). The squash merge-queue rewrites main's hashes, so if a plain git rebase origin/main replays an already-squashed commit use git rebase --onto <newmain> <oldbase> cleanup-pex-debug-prints-and-test-shaped-counter, then git push --force-with-lease. A rebase re-triggers CI; only rebase if the queue rejects for staleness, to avoid restarting the ~100-min remote run needlessly.
5. Merge #659 once both lanes are green — via the merge queue (GraphQL enqueuePullRequest; repo auto-merge is OFF). Merge is PRE-APPROVED per the original handoff. Never --admin/bypass; never merge red. Memory urlresolver-stress-test-load-flakes: enqueue UrlResolver PRs only on a drained executor.
6. Close out THIS handoff once #659 is merged: delete handoffs/2026-07-05-urlresolver-656-cleanup-659-and-netlab-oom-supersession.md via a PR and merge it (handoff-dir changes are pre-approved). Then mark tasks done.
7. Poll CI ≤ once per 60 s.

---

4. The netlab-hosted OOM (context; largely DONE by other sessions — see the NetLab handoff)

#659's testNetlab* failures (could not acquire a READY netlab environment from url://netlab-hosted/) were caused by the NetLabManagerServer manager OOM-crash-looping at -Xmx128m on prod (198.199.106.165). I investigated it forensically and shipped a manager-side eviction fix — which turned out to be treating the symptom.

CORRECTED root cause (found by later sessions, now the accepted one): the leak is resolver-side. SandboxedConnection.close() (UrlResolver src/foundation/url/resolver/sandbox/SandboxedConnection.kt) closed the RPC stream but never called sjvm.close(); each SJVM runs a single-thread "SJVM context" dispatcher thread that is a GC root, so every sandboxed connection leaked its whole SJVM. The manager opens one sandboxed TopologyService per provisioned worker → one leaked SJVM per env → 128 MB heap exhausted. Manager-side eviction (drop the map ref / call close()) can't help because it all routes through the close() that didn't free the SJVM.

The real fix (verify current state before acting):

• UrlResolver #688 — SandboxedConnection.close() now calls sjvm.close(); SandboxedProxyGenerator emits AutoCloseable proxies. At write time: OPEN (mergeStateStatus UNKNOWN). Note it may need reconciliation with the MERGED #669 (an earlier idle-autoclose approach) — the NetLab handoff says the two conflict and #688 supersedes #669; the auto-memory says #669 was closed-as-superseded but gh shows it MERGED — re-verify which is actually on main.
• NetLabManagerServer #82 — pin bump resolver 0.0.587→0.0.589 + protocol 0.0.327→0.0.328 + server 0.0.56→0.0.57. At write time: OPEN. Per the auto-memory it was DEPLOYED to prod url://netlab-hosted/ on 2026-07-03 (fabric write-file + kill→CN lazy respawn); the new manager held 6+ min under active provisioning with zero new OOM heapdumps and bounded live heap (~47 MB, SJVMImpl=1). Re-verify prod is still on the fixed jar and stable (ssh -p 23 root@198.199.106.165; check /root/ContainerNursery/heapdumps for dumps newer than the deploy).

My superseded work (for the record):

• NetLabManagerServer #71 — CLOSED (superseded). Manager-side WorkerForwarder.closeWorker + a single WorkerRegistry.remove onRemove eviction hook (more future-proof than main's per-site evict() calls) + 2 failing-first tests. It measurably reduced the prod OOM rate (frequent → ~1/18 min → ~0 for 27 min) which is what proved dereferencing helps partially — but it can't free the SJVM without the resolver sjvm.close(). All work is committed/pushed on branch fix-workerclient-sandbox-proxy-leak-oom (head 9537ad5) if any of it (e.g. the WorkerRegistry chokepoint hook, or the testWorkerRegistryRemoveEvictsWorkerProxy test) is worth salvaging on top of the resolver fix.

Next steps for the OOM thread (mostly owned by the NetLab handoff, listed for completeness):

1. Merge #688 (canonical resolver fix), reconciling with #669 if needed. Verify CI green.
2. Merge #82 (already deployed). Verify CI green.
3. Re-verify prod manager stability; re-run the netlab e2e (testNetlab* / W3WalletTests) against the deployed jar.

---

5. Uncommitted work / branches

Nothing is uncommitted (re-verified 2026-07-12). workspace/UrlResolver is clean, on branch cleanup-pex-debug-prints-and-test-shaped-counter @ 48585009, matching origin (pushed). All work is committed and pushed:

• UrlResolver #659 → branch cleanup-pex-debug-prints-and-test-shaped-counter @ 48585009 (OPEN; rebased 2026-07-12, supersedes the original 6a64393).
• NetLabManagerServer #71 → branch fix-workerclient-sandbox-proxy-leak-oom @ 9537ad5 (PR CLOSED/superseded; branch preserved).

---

6. Gotchas (all verified the hard way)

• Stale build cache = false PASS. For same-version src/ edits, kompile's buildMaven() serves the last-built jar (buildMaven() succeeded in 8ms, 0 recompiles). Purge ~/.aibuildcaches/<repo-path-with-slashes-as-underscores> AND ~/.cache/coursier/v1/bldbinary before any fail-first/green validation, or you validate against stale bytecode. This bit me twice.
• Squash merge-queue rewrites main. Force-updates mean your PR base may no longer be an ancestor; rebase with git rebase --onto <newmain> <oldbase> <branch>, and after a force-push GitHub's PR head/GH-Actions can lag minutes (an empty commit reliably re-triggers; the squash drops it).
• URL-route deploys via CN CLI: the route key for url://netlab-hosted/ is url:netlab-hosted (not netlab-hosted); upload-jar --route url:netlab-hosted writes the jar + restarts, but for a danger_jarfile it may NOT cycle the live process — kill the old PID so CN cold-starts the new jar, and verify the running process' start time is after the deploy (I hit a stale-jar-in-memory once).
• kompile test scripts allow only top-level functions — put fake classes as local classes inside the test function.
• (2026-07-12) Ground-truth buildtest runs on the host, not the dashboard. buildtest.kotlin.build 500s/times-out under load (its RPC to the backend is what the dashboard consumes). Read run state directly via HardwareControlFabric: cs launch community.kotlin.hardwarecontrolfabric:cli:0.2.6 -r https://kotlin.directory -- -H 198.199.106.165 -c ~/.config/hardware-control-fabric/fabric-client-cert.pem -k ~/.config/hardware-control-fabric/fabric-client-key.pem --ca-cert ~/.config/hardware-control-fabric/fabric-server-cert.pem read-file --remote-input-path /root/buildtest-data/runs/<id>/test-events.jsonl (and .../build.log). processes --top 10 shows whether buildtest-server-service.jar restarted (a new PID = a backend death that orphaned in-flight runs).
• (2026-07-12) HCF CLI launch --cmd takes ONE executable path — no shell/arg splitting. To append to the shared /root/cn-watchdog.audit you must read-file it, append locally, then write-file it back (small clobber race, but entries are minutes apart). Always log a lane entry there before touching the host so other sessions don't collide.
• (2026-07-12) A backend restart triggers a DigitalOcean droplet-provisioning thundering herd. Many runs re-provision at once and exhaust the DO account write rate-limit (ratelimit-remaining=0, with a provider reset=<time> in the error). Do NOT re-fire before the stated reset — every attempt just burns quota. Wait for the reset, then re-fire once.

---

7. Relevant PRs (state at write time — RE-VERIFY)

| PR | State @ write | Notes | |---|---|---| | UrlResolver #659 | OPEN @ 48585009 | The cleanup. Rebased + verified + pushed 2026-07-12; bld-build GREEN. Only remote-CI green outstanding (blocked by buildtest infra, not the diff) → merge (PRE-APPROVED). The one remaining item. | | UrlResolver #688 | OPEN | Canonical SJVM-close fix. Merge (reconcile with #669). NetLab-handoff territory. | | UrlResolver #669 | MERGED (per gh) | Earlier idle-autoclose. Conflicts with #688 per notes — verify what's on main. | | NetLabManagerServer #82 | OPEN | Pin bump to resolver 0.0.589. Deployed to prod 2026-07-03. Merge. | | NetLabManagerServer #71 | CLOSED | My superseded manager-side eviction. Branch preserved (9537ad5). | | UrlResolver #656 | MERGED | The PR whose leftovers #659 cleans up. |