Handoff: UrlResolver round-4 netlab tests (PR #632) — tests done & green locally, blocked on shared netlab-provisioning-storm CI infra

Written 2026-07-06 · ⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only when written (2026-07-06 ~20:05 UTC). Snapshots go stale: PRs merge, branches get rebased/force-pushed, CI re-runs, and the shared host's health changes hour to hour. Re-verify before acting:

• PR #632: gh pr view 632 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
• main CI: moid=$(gh api repos/CodexCoder21Organization/UrlResolver/commits/main --jq .sha); gh api repos/CodexCoder21Organization/UrlResolver/commits/$moid/check-runs --jq '.check_runs[]|select(.name=="kotlin.build (remote)")|{conclusion,summary:.output.summary}'
• Shared host health (198.199.106.165): ssh -n -p 23 root@198.199.106.165 'uptime; ls -lat --time-style=+%m-%d_%H:%M /root/ContainerNursery/heapdumps/*.hprof | head -5' — dump interval is the health proxy (minutes-apart = OOM-looping; hours-apart = comparatively healthy).
• A fresh query always wins over this document.

The date prefix 2026-06-11 is when this effort began (netlab tests + the watchman incident it uncovered); this handoff was authored 2026-07-06.

---

Mission summary

Original user request, verbatim:

"We want to have a bunch of comprehensive tests for UrlResolver repository which utilize the netlab to simulate more realistic network scenarios."

That is the whole ask. It expanded, in order, into:

1. Write the tests (the deliverable) — round-4 netlab integration tests covering degraded-network conditions no existing netlab test exercised.
2. A production-incident detour — verifying netlab health surfaced that the shared droplet 198.199.106.165 was in an OOM crash-loop (github-watchman-service-server dying every ~3 min), destabilizing ContainerNursery, the maven facade, and netlab provisioning. Fixing/helping-land that was prerequisite to getting any netlab test to run.
3. Getting the tests through CI — which hit a shared-infrastructure wall (the netlab-provisioning storm) that is the current blocker and the open decision.

The user's most recent steer (when asked how far to take the infra fix): "Check to see the current infra status, I think it may be mostly working already now." — Checked at write time: it is NOT. See "Current infra state" below.

---

What was found and done (the chain)

A. The deliverable — two new netlab tests (DONE, verified green locally)

Branch netlab-round4-impairment-chaos-tests, PR #632 (OPEN). Single commit, head 553c617c (rebased onto main on 2026-07-06). Four files changed:

• tests/testNetlabImpairedNatRelayScenarios.kts (new) — a NAT'd client resolves NAT'd providers across a relay circuit while tc netem degrades the path, in 5 phases on ONE provisioned droplet: baseline → 200 ms±40 ms + 5% loss on the provider edge → additionally 80 ms±15 ms + 2% loss on the relay hub → clear, then a 512 kbit/s rate cap with a 96 KiB relayed payload (deliberately below the legacy 128 KiB relay cap, so it exercises sustained constrained throughput, not chunking) → recovery.
• tests/testNetlabLinkFlapAndProviderMigration.kts (new) — baseline → rapid uplink flapping with a proven-breakage gate (a real logged [harness] resolve FAIL line is REQUIRED while severed — an await timeout is not accepted, so the disruption can never silently no-op) → crash-migration: the original provider is SIGTERM-killed (kill 1; container verified dead, stale announcement left behind), the URL is proven dead, then a standby provider on a different subnet with a new token registers the same url:// name via SERVE_SIGNAL; the client must reach the new provider, proven by the returned token.
• src/foundation/url/resolver/NetlabTestEnv.kt — added netem helpers setLatency / setPacketLossRate / setRateLimit / clearImpairment over the worker's netem RPCs (NetLabServiceServer ≥ 0.0.22), each asserting the tc exit code.
• src/foundation/url/resolver/NetlabHarness.kt — added BIG_RESPONSE_BYTES / EXPECT_BIG_BYTES harness mode (taken verbatim from the round-3 branch / PR #604 so the two merge trivially).

Verification (VERIFIED, observed firsthand): on 2026-06-16, after rebasing onto then-current main (protocol 0.0.303), a serialized local run passed both tests end-to-end against freshly provisioned managed netlab droplets — testNetlabImpairedNatRelayScenarios all 5 phases (including the netem RPCs executing with tc exit 0 and the rate-capped 96 KiB transfer) and testNetlabLinkFlapAndProviderMigration (flap gate + migration). foundation.url.resolver.buildMaven compiles clean. The tests themselves are correct — one of them (testNetlabImpairedNatRelayScenarios) even PASSED inside the contended CI run that failed the others.

Non-obvious bug caught and fixed during development (VERIFIED): NetlabTestEnv.awaitResolveResult(host, svc) greps the client container's whole accumulated stdout for [harness] resolve OK/FAIL service=<svc>. Reusing the same (client, serviceURL) across phases let a stale earlier OK satisfy a later phase instantly — a false pass. It was caught live by the flap-severed proven-breakage gate. Fix: every phase resolves a UNIQUE per-phase URL path (url://nlx-imp/p2-edge/ etc.); parseUrl/extractServiceIdentifier take the first path segment as the service id so paths don't affect routing, but each harness OK/FAIL line prints the full URL → collision-free markers, zero src changes needed.

B. The production-incident detour — watchman OOM crash-loop (FIXED, deployed)

While verifying netlab health, found 198.199.106.165 OOM-looping: github-watchman-service-server died every ~3 min from the known CallbackExecutor SJVM-sandbox-per-callback leak (it called openSandboxedConnection on every callback invocation, never reusing/closing — each open piles up parsed SJVM class data; 2026-06-11 heap dump showed ~118 MB of byte[] retained by JarClassPathEntry.classData). Each death wrote a ~140 MB heap dump, saturating the host and cascading into ContainerNursery / maven-facade / netlab-provisioning instability.

The fix already existed as GithubWatchmanServiceServer PR #37 ("reuse one sandboxed proxy per callback URL"); its failing CI was pure outage collateral. I verified its 3 regression tests pass locally (⚠️ gotcha: that test file holds three differently-named test functions, violating one-test-per-file — --test <filename> matches nothing; select by function name: testCallbackExecutorOpensConnectionOncePerUrl, ...OpensOneConnectionPerDistinctUrl, ...RetriesAfterFailedOpenThenReuses). PR #37 was MERGED and deployed 2026-06-11 by the owning session — that specific 2-minute crash-loop is gone (dumps went from every-2-min to hours-apart afterward). Also pruned ~17 redundant ~140 MB heap dumps (~2.3 GB reclaimed).

C. The CI blocker — netlab provisioning storm (ROOT-CAUSED, VERIFIED; unresolved)

kotlin.build (remote) is RED on PR #632. Root cause proven from the netlab-manager logs during the PR's CI run (2026-06-16): every failure was

SJVM host-side RPC interrupted during method=createDroplet (watchdog timeout, cancellation, or shutdown)

The manager's createDroplet call is an HTTP call to the DigitalOcean API executed inside an SJVM sandbox (DropletClient talks to url://digitalocean-droplets/ over a sandboxed connection). Under host load 30+, that HTTP round-trip does not complete within the SJVM watchdog window, so the watchdog kills it. The manager itself was HEALTHY (uptime ~1.7 h, not OOMing), and some createDroplets succeeded — so it is a provisioning STORM: every netlab test provisions its own DigitalOcean droplet via the single shared url://netlab-hosted/ manager, they run concurrently in the kompile suite, each retries createDroplet (via NetlabTestEnv.acquire()'s 6-round jittered backoff), and under CPU starvation most time out while a few get through.

This is shared-infra capacity, NOT a test defect, and it scales with the number of netlab tests:

• main alone was failing ~1 netlab test.
• Adding my 2 netlab tests pushed the same CI run to 6 netlab failures (the 5 pre-existing + my LinkFlap; my ImpairedNatRelay still passed).

So my PR does not introduce a defect, but it worsens a suite that is already over the shared manager's provisioning capacity.

---

Current infra state (write-time snapshot, 2026-07-06 ~20:02 UTC — RE-VERIFY)

The host is NOT healthy — the user's "may be mostly working now" is not borne out at write time:

• 198.199.106.165 load average 35.68 / 39.99 / 33.75 (2 vCPU box).
• ContainerNursery heap dumps at 19:36, 19:40, 19:42 on 2026-07-06 — i.e. every 2–4 minutes, a rapid OOM loop has RETURNED (worse than the hours-apart cadence seen right after the watchman fix). This is the separate ContainerNursery-itself OOM (~190–200 MB heap; hypothesized to be the UrlProtocol gossip outbound-buffer / dead-connection write-backlog leak — a different repo, NOT the watchman leak, NOT in scope for the UrlResolver test PR). cn-watchdog.sh restarts CN and now also reaps orphaned worker JVMs.
• netlab-manager-server uptime ~4726 s (~79 min) at write time.

CI, write-time (RE-VERIFY):

• main = 3c127de5 — kotlin.build (remote) FAILURE: 986 passed, 4 failed — testNetlabDiscoveryScenarios, testNetlabP2PIntegration, testNetlabResolutionUnderLatencyAndPacketLoss (all pre-existing netlab, same storm class) + testOpenPersistentRpcConnectionWaitsForGossipWhenNoCandidatesFound. main is red independent of PR #632.
• PR #632 = head 553c617c, state OPEN, mergeStateStatus UNSTABLE. Latest kotlin.build (remote) = FAILURE, but now a different mode: Build infrastructure error: Status poll for build run 588aa92f ... failed continuously for 338526ms, exceeding the transient-failure budget of 300000ms — i.e. the buildtest infra could not even report run status (consistent with the host being hammered at write time). bld-build (the old GHA check) is no longer present on this repo's PRs; only kotlin.build (remote) remains.

---

Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | State @ write time | Link | |---|---|---| | Round-4 netlab tests | OPEN, kotlin.build RED (infra), merge UNSTABLE, head 553c617c | https://github.com/CodexCoder21Organization/UrlResolver/pull/632 | | Branch | netlab-round4-impairment-chaos-tests (pushed to origin; local checkout no longer exists) | — | | Watchman OOM fix | MERGED + deployed 2026-06-11 (fixed the 2-min crash loop) | https://github.com/CodexCoder21Organization/GithubWatchmanServiceServer/pull/37 | | Round-3 netlab tests (sibling; source of the BIG_RESPONSE harness mode) | OPEN last checked | https://github.com/CodexCoder21Organization/UrlResolver/pull/604 | | netlab: drop url://netlab/ fallback (merged; why tests use only defaultNetlabCandidates()) | MERGED | https://github.com/CodexCoder21Organization/UrlResolver/pull/602 | | Latency/loss netlab test (merged; pre-existing, now red on main) | MERGED | https://github.com/CodexCoder21Organization/UrlResolver/pull/598 | | Dynamic-failure netlab test (merged; added RelayConfig/enableRelay to NetlabHarness that my rebase merged with) | MERGED | https://github.com/CodexCoder21Organization/UrlResolver/pull/599 |

Local memory files (this operator's persistent notes, /home/leslie/.claude/projects/-code/memory/): urlresolver_round4_netlab_tests_and_cn_oom_cascade.md (the authoritative running log of this effort), buildtest_service_oom_crashloop_sjvm_classdata.md (the watchman leak).

---

Next steps

Re-verify current state first (the three commands in the RE-VERIFY banner). Then:

1. This is the open decision the user must make — I asked "how far should I take this?" and the user replied "check the infra status, I think it may be mostly working." At write time it is not working (host load 35, CN OOM-looping every 2–4 min). So the premise for "just re-run CI and it'll pass" does not currently hold. Re-check host health; only if dumps are hours-apart AND load is well under ~10 is a clean netlab CI run plausible. The three options presented:
   • (a) Fix the shared infra (the only real path to green for the whole netlab suite). Two distinct sub-problems: (i) the createDroplet SJVM-watchdog-timeout under host CPU starvation + the unbounded concurrent-provisioning storm — fix at the owning layer in NetLabManagerServer (bound/queue concurrent droplet provisions; and/or stop the DO-API createDroplet from being subject to the aggressive generic SJVM watchdog — note the global "never increase timeouts without approval" rule, so prefer removing/segregating the watchdog for a legitimately-slow network call over bumping it); (ii) the separate ContainerNursery-itself OOM (~200 MB heap, ~every few min under load; hypothesized UrlProtocol gossip outbound-buffer/dead-connection leak) that keeps the box saturated. Both are cross-repo and sizable.
   • (b) Reduce PR #632's marginal footprint by bundling its two tests into ONE shared-droplet test (matching testNetlabDiscoveryScenarios's 9-host single-droplet design). Principled and in-scope, but does not achieve green — the 5 pre-existing netlab tests still storm and main is already red.
   • (c) Hold PR #632 for review as-is (tests verified green locally; CI red is pre-existing netlab-infra flakiness affecting main too). Do NOT merge while red per the global kotlin.build rule.
2. Do NOT merge PR #632 while kotlin.build (remote) is red (hard global rule). It also should not be merged without explicit user approval regardless.
3. If pursuing (a)-(ii) or diagnosing the CN OOM: the shared box has no local UrlResolver checkout anymore — clone fresh. Investigate the CN OOM with the investigate-broken-service skill; heap dumps are in /root/ContainerNursery/heapdumps/. Pull dumps off-host base64-chunked (fabric read-file is not binary-safe and caps ≲3 MB) and analyze locally.
4. When re-running PR #632 CI after any infra improvement, the watch-build skill is the right tool (avoids the silent-monitor / status-poll-timeout pitfalls this run already hit).

---

Reusable / operational knowledge

• The netlab test suite has a structural scaling limit: one DigitalOcean droplet provisioned per test file, unbounded concurrency, against a single -Xmx128m manager whose createDroplet runs in an SJVM sandbox with a watchdog. Every netlab test added raises the concurrent-provisioning storm and the failure rate for ALL netlab tests. The suite's own mitigation is to bundle many scenarios into one droplet (see testNetlabDiscoveryScenarios, 9 hosts / 4 phases); new netlab tests should follow that, not add fresh independently-provisioning droplets.
• NetlabTestEnv.awaitResolveResult false-pass trap: it matches against the client's whole accumulated stdout; never reuse a (host, serviceURL) pair across phases — use a unique per-phase URL path (service id = first path segment, so routing is unaffected).
• Facade-down workaround when kotlin.directory (443) flaps during a CN death: the maven backend also binds publicly at https://kotlin.directory:38809, and coursier launch --mode offline runs already-cached CLIs. The HardwareControlFabric daemon (mTLS, port 8443) is independent of ContainerNursery — use it (or SSH -p 23 as last resort) for host forensics.
• Fabric CLI for remote forensics: coursier launch [--mode offline] community.kotlin.hardwarecontrolfabric:cli:0.2.6 -r https://kotlin.directory -- -H 198.199.106.165 -c ~/.config/hardware-control-fabric/client_cert.pem -k ~/.config/hardware-control-fabric/fabric_key.pkcs8 --ca-cert ~/.config/hardware-control-fabric/server_cert.pem <cmd>. Its launch --args flag is naively space-split, so ship a script file and launch --cmd sh --args /tmp/x.sh --wait, then read back the output file.
• Local kompile builds during a facade flap need a retry loop that clears the stale failure cache between attempts: rm -rf ~/.aibuildcaches/_code_<repo>/buildRuleResultIndex (a "Connection refused"/"Premature EOF" download failure gets cached as a rule failure and re-served on the next run otherwise).
• CN heap-dump interval is the fastest health proxy for the shared box: ls -lat /root/ContainerNursery/heapdumps/*.hprof | head. Minutes-apart = actively OOM-looping (netlab provisioning will storm-fail); hours-apart = comparatively healthy.
• git on this operator's box needs identity set per-clone before committing: git config user.email admin.gmusic@gmail.com; git config user.name Leslie.