Handoff: UrlResolver #659 cleanup — code DONE, blocked on DigitalOcean droplet-provisioning failures in the remote CI lane

Written: 2026-07-12 ~16:50 UTC. This continues the #659 leg of the earlier handoff 2026-07-05-urlresolver-656-cleanup-659-and-netlab-oom-supersession.md — read that for the original mission framing; this document supersedes its "Next steps" for PR #659.

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below is a write-time snapshot. PRs merge, CI re-runs, merge queues shuffle, branches get force-pushed, and the buildtest host's picture changes hour to hour. A fresh query always wins over this document.

• PR: gh pr view 659 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Remote check-runs (the authoritative CI truth — the PR's statusCheckRollup lags): gh api "repos/CodexCoder21Organization/UrlResolver/commits/<HEAD_SHA>/check-runs" --jq '.check_runs[] | "\(.name) | \(.status)/\(.conclusion) | \(.details_url)"'
• buildtest run ground truth on host (beats the flaky HTTP frontend): ssh -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<runId>/build.log; tail -5 /root/buildtest-data/runs/<runId>/build.log'

---

1. Mission

Original request (verbatim intent): "drive [the 2026-07-05 handoff] to completion." That handoff's one unfinished, owned item is UrlResolver PR #659, a pure-cleanup PR removing two leftovers from the earlier #656 merge:

1. Ungated [PEX-*] debug diagnostics printed to stderr. The [PEX-OK] / [PEX-EMPTY] peer-exchange messages were emitted via unconditional System.err.println, violating the resolver's convention of routing diagnostics through the gated debugLogErr {} helper (enabled only with -Dfoundation.url.resolver.debug=true).
2. A test-shaped production global. A public, top-level val servicelessPeerVerificationCount: AtomicLong existed in production source only so a regression test could read it — production API shaped around a test.

The regression the counter guarded: a peer that advertises no services must be registered directly and must never enter the bounded/blocking reachability-verification queue (doing so causes an O(N²) dial storm that starves discovery).

---

2. What was found and done (the chain — all verified)

1. Re-verified the PR was still needed and DIRTY. PR #659 was OPEN with mergeStateStatus=DIRTY — its single commit was based on an old main and conflicted (UrlResolver.kt + a deleted stress test that main had since modified).

2. Delegated the intent-based rebase to codex gpt-5.6-sol (canonical invocation, background). It rebased onto current main, resolving conflicts by intent:

   • Started every conflicted UrlResolver.kt hunk from main's version. Discovered main had independently already gated the [PEX-OK] site — so the cleanup narrowed to gating the one remaining unconditional print, the [PEX-EMPTY] re-pull diagnostic (rewritten by #726), wrapping it in debugLogErr {} with string-building inside the lambda.
   • Removed the servicelessPeerVerificationCount declaration + increment.
   • Confirmed deletion of tests/stressTestServicelessPeerVerificationDoesNotStarveDiscovery.kts (its only assertion mechanism was the removed counter; verified it was the counter's sole consumer). Behaviour coverage remains in tests/stressTestConcurrentLateJoinersDiscoverServicePromptly.kts plus the branch's new public-API test tests/testOnPeersReceivedRegistersServicelessPeerWithoutVerification.kts.
   • Aligned that new test's @file:WithArtifact pins to current main (protocol 0.0.337, libp2p snapshot-10, SJVM 0.0.40, observable 0.3.13, logging 0.0.3, annotations 0.0.8).
   • Left the raw System.err.println calls at UrlResolver.kt ~lines 10735–11068 untouched — verified they are UrlProtocol2.close() cleanup-failure reporting, not [PEX-*] diagnostics, hence out of scope.

3. TDD fail-first evidence (verified, not assumed). With both build caches purged (~/.aibuildcaches/*UrlResolver* and ~/.cache/coursier/v1/bldbinary) before each run: reverting the routing guard (&& filteredServices.isNotEmpty() in onPeersReceived) made testOnPeersReceivedRegistersServicelessPeerWithoutVerification fail in ~2 s with the exact invariant message ("a SERVICE-LESS peer … must be registered DIRECTLY … routed to the verification queue"); restoring the guard made it pass. Build + both targeted tests green.

4. A parallel session raced the same PR. While codex finished, another session force-pushed its own equivalent rebase to the PR branch (head 48585009). It differs from my local rebase (51a3dbda, never pushed) in one 12-line UrlResolver.kt hunk where 48585009 is slightly better: it also corrects two stale "we do NOT silently swallow" comments (an original #656 cleanup item) and hoists all message-building inside the gated lambda. I adopted 48585009 rather than starting a force-push war, and re-verified it locally: both targeted regression tests pass 1/1 with caches purged. My 51a3dbda was discarded intentionally — nothing unsaved.

5. Netlab-OOM supersession thread (task from the parent handoff) — verified fully resolved in code, no residual work:

   • Main's src/foundation/url/resolver/sandbox/SandboxedConnection.kt now calls sjvm.close() on close (the leak the parent handoff chased). Confirmed via grep.
   • The canonical fix shipped via UrlResolver #669 — MERGED 2026-07-03. UrlResolver #688 (an alternative) is CLOSED; NetLabManagerServer #82 is CLOSED as obsolete — NetLabManagerServer main already pins foundation.url:resolver:0.0.600 (> the 0.0.589 it proposed). This thread needs nothing further.

---

3. The blocker: the remote CI lane cannot get a clean run (DigitalOcean droplet provisioning)

The PR's GitHub Actions lane (bld-build) PASSED (1h04m, run 29170928926) — full suite including the netlab e2e tests. The remote kotlin.build lane (buildtest, external kotlin-build-ci-test check-suite 78932865812) has never recorded a clean run across 4+ attempts, every failure infrastructure-classified, none the PR's code:

| Attempt (runId) | Outcome | Root cause (host-verified) | |---|---|---| | a039164c | 1082 passed / 2 failed | Both flakes, not the diff: testReconnectSucceedsViaGossipPropagatedServiceInfo missed a 15 s discovery window (its own error text shows the service was discovered); testNetlabDynamicFailureRecovery container harness didn't boot within 6 min. Host load ~80 at the time; run had been resumed after a service restart. | | 6b043a57 | provisioning_failure (0 tests) | DropletAllocationException: POST /v2/droplets waited in the DO account rate-limit queue, caller budget exhausted; ratelimit-remaining=0. | | 140f3161 | provisioning_failure (0 tests) | Same DO rate-limit signature. | | 31de760a | died mid-run ~05:12 UTC | Shards 0–6 started executing; shards 7/8/9 hit DropletAllocationException … listDroplets() … java.lang.NullPointerException … Failed to create droplet in all regions (sfo3, nyc3). Of the slots that ran, 275/275 executed tests were SUCCESS; the 816 "FAILURE" tally is un-provisioned shards, not code failures. |

Critical stale-state trap: GitHub's check-run for kotlin.build (remote) on head 48585009 still shows in_progress (details_url → run 31de760a) even though that run died ~05:12 UTC — the terminal conclusion webhook was never posted when the droplets died, so the check is wedged and will never resolve on its own. Do not wait on it. Confirm death on the host (build.log mtime frozen at 05:12), then force a fresh suite (see Next steps).

Prior evidence the code is green: the same commit 48585009 passed the GitHub Actions full suite, and an earlier superseded remote run (a607aabf) completed 100/100 PASSED before being re-requested away. The only thing missing is one clean, un-preempted remote run recorded against the PR head.

---

4. Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | State @ write | Notes | |---|---|---| | UrlResolver #659 | OPEN, mergeStateStatus=BEHIND, head 48585009, branch cleanup-pex-debug-prints-and-test-shaped-counter | The cleanup. Code complete + TDD-verified. GH Actions lane PASS; remote lane wedged (see §3). Merge is pre-approved (parent handoff §3 + user's "drive to completion"). | | UrlResolver #669 | MERGED 2026-07-03 | Canonical SJVM-close fix. Netlab-OOM thread — done. | | UrlResolver #688 | CLOSED | Alternative SJVM fix, superseded by #669. | | NetLabManagerServer #82 | CLOSED (obsolete) | main already pins resolver 0.0.600. | | UrlResolver #656 | MERGED | The PR whose leftovers #659 cleans up. |

No uncommitted or unpushed work anywhere. workspace/UrlResolver is clean at the pushed head 48585009.

---

5. Next steps (ordered; start by re-verifying)

1. Re-verify live state (banner commands). Specifically re-check whether the remote kotlin.build (remote) check on head 48585009 has since concluded green (a parallel session or the CI reconciler may have re-driven it while this was paused). If it is already green, skip to step 4 (merge).

2. If main has advanced far / the PR is BEHIND and the branch base is stale: rebase onto current origin/main first (this repo uses a squash merge-queue that rewrites main's hashes; use git rebase --onto <newmain> <oldbase> and git push --force-with-lease). Nothing in the diff is expected to conflict again, but verify the two targeted tests still pass with both caches purged.

3. Force a fresh remote run — do NOT wait on the wedged check. Only the single orchestrating operator should do this:

   • Confirm run 31de760a is dead on the host: ssh -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/31de760a/build.log' (mtime frozen ~05:12 UTC).
   • Check the DigitalOcean provisioning weather first. The repeated failures are DO droplet-allocation rate-limit / listDroplets NPE across sfo3+nyc3. If a burst is in progress, back off and let it drain (see buildtest DO 429 provision backoff knowledge) rather than hammering — re-requesting into an exhausted quota just burns attempts.
   • Warm webhooks, then re-request the suite:

     for i in 1 2 3; do curl -s -o /dev/null --max-time 60 https://github-webhooks.wasmserver.com/; done
     gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-suites/78932865812/rerequest
     

   • Verify the check's details_url run id CHANGED from 31de760a and the new /root/buildtest-data/runs/<newId> dir materialized before trusting anything.
   • Arm exactly one build-watchman as a tracked background job: cs launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 659 --to-merged. It auto-enqueues to the merge queue on green and follows to MERGED.

4. Merge #659 once both lanes are green — via the merge queue (GraphQL enqueuePullRequest; gh pr merge --auto is disabled here). Pre-approved. Never --admin/bypass; never merge red.

5. Close out the handoffs. Once #659 is MERGED: delete both this file and the parent 2026-07-05-urlresolver-656-cleanup-659-and-netlab-oom-supersession.md (the netlab-OOM thread it also tracked is verified done, §2.5), and merge the deletion (handoff deletions inherit the standing pre-approval).

6. If the remote lane keeps failing on DO provisioning after the burst drains, this is persistent shared-infrastructure friction worth a report-challenge (broken/rate-limited DO provisioning wedging kotlin-build-ci-test, with the wedged-check-run-conclusion bug as a second finding) — do not park the PR waiting for infra to quiet; keep re-driving on a drained quota.

---

6. Reusable / operational knowledge

• The buildtest status frontend flaps 503/500/timeout under host load; it is NOT proof a run died. Always ground-truth on the host: build.log mtime advancing = alive; frozen for >20 min = dead. build-watchman 0.0.12 classifies these correctly and never deletes on HTTP symptoms alone.
• A wedged GitHub check-run (in_progress forever because the terminal conclusion webhook was lost when its droplet died) will never self-resolve. Re-request the whole suite (check-suites/<id>/rerequest) and verify the details_url run id changes.
• DO droplet provisioning is the current fragile layer: signatures are DropletAllocationException … caller budget … exhausted … ratelimit-remaining=0 and … listDroplets() … NullPointerException … Failed to create droplet in all regions (sfo3, nyc3). Remedy is backoff, not retry-hammering; concurrent sessions re-requesting the same PR multiply the load.
• Stale-cache false PASS: kompile serves the last-built jar for same-version src/ edits. Purge both ~/.aibuildcaches/<repo-path-with-/-as-_> and ~/.cache/coursier/v1/bldbinary before any fail-first/green validation.
• The [PEX-*] sites run on detached background coroutines (peerExchangeJob) where an Effective { handler {…} } does NOT catch a tossed effect and an unhandled toss prints to stderr unconditionally — so debugLogErr (gated, silent by default), not an algebraic effect, is the correct sink for these specific diagnostics. If a reviewer re-raises "use effects," that's the answer; don't re-litigate.
• Host access (last resort): ssh -p 23 root@198.199.106.165 (key ~/.ssh/id_ed25519_20260706_035316). Append to /root/cn-watchdog.audit before acting on the host.
• codex delegation: codex exec -m gpt-5.6-sol -c model_reasoning_effort="high" --dangerously-bypass-approvals-and-sandbox "<brief>" from the target repo checkout.