Handoff: UrlResolver PR #760 — salvaged close()-promptness regression test, stuck one green check short on CI infrastructure trouble

Written: 2026-07-12 (effort began 2026-07-11). Every state snapshot below was verified at write time — RE-VERIFY before acting. PRs advance, CI re-runs, merge queues shuffle, branches get force-pushed, and the buildtest host's picture changes hour to hour. The authoritative re-checks:

• PR: gh pr view 760 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Remote buildtest run (ground truth beats the flaky frontend): ssh -i ~/.ssh/id_ed25519_20260706_035316 -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<runId>/build.log; grep -c test_completed /root/buildtest-data/runs/<runId>/test-events.jsonl'

---

1. Mission summary

User request: "Drive https://github.com/CodexCoder21Organization/PlanRepository/blob/main/handoffs/2026-07-05-close-scope-drain-hang-642-and-netlab-droplet-leak-diagnosis.md to completion."

That predecessor handoff had exactly one unfinished item (everything else in it was already resolved and its file was deleted as part of PlanRepository #681 on 2026-07-10): its §4 said that if main lacked an equivalent promptness assertion, the additive regression test from the closed UrlResolver #642 should be salvaged into a fresh PR.

I verified against live main that the test was genuinely missing and unique (see §2), salvaged it as UrlResolver #760, hardened it after an adversarial review, and have been driving its CI to green. It is one green check away from ready-to-merge, held up purely by shared-CI-infrastructure trouble, not by anything in the PR.

Standing user directives honored: delegate implementation to codex gpt-5.6-sol (see §7); do NOT merge — hand to the user for merge approval; never bypass CI/merge-queue.

2. What was found and done (the chain)

1. The predecessor handoff's other items were already resolved — re-verified live, not trusted from the doc: UrlResolver #617 MERGED, UrlResolver #642 CLOSED-superseded-by-#690, netlab leak/OOM fixed by NetLabManagerServer #68 + UrlResolver #669.

2. Confirmed the salvage test is genuinely unique coverage. The test — tests/testCloseCompletesPromptlyWithActiveBackgroundCoroutines.kts — asserts that closing a service-registered UrlProtocol2 node that has live background coroutines completes in under 3 s. It is not on main, and the closest existing close tests (stressTestCloseIsBoundedWhenRelayLinkIsDead.kts bounds close at 8 s; testCloseWithLiveRelayLinkAndInterruptedCallerReleasesPort.kts at 15 s) would not catch the historical 5-second scope-drain hang the test guards against. Verified by listing main's tree and reading those tests.

3. Salvaged it (via codex gpt-5.6-sol): branched test/close-promptness-guard off origin/main, cherry-picked the test file from origin/fix-close-scope-drain-hang (the still-existing #642 branch), and updated all stale @file:WithArtifact pins to match current main — protocol 0.0.307→0.0.337, jvm-libp2p snapshot-5→snapshot-10, sjvm 0.0.35→0.0.40, clocks 0.0.3→0.0.11, observable 0.3.6→0.3.13, logging 0.0.1→0.0.3, annotations 0.0.2→0.0.8. Ran the single test locally 3× green.

4. Adversarial review (codex gpt-5.6-sol) returned 3 findings, all fixed and pushed (head 9b2441430f8c84b5314ad1d1140b0590ed47c091):

   • HIGH — detection power: at the pre-fix ~25%-per-close failure rate, 12 iterations gave 3.2% false-green (above the org ≤1% bar). Raised iterations 12→20 (0.75^20≈0.3%). Re-ran 3× green locally (60/60 closes, ~28–41 s each, within the @Timeout(120)).
   • MEDIUM — timing accuracy: switched close-duration measurement from System.currentTimeMillis() to monotonic System.nanoTime(); rewrote the assertion-failure message so it no longer over-claims the scope-drain mechanism is proven (it now lists candidate slow phases to investigate).
   • LOW: added a justifying comment on the finally-block best-effort close() exception swallow (per org policy). PR body updated to reflect 20 iterations and the new provenance.

5. CI has been the whole battle — and every red was infrastructure or an unrelated pre-existing flake, never our test. Chronology of the bld-build (GitHub Actions) + kotlin.build (remote) required checks:

   • bld-build attempt 1 RED: testBytecodeFetchAggregatesRelayFailureDetails (pre-existing relay-error-message-equality flake; 1084/1085 passed). Re-ran.
   • attempt 2 RED: testNetlabDynamicFailureRecovery (pre-existing netlab flake; 1084/1085). Re-ran.
   • attempt 3 CANCELLED: runner logged "Socket read timeout … test may be running slowly" for ~1 h then hit the workflow time limit (slow/overloaded runner). Re-ran.
   • attempt 4 → SUCCESS ✅ (67 min, the full 1085-test suite green including our test).
   • kotlin.build (remote) run 44116119 RED and 0f6794d3 RED: both DigitalOcean droplet-creation rate-limit exhaustion at provisioning (build.log: "waited in the account rate-limit queue but caller budget 30000ms exhausted"). Backed off to the provider reset (03:46:41 UTC) rather than hammering, then re-requested.
   • remote run b0ea453b: got through provisioning and ran the suite cleanly to ~330 tests, then a netlab test (testNetlabResolutionUnderLatencyAndPacketLoss) failed on the same DO rate limit surfacing mid-run (the network-sim tests provision their own droplets). That run then wedged — its build.log froze at 05:12 UTC and the run cascaded to 766 errored tests (DO rate-limit killing netlab provisioning takes down all dependent tests), but the GitHub check never concluded and sat stuck IN_PROGRESS.

6. At handoff I re-drove the wedged remote check. Warmed the webhook receiver (3× curl https://github-webhooks.wasmserver.com/) and re-requested check-run 86612688633; it accepted. RE-VERIFY whether a fresh buildtest run id has materialized and whether it is green.

3. Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | What | State @ write | |---|---|---| | UrlResolver #760 | The deliverable. Salvaged close()-promptness regression test. Branch test/close-promptness-guard, head 9b2441430f8c84b5314ad1d1140b0590ed47c091. | OPEN, not merged. bld-build SUCCESS ✅; kotlin.build (remote) needs a green run (last re-request just fired). mergeStateStatus: BEHIND (branch 1 commit behind main). | | UrlResolver #642 | The closed PR the test was salvaged from (source fix superseded by #690). | CLOSED. | | UrlResolver #690 | The graceful-shutdown() fix that superseded #642's source change (already on main). | MERGED. | | PlanRepository #681 | Deleted the predecessor handoff (and 15 others) as resolved. | MERGED. |

The only branch produced this workstream is origin/test/close-promptness-guard (= PR #760). Nothing is uncommitted or unpushed anywhere; the salvage test is committed and pushed on that branch. The local ~/workspace/UrlResolver checkout mirrors it.

4. Next steps (for the picker-upper)

1. Re-verify current state first (commands in the banner). Specifically: is kotlin.build (remote) green yet on head 9b244143? Is bld-build still SUCCESS (a rebase or long delay can invalidate it)?
2. If the remote check is still red/wedged on DigitalOcean droplet rate-limits or a netlab flake: this is shared-infra, NOT the PR. Re-drive it: warm webhooks (for i in 1 2 3; do curl -s -o /dev/null --max-time 60 https://github-webhooks.wasmserver.com/; done), find the check-run id (gh api "repos/CodexCoder21Organization/UrlResolver/commits/9b2441430f8c84b5314ad1d1140b0590ed47c091/check-runs" --jq '.check_runs[]|select(.name=="kotlin.build (remote)")|.id'), then gh api --method POST ".../check-runs/<id>/rerequest". Verify a new buildtest run id appears (previous dead/failed runs: 44116119, 0f6794d3, b0ea453b). Ground-truth progress on the host, not the flaky frontend.
3. If the DO rate-limit persists across another attempt, file a report-challenge so the infra owners see the pattern (droplet rate-limit repeatedly failing UrlResolver netlab tests, each retry ~100 min). Do not weaken the test, reduce iterations, or raise timeouts to dodge it — the failures are external.
4. mergeStateStatus: BEHIND (1 commit): the repo uses a merge queue that tests against latest main at merge time, so a preemptive rebase is usually unnecessary — but if GitHub blocks the merge on "branch out of date," rebase the live remote tip (git fetch origin && git rebase origin/main on test/close-promptness-guard, git push --force-with-lease) and re-verify both checks. A push restarts CI.
5. When both checks are green: STOP and hand to the user for merge approval. Do NOT merge, do NOT enqueue (standing directive; this is a code PR, not a handoff PR).
6. After #760 merges: the predecessor handoff is already deleted, so the only cleanup is deleting this handoff (it becomes DONE at that point).

5. Reusable / operational knowledge

• codex gpt-5.6-sol invocation (from the target repo dir): codex exec -m gpt-5.6-sol -c model_reasoning_effort="high" --dangerously-bypass-approvals-and-sandbox "<prompt>"; resume the same session with codex exec <same flags> resume <session-id> "<prompt>" (the bypass flag MUST precede resume or it falls back to the bwrap sandbox and silently no-ops on this container). The gpt-5.6-sol quota was exhausted then reset during this session — the cap is transient/account-wide, not a standing block.
• Run only ONE test, not the suite, locally: scripts/test.bash --test tests/<name>.kts from the repo root. codex initially ran scripts/test.bash --test . (the entire 1085-test suite, hours) — watch for and correct that.
• buildtest frontend flaps HTTP 503 / read-timeout under load; this is degraded observability, NOT a dead run. Ground-truth on the host: ssh -i ~/.ssh/id_ed25519_20260706_035316 -p 23 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<id>/build.log; grep -c test_completed /root/buildtest-data/runs/<id>/test-events.jsonl; grep "\"error\":\"[^\"]" /root/buildtest-data/runs/<id>/test-events.jsonl'. A run is only truly wedged when build.log mtime AND content are frozen >20 min.
• build-watchman for CI shepherding: ~/bin/cs launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory --java-opt -Xmx512m -- --repo CodexCoder21Organization/UrlResolver --pr 760. It watches to checks-green then EXITS — re-arm it after every rerun. On this /code container the watchman JVM was repeatedly SIGTERM'd/killed (4 instances died); it is a tracked background job, so compensate by verifying PR state directly at each checkpoint rather than trusting the watcher is still alive. --java-opt -Xmx512m caps its heap (the box OOM-kills unbounded JVMs under load).
• DigitalOcean droplet rate-limit failures show classification=INFRASTRUCTURE/PROVISIONING: Failed to create droplet + a provider reset=<UTC> timestamp in the run's build.log. Back off to that reset before re-requesting; hammering just re-exhausts the window. It hits both the top-level buildtest droplet AND the netlab-worker droplets the network-sim tests spin up mid-suite.
• Known unrelated pre-existing flakes seen failing #760's CI (none implicate the diff): testBytecodeFetchAggregatesRelayFailureDetails, testNetlabDynamicFailureRecovery, testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup, testNetlabResolutionUnderLatencyAndPacketLoss.