← Priority list

Handoff: UrlResolver pending-dial-leak fix (824) + relay-withdrawal fix (826) reconciled as resolver 0.0.656, deployed to ContainerNursery; combined PR mid-CI, child-fleet bump outstanding

Original user request (verbatim intent): *"Looks like some kind of infrastructure error is happening, we need to resolve it. Use gpt 5.6-sol to do the heavy lifting."* — with explicit authorization to *"build locally, publish, merge and deploy."* Later, on seeing CI thrash, the user directed: *"we should build locally and deploy to unblock ourselves."*

hf-2026-07-22-urlresolver-dial-leak-plus-relay-withdrawal-reconciled-0656-cn-deploy-and-fleet-bump

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver pending-dial-leak fix (824) + relay-withdrawal fix (826) reconciled as resolver 0.0.656, deployed to ContainerNursery; combined PR mid-CI, child-fleet bump outstanding

Written 2026-07-22 ~16:20 UTC — RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time. PRs merge, CI re-runs, branches move, ContainerNursery restarts, other agent sessions deploy/roll back, and the host's socket/load picture changes minute to minute. Re-verify before acting on anything here. Authoritative re-checks:

  • PR: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • UrlResolver main tip: gh api repos/CodexCoder21Organization/UrlResolver/commits/main --jq '.sha[0:12]+" "+(.commit.message|split("\n")[0])'
  • Live production jar (which resolver is actually running): ssh -p 23 -i ~/.ssh/cn_host_diag -o IdentitiesOnly=yes root@198.199.106.165 'sha256sum /root/ContainerNursery/bin/container-nursery.jar; uptime; ss -Htn state syn-sent | wc -l'
    • ⚠️ At 16:20 UTC the live jar SHA was 581ff1ab75556d91… — which is NEITHER 0.0.654 (0939de87…) NOR our combined 0.0.656 (e8f29373…). A third actor deployed something between 15:00 and 16:20. Identify what 581ff1ab is (does it contain PendingDialAdmissionController AND the withdrawal unregisterFromRelay(long)?) BEFORE any further deploy. unzip -l the jar and grep for both classes.

Mission summary

Original user request (verbatim intent): "Looks like some kind of infrastructure error is happening, we need to resolve it. Use gpt 5.6-sol to do the heavy lifting." — with explicit authorization to "build locally, publish, merge and deploy." Later, on seeing CI thrash, the user directed: "we should build locally and deploy to unblock ourselves."

The infrastructure error was a host-wide degradation of ContainerNursery host 198.199.106.165: two public HTTPS names down (missing SNI routes), and the ContainerNursery supervisor JVM burning ~92% CPU on ~700 half-open (SYN-SENT) TCP sockets. Root cause of the CPU/socket storm: a pending-dial leak in the embedded UrlResolver/UrlProtocol networking layer. The effort expanded to: restore the routes, fix the dial leak, and — after discovering a second agent session had independently fixed a different real UrlResolver defect and the two deploys collided — reconcile both fixes into one artifact and get the host healthy.

What was found and done (the chain)

1. Two public routes were down — FIXED (operational, no PR)

productionhealth.kotlin.build and lambdaserver.wasmserver.com resolved to the host but had no ContainerNursery SNI route, so TLS handshakes were closed (TLS_UNKNOWN_SNI). Fresh Let's Encrypt certs were issued via the generate-tls-certificate skill (DNS-01; the Namecheap direct API rejected this machine's egress IP so the NamecheapProxy CLI was used), installed to /root/certs, and the two routes added dynamically via ContainerNursery POST /routes (no restart, hot siblings untouched). lambdaserver returned 200; productionhealth.kotlin.build completed TLS but its backend stalled — see item 4.

2. ContainerNursery ~92% CPU = UrlResolver pending-dial leak — ROOT-CAUSED + FIXED

Thread dumps showed the CPU in the libp2p event loop encrypting/flushing frames over ~700 SYN-SENT sockets. Mechanism: gossip/PEX sends to dead peers time out at the coroutine layer (~1s) and release their concurrency permit, but the underlying Netty TCP connect is not cancelled and lingers in the kernel's ~127s SYN-retry lifecycle; with a registry of thousands of stale cloud-peer addresses and a circuit breaker that only arms after a completed failure, hundreds of half-open dials accumulate. Fix (ours): a process-wide pending-dial admission controller — cap 24 total / 16 unverified / 8 reserved for known-good peers; single-flight coalescing on canonical (host, peerId, multiaddr); permit held until the transport future actually completes, not until the waiting coroutine gives up; timeout arms per-address exponential quarantine + circuit breaker; refusals are a distinct non-failure outcome (never quarantine an un-dialed peer); best-effort synchronous channel close on timeout. This bounds half-open sockets to tens instead of ~700. PR: https://github.com/CodexCoder21Organization/UrlResolver/pull/824 (branch fix/gossip-pending-dial-admission, head 5b8de7f5). Fail-first reproducer went 5/5-fail → 10/10-pass; full remote CI passed 100/100 test groups; two circuit-breaker regressions surfaced and were fixed (real transport failures — pending-timeout OR completed-failed — must still arm the breaker; admission refusal retries within the caller's budget rather than failing the send).

3. Buildtest CI false lease-death — already fixed by a concurrent effort

CI on the loaded host kept mass-failing PRs with the false-lease-death signature (all durationMs=0, "Test did not complete during build execution"). That coordinator bug was already fixed + deployed (BuildTestEmbedded PR 359 / BuildTestServerService PR 240, live since 2026-07-21 22:56 UTC). It is NOT the cause of our CI slowness — the cause is host starvation from the very dial leak we fixed (the circular dependency: our fix's merge gate runs on the host the bug is starving).

4. productionhealth WUI stall — pre-existing, separate defect

productionhealth.* completes TLS, sends GET, then stalls with no HTTP response. In-flight jstacks proved the WUI's Jetty threads wait on RPC response futures while the provider receives every call — RPC responses lost on the established UrlProtocol stream (the 2026-05-13 case-study defect class). The WUI/server were bumped to current resolver/protocol and redeployed (PRs 59/85 below), which changed the failure mode but did not fully clear it under host load. Verdict: load-induced under the dial storm; expected to clear once the host de-loads. A local marshaling benchmark ruled out a marshaling perf defect (100 health checks render in ~2.2s).

5. DEPLOY COLLISION with a second session — reconciled as 0.0.656

A second session (codex-stormfix) independently fixed a different real defect — relay withdrawal loss + snapshot resurrection (dead services resurrecting into peer registries): PR https://github.com/CodexCoder21Organization/UrlResolver/pull/826 (fix/relay-withdrawal-delivery-race), mechanisms = protocol 0.0.397 + authoritative unregisterFromRelay(flushTimeoutMs=5_000) before transport shutdown + Long.MIN_VALUE ordering for snapshot-derived membership. Timeline: they published 0.0.654 and deployed 12:14; we published 0.0.655 (admission fix) and deployed 12:30; they rolled us back to 0.0.654 at 12:33 (correctly — 0.0.655 lacked their fix; but 0.0.654 lacks ours). Neither single artifact had both fixes; the fixes are complementary and production needs both. Resolution: built combined/dial-admission-plus-withdrawal (our 5b8de7f5 + merge of their branch; also incorporated PR 825's bytecode single-flight, which was in their 0.0.654), verified both PRs' targeted tests green, published resolver 0.0.656 (0.0.654/0.0.655 were occupied — kotlin.directory is immutable), coordinated on both PRs + a merged challenge record BEFORE deploying, and deployed CN on 0.0.656 at ~13:57 UTC. Combined PR: https://github.com/CodexCoder21Organization/UrlResolver/pull/833.

6. Deploy verification (write-time)

Post-0.0.656 deploy, CN-scoped SYN-SENT dropped to 2–12 (from ~700) across 2/9/20-minute samples — the admission fix is working at the supervisor. BUT host-wide SYN-SENT stayed elevated (300–520) and load ~50–60, because ~20 child app JVMs still run OLD resolver versions and now dominate the host storm (per-process socket table: github-watchman ×2, aisupervisor ×2, kompile-CI WUI, dependabot WUI, netlab-manager, droplet-service, etc.). The supervisor is fixed; the fleet is not. Known deploy quirk: the restart tool's initial 128 MiB bootstrap OOMs on the cold-restoration burst; the watchdog auto-relaunches the same jar at -Xmx1024m — expected, wait for the watchdog PID.

Relevant PRs / refs (write-time state — RE-VERIFY)

| Item | State @ write | Link | |---|---|---| | UrlResolver combined PR (824+825+826, 0.0.656) | OPEN, mergeStateStatus BLOCKED, full CI mid-run (~412P/4F of 1128, host-side counters advancing; buildtest API 100-result pagination bug makes the web view lie — trust host test-events.jsonl) | https://github.com/CodexCoder21Organization/UrlResolver/pull/833 | | UrlResolver PR 824 (our admission fix) | OPEN — subsumed by 833; close/link on merge | https://github.com/CodexCoder21Organization/UrlResolver/pull/824 | | UrlResolver PR 826 (their withdrawal fix) | OPEN (third-party) | https://github.com/CodexCoder21Organization/UrlResolver/pull/826 | | UrlResolver PR 825 (bytecode single-flight, in 0.0.654) | OPEN (third-party) | https://github.com/CodexCoder21Organization/UrlResolver/pull/825 | | ProductionHealthServiceWui version bump | OPEN, BLOCKED (its only failing check was a host-starvation CI death; code deployed + locally green — re-run when host is healthy) | https://github.com/CodexCoder21Organization/ProductionHealthServiceWui/pull/59 | | ProductionHealthServiceServer version bump | OPEN, same as above | https://github.com/CodexCoder21Organization/ProductionHealthServiceServer/pull/85 | | PlanRepository deploy-collision challenge record | MERGED | https://github.com/CodexCoder21Organization/PlanRepository/pull/2710 | | ContainerNursery resolver-0.0.656 pin (fat-jar bump) | branch bump-resolver-0.0.656 prepared locally at /code/workspace/ContainerNursery (based on merged PR 535 head 8e8203c); PR not yet opened | (open one) | | DigitalOceanDropletServiceServer resolver pin | a third lane (claude-handoff-driver-lane5) claimed this child bump to 0.0.656; coordinate, don't duplicate | https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/92 (stale) |

Published artifacts: foundation.url:resolver:0.0.656 (combined — contains PendingDialAdmissionController, SandboxedProxyGenerator$AbandonedProxyCleanup, and the withdrawal unregisterFromRelay(long)/relay-dedup fingerprints; clean-cache verified). Do NOT deploy 0.0.654 (lacks admission fix) or 0.0.655 (lacks withdrawal fix). Never select protocol 0.0.395/0.0.396 (contaminated) — the good protocol is 0.0.397. CN fat jar with 0.0.656: SHA e8f293731253c648….

Next steps (ordered — RE-VERIFY current state first)

  1. Re-verify the live production jar SHA first (banner). If it is not e8f29373… (0.0.656 CN jar) and not something newer that provably contains both fixes, that is the top priority: identify what 581ff1ab… is; if it lacks the admission fix, the dial leak is live again. Do not enter a rollback war — if another actor is actively deploying, comment + coordinate via the host /root/cn-watchdog.audit file and the PR threads first.
  2. Drive combined PR https://github.com/CodexCoder21Organization/UrlResolver/pull/833 to merged. Its full suite was mid-run with ~4/412 failures — adjudicate each failure by name + evidence (real regression → fix; load-flake/known-flake → document). Use build-watchman 0.0.13 --to-merged; treat buildtest-API 500s/pagination as observability flaps and verify via host test-events.jsonl. On merge, close/link 824 and open the resolver 0.0.656 release-metadata PR against main and drive it merged.
  3. Open + merge the ContainerNursery 0.0.656 pin PR from the prepared bump-resolver-0.0.656 branch so the deployed jar matches a merged commit (currently deployed from an unmerged local build).
  4. FLEET-BUMP the child app JVMs to resolver 0.0.656 + protocol 0.0.397 and redeploy them — this is what actually de-loads the host (supervisor is already fixed). ~20 JVMs; coordinate with the droplet-service lane (item above) to avoid duplication. This is the remaining "resolve the infrastructure error" work: host load won't return to normal until the fleet is bumped.
  5. Re-run the failed CI checks on PRs 59 and 85 once the host is healthy (their failures were verified host-starvation deaths, not code) and merge them. Then re-check productionhealth.kotlin.build serves 200 (expected to clear once de-loaded; if not, chase the UrlProtocol established-stream RPC-response-loss defect — item 4, the 2026-05-13 class).
  6. Follow-ups already recorded on PR 824, don't lose them: (a) configure a ~10–15s Netty CONNECT_TIMEOUT_MILLIS in the org jvm-libp2p fork so kernel dials don't live ~127s (complements, not replaces, the ownership fix); (b) per-instance fair-share slices within the process-wide cap if multi-resolver-per-JVM tenancy ever becomes real.

Reusable / operational knowledge (learned the hard way)

  • Check for concurrent actors before ANY resolver/CN deploy. kotlin.directory is immutable, so "version 0.0.65x already taken" is itself a signal another session is mid-flight. stat the live jar mtime + diff its SHA vs expected before deploying; leave coordination breadcrumbs (PR comments + /root/cn-watchdog.audit on the host + a PlanRepository challenge record) BEFORE deploying, not after. On discovering a rollback by another actor: STOP, never overwrite in a loop, reconcile into ONE artifact containing all fixes. (Memory: deploy-collision-two-sessions-2026-07-22.)
  • Deploy-first is a legitimate way to break the CI-on-the-starved-host circular dependency — the artifact was 5/5 local + 100/100 remote before deploying, stronger evidence than most merges.
  • Host access: ssh -p 23 -i ~/.ssh/cn_host_diag -o IdentitiesOnly=yes root@198.199.106.165. Run data under /root/buildtest-data/runs/<id>/{run.json,build.log,test-events.jsonl}. CN jar at /root/ContainerNursery/bin/container-nursery.jar; timestamped pre-deploy backups *.pre-0.0.65x-*.
  • build-watchman: cs launch buildwatchman:build-watchman:0.0.13 -r https://kotlin.directory -- --repo <o>/<r> --pr <n> --to-merged. Its buildtest-API view has a 100-result pagination bug — a run "stuck at 100/100" is usually still advancing; confirm via host test-events.jsonl. Working coursier here: the JVM wrapper (native /usr/local/bin/coursier is wrong-arch); a downloaded x86_64 coursier was used.
  • Publish procedure (immutable registry): local unpushed build.kts version bump → foundation.url.resolver.buildMaven → inspect JAR for expected classes → publish via community-kotlin-maven-artifact-publishing:0.0.5 to https://api.kotlin.directory/upload → clean-cache resolve to verify byte-identical.

Why paused

The user asked to stop all in-progress work and loops and produce this report. The reconciliation agent (combined PR CI-shepherd + landing) was stopped mid-CI-watch; the recurring status-report loop was cancelled. No production mutation was in flight at stop time from this session. Nothing here is blocked on the user — it is paused by request and ready for a cold pickup per the next-steps list.