Handoff — Memory PR/Issue Audit → Rebase → Drive-to-Green

Date: 2026-07-05 Author: Claude (Opus 4.8) session Status: (updated 2026-07-10) Audit COMPLETE; 6 remaining UrlResolver PRs re-rebased onto current main, pin-synced, locally validated; still blocked on buildtest-server infra — 5 consecutive CI attempts on #605 were killed by 5 distinct infra mechanisms on 2026-07-10 (see the 2026-07-10 addendum at the bottom).

---

1. Original request (three sequential asks in one conversation)

1. Audit every PR/issue tracked in the agent's memory: figure out how many are still real vs obsolete, close the obsolete ones, and verify the non-obsolete ones are still a problem.
2. Get the still-real ones rebased onto current main and CI-green.
3. For the blocked ones, ensure they are rebased and green.

---

2. What is DONE

Phase 1 — Audit (complete)

Swept 121 PR/issue references from memory against live GitHub. Result: 5 obsolete (closed), 18 real (verified against current main).

Closed as obsolete (with explanatory comments on each):

• UrlResolver #601 — superseded by merged #607 (perf-test annotation skew fixed via build-kotlin-annotations 0.0.8, not this PR's 0.0.6).
• UrlResolver #603 — explicit [DIAGNOSTIC] throwaway; both fixes it combined (#607/#602) landed.
• UrlResolver #519 — superseded by merged #611 (per-peer bounded gossip queues) + upstream UrlProtocol fix.
• UrlProtocol #305 — superseded by merged #307, which fixed issue #294 and landed a stronger constrained-heap regression test.
• W3WalletDaemon issue #109 — fixed upstream in UrlResolver 0.0.552 (resourcePath now preserved on RPC sends).

Phase 2 — Rebase + green

All 18 real PRs were rebased onto current main (conflicts resolved by taking main's version pins; no logic changes).

MERGED + green (several merged by the user / a parallel session):

• W3Wallet security stack: daemon #106 · #107 · #108 · launcher #12 · extension #51 · harness #2 · tests #12.
• BuildTestEmbedded #218 — remote-builder kompile-cli 0.0.52→0.0.65 fix; merged + published as buildtest-embedded:0.0.198.
• NetLabManagerServer #54 — root-cause fix I authored this session (see §4).

Closed as obsolete during phase 2:

• BuildTestServerService #154 — its only change was a stale buildtest-embedded bump to 0.0.189; main already consumes buildtest-embedded:0.0.198 (which carries #218's kompile-cli 0.0.65 fix), so it was redundant end-to-end.

---

3. BLOCKED / in progress — the 7 UrlResolver PRs

All 7 are rebased onto current main, code-correct, and have ZERO real test failures. They cannot go green because of a buildtest-server-service infrastructure degradation — verified by the fact that UrlResolver main itself is red with the identical error:

foundation.url.protocol.sandbox.SandboxException:
SJVM virtual method invocation timed out after 120s:
buildtest/server/BuildTestServiceClientImpl.uploadChunk(Ljava/lang/String;I[B)V

uploadChunk streams the build's source tarball to url://buildtest/ over SJVM. It consistently times out at 120s for large repos: NetLabManagerServer + the whole W3Wallet stack (small uploads) went green, but UrlResolver is a big repo, so every upload exceeds the budget. Serial re-runs (one PR at a time, 3 attempts each, over 11 hours on 2026-06-22) all failed for this one reason. As of 2026-07-05, buildtest.kotlin.build is down again (HTTP 000) and UrlResolver main's last run (2026-07-04 20:39Z) is still failure on the same error.

This is the recurring buildtest/CN libp2p RPC congestion. It is the same workstream as the parallel-session handoffs in this folder — see 2026-07-05-buildtest-scheduling-cn-outage-and-getbuildrun-rpc-flake.md, 2026-07-05-ci-stabilization-and-fleet-redeploy.md, and the netlab-manager-oom-* handoffs. Do not fix buildtest infra without coordinating with that workstream (a restart would wipe an in-progress heap dump).

The 7 PRs (all OPEN, all rebased onto current main, all RED only due to the above)

| PR | Branch | What it is | Notes for pickup | |----|--------|-----------|------------------| | #605 | fix-openconnection-string-class-json-encoding | SOURCE FIX: openConnection(String::class) returned Map.toString() (invalid JSON) when the direct path won the direct-vs-relay race. Adds encodeResultAsString. | This is the ONLY non-test PR — a real product fix still absent from main (the earlier "0.0.542" was a burned external publish). Coordinate bumped to 0.0.560. Highest-value of the 7. | | #616 | add-bytearray-param-rpc-coverage-tests | Test-only: ByteArray RPC-param marshaling coverage across raw/sandbox/relay/stream paths. | Pure additive coverage. | | #631 | netlab-scenario-tests-v2 | Test-only: severe impairment / gossip-under-degradation / peers-going-offline. Owns the shared NetlabTestEnv netem helpers (setLatency/setPacketLossRate/setRateLimit/clearImpairment). | Merge this before #632. | | #632 | netlab-round4-impairment-chaos-tests | Test-only: netem impairment + link-flap + provider-migration. | STACKED on #631 (base branch = netlab-scenario-tests-v2) so it inherits the netem helpers and compiles. Must merge after #631; GitHub will retarget its base to main when #631 merges. | | #608 | netlab-nat-intermittent-relay-tests | Test-only: NAT↔NAT intermittent-relay-outage (outage/recovery + in-process session survival). | Independent. | | #643 | fix-netlab-upload-client-timeout | Test-only: netlab dynamic-failure upload test with a user-approved 180s uploadFile timeout. | Supersedes the auto-closed #625 (GitHub blocked reopening a force-pushed closed PR). The 180s timeout increase was explicitly approved by the user. | | #595 | netlab-relay-reconnection-after-connection-loss | Test-only DRAFT: relay reconnection after a real (provider-side) connection loss — the only netlab coverage of the slow reconnect path. | Keep as draft. |

---

4. Bonus root-cause fix delivered this session — NetLabManagerServer #54 (MERGED)

While investigating "167 leaked droplets" (which turned out to be a registry phantom-entry leak, not 167 live VMs — only ~10 real droplets existed), I found the root cause and fixed it:

• Bug: NetLabManagerServer.WorkerRegistry only removed an environment entry on an explicit client delete(). Nothing pruned entries when a droplet was auto-terminated by DigitalOcean (no notification to the manager), when a provision FAILED, or on manager restart. No reconcile/sweep existed → ~157 phantom READY entries accumulated.
• Fix (#54, MERGED, coord 0.0.39): new ReconcileService.reconcileOnce() lists live droplets and prunes registry entries with no matching live droplet (by id or netlab-worker-<env> name), guarded by a 15-min grace period (never prunes fresh CREATING_DROPLET entries) and fail-closed if listDroplets() throws. Runs at startup + every 5 min. TDD (failing test first), 31/31 tests, all CI green.
• Note: the DO droplet quota errors (netlab test "exceed droplet limit") were a separate capacity issue, NOT this leak — phantom registry entries don't consume quota. The user later confirmed there's usually plenty of droplet capacity; the earlier quota failures were transient bursts.

---

5. NEXT STEPS (in priority order — enough detail to drive to completion)

1. Restore the buildtest-server-service (the master blocker; owned by the parallel CI/CN workstream — coordinate first).

   • Symptom: SJVM ...uploadChunk timed out after 120s on every large-repo build (UrlResolver main + all 7 PRs). buildtest.kotlin.build intermittently down.
   • Investigate per the investigate-broken-service skill (OOM / libp2p gossip+mux-stream RPC leak in buildtest-server-service — see memory buildtest_server_gossip_leak_breaks_servicebridge_rpc, urlprotocol_libp2p_mux_stream_leak_per_rpc). Root cause is upstream in UrlProtocol; a url://buildtest/ restart via the ContainerNursery CLI clears the congestion as a temp mitigation.
   • Definition of unblocked: a fresh UrlResolver main kotlin.build run goes green.

2. Once UrlResolver main is green, re-trigger the 7 UrlResolver PRs — STRICTLY SERIALLY.

   • Concurrent triggering re-congests the uploadChunk path (that's what reddened them repeatedly). Fire one PR's kotlin.build, wait for terminal, then the next.
   • Trigger each via gh pr comment <n> --body "/retry" and gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-runs/<id>/rerequest.
   • There is also a known rotating single-test flake on the shared droplet (a different ~1 test fails per run, e.g. appendBuildRuleResultConcurrentWritesDoNotLoseRules); retry up to ~3× per PR. Never skip/weaken a test to get green.
   • Ready-to-use driver script pattern is in the session; or re-create a serial loop that rerequests + waits per PR.

3. Merging the 7 PRs (maintainer decision — NOT yet authorized).

   • Merge #631 before #632 (stack). #605 (source fix) is the highest-value. The netlab test PRs were historically "leave open"; get explicit approval before merging.

4. W3WalletTests #12 cleanup (merged, but red non-kotlin.build checks).

   • "Run Playwright suite" was red only on testClientSecretAuthentication.spec.ts because CI built the daemon from main before the daemon client-secret stack merged. Now that #108 is merged, a fresh W3WalletTests main run should build a daemon with the feature → the spec should pass. Re-run and confirm.
   • "NetLab NAT-traversal e2e" is red on W3WalletTests main too (pre-existing / capacity) — separate from this task.

---

6. All relevant PRs + current status (verified 2026-07-05)

| PR / Issue | Status | |---|---| | UrlResolver #605 · #608 · #616 · #631 · #632 · #643 · #595 (draft) | OPEN, rebased, RED (buildtest infra only — not the PRs) | | NetLabManagerServer #54 | MERGED (reconcile fix) | | BuildTestEmbedded #218 | MERGED (+ published 0.0.198) | | W3WalletDaemon #106 / #107 / #108 | MERGED | | W3WalletDaemonLauncher #12 · W3WalletExtension #51 · W3WalletTestHarness #2 | MERGED | | W3WalletTests #12 | MERGED (Playwright + NetLab e2e checks need a post-merge re-run — see step 4) | | BuildTestServerService #154 | CLOSED (obsolete/superseded) | | UrlResolver #601 · #603 · #519 · UrlProtocol #305 · W3WalletDaemon issue #109 | CLOSED (obsolete, during audit) |

---

7. Uncommitted/unsaved work

None. All work is already pushed to the PR branches listed above (verified: every local working clone had 0 dirty files and 0 unpushed commits). Nothing is stranded locally; the linked PR branches ARE the deliverables.

---

8. Key gotchas for whoever picks this up

• buildtest uploadChunk 120s SJVM timeout = buildtest-server congestion, not your code. Confirm by checking whether the repo's own main is red on the same error before touching a PR.
• Serialize UrlResolver kotlin.build triggers — concurrent triggers re-congest buildtest and cause cascading uploadChunk timeouts / getBuildRun interrupts.
• #632 is stacked on #631 — don't merge out of order; don't duplicate the netem helpers.
• #605 carries a real source fix (coord 0.0.560) not yet in main — treat it as more than a test PR.
• The "167 droplets" alarm was a registry phantom leak (fixed via #54), NOT live VMs; real droplet list via digitaloceandropletcli:digitalocean-droplet-cli list --json (strip ^\[[A-Z] P2P log lines before parsing).
• Related parallel-session handoffs in this folder cover the buildtest/CN outage + netlab OOM — read them before touching shared infra.

---

9. 2026-07-10 session addendum (second drive-to-completion attempt)

A follow-up session re-drove this handoff on 2026-07-10 (14:40–21:45Z). State as of writing:

Resolved since 2026-07-05

• #643 is CLOSED as superseded (2026-07-06, verified legitimate) — its retry/hardening landed on main in stronger form (uploadNetlabClasspathJars dedup + jittered backoff). Do not resurrect.
• Step 4 (W3WalletTests) is fully resolved — W3WalletTests main went all-green 2026-07-07, including the Playwright suite AND the NetLab NAT-traversal e2e (fixed via W3WalletTests #29). Nothing left there.

Re-rebase (2026-07-10) — all 6 remaining PRs sit on main 0359ae41

Supersession re-audit: all 6 survive (#605's encodeResultAsString fix is still absent from main; no test-name collisions). Rebased tips (force-pushed for #605 only; the other 5 are LOCAL-ONLY, see Next steps):

| PR | New tip | Rebase notes | |----|---------|--------------| | #605 | 32d9c2fb (PUSHED) | build.kts version-history conflict; fix entry re-slotted as coordinate 0.0.615 (main is at 0.0.614) | | #616 | d8e50c10 (local) | clean rebase | | #631 | ff419544 (local) | main now has its own netem helpers — dropped the PR's duplicates, adapted tests to main's API (setPacketLossRate->setPacketLoss, exit codes asserted via assertEquals(0, ...)), added missing setRateLimit in main's idiom; kept apply(prune=) + pollResolveRepeatedly | | #632 | c9f749da (local) | re-stacked on new #631 tip; same test adaptations | | #608 | 4dc43459 (local) | kept main's evolved block + its new countMarker/awaitMarkerCount; its apply(prune=) made byte-identical to #631's so either merges cleanly after the other | | #595 | e3238124 (local) | clean rebase, keep draft |

Pin sync (REQUIRED, not cosmetic): every new test file's @file:WithArtifact pins were aligned to main's canonical set (protocol 0.0.337, SJVM 0.0.40, jvm-libp2p snapshot-9, clocks 0.0.11, observable 0.3.12, logging 0.0.3, annotations 0.0.8) — kompile's global test scanner fails the whole suite on cross-file pin conflicts (see open UrlResolver #734, which unifies to the same versions).

Local validation: all 5 source-touching branches compile clean via buildMaven; #605's two regression tests (direct + relay path) PASS locally end-to-end; #605's bld-build full-suite attempt completed 1064/1067 with only live-netlab infra flakes. The code is sound.

The new blocker: buildtest-server instability (NOT uploadChunk congestion)

The 2026-07-05 uploadChunk blocker is gone. The 2026-07-10 blocker is the buildtest service OOM-restarting and its watchdogs killing HEALTHY runs. Five consecutive kotlin.build attempts on #605 head 32d9c2fb died to five distinct infra mechanisms:

1. 004acb84 — consumer RPC to url://buildtest/ closed mid-getTestResults during resume.
2. 0090faa9 — hard 180-min CI timeout (run survived a load-89 host window, a mid-run service restart, and a shard's DO droplet-create rate-limit wait; 756 test events, only the known testNetlabP2PIntegration live-infra flake errored).
3. 19039c11 — lease-staleness watchdog killed the run while shards were verifiably testing (107 events completed seconds before the kill); the OOM-restart severed lease renewal.
4. 2faa6840 — provisioning watchdog killed it while a shard re-provision sat in the DO rate-limit queue; 853 test events, ZERO failures at kill time.
5. 12a95f08 — status poller got UNKNOWN_METHOD - No service registered for identifier: buildtest continuously for 5 min (service dropped off the url:// registry).

Full forensics + suggested fixes filed for the infra owners: challenges/2026-07-10-1944-buildtest-server-oom-restart-cycle-kills-healthy-in-flight.md. The preserved OOM heapdump is /root/ContainerNursery/heapdumps/java_pid791492.hprof on prod. DO account droplet-create quota windows observably reset at :31 past the hour; retries themselves consume that quota, so blind rapid retries are counterproductive.

Next steps for pickup

1. Wait for (or fix) buildtest-server stability — coordinate with the CI/CN workstream; the challenge report above names the defects. Definition of ready: a kotlin.build run completes without watchdog interference (a green run on any large repo suffices).
2. Re-trigger #605's suite (gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-suites/78748550382/rerequest after warming https://github-webhooks.wasmserver.com/ 3x), watch with build-watchman 0.0.11. Also re-run its bld-build (gh run rerun 29104352877 --repo CodexCoder21Organization/UrlResolver --failed) AFTER kotlin.build is green (both consume DO quota).
3. Push the 5 local rebased branches serially as each predecessor goes green (--force-with-lease), order: #616 -> #631 -> #632 -> #608 -> #595. The local clone with all rebased branches lives at /code/workspace/UrlResolver in the 2026-07-10 session sandbox; if lost, redo per the rebase notes above (2-3h of work, mechanical given the table).
4. Merging remains a maintainer decision — #631 strictly before #632 (stack); #605 carries the real source fix (coordinate 0.0.615).