HHandoff
← Priority list

Handoff: W3Wallet NAT-flow regression reds netlab-e2e — fix code-complete across 4 PRs, blocked on UrlResolver #761 CI + a clean hosted validation

Original user request, verbatim: **"Proceed with [challenge 2026-07-11-1223 — netlab-e2e on w3wallettests main is red in both 2026-07-11 runs] now."** i.e. root-cause and durably fix whatever makes the across-NAT `netlab-e2e.yml` workflow fail on [W3WalletTests](https://github.com/CodexCoder21Organization/W3WalletTests) `main`. Per the global CLAUDE.md division of labor, implementation was delegated to codex gpt-5.6-sol; the orchestrator defines the problem and does the final review.

hf-2026-07-11-w3wallet-nat-flow-regression-reds-netlab-e2e

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: W3Wallet NAT-flow regression reds netlab-e2e — fix code-complete across 4 PRs, blocked on UrlResolver #761 CI + a clean hosted validation

Written: 2026-07-12 (effort began 2026-07-11).

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time (2026-07-12 ~17:00 UTC). PRs merge, CI re-runs, branches get force-pushed, the netlab-manager and buildtest fleet recover/degrade hour to hour, and DigitalOcean droplet quota shifts. A fresh query always wins over this document. Re-verify before acting:

  • PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • Published artifact: curl -s https://kotlin.directory/api/list-artifacts | grep 'foundation.url:resolver'
  • netlab-manager health (before any hosted validation): coursier launch --main-class netlabcli.MainKt netlabcli:netlab-cli:0.0.21 -r https://kotlin.directory -- --service-url url://netlab-hosted/ health
  • Last green vs red netlab-e2e runs: gh run list --repo CodexCoder21Organization/W3WalletTests --workflow netlab-e2e.yml --limit 20 --json databaseId,conclusion,createdAt,headBranch,event

1. Mission summary

Original user request, verbatim: "Proceed with [challenge 2026-07-11-1223 — netlab-e2e on w3wallettests main is red in both 2026-07-11 runs] now." i.e. root-cause and durably fix whatever makes the across-NAT netlab-e2e.yml workflow fail on W3WalletTests main. Per the global CLAUDE.md division of labor, implementation was delegated to codex gpt-5.6-sol; the orchestrator defines the problem and does the final review.

Where it stands: the fix is code-complete across 4 open PRs (all on branch fix/netlab-nat-regression), but it is NOT merged and NOT yet proven end-to-end. The single authorized hosted validation run was infrastructure-blocked on 4 of 7 suites, and the UrlResolver PR's aggregate CI is red (claimed unrelated — see §4). Two user/next-agent actions gate completion: (a) triage/clear UrlResolver #761 CI, (b) one clean hosted netlab-e2e validation.

2. Root cause (VERIFIED — and note the mid-investigation correction)

The original challenge's diagnosis was WRONG and was formally corrected. challenge 2026-07-11-1223 claimed the redness was netlab environment provisioning timing out under DigitalOcean quota contention (environment '…' did not become READY within 18 minutes). That is not the cause. Verified from the raw Actions log archives and the netlab-manager logs on the prod host:

  • The did not become READY strings are jest code-context frames printed around each console.warn at netlab-lifecycle.ts:109 (the throw at line 112 shows as surrounding source context for every probe warning) — no such error was actually thrown.
  • The environments provisioned fine and reached READY (manager logged Environment w3w-… is READY; one was even reaped later as READY-but-idle).

Correction filed and merged: challenge 2026-07-11-1532.

The real cause is a reproducible W3Wallet NAT-flow regression introduced by the 2026-07-08 billing/permission rollout across the W3Wallet family (which the netlab-e2e workflow clones at their mains at run time, so those repos change the gate's behavior without any W3WalletTests commit):

Last GREEN main netlab-e2e run: 2026-07-07T18:16Z (run 28888596730). The tell that this is the right class of bug: JVM-demo #96 "Billing failure must not mask a healthy wallet connection on page render" (merged 07-09) fixed exactly the browser-JVM flow — the one flow that recovered on its own — while the other three flows never got the equivalent fix.

3. What was found and done (the chain)

  1. Corrected the diagnosis (§2): jest code-frames misread as errors; provisioning/quota ruled out via raw logs + manager logs; correction challenge filed.
  2. Round 1 (acceptance run 29164911894) took the suite from 2-3/6 → 5/7 passing. Fixed nat_traversal_basic, nat_relay_unavailable, and kept the JVM flows green via the shared-relay-transport + lazy-approval + relay-readiness work. 2 remained: test_daemon_restart_reregisters (after daemon restart the new peer ID never propagated through the relay) and test_browser_nat_js_demo (coin flow).
  3. Round 2 root-caused the daemon-restart failure into UrlResolver (not the demos): persistent RPC handed peer-scoped /p2p-circuit/ addresses to the native TCP/libp2p dialer, which has no usable circuit-v2 reservation for them, so a correctly-resolved NAT'd daemon failed with NothingToCompleteException instead of using the application-level relay-forwarding path; and relay-forward failures were misclassified as server errors so the persistent connection never re-resolved the restarted daemon's new peer ID. This is the durable root-cause fix (UrlResolver #761). The JS coin flow was fixed by installing event handlers before the UI publishes its connected/readiness state (JS demo #18).
  4. Published foundation.url:resolver:0.0.634 (approved by the orchestrator; additive upstream artifact per CLAUDE.md). 0.0.632/0.0.633 were taken by other operators, so the next-free-patch rule selected 0.0.634; local+remote JAR/POM SHA-256 matched.
  5. Ran exactly ONE authorized hosted validation (run 29178784649) — 3/7 suites passed; 4 were infrastructure-blocked, not code-failed (see §5). The netlab-manager was too unhealthy for a final env listing, leaving a possibly-leaked env w3w-browser-nat-jvm-1783831629459. Recorded as challenge 2026-07-12-0505. At write time (2026-07-12 ~16:57 UTC) the netlab-manager is HEALTHY again (latency ~2s) — the 07-11 03:49 failures were a transient manager/buildtest incident.

Nothing is unsaved. All work is pushed to the four fix/netlab-nat-regression branches (local codex worktrees under /code/workspace/workspace/*-round2 are clean and disposable). No build artifacts committed.

4. Relevant PRs / refs (write-time state 2026-07-12 ~17:00 UTC — RE-VERIFY)

All fix PRs share branch fix/netlab-nat-regression so the netlab-e2e workflow's same-branch sibling checkout pairs them in one run.

| PR | Purpose | Write-time state | |---|---|---| | UrlResolver #761 "Fix circuit-only persistent RPC through application relays" — the root-cause fix; head SHA 5de61706e034e73fd658ca48e582861f5651dbbb | Circuit-only peers use application-level relay forwarding; relay-forward failures classified recoverable → reconnect re-resolves; String results encoded as JSON consistently; legacy JVM ABI preserved. Ships resolver:0.0.634. | OPEN, CI RED (bld-build + kotlin.build (remote) fail), mergeStateStatus BLOCKED | | W3WalletJvmServerSideCoinCollectorDemo #98 | Typed wallet reads + billing RPC share one relay-aware UrlProtocol2; pins resolver:0.0.634. 67/68 tests (1 intentional browser skip). | OPEN, CI GREEN (CLEAN) | | W3WalletJavascriptCoinCollectorDemo #18 | Install event handlers before UI publishes connected/readiness state; adds Playwright regression suite. | OPEN, CI GREEN (CLEAN) | | W3WalletTests #32 | Test orchestration: unattended permission approvals, capability-type counting, bounded P2P-readiness polling, same-branch sibling checkout, self-describing 18-min readiness-deadline error. | OPEN, non-netlab checks green; mergeStateStatus BLOCKED on the NetLab NAT-traversal e2e required check | | foundation.url:resolver:0.0.634 | The published resolver root-cause fix (matches #761 source). | Published, immutable. Consumers pin 0.0.634. (Latest published is now 0.0.637 from unrelated operators — do NOT assume "latest" == our fix.) |

On UrlResolver #761's red CI (codex's "unrelated" claim — substantiated but UNVERIFIED by the orchestrator): the PR body reports the remote aggregate named 3 failing tests: testPeerScopedAliasCircuitOnlyUnreservedRelayReproducesNothingToComplete (the branch's own regression test — passes when selected directly on the same rebased head, i.e. flaky only under aggregate load), plus pre-existing flaky stress tests testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup, testStressPeerExchangeRaceCondition, and stressTestRegistrationAfterWithdrawalRaceUnderContention. Two unrelated-breakage challenges back this up: 2026-07-12-0217 (3 newly-merged-to-main tests don't compile — absent helper APIs pollResolveRepeatedly/setRateLimit) and 2026-07-12-0437. This needs independent confirmation before merge — a red PR does not merge; see next steps.

5. The single hosted validation's per-suite result (run 29178784649)

Overall 3/7 suites, 4/8 tests passed, but 4 were infrastructure-blocked before reaching their target assertions, not code-failed:

| Suite | Result | |---|---| | Basic NAT traversal | ✅ Passed | | Browser staging integrity | ✅ Passed | | Lifecycle diagnostics | ✅ Passed (2 tests) | | Daemon restart | ⛔ Infra: manager discovery returned zero peers | | Relay unavailable | ⛔ Infra: chunk upload interrupted | | Browser JVM | ⛔ Infra: repeated manager bytecode timeouts during upload | | Browser JavaScript | ⛔ Infra: manager bytecode timeout during env creation |

So the fix is not disproven — the suites that ran passed — but the ones that specifically exercise the resolver + JS fixes (daemon restart, browser JVM/JS) never got a clean shot. A fresh validation on a healthy manager is the acceptance gate.

6. Next steps (ordered — RE-VERIFY §4 first)

  1. Re-verify all of §4 with the banner commands. In particular re-check whether #761's CI is still red and whether the manager is healthy.
  2. Triage UrlResolver #761's red CI. Confirm codex's claim that all 3 aggregate failures are unrelated/flaky (the branch's own regression passes when selected directly; the other 3 are pre-existing stress-test flakes per challenges 2026-07-12-0217/0437). If confirmed, a rerun of the remote check may go green; if a failure is genuinely the fix's fault, route it back to codex gpt-5.6-sol (do not hand-fix — CLAUDE.md division of labor). The PR cannot merge red.
  3. Once #761 is mergeable AND the netlab-manager is healthy, dispatch ONE fresh hosted netlab-e2e validation on the branch: gh workflow run netlab-e2e.yml --repo CodexCoder21Organization/W3WalletTests --ref fix/netlab-nat-regression. Watch it (~90-150 min). Success = all 7 suites pass, especially the 4 previously infra-blocked (daemon restart, relay unavailable, browser JVM, browser JS). Respect the one-run-at-a-time droplet-budget discipline; do not spam full runs.
  4. If the validation is green, the 4 PRs are ready for the USER's merge decision (do NOT merge without explicit approval — code PRs are not pre-approved). Merge order matters: UrlResolver #761 first (consumers depend on resolver:0.0.634, already published so no republish needed), then the three consumers #98 / #18 / #32. After all merge to their mains, re-run netlab-e2e on W3WalletTests main to confirm the gate is green there (main clones the consumer mains at run time).
  5. Clean up leaked netlab droplets. The 07-11 run flagged w3w-browser-nat-jvm-1783831629459 as possibly leaked (couldn't be targeted while the manager was unhealthy). Now the manager is healthy: netlabcli … list for stray w3w-* envs and netlabcli … delete <name> any orphans (each is a live DO droplet costing money). Also check /root/ContainerNursery/netlab-manager-state.json on the prod host for stale entries.
  6. Cancel this session's status-report cron loop. A /status-report recurring cron job (created this session) may still be firing every 20 min — CronList then CronDelete <id> if a new agent inherits the session, or it dies with the session.

7. Reusable / operational knowledge

  • The netlab-e2e workflow clones the W3Wallet family at their MAINS at run time (W3WalletDaemon, W3WalletExtension, both demos) — so a regression can enter via any of those repos with zero W3WalletTests commits, and a fix branch validates them together only via the same-branch sibling checkout (push the fix to identically-named fix/netlab-nat-regression branches in every repo touched). This is why all 4 PRs share one branch name.
  • jest code-context frames are not errors. The actual thrown error is the text directly under the ● test header; the >-arrowed throw new Error(...) lines are source context printed around every console.warn on the same function. Misreading them sent the original challenge down the wrong path — always read the raw log archive (gh api repos/<o>/<r>/actions/runs/<id>/logs, unzip with python zipfile, look under the ● markers).
  • Resolver published-version races: other operators publish foundation.url:resolver:0.0.6xx frequently; "latest published" is NOT our fix. Our fix is pinned at 0.0.634 by the consumer PRs; it is immutable. Don't bump consumers to a newer number without confirming that number contains this fix.
  • codex gpt-5.6-sol long-session pattern: the round-2 session ran ~6h and paused at an artifact-publish approval gate; a continuation task file (grant approval + finish) re-dispatched it cleanly from its worktree state. Usage limits gate NEW session starts (retry every 10 min behind a wrapper); already-running sessions continue past the limit. See the codex-gpt55-invocation memory.
  • netlab-manager on prod: url://netlab-hosted/ (route key url:netlab-hosted: in ContainerNursery), state persisted at /root/ContainerNursery/netlab-manager-state.json, MAX_LIVE_ENVIRONMENTS=60, ENV_IDLE_TTL_MINUTES=30, provisions DO droplets on demand. Health-check with netlabcli … health (manager liveness only — do NOT create/apply in a health check; those provision real droplets). It was transiently unhealthy 2026-07-11 03:49 (discovery zero-peers + bytecode-upload timeouts) and recovered by 2026-07-12 ~16:57Z.
  • Do not --admin/bypass CI. UrlResolver #761 must go green (or its red be proven infra/flake and rerun green) the honest way.