Handoff: w3wallet/LambdaServer resolver release chain — blocked on the shared buildtest executor, not on code

Written: 2026-07-21 ~23:40 UTC. This continues (and supplements — does not replace) the paused 2026-07-20-w3wallet-lambdaserver-resolver-release-chain.md and the three handoffs it lists at its top.

RE-VERIFY EVERYTHING BEFORE ACTING. Every state below is a write-time snapshot and drifts fast — over the ~10 hours before this was written, foundation.url:protocol advanced 0.0.393 → 0.0.397 and foundation.url:resolver reached 0.0.653, all from parallel lanes, while this chain's PRs did not move. Authoritative re-checks: gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt per PR; curl -s https://kotlin.directory/foundation/url/<protocol|resolver>/maven-metadata.xml | grep latest for published artifacts; gh api graphql for the UrlResolver merge-queue contents (query in "Reusable knowledge"). A fresh query always wins over this document.

Mission summary

Drive three older PlanRepository handoffs to genuine completion by delegating the heavy lifting to parallel codex gpt-5.6-sol lanes, per the user's standing request to "drive [the 2026-07-20 handoff] to completion." The user pre-approved publishing and merging for this workstream's own PRs; every PR targets main/master directly. The chain, in order:

1. Land the owning-layer relay-race fix at the UrlProtocol layer and publish the protocol artifact.
2. Re-pin, validate, and land the UrlResolver release (#801) plus its train (#816, #818); publish the resolver artifact; close superseded PRs.
3. Re-pin LambdaServerWui to the final resolver, open its PR, merge, publish LambdaWui:0.0.8.
4. Fix + land W3WalletTests #32 then #30.
5. Deploy the layered LambdaServer production topology + canaries + ProductionHealth; merge the docs branches.
6. Once all acceptance conditions are live-verified, delete all four handoffs.

What was found and done (the chain)

✅ Protocol layer — DONE

• UrlProtocol #392 merged 2026-07-21 03:56 UTC. Its original serialized-relay-registration code fix was subsumed mid-rebase by a parallel lane's UrlProtocol #393 (merged 2026-07-20 23:37 UTC, published foundation.url:protocol:0.0.393), which lands the authoritative relay lifecycle fix (doRegisterWithRelay now @Synchronized; authoritative ServiceWithdrawal gets a bounded 5 s reliable-delivery budget; a client keeps its relay listener until the relay ACKs unregister commit; a stale controller's no-op unregister can't supersede a pending replacement registration). #392 was therefore reduced to just its concurrency regression test (tests/testConcurrentScheduledReconnectCannotClearExplicitRegistration.kts, commit cd36d4b6…), validated locally 10/10 focused + 5/5 withdrawal-stability, and shepherded to merge.
• The mechanism (verified, from the PR bodies + codex logs): a scheduled relay reconnect called the private registration routine without the synchronization explicit registration uses, so an older parked retry could clear a newer live relay connection when it later failed. The fix serializes the whole listener/keepalive/connection replacement transaction for all callers.
• At write time protocol latest is 0.0.397 (parallel lanes kept publishing); 0.0.393 is what this chain consumes.

🔴 Resolver layer — BLOCKED on the executor, NOT on code

• UrlResolver #801 — OPEN/BLOCKED, branch fix/netlab-nat-regression, head a2d6eec7…. Carries: the protocol-0.0.393 re-pin (was 0.0.384; codex bumped the pin across build + README + 1,117 test files, commit 1637f8c6…), bug (d)'s gossip-fanout reservation fix ("reserve only peers actually selected, not during eligibility checking"), and a relay-aware raw RPC fallback made at-most-once and deadline-bound (4 commits force-pushed onto the re-pin by a co-driving lane: 1613c85a add fallback, 7fe65922 at-most-once+deadline, f0fce12f candidate-agnostic regression, a2d6eec7 align regressions w/ canonical protocol+release). Its bld-build check is SUCCESS; its required kotlin.build (remote) check is FAILURE — the sole remaining blocker.
• UrlResolver #816 — OPEN/CLEAN, branch fix/netlab-bulk-rpc-attachments, head de01107d…. Bulk-RPC binary-attachment work (streams large payloads over the protocol's memory-accounted attachment path so they can't OOM small-heap consumers). At write time both its checks are green (bld-build SUCCESS, kotlin.build (remote) SUCCESS) and it is CLEAN — but it is not currently in the merge queue (queue holds only #788, #819 at write time). It was evicted 3× earlier by checks_timed_out. It looks ready to enqueue — re-verify then enqueue it.
• UrlResolver #818 — OPEN/DIRTY. Carries bug (c)'s Any?/Object sandbox-unmarshalling fix; base was retargeted to main. Needs rebase onto current main after #816 lands.

The blocker, proven with evidence (six consecutive kills of #801's kotlin.build (remote) check):

1. Relay-dispatch SandboxException/UrlResolutionException errors against performancetest — a service that had been bounced by ContainerNursery restarts (uploadCoRunChunk … connection closed while requests pending).
2. A run frozen 24 min at 88/100 (host-confirmed both build.log and test-events.jsonl frozen) — deleted-CLI could not traverse the congested endpoint.
3. Two 180-minute build-timeout mass-fails (Build timed out after 180 minutes … 0 passed, 1134 failed) caused by the executor's Metaspace crash-loop consuming the budget across restarts.
4. Rerequest race: a fresh dispatch re-points the GitHub check at a new run, orphaning the run already executing — a run (4c0cd82a) that COMPLETED with 0 failures was discarded this way because the check had been reassigned to a newer run (2488efeb).
5. A new buildtest-server bug: Dynamic-dispatch event-ingestion/dispatch loop on shard N failed … java.lang.IllegalArgumentException: A dynamic per-test terminal event for spec '…' — the ingestion loop throws on a per-test terminal event and instantly fails the entire run (0 passed / 1134 failed).

Against all that, the code's own evidence is uniformly green: local focused oracles 15/15 (gossip) + 5/5 (relay-withdrawal) + 3/3 (restart/reattach); a codex triage of the three tests CI had flagged found them all executor-load flakes (testSendRpcRequestRelayFallbackHonorsTotalDeadline 4/5, the two stress tests 5/5 — see the run's TRIAGE.md, summarized in the PR comment thread); and CI runs that did execute reached 1132/1134 (2 triaged infra fails), then 709/0 COMPLETED, then 411/0 COMPLETED — repeatedly green, repeatedly discarded by the executor faults above. This is decoupling of check outcome from code quality, not a code defect.

Three challenge records were filed and merged to PlanRepository/challenges this session:

• 2026-07-21-0139-droplet-service-death-starves-entire-buildtest-fleet-cn.md — a dead lazily-routed CN url: app (droplet-service) is never resurpawned; buildtest-server's single reaper thread blocks dialing it and starves provisioning/deletion/admission fleet-wide.
• 2026-07-21-0623-buildtest-server-metaspace-oom-crash-loop-cn-s-hardcoded.md — root cause of the timeouts: ContainerNursery's danger_jarfile launcher hardcodes -XX:MaxMetaspaceSize=128m on the command line for every app; buildtest-server loads sandboxed test/bytecode classes per run so metaspace grows monotonically and it ExitOnOutOfMemoryErrors every ~60–110 min. A route-level JAVA_TOOL_OPTIONS override cannot win against the explicit flag.
• 2026-07-21-1312-buildtest-dynamic-dispatch-event-ingestion-crash-instantly.md — the ingestion-loop IllegalArgumentException + the rerequest-orphaning + duplicate-run-stacking that discards green runs.

Publishing status — needs verification

• foundation.url:resolver:0.0.653 is published at write time (pom present, HTTP 200) — but this chain's PRs #801/#816 have not merged (no UrlResolver PR merged since #812 on 2026-07-20 11:19 UTC), so 0.0.653 was published by a different lane/effort, and its contents almost certainly do NOT include #801/#816's fixes. There is even an older 2026-07-05-urlresolver-653-marshaling-salvage-and-flake-findings.md handoff referencing a 0.0.653. Do not assume the resolver release step is done. After #801 (and #816) actually merge, probe for the next free foundation.url:resolver version (0.0.654+ at that time) and publish that, feeding the real published version into the LambdaServerWui step.

Downstream — untouched, all waiting on the resolver merges

• LambdaServerWui WIP branch feature/three-source-registration-e2e (head was 4bed9c3 at the prior handoff; re-verify) — the three-source registration E2E. Re-pin to the final resolver, open its direct-main PR, merge, publish community.kotlin.lambdaserver:LambdaWui:0.0.8. Visuals verified UNCHANGED vs main — no webapp-screenshots needed.
• W3WalletTests #32 (OPEN/UNSTABLE) — its one red suite test_daemon_restart_reregisters is the downstream reproducer of the typed relay-delivery bug fixed upstream; expected fix is the resolver release + a workflow relay-harness pin bump + a clean 7-suite hosted NetLab gate, not new product code. W3WalletTests #30 (OPEN/UNSTABLE) — FU4 cross-NAT negative assertion, gated additionally on the shared host's 15-min load < 8.
• LambdaServer production-topology deploy + docs-branch merges (steps 9–11 of the 2026-07-18 closeout handoff) — not started.

Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | State @ write | Notes | |---|---|---| | UrlProtocol #392 | MERGED 07-21 03:56Z | reduced to test-only; the code fix rode #393 | | UrlProtocol #393 | MERGED 07-20 23:37Z | parallel lane; published protocol 0.0.393 | | UrlResolver #801 | OPEN / BLOCKED | bld-build SUCCESS, kotlin.build (remote) FAILURE; branch fix/netlab-nat-regression, head a2d6eec7…. Latest failing run 14669ad8. | | UrlResolver #816 | OPEN / CLEAN, both checks green | branch fix/netlab-bulk-rpc-attachments, head de01107d…; NOT in queue at write time — enqueue after re-verify | | UrlResolver #818 | OPEN / DIRTY | rebase onto main after #816 lands | | UrlResolver #797 | OPEN | superseded by #801 — close (comment linking #801) once #801 lands | | UrlResolver #802 | CLOSED already | — | | UrlResolver #804 | OPEN | superseded by #801 — close once #801 lands | | UrlResolver #806 | CLOSED this session | superseded; comment posted | | W3WalletTests #32 | OPEN / UNSTABLE | needs resolver release + pin bump + NetLab gate | | W3WalletTests #30 | OPEN / UNSTABLE | FU4 negative assertion; gated on host load < 8 | | PlanRepository #2482 | MERGED | challenge: droplet-service death | | PlanRepository #2492 | MERGED | challenge: metaspace crash-loop | | PlanRepository #2513 | MERGED | challenge: dispatch-loop crash |

No unsaved/uncommitted work is being dropped: both codex lanes committed and pushed their branches (protocol #392 head cd36d4b6… pushed; resolver #801 re-pin 1637f8c6… + the fallback commits through a2d6eec7… pushed). Local checkouts under /code/ws/{urlprotocol-392,urlresolver-801} are working copies only.

Next steps (exact order for whoever resumes)

1. Re-verify current state first (banner). Especially: is kotlin.build (remote) on #801 green now? Did #816 get enqueued/merged by another lane? Is a resolver version > 0.0.653 published?
2. The blocker needs a USER DECISION — this is why the effort is handed off, not just paused. #801's code is proven green; only the broken shared executor stops it. Options, in recommended order:
   • (a, recommended) Authorize an upstream infra fix (goes beyond the original merge-only pre-approval, so it needs explicit user sign-off): make ContainerNursery's per-route MaxMetaspaceSize configurable like memoryLimitMb (buildtest-server needs 512m+) instead of the hardcoded 128m, and fix buildtest-server's event-ingestion loop to skip/quarantine a malformed per-test terminal event instead of failing the whole run. Both are documented in the merged challenge records above. This is the durable fix — it stops the attrition for every consumer, not just this chain.
   • (b) Continued attrition: keep re-driving #801's suite on a freshly-restarted buildtest-server generation (a controlled restart via the CN CLI resets the metaspace budget; a clean run finishes in ~40 min). This is how #392 landed. But six attempts over ~8 h failed and the new ingestion-crash bug worsens the odds; do NOT blindly rerequest — each rerequest orphans the in-flight run and discards green results. Warm webhooks, rerequest exactly once against check-suite 80698636209, then leave it alone and watch to completion.
   • (c) User/another operator takes over the executor; stand down on the chain until it's healthy.
3. When #801 goes green: it auto-enqueues under --to-merged; confirm it merges. Then enqueue #816 (verify CLEAN first), rebase #818 onto the new main, land it. Close superseded #797 and #804 with comments linking #801.
4. Publish the resolver: probe https://kotlin.directory/api/list-artifacts (401 = masked-absent, so confirm the exact pom URL) for the next free foundation.url:resolver version at that time (0.0.654+; 0.0.653 is already taken by an unrelated publish), publish via the community-kotlin-maven-artifact-publishing CLI (--api-url api.kotlin.directory/upload; buildMaven() only builds locally — see the kotlin-directory-publish-mechanism memory), and record the actual published version.
5. LambdaServerWui: re-pin feature/three-source-registration-e2e to that resolver version, validate, open its direct-main PR, merge, publish LambdaWui:0.0.8 (no screenshots — visuals unchanged).
6. W3WalletTests: bump #32's workflow relay-harness resolver pin to the final version, rerun the 7-suite hosted NetLab gate to green, merge; then #30 (gated on shared-host 15-min load < 8, which never once cleared in the original session).
7. LambdaServer production topology + canaries + ProductionHealth per the 2026-07-18 closeout handoff steps 9–11; merge the DocumentationRepository + HostedServiceDocumentation lambdaserver branches.
8. Only once every acceptance condition is live-verified, delete all four handoffs (the three at the top of the prior handoff + the prior handoff itself + this one) via the create-handoff completion flow.

Reusable / operational knowledge

• Executor = 198.199.106.165. SSH last-resort: ssh -p 23 -i ~/.ssh/id_ed25519_20260706_035316 root@198.199.106.165. Read/append /root/cn-watchdog.audit before acting on the host; every intervention this session is logged there.
• Controlled buildtest-server restart (resets metaspace budget): coursier launch containernurserycli:container-nursery-cli:0.0.16 -r https://kotlin.directory -- restart --route-key "url:buildtest:" --url https://api.nursery.wasmserver.com — the CLI prints Failed: Unknown response format but the restart does take effect (verify the new buildtest-server-service.jar java pid's etime reset). CN CLI cannot force-start a dead lazily-routed url: app (Container not found) — only a full CN restart (watchdog) resurrects those.
• Ground-truth a run (frontend flaps 503 under load — that is NOT run death): stat -c "%y" /root/buildtest-data/runs/<id>/{test-events.jsonl,build.log} and read run.json status/testsPassed/testsFailed. A run is only wedged if both files are frozen >20 min. Runs survive service restarts ("Resuming build after service restart") — a restart mid-run means slower, not dead.
• Merge queue is mandatory; gh pr merge --auto is rejected. Enqueue via GraphQL: gh api graphql -f query='mutation($pr:ID!){enqueuePullRequest(input:{pullRequestId:$pr}){mergeQueueEntry{position state}}}' -f pr="$(gh pr view <n> --repo <o>/<r> --json id --jq .id)". Inspect queue: gh api graphql -f query='query{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:8){nodes{pullRequest{number} state position}}}}}'.
• gh pr edit --base fails with a GraphQL "Projects (classic) deprecated" error on some repos; retarget via gh api -X PATCH repos/<o>/<r>/pulls/<n> -f base=main.
• build-watchman (coursier launch buildwatchman:build-watchman:0.0.13 -r https://kotlin.directory -- --repo <o>/<r> --pr <n> --to-merged) 0.0.13 recognizes the "already in queue" enqueue rejection as success — safe to relaunch against an already-enqueued PR. Run it as a tracked background job and re-arm after any rerun; its exit is an event, not "still watching."
• Never launch two codex exec resume against the same session id concurrently — both exit 144.
• The two codex lanes ran the project-standard non-interactive invocation with per-lane clones under /code/ws/{urlresolver-801,urlprotocol-392}/; per-lane journals STATUS.md, TRIAGE.md, codex-*.log live in those dirs (working machine, not a repo).