Deploy the merged Handoff archived-list crash fix and list-page fix (pending approval), and land HandoffApi PR 19

RE-VERIFY: everything below is a snapshot taken at 2026-10-14 20:16 UTC. Check each PR again before acting:

gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergedAt

Check what is deployed with ContainerNursery CLI 0.0.20 routes. Check buildtest health with https://buildtest.kotlin.build/log/raw?id=<run>.

Mission

The user asked: "Fix the Archived list crashes the service and also fix The list page sometimes fails." Both bugs are in the Handoff service (url://handoff/) and its web interface at https://www.handoff.wasmserver.com/.

Status: both fixes are merged to main. Neither is deployed. The user has not approved a production deploy, so ask before deploying. One documentation-only PR, HandoffApi PR 19, has not merged yet.

What was found and done

1. Archived-list crash

Symptom. Calling listHandoffs(archived=true), for example with handoff-cli list --archived, made the service JVM die with OutOfMemoryError at -Xmx128m. ContainerNursery then restarted it, which dropped all live claims.

Mechanism (confirmed). HandoffEmbedded.listHandoffsJson put every archived handoff's full Markdown body into one reply:

• 871 archived handoffs, about 10.1M characters in total, which is about 20 MB as UTF-16 because 532 of them contain non-Latin-1 text.
• The transport (Libp2pRpcProtocol.encodeMessage → JSONObject.quote) then copied that string into a frame StringBuilder that grew from 19 MB to 38 MB.
• Even with enough heap, the 10.28 MB frame was over the client's 10,000,000-byte frame limit.

Evidence.

• The original 2026-10-07 heap dumps had been pruned, so the investigator reproduced the crash locally from a copy of production's state.json.
• An owner walk of a -Xmx96m dump shows a 20.28 MB listing string and an 18.87 MB frame buffer. The OOM stack is in jsonWireString.
• After the fix, the same production data gives a 0.93M-character reply, and the server survives at both 96m and 128m.

Fix. Listings, both priority lists and the archive, now return body: "". Full bodies come from getHandoffJson. listAllOpenHandoffsJson is unchanged and still returns bodies.

Tests.

• HandoffEmbedded testListingsCarryNoBodiesAtProductionArchiveSize: fails 4/4 before the fix, passes after.
• HandoffServiceServer testArchivedListingUnderProductionHeap: real child process at -Xmx128m, real url:// client. Fails 3/3 before, passes after.
• Two git-sync tests now read bodies with getHandoffJson. They make the same full-text comparison as before.

Consumer review: no data loss.

• Git sync uses listAllOpenHandoffsJson and getHandoffJson.
• The web interface's archive and home pages render only title and summary.
• The editor loads the full body with getHandoffJson.

2. List page sometimes fails

Symptom. About 1 in 12 loads of /lists/default showed "Priority list unavailable" with AmbiguousRpcRequestException … closed while requests were still pending … EOFException.

Mechanism (confirmed). The web interface shipped foundation.url:resolver:0.0.1203. When a gossip child that borrows the shared root connection to the provider fails or takes more than 1 s to negotiate, that version calls conn.close() on the root, killing pending url://handoff/ requests.

• jdb attached to the deployed jar stopped at that close, on the 1 s timeout.
• Production sockets show the web interface closing the connections.
• All failures happened within about 5 minutes of a web interface process start.

Fix. Already fixed upstream in resolver 0.0.1229 (https://github.com/CodexCoder21Organization/UrlResolver/pull/1105), which uses retireWhenIdle instead of close(). The PR moves the web interface to the service's transport family: resolver 0.0.1263, protocol 0.0.527, jvm-libp2p snapshot-27, observable 0.3.22. Web interface version is 0.0.29.

Test. testShippedTransportKeepsPendingRequestWhenGossipToProviderFails fails on 0.0.1203 and passes after the bump.

Second cause also removed. The old protocol pin, 0.0.509, was inside the 0.0.503–0.0.515 range that had an admission-close bug, fixed in 0.0.516 (https://github.com/CodexCoder21Organization/UrlProtocol/pull/579). So any drop in the failure rate after deploy cannot be credited to the gossip fix alone. The stalled-root closer retireIfStillStalled in Libp2pHostFactory was not ruled out.

Relevant PRs

Published artifacts: handoff:api:0.0.27 and handoff:embedded:0.0.36, built from the PR heads above. Do not push new commits to HandoffApi PR 19, so that the published 0.0.27 keeps matching what merges.

All five PRs passed an independent Opus adversarial review at these heads. Nothing remains unpushed locally.

Deployed vs merged (checked live at 20:16 UTC)

• Service route url:handoff: runs /root/ContainerNursery-uploads/jars/handoff-service-server-f294259c-20261014.jar. That build is HandoffServiceServer f294259 (PR 30, 0.0.28), deployed earlier today by someone else. It does not contain the crash fix, which is in e687597.
• Web interface route www.handoff.wasmserver.com:443 runs /root/ContainerNursery/apps/handoff-wui.jar, which predates PR 36. It does not contain the list-page fix.
• Do not call handoff-cli list --archived against production until the service is deployed. It still crashes the service.

Next steps

1. Ask the user whether to deploy. If approved:
   1. Back up /root/handoff-data on 198.199.106.165. HardwareControlFabric certs are missing on the /code box, so SSH (-p 23) was used last time.
   2. Build the HandoffServiceServer fat jar locally from main at e687597 or later. Remote builds can't return artifacts over 32 MiB.
   3. Point route url:handoff: at a new jar name, using ContainerNursery CLI 0.0.20 update-route --facade url --domain handoff --image <new>, then run upload-jar --route url:handoff:. Keep the f294259c jar for rollback.
   4. Verify the service: handoff-cli health, a handoff count unchanged at about 93 open, and list --archived returning without the service restarting.
   5. Build the HandoffWui fat jar from main at b7f5243 or later. Save the current handoff-wui.jar on the server as a rollback copy, then upload it to www.handoff.wasmserver.com:443.
   6. Load /lists/default at least 40 times and confirm no "Priority list unavailable". Also screenshot / and /lists/default.
2. Land HandoffApi PR 19. If GitHub evicts it again, re-enqueue with build-watchman 0.0.21 --to-merged --idle-timeout-minutes 300 once buildtest's queue is short.
3. Follow-ups (not requested):
   • listAllOpenHandoffsJson (HandoffEmbedded.kt:1511) still returns every open body in one reply. HandoffWui AgentPage.kt:98,105 calls it and doesn't need the bodies.
   • getHomePageProjection may build full records too. Not checked.
   • The investigator suggests the web interface's /archive page may have caused some list-page failures by crashing the service. Not verified against logs.

Infrastructure notes

BuildTest restarted twice on 2026-10-14, around 15:42 and 17:05 UTC.

• Each restart sat about 25–30 min in RECOVERING_LOCAL_STATE.
• Each one lost every in-flight run: GitHub still showed the checks as in_progress while /log/raw said "Build run not found".
• After the first restart, droplet creation stalled (createDropletAsync 30 s, listDroplets 60 s).
• Stale reservations again held 48–90 of the 200 slots.

What worked: re-request each lost check suite once BuildTest is healthy, after warming https://github-webhooks.wasmserver.com/.

Re-requesting a suite cancels the old run before the replacement appears. build-watchman reads that cancelled state as RED, so start the watcher only after the new run shows in_progress.

Challenge record: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-14-1647-buildtest-cannot-create-build-droplets-after-its-2026-10.md

Reusable knowledge

• Version availability on kotlin.directory: a version exists only if the POM body contains <artifactId>. The URL returns 200 either way.
• Stale HandoffEmbedded builds: HandoffEmbedded tests can load a stale build from ~/.cache/coursier/v1/bldbinary/repository/handoff/embedded/<v>, even with a fresh --cache-location. Delete that directory before measuring.
• Editing and creating PRs: gh pr edit and gh pr create fail on these repos. Use the REST API (gh api -X PATCH .../pulls/<n> -F body=@file).
• Investigation notes: /tmp/claude-1000/-code/e0a3189d-6141-446f-b037-169c14104a96/scratchpad/out/oom-findings.md and out/rpc-findings.md. These are local and may be cleared; the key facts are copied above.