Handoff site outage and cold-start work: HandoffWui listener fix rolled back after production acceptance failed

Written 2026-07-29 16:28 UTC — RE-VERIFY BEFORE ACTING

This is a write-time snapshot. Re-check pull requests with gh pr view, inspect the www.handoff.wasmserver.com:443 route with ContainerNursery CLI, and time a real cold request before changing production. The live service and route are authoritative.

Mission summary

The original work was to diagnose and repair https://www.handoff.wasmserver.com/, deploy the current interface, and reconcile the URL transport versions behind it. The relay fanout and modern-interface changes from the earlier work are merged. On 2026-07-29 this lane was specifically authorized to deploy the merged HandoffWui listener-before-backend cold-start change and prove that a first request receives HTTP bytes within a few seconds.

That deployment did not meet its production acceptance criterion and was rolled back to HandoffWui 0.0.3. Production is serving the real homepage again, and the route configuration was not changed.

What was found and done

1. HandoffWui listener fix is merged at 4e2d4bbef23275667d2eea6215a8f9597854db5a, with its remote check successful. Current main is b166b3c9acd5616d9c7734b77593b17bef6b9fc7 and contains that merge.
2. Current main was freshly cloned, fetched, and rebased. A local nice -n 19 fat-JAR build succeeded and produced 57,760,301 bytes with SHA-256 f470976fdc768955b2914f857c559a66f78ebe369d2dde09dee6cabea9ed8271.
3. Before deployment, production reported HandoffWui 0.0.3. Its first request after apparent idle returned the real 147,225-byte homepage in 20.440 seconds, with 20.290 seconds to first byte; the immediate follow-up took 3.619 seconds.
4. The mechanism in the merged WUI source is correct at process level: server.start() runs before a separate backing-service initializer thread. Public-HTTP tests hold initialization open and assert the exact 503 warming page with Retry-After: 5 in under two seconds, then prove the normal page is served after readiness.
5. The new artifact was uploaded only to www.handoff.wasmserver.com:443. ContainerNursery confirmed the exact SHA and restarted only that container. No route update was issued.
6. Production acceptance failed. The immediate external request timed out after 15.002 seconds with HTTP 000 and zero bytes. Container logs show Jetty did not start until about 24 seconds after upload completion/request start; ContainerNursery recorded the eventual correct 503 warming response only after 25,254 ms. A later homepage request also returned server-side 503 only after 30,085 ms, while /health returned 200 in 0.167 seconds after the process was up.
7. This separates two mechanisms. The WUI change works after the JVM begins executing it, but the external cold path is still delayed before Jetty is reachable, in ContainerNursery/JVM launch. The observed new first-byte time was worse than the version-0.0.3 baseline and violated “within a few seconds, never a zero-byte timeout.”
8. A rollback artifact was built remotely from exact version-0.0.3 source commit 2f699d27dc9f254b1698869286c3b6ca6cbf2f35. Buildtest run 0c6eafac completed successfully with 22 tests passed and zero failed. The normal artifact stream failed, so the exact output was recovered through resumable BuildTestApi.getArtifactChunk calls.
9. The recovered rollback JAR is 57,728,411 bytes with SHA-256 5d34495679e21711e397ecf71f66a149d5b5c27f56f190530d58994ef2b45976. Its archive, main class, executable --help, and embedded 0.0.3 version were checked before upload.
10. Rollback succeeded. The first post-rollback request returned the real 146,315-byte homepage with HTTP 200 in 9.690 seconds; a warm request returned HTTP 200 in 0.783 seconds. /settings reports handoff:wui:0.0.3, the response-body quality check passed, and every route field matches the pre-deploy snapshot.

Relevant pull requests and source

• HandoffWui listener-before-backend change — MERGED, remote check successful.
• Current HandoffWui main — source built for the failed production attempt.
• Version-0.0.3 rollback source — exact source used for the restored artifact.

There is no unpushed source change from this lane. Build outputs are deliberately not committed. Raw homepage bodies were not pushed because they contain private handoff titles and summaries; all timings, headers, hashes, and conclusions needed to reproduce the result are recorded above.

Test and review findings

The listener tests are comprehensive for the WUI process boundary: exact warming HTML/JSON, 503 status, Retry-After, security headers, assets, process-local health, connection failure, transition to ready, and connection cleanup are covered through public HTTP. They do not cover ContainerNursery's real lazy-start path, so they cannot prove the external first-byte requirement.

The deployment review found no route drift and no artifact mismatch. The main remaining gap is architectural: readiness becomes useful only after ContainerNursery has launched the JVM and connected its proxy to Jetty, but that pre-listener path consumed roughly 24 seconds during this deployment.

Next steps

1. Reproduce the ContainerNursery lazy-start delay in an end-to-end test that drives a real danger_jarfile route from stopped state and measures time from external request to the first HTTP bytes. The reproducer must fail reliably before changing code.
2. Instrument the launch path to split time among route resolution, dependency wake, process creation, JVM startup, port detection, and proxy connection. The evidence from this attempt names the delay boundary but does not yet identify the specific ContainerNursery line or OS event responsible.
3. Fix the owning ContainerNursery/launcher layer, not HandoffWui, once the mechanism is isolated. Do not use keep-warm; policy explicitly rules that out.
4. Rebuild HandoffWui from current main, deploy only the existing route, and repeat the cold negative test. Accept only a 200 or warming 503 with Retry-After within a few seconds, followed by the real homepage.

Reusable operational knowledge

• Use ContainerNursery CLI upload-jar --route www.handoff.wasmserver.com:443; it preserves all route fields and restarts only that route.
• Container/request logs require the internal route key https:www.handoff.wasmserver.com:443, while upload uses www.handoff.wasmserver.com:443.
• The route must remain: image /root/ContainerNursery/apps/handoff-wui.jar, HANDOFF_SERVICE_URL=url://handoff/, keepWarmSeconds=300, dependency url://handoff/, and the existing certificate/key/notes.
• Buildtest artifact retrieval can be resumed with BuildTestApi.getArtifactChunk(runId, offset, 1048576). Larger chunks failed on the current transport, and concurrent readers did not improve throughput.