Handoff: Make the droplets web app cold start fast

Written 2026-10-21T19:21:49.333712+00:00

RE-VERIFY: This is a snapshot. Check current main and the branch head with git, the HTTPS route with ContainerNursery routes, and the current access decision before acting. The configured production filename identifies9b011872, but its byte digest has not been verified.

Design decisions in effect

Owner rules dated 2026-10-21:

• "Never raise a ContainerNursery route keepWarmSeconds beyond 5 minutes to hide a cold start; fix startup/first-request instead."
• "No JVM tuning flags on JVMs we launch (no CDS/AppCDS archives, no -XX flags, no heap changes). Look for structural/algorithmic cost in our own code and dependencies."
• "No timeouts raised, no caching that hides wrong data."
• "Read-only on production. No deploys, restarts, or route edits."
• "If the dominant cost turns out to be in ContainerNursery or an upstream library, stop after the reproducer and report; do not patch around it in the WUI."
• No merges, queue actions, or Maven publications. A change needs a reliable initially red test, a reversal control, and a PR opening with the user's stated provenance. Do not watch CI after opening that PR.
• Use local targeted tests; use the machine-wide suite lock only if a shared contract changes. Do not change existing test budgets or iteration counts. No reflection or mocks.
• SSH is permitted only if both supported APIs are down. Both respond. A specific read-only SSH exception for retrieving the JAR is pending; no exception has arrived.

Mission

The owner reported that https://droplets.wasmserver.com/ "is super unreliable, frequently rendering various errors/notices due to bugs in UrlResolver or UrlProtocol or something else" and asked for the underlying root cause to be fixed. The HTTPS route stops the WUI after 5 idle minutes, so many human visits start a JVM.

This lane must profile the exact configured JAR and main, explain the 4–10-second cold-visit spread, and write a reliable initially red test before a structural change in its owning layer. It has not established that production-specific mechanism or delivered a fix.

What was found and done

1. Read the prior timing lane in full and used a fresh checkout. Main remains 9b0118727a17c4163beebcb6ff80d614826ebed2. The local build succeeds and produces40446520 bytes, SHA-256 ce3d4684f4d4befca62771853b8c46125e7324d4cef2de35e539e6f012eb4e59.
2. Preserved lightweight5ms wall-stack sampler and a separate timestamp agent, each with plain controls. Main spends its early local time in first-use class loading, Jetty construction/start, and then rendering/date/string helpers. Resolver setup is on the background initializer after the listener opens. A ready backend is not required for the first page.
3. Empty-history main plain first-page totals are 226/244 ms. A one-CPU diagnostic yields413/420 ms. The timestamp agent has substantial overhead and must not be used as an absolute production estimate. There are 15 diagnostic JVM runs total; no4–10-second production delay was reproduced.
4. Retrieved the exact production history through the supported text API: /root/ContainerNursery/data/last-known.json,5168 bytes,3 regions, SHA-256 594d8f7966713e1183ffbf0bdb5261b71219af1b271b854d06221eaad6b75df7. Plain main with this exact input totals 285/275 ms to first page. Releasing only this local JAR's file pages before launch yields265/253 ms. A fincore control confirms zero resident pages after the same local advice call. Thus neither saved input nor local file-page state reproduces the production delay.
5. Aligned six production visits. Five have WUI startup records. Estimated request-to-JVM-uptime-origin spans 76–563 ms; JVM-origin-to-listening spans 2668–6889 ms; listening-to-request-completion spans 533–2763 ms. Most of the spread is after the JVM uptime reference, not a several-second pre-JVM gap. Exact historical OS process-spawn timestamps are absent, and the 17:13 WUI sequence is unavailable. The records locate the large changing phases but do not identify their code-level cause.
6. Current file lookup reaches the API size check and reports the production JAR's length40446520bytes, matching the local build. Earlier worker path misses are superseded by this current result. Equal length is not a digest match. HCF read-file uses text and has a 10 MB cap, so it cannot retrieve this JAR intact. CN has upload/list/delete handlers and no JAR-content GET handler. A .jar.sha256 sidecar is unavailable. Proc text reads return zero bytes because the daemon's bounded reader uses reported length; this is not evidence of empty process arguments. No SSH was used.
7. Production has existing Serial GC/free-ratio/stack/direct-memory options and 128 MiB heap/metaspace caps. None was changed or supplied by the lane. Local runs use default JVM settings, so launch configuration is an additional comparison limit. No unmeasured flag effect is presented as a cause.
8. The existing testColdSetupServesFirstFrame passes before and after a successful fetch/rebase. Latest XML says success=true,time 1576 ms. CLI 0.0.124 emits no test-attempt counter; output has no retry labels. Do not claim the requested XML first-attempt bar. No existing test changed, no new reproducer was written, and no structural fix or reversal control exists.
9. No shared public contract changed, so no complete suite was run or queued. No PR was opened, no CI was watched, and no deploy, restart, route edit, publication, merge, or queue action occurred.

Relevant refs

Primary findings: https://github.com/CodexCoder21Organization/DigitalOceanDropletWui/blob/wip/lanePcold2-startup-profile/investigations/lanePcold2/findings.md

Prior lane: https://github.com/CodexCoder21Organization/DigitalOceanDropletWui/blob/wip/lanePcold-request-timing/investigations/lanePcold/findings.md

Configured live image: /root/ContainerNursery/apps/digitalocean-droplet-wui-v0.0.16-main-9b011872.jar at https://droplets.wasmserver.com/. Latest route recheck retained this filename and 300-second keep-warm value. This lane deployed or published nothing; its branch contains diagnostic material only.

Next steps

1. Re-verify current state first, including the user's pending read-only SSH decision. Both APIs respond, so the existing rule still disallows SSH unless a specific exception arrives. Do not infer approval from elapsed time.
2. Obtain the exact JAR through an approved byte-preserving method, verify its digest against the rebuilt JAR, and repeat the preserved lightweight profile with the same history. The history is already readable through HCF; the remaining file-access need is the JAR only.
3. Establish a production-specific condition that explains the extra JVM/HTTP seconds. Current evidence does not support a particular WUI algorithm, a multi-second CN pre-JVM delay, or a saved-history/file-page cause. Do not replace missing evidence with a local elapsed-time threshold.
4. Write the reliable public-API reproducer in the owning layer. If the cause belongs upstream, stop after the reproducer and report, as this lane requires. If it belongs in WUI, apply the structural change, show the test's red/green reversal, verify the requested attempt counts, and open the PR with the required provenance and footer. No production change is authorized.

Operational knowledge and omissions

• The repository branch contains all analysis and diagnostic source needed to resume. Large raw logs, built JARs, and private operational history are intentionally excluded. Numeric timings, the history digest/size, and selected startup events are preserved.
• For HCF use the existing alternate client_cert.pem, fabric_key.pkcs8, server_cert.pem names under /home/leslie/.config/hardware-control-fabric/. Do not publish those files.
• New source checkouts must be fresh and under the lane's own workspace. Other lanes' directories must not be edited or used for builds.
• Read the Testing Architecture sections Test-Driven Development and Never Calibrate a Reproducer to a Machine before tests: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md
• The old PlanRepository handoff README describes a git merge path; the current create-handoff skill uses the central service. This handoff uses the service and no merge/queue operation.