BuildTest backend deployed, but two verification gates failed

RE-VERIFY: This is a write-time snapshot verified at 2026-07-30T04:01:05Z. Re-check the exact ContainerNursery route url:buildtest:, https://buildtest.kotlin.build/api/test-results?id=f3ed0075, /root/cn-watchdog.audit, and the current build-run records before acting.

Mission summary

Build the BuildTest backend fat jar from merged BuildTestServerService main and deploy it only to the ContainerNursery route url:buildtest:. Success required all four pre-registered gates: exact server jar hash, RUNNING container with a clean startup log, the specified test-results endpoint returning JSON with FAILED entries first, and no pre-deploy in-flight run lost.

What was found and done

1. A fresh clone was built from merged main commit e85f6a71b76ed39ece00124cb3a2912d847a7e48. build.kts pins buildtest.embedded:buildtest-embedded:0.0.61206.
2. The documented build command produced build-fats/buildtest-server-service.jar, 145,112,105 bytes, SHA-256 ccc595de3ed6b2af8a511298af5b7c5d2669d2bcd4c9ea48caf303f48d55489b. The jar contains exactly one buildtest/embedded/TestResultSeverityClassificationCache.class.
3. /root/cn-watchdog.audit was read first through HardwareControlFabric. No unmatched recent deployment intent existed. An INTENT line was appended at 2026-07-30T03:46:47Z.
4. Pre-deploy route settings were preserved: memoryLimitMb=4096, maxMetaspaceSizeMb=512, startupTimeoutSeconds=300. The old jar was 145,050,777 bytes with mtime 2026-07-29 13:32:34.653745296 +0000.
5. Eleven nonterminal runs were recorded before restart: 76a4c4f2, f6245e08, 2c9f88a4, aa76865e, 00eb030a, 6e268744, 0fbe0e6e, 560531f3, 32d0af8a, 68f2a66a, and 00f0140b.
6. container-nursery-cli upload-jar uploaded the jar to only url:buildtest: and restarted that route. Independent HardwareControlFabric checks confirmed the live server file has the expected size and SHA-256.
7. At 2026-07-30T04:01:05Z, ContainerNursery reported url:buildtest: RUNNING on host port 34545. All 11 pre-deploy run IDs were still present; 2c9f88a4 had completed, and the other ten were TESTING. This proves restart continuity.
8. Startup was not clean. The old generation logged an AmbiguousRpcRequestException during shutdown when its digitalocean-droplets connection closed with requests pending. More importantly for the pre-registered gate, the new generation's stderr contains AmbiguousRpcRequestException/InterruptedException stack traces from interrupted renewDroplet calls and two DropletServiceCallStalledException warnings at 30 seconds. No OutOfMemoryError was found.
9. The exact functional endpoint did not pass. It first returned HTTP 500 for a stale WUI-to-backend connection, then HTTP 503 while rebuilding test-results:f3ed0075:0:100, and then HTTP 500 because getTestResultsPaginated(offset=0, limit=100) exceeded the backend RPC handler's 30-second limit. At 2026-07-30T04:01:05Z it still returned HTTP 503, so JSON ordering could not be checked.
10. A DONE-WITH-FAILED-GATES audit line was appended at 2026-07-30T03:58:00Z. No further upload, restart, config, run, DNS, or other route action was performed.

Result against the pre-registered gates

• (a) PASS: server jar has SHA-256 ccc595de3ed6b2af8a511298af5b7c5d2669d2bcd4c9ea48caf303f48d55489b.
• (b) FAIL: the route is RUNNING, but the new generation's startup/current stderr contains exception stack traces and stalled lease-renewal warnings.
• (c) FAIL: https://buildtest.kotlin.build/api/test-results?id=f3ed0075 does not return result JSON, so FAILED-first ordering remains unverified.
• (d) PASS: every pre-deploy in-flight run remained present and progressed.

The deployment is therefore not a success under the fixed four-gate definition.

Relevant PRs and refs

• Merged change: https://github.com/CodexCoder21Organization/BuildTestServerService/pull/266
• Merge commit: https://github.com/CodexCoder21Organization/BuildTestServerService/commit/e85f6a71b76ed39ece00124cb3a2912d847a7e48
• All source work is already on merged main. No source changes, commits, stashes, or local-only source branches were created in this session.
• The built jar is intentionally not committed or attached because it is a build artifact. Its deployed server copy is identified by the exact hash above.

Next steps

1. Re-check whether the exact test-results endpoint has eventually materialized. If it returns HTTP 200, capture the JSON and assert that all leading entries are FAILED before any non-FAILED entry.
2. If it still times out, diagnose why getTestResultsPaginated for f3ed0075 exceeds 30 seconds. Do not add a timeout or retry as a substitute for finding the mechanism.
3. Review the new-generation lease-renewal stack traces and determine whether they are a one-time restart recovery effect or an active droplet-service problem. The fixed success gate requires a clean startup log, so do not relabel the gate as passing without a new clean generation.
4. Any restart of the BuildTest WUI, another backend upload, route configuration change, or action against another container requires fresh user authorization. The original authorization covered only the completed url:buildtest: upload/restart.

Reusable operational knowledge

• The exact ContainerNursery key for this URL route is the literal url:buildtest:; omitting the empty port segment produces misleading route-not-found behavior.
• upload-jar both replaces the configured image and restarts the route; do not issue a second restart automatically.
• The local detailed evidence file for this session is /tmp/claude-1000/-code/21df1a0f-ec10-416f-b696-1ca6954f87a0/scratchpad/out/r2-findings.md. Its important evidence is copied into this handoff because the scratch workspace is disposable.