RE-VERIFY 2026-10-23 11:54 UTC: original PID 3353611 remains in application local-state recovery. Re-check healthDetailed before any operation.

DEPBT2 current findings: startup is actively parsing and canonicalizing retained build-run JSON. Five samples show worker CPU and read_bytes advancing. There are 23984 run.json files totaling 19342156412 bytes. Open record sampled: /root/buildtest-data/runs/87352d4f/run.json. CN startupTimeoutSeconds=900 applies to TCP readiness; the listener is already accepting connections, so it is not a timer for application recovery. No process or route change has been performed.

Evidence: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/DEPBT2-recovery-evidence/lane-evidence/DEPBT2 (head 5b8b99e9). Prior branch remains relevant: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/DEPBT-deployment-evidence .

Next: wait for original READY; measure elapsedMs; only then one retained-candidate redeploy through CN CLI 0.0.23 with unchanged settings and application-readiness window 1.5 times the measured recovery. Real CI allocation and a 30–45 minute candidate census remain pending. Candidate remains inactive; original digest 203ac9c3738807f914d0e7f72a8750b520e5031f191845cda3d9e7847c944a66 remains live.

Prior record (historical snapshot follows):

Complete buildtest deployment after checking startup recovery

Written 2026-10-23 UTC.

RE-VERIFY: This is a write-time snapshot from lane DEPBT, refreshed 2026-10-23T11:39:40.612Z. Use CN CLI 0.0.23 jar-history and a direct public healthDetailed RPC before any action. The previous artifact has been restored; application readiness remains incomplete.

Owner requirements, 2026-10-23

• "Use CN CLI 0.0.23 only."
• "Never launch anything through HCF launch."
• "Never restart, stop or reconfigure ContainerNursery itself, or any service other than buildtest."
• "Never interrupt a management command mid-request."
• "Never raise any timeout. Never merge or publish."
• "If verification fails, roll back to the recorded artifact and report."
• The only authorized candidate is BuildTestServerService commit 62ec2f9341cb07f5ff38518a09d2886731188c1d. This attempt followed the rollback-and-report path; obtain orchestrator direction before another deployment attempt.

Mission

The owner reported unreliable responses at https://droplets.wasmserver.com/ and authorized appropriate deployments after merges. DEPBT was to deploy the already-tested buildtest main revision, verify a new droplet allocation, and compare 30–45 minutes of retained run history with the earlier 48-hour census. Artifact replacement and rollback were completed; successful candidate readiness, new allocation proof and the requested observation window were not completed.

Findings and actions

DSTALL5 established shared network-worker execution of peer-registry publication in protocol 0.0.1421. Published protocol 0.0.1427 first changes this behavior. The selected candidate contains protocol 0.0.1431 and resolver 0.0.1332. DSTALL5 ran its complete service suite at the exact candidate: 281/281 first-attempt passes. DEPBT made no production source edits and did not repeat that suite.

A fresh clone was fetched and rebased, then checked out at the exact candidate before scripts/build.bash --local buildtest.server.buildFatJar ../buildtest-62ec2f93.jar. Build exit 0. Candidate SHA-256: 7ba9b078d89c305a361e6e32288aa3947411bdb349cc7a403d17f1619f3a93ba, size 158906322. All 1417 protocol classes match published 0.0.1431 and all 1442 resolver classes match published 0.0.1332, zero differences.

The route is url:buildtest:, using /root/ContainerNursery-uploads/jars/buildtest-server-e7ce3ecb3b62-embedded-0.0.69274771-journal-0.0.18.1-20261021T070333Z.jar. The image path stayed unchanged. Route comparison before/after is exactly equal. Settings include memoryLimitMb=4096, maxMetaspaceSizeMb=512, startupTimeoutSeconds=900, keepWarmSeconds=-1; JAVA_TOOL_OPTIONS includes SerialGC, Xms1024m, Xmx2048m, Xss512k, two Netty direct/heap arenas, MaxDirectMemorySize=512m. The CN process command also adds Xmx4096m and diagnostic heap options; these existing settings were preserved. Other route variables are in route-buildtest-before.json.

Before deployment, direct healthDetailed returned READY in 2119ms. Fleet samples reported 24 then 25 active runs, with 30 then 33 pending. A 10-row current list succeeded in 20708ms, while 50/100-row requests reached the existing 30000ms call bound. Seven pending IDs were captured: 37aa7662, 6564bb50, 56edf6ce, 3ca775a2, 5f7d7536, 6b56e11b, 670de212. Full active-ID enumeration was not obtained.

The WUI returned HTTP 200 but /api/runs reported outOfDate=true and refreshFailing=true, with a snapshot over 22 hours old. Do not use that view as a current-run census. CN server logs before deployment recorded buildtest projection journal errors with java.io.IOException: No space left on device. Current volume free space was not measured; the successful upload shows that artifact replacement itself had room at that moment. These earlier storage errors have not been established as the cause of local-state recovery duration.

CN upload completed exit 0, replaced the artifact at 2026-10-23T11:17:25.180Z and restarted exactly one container. PID 3317253 started at 11:18:11.710Z; CN marked the candidate active at 11:18:27.600Z. All route settings and JVM settings except generated java.io.tmpdir matched the original values. Startup logs contained no error events. The first public page remained HTTP 200 with unchanged notice text.

The preselected historical reference was 321.819 seconds from the old artifact upload timestamp to CN serving activation. The candidate still reported STARTING/RECOVERING_LOCAL_STATE at 11:22:42 UTC and again beyond that reference, with no completed startup phases. Important limit: that old history measures CN serving activation, not an earlier application READY event. It therefore does not establish a candidate startup regression. The lane used the conservative reference selected before upload and followed the requested rollback path when verification did not complete.

CN rollback-jar --route url:buildtest: --sha256 203ac9c3738807f914d0e7f72a8750b520e5031f191845cda3d9e7847c944a66 completed exit 0 and restarted exactly buildtest. Current original SHA-256 is 203ac9c3738807f914d0e7f72a8750b520e5031f191845cda3d9e7847c944a66, size 158246917. Restored process PID 3353611 started 2026-10-23T11:24:09.810Z. Final restored health at 2026-10-23T11:39:40.612Z remains STARTING/RECOVERING_LOCAL_STATE, zero of three completed phases, elapsedMs=923840. The original application has not reached READY beyond its unchanged 900-second reference. Final public response remains HTTP 200 in 0.539 seconds with unchanged notice text. Exactly one buildtest process was observed. The readiness gap requires investigation; no cause is established.

The historical 48-hour retained-record census was 6312 completions, 566 provisioning-limit outcomes (8.97%), and 412 service-call-stall records: createDropletAsync 218, getDropletCreationStatus 180, addSshKeyToDroplet 10, listDroplets 4. There is no qualifying 30–45 minute candidate census; unmeasured must not be reported as zero.

Relevant refs and branch state

Candidate source: https://github.com/CodexCoder21Organization/BuildTestServerService/commit/62ec2f9341cb07f5ff38518a09d2886731188c1d . The candidate is on origin/main. The original artifact is named for https://github.com/CodexCoder21Organization/BuildTestServerService/commit/e7ce3ecb3b62db2a53fc10e1788cfa3985b38d3b ; DSTALL notes associate it with that source, but this commit is not an ancestor of current origin/main. DEPBT did not independently reconstruct its assembly; rollback restored its exact original bytes. This lane introduced no new unmerged production code. Both current and candidate bytes remain in CN history; inspect jar-history for the candidate retained path.

Next steps

1. Recheck the original artifact health and save its actual process-start-to-READY duration. If readiness remains incomplete, investigate local-state recovery and the prior projection/storage errors before repeating an artifact replacement. Do not claim the protocol pin caused this delay without comparative evidence.
2. Ask the orchestrator to decide the next attempt using a valid application-readiness reference. The existing configured 900-second startup setting was never changed.
3. After a successful candidate attempt, capture the full active run IDs, verify one real allocation without a createDropletAsync stall, and perform the original 30–45 minute retained-run census. Use current RPC data rather than the WUI projection.

Operational notes

• All CN operations used CLI 0.0.23. No HCF launch, other service operation, merge or Maven publication occurred.
• Working HCF files here are ~/.config/hardware-control-fabric/client_cert.pem, fabric_key.pkcs8 and server_cert.pem. The certificate names in the brief are absent here.
• HCF read-file caps each file at 10MB. The original 158MB JAR could not be downloaded; CN upload atomically retained it and both retained digest and size were verified.
• CN single-container endpoint returned not found for the encoded URL route. containers and jar-history work. container-logs returned zero rows despite a nonzero total; raw stdout/stderr were read with HCF.
• Raw logs are at /root/ContainerNursery/data/container-logs/buildtest-server-e7ce3ecb3b62-embedded-0.0.69274771-journal-0.0.18.1-20261021T070333Z-4e371663.stdout.log and the matching .stderr.log. CN derives the basename from the JAR stem and eight-digit Java path hash. Initial class-name inventories are not error events.
• Public Java probe source is in lane-evidence/DEPBT/ServiceProbe.java. It calls UrlProtocol2.sendRpcRequest with unchanged 30000ms call bounds. Compile against the candidate fat JAR and run healthDetailed, getFleetStatus, listBuildRunsPaginated 0 10, or getBuildRun RUN_ID. RunCensus.java compiles but was not used for an accepted census.
• Build outputs, full global route data, full host process data and downloaded source/reference documents are deliberately excluded from the evidence branch. The service-specific route metadata has no account keys. The exact candidate bytes remain retained by CN; rebuilding from the linked commit is another option.
• Source reading: full DSTALL5 findings/final notes; BuildTestServerService README; BuildTestWui README; CN ManagementAPI.md; deployment, GitHub lookup, status-report and handoff skills; handoff format README. No application tests were written in this operational lane.