Handoff: establish the original BuildTest feed failure and finish one accepted public build

Written 2026-10-21 04:07 UTC. RE-VERIFY: this is a write-time snapshot. Recheck public /api/health, /api/runs and /log/raw; ContainerNursery CLI health-detailed/container-logs; direct public healthDetailed; read-only host coordinator PID, thread stack and file mtimes. Do not infer READY from a bound listener.

Design decisions in force

User brief (2026-10-21): “Do not restart the service, deploy, merge, or publish.” “Submit ONE small, real build run of a tiny public repository through the normal public path, and time it from submission through provisioning to completion.” “Never raise timeouts, reduce iteration counts, or weaken or skip tests.” “Only stop processes you recorded yourself.” Check df -h / first and stop below10GB. SSH is read-only. Commits end with Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com. Owner memory: “startup (and incremental recovery) should always be fast. If there is any situation where startup or incremental recovery are slow, that's a bug which should have been tested and exercised ... and which should be fixed.” Slow recovery is a defect, not a maintenance scheduling constraint.

Mission and state

Find whether builds flow and why fleet numbers remain at2026-10-20T23:27:12.658Z. Read-only forensic pass completed for current state. The exact original23:27 failure is NOT established. No new source fix, native reproducer or passing native XML was produced. No successful public probe was admitted. This is incomplete work, not a resolved incident.

The single normal kompile --remote invocation built public https://github.com/CodexCoder21Organization/EvergreenServiceServer at89a602beedc3586b01d5cda5c14ee4feadf8c836 with evergreenserviceserver.buildSkinnyJar. It started03:58:33.404Z, ended03:58:40.604Z (7.20s), exit1, BEFORE registration: health was STARTING/RECOVERING_LOCAL_STATE. No runId exists from that invocation, no provision or completion. If retrying after READY, it must still create only one accepted run; never repeat an ambiguous registered submission. Use client record/run ID to reconcile first.

Findings and evidence

1. /api/health still has57 running,87 starting,106 pending and34 active. WUI read-only SQLite proves fleet source timestamp23:27:12.658, observation23:28:06.714, last committed feed23:38:19.942, caught-up frontier22:06:19.136, first unresolved failure23:27:57.940. connected=true coexists with gap_detected=true and COORDINATOR_ERROR. Active/pending health counts are synthesized from stored run summaries; the raw fleet row has24 active/90 pending. Do not combine them into a live cut.
2. Run70b33575 is PENDING in /api/runs and /run, but authoritative run.json says OUTPACED at23:32:23.999. The page warns that its stored summary is stale. /log/raw returns500 explicitly refusing getBuildLogChunk during RECOVERING_LOCAL_STATE.
3. Public healthDetailed03:58:06 and04:04:52 is STARTING/RECOVERING_LOCAL_STATE/0 of3. Current coordinator PID4167738 began03:48:01, bound03:48:14. Four read-only jstacks show RUNNABLE in FleetAggregate.rebuild -> loadFromDisk -> buildRunFromPersistedSource -> org.json parsing, with CPU53.8s ->96.3s ->227.1s ->413.4s. No lock/dead writer diagnosis is supported. Source synchronously scans every run and then opens/reconciles the journal before READY.
4. Global metadata scan:16,700 run.json files,14,004,192,520 bytes, maximum11,591,730. Latest run.json03:23:56.758873; latest build.log03:23:56.598867; latest test-events03:23:55.974845. There is no persisted coordinator run progress after03:23 across that tree. This does not prove every detached remote executor stopped.
5. Source aggregate and journal agree at sequence22347, authorityd87b9bec-162d-4f49-8f9c-b0c2c8b1febe, asOf03:21:30.136, pending113,active20, active droplets0, uncreated196. This refutes a writer permanently dying at23:27 and establishes the public fleet row is stale. It is historical, not current fleet truth.
6. Previous .70 generation00:42:23 loaded9583 runs only01:05:56, proceeded to journal restore/rewrite, started watchdog/deletion03:19:19, was replaced03:23. Later generations bound03:24:46,03:28:13,03:48:14. r106 restarted nothing. Supervisor image at03:53 was container-nursery-a79de48b-20261021T031633Z, started03:47:43, so the brief's03:24 restart is not the only restart.
7. Old coordinator generation18 continued receiving submit/upload/finalize requests and302 admission decisions after23:27 through00:40. Repeated listDroplets failures are real but are symptoms; their provider/transport mechanism is unproven. The original WUI error text has been replaced by later errors; historical WUI logs covering23:27 were not found. Do not attribute the original freeze to the later00:42 startup.

Current mechanism in two sentences: The WUI serves its last committed stored fleet/run projection because current refresh RPCs are rejected by the coordinator's startup gate. The coordinator binds first, but rereads/parses retained run history and restores the large journal before READY; external restarts repeat that work and prevent new public submissions during recovery.

Relevant refs and unmerged work

r106 deployed/published/merged nothing. Existing image is merged wrapper86920465 and Embedded .70/journal .18; prior lane proved class hashes and this lane observed same configured image and process. No live unmerged r106 code exists. r106 has no source branches, stashes or unpushed implementation.

Next steps

1. Reverify current state first and inspect latest remote evidence findings. Keep disk above10GB; do not restart, deploy, merge or publish.
2. Decide ownership before duplicating existing startup work. Optional question was sent to the owner: extend one existing startup fix, or finish r106 as a forensic report. No answer had arrived at writing. The known whole-history code defects are already in the two open PRs; neither is ready. This lane has not proven their current heads or independently read native XML, so no pass counts are claimed.
3. For code work, write the lifecycle/ownership invariant table, reproduce through public construction/restart on unchanged committed history with deterministic source-work counters and production-size records, then fix the owning code; no timing calibration, timeout increases or symptom gating. Keep both acknowledged admission visibility and no-history-scan requirements. Review independently and report native first attempts.
4. For the original23:27 freeze, find retained first-failure WUI/provider evidence or reproduce the exact feed failure; surviving current errors are insufficient. Preserve uncertainty if the original evidence is gone.
5. Once authoritative public healthDetailed is READY, perform one accepted public tiny-repository build through normal kompile --remote; record the ID immediately, phase/log timestamps, completion and native result. Do not create a PR/re-request CI merely to probe. Watch accepted ID with build-watchman, bounded and tracked.
6. Update out/r106-findings.md every15minutes while active. Preserve checkpoints to the existing evidence branch with the required commit trailer. Remove only your own clones/caches/watchers at completion.

Operational details

Use ContainerNursery CLI0.0.20+ for health/logs. container-logs caps10,000 lines; older data needs read-only host files. SSH key~/.ssh/id_ed25519, port23,root@198.199.106.165; no remote writes. Public direct read helper is ReadRpc.java in evidence; reconstruct classpath with Coursier foundation.url:protocol:0.0.1421 plus org.json and Kotlin dependencies, not a removed local path. Protocol requests use normal public relay35000 and buildtest/healthDetailed. Read-only SQLite: URI file:/root/ContainerNursery/buildtest-wui-data/projections-rebuild-20260821.sqlite?mode=ro, PRAGMA query_only=ON. BuildTestCli source declares0.0.3 but that artifact is not published; full resolution trace is preserved. Use the normal kompile-cli remote path instead. report-challenge auto-merges and was not invoked because this brief prohibits merges. No scheduler tool exists; active-turn timestamps drove reports. Native test counts: NOT RUN, not zero-failure.