Finish clean full-suite verification for the BuildTest restart watchdog PR

Written 2026-10-21 UTC. RE-VERIFY: This is a write-time snapshot. Recheck the PR head/state/checks with gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1570 --json state,headRefOid,isDraft,statusCheckRollup,mergedAt; verify every branch with git ls-remote. Stored BuildTest WUI summaries may be out of date; query the exact raw-run endpoint and retain readiness errors.

Design decisions in force (2026-10-21, user wording)

• “Your scope is how runs that span a restart are treated, and the stall message and result accounting.”
• “That slow-startup fix is owned by a separate lane, so do not change startup recovery speed here.”
• “startup (and incremental recovery) should always be fast. If there is any situation where startup or incremental recovery are slow, that's a bug which should have been tested and exercised by https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos and which should be fixed.” This lane does not own that startup-speed change.
• “A run that is PENDING with old provisioning progress before a coordinator restart, with a long gap during the restart, is not failed by the stall watchdog after recovery. Its stall clock restarts from the moment the restarted coordinator first owns it.”
• “The stall message reports the true elapsed time.”
• “A run that fails in provisioning reports its tests as not run, not as failed. Do this only if it is a small, contained change. If it is not, record it in the findings as a separate follow-up and leave it out of this PR.” The existing classification made this contained; it is included.
• “Use the repository's Clock abstraction (ManualClock) for time; do not sleep. Show each test failing on main, with the failure output, before fixing anything.” The elapsed-message allegation is refuted rather than changed: its control passes main.
• “Fix the defect at its cause. As a mutation control, show that reverting the fix makes the tests fail again.”
• “Run the new tests plus the neighbouring watchdog and recovery tests locally, and report first-attempt pass counts from the runner's own output. Run the full suite once from the head you push, per AGENTS.md, and report its counts.”
• “Run df -h / first.”
• “Work only in /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/r110. Clone repositories fresh there.”
• “NEVER stop any codex process. Stop only build/test JVMs that you started yourself, by their recorded PID.”
• “At most one full scripts/test.bash may run on this machine at a time. Check uptime before a heavy build. If load is above 20, run targeted tests only.”
• “Do not merge, enqueue, deploy, publish, or restart any production service.”
• “Do not increase any timeout or deadline, do not reduce iterations, and do not weaken or skip tests.”
• “Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. The PR description ends with 🤖 Generated with [Claude Code](https://claude.com/claude-code).”
• “Write a checkpoint to out/r110-progress.md after every milestone.”
• “Write out/r110-final.md: the mechanism, the PR link, the test evidence, the mutation control, and anything you could not do. Remove your clone and any lane caches. Kill any wait loop you started.”

Mission and remaining work

The code, fail-first tests, original main-reversion control, two independent reviews, push, and PR are finished. Clean full verification remains: the required local full runner exited137 during fixture preparation, and the otherwise green sixteen-shard CI contains three first-attempt failures hidden by automatic retries. Keep the PR draft until that remaining evidence is resolved or the owner explicitly accepts the limitations. Do not call retry-green a clean first-attempt suite. The last local preflight at13:35 UTC was load31.11, above the explicit full-run limit20. No larger heap, timeout, runner, or symptom workaround is proposed; the memory-termination producer is not established.

What was found and done

On main, BuildTestEmbeddedService.kt:28539 derives execution start from the historical provisioning phase; BuildWatchdog.kt:1604 clamps current droplet-less progress to that old start. BuildTestEmbeddedService.kt:28099 then rejects the watchdog's stale cutoff using current admission activity, so a recovered run can remain provisioning indefinitely after its fresh window; the provisioning failure callback also omitted the existing not-run classification, causing unfinished-test finalization to mark unstarted discoveries failed.

The change records a coordinator-owned recovery baseline only for restored pending runs without assigned droplets or unresolved cleanup, advances it to the first actual grant, and preserves that grant across requeues. The watchdog and final service decision agree on that bounded interval, including backward wall-clock correction. Terminal deletion and service close discard the coordinator-owned records. Provisioning failures before any test starts preserve discovered rows as PENDING and report zero failed tests; a durable started-test control retains ordinary failure finalization. Startup recovery speed and existing deadlines are unchanged.

Fixed-head references: BuildWatchdog.kt:1609,2600; BuildTestEmbeddedService.kt:6328,28112,28614,29915,44997,45003.

Production evidence

The available stored record does not establish the brief's immediate-after-recovery failure. It records completion at 07:13:26.631 UTC, before the stated 07:13:41 restart. The named progress timestamp is 07:07:54.708 UTC: a 5m31.923s gap, accurately rendered as five whole minutes. The page warns that its stored summary is out of date; check delivery time is not proof of failure execution time. A deterministic 100-minute message test passes unchanged main, so no elapsed-time arithmetic change was justified.

Failed check supplied in the brief: https://github.com/CodexCoder21Organization/UrlProtocol/runs/119087524045

Test evidence

Counts below are first attempts parsed from the runner's native XML, rather than a retry's final summary. All tests use public APIs and in-process components; time is driven by ManualClock without sleeps or mocks.

Both failing main cases failed every automatic retry, as did both failures after reverting the production fix. The 14-case final sweep needed zero retries. Additional real-service requeue and backward-clock cases failed before their respective repairs and passed in that sweep (service-regrant-baseline-ready.xml, backward-clock-baseline.xml). Independent test and code reviews are recorded in r110-test-review.md and r110-code-review.md; all material findings were addressed.

The final 14-case run preceded a clean rebase that changed only main's dependency/version pin: source, tests, and README diff against the reviewed head is empty. The final source compiled after that rebase.

CI workflow https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39600156974 completed with all 16 shards green. The native first-attempt counts are 5,009 passed, 3 failed, 5,012 total, followed by 3 automatic retries that passed. All seven new tests passed first attempt. No CI job was manually rerun. The failures and full stacks are in r110-ci-triage.md: two projection-storage timeouts do not load the changed production files; the assigned TESTING recovery case lies outside the changed watchdog branches and rejects an unchanged readiness-message variant. This is a green CI verdict with known first-attempt failures, not a clean full-suite run.

Incomplete verification

The required full local suite was launched once from the exact pushed head. It compiled the service but exited 137 during fixture preparation; no XML or test counts were emitted. The OS memory-kill counter rose from 19 to 21 during the run, but available evidence cannot identify which process caused the memory use or attribute each kill. This is neither a full-suite pass nor thousands of failed tests. A second full run was prohibited by the house rule while load remained above 20.

An expanded latest-main mutation build also ended before tests with exit 137. Its six-case remote replacement remained pending through the client's existing 30-minute budget, waiting for admission capacity: https://buildtest.kotlin.build/run?id=e8a37dd5. Neither is counted as mutation evidence; the completed original four-case mutation remains valid. No timeout, deadline, iteration count, or existing assertion was weakened.

Relevant PRs and refs

Artifact directory: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/r110-verification-evidence/handoffs/artifacts/2026-10-21-r110

No code was deployed or published by this lane: all mutations were local GitHub branches, the requested draft PR, and this handoff. The execution record contains no production lifecycle/deploy/publish command. No claim is made about production changes performed by other sessions.

The source checkout was clean with no stashes or extra worktrees before cleanup. All source commits are on the verified PR remote; original reproducer is separately pushed. Native outputs, reviews, progress, and failure stacks are on the evidence branch. JARs, caches, static downloaded docs and repository-search dumps are intentionally omitted. The lane clones/cache are now removed per the brief. Final cleanup and branch equality are recorded in the service report.

Next steps

1. Read ~/.claude/CLAUDE.md, ~/.codex/AGENTS.md, the BuildTestEmbedded README, the linked Testing Architecture guide (especially extreme-condition testing), and both memory notes named in the original brief. Re-verify live PR/head and all native reports. The finished source/reviews need verification, not reinvention.
2. Run df -h / and uptime; inspect other full test invocations without stopping them. Clone fresh within the lane workspace, check out the PR branch, and fetch/rebase before any build. A changed source head requires fresh tests and review of its actual diff; do not assume the original evidence covers it.
3. Investigate the three first-attempt CI failures locally through public APIs. r110-ci-triage.md contains exact stacks and path exclusions; do not broaden the readiness diagnostic regex or raise the30second projection runner limits as a guess. Two projection cases do not load either modified production file; the assigned TESTING case does not enter the new recovery floor or provisioning-watchdog branch. These facts do not identify the projection timing mechanism. Do not rerun CI to diagnose them. Keep unrelated repairs in separate PRs.
4. Once the full-run house conditions permit, obtain a complete local full report from the exact head being pushed. Original environment was JAVA_TOOL_OPTIONS='-Xmx1024m -XX:ActiveProcessorCount=2'; the full invocation uses scripts/test.bash --local --test . --log <absolute-results.xml>. A targeted example is scripts/test.bash --local --test testPendingRestartGetsFreshProvisioningWindow --log <absolute-results.xml>. Do not increase budgets/heaps or reduce iterations; capture native attempt-history so first failures remain visible. The prior local full attempt compiled source, then was killed in fixture preparation with no test counts.
5. Update PR/report with measured counts and evidence; remove draft only when verification is honestly complete. Stop at review: no merge, enqueue, deploy, publish, or production restart is authorized. Clean only the resumed lane's own build/test/monitor processes by recorded PID and remove its clone/cache after pushing all necessary work.

Reusable operational knowledge

• Required docs read in this lane: both global instruction files; Testing Architecture end to end, Engineering Philosophy, both BuildTest memory notes; README overview/recovery/watchdog/admission/build sections; docs/restart-recovery-invariants.md; PlanRepository README/handoff guide; Code Review host-overload guidance. README is enormous; only the named relevant sections were reviewed in detail. The legacy handoff-merge exception conflicts with current service-based create-handoff instructions and the explicit no-merge rule; the current CLI workflow and user prohibition won.
• Use native XML <attempt-history> to recover first attempts; top-level <test success=true> may conceal a failure followed by a retry. summarize-ci-results.py also verifies unique identities across all16 shards.
• gh pr edit failed because this installed gh requests deprecated Projects classic fields. Exact body updates succeeded through REST PATCH with a JSON input file, followed by byte-for-byte/footer verification. No issue was automatically merged to report this: report-challenge's auto-merge conflicts with the brief; r110-challenge.md preserves the record.
• build-watchman0.0.21 watched the PR to GREEN without a merge option. Its printed enqueue suggestion is not authorization. The installed paused remote-check app's zero-run suite was corroborated against base history; it was not blindly redispatched.
• Optional remote mutation https://buildtest.kotlin.build/run?id=e8a37dd5 is six selected cases, not5010 executions: other discoveries are Excluded. The client ended after its normal30minute pending budget; no mutation verdict exists. The raw-log endpoint later returned500 naming RECOVERING_LOCAL_STATE, saved in remote-mutation-raw-error.txt. The watcher miscounted100 excluded rows as completed; do not interpret that as tests running. No remote run was cancelled or production service changed.
• The interrupted local run's OS memory-kill counters and force-stop are observations. Kernel logs were inaccessible; no precise producer/termination attribution is established. Do not turn host pressure into a proposed code fix or a hardware-upgrade request.