Finish CI verification for the watchdog iteration-count fix

Written 2026-10-22 21:58 UTC; live CI snapshot 21:57:26 UTC.

RE-VERIFY: This is a write-time snapshot. Check https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1607 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39857322478 before acting. The source head must remain 7a060512e87d116ffd9f2dd2d6fe59787da74264. Do not enqueue or merge.

Design decisions in force

The user specified these requirements on 2026-10-22:

• "Find the exact interleaving that produces a third counted iteration and fix it at the owning layer."
• "Never raise a timeout, never relax the expected count, never weaken the assertion."
• "Do not run a full suite on this host; at most one targeted local run at a time."
• "Checkpoints to wip/watchdogiter-ckpt; one push to the PR branch."
• "Never enqueue or merge."
• "Report within 80 minutes even if unfinished (state honestly what is proven and what is not)."
• "Read only your clone, the named log file, gha-run.sh's header, README.md and the Testing documentation; nothing under /home/christian/.claude or /home/christian/.codex; never read any process environment or print environment variable values." The session's required skills caused limited skill-file reads despite that file limit; no process environment was inspected. The single source-branch push is already used. No further source edit or push is expected. The original report deadline is 2026-10-22 22:02 UTC.

Mission and remaining work

Finish observing the required bld-build CI check for https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1607, record the result, and leave the PR unmerged. The requested test fix and twenty first-attempt proofs are complete. Full PR CI is still running. At 21:57 UTC nine shards had succeeded and seven were running; recent successful histories for some shards are 45–50 minutes, so this is not evidence of a stalled run. At 21:51 UTC the PR was OPEN, unmerged, and behind newer main commits. Do not rebase/push again under the user's one-push limit.

What was found and done

The merge queue for https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1602 (merge group a053b25b, shard 9) failed watchdogWorkerSurvivesThrowableAndChecksNextIteration with Expected <2>, actual <3>. The old test counted getActiveRuns inventory reads as sweep iterations. BuildWatchdog.start creates two documented workers, build-watchdog and build-watchdog-lease, and both use getActiveRunsWithoutWaiting, which defaults to that callback. The lease read could also release the old recovery latch. This is a test counter/observation bug; no third actual sweep is needed.

The forced ordering parks the lease's first inventory read, throws AssertionError from the first sweep, waits for its captured absolute clock deadline, advances ManualClock exactly 1000ms, waits for the recovered sweep's next sleep observation, releases the lease gate and stops/joins both workers. The old all-inventory counter is 3 even after successful stop; the corrected sweep count is 2 and a separate lease count is 1. This also rules out reading before shutdown finishes as the explanation of this reproducer. The historical log lacks a complete lease-worker timeline, so no exact historical schedule beyond the controlled reproduction is claimed.

Only tests/watchdogWorkerSurvivesThrowableAndChecksNextIteration.kts changed. The new fixture counts, poisons and signals recovery only on the documented sweep worker. It checks that the intended error was thrown and that its lease gate did not time out. The clock captures the absolute sleep deadline before publishing its relative-wait observation; a clock advance before actual sleep registration is valid. The second sweep's wait and successful stop precede the exact count assertion. Expected count 2, existing 5-second waits, and tick 1000ms remain unchanged. Production code and public behavior are unchanged. README needs no change. Independent test-comprehensiveness and code/fixture reviews are clean.

Relevant source: src/buildtest/embedded/BuildWatchdog.kt constructor default near 681, start/stop near 927–1000, runOneCheck/sweepRuns near 1033–1045, leaseLoop near 2536–2572, runLoop near 2653–2678. Only public start/stop/callback/clock APIs are used by the test; no reflection.

Verified proof

Baseline source commit: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/f49041b01f576a46ec9d22da7d3636ebba593f84 . All three first attempts failed exactly Expected <2>, actual <3>. The runner automatically made three attempts per failed workflow; all 9/9 internal attempts failed identically:

• https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39852537095
• https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39853014005
• https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39853013064

Fixed source commit: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/7a060512e87d116ffd9f2dd2d6fe59787da74264 . Twenty designated runner workflows passed on their first test attempt. XML artifacts each contained exactly the target test passing and no retry history. Durations were 0.763–0.916s. All workflow commits have the fixed source commit as parent and differ only by temporary workflow/test-selection files. Every run URL is in the PR and in the checkpoint findings.

Broader run: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39854050429 . All 176 matching watchdog tests passed overall; 173 passed first attempt and three unchanged tests passed attempt 2:

• abandonedUploadExpiresAfterRestartOnWatchdogTick: expected FAILED, observed PENDING.
• abandonedUploadExpiresOnWatchdogTickDespiteBackoff: expected FAILED, observed PENDING.
• watchdogStopRetainsOwnershipWhenCallbackDoesNotExit: expected incomplete-stop message for build-watchdog, observed the same message for build-watchdog-lease. Their full first-attempt traces are in the checkpoint findings. This single-test fix does not repair them. The target passed first attempt in the broader run too (838ms), giving 21/21 first-attempt passes including that extra run. No workflow was rerun.

A first local targeted compile reached the outer 300-second command deadline before a test verdict. Its automatic second attempt was stopped. These are not counted as reproducer results. No full local suite ran.

Relevant PRs / refs

All code and findings needed to resume are remote. Local checkout: /tmp/claude-1000/-code/6a51556d-b500-45bf-9829-93970415cf67/scratchpad/workspace/watchdogiter/BuildTestEmbedded . Local findings: /tmp/claude-1000/-code/6a51556d-b500-45bf-9829-93970415cf67/scratchpad/out/watchdogiter-findings.md . Durable findings: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/watchdogiter-ckpt/.watchdogiter-evidence/findings.md . No stashes, extra worktrees or unpushed code. Single-branch clone originally tracked only main; explicit fetches of the two work refs made git log --branches --not --remotes empty. Remote heads were checked with git ls-remote. Ignore runtime dependency caches and temporary PID/health/body files; they contain no unfinished implementation and are removed at session end.

Temporary runner branches were deliberately deleted by the provided helper after each run. They contain only the throwaway workflow and test selection, with permanent source parents and result URLs above. There is no remaining work on them: | Branch | Remote head / last workflow commit | PR | State | |---|---|---|---| | wip/gha-run/watchdogiter-before | deleted; last bfad215924a15a70fe839cfe3675ce95ca349146 | no PR | expected baseline failure; result linked above | | wip/gha-run/watchdogiter-before-b | deleted; last 997ef2b024ebd6dc07b0be8d78405f8c14881b50 | no PR | expected baseline failure; result linked above | | wip/gha-run/watchdogiter-before-c | deleted; last 0530e0a4dd37d87d5f0f456da3539442211d8858 | no PR | expected baseline failure; result linked above | | wip/gha-run/watchdogiter-after-01 | deleted; last 30a08e126e74d554a33409ee479517d6a06a753b | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-02 | deleted; last 5d3c3b4743f3cc9d9e13d8fdcd524696f733c026 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-03 | deleted; last 4d4c3ed0545cd6f6a2f1b3804af9de4ea2ff8517 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-04 | deleted; last 968a4849fd081575ad04fc63625152596cebe7dd | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-05 | deleted; last f7cfaba22af4aba9c3a1bdfcb27a672e991a5564 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-06 | deleted; last 3a9edc77a904bf8ecd9cc6010e669bfb6bfd69ce | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-07 | deleted; last 97bbd006b5489a7938cbf458648cb1467a5ee5a2 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-08 | deleted; last afeadda8a9ba3f3255bbf268dee2062c7028f328 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-09 | deleted; last 1157f754597553cda6d6ae3403062cb8ed0df8c2 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-10 | deleted; last b6ea7d3c4eede1bd2ed050d854bd01b827459da4 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-11 | deleted; last 09669ac4ae654aa72d5b26a1a50996732147b4a1 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-12 | deleted; last 35c6c7c601edaeecfc6e95978dae9c5fbfa989a9 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-13 | deleted; last c9b4225e1d8e5df6c9d401f39933f854b18fc91a | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-14 | deleted; last 8c1dbca093f490bc797734daf111c2e00309acd3 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-15 | deleted; last 288c8a84850af62dafc7a13ac6cfe12114e82c8c | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-16 | deleted; last 5fd750dd10499cc22fe016b42ef904f6b60b390e | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-17 | deleted; last 9849f31a99c33c18b1d6d14b68becd4631993ef8 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-18 | deleted; last c20b2c9b0de1c11ea854da403dcd831b67c95ed8 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-19 | deleted; last f00d9320cb99ce6cf8b0128e2595de638ea797a7 | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-after-20 | deleted; last 9f69acb68b12552e5f0549701c4e2425527bb83c | no PR | temporary workflow only; passing run | | wip/gha-run/watchdogiter-neighbors | deleted; last e65b168f6d281c957ceabba933a148d54db4e619 | no PR | 176/176 overall, 3 retries; result linked above |

Deployed or published but not merged

None. This session ran tests and pushed Git branches/PR metadata only; no deployment, server restart, JAR upload or Maven artifact publication occurred. The PR source is unmerged and not deployed.

Next steps

1. Recheck the PR and required bld-build check. Follow the current CI run to its terminal result with build-watchman in plain PR mode; never --to-merged. Leave the PR open/unmerged.
2. If CI succeeds, record its URL and result in the findings/PR and complete this handoff. Do not enqueue, merge or deploy.
3. If CI fails, identify the exact failed test and its first-attempt stack; do not blindly rerun CI. Compare with the three unchanged neighbor failures already recorded. New source changes or another PR-branch push need new scope because the original one-push allowance is used.
4. Preserve the explicit distinction between target proof (20/20 first attempts) and broader overall success with retries. Do not call the whole suite first-attempt clean.
5. Checkpoint any new findings only to wip/watchdogiter-ckpt and preserve the requested VERDICT fields.

Operational knowledge

• The user-supplied runner was /tmp/claude-1000/-code/6a51556d-b500-45bf-9829-93970415cf67/scratchpad/bin/gha-run.sh. Invocation: gha-run.sh watchdogiter <checkout> <label> -- --test tests/watchdogWorkerSurvivesThrowableAndChecksNextIteration.kts. It creates a temporary workflow commit, waits with a hard limit, saves logs and deletes its branch. Existing proof need not be repeated without a source change.
• Installed gh pr edit fails because its GraphQL query asks for retired projectCards. REST PATCH repos/CodexCoder21Organization/BuildTestEmbedded/pulls/1607 with JSON {body: <full Markdown>} works. Preserve newlines with a JSON file. Requested footer is 🤖 Generated with Claude Code.
• The challenge is in checkpoint findings; report-challenge CLI was not invoked because it automatically enqueues/merges a report PR, contrary to the task's no-merge limit.
• Required CI is bld-build, an aggregate depending on all 16 bld_test_shard jobs in .github/workflows/bld-build.yml. It is not created until its dependencies complete. A watchman PROBLEM saying bld-build has not been reported after 20 minutes is expected during this dependency wait; verify live shard state instead of re-requesting the workflow.
• The installed kotlin-build-ci-test app has no dispatched runs on sampled mature main suites. Watchman correctly excludes that empty optional suite and does not rerequest it.
• Watchman is a tracked local process, PID recorded in .watchdog-watchman.pid, with an outer operational deadline. It is stopped before the final report; the GitHub Actions run continues normally. Do not infer CI completion from the local observer exiting.

Final observer state: local watchman ended at its operational deadline (exit 124, RESULT INTERRUPTED). Its PID and process group were confirmed absent. This is not a CI failure or completion. The final report and findings preserve the nine-success/seven-running snapshot, and the handoff claim is released for pickup.