Current r62 checkpoint — 2026-10-19 05:05 UTC
Re-verify live state before acting; all timestamps below are observations. The PR remains OPEN and draft at https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1530, source head892e20d2039c3b582e519260a3e43d376f744dee after fetch/rebase on unchanged main and explicit force-with-lease against incoming70b442c69422f0f484a9a3e75fa8e24cc73696bc. No ready/merge/enqueue/publish/deploy action is authorized.
The clean rebased candidate892e20d2039c3b582e519260a3e43d376f744dee is durably saved at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r62-matched-baseline . It is based on current main daf1d2ad5148ea8a6843a96f53298f2a133345b8; its production service blob is unchanged from incoming70b. The existing PR branch was pushed at05:01UTC and remains draft. Final-head GitHub Actions run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39202422075 has16test-shard jobs IN_PROGRESS at05:04; required bld-build has no result yet. Root owns a strict watch-only observer bounded before05:35; no CI re-request was made. All nine existing test files match main. The approved resident-runner test hash remains6aa95cd20b0c93ce54d8c85d8dc8ec13c10f979714b307ec0426cbcf168e3af9.
Evidence checkpoint8b0b4e4535 is at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r62-guard-evidence/handoff-artifacts/r62 . The linked r62-report.md and root-classification.md give every counted run and explicit missing slots. The audit has88 selected canonical/highest1/1 XML results, zero row-identity problems. Eight guards have five results per side, standalone plus nine-test batches. Terminal restart has four per side; original singleton runs bb0602cb and34d8d4bd have returned no XML, and are not counted as failures.
Five known main failures now have issues and are listed in the PR description (opening why paragraph preserved; body changed only through REST):
- cleanup: https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1540 (main3pass/2fail, PR3pass/2fail).
- delayed reply: https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1541 (main4/1, PR5/0).
- resumed run: https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1542 (main4/1, PR4/1).
- F1377e: https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1543 (main4/1, PR2/3).
- cross-run publication: https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1544 (main4/1, PR5/0).
Dynamic deadline, review abnormal exit and released activation each pass5/5 on both sides and remain inconclusive; terminal restart remains inconclusive at4/4. No PR-only cause is proven; no production or original guard test edit has been made. Main and PR failure forms are not assumed identical. The delayed-reply original error is omitted in the XML; repeated read-only recovery failed OUTBOX_FULL before returning the field, and the issue records that limitation.
At04:56:50 fresh read-only attach probes for the original terminal runs failed exact health checks: STARTING recovering local state (elapsed~532s). The attempt-history endpoint also returned READ_FAILED. Original clients exited1 at04:58:38/48 after failed15-minute lease renewal during recovery, with no XML and no remote cancellation. Both original observers are stopped/reaped. Remaining read-only attach readiness probes have finite180s bounds and a05:10 readiness cutoff. No server change was initiated by r62. No test was resubmitted during recovery. This is infrastructure evidence, not a guard verdict.
Pending execution plan: recover existing XML if service permits, count each run once. Once originals stop and exact service health is OK, run matched original common-selector contexts1f7f5983 and0d42f9a2, plus a new separately identified terminal singleton only if original XML is missing. Plan and helper are in evidence; per-invocation25min bounds unchanged, campaign05:35 harddeadline. Stop rather than submit if still recovering at05:10. Finish classification/issues, fix only demonstrated PR causes with unchanged guards, then fetch/rebase, confirm PR OPEN, force-with-lease push and report final-head CI. Final cleanup must stop/reap all owned local waits and remove clones. If service recovery persists, do not claim the four inconclusive tests resolved or fabricate their missing evidence. Old-head17green checks are not final-candidate CI.
The older text below is historical and is superseded where it disagrees with this r62 checkpoint or the narrower r62 user scope recorded below.
Handoff: Resolve expanded provisioning guard failures and verify final CI
Written 2026-10-19T03:26:53.201173+00:00 for the2026-10-19 review repair.
RE-VERIFY: Every state below is a write-time snapshot. First run gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1530 --json state,isDraft,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, respecting the once-per-60-seconds status rule. Fetch current main and compare the recorded code before editing.
Design decisions in force
Recorded2026-10-19 from the user's r55 brief. These constraints remain in force for anyone continuing this work.
Start from the PR's current head.
If main passes, fix the production code so the test passes without changing it, while keeping the PR's goal: shards that are only testing don't burn the budget.
Changing this test, or any assertion, is NOT allowed. If you are convinced the test itself is wrong, stop and write that up with evidence in your findings and report; do not change it.
Also find every other existing guard test that exercises provisioning-deadline or cleanup paths, not just the PR's 25 selected tests, and run them. Fix anything still failing at its cause. Correct the PR description sentence "existing targeted guards preserve their original assertions" so it names the guards actually run.
Either the call is redundant, in which case remove it and say why, or it is load-bearing, in which case add a test that fails when only that line is removed and show it fails. The replacement-limit branch (around :23196-23203) also has no test. Add one that fails when that branch's pause is removed.
Rewrite that paragraph in order so it matches the code's actual pause and resume rules, including that cleanup stays within the provisioning bounds.
Read ~/.claude/CLAUDE.md, https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/playbooks/DELEGATION_BRIEFS.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md first.
Clone fresh. Use ~/bin/cs, never /usr/local/bin/coursier. Targeted remote runs only (scripts/test.bash --remote --test <name>). Record first-attempt counts from the runner XML.
Each new test must fail on the mutation it targets (remove only that one line or branch) and pass with it restored.
Do not change timeouts, iteration counts, the 600000 ms limit or any assertion, and do not disable tests. Keep the owner-approved two-line change in tests/residentRunnerEventsDoNotReadRunJson.kts.
Before pushing: git fetch origin && git rebase origin/main, confirm the PR is OPEN, push with --force-with-lease. End each commit message with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. Edit the PR description with REST (gh api -X PATCH repos/CodexCoder21Organization/BuildTestEmbedded/pulls/1530 -F body=@file), keeping its opening why-paragraph.
Do NOT mark ready, merge, enqueue, publish or deploy. Do not re-run CI blindly. Report the CI state of the final head, waiting for it with a hard deadline.
Every background wait has a hard deadline and is killed before you finish. Wait on a PID, never on pgrep -f.
Write progress to out/r55-findings.md as OBSERVED <UTC> — … lines. About 90 minutes. Remove your clone when you finish.
Each finding: what changed, the SHA, the pinning test with its fail-before/pass-after or mutation evidence, the full list of guard tests run with first-attempt counts, and the final head's CI state.
Final main drift: main is now https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/daf1d2ad5148ea8a6843a96f53298f2a133345b8. Tested source70b is based on7485f604 and is BEHIND; the later main is not validated by these results. Rebase and re-verify before a source push.
Current r62 task — 2026-10-19
The following user instructions narrow the earlier guard-fix scope for this session and supersede the old direction to fix every failure in this PR:
For each of the nine, run matched comparisons on the PR head and on current main (7485f604 or newer): at least 5 first-attempt runs per side, at the same time, counted from the runner XML. Run each test in its own invocation, as well as within a batch, so that batch-load timing is separated from the PR's behavior.
Classify each one as PR-caused (main passes, PR fails), already on main, or inconclusive (add runs until it isn't).
For each PR-caused test: name the mechanism, fix the production code at its cause on the PR branch, and show it flips with counts. Never change those tests.
For each test that is already on main: record the counts and the failure text. Do not fix it in this PR. List it in the PR description as a known flake on main, and open a GitHub issue for each, with the evidence.
Rebase onto main, push, and report CI on the final head.
Do NOT mark ready, merge, enqueue, publish or deploy. Do not re-run CI blindly.
Every background wait has a hard deadline and is killed before you finish. Wait on a PID, never pgrep -f.
Write progress to out/r62-findings.md as OBSERVED <UTC> — … lines. Checkpoint after the classification table, before any fix. About 2 hours. Remove your clone when done.
At03:47UTC, main is pinned atdaf1d2ad5148ea8a6843a96f53298f2a133345b8 and the clean rebased PR candidate is892e20d2039c3b582e519260a3e43d376f744dee. Its service blob is identical to the incoming70b442c6 service; nine unchanged guard blobs are identical to main. Source checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r62-matched-baseline. Initial plans/evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r62-guard-evidence/handoff-artifacts/r62. No new guard verdict yet; six targeted clients are progressing through provisioning/building under hard bounds. A pre-test provider-status timeout caused requeue; this is not counted as a guard failure. No production code was changed by r62.
Mission summary
The user asked to fix the final-review findings after the shard fix on https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1530 (“Pause provisioning charge while dynamic shards are testing”). Full UrlResolver suites had failed with “Provisioning deadline exceeded”, blocking fixes for a TinySite production outage. A prior lane repaired initial mandatory-shard/cleanup charging but incorrectly left the no-create guard red. This lane proves that no-create regression, fixes it without changing the existing test, removes a redundant settlement pause, adds public replacement-limit release coverage with a single-call mutation, and rewrites the README paragraph. Expanded guard testing exposed further failures; they must be understood and fixed or refuted with evidence before calling this brief done. No production deployment is authorized.
What was found and done
-
Three matched first attempts on historical mainb28d0280 pass the unchanged no-create test; three on incoming7e631db2 fail. The last no-droplet create removed its provisioning worker before backoff, pausing the charge and letting retries outlive600000ms. The fix keeps a worker active through no-create backoff but retains the pause after a physical replacement fails boot. Historical fixed source passes7/7; full XML/run IDs and current-base results are embedded below.
-
The settlement helper’s extra pause is redundant because readiness, worker-finally, or startup recovery already owns it; removed. The replacement-limit path starts no create, so its extra helper pause is redundant too. The new test pins the load-bearing release call, holds two sibling leases for600001ms, observes no fourth replacement for the lost shard, then replaces a second sibling and observes public TESTING history. Removing only the release call fails at public dropletId expected0/actual6 on14a and again on rebased70b. Do not misrepresent this as proof that a redundant pause is necessary.
-
Main advanced to7485f604 during review when https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1531 merged an independent SSH spool recovery change. The candidate was rebased to70b442c69422f0f484a9a3e75fa8e24cc73696bc; service/README/new-test blobs did not change. A fresh183-guard plus11-critical-selector matrix and a same-head mutation were run to check that integration, rather than silently relabelling old proof as new. The rebased independent review found no missing fixture command or provider implementation call.
-
The initial expanded matrix has six failing first attempts; the current-base matrix has nine failures, twelve unique failures across both matrices. Two also fail on unchanged main, and a main-context comparison adds another main-only failure. No fixture or production mechanism is reliably reproduced yet. In particular, SSH preparation timing and manually advanced clock observations are evidence to investigate, not a root-cause conclusion. The bytecode shows FakeDropletService0.0.20 shares one static EC host key; repeated RSA key generation is refuted. Diagnostic-only phase-print patches are preserved but never built/run. Both delayed-reply full timeout wrappers were recovered, while the original consoles remain unavailable; that evidence gap is explicit.
Relevant PRs / refs
| Repository |
Branch |
Verified remote head |
PR |
What is on it |
State |
| BuildTestEmbedded |
fix/pause-provisioning-while-testing-r27 |
70b442c69422f0f484a9a3e75fa8e24cc73696bc |
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1530 |
PR branch with the reviewed repair |
Draft; inspect final CI snapshot below |
| BuildTestEmbedded |
wip/r55-rebased-review-findings |
70b442c69422f0f484a9a3e75fa8e24cc73696bc |
no PR |
Rebased repair candidate70b |
Current-base validation recorded below |
| BuildTestEmbedded |
wip/r55-review-evidence |
611174f1bcfa18082e8e7f4035d9e2c0e336bb04 |
no PR |
Intentional forensic records, runner XML, full available errors, scripts and diagnostic patches |
Evidence branch only; rebased onto later main for preservation, code context unvalidated; never merge into the product |
| BuildTestEmbedded |
wip/r55-review-findings |
14a429c9fd6423847aa88b5b100821a126d7a819 |
no PR |
Pre-rebase repair14a |
Historical valid proof; superseded by rebased candidate |
| BuildTestEmbedded |
wip/r55-settlement-pause-removal |
c0d049ed50d7dcd889ad2be9231650fa85f7a2fb |
no PR |
Pause-removal-only checkpoint |
Three named pause guards pass1/1; superseded |
| BuildTestEmbedded |
wip/r55-superseded-limit-prototype |
61c63fb578b3acef472ebed026d81fcf2338eba2 |
no PR |
Earlier limit-test/prototype history including two local commits |
Superseded; do not adopt over final public test |
No code was deployed or published by this effort. Command/action records contain only source pushes, REST description changes, targeted remote test workspaces and read-only CI observation; there was no production deployment, container restart, Maven publication, merge, queue entry or readiness change. Ephemeral remote test workspaces are not production deployments. No production state claim is made.
All touched worktrees were enumerated in worktree-sweep/. Dirty diagnostic patches are preserved there and in deletion-profile-{main,pr}.diff. No stash was left. The superseded prototype commits are also remote. Jars, caches and build outputs were deliberately excluded and deleted during final cleanup. The raw evidence below is on https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r55-review-evidence/handoff-artifacts/r55.
Full report and first-attempt ledger
BuildTestEmbedded final-review repair evidence
PR: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1530
Written 2026-10-19T03:26:53.032375+00:00. The current candidate is 70b442c69422f0f484a9a3e75fa8e24cc73696bc. Main advanced during review to 7485f604; the candidate was rebased without changing its provisioning service, README or final limit-test blobs. Historical results and current-base integration results are recorded separately below.
No-create deadline regression
The create that returned no droplet removed its worker from the active-attempt set before retry backoff. With the last worker removed, the deadline wait saw a paused budget and could keep scheduling no-create requests indefinitely. Retaining that worker through backoff lets the existing remaining-budget wait enforce the unchanged 600000 ms limit. Backoff after an allocated replacement fails boot still pauses, so testing-only time remains excluded.
Fix after rebase: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/2b1bae4eccb8ccdcf04e1cf9b581688288b1d549
The unchanged e2eDynamicNoCreateBudgetReturnsReservation passes 3/3 on main b28d0280, fails 0/3 on incoming 7e631db2 at its terminal-at-budget assertion, then passes 7/7 on fixed source. Each record has canonical/highest attempt 1. The pre-registered seven fixed passes meet P<=0.5^7<1% against an unchanged failure rate>=50%. Full original error is in no-create-before-full-stack.txt.
Finding 1: cleanup, mandatory shard zero and expanded guards
Both cleanup-deadline guards and the retention/mandatory-zero guard pass on the incoming head and repaired source. The initial guard from the prior lane is retained at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/54130618058e55bdc035cb07311925fc452fafca.
| Named guard |
Main b28d0280 |
Incoming7e631db2 |
Repaired source |
cleanupDeletionNeverConfirmedEndsAtProvisioningDeadline |
not run |
3/3 |
1/1 |
ambiguousCleanupListingThatNeverResolvesEndsAtProvisioningDeadline |
not run |
3/3 |
1/1 |
e2eDynamicNoCreateBudgetReturnsReservation |
3/3 |
0/3 |
7/7 |
testRetentionMaintenanceWakeKeepsMandatoryShardZeroProvisioning |
3/3 |
3/3 |
1/1 |
The expanded inventory has 183 actual runnable selectors: 25 prior selectors, 55 additional budget/deadline guards, 59 allocation-cleanup guards and 44 adjacent retry/reservation guards. Selection evidence is in guard-discovery-findings.md and guard-inventory.tsv. All 183 ran once; 177 distinct guards pass first attempt and six fail. Including deliberate no-create and critical repeats, 185/191 fixed-source selected records pass, all canonical/highest 1. These failures are preserved, not replaced by reruns; cause/comparison investigation is ongoing. See extra-guard-failures.md and extra-guard-root-analysis.md.
Finding 2: settlement pause and replacement limit
The extra pauseProvisioningBudgetIfIdle inside persistCompletedRecoveredShardRelease is redundant: readiness/worker-finally owns the normal pause; startup recovery owns the persisted pause before recovered owners attach. The replacement-limit branch creates no new worker. The redundant call is removed in https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/eda0b306e4ca834c276594125451476f4a840570. With only that call removed, testOrchestratorFinishedDynamicShardLossKeepsProvisioningBudgetPaused, testDynamicReplacementRetryBackoffPausesProvisioningBudget and testReadinessLossBeforeLastBootAttemptEndsPausesProvisioningBudget each pass 1/1 on attempt 1 (run9af684fc).
The new testDynamicReplacementLimitReleasesSlotWithBudgetPaused pins the three-generation limit and public slot release; holds sibling leases for 600001 ms; then proves remaining budget by creating a second lost sibling's replacement and observing TESTING history. The mutation removes only the replacement-limit branch's persistCompletedRecoveredShardRelease call. This is coverage of the load-bearing release call, not evidence that the removed redundant internal pause was load-bearing. At 14a429c9, the exact final test passes 1/1 in run7c40d8cf and fails 0/1 in runece61516 after removing only the release call: “The released shard-zero assignment must be cleared. Expected <0>, actual <6>.” Both canonical/highest attempts are 1. Prototype evidence is superseded and excluded from this proof. Fresh current-base results are listed below.
The first public fixture revision (4b89) failed at the final unchanged phase-count assertion: lease acceptance precedes service test-start event ingestion. The added public-history wait in 14a429 synchronizes that observation without changing assertions, waits, limits, iterations or production code. Its final 14a restored/mutation pair establishes the exact public release assertion above. No assertion was changed.
Finding 3: README ordering and rules
The no-create fix commit (2b1bae4e after rebase) introduces the ten-minute budget before its persisted charge. It explains testing/replacement pauses, no-create backoff charging, resume at provider create and bounded whole-run allocation cleanup, and removes the stale final-sibling wording. README code review is in final-code-review.md; test review is in final-test-review.md.
Full existing guard list: XML first-attempt pass counts
All rows below are selected tests, excluding filtered-out records. Canonical and highest attempt are 1 on every listed execution. Fixed-source runs:6065e986,f0916dc7,5afb3b42,1931f870,db763b6b,86e82940 and601bde52, linked as https://buildtest.kotlin.build/run?id=<id> in runner-ledger.json.
| Guard actually run |
First-attempt passes |
admissionWaitAfterFailedBootWithoutConsumerHeartbeatSurvivesWhileQueued |
1/1 |
admissionWaitDoesNotConsumeProvisioningBudget |
1/1 |
allocationCleanupStartsDeletionWorkerWhenPeriodicWorkersDisabled |
1/1 |
ambiguousCleanupListingReentersAdmission |
1/1 |
ambiguousCleanupListingThatNeverResolvesEndsAtProvisioningDeadline |
1/1 |
ambiguousCleanupListingThroughSandboxBridgeReentersAdmission |
1/1 |
ambiguousCreateAcrossRestartReapsItsDropletOnceAndKeepsPeer |
1/1 |
ambiguousWriteFailureCleanupListingReentersAdmission |
1/1 |
ambiguousWriteFailureCleanupListingThroughSandboxBridgeReentersAdmission |
0/1 |
badGatewayGhostReapedBeforeRestartKeepsLiveRun |
1/1 |
buildRunImplDropletRecordsJsonRoundTrip |
1/1 |
cancelAfterFailedBootAdmissionReparkCancelsPromptly |
1/1 |
cleanupClaimHaltAfterFailureBeforeRecordKeepsOwnership |
1/1 |
cleanupClaimHaltAfterIdBeforeRunRecordKeepsOwnership |
1/1 |
cleanupClaimHaltBeforeFailureRecordKeepsSingleMachineOwnership |
1/1 |
cleanupClaimHaltBeforeFailureRecordKeepsSingleMachineOwnershipWithSpareCapacity |
1/1 |
cleanupClaimHaltBeforeFirstVisibilityKeepsOwnership |
1/1 |
cleanupClaimHaltBeforeInlineDeletionKeepsOwnership |
1/1 |
cleanupClaimHaltDuringQuotaRetryKeepsOwnership |
1/1 |
cleanupClaimHaltInFlightHasBoundedRecovery |
1/1 |
cleanupClaimHaltPreservesWatchdogBoundary |
1/1 |
cleanupClaimHaltSameCallInlineQuotaRetryKeepsOwnership |
1/1 |
cleanupDeletionNeverConfirmedEndsAtProvisioningDeadline |
1/1 |
cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership |
0/1 |
cleanupLegacyRunWithoutClaimKeepsNormalAdmission |
1/1 |
cleanupListingRejectedByClosedRequestBudgetRequeuesRun |
1/1 |
cleanupListingRetriesWhileRequestBudgetResetIsFarAhead |
1/1 |
cleanupResolvedClaimThenRestartUsesNormalAdmission |
1/1 |
closeEndsAllocationCleanupWaitWhoseInterruptWasConsumed |
1/1 |
closeOrdersItselfAgainstOperatorPassesAlreadyActing |
1/1 |
committedOwnershipResolutionRetainsDeletionInventoryAfterRestart |
1/1 |
completeRunWithUnresolvedEarlierAttemptIsNotRewrittenAndItsDropletIsReaped |
1/1 |
concurrentRunWritersPreservePausedProvisioningBudget |
1/1 |
deletedOwnedIdDoesNotReopenAllocationAfterRestart |
1/1 |
e2eAdmissionPressureCleanupProgressSurvivesWatchdogAndReadmits |
1/1 |
e2eAdmissionPressureCleanupWithoutProgressStillFailsWatchdog |
1/1 |
e2eAllocationRequeueBackoffWakesAtRetryTimeWhenClockAdvancesBeforeSleep |
1/1 |
e2eAllocationRequeueBackoffWakesAtRetryTimeWhenClockStepsBackBeforeSleep |
1/1 |
e2eAllocationRequeueWaitsForDropletCleanup |
1/1 |
e2eCleanupListingFlapDoesNotRefreshWatchdog |
1/1 |
e2eCloseDuringDeferralLeavesRunResumableWithoutFollowerCreates |
1/1 |
e2eDelayedCompletedDropletReplyDoesNotExhaustProvisioningDeadline |
0/1 |
e2eDropletlessSiblingRetryChargesLiveRequeueWindow |
1/1 |
e2eDynamicDeferredFollowersAreNotReleasedWhilePreparationOutlastsProvisioningBudget |
1/1 |
e2eDynamicFinalNoCreateFailureReleasesReservation |
1/1 |
e2eDynamicFollowerWokenAfterQueueFinishedNeverRequestsDroplet |
1/1 |
e2eDynamicFollowersAreNotCreatedWhenShardZeroFinishesTheQueueFirst |
1/1 |
e2eDynamicNoCreateBudgetReturnsReservation |
7/7 |
e2eDynamicProvisioningDeadlineReleasesNeverActiveShardToSurvivors |
1/1 |
e2eDynamicProvisioningDeadlineWithUnprovisionedShardZeroStillFailsRun |
1/1 |
e2eDynamicProvisioningDeadlineWithoutReadyShardStillFailsRun |
1/1 |
e2eDynamicReplacementKeepsAdmissionReservation |
1/1 |
e2eDynamicReplacementProviderFailureUsesClockBackoff |
1/1 |
e2eDynamicRestartOwedReplacementRetainsReservation |
1/1 |
e2eDynamicSixNoCreateFailures |
1/1 |
e2eFailedBootRestoresAdmissionCapacityPromptly |
1/1 |
e2eOverlappingDeletionWaitsPauseWhenLastAttemptBecomesReady |
1/1 |
e2eProvisioningAdmissionPauseWakesAtResumeTimeWhenClockAdvancesBeforeSleep |
1/1 |
e2eProvisioningAdmissionPauseWakesAtResumeTimeWhenClockStepsBackBeforeSleep |
1/1 |
e2eProvisioningDeadlineFailsPendingRunWhileSiblingBoots |
1/1 |
e2eProvisioningRetryBackoffWakesAtRetryTimeWhenClockAdvancesBeforeSleep |
1/1 |
e2eProvisioningRetryBackoffWakesAtRetryTimeWhenClockStepsBackBeforeSleep |
1/1 |
e2eProvisioningWallClockExpiresAcrossRequeues |
1/1 |
e2eRateLimitExhaustionRequeuesInsteadOfHardFailing |
1/1 |
e2eRateLimitRequeueBackoffExitsOnTerminal |
1/1 |
e2eRepeatedFailedBootRequeuesKeepCountingProvisioningBudget |
1/1 |
e2eRestartDuringFailedDeletionWaitKeepsPausedBudget |
1/1 |
e2eRestartQueuedRunSurvivesQuotaSaturationWithoutFailing |
1/1 |
e2eShardZeroProvisioningExhaustedCreatesNoFollowerDroplet |
1/1 |
e2eShardedFailedBootRetryDoesNotDiscardSiblingReservations |
1/1 |
e2eShardedQuotaRetryDoesNotMaskSiblingProvisioningFailure |
1/1 |
e2eSiblingReadinessAfterRequeuePausesProvisioningBudget |
1/1 |
e2eSiblingReadyDuringRetryBackoffPausesProvisioningBudget |
1/1 |
e2eStaleKillReleasesAdmissionReservationsWhileProvisioningRetries |
1/1 |
e2eStaleKillStopsZombieProvisioningRetries |
1/1 |
e2eStalledCreateReleasesAdmissionAndRelaunches |
1/1 |
e2eWholeRunRequeueClearsFailedDropletAdmissionHold |
1/1 |
failedBootReplacementDrainsBeforeQueuedRecoveredReservation |
1/1 |
lateGhostReapedBeforeRestartKeepsLiveRun |
1/1 |
legacyNameClaimSettlesCleanupBeforeDeadline |
1/1 |
legacyProvisionedRunResumesWithoutProvisioningDeadlineAfterRestart |
1/1 |
malformedProvisioningBudgetFieldsUseSafeFallbacks |
1/1 |
nonAmbiguousCleanupListingThroughSandboxBridgeFailsRun |
1/1 |
ownershipLedgerIdsSurviveRestartWithoutRunObservations |
1/1 |
pendingDetailsReportRealProvisioningRetryTime |
1/1 |
providerRefusedCleanupListingAfterCreateStallFailsRunWithPrimaryFailure |
1/1 |
provisioningDeadlineFailureLogNamesProvisioningDeadline |
1/1 |
provisioningKeepsJustCreatedDropletThroughTransientNotFound |
1/1 |
provisioningWaitsOutAdmissionRefusalWithoutDiscardingItsDroplet |
1/1 |
queuedV4dRunRemainsPendingWhenReadByCurrentService |
1/1 |
quotaCapacityRecoveryProceedsBeforeTotalWindow |
1/1 |
quotaGhostCleanedInlineKeepsLiveRunAcrossRestart |
1/1 |
quotaRefusalThenRetryKeepsLiveRunAcrossRestart |
1/1 |
quotaSaturatedRunExposesPendingReason |
1/1 |
quotaSaturationRemainsPendingBeyondFormerTotalWindow |
1/1 |
regionFallbackAfterOrdinaryErrorKeepsLiveRunAcrossRestart |
1/1 |
repeatedCleanupListingStallsFailOnlyAtWatchdogBoundary |
1/1 |
repeatedCleanupListingStallsRetainRunAndReadmit |
1/1 |
replacedShardAndInterruptedAllocationAreTreatedSeparatelyAfterRestart |
1/1 |
replacementInFlightAtRestartKeepsOtherShards |
1/1 |
replacementPendingRunKeepsUnrecordedDropletUntilItEnds |
1/1 |
requeueBoundaryPausesProvisioningBudgetBeforeQueuePersistence |
1/1 |
residentRunnerEventsDoNotReadRunJson |
1/1 |
restartAfterUnknownCreateOutcomeStillCleansUpAllocation |
1/1 |
restartDuringActiveProvisioningEpisodeKeepsRemainingBudget |
1/1 |
restartDuringInFlightCreateStillCleansUpAllocation |
1/1 |
restartedProvisioningDeadlineResumesAccumulatedActiveTime |
1/1 |
resumedShardedRunNeedsDynamicSurvivorBeforeReplacingMissingShards |
1/1 |
reviewSettledFollowerAbnormalExitReturnsReservation |
1/1 |
runCancelledDuringCancelledCleanupListingEndsCanceled |
1/1 |
startupMetadataIsCommittedBeforeConstructionReturns |
1/1 |
startupServesWhilePersistedRecoveryBacklogIsStillRunning |
1/1 |
stoppedOwnedDropletStillRequiresConfirmedAbsence |
1/1 |
surplusDropletListedAtRestartIsDeletedWithoutTouchingRun |
1/1 |
testAccountQuotaRetryContinuesPastThreeAttempts |
1/1 |
testAssignedFollowerExceptionRetainsPhysicalChargeUntilDeletion |
1/1 |
testBudgetPauseRacingFollowerCreateKeepsProvisioningDeadline |
1/1 |
testCancellationDuringHeldAllocationCleanupStaysTerminal |
1/1 |
testCompactRunProjectionKeepsActiveQueuedProvisioningPhase |
1/1 |
testCompactRunProjectionKeepsLegacyQueuedProvisioningPhase |
1/1 |
testCompactRunProjectionOmitsOnlyQueuedProvisioningPhase |
1/1 |
testDeferredFollowerWaitEndsAtShardZeroPreparationDeadline |
1/1 |
testDynamicReplacementRetryBackoffPausesProvisioningBudget |
1/1 |
testEveryUncreatedSlotRetriesOnceBeforeDeadline |
1/1 |
testF1377eLateReadyFollowerReservationPreventsDeletion |
1/1 |
testFollowersProvisionFromRemainingBudgetAfterShardZero |
1/1 |
testHungSeedPreparationReachesDeadlineAndStopsFollowers |
1/1 |
testHungSeedPreparationWithStaleConsumerKeepsAbandonmentVerdict |
1/1 |
testLateDynamicReplacementAfterReopenExcludesTestingFromProvisioningBudget |
1/1 |
testLateDynamicReplacementExcludesTestingFromProvisioningBudget |
2/2 |
testLateDynamicReplacementKeepsGenuineProvisioningDeadline |
2/2 |
testLateDynamicReplacementsOverlapWithoutPausingProvisioningBudget |
1/1 |
testOrchestratorFinishedDynamicShardLossKeepsProvisioningBudgetPaused |
1/1 |
testProvisioningBudgetResumesWhenDeferredFollowerStartsCreatingItsDroplet |
1/1 |
testProvisioningRetryInterruptionKeepsOriginalCause |
1/1 |
testProvisioningRetryInterruptionReportsAndPreservesStatus |
1/1 |
testQuotaAllocationFailureRetainsHealthySibling |
1/1 |
testReadinessLossBeforeLastBootAttemptEndsPausesProvisioningBudget |
1/1 |
testReleasedFollowerActivationUsesExistingProvisioningBound |
0/1 |
testReleasedFollowerCachePublicationWaitUsesRunDeadline |
1/1 |
testReleasedFollowerReturnsOriginalReservationWithReplacementAllowance |
1/1 |
testReleasedFollowerSettlementSurvivesRepeatedRestarts |
1/1 |
testReleasedFollowerStoppedBeforeCreateReturnsRetainsReservation |
1/1 |
testReleasedFollowerWithLostStatusReplyIsSettledWhileRunLives |
1/1 |
testReleasedGoneResultReservationWaitsForProvisioningUnregister |
1/1 |
testReleasedNoDropletReservationRequiresThreeCompleteObservations |
1/1 |
testReleasedNoDropletReservationSettlesWithinObservationWindow |
0/1 |
testReleasedShardDeletionWaitRetainsCause |
0/1 |
testReleasedUncreatedSlotLateDeletionKeepsPrimaryReservation |
1/1 |
testReleasedUncreatedSlotsWakeQueuedRun |
1/1 |
testReleasedUnprovisionedDynamicSlotsAdmitNextRun |
1/1 |
testReleasedUnresolvedCreateKeepsAdmissionReservation |
1/1 |
testRepeatedAdmissionProvisioningCyclesGetOneFreshWindow |
1/1 |
testRestartedServiceReapsStalledProvisioningAtBudget |
1/1 |
testRetentionMaintenanceWakeKeepsMandatoryShardZeroProvisioning |
1/1 |
testRev1421aAssignedReleasedFollowerInstallsWorkspaceCache |
1/1 |
testRev1421cCrossRunPublicationFailureKeepsRunArchive |
1/1 |
testRev1421cLateCompletedArchiveIsNotInstalled |
1/1 |
testReviewRetentionRestartDoesNotCompleteUnresolvedProvisioning |
1/1 |
testSupersededBudgetPauseKeepsProvisioningDeadline |
1/1 |
testTerminalRestartAdmissionMatchesExistingBehavior |
1/1 |
testTerminationTimerDependencyOutageRetainsHealthySibling |
1/1 |
testUnansweredCreationKeepsReservationAfterSiblingCleanup |
1/1 |
testUnansweredCreationKeepsReservationDuringAllocationCleanup |
1/1 |
testWholeRunRequeueAfterDeletionAlreadyConfirmed |
1/1 |
timerSetupFailureFallbackKeepsLiveRunAcrossRestart |
1/1 |
twoAmbiguousCleanupListingsThenComplete |
1/1 |
twoRunsSharingStalledCleanupListingsBothRetainAndReadmit |
1/1 |
watchdogDoesNotFailProvisioningBuildBeforeFiveMinutes |
1/1 |
watchdogFailsProvisioningBuildAfterFiveMinutes |
1/1 |
watchdogFrozenProvisioningReportsLastProgress |
1/1 |
watchdogGlobalCeilingDoesNotExpireProviderQuotaPending |
1/1 |
watchdogKeepsFailedBootReplacementPendingBeyondFormerAdmissionLimit |
1/1 |
watchdogKeepsRateLimitedBuildPendingBeyondFormerAdmissionCeiling |
1/1 |
watchdogLegacyProvisioningRunGetsFreshBudgetAfterRestart |
1/1 |
watchdogLogInspectionIsBoundedAndProvisioningOnly |
1/1 |
watchdogNeverExpiresAdmissionAcrossRequeueCycles |
1/1 |
watchdogNeverExpiresGenuineAdmissionQueueWait |
1/1 |
watchdogProvisioningProgressSurvivesBeyondWallClockBudget |
1/1 |
watchdogProvisioningProgressSurvivesServiceRestart |
1/1 |
watchdogProvisioningRunWithMissingDropletIsJudgedOnce |
1/1 |
watchdogProvisioningTimeoutDestroysDropletIfPresent |
1/1 |
watchdogProvisioningTimeoutDoesNotAffectOtherStatuses |
1/1 |
Rules and terminal state
The owner-approved tests/residentRunnerEventsDoNotReadRunJson.kts SHA256 remains6aa95cd20b0c93ce54d8c85d8dc8ec13c10f979714b307ec0426cbcf168e3af9. The r55-only diff from rebased incoming54130618058e55bdc035cb07311925fc452fafca adds only the new limit test; it changes no existing test. Main’s separate merged SSH change modifies its own tests. No existing assertion, iteration, timeout or600000ms limit changed. Targeted remote calls only, no CI reruns, no readiness change/merge/enqueue/publication/deployment.
Final-head CI and cleanup are recorded in the terminal state below.
Current-base integration results
Candidate https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/70b442c69422f0f484a9a3e75fa8e24cc73696bc includes main7485f604. All selected direct verification remains targeted remote testing. 190/199 selected records pass on attempt one; expected199 records across184 unique tests if all seven commands complete. Missing records are not passes.
| Existing guard actually run |
Historical repaired source |
Current base |
admissionWaitAfterFailedBootWithoutConsumerHeartbeatSurvivesWhileQueued |
1/1 |
1/1 |
admissionWaitDoesNotConsumeProvisioningBudget |
1/1 |
1/1 |
allocationCleanupStartsDeletionWorkerWhenPeriodicWorkersDisabled |
1/1 |
1/1 |
ambiguousCleanupListingReentersAdmission |
1/1 |
1/1 |
ambiguousCleanupListingThatNeverResolvesEndsAtProvisioningDeadline |
1/1 |
2/2 |
ambiguousCleanupListingThroughSandboxBridgeReentersAdmission |
1/1 |
1/1 |
ambiguousCreateAcrossRestartReapsItsDropletOnceAndKeepsPeer |
1/1 |
1/1 |
ambiguousWriteFailureCleanupListingReentersAdmission |
1/1 |
1/1 |
ambiguousWriteFailureCleanupListingThroughSandboxBridgeReentersAdmission |
0/1 |
1/1 |
badGatewayGhostReapedBeforeRestartKeepsLiveRun |
1/1 |
1/1 |
buildRunImplDropletRecordsJsonRoundTrip |
1/1 |
1/1 |
cancelAfterFailedBootAdmissionReparkCancelsPromptly |
1/1 |
1/1 |
cleanupClaimHaltAfterFailureBeforeRecordKeepsOwnership |
1/1 |
1/1 |
cleanupClaimHaltAfterIdBeforeRunRecordKeepsOwnership |
1/1 |
1/1 |
cleanupClaimHaltBeforeFailureRecordKeepsSingleMachineOwnership |
1/1 |
1/1 |
cleanupClaimHaltBeforeFailureRecordKeepsSingleMachineOwnershipWithSpareCapacity |
1/1 |
1/1 |
cleanupClaimHaltBeforeFirstVisibilityKeepsOwnership |
1/1 |
1/1 |
cleanupClaimHaltBeforeInlineDeletionKeepsOwnership |
1/1 |
1/1 |
cleanupClaimHaltDuringQuotaRetryKeepsOwnership |
1/1 |
1/1 |
cleanupClaimHaltInFlightHasBoundedRecovery |
1/1 |
1/1 |
cleanupClaimHaltPreservesWatchdogBoundary |
1/1 |
1/1 |
cleanupClaimHaltSameCallInlineQuotaRetryKeepsOwnership |
1/1 |
1/1 |
cleanupDeletionNeverConfirmedEndsAtProvisioningDeadline |
1/1 |
2/2 |
cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership |
0/1 |
0/1 |
cleanupLegacyRunWithoutClaimKeepsNormalAdmission |
1/1 |
1/1 |
cleanupListingRejectedByClosedRequestBudgetRequeuesRun |
1/1 |
1/1 |
cleanupListingRetriesWhileRequestBudgetResetIsFarAhead |
1/1 |
1/1 |
cleanupResolvedClaimThenRestartUsesNormalAdmission |
1/1 |
1/1 |
closeEndsAllocationCleanupWaitWhoseInterruptWasConsumed |
1/1 |
1/1 |
closeOrdersItselfAgainstOperatorPassesAlreadyActing |
1/1 |
1/1 |
committedOwnershipResolutionRetainsDeletionInventoryAfterRestart |
1/1 |
1/1 |
completeRunWithUnresolvedEarlierAttemptIsNotRewrittenAndItsDropletIsReaped |
1/1 |
1/1 |
concurrentRunWritersPreservePausedProvisioningBudget |
1/1 |
1/1 |
deletedOwnedIdDoesNotReopenAllocationAfterRestart |
1/1 |
1/1 |
e2eAdmissionPressureCleanupProgressSurvivesWatchdogAndReadmits |
1/1 |
1/1 |
e2eAdmissionPressureCleanupWithoutProgressStillFailsWatchdog |
1/1 |
1/1 |
e2eAllocationRequeueBackoffWakesAtRetryTimeWhenClockAdvancesBeforeSleep |
1/1 |
1/1 |
e2eAllocationRequeueBackoffWakesAtRetryTimeWhenClockStepsBackBeforeSleep |
1/1 |
1/1 |
e2eAllocationRequeueWaitsForDropletCleanup |
1/1 |
1/1 |
e2eCleanupListingFlapDoesNotRefreshWatchdog |
1/1 |
1/1 |
e2eCloseDuringDeferralLeavesRunResumableWithoutFollowerCreates |
1/1 |
1/1 |
e2eDelayedCompletedDropletReplyDoesNotExhaustProvisioningDeadline |
0/1 |
0/1 |
e2eDropletlessSiblingRetryChargesLiveRequeueWindow |
1/1 |
1/1 |
e2eDynamicDeferredFollowersAreNotReleasedWhilePreparationOutlastsProvisioningBudget |
1/1 |
1/1 |
e2eDynamicFinalNoCreateFailureReleasesReservation |
1/1 |
1/1 |
e2eDynamicFollowerWokenAfterQueueFinishedNeverRequestsDroplet |
1/1 |
1/1 |
e2eDynamicFollowersAreNotCreatedWhenShardZeroFinishesTheQueueFirst |
1/1 |
1/1 |
e2eDynamicNoCreateBudgetReturnsReservation |
7/7 |
7/7 |
e2eDynamicProvisioningDeadlineReleasesNeverActiveShardToSurvivors |
1/1 |
0/1 |
e2eDynamicProvisioningDeadlineWithUnprovisionedShardZeroStillFailsRun |
1/1 |
1/1 |
e2eDynamicProvisioningDeadlineWithoutReadyShardStillFailsRun |
1/1 |
1/1 |
e2eDynamicReplacementKeepsAdmissionReservation |
1/1 |
1/1 |
e2eDynamicReplacementProviderFailureUsesClockBackoff |
1/1 |
1/1 |
e2eDynamicRestartOwedReplacementRetainsReservation |
1/1 |
1/1 |
e2eDynamicSixNoCreateFailures |
1/1 |
1/1 |
e2eFailedBootRestoresAdmissionCapacityPromptly |
1/1 |
1/1 |
e2eOverlappingDeletionWaitsPauseWhenLastAttemptBecomesReady |
1/1 |
1/1 |
e2eProvisioningAdmissionPauseWakesAtResumeTimeWhenClockAdvancesBeforeSleep |
1/1 |
1/1 |
e2eProvisioningAdmissionPauseWakesAtResumeTimeWhenClockStepsBackBeforeSleep |
1/1 |
1/1 |
e2eProvisioningDeadlineFailsPendingRunWhileSiblingBoots |
1/1 |
1/1 |
e2eProvisioningRetryBackoffWakesAtRetryTimeWhenClockAdvancesBeforeSleep |
1/1 |
1/1 |
e2eProvisioningRetryBackoffWakesAtRetryTimeWhenClockStepsBackBeforeSleep |
1/1 |
1/1 |
e2eProvisioningWallClockExpiresAcrossRequeues |
1/1 |
1/1 |
e2eRateLimitExhaustionRequeuesInsteadOfHardFailing |
1/1 |
1/1 |
e2eRateLimitRequeueBackoffExitsOnTerminal |
1/1 |
1/1 |
e2eRepeatedFailedBootRequeuesKeepCountingProvisioningBudget |
1/1 |
1/1 |
e2eRestartDuringFailedDeletionWaitKeepsPausedBudget |
1/1 |
1/1 |
e2eRestartQueuedRunSurvivesQuotaSaturationWithoutFailing |
1/1 |
1/1 |
e2eShardZeroProvisioningExhaustedCreatesNoFollowerDroplet |
1/1 |
1/1 |
e2eShardedFailedBootRetryDoesNotDiscardSiblingReservations |
1/1 |
1/1 |
e2eShardedQuotaRetryDoesNotMaskSiblingProvisioningFailure |
1/1 |
1/1 |
e2eSiblingReadinessAfterRequeuePausesProvisioningBudget |
1/1 |
1/1 |
e2eSiblingReadyDuringRetryBackoffPausesProvisioningBudget |
1/1 |
1/1 |
e2eStaleKillReleasesAdmissionReservationsWhileProvisioningRetries |
1/1 |
1/1 |
e2eStaleKillStopsZombieProvisioningRetries |
1/1 |
1/1 |
e2eStalledCreateReleasesAdmissionAndRelaunches |
1/1 |
1/1 |
e2eWholeRunRequeueClearsFailedDropletAdmissionHold |
1/1 |
1/1 |
failedBootReplacementDrainsBeforeQueuedRecoveredReservation |
1/1 |
1/1 |
lateGhostReapedBeforeRestartKeepsLiveRun |
1/1 |
1/1 |
legacyNameClaimSettlesCleanupBeforeDeadline |
1/1 |
1/1 |
legacyProvisionedRunResumesWithoutProvisioningDeadlineAfterRestart |
1/1 |
1/1 |
malformedProvisioningBudgetFieldsUseSafeFallbacks |
1/1 |
1/1 |
nonAmbiguousCleanupListingThroughSandboxBridgeFailsRun |
1/1 |
1/1 |
ownershipLedgerIdsSurviveRestartWithoutRunObservations |
1/1 |
1/1 |
pendingDetailsReportRealProvisioningRetryTime |
1/1 |
1/1 |
providerRefusedCleanupListingAfterCreateStallFailsRunWithPrimaryFailure |
1/1 |
1/1 |
provisioningDeadlineFailureLogNamesProvisioningDeadline |
1/1 |
1/1 |
provisioningKeepsJustCreatedDropletThroughTransientNotFound |
1/1 |
1/1 |
provisioningWaitsOutAdmissionRefusalWithoutDiscardingItsDroplet |
1/1 |
1/1 |
queuedV4dRunRemainsPendingWhenReadByCurrentService |
1/1 |
1/1 |
quotaCapacityRecoveryProceedsBeforeTotalWindow |
1/1 |
1/1 |
quotaGhostCleanedInlineKeepsLiveRunAcrossRestart |
1/1 |
1/1 |
quotaRefusalThenRetryKeepsLiveRunAcrossRestart |
1/1 |
1/1 |
quotaSaturatedRunExposesPendingReason |
1/1 |
1/1 |
quotaSaturationRemainsPendingBeyondFormerTotalWindow |
1/1 |
1/1 |
regionFallbackAfterOrdinaryErrorKeepsLiveRunAcrossRestart |
1/1 |
1/1 |
repeatedCleanupListingStallsFailOnlyAtWatchdogBoundary |
1/1 |
1/1 |
repeatedCleanupListingStallsRetainRunAndReadmit |
1/1 |
1/1 |
replacedShardAndInterruptedAllocationAreTreatedSeparatelyAfterRestart |
1/1 |
1/1 |
replacementInFlightAtRestartKeepsOtherShards |
1/1 |
1/1 |
replacementPendingRunKeepsUnrecordedDropletUntilItEnds |
1/1 |
1/1 |
requeueBoundaryPausesProvisioningBudgetBeforeQueuePersistence |
1/1 |
1/1 |
residentRunnerEventsDoNotReadRunJson |
1/1 |
1/1 |
restartAfterUnknownCreateOutcomeStillCleansUpAllocation |
1/1 |
1/1 |
restartDuringActiveProvisioningEpisodeKeepsRemainingBudget |
1/1 |
1/1 |
restartDuringInFlightCreateStillCleansUpAllocation |
1/1 |
1/1 |
restartedProvisioningDeadlineResumesAccumulatedActiveTime |
1/1 |
1/1 |
resumedShardedRunNeedsDynamicSurvivorBeforeReplacingMissingShards |
1/1 |
0/1 |
reviewSettledFollowerAbnormalExitReturnsReservation |
1/1 |
0/1 |
runCancelledDuringCancelledCleanupListingEndsCanceled |
1/1 |
1/1 |
startupMetadataIsCommittedBeforeConstructionReturns |
1/1 |
1/1 |
startupServesWhilePersistedRecoveryBacklogIsStillRunning |
1/1 |
1/1 |
stoppedOwnedDropletStillRequiresConfirmedAbsence |
1/1 |
1/1 |
surplusDropletListedAtRestartIsDeletedWithoutTouchingRun |
1/1 |
1/1 |
testAccountQuotaRetryContinuesPastThreeAttempts |
1/1 |
1/1 |
testAssignedFollowerExceptionRetainsPhysicalChargeUntilDeletion |
1/1 |
1/1 |
testBudgetPauseRacingFollowerCreateKeepsProvisioningDeadline |
1/1 |
1/1 |
testCancellationDuringHeldAllocationCleanupStaysTerminal |
1/1 |
1/1 |
testCompactRunProjectionKeepsActiveQueuedProvisioningPhase |
1/1 |
1/1 |
testCompactRunProjectionKeepsLegacyQueuedProvisioningPhase |
1/1 |
1/1 |
testCompactRunProjectionOmitsOnlyQueuedProvisioningPhase |
1/1 |
1/1 |
testDeferredFollowerWaitEndsAtShardZeroPreparationDeadline |
1/1 |
1/1 |
testDynamicReplacementRetryBackoffPausesProvisioningBudget |
1/1 |
2/2 |
testEveryUncreatedSlotRetriesOnceBeforeDeadline |
1/1 |
1/1 |
testF1377eLateReadyFollowerReservationPreventsDeletion |
1/1 |
0/1 |
testFollowersProvisionFromRemainingBudgetAfterShardZero |
1/1 |
1/1 |
testHungSeedPreparationReachesDeadlineAndStopsFollowers |
1/1 |
1/1 |
testHungSeedPreparationWithStaleConsumerKeepsAbandonmentVerdict |
1/1 |
1/1 |
testLateDynamicReplacementAfterReopenExcludesTestingFromProvisioningBudget |
1/1 |
1/1 |
testLateDynamicReplacementExcludesTestingFromProvisioningBudget |
2/2 |
2/2 |
testLateDynamicReplacementKeepsGenuineProvisioningDeadline |
2/2 |
2/2 |
testLateDynamicReplacementsOverlapWithoutPausingProvisioningBudget |
1/1 |
2/2 |
testOrchestratorFinishedDynamicShardLossKeepsProvisioningBudgetPaused |
1/1 |
2/2 |
testProvisioningBudgetResumesWhenDeferredFollowerStartsCreatingItsDroplet |
1/1 |
1/1 |
testProvisioningRetryInterruptionKeepsOriginalCause |
1/1 |
1/1 |
testProvisioningRetryInterruptionReportsAndPreservesStatus |
1/1 |
1/1 |
testQuotaAllocationFailureRetainsHealthySibling |
1/1 |
1/1 |
testReadinessLossBeforeLastBootAttemptEndsPausesProvisioningBudget |
1/1 |
2/2 |
testReleasedFollowerActivationUsesExistingProvisioningBound |
0/1 |
0/1 |
testReleasedFollowerCachePublicationWaitUsesRunDeadline |
1/1 |
1/1 |
testReleasedFollowerReturnsOriginalReservationWithReplacementAllowance |
1/1 |
1/1 |
testReleasedFollowerSettlementSurvivesRepeatedRestarts |
1/1 |
1/1 |
testReleasedFollowerStoppedBeforeCreateReturnsRetainsReservation |
1/1 |
1/1 |
testReleasedFollowerWithLostStatusReplyIsSettledWhileRunLives |
1/1 |
1/1 |
testReleasedGoneResultReservationWaitsForProvisioningUnregister |
1/1 |
1/1 |
testReleasedNoDropletReservationRequiresThreeCompleteObservations |
1/1 |
1/1 |
testReleasedNoDropletReservationSettlesWithinObservationWindow |
0/1 |
1/1 |
testReleasedShardDeletionWaitRetainsCause |
0/1 |
1/1 |
testReleasedUncreatedSlotLateDeletionKeepsPrimaryReservation |
1/1 |
1/1 |
testReleasedUncreatedSlotsWakeQueuedRun |
1/1 |
1/1 |
testReleasedUnprovisionedDynamicSlotsAdmitNextRun |
1/1 |
1/1 |
testReleasedUnresolvedCreateKeepsAdmissionReservation |
1/1 |
1/1 |
testRepeatedAdmissionProvisioningCyclesGetOneFreshWindow |
1/1 |
1/1 |
testRestartedServiceReapsStalledProvisioningAtBudget |
1/1 |
1/1 |
testRetentionMaintenanceWakeKeepsMandatoryShardZeroProvisioning |
1/1 |
2/2 |
testRev1421aAssignedReleasedFollowerInstallsWorkspaceCache |
1/1 |
1/1 |
testRev1421cCrossRunPublicationFailureKeepsRunArchive |
1/1 |
0/1 |
testRev1421cLateCompletedArchiveIsNotInstalled |
1/1 |
1/1 |
testReviewRetentionRestartDoesNotCompleteUnresolvedProvisioning |
1/1 |
1/1 |
testSupersededBudgetPauseKeepsProvisioningDeadline |
1/1 |
1/1 |
testTerminalRestartAdmissionMatchesExistingBehavior |
1/1 |
0/1 |
testTerminationTimerDependencyOutageRetainsHealthySibling |
1/1 |
1/1 |
testUnansweredCreationKeepsReservationAfterSiblingCleanup |
1/1 |
1/1 |
testUnansweredCreationKeepsReservationDuringAllocationCleanup |
1/1 |
1/1 |
testWholeRunRequeueAfterDeletionAlreadyConfirmed |
1/1 |
1/1 |
timerSetupFailureFallbackKeepsLiveRunAcrossRestart |
1/1 |
1/1 |
twoAmbiguousCleanupListingsThenComplete |
1/1 |
1/1 |
twoRunsSharingStalledCleanupListingsBothRetainAndReadmit |
1/1 |
1/1 |
watchdogDoesNotFailProvisioningBuildBeforeFiveMinutes |
1/1 |
1/1 |
watchdogFailsProvisioningBuildAfterFiveMinutes |
1/1 |
1/1 |
watchdogFrozenProvisioningReportsLastProgress |
1/1 |
1/1 |
watchdogGlobalCeilingDoesNotExpireProviderQuotaPending |
1/1 |
1/1 |
watchdogKeepsFailedBootReplacementPendingBeyondFormerAdmissionLimit |
1/1 |
1/1 |
watchdogKeepsRateLimitedBuildPendingBeyondFormerAdmissionCeiling |
1/1 |
1/1 |
watchdogLegacyProvisioningRunGetsFreshBudgetAfterRestart |
1/1 |
1/1 |
watchdogLogInspectionIsBoundedAndProvisioningOnly |
1/1 |
1/1 |
watchdogNeverExpiresAdmissionAcrossRequeueCycles |
1/1 |
1/1 |
watchdogNeverExpiresGenuineAdmissionQueueWait |
1/1 |
1/1 |
watchdogProvisioningProgressSurvivesBeyondWallClockBudget |
1/1 |
1/1 |
watchdogProvisioningProgressSurvivesServiceRestart |
1/1 |
1/1 |
watchdogProvisioningRunWithMissingDropletIsJudgedOnce |
1/1 |
1/1 |
watchdogProvisioningTimeoutDestroysDropletIfPresent |
1/1 |
1/1 |
watchdogProvisioningTimeoutDoesNotAffectOtherStatuses |
1/1 |
1/1 |
New limit test restored: 1/1. Current-base release-call mutation: 0/1. The mutation must fail at the targeted release assertion; inspect limit-rebased-mutation-stack.txt for the exact failure.
Current-base failures (all canonical/highest 1):
| Guard |
First attempt |
Full available evidence |
testF1377eLateReadyFollowerReservationPreventsDeletion |
0/1 |
rebased-guard-failures/testF1377eLateReadyFollowerReservationPreventsDeletion.md |
testTerminalRestartAdmissionMatchesExistingBehavior |
0/1 |
rebased-guard-failures/testTerminalRestartAdmissionMatchesExistingBehavior.md |
resumedShardedRunNeedsDynamicSurvivorBeforeReplacingMissingShards |
0/1 |
rebased-guard-failures/resumedShardedRunNeedsDynamicSurvivorBeforeReplacingMissingShards.md |
reviewSettledFollowerAbnormalExitReturnsReservation |
0/1 |
rebased-guard-failures/reviewSettledFollowerAbnormalExitReturnsReservation.md |
testReleasedFollowerActivationUsesExistingProvisioningBound |
0/1 |
rebased-guard-failures/testReleasedFollowerActivationUsesExistingProvisioningBound.md |
testRev1421cCrossRunPublicationFailureKeepsRunArchive |
0/1 |
rebased-guard-failures/testRev1421cCrossRunPublicationFailureKeepsRunArchive.md |
cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership |
0/1 |
rebased-guard-failures/cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership.md |
e2eDelayedCompletedDropletReplyDoesNotExhaustProvisioningDeadline |
0/1 |
rebased-guard-failures/e2eDelayedCompletedDropletReplyDoesNotExhaustProvisioningDeadline.md |
e2eDynamicProvisioningDeadlineReleasesNeverActiveShardToSurvivors |
0/1 |
rebased-guard-failures/e2eDynamicProvisioningDeadlineReleasesNeverActiveShardToSurvivors.md |
There are nine distinct current-base failures, so 174/183 existing guards pass, or 175/184 unique tests including the passing new test. With repeated critical/no-create selectors,190/199 records pass. These are incomplete guard-fix results, not a DONE claim.
Expanded-guard investigation remains open
The initial six failures remain evidence even if a fresh execution passes. Two also fail on unchanged main: cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership (30s timeout sampled in ManualClock.advanceBy) and testReleasedShardDeletionWaitRetainsCause (30s timeout in a serial ten-round real-SSH scenario). The first main context batch also finds a main-only failure in cleanupListingRetriesWhileRequestBudgetResetIsFarAhead: the clock reaches300 virtual seconds while initial analysis is still queued; that guard passed on the repaired PR source. None is a reliable deterministic mechanism yet. The remaining four initial failures are not classified as PR-caused from a single matched comparison.
| Initially failing guard |
Historical fixed source |
Main controls including context |
Current base |
ambiguousWriteFailureCleanupListingThroughSandboxBridgeReentersAdmission |
0/1 |
2/2 |
1/1 |
cleanupLegacyInventoryClaimBeforeFailureKeepsOwnership |
0/1 |
0/1 |
0/1 |
e2eDelayedCompletedDropletReplyDoesNotExhaustProvisioningDeadline |
0/1 |
1/1 |
0/1 |
testReleasedFollowerActivationUsesExistingProvisioningBound |
0/1 |
1/1 |
0/1 |
testReleasedNoDropletReservationSettlesWithinObservationWindow |
0/1 |
1/1 |
1/1 |
testReleasedShardDeletionWaitRetainsCause |
0/1 |
1/2 |
1/1 |
Full available errors/console are in extra-guard-failures/, main-context-guards3-failures.txt and the XML. The delayed-reply guard’s original838KB console was omitted by the bulk XML; the complete964-character timeout wrapper was recovered from the WUI, but the original full console remains unavailable. The current-base delayed-reply failure’s complete1,062-character wrapper was also recovered separately in rebased-delayed-reply-stack.txt; its output is likewise empty. Bytecode inspection confirms FakeDropletService0.0.20 uses a shared static EC host key, refuting a repeated RSA-generation explanation. Diagnostic-only deletion timing patches were preserved but never built or run, so no timing conclusion is claimed.
The current-base diagnostic failure is documented in rebased-diagnostic-static-evidence.md: the test accepts budget exhaustion only at addSshKeyToDroplet, but the unchanged main/PR code sets awaitGuestReadiness before checking the same budget. The exact observed pair follows that existing block. Changing the reported stage to satisfy the regex would be wrong. No assertion was widened. Current-main control: not run; it is excluded from the PR-source matrix. This identifies an allowed-message gap, but a deterministic ordering is still needed before any harness repair; a control pass alone is not a root cause.
The sole unchanged-current-main control is run https://buildtest.kotlin.build/run?id=2969f958, confirmed by exact project/test identity. The bounded local command ended without producing runner XML; the last console state was BUILDING. No first-attempt verdict is available or counted. The local command and read-only watcher were stopped and reaped by03:23 UTC. The remote run may continue; re-verify it before any further action, and do not blindly rerun it.
Review and evidence
Independent production/test passes are recorded in final-code-review.md, final-test-review.md, final-parent-review.md and final-rebased-review.md. The rebased review traces the new guarded SSH spool reads through the real fixture shell and finds no missing DropletService implementation call. No additional test was run by that review. runner-ledger.json carries every selected row, canonical/highest attempt, XML filename and mapped run ID. run-id-map.json, rebased-run-filter-map.json and MANIFEST.json allow exact run/source reconciliation.
Finding disposition
- No-create regression: fixed at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/2b1bae4eccb8ccdcf04e1cf9b581688288b1d549 with matched3-per-side historical controls and7 fixed first-attempt passes; current-base counts above remain separate.
- Finding1 named guards: verified as listed; expanded-guard root-cause work remains incomplete wherever the evidence above does not establish a cause. No failure is waived by a later pass.
- Finding 2: redundant helper pause removed at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/eda0b306e4ca834c276594125451476f4a840570; replacement-limit release has the exact mutation/restored evidence above. The branch’s extra pause is redundant too; the test deliberately proves release rather than pretending a redundant pause is necessary.
- Finding 3: README ordering and actual rules updated in the no-create fix commit; no separate production deployment.
Terminal state
Observed 2026-10-19T03:25:07.922911+00:00: PR is OPEN, draft=True, head=70b442c69422f0f484a9a3e75fa8e24cc73696bc, merge status=BEHIND. Check rollup: 2 COMPLETED/SUCCESS; 14 IN_PROGRESS/no conclusion. Required bld-build: not reported. Workflow links: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39193278098. No CI rerun, readiness change, merge, enqueue, publication or deployment was performed.
Verified 2026-10-19T03:26:52.706345+00:00: all owned clone/worktree directories and development build-output directories were removed. workspace/ no longer exists. Every tracked local wait PID is absent or reaped; all three agents finished. Forensic records remain in out/ and on the verified remote evidence branch. No remote CI job or pending control run was cancelled or rerun.
At the final snapshot main had advanced again to https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/daf1d2ad5148ea8a6843a96f53298f2a133345b8. The tested PR head70b remains based on7485f604 and is BEHIND; this later main was not integrated or tested here. Rebase and validate before a further source push.
Current-base run links: https://buildtest.kotlin.build/run?id=0d42f9a2, https://buildtest.kotlin.build/run?id=1f7f5983, https://buildtest.kotlin.build/run?id=65b14e49, https://buildtest.kotlin.build/run?id=7374bbc5, https://buildtest.kotlin.build/run?id=7455a52d, https://buildtest.kotlin.build/run?id=7fc6fd12, https://buildtest.kotlin.build/run?id=f588f6f1. Single-call mutation: https://buildtest.kotlin.build/run?id=e8d197e0.
Next steps
- Re-verify PR/main/CI state first. Clone fresh under workspace/. Read the repository README/AGENTS plus the documentation links quoted above. Do not deploy or merge.
- Read the exact first-attempt failures in the forensic evidence. A pass in a later batch does not resolve the earlier failure. Establish an end-to-end mechanism and a deterministic public-API reproducer before editing production code or harnesses. Start with the deletion-wait scenario’s serial SSH phases and the clock-driven startup/cleanup ordering, using the preserved never-run phase-print patches only as a starting point to review. No larger timeouts, fewer iterations, changed assertions or hardware changes.
- Keep historical mainb28 comparisons distinct from current-main integration. Once a mechanism is named, force its exact condition instead of tuning the test to a host. Fix each failing guard at the owning layer. Do not guess that a single timeout frame or frontend503 is the cause.
- Verify every new test with only its target line/branch removed, then restore it and record first-attempt runner XML metadata. Preserve the no-create/provider deadline contract and the testing-only pause together.
- Before pushing, fetch/rebase latest main and verify the PR remains OPEN. Use explicit force-with-lease, keep the opening why paragraph when updating the PR body through REST, and end commits with the required coauthor trailer. Direct runs remain named targeted remote tests only.
- Observe final-head CI with build-watchman under a hard deadline; no blind CI reruns. If a check is red, reproduce it locally through the authorized remote test client before a fix. Leave the PR draft/open and report actual final CI state. Remove the fresh clone and stop/reap all tracked local observers before finishing.
Reusable operational knowledge
Use /home/helena/bin/cs, never /usr/local/bin/coursier. Targeted command: scripts/test.bash --remote --test <exactName> --log /absolute/path/result.xml; the client is wrapped in a hard timeout. Successful XML may omit run-id, so map the first filtered-out /api/test-results?id=<run>&page=1&limit=100 row’s selector string to the exact launch plan; failed XML includes run-id. WUI pagination is1-based and limit<=500; offset is unsupported. Whole-roster filtered rows are not selected passes. Canonical/highest attempt metadata must be1 before recording first-attempt success. The WUI’s first100 rows are mostly filtered-out tests and cannot establish selected completion. Its /log/raw lacks Last-Modified, so a watchman freshness warning is not executor failure. Seven progress endpoint503 snapshots are preserved without a cause claim. The report-challenge CLI would automatically enqueue/merge and therefore was not invoked under the user’s explicit prohibition; an unfiled record is preserved.
The final CI observer used an absolute2026-10-19 03:25 UTC deadline. All local tracked waits are stopped before the final handoff; remote GitHub jobs, if still running, remain authoritative and were not cancelled or rerun. The user’s optional question about continuing beyond roughly90minutes had no answer at write time unless later direction is recorded. This handoff preserves remaining work rather than declaring the six initial failures resolved.
r62 continuation after local disk cleanup — 2026-10-19
Owner direction: "run df -h / before every clone or build. Stop and report if free space is below 10 GB. Delete your clone and its ~/.aibuildcaches entries when you finish." The initial check has32GB available; source clones are complete and clean at recorded pins. Existing evidence/finished runs are retained, not repeated. Five main issues and88selected first-attempt XML rows remain authoritative. Exact delayed-reply1062-character error has now been recovered via read-only saved result access and added to https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1541 .
At05:23UTC BuildTest remains STARTING after35minutes. Read-only coordinator thread snapshots show active ProjectionChangeJournal.loadCommittedRecords and index writes, then line parsing; journal fd is at1331953664 of2022328465bytes at05:23:41, so recovery is advancing rather than proven deadlocked. No service change. Both original terminal attach probes were refused at health, no XML. New matched batches are gated on healthOK. Continuation/checkpoint evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/79d997f753 ; https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r62-guard-evidence/handoff-artifacts/r62 . The source head remains892e20d2; final-head CI proceeds independently.