← Priority list

Handoff: BuildTestEmbedded reservation-idempotency proof blocked by an already-owned map invariant

Resolve the reservation-idempotency versus create-stall-recovery conflict on [BuildTestEmbedded PR](https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/314) using branch `fix/deterministic-admission-crash-recovery`. The approved plan required both an ownership/already-released guard and a corrected end-to-end fixture in which run B's create completes normally. It also required mutation evidence that the corrected test fails without the new guard because run A's late exit releases B's slot and admits run C, then passes with the guard. If another genuine conflict appeared, the instruction was to stop and report rather than guess.

hf-2026-07-13-buildtest-reservation-idempotency-stall-recovery-conflict

Edit handoff

Title Summary Markdown body

# Handoff: BuildTestEmbedded reservation-idempotency proof blocked by an already-owned map invariant

Written 2026-07-13 UTC.

**RE-VERIFY:** This is a write-time snapshot. Before acting, re-check the product PR with `gh pr view 314 --repo CodexCoder21Organization/BuildTestEmbedded --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefName,headRefOid` and re-run the named test from the existing `/code/BuildTestEmbedded-task19` checkout. A fresh query and fresh-source test run supersede every state below.

## Mission summary

## What was found and done (the chain)

1. The existing test was reproduced against the original branch. Its old fixture blocked run B's `createDropletAsync`, advanced `ManualClock` beyond the new 60-second create-stall bound, and then observed run C being admitted. That is run B's legitimate stall recovery, not proof that run A double-released capacity.
2. The product already has the requested ownership and already-released semantics in both `origin/main` and the PR head: `admissionReservations` is `MutableMap<String, Int>`, keyed by run ID, and every normal release is `admissionReservations.remove(runId)` while holding `admissionLock`. Removing absent run A cannot remove run B's different key, and repeating it is idempotent. Relevant locations at write time are `src/buildtest/embedded/BuildTestEmbeddedService.kt` around lines 255 and 525.
3. A prototype helper made that existing invariant explicit and suppressed the stale release's unnecessary wake/recompute side effects. A prototype fixture moved run A's uncancellable block to its real post-create provisioning worker and let run B's provider `createDroplet` return a real droplet.
4. The prototype initially appeared to pass even under deliberately broken mutants. This was invalid evidence: kompile reused a stale compiled root `build.kts` containing artifact `0.0.276` after the checkout had changed to `0.0.277`. `javap -v` showed the stale constant and no `0.0.277` artifact existed. This infrastructure issue is permanently recorded in the [challenge report](https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-13-1541-buildtest-kompile-cache-reused-a-stale-root-build-kts.md).
5. After deleting only the generated cache for this checkout and rebuilding from source, the prototype test failed even with the explicit helper. The decisive output was:

```text
   java.lang.AssertionError: Run A's late worker exit must not release run B's still-held reservation or spuriously grant run C; ... createdNames=[buildtest-<A>, buildtest-<B>, buildtest-<C>].
   TESTS FAILED (0/1 tests completed successfully, 1 failed)
   ```

6. Inspection found why: the attempted fixture blocked run B inside the returned droplet's first `id` getter. `createBuildDroplet` consumes `droplet.id` before `recordReconciledDropletCreated`, so run B had **not** completed the service-level create/handoff normally. The prototype therefore still violated the approved fixture condition and could expose legitimate provisioning/stall behavior.
7. If run B is allowed to complete `createBuildDroplet` and `recordReconciledDropletCreated` normally, reverting the proposed helper to the current `admissionReservations.remove(runId)` cannot produce the demanded mutation outcome: run A's absent key cannot delete run B's key, and the locally observed B droplet continues to occupy capacity even if reservation accounting changes. A test may detect the current implementation's unnecessary wake/re-check after stale release, but it cannot truthfully show B losing its slot or C being granted due to `remove(A)`.
8. This is the new genuine conflict: the requested fail-without-new-guard / B-loses-slot evidence assumes a scalar or unowned reservation implementation, while the actual implementation already has per-run ownership. Manufacturing a mutant that removes an arbitrary map entry would demonstrate a newly invented defect, not removal of the requested guard from the current code.
9. Per the stop-on-conflict instruction, no product commit was made and nothing was pushed. The three prototype files were restored to the branch head; `git status --short` was empty at write time. Generated test/build cache remains outside the repository and no build artifact is tracked.

## Relevant PRs / refs

| Resource | Write-time state |
|---|---|
| [BuildTestEmbedded PR](https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/314) | OPEN, `mergeStateStatus=BLOCKED`, head `9f926f7e14b02f101cc158dc11b29d08f6975bae`; remote check failed at [buildtest run](https://buildtest.kotlin.build/run?id=1a44e503) |
| Product branch | `fix/deterministic-admission-crash-recovery`, clean at head `9f926f7e14b02f101cc158dc11b29d08f6975bae`; no new product commit or push |
| Base branch observed locally | `origin/main` at `47733108f69572695a715a9d6674658b7ac9ad74`; the product branch rebased cleanly before investigation |
| [Kompile stale-cache challenge](https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-13-1541-buildtest-kompile-cache-reused-a-stale-root-build-kts.md) | Merged permanent infrastructure record |

## Next steps

1. Re-verify current PR state and source first using the command in the banner.
2. Obtain the user's decision on the conflicting proof requirement. The evidence supports two honest options:
   - Treat the existing per-run map removal as the ownership/idempotency guard, correct the fixture so B fully completes creation, and preserve the no-C-grant assertion. This test will also pass after reverting an explicit no-op helper because the underlying ownership guard predates the helper.
   - Add and test a narrower new contract: an absent stale release must not wake admission waiters or recompute capacity. Removing that new boolean/no-op side-effect guard can make a corrected test fail on an extra admission re-check, but B will not lose its slot and C will not be granted.
3. Do not use an arbitrary-owner mutant such as `admissionReservations.remove(admissionReservations.keys.first())`; it does not represent removal of the current guard and would create misleading “teeth” evidence.
4. Once the user chooses the intended observable contract, redesign the fixture with B completely through `createBuildDroplet` and `recordReconciledDropletCreated`; do not block any getter used by that path and do not advance B's create-stall clock past 60 seconds.
5. Then perform the requested 15-or-more CPU-spinner repetitions of the six named crash-recovery tests, full post-rebase tests, adversarial review, product commit, force-with-lease push, and CI monitoring. Do not merge the product PR.
6. When the work is genuinely resolved, delete this handoff through the handoff skill and merge that deletion.

## Reusable / operational knowledge

- `scripts/test.bash --test <name>` can report a successful selected test while resolving an old locally cached product artifact. Before accepting mutant evidence, inspect the cached buildscript coordinate with `javap -v` and verify the expected artifact exists beneath `$HOME/.aibuildcaches/_code_BuildTestEmbedded-task19`.
- The fresh-source workaround used here was to delete only `$HOME/.aibuildcaches/_code_BuildTestEmbedded-task19`, then rerun the selected test. This generated cache is checkout-scoped; no repository source was deleted.
- A provider method returning a `Droplet` object does not mean BuildTestEmbedded's create path has completed. `createBuildDroplet` immediately reads metadata such as `droplet.id`, and `createBuildDropletWithRetry` reads the ID again before calling `recordReconciledDropletCreated`.
- The existing map key is the reservation owner. A stale `remove(runAId)` is already unable to remove `runBId`; any proposed test must be evaluated against that actual invariant rather than a scalar-capacity mental model.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: BuildTestEmbedded reservation-idempotency proof blocked by an already-owned map invariant

Written 2026-07-13 UTC.

RE-VERIFY: This is a write-time snapshot. Before acting, re-check the product PR with gh pr view 314 --repo CodexCoder21Organization/BuildTestEmbedded --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefName,headRefOid and re-run the named test from the existing /code/BuildTestEmbedded-task19 checkout. A fresh query and fresh-source test run supersede every state below.

Mission summary

Resolve the reservation-idempotency versus create-stall-recovery conflict on BuildTestEmbedded PR using branch fix/deterministic-admission-crash-recovery. The approved plan required both an ownership/already-released guard and a corrected end-to-end fixture in which run B's create completes normally. It also required mutation evidence that the corrected test fails without the new guard because run A's late exit releases B's slot and admits run C, then passes with the guard. If another genuine conflict appeared, the instruction was to stop and report rather than guess.

What was found and done (the chain)

The existing test was reproduced against the original branch. Its old fixture blocked run B's createDropletAsync, advanced ManualClock beyond the new 60-second create-stall bound, and then observed run C being admitted. That is run B's legitimate stall recovery, not proof that run A double-released capacity.
The product already has the requested ownership and already-released semantics in both origin/main and the PR head: admissionReservations is MutableMap<String, Int>, keyed by run ID, and every normal release is admissionReservations.remove(runId) while holding admissionLock. Removing absent run A cannot remove run B's different key, and repeating it is idempotent. Relevant locations at write time are src/buildtest/embedded/BuildTestEmbeddedService.kt around lines 255 and 525.
A prototype helper made that existing invariant explicit and suppressed the stale release's unnecessary wake/recompute side effects. A prototype fixture moved run A's uncancellable block to its real post-create provisioning worker and let run B's provider createDroplet return a real droplet.
The prototype initially appeared to pass even under deliberately broken mutants. This was invalid evidence: kompile reused a stale compiled root build.kts containing artifact 0.0.276 after the checkout had changed to 0.0.277. javap -v showed the stale constant and no 0.0.277 artifact existed. This infrastructure issue is permanently recorded in the challenge report.

After deleting only the generated cache for this checkout and rebuilding from source, the prototype test failed even with the explicit helper. The decisive output was:

java.lang.AssertionError: Run A's late worker exit must not release run B's still-held reservation or spuriously grant run C; ... createdNames=[buildtest-<A>, buildtest-<B>, buildtest-<C>].
TESTS FAILED (0/1 tests completed successfully, 1 failed)

Inspection found why: the attempted fixture blocked run B inside the returned droplet's first id getter. createBuildDroplet consumes droplet.id before recordReconciledDropletCreated, so run B had not completed the service-level create/handoff normally. The prototype therefore still violated the approved fixture condition and could expose legitimate provisioning/stall behavior.
If run B is allowed to complete createBuildDroplet and recordReconciledDropletCreated normally, reverting the proposed helper to the current admissionReservations.remove(runId) cannot produce the demanded mutation outcome: run A's absent key cannot delete run B's key, and the locally observed B droplet continues to occupy capacity even if reservation accounting changes. A test may detect the current implementation's unnecessary wake/re-check after stale release, but it cannot truthfully show B losing its slot or C being granted due to remove(A).
This is the new genuine conflict: the requested fail-without-new-guard / B-loses-slot evidence assumes a scalar or unowned reservation implementation, while the actual implementation already has per-run ownership. Manufacturing a mutant that removes an arbitrary map entry would demonstrate a newly invented defect, not removal of the requested guard from the current code.
Per the stop-on-conflict instruction, no product commit was made and nothing was pushed. The three prototype files were restored to the branch head; git status --short was empty at write time. Generated test/build cache remains outside the repository and no build artifact is tracked.

Relevant PRs / refs

| Resource | Write-time state | |---|---| | BuildTestEmbedded PR | OPEN, mergeStateStatus=BLOCKED, head 9f926f7e14b02f101cc158dc11b29d08f6975bae; remote check failed at buildtest run | | Product branch | fix/deterministic-admission-crash-recovery, clean at head 9f926f7e14b02f101cc158dc11b29d08f6975bae; no new product commit or push | | Base branch observed locally | origin/main at 47733108f69572695a715a9d6674658b7ac9ad74; the product branch rebased cleanly before investigation | | Kompile stale-cache challenge | Merged permanent infrastructure record |

Next steps

Re-verify current PR state and source first using the command in the banner.
Obtain the user's decision on the conflicting proof requirement. The evidence supports two honest options:
- Treat the existing per-run map removal as the ownership/idempotency guard, correct the fixture so B fully completes creation, and preserve the no-C-grant assertion. This test will also pass after reverting an explicit no-op helper because the underlying ownership guard predates the helper.
- Add and test a narrower new contract: an absent stale release must not wake admission waiters or recompute capacity. Removing that new boolean/no-op side-effect guard can make a corrected test fail on an extra admission re-check, but B will not lose its slot and C will not be granted.
Do not use an arbitrary-owner mutant such as admissionReservations.remove(admissionReservations.keys.first()); it does not represent removal of the current guard and would create misleading “teeth” evidence.
Once the user chooses the intended observable contract, redesign the fixture with B completely through createBuildDroplet and recordReconciledDropletCreated; do not block any getter used by that path and do not advance B's create-stall clock past 60 seconds.
Then perform the requested 15-or-more CPU-spinner repetitions of the six named crash-recovery tests, full post-rebase tests, adversarial review, product commit, force-with-lease push, and CI monitoring. Do not merge the product PR.
When the work is genuinely resolved, delete this handoff through the handoff skill and merge that deletion.

Reusable / operational knowledge

scripts/test.bash --test <name> can report a successful selected test while resolving an old locally cached product artifact. Before accepting mutant evidence, inspect the cached buildscript coordinate with javap -v and verify the expected artifact exists beneath $HOME/.aibuildcaches/_code_BuildTestEmbedded-task19.
The fresh-source workaround used here was to delete only $HOME/.aibuildcaches/_code_BuildTestEmbedded-task19, then rerun the selected test. This generated cache is checkout-scoped; no repository source was deleted.
A provider method returning a Droplet object does not mean BuildTestEmbedded's create path has completed. createBuildDroplet immediately reads metadata such as droplet.id, and createBuildDropletWithRetry reads the ID again before calling recordReconciledDropletCreated.
The existing map key is the reservation owner. A stale remove(runAId) is already unable to remove runBId; any proposed test must be evaluated against that actual invariant rather than a scalar-capacity mental model.