Handoff: restart CLI test-comprehensiveness review complete, remote CI blocked by buildtest provisioning incident

Written 2026-07-10 at approximately 16:35 UTC.

RE-VERIFY BEFORE ACTING: every state below is a write-time snapshot. Re-check the code PR with gh pr view 17 --repo CodexCoder21Organization/ContainerNurseryProductionRemoteRestartCli --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefName,headRefOid; re-check the buildtest fleet via /root/buildtest-data/runs/*/run.json on root@198.199.106.165 (SSH port 23) or the owning service API. Fresh state always supersedes this document.

Mission summary

Review the test comprehensiveness of ContainerNurseryProductionRemoteRestartCli's adoption-aware restart PR: compare the branch to main, assess behavior/fidelity/edge cases/assertions/hermeticity, fix material gaps, run the complete suite, commit and push the work, and ensure CI is green without merging the code PR.

The review, fixes, local verification, commit, push, and PR-description update are complete. The sole unfinished requirement is obtaining a remote green check. Two remote attempts failed in buildtest provisioning before any project test executed during a documented system-wide incident.

What was found and done (the chain)

1. Read the repository README, global instructions, the complete testing architecture document, and engineering philosophy. Compared the PR branch with origin/main and audited every existing scenario.
2. Existing coverage already verified exact main-process discovery for JAR/class launches, managed-child exclusion, graceful handoff without SIGKILL, bounded escalation with the complete ADOPTION FORFEITED message, and explicit cold broad-kill behavior.
3. Material gaps were identified: no real JAR-upload restart path; no zero-process fresh start; no SSH-command-failure behavior; no missing-replacement failure; insufficient sequence-order assertions; actual launch-path handoff environment only indirectly covered; one validation test did not assert its full error text.
4. Added real, hermetic Apache MINA embedded-SSH scenarios (no mocks, reflection, disabled tests, external service dependency, or raised timeout):
   • tests/testColdRestartWithJarUpload.kts streams a valid 40 MiB-plus JAR over real SFTP, exercises cold=true, kills simulated main/child processes, and verifies the actual launch contains CN_RESTART_HANDOFF=1.
   • tests/testFreshStartWithoutExistingProcesses.kts invokes the real CLI entry point with no existing process and verifies no signal plus handoff-enabled launch.
   • tests/testSigtermCommandFailureAbortsRestart.kts, tests/testLaunchCommandFailureAbortsRestart.kts, and tests/testMissingReplacementProcessFailsRestart.kts cover the three process-control failure paths with complete diagnostic-message assertions.
   • Existing graceful, escalation, cold, no-upload, and validation scenarios were strengthened with strict command ordering, actual launch-command assertions, and full error text.
5. Two defects were reproduced red before implementation:
   • A nonzero SIGTERM command was only logged, after which the replacement launched over a potentially live main process.
   • A replacement launch command could return success without producing a new main process and the restart still reported success.
6. Fixed src/community/kotlin/ssh/restart/containernursery/RestartContainerNursery.kt so signal and launch command failures abort with exit code plus combined nonblank stdout/stderr, and so absence of a replacement main after launch is a hard failure with the production log path. Updated the README's public failure contract.
7. The complete post-rebase local suite passed: ALL TESTS PASSED (23/23 tests completed successfully), exit code 0. A second fresh-cache run and focused upload-path run also passed. git diff --check passed, generated artifacts were removed, and an adversarial review found no remaining material gap.
8. Committed and pushed commit 35ad1805d3fb054099dabf9c41124fafda5a091b on adoption-aware-restart. At write time the local branch exactly matched origin/adoption-aware-restart; no work was unsaved. The PR description now reports 23/23 and the follow-up coverage.
9. Remote CI attempt dc2d3c20 failed after its run remained provisioning with dropletId=0; all 23 tests were synthesized as zero-duration “did not complete” failures. Retry 7ad64a97 eventually obtained droplet 583674851, but provisioning exceeded the five-minute watchdog before SSH/cloud-init finished; the run reported 0/0 tests. Neither attempt executed this project's tests.
10. The incident was verified system-wide rather than inferred from this PR. Buildtest logs showed 30-second DropletServiceClientImpl.listDroplets() invocation timeouts in provisioning/recovery/reaping; the droplet service was handling a large concurrent creation/listing storm; the host had load between roughly 70 and 104; two kotlin-build-ci JVMs were alive; many unrelated runs failed from provisioning watchdogs or 15–17-minute stale leases. At the final 16:35 UTC sample, the host load was 87.61 and 22 runs were nonterminal (PROVISIONING=5, TESTING=17) with zero terminal runs in the previous five minutes. No production process was restarted because the root cause was not yet fixed and a restart would not remove the request storm.

Relevant PRs / refs

| Item | Write-time state | Reference | |---|---|---| | Restart CLI code PR | OPEN, merge state BLOCKED solely by remote check; head 35ad1805d3fb054099dabf9c41124fafda5a091b; do not merge without user approval | https://github.com/CodexCoder21Organization/ContainerNurseryProductionRemoteRestartCli/pull/17 | | Review/fix commit | Pushed to adoption-aware-restart | https://github.com/CodexCoder21Organization/ContainerNurseryProductionRemoteRestartCli/commit/35ad1805d3fb054099dabf9c41124fafda5a091b | | Remote run 7ad64a97 | FAILURE before tests (provisioning watchdog) | https://buildtest.kotlin.build/run?id=7ad64a97 | | Project-cache challenge | Merged record | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-10-1449-kompile-project-cache-reused-an-artifact-incompatible-with.md | | Global bldbinary-cache challenge | Merged record | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-10-1510-kompile-tests-resolve-an-obsolete-global-bldbinary-jar.md | | First provisioning challenge | Merged record | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-10-1557-buildtest-marked-every-test-failed-after-digitalocean.md | | Systemic provisioning/stale-lease incident | Merged record | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-10-1613-buildtest-is-in-a-systemic-provisioning-and-stale-lease.md |

The GitHub check-suite ID for the current code head was 78746719440 at write time, with conclusion failure from the infrastructure-only 7ad64a97 run.

Next steps

1. Re-verify current state first using the commands in the banner. Confirm the PR remains open at the same head and has not already acquired a newer check run.

2. Confirm buildtest has genuinely recovered before retrying: active provisioning/installing counts should be low, unrelated recent runs should be reaching COMPLETED, leases should renew, and host load should no longer be in the observed 70–104 range. Do not use elapsed time alone as the signal.

3. If the fleet remains stalled, continue investigation/fix at the owning buildtest/droplet-service/kotlin-build-ci layer. Do not change this restart CLI or relax its tests to mitigate shared infrastructure failure. Use the investigate-broken-service skill and HardwareControlFabric first; the previous environment lacked local HCF certificates, so read-only SSH was used as the documented last resort.

4. Once recovered, re-request the current suite (after re-fetching its ID):

   suite_id=$(gh api repos/CodexCoder21Organization/ContainerNurseryProductionRemoteRestartCli/commits/35ad1805d3fb054099dabf9c41124fafda5a091b/check-suites --jq '.check_suites[] | select(.app.name == "kotlin-build-ci-test") | .id' | head -1)
   gh api -X POST "repos/CodexCoder21Organization/ContainerNurseryProductionRemoteRestartCli/check-suites/$suite_id/rerequest"
   

5. Use the watch-build skill/build-watchman to follow the new check to completion. If it reaches tests and fails, investigate the actual test output; do not conflate it with the two documented provisioning failures.

6. Confirm the check is green, the PR remains OPEN, the branch remains clean/pushed, and then report completion to the user. Do not merge the code PR.

Reusable / operational knowledge

• Kompile may resolve a stale same-coordinate artifact from both the project cache and the global bldbinary Coursier cache. During this review, an obsolete ssh-restart-containernursery:0.0.6 JAR masked the newly built API. A fresh verification required removing the project .kompile cache and /home/helena/.cache/coursier/v1/bldbinary/repository/community/kotlin/ssh/restart/containernursery/ssh-restart-containernursery/0.0.6. The two linked challenge records contain the evidence; this is a tooling workaround, not a project change.
• The upload-path test's large incompressible ZIP entry uses AES-GCM on the embedded SSH server to keep the real SFTP transfer inside the existing default timeout. No production timeout or test timeout was increased.
• Buildtest terminal FAILED plus testsTotal=0, or synthesized zero-duration failures for every declared test, requires inspecting run.json/build.log before treating it as a code failure.
• The code PR's comprehensive local result is 23/23 after mandatory fetch/rebase. The PR description was updated through the REST endpoint because gh pr edit failed on a deprecated Projects-classic GraphQL field.