Carrier correction complete; timer registration and first-attempt readiness unresolved
Updated 2026-10-09 05:34:26 UTC. This effort is NOT READY FOR ORCHESTRATOR REVIEW. Do not merge, enqueue, mark ready for review or deploy.
The request was to repair the obsolete carrier test on the existing draft, compare the real-droplet scenario five times on original draft and main, establish the timer failure's mechanism at its owner, and obtain a full required CI result with first-attempt accounting. The carrier change is complete. The exact upstream cause and two unnamed full-suite retry-only failures remain unresolved. This replaces the previous stale red-CI banner; previous lane evidence remains linked below.
Live source and PR
- Existing PR: https://github.com/CodexCoder21Organization/BuildTestServerService/pull/403 — freshly verified OPEN, DRAFT, unmerged; head dc44e0a5c9507a348ef446edfbdace8bcb48d518, mergeStateStatus CLEAN. Its original why paragraph is preserved byte-for-byte.
- Draft branch: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/sol-coordfixD2 ; source recovery branch: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/sol-fix403c . Both contain exact tested https://github.com/CodexCoder21Organization/BuildTestServerService/commit/dc44e0a5c9507a348ef446edfbdace8bcb48d518 . No source changes are local-only.
- Evidence: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/sol-fix403c-evidence/handoff-artifacts/fix403c . It contains raw first-attempt XML, compressed complete logs, incremental findings, both reviews, upstream plan/source fact-check, exact sample ledger and current PR body. This evidence branch is not for merging.
- Previous lane evidence: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/wip/sol-rr-d2-403-evidence/handoff-artifacts/rr-d2-403 . Earlier original source is https://github.com/CodexCoder21Organization/BuildTestServerService/commit/2473b7c6a17da4b44a6ed505244fd535130ee540 . Main comparison source is https://github.com/CodexCoder21Organization/BuildTestServerService/commit/9d936fc263b85881104c2c0d85c2212c3608088d .
Completed proofs and limits
Carrier uses ten distinct public async calls, actual provider entries, a virtual releaser, unchanged two-carrier cap and deadlines, exact result/error checks and bounded resource cleanup. https://github.com/CodexCoder21Organization/BuildTestServerService/actions/runs/37886369142 proves10/10 first passes plus a first-attempt RED product-only separate-monitor mutant, whose JVM trace reports MONITOR and zero provider entries. Retry attempts on the mutant all fail and receive no extra control credit. Final code/test reviews accepted.
Real-droplet comparison: https://github.com/CodexCoder21Organization/BuildTestServerService/actions/runs/37885074299 (sample1 each), https://github.com/CodexCoder21Organization/BuildTestServerService/actions/runs/37886824046 (samples2–5 each). Actual launcher0.0.114 and Temurin21, same partition environment. OriginalPR2/5 first passes, sample5 retry-only; main3/5 first passes, no retry-only. All five first failures report provisional timer upsert20000ms before DigitalOcean creation. Raw XML attempt-history is a sibling of test rows; ledger counts only attempt1, never retries. The temporary proof workflow's success means collection completed, not that all tests passed.
Full required Actions: https://github.com/CodexCoder21Organization/BuildTestServerService/actions/runs/37887003885 on dc44e0a5 — Actions success,251/251 eventual passes, 249/251 first attempts, two retry-only tests with UNKNOWN names. Raw console explicitly records both attempt2 warnings but omits names, and artifact API total_count=0. The console cannot supply a per-test first-attempt ledger. Do not call this READY. No final-head CI rerun was issued; an earlier superseded diagnostic-review head's run was canceled and receives no final-gate credit.
Root-cause boundary and exact next work
Read https://github.com/CodexCoder21Organization/BuildTestServerService/blob/wip/sol-fix403c-evidence/handoff-artifacts/fix403c/fix403c-upstream-plan.md end to end, its pinned sources/tests, central TESTING and PHILOSOPHY. The gateway persists auto-terminate-creation-<accountScope>-<requestId> and awaits WebCronService.createOrUpdateTimerByName before provider creation. AutoTerminationManager.kt:1642–1653 times out its outer future after20000ms. Current direct RPC request uses19000ms, but connection setup is outside that inner bound. WebCron has initialization, append/commit/application and response phases; evidence has not identified which wait is occupied. Timer callback firing is not needed: registration fails first.
ContainerNurseryCLI0.0.20/0.0.21 return zero records, including gateway total_lines38290 with empty lines; no deployed bytecode identity, gateway worker stack or far-side request/response correlation was obtainable. A production image filename is not version proof. The existing inner/outer-bound fix upstream is not established as the current cause. Exact owner within gateway/WebCron/transport remains UNPROVEN; no consumer mitigation or guessed upstream patch was made.
Next obtain permitted read-only deployed identity and correlate one failing requestId through gateway send, WebCron receive/append/commit/apply/return and gateway receive. Reproduce the observed ordering with real WebCron storage/server and real loopback URL peers through public gateway createTimer, using a local provider stand-in. The plan names fixture APIs, existing tests, all resource invariants and deterministic trigger requirements. Establish reliable baselineRED/fixPASS/mutantRED at the actual owner, then verify the original consumer. Separately recover/name and reproduce the two full-suite retry-only failures before a new final CI gate. Do not blindly rerun CI, add retries/sleeps, raise timeouts, loosen assertions or guess a fix.
Remote proof branches
All throwaway workflows are unmerged; no local-only workflow is needed to resume:
- https://github.com/CodexCoder21Organization/BuildTestServerService/tree/gha-run/fix403c-final-carrier (05a015ad1bf82f5cb045e9e1929aaa6f1f925ac9)
- https://github.com/CodexCoder21Organization/BuildTestServerService/tree/gha-run/fix403c-remaining (203a274ab3db157c59e933457eefcf34ee3801b8)
- https://github.com/CodexCoder21Organization/BuildTestServerService/tree/gha-run/fix403c-compare (3d598f72e62c70e1910553e9a8f0dbb30804b3c5)
- https://github.com/CodexCoder21Organization/BuildTestServerService/tree/gha-run/fix403c-carrier (8e185763d96a032804a9eda546e1925300912c2c; initial corrected carrier gate)
- https://github.com/CodexCoder21Organization/BuildTestServerService/tree/gha-run/fix403c-mutant (508d5264b47334e463de74d900f482c16debaee3; initial corrected control)
Execution state
Owned watchers have exited and all requested proof/CI runs are terminal. GPT-6-luna max finished drafting, test/code review and narrow source-plan fact-check; no delegated diagnostic or ongoing task remains. Owned checkout/cache cleanup is complete, with its outcome appended to findings and cleanup-audit.json. Fresh resumption must clone under its own workspace directory. Read-only upstream clones had no edits and are recoverable at pinned commits in the plan.
Deployed or published but unmerged: none produced by this effort. No production deployment/publication occurred. All code is on remote branches, existing draft retained. Challenges are recorded in evidence because the report-challenge CLI automatically merges, which this brief prohibits.
Cleanup confirmed 2026-10-09 05:35:58 UTC: owned checkout deleted; matching cache entries0; private diagnostics deleted; no owned build/watch process after final ps verification. Other lanes were not changed.