Repository · handoffs
← Handoffs
View on GitHub ↗
id: hf-2026-10-02-finish-the-production-shaped-coordinator-invariant-gate-and-establish-measured-red-and-green-rows url: url://handoff/handoffs/hf-2026-10-02-finish-the-production-shaped-coordinator-invariant-gate-and-establish-measured-red-and-green-rows title: Prepare production fixture cuts and complete the coordinator invariant gate summary: Prepare cold fixture cuts before measured phases and finish I1-I6 proofs; source and evidence checkpointed, current-source requests ran no tests, open required checks remain red. created: 2026-10-02T21:54:25.623Z completed: null blocked-reason: I2 owner green, fixture and instrumentation corrections, missing invariant rows, and exact-current-source runtime/CI proof remain dependencies:
Handoff: Prepare production fixture cuts and complete the coordinator invariant gate
RE-VERIFY BEFORE ACTING (2026-10-04 14:27 UTC): Claims and refs below were checked in this lane. The earlier 6/14 result is historical; it is not the present draft or current-main verdict. Re-fetch claims, main and PR state before continuing. Do not merge, enqueue, publish, deploy or complete this handoff; the supervisor owns those decisions.
Mission
Finish the production-shaped coordinator invariant gate requested by the operator on 2026-10-02, preserving measured fail-first rows and completing the promised storage-loss, work, availability, unavailable-delivery and deadline coverage. Original request: “There seem to be so many different root causes of these same classes of problems, and none of them have been caught by our unit tests. I don't understand why it is so hard to have coverage of the few important flows, such that all these problems would have been caught early by tests instead of showing up in production.” The draft remains incomplete, with known fixture corrections preserved and intended product rows still red.
Chain of findings
2026-10-04 12:18 UTC — original eight-row classification
OBSERVED: The saved handoff stops at 9dc04fc4 (6/14). The draft at https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1206 is OPEN, draft, head 4e24eca29990b1b30014dc8b0ce1d91a7ea1594b, with bld_test_shard_1/2/3 and bld-build FAILURE and shard 4 SUCCESS. Newer durable evidence at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate3-evidence-2026-10-02b (14dc12ab80d178a2fa67663b4a0464cd1b2caab1) has final first-attempt A5/B5/C5 verdicts 9/14, 9/14, 11/14. Retry-green is not counted as green. Those combined comparison source commits were local-only; they are not current-main evidence.
| Original failing selector | Classification supported by evidence | Present state |
|---|---|---|
| DeletedChild | Original payload precondition failed because generated retry journals omitted format/row-authority events. Corrected producer now reaches an actual missing test-events.jsonl repair failure. | Fixture correction already in draft. Product repair owner https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1198 remains OPEN with red checks. |
| PublicationWork | Product snapshot writer reopened 8,241,033 output-bearing bytes while publishing compact rows; exact equality/positive control succeeded first. | Readback implementation https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1202 MERGED, merge 59db2da309fdb76715445fd574308cc929047a06. C5 PASS. The current-source14-row request timed out PENDING and was canceled without executing tests; current-main row verdicts remain absent. |
| AttemptWork | Original JFR constant-pool reader exhausted heap before work assertion: instrumentation failure. Corrected FileHandle instrumentation measures 10,000 off-prefix large-value requests on old code. | Instrumentation correction already in draft. Same merged readback change filters prefix before value loading; C5 PASS. |
| PageWork | Generated journal omitted retry authority and selected result output was empty. | Corrected producer preserves 25MB payload/depth; final A5/B5/C5 PASS. |
| HeldStartupReads | Same generated-journal problem stopped selected-output assertion before availability proof. | Corrected producer final A5/B5/C5 PASS; retained-tail proof only, not held reset publication. |
| MetadataWork | Test exhausted its unchanged 30-second budget while awaiting startup, before measured metadata reads. This alone establishes neither a readback bug nor host-load cause. | Still fails in all final comparisons. Fixture uses internal BuildRunImpl/flushTestResultsProjection/processRunnerEvent; source-based review found public-API setup gap. Needs causal profiling and contract-correct fixture, without changing bound. |
| UnavailableTerminal | appendAll rejects unavailable journal writes; markUnavailableLocked clears pendingAppends; current-state repair yields CANCELED baseline but no intermediate PENDING rows. | Intended stronger I5 gate remains red. README describes current-state reset recovery; exact unavailable-event delivery requires an agreed stronger contract and product work, not assertion weakening. |
| Existing soak | Original retained service-thread mismatch; newer historical runs also fail thread counts or journal row bound with rotating outcomes. | Later single PASS is not a repair. No demonstrated mechanism/owner fix yet. Preserve unchanged selector. |
OBSERVED: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 is MERGED (068b5607bc15a53346d3ee6ad0bebd935aeba061). Its journal heap fix does not change these 14 row verdicts and cannot establish the distinct 200-run I2 production population. Do not duplicate the sibling heap comparison. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1207 and the paginated result-count sibling remain untouched.
OBSERVED: Fresh clone read root README; no root AGENTS.md exists. Required documents read: PlanRepository handoffs README / Triaging handoffs and body format; PHILOSOPHY / root cause over mitigation, refuted hypotheses; TESTING / public API, hermetic tests, fail-first, machine-independent work bounds and real concurrency; CODE_REVIEW / Claims That a Host Is Overloaded. The current service-backed handoff skill and lane brief override the older README's PR-based handoff completion procedure. report-challenge skill auto-enqueues and merges; this lane forbids that, so no such CLI was invoked.
OBSERVED: Source rebased onto current main 1de45a68db4c34d27e12494b2df00877d012e527 at local head 160227ec025a6d38988f65cd514b8ad6281b797f; build.kts conflict resolved by retaining both existing main fixture rules and new production fixture rule. No production implementation change. An initial 256MiB remote client attempt failed before submission in ByteArrayOutputStream while creating the workspace tar. The pre-submission OOM was observed in archive construction; its initial console log was overwritten by the later request, so a complete durable original stack is unavailable. Do not infer an executor failure from this client error. Bootstrap jars were moved outside checkout before a direct pinned kompile launcher submission, using a 768MiB client cap; this does not alter any test heap or timeout. Remote run aae65935 was accepted at160227ec025a6d38988f65cd514b8ad6281b797f, remained PENDING for the pinned client's30-minute limit, then was canceled by workspace close with0passed/0failed. build-watchman's RESULT:RED described the canceled run, not failing tests. The separate single-test request2495be4d also executed no tests and canceled after its client30-minute limit. Its public getBuildRun/getBuildLog records a specific createDropletAsync result stall during provisioning; no conclusive creation result was delivered, and cleanup-before-requeue preceded client workspace closure. Both runs are terminal; do not requeue to seek green.
INFER: The handoff cannot currently be declared ready or obsolete. Existing fixture corrections are worth preserving, but missing matrix/reset/deadline proof and unresolved intended-red product rows remain. Next: preserve this classification/source patch remotely, measure current main plus gate, then narrow MetadataWork's preparation mechanism and any first-attempt soak failure rather than rerunning CI.
Checkpoint measurements and review at 2026-10-04 13:14 UTC
- Source was rebased onto main1de45a68db4c34d27e12494b2df00877d012e527, retaining main fixture rules and the gate fixture rule. Remote WIP source head51eb77575d40216c41b2ff7c9435ed0ee5ff2f06 is recoverable without applying a patch. No production implementation change was made; do not update the existing draft until full suite and owner failures are resolved.
- Added public productionInvariantHeldResetReads: one2.5MB run,25MB results,depth4, held journal-owned reset serialization, eight lease writers and four feed/result/attempt readers. All callers must finish before barrier release; reset plus tail must reconstruct all nine runs and deliver each accepted renewal once. Fixed the new consumer's initial error of stopping at the earlier captured snapshot head instead of the separately captured accepted-writer head. The selected gate now has15 rows; historical A5/B5/C5 still refer to14.
- One small published-binary compatibility check at8.2MB passed1/1 with zero failed assertions after that consumer correction. The stronger25MB compatibility run exhausted128MiB heap during startup, before the held phase. It used published buildtest-embedded0.0.69274748, not current source. The full emitted diagnostic is Terminating due to java.lang.OutOfMemoryError: Java heap space; ExitOnOutOfMemoryError ended the VM without a Java stack. No current-source heap mechanism or fix is claimed, no size/heap/timeout/assertion was weakened, and the separately owned startup heap work was not duplicated.
- Current-source fourteen-row runaae65935 executed no tests. The frontend run list was stale (outOfDate=true,refreshFailing=true), initially omitted the accepted run, while direct coordinator RPC showed PENDING; later direct RPC timed out. A read-only HTTP adapter let build-watchman observe the final primary CANCELED status. This is execution/observability evidence, not a replacement test verdict. The distinct new held-reset selector has been submitted at source51eb77575d40216c41b2ff7c9435ed0ee5ff2f06; it ended CANCELED0passed/0failed before test execution. A corrected bounded build-watchman adapter confirmed that actual terminal state; an earlier adapter had expired before its watcher and was a local probe error, not a service failure.
- Independent read-only coverage review plus separate parent test-comprehensiveness and adversarial code reviews are preserved on the evidence branch. They confirm mandatory gaps: I1 full96-cell storage matrix/episode/deadline proof; I2 complete200-run mixed-population/growth proof (separately owned); I3 public metadata setup/list/full-parse/encoded-response/depth bounds; I4 all held phases and simultaneous overlap; I5 global durable order/process interruption; I6 restart original deadline/late callback/attempt/cleanup/feed proof. A passing individual phase does not complete the gate. No required job is enabled.
Lane checkpoint at 2026-10-04 14:27 UTC
- The original six-pass premise is superseded by the corrected historical9/14,9/14,11/14 ledgers. The production retry fixture and FileHandle work instrumentation are salvageable and already preserved on the source branch. No product implementation was changed in this lane.
- Current-source thin compilation passed for84 production files at51eb77575d40216c41b2ff7c9435ed0ee5ff2f06 on main1de45a68db4c34d27e12494b2df00877d012e527. Fixture/test compilation also passed. All134 declared dependencies match the available prior thin-artifact POM; the old implementation JAR was excluded from compiler/runtime. This is targeted source execution, not a full local suite or fat-JAR build.
- Two fixed current-source diagnostics both ended before the held phase under the same30-second outer guard.128MiB logs a full OutOfMemoryError stack in loadTestResultsReadSnapshot32898 during recoverTerminalDurationPublications30528.512MiB (the pre-existing production plan's heap tier) completes background recovery, then the first public attempt page waits on cold index preparation. The actual worker is RUNNABLE in AttemptHistoryIndex.readAttemptIndexNode / AttemptHistoryProjection.refreshAttemptHistoryTest373; partial index nodes21036112bytes, events35056944bytes. This outer diagnostic guard is not theI4 five-second held-call contract, so neither expiry is anI4 row verdict. No bound was increased after these results.
- Full output and all-thread dumps are on the evidence branch. The128MiB duration-readback path may overlap separately active recovery/heap owner work; no common mechanism with the paginated-result-count sibling is established. Do not duplicate owner fixes without that comparison. Cold migration must be counted separately perI3 and fixture preparation must precede bounded phase measurement.
- Corrected storage-matrix-ledger.md enumerates96 unique source cells:1 limited source-covered,2 RED/PARTIAL from paired child deletion reaching the real missing-events repair failure,2 conflicts with older whole-feed-refusal tests,3 ownership-not-applicable,88 missing. These are inventory classifications, not96 tests. Restoration/new episodes/original deadlines remain unproved. metadata-public-path.md supplies public ingestion routes and relevant existing e2e tests.
- The unchanged existing soak plus thread diagnostics passed once against prior published0.0.69274748 (26threads before/after). Replacement current-service workers were waiting normally; two close-time InterruptedException stacks are preserved. This single pass does not refute historical intermittent failures.
- Dedicated test-comprehensiveness and adversarial review sections are in findings-checkpoint.md; the cold-preparation finding remains explicit. Required aggregate checks on all open related PRs are red. No gate activation, new PR/update, merge, enqueue, deploy, publication, test weakening, sibling modification or handoff completion. This lane ends CHECKPOINT; supervisor may resume the still-valid mission.
Relevant PRs / refs
| Repository / branch | Head | Link | State / recommendation |
|---|---|---|---|
| BuildTestEmbedded / test/production-shaped-invariant-gate | 4e24eca29990b1b30014dc8b0ce1d91a7ea1594b | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1206 | Draft OPEN, required aggregate red. Keep fixture corrections and useful invariant tests. |
| BuildTestEmbedded / wip/L11-production-invariant-gate | 51eb77575d40216c41b2ff7c9435ed0ee5ff2f06 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L11-production-invariant-gate | Rebased gate and selected25MB held-reset candidate. WIP, current-source preparation diagnostics only, no held-phase/full-suite verdict; no PR. |
| BuildTestEmbedded / wip/L11-invariant-gate-evidence | d872364dd66e64f2bcabdfb2dea8da85c154875a | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L11-invariant-gate-evidence | Durable classification, PR snapshots, newer A5/B5/C5 ledgers, complete current-main rebased source patch. Includes corrected96-cell ledger/public-ingestion map, current-source compile/diagnostic stacks, compatibility/soak logs, primary canceled-run/provider evidence and complete source patch. Keep; no PR update. |
| BuildTestEmbedded / wip/invariantgate3-handoff-2026-10-02b | 4e24eca29990b1b30014dc8b0ce1d91a7ea1594b | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate3-handoff-2026-10-02b | Latest remote gate checkpoint. Keep. |
| BuildTestEmbedded / wip/invariantgate3-evidence-2026-10-02b | 14dc12ab80d178a2fa67663b4a0464cd1b2caab1 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate3-evidence-2026-10-02b | Full corrected-fixture comparison and reviews. Keep as historical evidence. |
| BuildTestEmbedded / wip/invariantgate2-handoff-2026-10-02b | cc1b0c6a09c03110929a3fff78ea6ec74e50c1ef | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate2-handoff-2026-10-02b | Superseded source; preserve until supervisor confirms salvage. |
| BuildTestEmbedded / wip/invariantgate2-tested-2026-10-02b | 9dc04fc4ab47213d6c5cbc90c75f6757af72304f | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate2-tested-2026-10-02b | Exact original 6/14 comparison. Historical proof, keep. |
| BuildTestEmbedded / wip/invariantgate2-evidence-2026-10-02b | aeb6a95a758b5c58e2b6c6e7561b4e273b872a75 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate2-evidence-2026-10-02b | Older complete evidence, keep. |
| BuildTestEmbedded / wip/invariantgate-handoff-2026-10-02b | 443bb7eb9c01def0b230948f3e5d20c92c10195e | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/invariantgate-handoff-2026-10-02b | Superseded source, preserve historical context. |
| BuildTestEmbedded / wip/heapowner3-handoff-2026-10-02b | ec24ae6a4c5af82b6142549ade9391d16fcd21d5 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/heapowner3-handoff-2026-10-02b | Earlier constructor heap comparison; do not duplicate this implementation. |
| BuildTestEmbedded / fix/projection-repair-missing-child-sources-2026-10-02 | 3a7293e1d73b8a9be56a48d91ec06e8aa84c3042 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1198 | OPEN, aggregate red; missing-child owner work not landed. |
| BuildTestEmbedded / merged readback correction | merge 59db2da309fdb76715445fd574308cc929047a06 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1202 | MERGED. Re-run gate on current main, no dependency publish in this lane. |
| BuildTestEmbedded / merged journal heap correction | merge 068b5607bc15a53346d3ee6ad0bebd935aeba061 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 | MERGED. Does not prove full 200-run population. |
| BuildTestEmbedded / disk-backed reset follow-up | 64f45f52590ee5877ba1e1eb7e8b9de3bcbd9b1d | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1223 | OPEN, aggregate red. Relevant to reset bounds, not adopted here. |
| BuildTestEmbedded / recovery sibling | a05b6cfc21b7e894feb84bc2548189d4afbd6d83 | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1207 | Separate active lane. Do not change it. |
Next steps
- Re-verify claims, origin/main, every PR head/check and execution health. Both remote requests are terminal without tests; addressed primary RPC logs are preserved. Resume source51eb77575d40216c41b2ff7c9435ed0ee5ff2f06, not an old local comparison. Prepare cold fixture/index cuts outside the measured held phase, through the existing public API and build-fixture pattern; retain25MB output/depth4/W8/R4/5-second held-call bounds and exact delivery assertions. Measure setup independently, then obtain one attributable current-source held-reset verdict without changing bounds or weakening the producer.
- Correct MetadataWork setup to service-owned ingestion using metadata-public-path.md: public submitBuild plus real local SSH spool, or the documented singleShardRunnerEventFixture callback with explicit transport-coverage limit. Its current internal BuildRunImpl/processRunnerEvent bypasses service ownership. Profile24-run preparation; the historical30-second expiry is not a measured metadata-readback verdict. Preserve all production sizes, deadlines and positive controls. Resolve the soak with a fail-first reproducer at its owning mechanism; one published-binary diagnostic PASS is not a fix.
- Finish every promised coverage row: storage 96-cell matrix with artifact variants, deadlines and once-only diagnostics; independent 200-run/mixed-size heap and population growth proof; complete parse/response-byte/depth bounds; held reset publication and simultaneous startup/maintenance/writers; unavailable queue global order/process interruption; restart deadline and late callbacks. Existing fixtures/selector names alone do not satisfy these rows.
- Recheck product owner PRs against actual failures; do not duplicate result-count or recovery sibling work. Current terminal convergence is distinct from exact intermediate-event delivery.
- Run full remote suite before pushing shared-contract changes, record counts, open/update PR only onto main after immediate rebase and state check. Complete separate test-comprehensiveness and adversarial review passes, address findings, then obtain green required checks. Supervisor final review and merge remain the only completion gate when implementation and proof are genuinely ready.
Operational knowledge
Fresh checkout /code/ws/L11/BuildTestEmbedded; evidence worktree /code/ws/L11/BuildTestEmbedded-evidence. Clone source branchwip/L11-production-invariant-gate at51eb77575d40216c41b2ff7c9435ed0ee5ff2f06 and rebase on current main. Evidence branchwip/L11-invariant-gate-evidence also preserves rebased-gate.patch, held-reset source, reviews and logs under incident-notes/L11-invariant-gate. No build binaries or credentials are stored there.
Use PATH=/code/ws/bin:$PATH. Native /usr/local/bin/coursier is wrong architecture. Current repo scripts now bootstrap a verified JVM coursier jar themselves. The remote workspace client packages bootstrap jars too; move bootstrap outside the source before launching pinned kompile, export BUILDTEST_TEST_WORKSPACE to the source checkout, and keep heavy execution remote. The initial 256MiB client ran out of heap in archive construction before run submission; no test verdict resulted. Current client uses 768MiB, timeout 2700; no test heap/timeouts changed. All local client/session PIDs must be stopped or terminal before leaving.
Historical labels: I1–I6 are the six invariant rows in docs/production-invariants.md; heapowner3 is the previous journal-constructor heap owner, hotjson3 the previous metadata/readback owner, lostterminal5 the previous unavailable-append owner. These are historical labels, never source or new PR wording. The stronger unavailable-delivery and affected-source availability contracts are intentional new gate requirements, not claims that current README already promises exact audit history.
No production deploy, publication, merge, enqueue, test weakening or sibling branch modification occurred. The current service-backed handoff CLI replaces the older PlanRepository handoff PR procedure. report-challenge CLI auto-enqueues/merges and therefore cannot run within this lane's explicit prohibition; preserve tooling friction in durable evidence for the supervisor instead.
The BuildTestCli README pins0.0.3, which returned HTTP404 during this session. A temporary read-only Java client used the pinned kompile public UrlProtocolBuildService.rpc("listBuildRunsPaginated",offset0,limit100) to find the run; the source is preserved. The HTTP adapter forwards that actual response unchanged to build-watchman and performs no retries or fabricated statuses. Stop its JVM when the watcher ends. Every local process started by this lane must be terminal before exit.
Resume the isolated source check with local-current-source-plan.md only after re-verifying current refs. It is not a prescribed128MiB production bound:128MiB was a small compatibility diagnostic;512MiB is the documented production tier. The fixture/test was unchanged between both current-source diagnostics. Both guard terminations occurred before arming reset, so they cannot be used as evidence that held readers/writers violateI4. Prepare cold cuts separately; do not change the original measured bounds to compensate. Owned diagnostic temp roots were inventoried and removed after process exit.