RE-VERIFY: refreshed2026-10-09 08:18UTC. Live component state is in the appended pagefix2 section; the earlier snapshot is retained as provenance for the broader normalized-store work.

Handoff: Finish the durable incremental per-run result-page projection

Written: 2026-10-09 06:44 UTC.

RE-VERIFY: This is a write-time snapshot of unfinished work. Check remote branch heads with git ls-remote, each linked Actions run with gh run view --json status,conclusion, raw first-attempt XML artifacts, and PR state with gh pr list --head wip/sol-pagefix1 --state all. Nothing below authorizes merging, queueing, marking ready, or production deployment.

Mission summary

The request was “implement the persisted projection so a run page read never re-decodes whole sources” in BuildTestEmbedded, addressing “the root cause of projection update slowness.” The prior production-sized fixture showed 76–85% sampled CPU in whole-source decodes and full result publication, versus 0.2% selected-row work. The brief requires a durable per-run normalized row/order/count/checkpoint projection, incremental committed-event updates, bounded reads after restart, deterministic red-first tests, regression10/10 and neighboring3/3 first attempts, unchanged fixture before/after timing in the same Actions partition, a product-revert mutation control, one full four-shard CI, and a draft why-first PR. The original lane has an 85-minute deadline ending07:10UTC; it must stop there and must not claim READY while this work remains unfinished.

The design is prescribed by https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/playbooks/SLOW_SERVICES.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/STATE_MANAGEMENT.md. Read the complete previous design at https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-pagework1-evidence/handoff-artifacts/pagework1/ANALYSIS.md, and testing guidance at https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md before implementation or tests.

What was found and done

1. Reproduced one child JFR profile on unchanged main f87fd2d969d2c78b3a17104f7ca65583f5ca012a, actual toolchain0.0.112, original200×2000×16000 fixture/full outputs, at https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37890376471. The200 PAGE_WORK calls sum96071ms, median480.5ms. Main page samples12043: full snapshot publication4796, run-load/decode3875, terminal-status decode1840, selected row/metadata22. Recompute87.3%; selected0.18%. Native FileRead event sums include stacked read events; do not present them as exact logical bytes. This profile measures first legacy conversion, not a converted cold restart.
2. Added production-sized public-API cold-restart regression tests/testPersistedRunPageRestartReadsOnlySelectedRow.kts and its isolated child fixture/rule. First service converts all200 histories through public APIs, closes; new service visits32 startup directories; one page targets an observed outside-cycle run. Existing result/rebuild observers and an Okio forwarding filesystem count work. Main first attempt at https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37891779103 fails: Cold-restart page should read zero bytes from target run.json, but read 4628026. Target00000020; resultBytes5555664; decodedRows2001; rebuilds0; publications0. An earlier missing-annotation compile failure is not credited as reproduction.
3. Mechanism refinement: getTestResultsPaginated calls repairStartupEvidenceForRead before the selected-row path. Startup marks previously converted terminal histories pending. Addressed repair loads the large run twice and full committed result list to recount, even when no named rebuild callback occurs. Initial legacy repair also publishes full snapshots and status-parses the run. Evidence and original line references are in the linked FINDINGS.md. A cold cache miss still reads all94946bytes of test-results.page-inputs-v1 for2000rows.
4. Implemented only the compact completion component: TerminalResultPageProjection.kt persists terminal status/counts with run/event/result length, precise mtime, inode identity and Unix change time; ordered-field checksum; maximum8192-byte reads; atomic publication only after committed reconciliation and exact streamed run-count agreement. It reuses the strict scalar JSON reader in UploadRequestOwnerReader.kt. Service repair validates this completion before opening full sources. Source changes/corruption refuse it; changed completion must bypass the old cheap result marker and final-utilization shortcut and evict the event projection before authoritative fold. Fast validation holds the existing terminal boundary; close guards suppress publication; warnings are deferred past ownership and guard warning-handler re-entry. Unsupported filesystem change time refuses this shortcut. Every file handle is operation-owned. No reflection or new timeout.
5. First component head bbe6ce88f75c8d12c30b4b2c7e219ca2bedc726f passed once at https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37893335994 (114302ms, raw XML success=true, no retries). Review followups at3dd4d340970125f458a7a3f3a63eefbdc3eba648 are separately being gated. Passing this narrow test is NOT proof of total bounded I/O or full task completion.
6. Two isolated source-generation controls at1ec46c03d94c9e4b236caa12485432cc02f31d86 were independently locally red, then first Actions red at https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37894503996. testTerminalEventLogGenerationInvalidation.kts rewrites SUCCESS→FAILURE without changing byte length/inode and restores oldmtime; outside-cycle page incorrectly keeps PASS. testTerminalRunRecordGenerationInvalidation.kts rewrites run counts while event remainsPASS; after reconciliation/another restart, page still reads1736run bytes versus0. Each owns33real histories/FakeDropletService; public event-cache eviction models cold process; each checks public counts/output after reconciliation. Root cherry-picked these at1c3f779ccdc19e6b4a7ca02e9fd4b8274288e63d; this adds tests only.
7. IMPORTANT REMAINING DESIGN GAP: there is no durable normalized result row/order/checkpoint store, no incremental event-path durable row updates, no persisted migration progress. Existing projection remains full in-memory replay plus full flat export, and current cold page still scans all compact descriptors. Do not describe the compact completion as the prescribed full projection fix. No requirement is silently waived or refuted.

Relevant PRs / refs

Related bulk-journal PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1354 was OPEN/BLOCKED at06:42UTC, head8657b7c16138234a1dce4af3886d342036e462ed. Its original brief head wasc24632ff. Current component does not touch PagedJournalRecords.kt or ProjectionChangeJournal.kt. If continuation touches either, stack on its verified latest branch wip/sol-scale1 and name that dependency in the PR body.

Deployed/published but unmerged: none. This lane performed no deploy, service restart, Maven artifact publication, coordinator or production-host operation. Only remote Git branches and isolated Actions test jobs were written.

Next steps

1. Re-verify current state and download raw first-attempt XML. Record actual results of regression repetition run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37894230681 (10 separate logs), neighboring contract run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37894396762 (3 batches), and product-revert run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37894424086. Source-control fixed run was submitted under gha-run/pagefix1-generation-controls-fixed; recover its URL from Actions head/branch. Do not infer passing gates from workflow success before auditing first XML attempts.
2. Design/implement the actual durable normalized result store and incremental committed-event reconciler. Existing hooks: RunEventProjection.put, identity migration/drop, output materialization, clock/authority/XML dependency changes; all committed append paths including appendTestEvent, appendAndProjectTestEvent and synthetic terminal events. Rows/count/order/checkpoint must commit atomically. A flat JSON compatibility export cannot be a prerequisite for one-row reads. Duration averages affect severity ordering and require coherent invalidation; do not persist a stale ordering from one global duration map. Keep public counts/status/output/order exact and memory bounded.
3. Preserve32-directory startup cycle contract (startupTerminalRepairHonorsDirectoryCycleBudget:32/64/65 across cycles and0restart rebuilds). Legacy conversion can scan once through the same reconciler but must persist progress and not retain all historical state in heap. Verify changes in unsupported-ctime filesystems without guessing at a portable generation proof.
4. Add tests for every invariant row: source/run replacement, exact event span/affected row incremental work, cache eviction, alias/XML/clock/severity changes, output chunks, offset/limit/empty pages, reader mutation, callback re-entry, close/deletion mid-publication, crash between rows and checkpoint. Existing new tests only count injected run-body reads and result observers; native reads and compact metadata reads are uncovered. Use genuine public ingestion and real dependencies; never mocks/reflection, weaker assertions, larger timeouts, fewer iterations, or artificial CPU load.
5. After full design passes reliable red-first regressions, run exact gates10/10 and existing3/3 on actual pin0.0.112, product-only revert red, unchanged production fixture before/after in same Actions partition, independent tests/code review. Open draft why-first PR only after prescribed targeted and mutation gates; quote goal/provenance, prior76–85% and new87.3% measurements, link both documents, include coauthor and Claude footer. Fetch/rebase immediately before builds/PR, run required tests after rebase. Run one full four-shard PR CI and audit raw first-attempt XML. Never merge/enqueue/mark ready/deploy. READY final phrase only after ALL brief requirements proven.

Reusable / operational knowledge

Use fresh checkout under the session workspace. Current actual kompile pin is0.0.112;0.0.121 is expressly not a gate. Multi-test gates use the supplied tools/gha-run.sh (Actions execute --local on their own machine), maximum4 simultaneous lane partitions. Local slots only for short targeted/compile work; no local full suite. The helper's75-minute limit exceeds this lane's deadline; track it with an outer shorter timeout and cancel owned Actions jobs on final cleanup. Never touch the coordinator/WUI/production host.

The evidence branch's handoff-artifacts/pagefix1 retains helper source, native JFR consumer, baseline sampled stacks/counts, and explicit coverage limits. Raw JFR/GC/console/XML are in linked Actions targeted-test-results artifacts; JFR includes environment events, so only derived data is committed. Build jars/caches are deliberately excluded. Artifact retention is finite; download evidence on pickup.

One first contract push was rejected with HEAD -> gha-run/pagefix1-three-contracts (failed) before any test existed; verified no remote ref and used a distinct-label push successfully. No failing test was requeued. report-challenge CLI auto-enqueues/merges and was not invoked because this brief prohibits that; the observation is retained in findings. An initial controls fixture used unsupported test_completed.output; corrected to genuine test_console_output.entries before crediting red. The handoff README still mentions git merging, while current skill uses the central CLI; follow the current service process.

Pagefix2 handoff

Written 2026-10-09 08:12 UTC. RE-VERIFY branch heads, raw first-attempt XML and PR state before acting. This is the compact terminal-completion component, not the full normalized-store mission. No merge, queue entry, mark-ready, publication, deployment or restart was performed or authorized.

The user asked to "solve the underlying root cause of any projection update slowness". The component persists terminal status/counts against committed run/event/result generations, and central committed header publication advances that evidence only when counts and unchanged sources agree. Cold reads still scan compact descriptors; zero injected run-source bytes is not zero native disk I/O. Durable normalized rows/order/counts/checkpoints and incremental events remain explicitly separate work.

Relevant branches

Findings and reproductions

The initial gate found that a public deletion in the snapshot-publication callback removed event/results but its writer continued to validate the removed source and threw. The deterministic unchanged publication-deletion test failed locally and in Actions before the fix. Independent review then identified startup keeping the captured terminal record after deletion; its public startup-callback test failed first attempt on recreated .recovery-complete-v1. Terminal checks at callback returns, outer read/startup continuations, and marker creation now abandon that work. No exceptions are suppressed. Local targeted tests changed to PASS; exact final Actions results are recorded below.

A warning handler re-entering a page could queue a second warning while the guard was true, then deliver it after the guard cleared. An exact-count public callback reproducer failed first attempt with two full identical warnings and passed after the scheduling-time guard. The proposed no-handler exception premise was refuted by a genuinely separate ordinary JVM that returned correct truth and the warning without any caller handler; no speculative catch was added.

The production regression retains200runs×2000rows,16000plan entries,2600-character output,512m SerialGC/192MiB retained heap and original300/360s child/parent budgets. A real Runtime.halt77 between snapshot and completion publication precedes ordinary conversion and measurement in separate JVMs. All child readers disable unrelated public-row backfill. Exact last/beyond boundaries and returned-row mutation are checked.

Evidence

• Historical ledger: raw historical stronger PASS1/mutation FAIL1, controls, four neighbors3/3, and unchanged paired timing.
• Initial current-head audit: mutation first FAIL124810ms; event/run controls first FAIL; initial contracts11PASS/1FAIL deletion; cancelled old-head regression2PASS with no final-head credit. Nested retries get zero credit.
• Findings distinguish OBSERVED and INFER and name all documents read.
• Paired initial-conversion timing: main200pages sum99590ms median490.5 mean497.950 range6–914; component sum98675 median498.5 mean493.375 range6–838. It does not establish faster first conversion. Run.

Final gates on1bba4d6f7: five A, five B, contracts3. GATES_PENDING

Next steps

NEXT_PENDING. Keep the normalized durable row/order/count/checkpoint store and incremental event projection as the broader handoff's remaining design work; do not claim this component finishes that architecture.

Infrastructure notes and cleanup

The first contracts submission failed before any workflow with remote fatal error in commit_refs; one resubmission succeeded, not a test retry. Report-challenge automatically enqueues/merges its record and was not invoked because this brief forbids those actions; the failure is preserved in findings. No build jars/caches were committed. CLEANUP_PENDING

Pagefix2b service regression checkpoint

Written2026-10-09 10:05UTC. RE-VERIFY: this is a write-time snapshot; recheck the PR head, full CI artifacts and branch refs before resuming. Existing draft https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1366 is OPEN, unmerged, atb4d16b1da33c2e504ee3f9ee9fb0a11ab4203a64.

Mission and cause

The pagefix2b brief asks to repair seven failures from full CI37905103470 while retaining the compact terminal-completion component's contracts. This round does not implement the broader durable normalized-store mission. All seven reproduced first attempt on unfixed rebased0b075e500 in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37912352668 before product edits.

Three mechanisms: initial completion probe took the noninterruptible terminal boundary before the existing read handoff, blocking committed readers and interrupted missing-snapshot readers; the current-completion path cleared pending repair before adopting a peer-repaired resident record; the stale-completion classifier replayed a legitimate result+marker replacement even when authoritative run/events were unchanged. The fix admits readers first, adopts changed peer run records before completion, and recognizes a current marker-backed result replacement without supplying old page scalars. Exact current-page reads remain strict. Ctime-only changes, unmarked replacement, damaged completion and changed run/events still reconcile. Tests, workload, assertions, timeouts, actual kompile pin0.0.112 and completion schema remain unchanged.

Durable refs

• Product checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-pagefix2b
• PR topic: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-pagefix2
• Evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-pagefix2b-evidence/handoff-artifacts/pagefix2b
• Product-revert proof: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/sol-pagefix2b-mutation-proof

Live refs checked10:05UTC:

b4d16b1da33c2e504ee3f9ee9fb0a11ab4203a64	refs/heads/wip/sol-pagefix2
b4d16b1da33c2e504ee3f9ee9fb0a11ab4203a64	refs/heads/wip/sol-pagefix2b
045c73961c6f478dc9cc66a38e54902c3b3a06e5	refs/heads/wip/sol-pagefix2b-evidence
363ffa49a409cca4113d6c85798944168378407d	refs/heads/wip/sol-pagefix2b-mutation-proof

Verification

• Seven unchanged service regressions each3/3 plus deterministic peer-adoption3/3:24/24 raw first passes, no retries; https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37913415164
• Production-sized cold restart regression3/3 raw first passes, no retries;193746/186471/180560ms; https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37913475157 -13component/neighbor contracts once:13/13 raw first passes, no retries; https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37913478884
• Same-test all-src revert to main48c143b35: three first failures, including cold target run read4,628,026 bytes vs required zero and both generation controls; https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37912610131
• Short local missing-snapshot proof: first pass3101ms.
• Independent test and code reviews complete. CODE-REVIEW.md found no actionable delta defect; TEST-REVIEW.md records existing assertion limits. Marker describes freshness of a committed pair, not writer provenance; the sentinel deliberately updates both files. The suggested missing completion signal was refuted by blocked-reader stacks: interrupted thread was stuck on the initial lock, and test cleanup was joining it. A speculative deletion-hook ownership cycle was refuted as newly introduced because the parent had that same order.
• Normal four-shard CI started once: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37915259503 on exactb4d16b1. First-attempt audit pending. Separate kotlin-build-ci-test suite queued with zero runs matches mature main suites; known outage, no rerequest or restart.

Next steps

Finish bounded normal CI watch, download all four raw XML and audit first attempts against the recorded main XML. Classify every first failure; no retry credit, no blind reruns. Update PR body Round pagefix2b and this evidence with exact outcome. READY requires the requested full audit, not workflow success alone. Do not merge, enqueue, mark ready, deploy, publish or restart services.

Deployed/published but unmerged: none. Only Git refs and isolated Actions jobs were written; command history contains no deployment, Maven publication, production-host or service restart action. Broader normalized-store handoff remains active and must not be completed for this component-only result.