Finish the filesystem read-state correction and qualify the existing pull request

Written 2026-10-27 07:52 UTC. This update supersedes the earlier fairness-only readiness claims preserved below.

RE-VERIFY: This is a write-time snapshot. Check https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41 with gh pr view <url> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, and compare git ls-remote with the branch table before acting. The correction is on a separate candidate branch; the existing PR head was deliberately not updated before qualification.

Design decisions in force

On 2026-10-27 the supervisor authorized: "replace the fairness toggle with a read-state design that guarantees independent lifecycle and read checks never wait behind unrelated manager storage work."

The supervisor also required: "The test must pass; also keep the original 30 s bound and every iteration count, and the existing regression." The original nine workers, 10,000 reads each, 25-second worker bound and 10/2-second readiness/cleanup bounds remain unchanged. No test assertion was weakened or removed. Do not publish, merge, enqueue, deploy, restart services or change rulesets. Stop the existing PR at an open rebased head with required bld-build green; the supervisor makes every merge decision.

Mission summary

Finish verification and any remaining corrections on the candidate branch below, run the full suite under the machine-wide lock, update the existing PR and rewrite its description around the read-state design, then get required bld-build green. The fairness premise was refuted: nonfair locking still permits the old queue ordering. Controlled public-API regressions now fail on both main and the old PR implementation without timing a fixed linear workload.

The latest source checkpoint is 120f2aad737d273adec6dc26993fa288b50bb84f, not PR-READY. Its focused run is still in preparation at this write time. No final-head full suite or CI has run. The supervisor ordered this lane to checkpoint and stop before about 08:00, with a relaunched lane finishing qualification. The original three historical filesystem deadline mechanisms remain unresolved and are not claimed fixed by this correction.

What was found and done

1. Main and the old PR both take the manager state lock for lifecycle checks, cached opens and quota/expiration getters. Holding a real descriptor-storage callback while another thread makes those public calls forces an unbounded dependency on that callback returning. The native regression failed first attempt on fair main and on the nonfair PR implementation. This is a contract/order discriminator, not a speed comparison.
2. A clock callback can close the manager while openFilesystem checks expiration; old code then returns successfully. A public regression covers cached and uncached close/delete/expire callbacks. The first read-state candidate passes that scenario.
3. The first read-state candidate only removed read-check locking. It remained unready: first-attempt 1/4 focused passes, with both progress scenarios still blocked and the original 90,000-read guard failing all three native attempts at its unchanged worker bound. The remaining code path is OkioSimpleFileSystem.unpin() -> onPinsDrained -> manager reclaimPendingDeletionIfUnpinned, which takes the global state lock on every last-source close even for an existing descriptor.
4. Review added deterministic regressions for a nested later expiration, a nested deletion, last-pin release while deletion storage is held, and a storage error reported after journal removal. Every scenario was observed red on the first candidate before the corrective implementation. Nested expiration returned 1000 instead of the later 2000; nested deletion left the lifecycle lookup readable; pin close waited for storage; post-apply expiration remained null instead of the persisted 1000.
5. Latest commit 120f2aad addresses those conditions: read values are immutable records in a ConcurrentHashMap; completed handles are safely published in a concurrent cache while construction remains serialized. Lifecycle publication runs in finally and reads the current authoritative descriptor by UUID, including removal, so a later nested transition or post-apply error is retained. Closed/recovery flags remain authoritative. Pin-drain cleanup returns without the manager lock when a published descriptor exists and recovery is not required; successful deletion already performs a post-transition pin check, covering pins drained before publication. Recovery retains the locked cleanup path. These final corrections still require green evidence; they are not claimed verified.

Resource	Owner/mutation	Reader consistency	Close/failure/restart
Working descriptor/revision bookkeeping	Manager, under existing state/finalization/lifecycle locks	Combined info/usage/list/watch APIs retain their existing ordering	Recovery flag rejects use after an incomplete journal; restart rebuilds authoritative descriptors
Published lifecycle values	Manager copies current UUID/max/expiration values when a transition exits	Atomic immutable per-UUID value; mutable descriptor objects never escape	Current authority is re-read after callback-capable storage, including finally on failures; delete removes the value; construction republishes recovered records
Cached handles	Construct once under state lock; publish only after hooks are installed	Cached opens and handle checks avoid unrelated manager storage locking	Handle data operations retain lifecycle/tree/path admission; stale handles check terminal state
Pin release / pending deletion	Existing handle pin counter and manager deletion journal	Active descriptors need no global cleanup check	A pin drained before deletion publication is picked up by the deleter's own post-check; absent descriptors and recovery take existing reclamation path
Closed/recovery flags	Existing atomic/volatile flags	Checked around lifecycle evaluation and after reentrant clock callbacks	Close stays terminal; cleanup/root ownership code is unchanged

Native first-attempt receipts

true/false below is the first native attempt, using attempt-history when present. Total runner duration includes startup/cleanup and is reported, not the new regression verdict.

Source	Test	First-attempt success	Runner duration
Fair main	testManagedLifecycleReadsProgressDuringUnrelatedStorage	false	8328 ms
Original nonfair PR	testManagedLifecycleReadsProgressDuringUnrelatedStorage	false	7471 ms
Original nonfair PR	testManagedLifecycleReadStateTracksTransitions	false	6906 ms
Clock callback old-head baseline	testManagedLifecycleReadRechecksClockCallbacks	false	1917 ms
First snapshot candidate (NOT green)	testManagedLifecycleReadsProgressDuringUnrelatedStorage	false	10784 ms
First snapshot candidate (NOT green)	testManagedLifecycleReadStateTracksTransitions	false	10019 ms
First snapshot candidate (NOT green)	testManagedLifecycleReadRechecksClockCallbacks	true	5774 ms
First snapshot candidate (NOT green)	testConcurrentManagedReadersCompleteWithoutPerCheckHandoffs	false	30691 ms
Candidate review regressions	testManagedLifecycleReadStateKeepsReentrantTransition	false	9172 ms
Candidate review regressions	testManagedLifecycleReadStateKeepsReentrantDeletion	false	7049 ms
Candidate review regressions	testManagedPinDrainDuringDeletionStorage	false	16196 ms
Post-apply error regression	testManagedLifecycleReadStateSurvivesJournalRemovalFailure	false	7093 ms

Raw XML and complete stacks are retained in /home/u/lanes/hd27/out/hq057-*.xml and /home/u/lanes/hd27/out/hq057-findings.md. The source tests and their red checkpoints are on the remote branches below. Evidence archive /home/u/lanes/hd27/out/hq057-native-evidence.tar.gz contains XML/findings; a one-year Filedrop upload failed without returning a URL. Critical observations/counts are embedded here and in status reports so the next reader need not reconstruct them. No jars, dependency caches, raw JFR or credentials were included.

Relevant PRs / refs

All rows refer to CodexCoder21Organization/SimpleFileSystemServiceEmbedded unless noted. Remote SHAs were verified with git ls-remote at 07:46 UTC.

Branch / PR	Remote head	What it contains / state
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/fix/hq057-read-state-candidate	120f2aad737d273adec6dc26993fa288b50bb84f	Latest correction plus all seven new public tests; final verification incomplete, no PR
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/fix/hq057-lifecycle-red-evidence	bd1961ea06796a412b6d57be01bf1965635106b8	Corrected test-only old-PR baseline; deterministic regressions red, no implementation fix
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/fix/hq057-main-red-evidence	fd93c9587896c398295ee5105d58b736da26afec	Corrected test-only fair-main baseline; progress test red
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41 / https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/fix/hq031-concurrent-read-handoffs	776fe204bc026eb9687d7ffb85fbb2ac521ba132	OPEN, old fairness-only head, old required bld-build SUCCESS; review findings mean NOT ready. This lane did not push to this PR branch
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/main	38495dcf3d37c531a37e1e78c054a16201a6baf5	Unchanged baseline
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/evidence/hq031-manager-read-cost	fc9c4794e3f77e0344b05af6ae2ea872242f2806	Prior lane's historical diagnostic patch
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/tree/fix/hq014-fakefilesystem-readers	bd9e3075b1f271333134184e46a01c5fe36fa961	Prior consumer checkpoint, unchanged; not qualified or published by this lane
https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43	d5c03fe5f3b9c0389f6b29c67c28526a6ecdcc49	MERGED 2026-10-26; this does not establish the historical deadline cause
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1677	2fdaa9ba1cef4fe839fb8518ff8f4e2ea54c910d	OPEN at final live check; original coordinator dependency remains separately tracked

No production deployment, service restart, queue action, merge, ruleset change or Maven publication occurred. Embedded 0.0.34 is prepared only; its POM still returns 404 at 07:46 UTC. Do not pin or publish it as if qualified.

Next steps

1. Re-verify current state first, including the existing PR state, candidate branch SHA and claim. Start from the candidate remote branch, not the old PR head. Local checkout is /home/u/lanes/hd27/ws/hq057/workspace/SimpleFileSystemServiceEmbedded; its local branch name is the old PR branch name, but only the separate candidate remote was pushed. No stash or unpushed commit remained at the write-time sweep.
2. Read Testing architecture, Engineering Philosophy and the racing-CI guide before changing code. All current bounds and counts are fixed. Re-run focused verification on the actual final head; do not reuse first-candidate failures as final results. The eight relevant scenarios are:
   • testManagedLifecycleReadsProgressDuringUnrelatedStorage
   • testManagedLifecycleReadStateTracksTransitions
   • testManagedLifecycleReadRechecksClockCallbacks
   • testManagedLifecycleReadStateKeepsReentrantTransition
   • testManagedLifecycleReadStateKeepsReentrantDeletion
   • testManagedLifecycleReadStateSurvivesJournalRemovalFailure
   • testManagedPinDrainDuringDeletionStorage
   • testConcurrentManagedReadersCompleteWithoutPerCheckHandoffs
3. Use scripts/test.bash --local --test <name> with repeated --test selectors and --log. Only one local build/test command per lane. Native preparation for a new source head took about 12–13 minutes here; that is outside the test-body bound. If a final-head test fails, diagnose and fix it test-first; do not retry unchanged past it or weaken bounds/counts. Source and test review must confirm deletion cleanup is not lost around publication and that callbacks retain the latest state.
4. Run the full suite once on the exact final head under flock /home/u/lanes/hd27/kompile-full-suite.lock (expected 353 tests after the seven additions; verify actual discovery). The supervisor explicitly authorized local full suites on 07:10 UTC because remote returns STARTING during recovery. Preserve first-attempt counts, not only final green counts. No final-head full suite has run yet.
5. Only after qualification, verify https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41 is still OPEN, fetch/rebase, update its original head branch with the qualified candidate, rewrite its description/title around the read-state design and include the ownership table and red/green receipts. The original fairness-only motivation is refuted. Current effective requirements contain bld-build only; workflow already uses both --local-and-remote commands and 0.0.124 pins, so no upgrade PR is needed. Watch with build-watchman 0.0.22 without --to-merged. Report PR-READY only when that exact rebased head has required bld-build green.
6. Do not publish. The supervisor owns merge/publish/pin decisions and the wider original three-deadline investigation. Do not complete/archive the whole handoff just because this PR becomes ready.

Operational knowledge

Remote health currently rejects dispatch with STARTING: recovering local state (elapsedMs=7180182) instead of OK; it is not a test failure. Handoff CLI sometimes fails fetching its 2.8 MB service bytecode with partial chunks or a bounded-dial-task limit; bounded later clients have succeeded. Reports that succeed say usage attachment failed on inherited supervisor transcript line 19358; the report body is still stored. report-challenge CLI automatically merges, so the lane prohibition prevented invoking it. Filedrop upload returned no usable drop. No production operation was used to work around these service errors.

Read documents: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md ; https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md ; https://my.externalbrain.wasmserver.com/doc/github-builders-racing-remote-ci.md ; https://github.com/CodexCoder21Organization/PlanRepository/blob/main/handoffs/README.md . Explicit lane racing-CI instructions supersede older central text requiring informational remote checks. Preserve the exact 25-second worker, 30-second body and all stress iteration limits.

Earlier historical work (preserved; readiness claims above supersede older claims)

Review the manager read correction and finish the filesystem deadline diagnosis

Written 2026-10-27 03:57 UTC.

RE-VERIFY: This is a write-time snapshot. Check https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41 with gh pr view <url> --json state,headRefOid,statusCheckRollup,mergeStateStatus,mergedAt, compare remote refs with git ls-remote, and check artifact publication before changing a consumer pin. The worker did not merge, enqueue, deploy, publish, or complete this handoff.

Design decisions in force

Supervisor direction on 2026-10-27: "keep the 25-second worker bound and every other bound unchanged; do not ask to raise them. If the guard is too close to its bound, the work is to find and remove the cost that makes a healthy run take 24 s."

All bounds, iterations and assertions remain unchanged. No production action or artifact publication was authorized for this lane. bld-build is the only effective required check in both filesystem repositories; informational remote checks are not a gate. The supervisor owns every merge decision.

Mission summary and remaining work

The added reader guard's cost is corrected upstream and PR-ready at https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41. Review/merge and publication/pin remain with the supervisor. The original three historical filesystem deadline mechanisms remain unresolved; this separate correction does not establish their cause. Their remote baseline/coordinator dependency is still tracked by url://handoff/handoffs/hf-2026-10-26-restore-coordinator-progress-and-cleanup-for-filesystem-baseline-run-b2e94bd0 and active https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1677. Do not archive this handoff merely because the adjacent guard correction is ready.

What was found and done

The server's separate OpenFile null-entry error has a published FakeFileSystem 3.4.0.1 correction. Downloaded artifact bytes match SHA-256 ce679551d2812f30a71f3a883e84eeb0c282b470b37af722a452ac5a23909715. Its consumer regression through real client -> RPC handler -> Embedded manager still exceeded the unchanged 25-second worker bound after that dependency correction.

The measured mechanism is a forced handoff through the fair manager bookkeeping lock on repeated independent reads. Nine worker stacks queued on the same FairSync despite only 0.27–0.34s CPU each at 14.6s elapsed. The direct manager public regression failed all three normal-runner attempts (28.401/29.929/29.466s total runner time) at Future.get(25 seconds), plus a separate prepared-test execution. Virtual workers alone remained too close to the bound and eventually failed; that experiment was reverted, preserving the original nine platform workers and all 90,000 reads.

The fix changes only the private state-lock fairness setting. Critical sections, conditions, ready-commit revisions, separate fair finalization and per-filesystem lifecycle locks remain intact. Version 0.0.34 is prepared, not published. A matched consumer profile with the normal runner JVM flags reduces sampled manager parks from 5,336 / 81.313 aggregate seconds to 121 / 2.029 seconds. No timeout, iteration, assertion, memory cap or old test was changed.

Validation: corrected direct target passes first attempt (8.672s total); seven fresh upstream JVM runs pass (6.569–10.270s body), seven original consumer platform-worker runs pass against the local corrected manager (9.369–16.603s body), and eight lifecycle/reentry/queued-recovery/ready-group neighbors pass first attempt. Complete source and test reviews found no remaining defect. Full independent qualification is 346/346 first-attempt passes, zero retries, at https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/actions/runs/40445370938. PR-triggered confirmation is also 346/346 first-attempt passes with zero attempt-history records at https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/actions/runs/40446019176. New guard totals are 3.295s and 7.830s respectively. Required bld-build is SUCCESS; PR is OPEN at 776fe204bc026eb9687d7ffb85fbb2ac521ba132. GitHub's UNSTABLE label reflects pending informational remote checks. No enqueue or merge.

Relevant PRs / refs

Repository	Branch	Remote head	PR	State
SimpleFileSystemServiceEmbedded	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/fix/hq031-concurrent-read-handoffs	776fe204bc026eb9687d7ffb85fbb2ac521ba132	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41	OPEN, rebased on main 38495dc, required bld-build green, two 346/346 first-attempt full suites
SimpleFileSystemServiceEmbedded	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/tree/evidence/hq031-manager-read-cost	fc9c4794e3f77e0344b05af6ae2ea872242f2806	none	Data-only candidate patch checkpoint; compiled source is the initial test-only baseline
SimpleFileSystemServiceServer	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/tree/fix/hq014-fakefilesystem-readers	bd9e3075b1f271333134184e46a01c5fe36fa961	none	Source tree exactly equals prior 6ff5574; published fake-storage pin plus consumer regression. Needs authorized Embedded 0.0.34 publication/pin before normal consumer qualification
SimpleFileSystemServiceServer	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/tree/main	5e7f084bec176e09a781e6ce8ad99892ef2992bf	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43	Already merged before this lane, required bld-build SUCCESS

Patch checkpoint: https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/blob/evidence/hq031-manager-read-cost/handoff-artifacts/hq031/manager-read-cost.patch. Applied to main 38495dcf3d37c531a37e1e78c054a16201a6baf5, it exactly reconstructs tree 7cce7231ad40dfc42b5da97c8f69062abfb2488b, verified through an isolated Git index. No partial source is only local.

No deployment or Maven publication occurred in this lane. Version 0.0.34 exists only as build output from the unmerged PR, not as a globally published artifact. Do not pin it before verifying publication. The already published FakeFileSystem correction above predates this lane.

Next steps

1. Supervisor review/merge decision for https://github.com/CodexCoder21Organization/SimpleFileSystemServiceEmbedded/pull/41. Worker stops at PR-ready, never enqueues or merges.
2. If authorized, publish Embedded 0.0.34 from the reviewed source using the publishing workflow. The lane's explicit no-publication rule prevented doing this. Then update the server's Embedded dependency pins on the preserved consumer branch, retain FakeFileSystem 3.4.0.1 and the original platform-worker guard, run normal full qualification and open its separate PR.
3. Re-verify the coordinator/baseline dependency and the original three deadline evidence before resuming that diagnosis. There is still no proven common explanation or reliable matched remote full-suite baseline. Do not call CPU contention alone a completed answer and do not treat this new manager correction as fixing those historical deadlines.
4. Only the supervisor may complete/archive this handoff when the whole remaining mission is verified.

Operational knowledge

Checkouts: /home/u/lanes/hd27/ws/hq031/workspace/SimpleFileSystemServiceEmbedded and its evidence worktree; reused server checkout /home/u/lanes/hd27/ws/hq014/workspace/SimpleFileSystemServiceServer as explicitly directed. Incremental findings and native XML/log receipts: /home/u/lanes/hd27/out/hq031-findings.md and adjacent hq031 files. Generated cache (434 MiB), launcher directory, compiled probe binaries and raw JFR recordings were removed after verification; no build/test output was committed. Portable profile count summaries and Java driver source remain outside the repositories.

The local full-suite slot was occupied by another lane. Two bounded acquisition attempts executed zero tests. After local red/green, repeated public regressions, neighboring tests and profiles were complete, a single workflow_dispatch of the existing racing bld-build qualified the exact head on an independent GitHub runner before PR creation; no CI rerun was used to diagnose or hide a failure.

Handoff reports store successfully but usage parsing points at a malformed inherited supervisor transcript line. Filedrop upload connected but timed out during createDrop, returning no URL; do not count that attempted archive as a remote backup. Earlier lane checkpoint remains https://filedrop.wasmserver.com/drop/bde7401739. All source is remote and essential failure/profile evidence is in Handoff reports. Old gh pr edit fails on deprecated projectCards; REST PATCH with a JSON body file updates descriptions correctly. report-challenge CLI auto-enqueues/merges, so it was not invoked under this lane's prohibition. Informational remote log/status probes returned 404/503; they were never treated as failed required checks or evidence a runner died.

Historical original investigation (preserved)

Handoff: Find and fix the cause of the three filesystem test deadlines after completing a reliable main baseline

Written 2026-10-26 UTC.

RE-VERIFY: Every state below is a snapshot. Re-check the PR with gh pr view https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43 --json state,headRefOid,mergeStateStatus,statusCheckRollup, compare its remote branch with git ls-remote, and read the two full-suite coordinator run records before proceeding. A fresh query wins.

Design decisions in force

The original user asks to "find why three SimpleFileSystemServiceServer tests miss deadlines on the remote build service" for https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43. The original rules include: "A first-attempt failure is a defect until explained: either a regression from the upgrade or a test/product inefficiency. Never raise a deadline or cut iterations, and CPU contention alone is not an acceptable final answer." Also: "No reflection, no mocks, no timeout increases, no iteration cuts, no weakening or deleting assertions. Use the injected Clock." "No notes files in repos." "Do the work yourself; no other model sessions." "Never merge or enqueue." "kotlin.build (remote) is informational ... the required check is bld-build." No production deployment was requested.

The brief asks for targeted remote comparisons, then says targeted runs stay local. This session used remote runs for the explicitly requested main/PR comparison and local runs for diagnosis/profiling. All full suites used scripts/test.bash --remote --test . --log <file>; no local full suite or suite-lock wait occurred.

Mission summary

Finish a reliable full-suite main baseline, obtain a consistent public-path reproduction of the underlying deadline mechanism, and fix it at its cause without changing coverage or budgets. Eighteen requested targeted comparisons are complete and clean on first attempt. A full PR suite reproduced two requested deadlines, but the full main run stalled at the coordinator and lost its lease, so it cannot establish a matched full-suite regression or justify a code fix. The blocker is url://handoff/handoffs/hf-2026-10-26-restore-coordinator-progress-and-cleanup-for-filesystem-baseline-run-b2e94bd0.

What was found and done

L155 deadline investigation — 2026-10-26 UTC Status: blocked; no code-level cause or fix established. Request: find why the storage-stop sweep, session-limits test, and concurrent-close test missed first-attempt deadlines on the remote build service. PR: https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43 Main: 2f8d1553b5d9cd42a792ca5dab2274002fd763ea PR head: 60f7629a8018388672cdaa7c345807836d224094

Requested comparison: all 18 targeted remote executions passed first attempt, zero retries. Each journal has one test_started and one successful test_completed. The remote XML has no attempt history; journal audit is included. Every run used scripts/test.bash --remote --test tests/<file>.kts --log <file>. Main and PR samples were interleaved. Targeted local profiling used --local; no local full suite or local suite lock was used.

Raw targeted times: main sample 1: testCloseWhileClientsOpenAndCloseSessions: 9.301s; https://buildtest.kotlin.build/run?id=caf61fc1 main sample 1: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 43.793s; https://buildtest.kotlin.build/run?id=84732ebb main sample 1: testSessionLeaseAndStagingLimits: 6.436s; https://buildtest.kotlin.build/run?id=002f7737 main sample 2: testCloseWhileClientsOpenAndCloseSessions: 6.103s; https://buildtest.kotlin.build/run?id=b4680fa0 main sample 2: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 20.909s; https://buildtest.kotlin.build/run?id=db226531 main sample 2: testSessionLeaseAndStagingLimits: 10.658s; https://buildtest.kotlin.build/run?id=ebcd01dc main sample 3: testCloseWhileClientsOpenAndCloseSessions: 12.890s; https://buildtest.kotlin.build/run?id=ed62e004 main sample 3: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 32.207s; https://buildtest.kotlin.build/run?id=e09d97c1 main sample 3: testSessionLeaseAndStagingLimits: 11.409s; https://buildtest.kotlin.build/run?id=395f198a pr sample 1: testCloseWhileClientsOpenAndCloseSessions: 10.117s; https://buildtest.kotlin.build/run?id=015e026d pr sample 1: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 21.876s; https://buildtest.kotlin.build/run?id=5e8abfdb pr sample 1: testSessionLeaseAndStagingLimits: 8.802s; https://buildtest.kotlin.build/run?id=3481d980 pr sample 2: testCloseWhileClientsOpenAndCloseSessions: 7.354s; https://buildtest.kotlin.build/run?id=5d484afd pr sample 2: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 17.110s; https://buildtest.kotlin.build/run?id=4e7f00d0 pr sample 2: testSessionLeaseAndStagingLimits: 5.542s; https://buildtest.kotlin.build/run?id=27a2fcd7 pr sample 3: testCloseWhileClientsOpenAndCloseSessions: 18.617s; https://buildtest.kotlin.build/run?id=20112e97 pr sample 3: testEmbeddedUpgradeLosesNoLegacyFileWhenStorageStopsAtAnyOperationOfTheFirstMutation: 28.735s; https://buildtest.kotlin.build/run?id=8b221823 pr sample 3: testSessionLeaseAndStagingLimits: 9.806s; https://buildtest.kotlin.build/run?id=f6978639

Extra full-suite comparison: PR run https://buildtest.kotlin.build/run?id=e8c14a6b completed with 103/108 first-attempt passes and 108/108 final passes. Its raw journal has 119 completion records: 111 SUCCESS, 7 FAILURE, 1 CANCELED. Do not confuse completions with unique tests. First-attempt failures: large filesystem read (FakeFileSystem OpenFile null entry); session maintenance (workers miss deadline); pinned chunk-state coverage (30-second deadline); concurrent-close (25-second Future.get deadline); session limits (30-second deadline). Concurrent-close: first failure 28.699s, 11.720 process-tree CPU seconds, 62.888 summed runnable-wait seconds; attempt 2 SUCCESS 35.500s, attempt 3 SUCCESS 12.006s. Session limits: first failure 33.364s, 9.560 CPU seconds, 58.773 summed runnable-wait seconds; attempt 2 FAILURE 32.920s, attempt 3 SUCCESS 12.893s. Storage-stop sweep: first-attempt SUCCESS 138.554s on PR, 110.908s on main. Main run https://buildtest.kotlin.build/run?id=b2e94bd0 did not finish. Client exited ERROR after failing to confirm lease renewal for 903.227s against a 900s lease. Coordinator last reported 14/106; its journal has 18 successful first-attempt completions. Session limits and concurrent-close never completed in this full main run. All four main executors were idle with no test child; the primary runner stack awaited PreparedLeasePipeline input, and preparation workers awaited their work queue. Coordinator log stopped progressing at 15:29 UTC despite queued tests. A cancellation RPC timed out. After verifying each command line contained b2e94bd0 and the buildtest runner entry, the four owned idle executor JVMs were terminated by exact PID; all four were verified gone. A subsequent cancellation RPC returned: Cannot cancel build run b2e94bd0: a cancelled operation is already in progress. The coordinator still showed TESTING at the last snapshot; this is an unresolved cleanup/bookkeeping state, not running test code.

Mechanism evidence and limits: Original failures are preserved from the supplied Filedrop archive and independently from coordinator events. Their CPU times overlap passing targeted-run CPU ranges. The original dispatcher explicitly recorded that CPU admission would reject these cases on 2-vCPU droplets, then admitted them because CPU admission was OBSERVE. fitsCpu returns true after cpuFits rejects under OBSERVE. Source: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/5f720d2a10b9ad20bf4f1383d2574cf6a40abf6f/src/buildtest/embedded/LeaseScheduler.kt#L3528-L3552 The existing public-service test cpuAdmissionObserveModeRecordsAdvisoriesAndAdmitsLikeMain explicitly expects all four cases in one lease on 2000 millicores while recording three rejection advisories. It was read, not executed in this task. This identifies a concrete admission decision and measured queue delay. It does not by itself satisfy the requested diagnosis of an upgrade regression or removable test/product inefficiency. No code fix is claimed. Two target tests use the handler in process; only session-limits uses the resolver/network. There is no evidence of an 8-KiB pull regression here. Storage profile: 242/474 execution samples include namespace index loading/conversion, mostly the intended first-mutation migration/recovery repeated at each stop point; 108472 FakeFileSystem file-open backtrace Exception events. Concurrent-close profile: 40000 FakeFileSystem source-open backtraces and 40000 expected post-close IllegalStateExceptions; four stores, 10000 iterations per store. No avoidable work was proved. Local main uninstrumented close: 2.388s; session limits: 3.489s. Storage with sampling profile: 11.220s; with exception recording: 16.999s.

State: no code changes, no commits, no new PR, no merge, no queue entry, no deploy, no publication, no other model session. All four source checkouts were clean. PR is OPEN/CLEAN at the requested SHA; required bld-build was freshly verified green. The user forbids notes in repos. Investigation data is therefore preserved through Filedrop, not committed to a source branch. Filtered execution samples and summaries are included. Raw JFR files are excluded because their general-purpose recordings can contain the process environment; the useful method samples and statistics are exported separately without that environment. Remaining: obtain a functioning main full-suite baseline and a reliable public-path reproduction, then identify and fix the underlying inefficiency or exact upstream cause without changing deadlines, iterations, or assertions. Do not present CPU contention alone as the completed diagnosis.

Relevant PRs / refs

Repository	Branch	Verified remote head	PR	State
SimpleFileSystemServiceServer	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/tree/main	2f8d1553b5d9cd42a792ca5dab2274002fd763ea	none	Baseline; no changes by this effort
SimpleFileSystemServiceServer	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/tree/l134-resolver1337-libp2p37	60f7629a8018388672cdaa7c345807836d224094	https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/43	OPEN/CLEAN, required bld-build green; not merged or enqueued

All four fresh checkouts (server, its PR worktree, SimpleFileSystemServiceEmbedded, kompile-cli) were clean, with no stash or local-only commit. No source work is unpushed. This session made no publication or deployment. The user forbids notes in repositories, so evidence was backed up outside source repositories instead of creating a notes branch.

The comparison archive is https://filedrop.wasmserver.com/drop/99aa9020e1 (one-year expiry). It carries all eighteen targeted XMLs, first-attempt events and original deadline evidence. Final Filedrop upload failed with All 0 peers for service 'filedrop' failed to connect (direct and relay); the full final archive was therefore saved on the coordinator, outside every repository, with a verified SHA-256:

• Host: 198.199.106.165
• Path: /root/buildtest-data/investigation-evidence/L155/20261026T160919Z-final-deadline-evidence.tar.gz
• SHA-256: 1485d2ce5abf21877a8ff1600a313fb796257b23b0bffd1b78ca4107156c3d20
• Size: 649017 bytes

This archive contains the final report, all comparison results and attempt-start audits, original and new full-suite events, complete first-attempt error stacks, executor thread dump, ownership checks and stop confirmations, filtered profiling samples and summaries, required-check output, and the scheduler source observations. Use Fabric read-file when configured; the session's fallback was ssh -p 23 root@198.199.106.165 because the local Fabric client certificate file was absent. Raw JFR recordings were intentionally excluded because general-purpose recordings can include process environment; the useful execution samples and counts are exported without that environment.

Next steps

1. Re-verify current state first. Read the blocker and reconcile main run b2e94bd0's nonterminal cancellation record and owned droplet cleanup before trusting its status.
2. Get a functioning main full-suite baseline; do not interpret the interrupted main run or the PR's retried final XML as clean first-attempt evidence. Do not blindly requeue a stalled run.
3. Separate observation from inference. The exact OBSERVE admission branch and measured queue delays are facts; they have not established the requested removable test/product inefficiency. The profile's namespace conversion work is expected by the storage-stop scenario. There is no proof of an 8-KiB pull regression, and two of the three target tests use the RPC handler in process.
4. Build a reliable failing test against the narrow public path, under a deterministic triggering condition once understood. Never use artificial CPU load or machine-tuned amplification, and do not change deadlines, iterations, assertions, or the clocks' contracts.
5. If the upgrade is worse, fix at its cause in this repository or report the exact resolver/protocol line as the brief permits. If main fails equally and a test/product inefficiency is proved, use a separate branch from main with fail-first evidence. Independently review tests and code, run required verification, push each milestone and verify the remote SHA, and open/update a PR. No merge, enqueue, or production deployment.
6. Track the full-suite FakeFileSystem OpenFile null entry and the other first-attempt failures separately; do not silently call the full suite clean or claim those failures are explained by CPU measurements.

Complete first-attempt failure stacks from the additional PR full suite

testLargeFileReadOverUrlWithConcurrentSmallCalls

simplefilesystem.SimpleFileSystemException: Cannot invoke "okio.fakefilesystem.FakeFileSystem$OpenFile.getCanonicalPath()" because "it" is null
	at simplefilesystemservice.client.SimpleFileSystemClientKt.throwRemote(SimpleFileSystemClient.kt:477)
	at simplefilesystemservice.client.SimpleFileSystemClientKt.envelopeValue(SimpleFileSystemClient.kt:446)
	at simplefilesystemservice.client.SimpleFileSystemClientKt.value(SimpleFileSystemClient.kt:442)
	at simplefilesystemservice.client.SimpleFileSystemClientKt.access$value(SimpleFileSystemClient.kt:1)
	at simplefilesystemservice.client.RemoteSimpleFileSystem.readUtf8(SimpleFileSystemClient.kt:154)
	at simplefilesystemservice.server.tests.TestLargeFileReadOverUrlWithConcurrentSmallCallsKt.testLargeFileReadOverUrlWithConcurrentSmallCalls(testLargeFileReadOverUrlWithConcurrentSmallCalls.kt:111)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at community.kotlin.kompile.testrunner.bootstrap.BootstrapRunner.main(BootstrapRunner.java:449)

testSessionMaintenanceWhileClientsOpenAndClose

java.lang.AssertionError: Both clients must finish opening and closing 2,000 sessions.
	at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16)
	at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:652)
	at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11)
	at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:662)
	at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11)
	at kotlin.test.AssertionsKt__AssertionsKt.assertTrue(Assertions.kt:44)
	at kotlin.test.AssertionsKt.assertTrue(Unknown Source)
	at simplefilesystemservice.server.tests.TestSessionMaintenanceWhileClientsOpenAndCloseKt.testSessionMaintenanceWhileClientsOpenAndClose(testSessionMaintenanceWhileClientsOpenAndClose.kt:82)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at community.kotlin.kompile.testrunner.bootstrap.BootstrapRunner.main(BootstrapRunner.java:449)

testPinnedReadChunkStateRejectsIncompleteTransport

java.lang.RuntimeException: Test 'testPinnedReadChunkStateRejectsIncompleteTransport' timed out after 30000ms, stuck at: net.javadeploy.sjvm.helper.AvianStdLibHelper.registerNativeOverrides(AvianStdLibHelper.kt:311)
	at kompile.WorkspaceKt.executeTestInSeparateJvm(Workspace.kt:4748)
	at kompile.WorkspaceKt.access$executeTestInSeparateJvm(Workspace.kt:1)
	at kompile.WorkspaceKt$prepareTests$4$executeAction$1.invoke(Workspace.kt:6740)
	at kompile.WorkspaceKt$prepareTests$4$executeAction$1.invoke(Workspace.kt:6725)
	at kompile.PreparedTestRun.execute(Workspace.kt:4396)
	at kompile.LocalBuildWorkspace$prepareTestBatchInternalOpen$executePrepared$1.invoke(Workspace.kt:1981)
	at kompile.LocalBuildWorkspace$prepareTestBatchInternalOpen$executePrepared$1.invoke(Workspace.kt:1961)
	at kompile.PreparedTestBatch.executeOnce(Workspace.kt:348)
	at kompile.PreparedTestBatch.execute(Workspace.kt:304)
	at buildtest.runner.PreparedLeaseBatch.execute(PreparedLeaseBatch.kt:22)
	at buildtest.runner.MainKt.runOneExecution(Main.kt:1774)
	at buildtest.runner.MainKt.runOneExecution$default(Main.kt:1149)
	at buildtest.runner.MainKt$preparePipelineWork$warmEntry$1.invoke(Main.kt:1087)
	at buildtest.runner.MainKt$preparePipelineWork$warmEntry$1.invoke(Main.kt:1086)
	at buildtest.runner.PreparedLaneExecution.thread$lambda$0(PreparedLaneExecution.kt:129)
	at java.base/java.lang.Thread.run(Thread.java:1583)

testCloseWhileClientsOpenAndCloseSessions

java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
	at simplefilesystemservice.server.tests.TestCloseWhileClientsOpenAndCloseSessionsKt.testCloseWhileClientsOpenAndCloseSessions(testCloseWhileClientsOpenAndCloseSessions.kt:128)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at community.kotlin.kompile.testrunner.bootstrap.BootstrapRunner.main(BootstrapRunner.java:449)

testSessionLeaseAndStagingLimits

java.lang.RuntimeException: Test 'testSessionLeaseAndStagingLimits' timed out after 30000ms, stuck at: net.javadeploy.sjvm.util.Sha256PlatformKt.sha256HexAccelerated(Sha256Platform.kt:4)
	at kompile.WorkspaceKt.executeTestInSeparateJvm(Workspace.kt:4748)
	at kompile.WorkspaceKt.access$executeTestInSeparateJvm(Workspace.kt:1)
	at kompile.WorkspaceKt$prepareTests$4$executeAction$1.invoke(Workspace.kt:6740)
	at kompile.WorkspaceKt$prepareTests$4$executeAction$1.invoke(Workspace.kt:6725)
	at kompile.PreparedTestRun.execute(Workspace.kt:4396)
	at kompile.LocalBuildWorkspace$prepareTestBatchInternalOpen$executePrepared$1.invoke(Workspace.kt:1981)
	at kompile.LocalBuildWorkspace$prepareTestBatchInternalOpen$executePrepared$1.invoke(Workspace.kt:1961)
	at kompile.PreparedTestBatch.executeOnce(Workspace.kt:348)
	at kompile.PreparedTestBatch.execute(Workspace.kt:304)
	at buildtest.runner.PreparedLeaseBatch.execute(PreparedLeaseBatch.kt:22)
	at buildtest.runner.MainKt.runOneExecution(Main.kt:1774)
	at buildtest.runner.MainKt.runOneExecution$default(Main.kt:1149)
	at buildtest.runner.MainKt$preparePipelineWork$warmEntry$1.invoke(Main.kt:1087)
	at buildtest.runner.MainKt$preparePipelineWork$warmEntry$1.invoke(Main.kt:1086)
	at buildtest.runner.PreparedLaneExecution.thread$lambda$0(PreparedLaneExecution.kt:129)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Reusable operational knowledge

Remote XML currently omits attempt history. Read test-events.jsonl, deduplicate by project/name/attempt, and inspect starts as well as completions. The eighteen targeted journals each have one start and one successful completion; there are no hidden retries in those samples. Full-run raw completion counts include retries/cancellations and must not be treated as unique test counts.

A local plain-Java helper used the public kompile.cli.UrlProtocolBuildService API to read the service, without reflection. An attempted cancellation timed out; after owned executor termination a later call reported Cannot cancel build run 'b2e94bd0': a cancelled operation is already in progress. Main's client had already exited ERROR after 903227ms of unconfirmed renewal against a 900000ms lease. The four exact run-owned JVM PIDs were checked for the run ID before termination and verified gone; no other run or production process was killed.

No model sessions were delegated. No active goal or recurring timer was created. Local test clients and watchers have ended; failed evidence-upload clients were bounded by timeouts. Re-check live ownership and status before any action.

Final live-state update

Main run b2e94bd0 is now CANCELED, with completedAt 1793030820996 and 19/106 reported passes. Its earlier nonterminal cancellation is historical. All four owned runner JVMs were verified stopped. The reason work stopped being delivered while queued tests remained is not established; this is still an incomplete baseline. The final archive, including this update, is on 198.199.106.165 at /root/buildtest-data/investigation-evidence/L155/20261026T160919Z-final-deadline-evidence.tar.gz, verified SHA-256 1485d2ce5abf21877a8ff1600a313fb796257b23b0bffd1b78ca4107156c3d20. Its path supersedes the earlier archive snapshot.