Handoff: SimpleFileSystemDurableEmbedded Round 3 measured memory, but the required cgroup rig is unavailable

Written 2026-07-28 UTC.

RE-VERIFY: This is a snapshot verified at 2026-07-28 06:22:00 UTC. Before acting, re-run gh pr view 3 --repo CodexCoder21Organization/SimpleFileSystemDurableEmbedded --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefName,headRefOid; confirm the remote branch with git ls-remote origin refs/heads/fix/prestart-shared-cockroach-fixture; and inspect the execution host's cgroup mount, controllers, memory, and swap. Fresh live evidence supersedes every state claim below.

Mission summary

The user requested a third investigation round on SimpleFileSystemDurableEmbedded, with memory as a strict test-before-fix gate:

1. Measure process-level peak RSS for the unthrottled four-way 60-test dispatch at 9f60dc5cb60cfbf9a7f887b299041c67c693ed0a, including Kompile, every test/helper JVM, CockroachDB, and its in-memory store.
2. Reproduce the CI failure roster inside one cgroup v2 constrained to memory.max=3800M, memory.swap.max=0, two CPUs, and moderate steal spinners.
3. Only if that rig reproduces, reduce helper/infrastructure memory without changing any test workload, iteration count, assertion, or timeout.
4. Require two consecutive 60/60 capped runs plus one capped run at 12% CPU duty before one push and one CI observation. Do not merge; stop on a red CI result.
5. If the capped rig passes, stop and report measurements rather than guess-fix.

What was found and done

1. Live branch and CI were re-verified. At write time, https://github.com/CodexCoder21Organization/SimpleFileSystemDurableEmbedded/pull/3 was OPEN, merge state BLOCKED, branch fix/prestart-shared-cockroach-fixture, remote head 9f60dc5cb60cfbf9a7f887b299041c67c693ed0a. Its only check was completed FAILURE at https://buildtest.kotlin.build/run?id=a23b5fbc with 43 passed and 17 failed.

2. A fresh checkout was used and rebased without changing the head. The checkout was /code/codex-lanes/prestart/SimpleFileSystemDurableEmbedded/workspace/SimpleFileSystemDurableEmbedded-round3. git fetch origin && git rebase origin/main reported it was up to date and left the SHA at 9f60dc5c. The checkout is clean, has no stash, has no local-only commits, and the local SHA matches the remote branch SHA. No source or test file was changed.

3. The exact four-way dispatch was measured uncapped. The run invoked Kompile directly with the directory selector, not the repository wrapper's sequential batches:

   jars/KompileCli.jar \
     --cache-location /home/leslie/.aibuildcaches/_code_codex-lanes_prestart_SimpleFileSystemDurableEmbedded_workspace_SimpleFileSystemDurableEmbedded-round3 \
     -w /code/codex-lanes/prestart/SimpleFileSystemDurableEmbedded/workspace/SimpleFileSystemDurableEmbedded-round3 \
     --test . \
     --log /tmp/round3-current-unthrottled/tests.xml
   

   A 50 ms /proc sampler followed the whole descendant tree, including detached fixture process groups, and retained each PID/start-time pair's VmHWM, sampled anonymous/file RSS, and swap. The run passed 60/60. Peak concurrent RSS was 3,011,252 KiB (2.871 GiB); concurrent anonymous RSS was 2,320,740 KiB (2.213 GiB); every sampled process reported zero swap.

4. The footprint is close enough to a 4 GB host ceiling that the memory hypothesis remains plausible, but it is not proven. The long-lived Kompile JVM peaked at 723,168 KiB RSS (706.2 MiB), the primary CockroachDB process at 768,484 KiB (750.5 MiB), and the primary fixture daemon at 101,400 KiB (99.0 MiB). Across the run, 68 test-runner JVM invocations (60 top-level tests plus scenario-owned nested probes) peaked between 37.5 and 167.0 MiB each. At the peak instant, the primary CockroachDB node, two scenario-owned CockroachDB nodes, the Kompile runner, four top-level test workers, one nested test runner, four fixture/probe JVMs, and the primary fixture daemon were live. This leaves about 650 MiB below the user's estimated 3.5 GiB usable ceiling before cgroup-charged unmapped page cache and kernel memory, but aggregate RSS also double-counts shared file-backed pages; only the requested cgroup experiment can settle the net effect.

5. CockroachDB store data was measured during the live run. The primary process command line configured --store=type=mem,size=640MiB, --cache=64MiB, --max-sql-memory=128MiB, --max-tsdb-memory=32MiB, --max-disk-temp-storage=128MiB, and --max-go-memory=512MiB. crdb_internal.kv_store_status reported capacity 671,088,640 bytes, available 671,088,640 bytes, and used 0 bytes; SHOW DATABASES counted 19 databases (the system/default databases plus the 16 schema-ready fixture databases). Cockroach's heap profile during warmup showed HeapAlloc=98,854,808, HeapInuse=156,114,944, and Sys=310,764,872 bytes, while process RSS continued climbing and ultimately reached 750.5 MiB. The SQL used=0 counter therefore does not describe the process's actual anonymous footprint for this in-memory fixture.

6. The required cgroup rig cannot be created from this execution environment. Cgroup v2 is mounted as cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime), and mkdir /sys/fs/cgroup/round3-probe fails with Read-only file system. There is no writable delegated subtree, systemd-run, Docker, Podman, nerdctl, sudo, or container socket. unshare -UrC true fails with unshare: unshare failed: Operation not permitted. Per-process address-space limits and a global memory hog were rejected as substitutes because they do not reproduce aggregate cgroup reclaim/page-cache behavior and the latter could disrupt unrelated workloads. This infrastructure gap is recorded at https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0607-read-only-cgroup-namespace-blocks-memory-faithful-local-ci.md.

7. The investigation stopped at the mandatory gate. There was no capped reproduction, no footprint fix, no verification run, no new commit, no push, and no new CI run. The memory mechanism remains a hypothesis rather than an identified root cause.

Measurement artifacts

• Every observed process's normalized peak RSS/anonymous/file/swap values
• All processes live at the 2.871 GiB peak instant
• 50 ms aggregate RSS timeline
• Passing 60-test XML report

The raw 318 MiB per-process sample stream was intentionally not committed; it is a disposable measurement output, while the complete per-process high-water table and aggregate timeline retain the evidence needed to continue.

Relevant PRs and refs

| Reference | Write-time state | |---|---| | https://github.com/CodexCoder21Organization/SimpleFileSystemDurableEmbedded/pull/3 | OPEN, merge state BLOCKED, CI FAILURE at run a23b5fbc, head 9f60dc5cb60cfbf9a7f887b299041c67c693ed0a | | Remote branch fix/prestart-shared-cockroach-fixture | Remote head 9f60dc5cb60cfbf9a7f887b299041c67c693ed0a; no Round 3 source changes | | Failed CI run a23b5fbc | Completed FAILURE, 43 passed / 17 failed / 60 total | | Cgroup-environment challenge | Merged record of the exact local blocker |

Nothing is unsaved in the implementation checkout. The only new work is this handoff and its measurement summaries.

Next steps

1. Re-verify current state first using the commands in the banner. Do not assume the branch, PR check, or cgroup environment is unchanged.
2. Obtain an execution boundary with a writable cgroup v2 subtree (or an already-authorized disposable 4 GB runner). Put the entire Kompile dispatch and all detached descendants into the same cgroup before they spawn; set memory.max=3800M and memory.swap.max=0; retain the prior two-CPU affinity and moderate steal-spinner duty. Confirm the primary fixture daemon and every CockroachDB/helper PID remain in that cgroup.
3. Run the unchanged head's direct --test . dispatch and record the 60-test tally and failure roster. If it passes under the cap, stop and report it exactly as the user requested—do not change code.
4. Only if it reproduces near the CI roster, read the authoritative testing architecture before touching tests, then add a reliably failing infrastructure-level reproducer. Reduce only fixture/helper footprint: candidates the user authorized are a smaller or lazy schema-ready database pool, a lower bounded in-memory Cockroach store, explicit helper JVM heap caps, and removal of duplicate long-lived JVMs. Do not change test iterations, assertions, workloads, or timeouts.
5. Measure each change independently against the same rig and report before/after concurrent aggregate RSS plus affected process high-water RSS. Retain only changes supported by the reproducer.
6. Require two consecutive 60/60 runs inside the memory cap and one additional 60/60 run at 12% CPU duty with the cap. Then follow the repository's rebase/test/review process, push once, and use the build watcher for exactly one CI result. Stop on red and do not merge.
7. Delete this handoff through the pre-approved handoff deletion workflow only after the requested capped reproduction/fix/verification/CI sequence is genuinely complete.

Reusable and operational knowledge

• scripts/test.bash --test . is not the CI-equivalent four-way dispatch on this branch: it expands the full suite into sequential explicit-file batches. Use the Kompile launcher directly with --test . when reproducing BuildTest dispatch.
• The workspace-specific fixture namespace is the first 16 hex characters of SHA-256 over the canonical workspace path without a newline, under /tmp/simplefilesystem-durable-shared-cockroach-v2-<hash>.
• Detached fixture daemons become children of PID 1. A process-tree sampler must discover and retain them before reparenting, or—preferably—the cgroup itself should be the authoritative process inventory.
• PID alone is not a stable measurement key. Pair PID with /proc/<pid>/stat start ticks.
• Aggregate simultaneous RSS is meaningful; summing each process's lifetime high-water RSS is not, because those peaks occur at different times. The committed artifact contains both the per-process high-water values and the actual concurrent peak snapshot.
• The fresh checkout needed a working arm64 Coursier copied from the prior workspace because the system Coursier is the wrong architecture; this did not change tracked repository state.