Finish the kompile temp-directory leak fixes and apply the 25 GB per-container Docker limit on helena-cooper

G3 build-kotlin-jvm runner verification — 2026-10-28 00:35 UTC

RE-VERIFY (G3 only): This section supersedes G2's missing directory/full-suite evidence. Recheck git ls-remote for the linked branch and run https://github.com/CodexCoder21Organization/build-kotlin-jvm/actions/runs/40607379582 . Other workstreams below retain their own dated snapshots.

Design decisions in force

Original request: "Fix the kompile temp dir leak upstream." G3's brief says: "Not yours: merging, enqueueing, marking ready, publishing, deploying, editing other repositories." It also says: "If a requirement is impossible, stop and say so with evidence — a complete result." Those constraints are why G3 stopped instead of attempting a receiving-runtime change or opening a draft described as fully verified.

What was verified

G3 added a temporary push-triggered workflow with separate fixed, reverted, branch-full-suite, and main-full-suite jobs. All executed on GitHub Actions runners with JDK 21 and CLI 0.0.91 pinned by the tracked wrapper. No host build/test was run and no test or production source was edited. The workflow was removed after evidence capture.

• Targeted fixed: 4 passed / 0 failed, first attempt. Both directory consumers completed in 9–10 seconds.
• Targeted reverted: 0 passed / 4 failed, each by its intended assertion: NotSerializableException for Manifest/MavenPrebuilt; FileNotFoundException writing the POM for both metadata-only builders. Only the four production fix lines were reversed.
• Branch full suite: 274 passed / 2 failed / 276 total.
• Main full suite at ff0f9ec2aaec66c18ad4fcd7e76ef11ce0dced55: 269 passed / 0 failed / 269 total.
• All four jobs are terminal in Actions run attempt 1. No manual reruns. The two branch failures are newly introduced tests absent from main, so do not call them pre-existing main failures.

Remaining failure and evidence

testPublishedBuildJarTransportsManifest and testPublishedBuildKotlinTransportsDeclarations now pass child-side serialization but fail their consumer exit-code assertions with receiving-side InvalidClassException: build.kotlin.jvm.Manifest; class invalid for deserialization and the matching MavenPrebuilt error. This is beyond the previous host deadline exits: real runner assertions now identify the next boundary.

Observed mechanism: CLI 0.0.91 POM pins build-kotlin-jvm 0.0.23 and executionenvironment 0.0.35. Bytecode inspection of rules 0.0.23 shows both records lack Serializable. The receiving BootstrapProtocolObjectInputStream special-cases only TcpMessage/SerializableFailure and delegates these argument class names to the ordinary parent resolver. TcpCommunicationHandler reads/deserializes the object before handling the message, so the parent loads the older class and rejects it before published-rule conversion. The third new published-artifact consumer test passed; it does not negate these two receiving-path failures.

The four edits therefore prove local value serialization and directory behavior, but are insufficient for the pinned older-parent/current-child transport path. No runtime fix, wrapper version change, publication, deployment, merge or enqueue was attempted. Runtime/CLI compatibility needs an authorized owner before these two full-path tests can become green. Do not weaken/delete them, raise their budgets, or retry unchanged.

Relevant branch and evidence

Evidence: verification and requirement audit, complete targeted assertions/XML and reverted diff, suite counts, complete remaining stacks and receiving stream bytecode. The linked run artifacts retain raw logs/XML and exact source heads. No build outputs, caches, or downloaded raw artifacts were committed as source.

Nothing was published or deployed in G3, and no PR was opened. The tracked Actions watcher exited on the terminal failed run; no owned build/test/watch process remains. Remote and local head match; no stash or unpushed commit. The unchanged source/tests/build/wrapper trees match the runner commit a00ed324f814924e15c59f1208991d47b93f1793.

Next steps

1. Re-verify live branch/PR/runner state, then decide and authorize the receiving-runtime compatibility work across the older parent and current published argument types.
2. Preserve the two failing public CLI tests as the consumer reproducer. Reproduce at the owning runtime layer before implementing any change; then verify fixed and reverted behavior without relaxing assertions or limits.
3. Re-run the full branch suite from the final code, obtain first-attempt counts, and only then open the requested draft with the original request, blocked CLI branch, challenge links and all runner evidence. No merge, ready transition, enqueue, publish or deployment is authorized by G3.

REST quota initially rejected lookups until approximately 00:23 UTC; GraphQL status reads worked. Wait for reset after rejection instead of looping. All runner monitoring used at least two-minute intervals; once REST recovered, a bounded gh run watch --interval 120 followed the run to completion.

B4 kompile-core bounded checkpoint — 2026-10-28 01:26 UTC

B4 stopped within its explicit120-minute limit. Not DONE: exact final pushed-head full-suite gate is unfinished. Draft target https://github.com/CodexCoder21Organization/kompile-core/pull/444 is OPEN/isDraft=true. Final source https://github.com/CodexCoder21Organization/kompile-core/commit/7f181842b1fcf9c623d7ad5a3353b58e2612cf12 is pushed; worktree restored byte-for-byte and no code is uncommitted. No merge, queue, ready transition, publication, deploy or other-repository edit. Nothing from B4 was deployed or published.

Durable branches:

• Source https://github.com/CodexCoder21Organization/kompile-core/tree/fix/tmp-dir-leak-tests
• Final proof https://github.com/CodexCoder21Organization/kompile-core/tree/notes/b4-final-retention-2026-10-28 (latest notes commit758356e10; green/reverted-red traces, brief-state.md, materialization review and partial suite records).
• Earlier four source mutation records https://github.com/CodexCoder21Organization/kompile-core/tree/notes/b4-validation-2026-10-28

Completed: group/session wrapper containment, bounded TERM/KILL and root retention; eight focused wrapper greens plus startup/TERM-boundary and removed-signal reds. Resolve-owned synthetic Maven stub has six green success/error/suppression cases, genuine baseline red and identical-harness resolver-revert red. All eight local lifecycle invocations and six fixed/main/source-mutation controls completed at637; nested exceptional scenario passes main as well, explicitly refuting its claimed old-regression role. Allocation audit gaps are listed with reasons per brief, not swept broadly. Late callback recreation remains explicitly excluded as a separate launcher follow-up.

Full-suite comparison exposed a real retained-path regression: deleting the first result then resolving recreated the same path. Main passes its existing test. New public close-lifetime test reproduces by assertion; final repair passes it and the unchanged existing CI-failing test. Independent review rejected per-call producer generations because identical outputs accumulated copies; corrected public File-input test proves two rule executions and identical bytes then fails on different random producer parents. Final source uses a content/layout-keyed producer-classpaths namespace separate from stable consumer restoration, indexes exact retained paths after copying, and preserves preparation behavior. All three focused final retention tests pass first attempt. Both repairs have assertion red/green/reverted-red. Resolver source restored after both revert runs.

CI/evidence: final exact-head fullsuite https://github.com/CodexCoder21Organization/kompile-core/actions/runs/40614222439 and fat-JAR run https://github.com/CodexCoder21Organization/kompile-core/actions/runs/40614222400 were pending at last check. Final first-attempt counts are NOT established. Older637 suite https://github.com/CodexCoder21Organization/kompile-core/actions/runs/40610072487 has shards2/3/4 collected:1,095 first passes/1first failure,1,096identities; missing shard1 means wrapper full-run outcome still unknown. Its only available failure is the retained-path assertion, which main passes and final source now passes locally. Comparison2bf run https://github.com/CodexCoder21Organization/kompile-core/actions/runs/40607556285 has1461firstpass/3fail,1462finalpass/2fail; older wrapper predates final fixes, admission-drain also first-fails on main. Main run https://github.com/CodexCoder21Organization/kompile-core/actions/runs/40585381738 has1461firstpass/1fail,1462finalpass. Full traces and per-failure comparisons are in notes. No unchanged CI job was requeued; obsolete intermediate runs cancelled, final7f and637 comparison preserved.

Resume: verify remote source/evidence tips; collect exact7f full-suite XML and quote first-attempt counts using notes/.../summarize-suite.py; compare each failure with the main reference. Read637 shard1 result for final wrapper behavior. If any failure remains, reproduce the exact public path locally and fix test-first; never raise timeouts, reduce iterations or use CI retries as diagnosis. The final7f default all-eight lifecycle invocation was not rerun (those records are637); final full wrapper/lifecycle suite gate remains required. Update the accurate draft PR body and evidence; do not merge/enqueue/mark ready/publish/deploy. Brief checklist and exact trace paths are in the final proof branch. Owned local probes/watcher ended; no B4 agent remains running. Preserve all unrelated handoff sections.

G2 build-kotlin-jvm checkpoint — 2026-10-27 23:59 UTC

RE-VERIFY (G2 only): branch and remote equal 87509142de45d6f3330abbed91e5204d9dd355a4; no PR exists for this branch. The separate CI migration https://github.com/CodexCoder21Organization/build-kotlin-jvm/pull/75 is OPEN at 4a19aaa169a89a36577f11aebe862397c407706c. Other workstreams retain their own dated snapshots below.

The lost fix is now pushed: Manifest and MavenPrebuilt implement Serializable, and both Maven builders call output.mkdirs() before writing the POM. The two current-source serialization tests have complete red → green → reverted-fix red proof (0/2, 2/2 first attempt, 0/2). Both metadata-only consumers have assertion reds with FileNotFoundException before their respective fix; their fixed-source runs reached operator deadlines without normal XML results, so directory green and reversion remain unverified. The three original publisher regressions stopped in bootstrap compilation before target behavior. A full suite was attempted once in race mode from pushed source commit 4c90ee868f9e1c10f23062f8e1803d4d3de62e3f, reached its 660-second bound, and emitted only interrupted placeholder counts; first-attempt counts are unknown. No failure is called pre-existing without a main comparison.

All source, tests and evidence are durable at https://github.com/CodexCoder21Organization/build-kotlin-jvm/tree/fix/manifest-serializable-rule-params. Read https://github.com/CodexCoder21Organization/build-kotlin-jvm/blob/fix/manifest-serializable-rule-params/notes/manifest-rule-transport/final-verification.md first, then the audit, exact error stacks and draft PR provenance linked beside it. This lane made no merge, enqueue, publication, deployment or other-repository edit. All its build/test processes stopped and both checkouts are clean; the generated ignored outer launchers were removed after validation. No PR was opened because the prerequisite validation is incomplete.

Resume by resolving publisher preparation without increasing limits, establishing normal directory green and source-reverted red results, then completing the full suite with first-attempt counts and main evidence for any existing-failure claim. Only then open the draft PR, preserving the original request and CLI/challenge links in notes/manifest-rule-transport/draft-pr.md. Source compiles against buildscript 0.0.33; copying an ignored newer CLI into the fixture exposed NoSuchMethodError against 0.0.44, so read runner-version-failure.md before changing the wrapper pin. No merge, publish or deploy was authorized for G2.

Written 2026-10-27 22:30 UTC. RE-VERIFY: this is a write-time snapshot. Re-check:

• PR state with gh pr view <url> --json state,headRefOid,mergeStateStatus;
• branch heads with gh api repos/CodexCoder21Organization/<repo>/branches/<branch>;
• the host with ssh root@64.225.115.174 'docker info | grep -E "Storage Driver|Backing"; cat /root/docker-switch-2026-10-27.log'.

B3 stop condition — 2026-10-27 23:18 UTC

RE-VERIFY (B3 only): https://github.com/CodexCoder21Organization/kompile-core/pull/444 is OPEN, draft and unmerged. Local and remote fix/tmp-dir-leak-tests equal 326b112d37cde8b2ce9f971986f2a78a70c98486. Other workstreams below retain their earlier snapshot; B3 did not re-verify or modify Docker, build-kotlin-jvm or kompile-cli.

The current brief explicitly excludes the late launcher callback that recreates output after cancellation; it is planned as a separate PR. The old callback-drain stop condition below is superseded for B3 only. Existing diagnostic notes remain; no timer or deferred deletion was added.

B3 reached a different stop condition. A real nested JVM remains alive after its parent exits 23, but the parent's ProcessHandle is unavailable, and the wrapper has returned and deleted its root. The deliberate assertion-red experiment took 8.687s. Teardown terminated the exact child and verified neither PID remained. ProcessHandle.descendants() is a snapshot and cannot recover the former relationship after reparenting. This refutes the ProcessHandle-only broad failure-tree guarantee; it does not claim OS process containment is impossible. A stronger ownership/registration contract or an explicit live-parent scope must be decided and tested. No sampling workaround was added.

• Source, full assertion and teardown evidence.
• Complete state of every brief item and resume steps.
• Wrapper red/green/mutation evidence.
• All allocation sites, owners and further gaps/reasons.
• Independent test review.

The wrapper fix owns root allocation through command/start failure, stops captured live descendants and waits before root deletion, retains the root when exit cannot be established, and preserves original/suppressed failures. Five focused helper checks pass. Restoring the original lifecycle or unconditional root deletion fails by assertion. The final helper source is unchanged from the green source at https://github.com/CodexCoder21Organization/kompile-core/commit/9b859da82 . These are focused helper results, not a full suite result.

The remaining core regressions are drafted and pushed: synthetic stub success/generation/fetch failure, both scanner families, immediate staging checks, cancellation/timeout/nonzero/overlap/nested-error cases, and combined body/cleanup errors and duration. Their exact final runtime and required mutations have NOT run. The production synthetic-stub ownership fix is still open. The audit covers 25 direct calls and three indirect references, with reasons and further gaps; no blanket all-allocations-clean claim.

Two race runtime builds and a manager-only race build each exited 124 at the unchanged external 600s command bound, with no artifact or runner record. Remote emitted ConnectionClosedEffect; local stacks showed manager compilation. Current first-attempt full-suite pass/fail counts and fresh main comparisons are unavailable. Historical 355/7 counts are not B3 results. No CI job was requeued.

Deployed/published by B3: nothing. No merge, enqueue, ready, publication or deployment command was issued. Source/text evidence is pushed; compiled classes, caches and binaries are intentionally excluded. The working tree and stash are clean; all three workers finished and no task-owned runner or fixture remains.

Next: decide/test process containment or clarify the failure-tree contract; obtain a current-source runtime, finish every remaining assertion-red/green/reverted-fix control, then rebase and run the full race-mode suite on the exact pushed head and compare each failure against main. Keep the PR draft until complete.

Design decisions in force

• 2026-10-27, owner: "Fix the kompile temp dir leak upstream, delegate your work to gpt 6.1 sol".
• 2026-10-27, owner: "each of these docker containers should have a 25gb disk limit such that no one container grows rapidly and unbounded and creates an issue of disk space for the other containers".
• 2026-10-27, owner: "Let's go with (B) free XFS filesystem stored in a file on the existing disk."
• 2026-10-27, owner: "Resume your work. I did restart the containers, yes apply the 25gb limit".
  • The switch was not yet run: when the script was ready, new agent sessions were already running and the restart would end them. The owner chooses the moment.
• 2026-10-27, owner (CI): when the remote build service blocks a PR, race CI on GitHub runners per https://my.externalbrain.wasmserver.com/doc/github-builders-racing-remote-ci.md.
  • Use --local-and-remote for builds and tests.
  • kotlin.build (remote) is informational in repos that require bld-build / bld-all-tests.
• Shared build cache (~/.aibuildcaches): split into its own design handoff: url://handoff/handoffs/hf-2026-10-27-design-a-build-cache-shared-between-kompile-checkouts-aibuildcaches-design-only-for-fable.

What was found

On 2026-10-27, helena-cooper's 309 GB disk filled twice. About 180 GB was per-run temp directories under each agent container's /tmp. Kompile builds and tests created them and never deleted them: resolveDependencies-artifact-* (up to ~200 MB each), test-scratch* (30,443 in one container), qtcompile*, qtbuild*, launcher dirs, and others. Two rounds of manual cleanup freed 68 GB and then ~80 GB, including orphaned ~/.aibuildcaches.

Leak sources, measured with a private java.io.tmpdir against the real scripts/test.bash / scripts/build.bash entry points:

• JVM test launcher (community.kotlin.kompile.testrunner.jvm): already fixed on main by https://github.com/CodexCoder21Organization/community.kotlin.kompile.testrunner.jvm/pull/57 and published (0.0.35).
• Build rules (build-kotlin-jvm): resolveDependencies-* and cs-urlhandler-* are already fixed on main and published (0.0.36). One build with the old 0.0.23 rules left 106 resolveDependencies-artifact dirs.
• kompile-core: a real remaining leak of test-scratch* and qtbuild*, being fixed in https://github.com/CodexCoder21Organization/kompile-core/pull/444. Reviews of head c2b2589 also found:
  • (a) a cancelled test's late build callback can recreate an already-cleaned output dir;
  • (b) workspace-stub-jar from file-level @MavenArtifactCoordinates (WorkspaceCoordinateResolver.kt ~288) has no owner;
  • (c) the PR's own test wrapper kills only its immediate child on timeout, and leaks its root if process start fails;
  • plus six test-coverage gaps: stub-jar not scanned, cancellation after allocation, nested qtbuild exception path, per-execute (not just end-of-workspace) qtbuild assertions, combined test failure plus cleanup failure, and two overlapping execute calls.
  • Its GitHub bld-all-tests shards failed (1 and 4 failed; 2 and 3 cancelled) and were not yet diagnosed.
• Propagation is the main gap. kompile-cli 0.0.126 (latest) still loads launcher 0.0.29, manager 0.0.210 and rules 0.0.23. Most repositories pin older CLIs: 435 pin 0.0.65, 42 pin 0.0.124, and 8 pin 0.0.126.
• Upgrading the CLI to build rules 0.0.36 breaks the CLI's own fat-jar build with NotSerializableException: build.kotlin.jvm.Manifest. Kompile-executionenvironment's BuildscriptRunner serializes rule arguments, so Manifest and MavenPrebuilt must be Serializable.
• Version skew: manager 0.0.212 embeds bootstrap runner 0.0.16, while launcher 0.0.35 declares bootstrap 0.0.18.

The ownership rules the kompile-core fix must satisfy (written for the second round):

• Each per-attempt dir is owned by the attempt arena. Close stops new registrations, waits for in-flight holders through existing cancellation, then deletes. A late registration deletes its own output and fails.
• workspace-stub-jar is owned by the resolve call and deleted when it returns or throws, after Coursier copied it.
• qtbuild staging is gone right after each execute.
• Never delete by prefix or age.
• The test wrapper owns its root and the whole process tree inside a finally.
• The original failure stays primary, with the cleanup failure suppressed. There is exactly one terminal result per attempt, and durations are measured after cleanup.
• No new or raised timeouts.

Relevant PRs / branches

Nothing from this effort is merged, published, or deployed.

Docker 25 GB limit (helena-cooper, 64.225.115.174, root over SSH)

• Prepared:
  • /var/lib/docker-xfs.img (32 GiB, sparse) is mounted at /var/lib/docker-xfs as XFS with prjquota; project quota accounting and enforcement are ON; it has an fstab entry;
  • /etc/docker/daemon.json.staged sets data-root /var/lib/docker-xfs/docker, overlay2, overlay2.size=25G, and containerd-snapshotter: false;
  • there is no live /etc/docker/daemon.json.
• Background: Docker 29.1.3 currently uses the containerd overlayfs snapshotter on ext4, where --storage-opt size is silently ignored (verified: a container wrote 2 GB under a 1 GB cap).
• The switch script /root/docker-switch-2026-10-27.sh is ready and syntax-checked. It:
  1. grows the image to 230G (losetup -c, xfs_growfs);
  2. docker saves all images to /var/lib/docker-xfs/images-backup-2026-10-27.tar; images include claude-code:latest and aicli-guest-codex images used by the AiCli supervisor running on the host;
  3. installs the staged config and restarts docker (this ends every container);
  4. verifies overlay2 on xfs;
  5. reloads the images;
  6. proves the limit: the default root is about 25G, a 500 MB write under a 1G cap succeeds, and a 2 GB write fails;
  7. rolls back automatically (removes daemon.json, restarts) on any failed check.
• Run it detached on the host, at a moment the owner chooses: ssh root@64.225.115.174 'setsid nohup /root/docker-switch-2026-10-27.sh >/dev/null 2>&1 &'. Then read /root/docker-switch-2026-10-27.log.
• Afterwards: once healthy, reclaim the old /var/lib/containerd and /var/lib/docker data. Bind-mounted host folders (e.g. ~/docker-claude-code/_.codex, 37 GB of codex session logs) are not covered by the limit and need their own retention rule.

Next steps

1. When the owner says the host is clear, run the switch script and verify its log.
2. Second fix round on https://github.com/CodexCoder21Organization/kompile-core/pull/444, against the ownership rules above:
   • diagnose the failed bld-all-tests run against main first;
   • test-first for each finding, with red, green and red-again-with-fix-reverted evidence;
   • full suite with --local-and-remote;
   • then fresh-reader reviews of the new head.
3. Resolve the receiving-runtime/current-rule compatibility issue described in the G3 checkpoint, then complete the full branch suite and open the draft PR. G3 proved directory and value fixes but found two receiving-path failures; main passed. Runtime changes and publication need scope beyond G3.
4. Finish the kompile-cli adoption (bump to the newly published rules), and publish a kompile-core release with the PR 444 fix plus the matching embedded bootstrap.
5. Roll the new kompile-cli out to repositories (the kompile-cli-bump skill).