Handoff: Find and correct the startup memory increase before updating the PR

Written 2026-10-21T19:02:59.214885+00:00.

RE-VERIFY: This is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/58 with gh pr view 58 --repo CodexCoder21Organization/ExternalBrainWui --json state,headRefOid,mergedAt,statusCheckRollup, and verify remote branch SHAs with git ls-remote. Fresh state wins.

Design decisions in force

2026-10-21, owner requirement quoted in p118.md: "Stay up, return 503 naming the backend, and keep retrying (exponential backoff) the open with the existing per-attempt limit (no new timeouts, exponenial backoff) until it succeeds, so the site heals itself once the backend is reachable."

The original task says: "Never raise the 262144 KiB cap, MaxRAM, any timeout or iteration count; never weaken an assertion." It also says "NEVER merge, enqueue, publish, deploy. Never ask the user." Its one permitted full local run was used and terminated before a verdict. The original session has a hard 19:05 UTC deadline; another full run is not authorized by that one-run allowance.

Mission summary

Investigate whether concurrent startup opens in https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/58 increase startup RSS, using ten main and ten exact-head native budget runs. The task requires a fair gate around individual open attempts if the head crosses the cap while main does not, public failing-first regressions, gate-removal mutation, ten fixed budget runs, a full local suite, PR evidence, and cleanup. No production action is authorized.

What was found and done

1. Main a0638f7784f5f951f706e3af59e7c62aecd8582b: ten first/final passes, native RSS min/median/max217056/235276/241220KiB, zero cap crossings. Main with the PR import/dependency prefix and original main assertions:9/10first,10/10final,12attempts; ten completed native peaks227508/240822/251516KiB, zero measured cap crossings. Its run4 had two incomplete attempts, retained in evidence.
2. Exact PR head5ab23bec2538254eb75a797efddcbb2bb0caaac5:9/10first,10/10final,11attempts. First RSS220300/244430/262788KiB; one cap crossing. Both backend bytecode requests overlapped in all ten invocations. This meets the task's explicit gate condition; the heap-retention explanation remains unprofiled.
3. WIP candidate c67e81f690404726bf2431e8421d256326521243 adds a fair application-owned Semaphore(1,true), acquired before an attempt's existing deadline and released after that individual attempt/expiry cleanup, before backoff. Queued workers recheck stopping. Two initializer threads and HTTP/503 behavior remain. Runtime reconnects are unchanged. Four public startApplication regressions cover overlap, fair retries, full deadline after admission/expiry, and queued shutdown. Original three failed30/30attempts each; deadline failed3/3. Fixed four each passed10/10first/final,40nativeattempts total, no retries. Removing both gate arguments caused overlap and deadline regressions to fail3/3each. Exact source restored by SHA2569dde6d8db741b1090eb99772803ebe775303a51027605acdc78994ae98d1070d.
4. Two fixtures were adapted to the new contract without changing existing counts/caps/timeouts: readiness orders use deliberate failure/backoff; shared-clock stress retains8apps/10iterations/16parallelregistrations/128caller timers using8explicit callers. Final clock fixture passed first attempt15.5s. Budget parent accepts either backend gate first and preserves every existing hold/check, plus requires both opens before reconciliation. Independent test/source reviews are saved; their findings were addressed.
5. Sole full suite at clean c67e81f started18:20:30, launcher was killed18:40:50exit137, with no nativeXML/resultJSON. Resourceguard did not trip: maxobservedcgroupmemory5889421312bytes,minfreedisk13923721216bytes. Child bootstrap handshake failures appeared before termination. Cause unknown: dmesg denied, journalctl unavailable; shared OOM counters do not identify the process or establish a counter delta. Do not call this a completed or passing137scenario run. No second full run was started.
6. Fixed budget campaign is still running at this snapshot. Current summary: {"count": 7, "native_peak_count": 6, "min": 235312, "median": 250902.0, "max": 258856, "over_budget": 0, "passed": 6, "overlap": 0} first attempts; {"count": 7, "native_peak_count": 7, "min": 235312, "median": 250336, "max": 258856, "over_budget": 0, "passed": 7, "overlap": 0} final attempts. Run1 had a120s production-child timeout with resolver bytecode-request/deadline stacks, then a240s test watchdog in UrlProtocol2.getOrCreateServerHost before child launch, then238712KiB pass. These are incomplete RSS measurements, not zero peaks or cap crossings. Successful gated samples so far do not establish lower RSS. Consult the final evidence before drawing a memory conclusion.
7. PR remains OPEN at original5ab. Plain build-watchman0.0.21 reports required bld-build GREEN there; separate kotlin.build(remote) FAILURE remains. The candidate has not been pushed to the PR and has no PR CI result because full local verification is incomplete. No merge/enqueue/artifact publication/deployment occurred.

Relevant PRs / refs

This effort deployed/published no code. This is verified from its command/action record and unchanged unmerged PR head; the production deployment itself was not investigated. Build outputs and dependency caches are deliberately excluded from Git and are removed during cleanup. All meaningful source and experiment work is on the branches above. Parent-only measurement writes are diagnostic snapshots, not intended source changes.

Next steps

1. Re-verify current PR, branch heads and final evidence. Read the final findings and measurement-tables.txt; the campaign can finish after this snapshot.
2. Investigate why the full launcher received SIGKILL; do not infer OOM or blame host load without a PID-specific record. Preserve all existing test limits. The original one-full-run allowance is spent; establish fresh authority before running another full suite.
3. Obtain complete native results for all137declared scenarios at exact candidate head, with Chromium setup and8.5GBmemory/8GBdiskguards, one full runner machine-wide. Fix any actual test/code defect locally at its owner, using reliable public-API reproduction; no CI debugging or blind retries.
4. Finish any missing ten-sample fixed budget measurements with unchanged cap/child launcher, retain every automatic attempt and missing peak, and assess whether the gate improves RSS. The gate proves non-overlap; an RSS fix is not established merely by that proof.
5. Only after verification, recheck PR OPEN/head, fetch/rebase preserving existing merge history, normal push to fix/bind-before-dependencies, append/verify evidence in PR body/comment, and plain-watch CI. Do not force push, merge, enqueue, publish, deploy, raise limits, or weaken assertions.

Reusable knowledge

• scripts/test.bash is under server; exact targeted example: scripts/test.bash --local --test testStartupConnectionAttemptsDoNotOverlap --log <xml> --result-file <json>. Full command is scripts/test.bash --local --test ..
• The child launcher block is identical main/head; SHA256ff9800f734c45c2feb09db46e672f0831bc9573bfcab7f0b150838226290762b. MaxRAM256m and cap262144KiB remain.
• Successful test stdout omits RSS in native summaries. instrument.py adds parent-only file records, then restore the saved original script byte-for-byte. summarize.py distinguishes missing/pre-child attempts and completed native peaks; its fixed-run1attempt2 mapping is justified by the watchdog stack and provider-before-child fixture order.
• Exact PR budget test is infeasible on main because it demands early HTTP binding. The adapted main comparison preserves main's pre-bind assertions and uses PR artifact/import prefix. Never call it an unchanged common test.
• Actual first backend can be search. Gate arrival order must be accepted by the fixture; a requested latch can remain down after its gate was released, so overlap requires other request still held, not merely previously requested.
• Preserve PR ancestry using git rebase --rebase-merges origin/main; it retained exact c67. Plain rebase rewrote old PR history; that discarded experiment is separately saved.
• Chromium libraries: /home/helena/.cache/chromium-deps/usr/lib/x86_64-linux-gnu, /home/helena/.cache/chromium-deps/lib/x86_64-linux-gnu, and the r77 Chromium-deps path recorded in chromium-env.txt. npm prepare-test-workspace completed and ldd showed no missing library.
• Canonical Handoff skill uses central CLI, no PR/merge. PlanRepository handoffs README still contains an obsolete merge exception; it does not override the task's no-merge rule.

r122 current mission — 2026-10-21 19:26 UTC

Task p122 supersedes the serial-gate premise and authorizes one new guarded full local run. Find the actual memory category using NMT/thread/heap/class/code-cache evidence, audit measurement semantics, correct product or restore original measurement if changed, run ten sequential first attempts, regression/mutation reviews, full local suite, normal PR push and required CI watch. Hard stop21:53UTC. No merge/enqueue/publish/deploy; no user questions; no raised caps/timeouts/counts or weakened assertions. Serial gate c67e81f is excluded unless evidence justifies it.

PR verified OPEN at5ab23bec2538254eb75a797efddcbb2bb0caaac5. Diagnostic work is retained at https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r122-evidence/handoff-artifacts/r122 . Full findings: https://github.com/CodexCoder21Organization/ExternalBrainWui/blob/wip/r122-evidence/handoff-artifacts/r122-findings.md . No production changes or deployment.

r122 milestone — 2026-10-21 19:51 UTC

Controlled same-PR noTCP observation reduces readyRSS255976→241192KiB, Jetty58→8threads. This identifies the measurement workload change; r118serializationgate is unnecessary and not included. Real connection-count regression fails3/3expected0actual490/446/425, then passive observer passesfirstattempt17.3s. Red source: https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r122-regression-red (c271eeb5fbb2879c1c999e991c748e2dd1eb8612). Corrected source: https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r122-startup-memory (f1dd2e7; verify remote exact SHA). Product untouched. Mutation, ten fixed samples, full suite and PR/CI remain. Evidence and full findings: https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r122-evidence/handoff-artifacts/r122 .