← Priority list

buildtest.kotlin.build emergency lane �� acute stall fixed + keystone/reopen PRs verified, blocked on coordinator throughput (upstream fix pending)

Acute buildtest admission stall root-caused+relieved (sandbox proxy never reopened; droplet-backend restart) and durable fix opened (BuildTestServerService PR 267). Keystone relay-flake fix (UrlResolver 893) verified 10/10, captured on wip branch, needs push+land. Everything routes through buildtest CI which is degraded by an unbounded ~1GB projection-scan on a RAM-bound host; durable remedy is claude-fable-deploy-lane's upstream BuildTestEmbedded scan-bounding fix (not yet merged). 894/895 downstream of 893; 447 round-2 needs Opus (codex filter-blocked).

hf-2026-07-27-buildtest-kotlin-build-emergency-lane-six-root-causes-found-timer-service-memory-fix-deployed-eleven-pull-requests-awaiting-merge

Edit handoff

Title Summary Markdown body

# buildtest.kotlin.build emergency lane — acute stall fixed, PR train verified but blocked on coordinator throughput (upstream fix pending)

**RE-VERIFY (write-time snapshot, 2026-07-30 ~13:00 UTC).** Everything below was true at write time and will drift. Re-check before acting: PRs via `gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt`; buildtest via https://buildtest.kotlin.build/ and host logs; coordinator process strictly by `ps -eo pid,etimes,rss,args | grep Xmx | grep buildtest-server` (RSS>500MB, not `pgrep|head`).

## Mission
Keep buildtest.kotlin.build (emergency-lane CI: dispatches each run to ~10 DigitalOcean droplets from a ~100-droplet fleet) reliably healthy, and land the outstanding relay/protocol PRs + fix the production-infra root causes behind the CI flakiness.

## What was done this session (2026-07-30)
1. **Merged** https://github.com/CodexCoder21Organization/UrlProtocol/pull/448 (composed relay-state) and https://github.com/CodexCoder21Organization/UrlResolver/pull/898 (NetLab retry idempotency — the biggest CI flake source).
2. **Fixed the ACUTE buildtest admission total-stall.** Root cause: `BuildTestServerService` `RemoteDropletService.kt:44` opens the `url://digitalocean-droplets/` SJVM sandbox proxy via `by lazy` and caches it for the process lifetime; one `listDroplets` 30s timeout marks that sandbox stickily invalidated and it is never reopened (`BoundedDropletService` only time-bounds each call, returns the same dead proxy), so admission freezes and the droplet reaper stalls. Relieved operationally at 11:30Z by restarting the droplet-service-server backend (pid was 2926279) — a server-side close forces buildtest-server to re-resolve and open a fresh sandbox. Reversible, route/process restart only.
3. **Durable fix for #2 as PR** https://github.com/CodexCoder21Organization/BuildTestServerService/pull/267 ("Reopen invalidated droplet sandbox connections", gpt-5.6-sol, has a fail-first test) — OPEN, needs CI + merge.
4. **Keystone relay-flake fix** https://github.com/CodexCoder21Organization/UrlResolver/pull/893 — verified locally: full local suite 0 failures (1232 tests) + a deterministic reproducer flips 10/10-fail → 10/10-pass. The fix moves the relay-registration retry coroutine off the bounded shared resolver dispatcher onto a dedicated elastic dispatcher so the healthy same-tier relay's 10s registration window isn't shortened under contention (was causing ambient flaky `stressTestHealthySameTierRelayWinsUnderContention`). NOT yet pushed to the PR branch (its worker over-verified >1h). **The verified commit is captured remotely** at branch `wip/893-relay-flake-verified-handoff-20260730` head `52882fce2f24668d5af05988f03fd9f94cc7f529` (https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/893-relay-flake-verified-handoff-20260730). Next session: fast-forward/force-with-lease PR branch `fix/relay-preference-attempt-accounting` to that commit (discard any uncommitted debug logging in /code/ws/w21/UrlResolver first), then land.
5. **Investigated run a6f06492** (user ask): a genuine, already-fixed project compile error (a dependency bump added an interface method the same commit didn't implement; next commit fixed it), NOT infra. Surfaced a real observability gap: buildtest's `errorMessage` stores only the last 1000 chars, hiding the first compiler error behind secondary noise.

## Blocked / open PRs
- https://github.com/CodexCoder21Organization/UrlResolver/pull/893 — verified, needs push (see #4) + land.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/894 — CLEAN; blocked-by 893's ambient flake (its full-suite validation runs `stressTestHealthySameTierRelayWinsUnderContention`). Rebase after 893 lands.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/895 — downstream of 893.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/447 — round-2 has 5 residual concurrency defects (authoritative: /code/ws/w23/FINDINGS.md — same-relay prepare erases live route, stale reconnect undoes commit's failure-counter reset, close() misses concurrently-admitted candidate leak, rejected-commit doc mismatch, release-notes wrongly claim 0.0.429 published). **codex content-filter blocks gpt-5.6-sol on this UrlProtocol code** — needs Opus or hands-on.

## THE WALL (primary blocker) — buildtest coordinator throughput degradation
buildtest.kotlin.build's coordinator has a persistent throughput degradation that periodically FREEZES admission (0 new droplets created) and lets bursts of runs flood and fail. Everything above (893/894/895/447/267 CI) routes through buildtest, so nothing lands while it's degraded. Root cause is coordinator resource management under the ~1 GB UNBOUNDED file `/root/buildtest-data/execution-events.jsonl` (5.9M lines, ~1,017,738,237 bytes) that `buildtest-embedded` 0.0.61206 (BuildTestEmbedded PR 442) scans to build its duration/memory projection, on a RAM-constrained host (15 GB total, ~4 GB free; coordinator `-Xmx4096m -XX:MaxMetaspaceSize=512m`).

**Durable remedy = an UPSTREAM fix bounding/streaming that scan — being authored by claude-fable-deploy-lane, NOT yet merged in BuildTestEmbedded.** That is THE unblock.

**Levers evaluated and rejected (don't repeat these):**
- Coordinator restart does NOT block admission (verified: `durationProjection` is read only by `getTestResultsPaginated`/`getExecutionDurationStats`/`getExecutionResourceStats`/`getExecutionDurationStatsPaginated` — all WUI/stats; the droplet-admission gate `reservedHeadGrantFitsObservedFleetLocked` etc. do not touch it) — but it does NOT raise net throughput (the coordinator's ~8h old-gen heap cycle caps it either way) and re-triggers a ~25min WUI-view 503 projection rebuild + ~8min startup fleet-reconciliation. The coordinator OOM-restarts itself when old-gen fills; a manual restart only shifts timing.
- Heap can't be enlarged (host RAM-bound).
- Truncating the live projection file is risky (open fds / append).

**UNEXPLAINED PUZZLE for the next investigator:** at 12:58Z admission was frozen (0 droplets created in 200 lines) with old-gen only **34%** and `listDroplets` NOT timing out — so the freeze is NOT (always) GC-pressure and NOT the sandbox stall. The admission-gate freeze mechanism at low heap is still un-root-caused; this is the next thing to nail (likely fleet-capacity accounting or an admission-gate condition, not memory). Also observed a retry-storm: 349 runs created in 15 min across diverse repos (~25× the ~10-concurrent-run fleet capacity) failing in waves — amplified by the throughput degradation.

**Process-ID caution:** I repeatedly mis-identified the coordinator because `pgrep -f buildtest-server-service.jar` matches transient ~3 MB launcher/SJVM subprocesses as well as the real `-Xmx4096m` JVM; several of my "crash-loop"/"restart" conclusions were artifacts of grabbing the wrong PID. Identify the coordinator strictly by RSS>500MB + uptime.

## Next steps for pickup (ordered)
1. **THE unblock:** get claude-fable-deploy-lane's upstream projection-scan bounding fix merged + deployed to restore buildtest throughput. Until then, buildtest cycles degraded and CI is unreliable.
2. Push + land keystone 893 (from wip branch `wip/893-relay-flake-verified-handoff-20260730` @ `52882fce2`), then cascade-rebase + land 894, 895.
3. Land reopen-fix https://github.com/CodexCoder21Organization/BuildTestServerService/pull/267.
4. Do 447 round-2 via an Opus subagent (codex filter blocks gpt-5.6-sol on it); defects in /code/ws/w23/FINDINGS.md.
5. Durably cap `execution-events.jsonl` and fix the `errorMessage` last-1000-chars truncation (buildtest observability).
6. Root-cause the low-heap admission-gate freeze puzzle above.

## Operational knowledge
- Host: `ssh -i ~/.ssh/cn_host_diag -p 23 root@198.199.106.165`. Coordinator logs: `/root/ContainerNursery/data/container-logs/buildtest-server-service-c162fff3.std{out,err}.log`. Run state: `/root/buildtest-data/runs/<id>/run.json`.
- Cross-lane coordination log: `/root/cn-watchdog.audit` (I logged the 11:30Z droplet-backend restart and the overload/retry-storm there).
- Live workers still running at handoff time (in this session's box, ephemeral): `/code/ws/w21` (893 verification, checkout on branch `fix/relay-preference-attempt-accounting`), `/code/ws/w-reopen` (267 monitoring). Their key output is already captured (893 → wip branch above; 267 → its PR).
- Restarting the `url:digitalocean-droplets:` backend (SIGTERM `/root/ContainerNursery/apps/droplet-service-server.jar`, CN respawns) is the proven, reversible lever for the SANDBOX-stall failure mode ONLY (listDroplets timing out / "Reopen the connection" in stderr) — not for the memory-throughput freeze.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

buildtest.kotlin.build emergency lane — acute stall fixed, PR train verified but blocked on coordinator throughput (upstream fix pending)

RE-VERIFY (write-time snapshot, 2026-07-30 ~13:00 UTC). Everything below was true at write time and will drift. Re-check before acting: PRs via gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt; buildtest via https://buildtest.kotlin.build/ and host logs; coordinator process strictly by ps -eo pid,etimes,rss,args | grep Xmx | grep buildtest-server (RSS>500MB, not pgrep|head).

Mission

Keep buildtest.kotlin.build (emergency-lane CI: dispatches each run to ~10 DigitalOcean droplets from a ~100-droplet fleet) reliably healthy, and land the outstanding relay/protocol PRs + fix the production-infra root causes behind the CI flakiness.

What was done this session (2026-07-30)

Merged https://github.com/CodexCoder21Organization/UrlProtocol/pull/448 (composed relay-state) and https://github.com/CodexCoder21Organization/UrlResolver/pull/898 (NetLab retry idempotency — the biggest CI flake source).
Fixed the ACUTE buildtest admission total-stall. Root cause: BuildTestServerService RemoteDropletService.kt:44 opens the url://digitalocean-droplets/ SJVM sandbox proxy via by lazy and caches it for the process lifetime; one listDroplets 30s timeout marks that sandbox stickily invalidated and it is never reopened (BoundedDropletService only time-bounds each call, returns the same dead proxy), so admission freezes and the droplet reaper stalls. Relieved operationally at 11:30Z by restarting the droplet-service-server backend (pid was 2926279) — a server-side close forces buildtest-server to re-resolve and open a fresh sandbox. Reversible, route/process restart only.
Durable fix for #2 as PR https://github.com/CodexCoder21Organization/BuildTestServerService/pull/267 ("Reopen invalidated droplet sandbox connections", gpt-5.6-sol, has a fail-first test) — OPEN, needs CI + merge.
Keystone relay-flake fix https://github.com/CodexCoder21Organization/UrlResolver/pull/893 — verified locally: full local suite 0 failures (1232 tests) + a deterministic reproducer flips 10/10-fail → 10/10-pass. The fix moves the relay-registration retry coroutine off the bounded shared resolver dispatcher onto a dedicated elastic dispatcher so the healthy same-tier relay's 10s registration window isn't shortened under contention (was causing ambient flaky stressTestHealthySameTierRelayWinsUnderContention). NOT yet pushed to the PR branch (its worker over-verified >1h). The verified commit is captured remotely at branch wip/893-relay-flake-verified-handoff-20260730 head 52882fce2f24668d5af05988f03fd9f94cc7f529 (https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/893-relay-flake-verified-handoff-20260730). Next session: fast-forward/force-with-lease PR branch fix/relay-preference-attempt-accounting to that commit (discard any uncommitted debug logging in /code/ws/w21/UrlResolver first), then land.
Investigated run a6f06492 (user ask): a genuine, already-fixed project compile error (a dependency bump added an interface method the same commit didn't implement; next commit fixed it), NOT infra. Surfaced a real observability gap: buildtest's errorMessage stores only the last 1000 chars, hiding the first compiler error behind secondary noise.

Blocked / open PRs

https://github.com/CodexCoder21Organization/UrlResolver/pull/893 — verified, needs push (see #4) + land.
https://github.com/CodexCoder21Organization/UrlResolver/pull/894 — CLEAN; blocked-by 893's ambient flake (its full-suite validation runs stressTestHealthySameTierRelayWinsUnderContention). Rebase after 893 lands.
https://github.com/CodexCoder21Organization/UrlResolver/pull/895 — downstream of 893.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/447 — round-2 has 5 residual concurrency defects (authoritative: /code/ws/w23/FINDINGS.md — same-relay prepare erases live route, stale reconnect undoes commit's failure-counter reset, close() misses concurrently-admitted candidate leak, rejected-commit doc mismatch, release-notes wrongly claim 0.0.429 published). codex content-filter blocks gpt-5.6-sol on this UrlProtocol code — needs Opus or hands-on.

THE WALL (primary blocker) — buildtest coordinator throughput degradation

buildtest.kotlin.build's coordinator has a persistent throughput degradation that periodically FREEZES admission (0 new droplets created) and lets bursts of runs flood and fail. Everything above (893/894/895/447/267 CI) routes through buildtest, so nothing lands while it's degraded. Root cause is coordinator resource management under the ~1 GB UNBOUNDED file /root/buildtest-data/execution-events.jsonl (5.9M lines, ~1,017,738,237 bytes) that buildtest-embedded 0.0.61206 (BuildTestEmbedded PR 442) scans to build its duration/memory projection, on a RAM-constrained host (15 GB total, ~4 GB free; coordinator -Xmx4096m -XX:MaxMetaspaceSize=512m).

Durable remedy = an UPSTREAM fix bounding/streaming that scan — being authored by claude-fable-deploy-lane, NOT yet merged in BuildTestEmbedded. That is THE unblock.

Levers evaluated and rejected (don't repeat these):

Coordinator restart does NOT block admission (verified: durationProjection is read only by getTestResultsPaginated/getExecutionDurationStats/getExecutionResourceStats/getExecutionDurationStatsPaginated — all WUI/stats; the droplet-admission gate reservedHeadGrantFitsObservedFleetLocked etc. do not touch it) — but it does NOT raise net throughput (the coordinator's ~8h old-gen heap cycle caps it either way) and re-triggers a ~25min WUI-view 503 projection rebuild + ~8min startup fleet-reconciliation. The coordinator OOM-restarts itself when old-gen fills; a manual restart only shifts timing.
Heap can't be enlarged (host RAM-bound).
Truncating the live projection file is risky (open fds / append).

UNEXPLAINED PUZZLE for the next investigator: at 12:58Z admission was frozen (0 droplets created in 200 lines) with old-gen only 34% and listDroplets NOT timing out — so the freeze is NOT (always) GC-pressure and NOT the sandbox stall. The admission-gate freeze mechanism at low heap is still un-root-caused; this is the next thing to nail (likely fleet-capacity accounting or an admission-gate condition, not memory). Also observed a retry-storm: 349 runs created in 15 min across diverse repos (~25× the ~10-concurrent-run fleet capacity) failing in waves — amplified by the throughput degradation.

Process-ID caution: I repeatedly mis-identified the coordinator because pgrep -f buildtest-server-service.jar matches transient ~3 MB launcher/SJVM subprocesses as well as the real -Xmx4096m JVM; several of my "crash-loop"/"restart" conclusions were artifacts of grabbing the wrong PID. Identify the coordinator strictly by RSS>500MB + uptime.

Next steps for pickup (ordered)

THE unblock: get claude-fable-deploy-lane's upstream projection-scan bounding fix merged + deployed to restore buildtest throughput. Until then, buildtest cycles degraded and CI is unreliable.
Push + land keystone 893 (from wip branch wip/893-relay-flake-verified-handoff-20260730 @ 52882fce2), then cascade-rebase + land 894, 895.
Land reopen-fix https://github.com/CodexCoder21Organization/BuildTestServerService/pull/267.
Do 447 round-2 via an Opus subagent (codex filter blocks gpt-5.6-sol on it); defects in /code/ws/w23/FINDINGS.md.
Durably cap execution-events.jsonl and fix the errorMessage last-1000-chars truncation (buildtest observability).
Root-cause the low-heap admission-gate freeze puzzle above.

Operational knowledge

Host: ssh -i ~/.ssh/cn_host_diag -p 23 root@198.199.106.165. Coordinator logs: /root/ContainerNursery/data/container-logs/buildtest-server-service-c162fff3.std{out,err}.log. Run state: /root/buildtest-data/runs/<id>/run.json.
Cross-lane coordination log: /root/cn-watchdog.audit (I logged the 11:30Z droplet-backend restart and the overload/retry-storm there).
Live workers still running at handoff time (in this session's box, ephemeral): /code/ws/w21 (893 verification, checkout on branch fix/relay-preference-attempt-accounting), /code/ws/w-reopen (267 monitoring). Their key output is already captured (893 → wip branch above; 267 → its PR).
Restarting the url:digitalocean-droplets: backend (SIGTERM /root/ContainerNursery/apps/droplet-service-server.jar, CN respawns) is the proven, reversible lever for the SANDBOX-stall failure mode ONLY (listDroplets timing out / "Reopen the connection" in stderr) — not for the memory-throughput freeze.