Handoff: BuildTestWui run-page data-path fix — complete and pushed, awaiting CI + user merge

Updated 2026-07-12 21:58 UTC (supersedes the 19:00 UTC snapshot; the stale-artifact blocker and every next-step from that snapshot are RESOLVED).

RE-VERIFY: every state below is a write-time snapshot. Before acting, run gh pr view 126 --repo CodexCoder21Organization/BuildTestWui --json state,mergeStateStatus,statusCheckRollup,mergedAt and check /tmp/codex-wui-progress.log on the work host. Fresh results override this document.

Mission summary

Fix buildtest.kotlin.build run pages loading slowly / never loading (user request 2026-07-12 ~13:30 UTC). Root cause was proven with live production thread dumps: the WUI marshaled every field of every test result through the interpreted SJVM sandbox (10,000+ crossings per page) behind one global cache lock with a 2s TTL; renders stalling >30s after the ~24KB header flush were killed by ContainerNursery's HTTPS-facade idle-read timeout (the "never loads" truncation at ~24.25KB).

Current state (verified at write time)

1. All work is complete and pushed. https://github.com/CodexCoder21Organization/BuildTestWui/pull/126 head e914a91f26b9b932e02aeec60a8a4537db4edc96, 14 commits, OPEN, unmerged; required check kotlin.build (remote) IN_PROGRESS at write time with build-watchman 0.0.12 armed on it. Full local suite 188/188 green (multiple times, pre/post rebase).
2. The 19:00 snapshot's verification blocker (stale test bytecode) was root-caused: kompile's buildMaven() output is cached in TWO layers — ~/.aibuildcaches/_code_BuildTestWui AND coursier's ~/.cache/coursier/v1/bldbinary/repository/buildtest/wui — under an unchanging version coordinate (0.0.76), so clearing only the first layer keeps serving pre-commit bytecode. Purge BOTH after every module source change. (Challenge records linked below.)
3. Fix inventory on the PR (each fail-first verified): per-run single-flight caches (484db58), immutable terminal-run snapshots (9b758a0), stale-while-revalidate (e082da1), bounded-concurrency duration-stats JS (1388b27), visible fetch-failure notices (0d38518), raw host-side RPC client replacing sandboxed per-field marshaling (b0a740e), production-exact nine-field build-rule wire parsing (88d6c06, fixed a review BLOCKER), 503+JSON error surfacing / run-lifecycle 404s + cache invalidation (2ec25a1, 228a0a6), bounded refresh executor + stale-poll UI indicator (7412fde, a6a9444, e25b48b), boundary/recovery test coverage (2c559b8, c580ea5), deterministic join-failure test (aab6795), deterministic transport-restart + load-stress-hardened concurrency tests (e914a91).
4. CI convergence across runs: 2 failures → 2 → 1 → (current run pending). Each red was root-caused, never retried blind: goldenScreenshots = known infra flake (verified green locally, no golden regeneration needed — diff only adds failure-only UI); the two test reds were under-synchronized TEST coordination (implementation proven correct), fixed deterministically and stress-verified 50–200 iterations under CPU starvation.
5. Upstream resolver defects (separate lane): the published foundation.url:resolver:0.0.600 retries server application errors and replaces the shared connection (proven downstream; NOT reproducible on UrlResolver HEAD — published-artifact/source drift). Regression coverage PR https://github.com/CodexCoder21Organization/UrlResolver/pull/768 is OPEN at 1092/1093 — the single failure is testGossipBroadcastStormWithStalledPeersBoundsQueuedMemory, which fails identically on that repo's current main (pre-existing baseline; challenge 2026-07-12-2106). A second published-0.0.600 defect (caller deadline overrun) is recorded in challenge 2026-07-12-2124. The WUI pin bump to a fixed resolver is DEFERRED until a corrected artifact is published.
6. Related but separate: backend handler-starvation issue filed with evidence at https://github.com/CodexCoder21Organization/BuildTestServerService/issues/211 (burst windows where all url://buildtest RPCs time out at 30s). A production latency probe ("wedge trap", /root/wedge-capture/ on 198.199.106.165) is still armed, polling run pages and thread-dumping backend/CN/WUI on stalls — use its poll.log for before/after latency evidence when the fix deploys.

Relevant PRs / refs

| Item | Write-time state | |---|---| | https://github.com/CodexCoder21Organization/BuildTestWui/pull/126 | OPEN, head e914a91f, check IN_PROGRESS; do NOT merge without user approval | | https://github.com/CodexCoder21Organization/UrlResolver/pull/768 | OPEN, 1092/1093; gated on that repo's pre-existing gossip retained-heap baseline failure | | https://github.com/CodexCoder21Organization/BuildTestServerService/issues/211 | OPEN — backend Dispatchers.IO starvation (stream-pull long-poll + inline lease-renewal RPCs) | | Challenges | 2026-07-12-1833 (resolver retries app errors), 2026-07-12-1846 (kompile cache-index write), 2026-07-12-1859 (stale module bytecode), 2026-07-12-2124 (resolver deadline overrun), 2026-07-12-2106 (UrlResolver main gossip baseline) |

Next steps

1. Watch the kotlin.build (remote) check on head e914a91f (build-watchman; re-arm after any rerun). If red: diagnose per the watch-build tables — do NOT retry blind; the only expected residual flake class is goldenScreenshots infra non-completion.
2. When green: request the USER's merge approval for https://github.com/CodexCoder21Organization/BuildTestWui/pull/126 (never merge without it).
3. After merge: propose deployment of the new buildtest-wui.jar to https:buildtest.kotlin.build:443 (container-nursery-deploy flow; log to /root/cn-watchdog.audit), then verify with the wedge trap's poll.log (baseline: 2–15s renders plus intermittent 30s-stall truncations; expect sub-second warm renders).
4. UrlResolver lane: PR 768 merges once that repo's owners fix their main gossip-baseline test; afterwards a corrected resolver artifact publish + WUI pin bump (separate PR) closes the two published-0.0.600 defects.
5. This handoff deletes (via create-handoff close-out) only when: PR 126 merged AND deployed AND verified via the trap, and the resolver follow-ups are either done or re-homed to their own tracking.

Reusable / operational knowledge

• kompile stale-artifact purge (BOTH layers, after every module source edit): rm -rf ~/.aibuildcaches/_code_BuildTestWui ~/.cache/coursier/v1/bldbinary/repository/buildtest/wui.
• Before every build: git fetch origin && git rebase origin/main.
• The WUI's /api/test-results?id= paginates at 100 — page or use the run page HTML when hunting CI failures (run records report true totals).
• Concurrency tests in this PR carry deterministic latch gates; when adding more, stress 50–100 iterations in one JVM under CPU starvation before trusting them on the 2-core CI droplet.