Build-log out-of-memory tail — Lane Q live disposition

Re-verified 2026-07-30. This replaces the stale remaining-work claims in the 2026-07-29 snapshot. Do not merge, enqueue, deploy, restart, or otherwise change production state from this handoff without explicit authorization.

L16 close-out refresh — 2026-07-31

The following is the current terminal state for the non-resolver work. The handoff remains open because the remaining daemon consumer change depends on an unpublished UrlResolver fix, and this lane is not allowed to modify that repository.

Non-resolver API cleanup — completed

• BuildTestApi PR https://github.com/CodexCoder21Organization/BuildTestApi/pull/55 is merged and buildtest.api:buildtest-api:0.0.55 is published.
• BuildTestEmbedded PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/458 is merged. Its 0.0.61402 JAR and POM were published once from the locally verified output; the remote JAR and POM matched the local SHA-256 values, and the POM depends on BuildTestApi 0.0.55. The already-existing 0.0.61401 publication had different bytes and still depended on API 0.0.54, so it was not overwritten.
• kotlin-build-ci PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/200 is merged. Its whole-log fallback now uses metadata plus bounded tail chunks, and its final local suite passed 390/390.
• BuildTestServerService PR https://github.com/CodexCoder21Organization/BuildTestServerService/pull/273 is merged. It consumes the published API and embedded artifacts, keeps the concrete legacy getBuildLog helper for compatibility, and supplies that concrete reader to the RPC handler at the production boundary because the method is no longer in BuildTestApi. Its post-rebase build and full local suite passed 70/70; its remote check also passed 70/70.
• The earlier bounded fallback PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/198 and flaky-test fix PR https://github.com/CodexCoder21Organization/BuildTestServerService/pull/256 were already merged and were reverified. No new flaky-test change is needed.

The first server full-suite attempt encountered a transient DigitalOcean provisioning transport failure. The unchanged post-rebase suite completed the same real-droplet test successfully. The failure and the later queue-side buildtest restart failure are recorded in https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-31-0806-real-droplet-server-test-fails-when-digitalocean-droplets.md and https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-31-0931-merge-queue-validation-loses-dynamic-test-dispatch-after-a.md.

Daemon OOM — resolver-gated residual

The heap-dump evidence still supports resolver-owned dial retention: the old resolver version allowed expired waiters to leave underlying libp2p dials and their Netty/Yamux graphs alive. No daemon-owned unbounded structure was found to be the cause. UrlResolver PR https://github.com/CodexCoder21Organization/UrlResolver/pull/835 remains merged, and the post-fix resolver publications are available; the current published resolver metadata reaches 0.0.827 (with protocol metadata reaching 0.0.430).

The separate exact-once relay registered-peer exclusion remains in https://github.com/CodexCoder21Organization/UrlResolver/pull/823, which is still OPEN/CLEAN and not published. The consuming https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/119 is still OPEN/DIRTY with its earlier failed check because it needs that resolver-side change. This lane did not push or merge UrlResolver changes.

Therefore the daemon dependency bump and consumer verification remain resolver-gated. The next owner must merge and publish the owning UrlResolver change, then update and prove the daemon consumer. Do not delete or complete this handoff until that residual is resolved.

Current verdicts

API cleanup — historical evidence

The merged final state of https://github.com/CodexCoder21Organization/BuildTestServerService/pull/255 restored the legacy route as enabled-by-default compatibility with a deprecation warning. Current server main still advertises and dispatches it.

A read-only check of the active production generation found zero exact getBuildLog request records and zero matching deprecation warnings through 2026-07-30 10:30 UTC. This is useful evidence, but it is only a pre-migration sample.

The prior handoff was wrong that all callers were migrated:

• kotlin-build-ci still called getBuildLog after a stream failure, so it materialized the whole log before truncating it.
• BuildTestEmbedded has a test launcher that still calls the method.
• BuildTestApi has 60 test implementations of the abstract method and five direct contract assertions.
• BuildTestEmbedded still implements and tests the whole-log cache.

Lane Q migrated the live production caller in https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/198 at 8e0302e95eb85dcca6fb0a944f5fd940b2e195cb. The fallback now reads metadata and one 512 KiB tail chunk, and SynchronizedBuildTestApi forwards chunk requests. Build-watchman drove remote run 69883c54 to terminal success; all checks on the PR head are green. During the run, the BuildTest web view twice returned a cached-view 503. Read-only host ground truth showed fresh successful test events and then Build finished with status: COMPLETED, so no rerun or run deletion was needed.

Red/green proof:

• Against current main, the final regression failed because getBuildLog was called.
• With only the direct fallback changed, it failed because the production wrapper forwarded zero chunk requests.
• The final test passed repeatedly under -Xmx512m -XX:+UseSerialGC.
• The adjacent successful-stream regression passed unchanged.
• The full suite passed 390/390 after rebase and the final review correction.

The ordered migration steps above are historical evidence from before the L16 close-out refresh. Their terminal results are recorded in the current section at the top of this handoff.

At the time of this historical section, the final interface and implementation deletions could not be independently green before their prerequisites merged and published. That prerequisite is now complete for the non-resolver work; the concrete legacy helper remains only at the compatibility boundary by design.

Flaky test — resolved before the old handoff was written

https://github.com/CodexCoder21Organization/BuildTestServerService/pull/256 merged on 2026-07-26, three days before the handoff.

Its measured baseline failed 13/13 times; 11 runs expired at 301.4–302.2 seconds. A raw filesystem control spent 281.5 seconds on 1,048,576 file creates, reads, and deletes. The real embedded path took 241.3 seconds with 64-byte chunks versus 4.9 seconds with 4 MiB chunks.

The fix replaced the timing-dependent filesystem window with a latch around the real embedded API delegate. It did not raise the timeout, reduce work, or move cleanup outside the test. The merged proof was 10/10 repeated passes, four adjacent tests, a 65/65 full suite, and green remote CI. Lane Q reran unchanged current main 5/5 successfully under the requested JVM limits. No new PR is needed.

Daemon out-of-memory — mechanism identified; owning fix already merged

The failed daemon was built with resolver 0.0.637. Its heap dump contains 7,034 Netty socket channels, 9,399 pipelines, 6,676 libp2p connections, 6,467 Yamux handlers, 35,031 futures, 16,549 relay records, 16,989 gossip diagnostic records, and only 220 bounded peer gossip queues.

Mechanism: resolver 0.0.637's one-second waiter can expire while the underlying libp2p dial and kernel socket continue. That version has no process-wide bound on underlying transport dials, so historical peers retain thousands of Netty, Yamux, future, and relay graphs until the 128 MiB heap is exhausted. The historical-peer diagnostics map is a second unbounded retention path, but its shallow size is not the main heap consumer.

A hermetic public-path reproducer used 500 unreachable TEST-NET peers, 100 genuine concurrent gossip callers, the real registry/queue/libp2p/kernel path, and no artificial machine load. The historical stack failed 3/3 with 434, 435, and 410 simultaneous SYN-SENT sockets against a 32-socket oracle. Changing only the dependencies to resolver 0.0.661, protocol 0.0.397, and jvm-libp2p snapshot 19 made the identical reproducer pass 3/3.

The owning fix is already merged and published: https://github.com/CodexCoder21Organization/UrlResolver/pull/835.

The daemon consumer is not independently green: https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/119 is open and red because the coherent newer stack exposes a separate double-delivery defect. That defect is isolated upstream in https://github.com/CodexCoder21Organization/UrlResolver/pull/823. Its checks are green, but it has not been merged or published. The next action is to review and merge that owning PR, publish the resolver, then update and prove the daemon consumer. Lane Q did none of those merge/publish actions.

Durable checkpoints

• API prerequisite branch: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/lane-q-bounded-log-fallback at 8e0302e95eb85dcca6fb0a944f5fd940b2e195cb.

• Evidence summary branch: https://github.com/CodexCoder21Organization/PlanRepository/tree/artifacts/lane-q-build-log-tail-2026-07-30 at 01c71858f23cded3169063785ac645da89467d89.

• Earlier dump-analysis tools and reports: https://github.com/CodexCoder21Organization/PlanRepository/tree/artifacts/cn-oom-forensics-2026-07-29 at 6af7e78cfb7747def85f8a2b15eac7f96a2361c7.

• L16 BuildTestEmbedded checkpoint branch: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/lane-l16-remove-whole-log-override at 451606db5cc447f9ce5c9ee930fc7c418de39518 (merged).

• L16 kotlin-build-ci checkpoint branch: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/lane-l16-remove-whole-log-api-consumer at eeb0a22b25f33cfe14255e49c79315bfd81694aa (merged).

• L16 BuildTestServerService checkpoint branch: https://github.com/CodexCoder21Organization/BuildTestServerService/tree/lane-l16-remove-whole-log-api-consumers at cc4d292d5b71f21f56b1d04c1b9065504fb6ae33 (merged).

The raw 176,678,834-byte heap dump remains local only and was not committed. Read-only host inspection and the one documented copy fallback were appended to /root/cn-watchdog.audit. No production process, route, deployment, or configuration was changed.