Handoff: Finish CI verification for the CI queued-event heap fix

Written 2026-10-22, 06:23 UTC.

RE-VERIFY: This is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/352 with gh pr view --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt before acting. Check the Actions run linked below.

Design requirements

Owner directives for this lane, dated 2026-10-22:

• "never raise -Xmx or add JVM flags"
• "never restart production services in this lane unless the brief says so"
• "never enqueue/merge/publish/deploy"
• "Watch your PR's CI with build-watchman 0.0.21 plain --pr."
• "Poll GitHub no faster than every 2 minutes." The brief requires first-attempt 10/10 proof and a red reversal, fixes in the owning repository, local targeted tests during iteration, and evidence only on wip/laneHCI-evidence. Production rollout and Maven publication require separate authorization. This handoff carries no merge or rollout authorization.

Mission summary

The owner reported that https://droplets.wasmserver.com/ "is super unreliable, frequently rendering various errors/notices due to bugs in UrlResolver or UrlProtocol or something else" and asked for the underlying root cause to be fixed; the ContainerNursery route stops the WUI after 5 idle minutes, so most human visits are cold starts.

The HCI lane investigates kotlin-build-ci on the same production host. The code fix, live ownership analysis, deterministic reproducers, first-attempt register, reversal and independent review are complete. The remaining work is to obtain the required CI verdict, address any actual relevant CI issue with a local reproducer, and finish the lane report. No merging, queue entry, publication or deployment is part of that work.

What was found and done

The original host sample reported 293 full collections taking 552.6 seconds in about 600 seconds. The lane captured a live histogram and a live heap copy of PID 455427, deployed/main revision https://github.com/CodexCoder21Organization/kotlin-build-ci/commit/f22fb5a104d124fef9b9c46c7fcec8d8991ae7db, at about 05:10 UTC. That was the sampled deployment; it has not been changed by this lane.

The live histogram has 59,586,936 byte-array bytes, 10,627,152 string bytes and 9,603,560 linked-map-entry bytes. A strong-reference dominator analysis identifies WebhookServlet.executor -> ThreadPoolExecutor.workQueue -> LinkedBlockingQueue (about 20,420,008 bytes) and DualRouteWebhookEventHandler.shadowExecutor -> ThreadPoolExecutor.workQueue -> ArrayBlockingQueue (about 8,704,944 bytes). Queued servlet callbacks hold the complete decoded event through arg$3. Individual callbacks hold up to 288,120 bytes. Main source producers are WebhookServlet.kt lines 300, 359, 366, 3013 and 3046 on the sampled revision.

Mechanism: pending primary and shadow callbacks retain complete GitHub event graphs, including check output and nested metadata that CI never consumes. Reproducer: real Jetty/HTTP requests with exactly 64 accepted events and a paused scheduling dependency, testing both queue stages against the same 8,388,608-byte public heap-use bound, with ManualClock-backed service objects. Unchanged source retains 67,839,024 primary bytes and 68,429,344 shadow bytes and fails all three recorded baseline attempts. Final source passes each queue scenario 10/10 on attempt 1. Restoring the original servlet source makes both memory scenarios and the direct caller-mutation scenario red; fixed source was restored and verified clean.

The fix selects consumed fields before primary queueing for the built-in CI handlers and at direct dual-route entry. Nested valid maps/lists are copied. Custom handlers still receive complete documents. A review caught that invalid PR association lists must remain complete for the existing diagnostic; that is fixed and checked against the full original text. Queue depth and worker scheduling are unchanged. No new interface method, fixture, persisted status model or existing test was changed.

Six new scenarios cover both queue stages, all five CI event families and journal fields, signed Unicode/custom documents, malformed JSON, exact invalid-association diagnostics and caller mutation before delayed shadow work. Nine existing webhook scenarios pass on their first attempts. Final pre-PR gate passes all six new tests on attempt 1. XML with no attempt-history is a single attempt; explicit histories were checked rather than trusting an eventual green exit. Canonical XML is preserved on the evidence branch.

Other measured owners include three SJVM JarClassPathEntry objects (about 17.9 MB), resolver bytecode caches (about 6.3 MB), the activity journal (about 3.5 MB) and captured PR watchers (about 4.9 MB). Their growth over time is unmeasured; this lane establishes the queued-event retention mechanism. Byte counts are estimates using histogram-derived instance sizes and aligned arrays. Reference.referent edges are excluded. No claim is made that the new code has been verified in production.

The checksum-verified heap copy was analysed locally outside git. All 62 temporary files under /root/heapdumps-laneHCI were removed from the host, leaving zero files. Heap bytes, dependency caches and generated build outputs are deliberately excluded from remote branches. The full histogram, ownership report, collection/analysis scripts, canonical test XML and summaries are remote.

Relevant PRs / refs

No code was deployed or published by this lane. The fix is unmerged and not running in production. The sampled production revision is on main as linked above.

Current CI snapshot: fresh 06:23 UTC PR snapshot: bld-build IN_PROGRESS; the 25-minute monitor cap ended with the CI run still active. The last normal watch assessment was within its typical duration. Actions run: https://github.com/CodexCoder21Organization/kotlin-build-ci/actions/runs/39730793171 The application build step was already successful in the 06:06 UTC detail snapshot; the workflow then runs the complete local suite on its GitHub runner. No CI run was restarted.

Final report: https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/wip/laneHCI-evidence/lane-evidence/hci-20261022/rHCI-final.md Incremental findings: https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/wip/laneHCI-evidence/lane-evidence/hci-20261022/rHCI-findings.md Canonical XML: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/laneHCI-evidence/lane-evidence/hci-20261022/verification

Next steps

1. Re-verify current state first using the PR and Actions links above. Do not assume the CI state in this snapshot is still current.
2. Continue the required-check watch with: coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/kotlin-build-ci --pr 352. Do not add merge or queue options. Do not poll PR status more frequently than two minutes.
3. On green, record the final CI verdict in rHCI-final.md and stop. On red, inspect the actual test evidence, reproduce the relevant issue locally, and correct its cause before updating the PR. Do not rerun CI unchanged.
4. If code work is needed, clone fresh into your own lane, read AGENTS.md and README.md, and fetch/rebase before each build. Use scripts/test.bash --local --test <scenario> --log <file>.xml. The two queue scenarios are testWebhookPrimaryQueueKeepsResidentBytesBounded and testWebhookShadowQueueKeepsResidentBytesBounded. Read https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md. No new full local suite is needed merely to repeat the already-verified change; if a shared contract changes, use the machine-wide complete-suite lock from the lane instructions.
5. Keep all further evidence on the evidence branch. There is no pending user question or additional production authorization.

Reusable / operational knowledge

• The actual fabric certificate filenames are client_cert.pem, fabric_key.pkcs8 and server_cert.pem under /home/leslie/.config/hardware-control-fabric; the longer filenames in the brief are absent here. No key contents are copied into evidence.
• HCF launch output is clipped and can wait on a child that produces too much output. The collection script emits small compressed replies. read-file has a 10 MB limit and an additional per-request heap budget. For binary transfer, use small base64 text chunks, one at a time, then verify the checksum; direct text file output changes arbitrary binary bytes.
• rg is absent on this host; find and grep work.
• Successful XML normally omits attempt-history. Failed standalone baseline reports include three explicit attempts. A multi-test reversal can report one red test record without history; use the actual XML rather than assuming three retries.
• Challenge-report CLI automatically merges its record, so it was not used under the lane's no-merge rule. Tooling observations are in the findings.
• The bounded CI watcher has stopped. No extra test or service process is intentionally left running.