Land and deploy the TinySite memory-leak fix and the BuildTest disk-full recovery fix

RE-VERIFY BEFORE ACTING. Everything below is a snapshot written 2026-10-22 03:30 UTC. Re-check each PR with gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,headRefOid,mergedAt, TinySite with curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://tinysite.wasmserver.com/, and BuildTest intake with the latest kotlin.build (remote) check-run on CodexCoder21Organization/UrlResolver main (gh api repos/CodexCoder21Organization/UrlResolver/commits/main/check-runs).

Design decisions in force (owner's words, verbatim)

• 2026-10-21, original request: "investigate the broken production service … follow TDD, figure out what went wrong, reproduce it in a test (TDD), fix it, merge and deploy." Scope named by the owner: TinySite (https://tinysite.wasmserver.com/), UrlResolver, UrlProtocol, SimpleFileSystemServiceServer, and the BuildTest build service with its droplet service.
• 2026-10-21: "yes, deploy after your merges" — deploy approval covers merges made by this effort. A bare TinySite restart was not approved (asked repeatedly; no answer).
• 2026-10-21: "startup (and incremental recovery) should always be fast. If there is any situation where startup or incremental recovery are slow, that's a bug which should have been tested and exercised by https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos and which should be fixed."
• 2026-10-21, on UrlProtocol slow readers: "I think we should do the things that the web server does." and "I think the shared buffer size should also have a cap per connection, and the shared buffer should have a maximum bounded size but also should shrink if it isn't being heavily utilized..."
• 2026-10-22, on stopping this session: "Update the handoff with the latest details and open questions. Kill the lanes which have been running for over an hour. Let other lanes finish gracefully and record the results but don't launch more."

Open questions awaiting the owner (unanswered at write time)

1. May TinySite be restarted whenever it degrades before the fix is deployed? The leak refills TinySite's 128 MB heap over hours; a restart clears it and loses no evidence (the heap evidence is saved on branch evidence/r114-tinysite-heap-20261021). Recommendation: yes, restart route https:tinysite.wasmserver.com:443 when responses exceed 5 s. TinySite was restarted by someone else around 01:30 UTC; at 03:27 it answered in 2.25 s (it answered in 0.25 s at 01:31), so it is degrading again.
2. Stream or reject oversized replies in UrlProtocol? https://github.com/CodexCoder21Organization/UrlProtocol/pull/765 implements the owner's slow-reader direction by streaming large replies with backpressure. Two tests already on UrlProtocol main encode the old rule: testDeclaredLongPollOversizedReplyKeepsSlotUntilDelivered (expects a >4 MiB reply to a non-reading peer to fail with exact OUTBOX_FULL) and testForwardedPendingWriteCompletionOnWriterInterrupt (expects one 37,873-byte write; streaming emits 16,384-byte frames). Option A (recommended): stream, and update both tests to assert the new behavior equally strictly. Option B: keep the size rejection. The first test's update is written, mutation-checked (fails with backpressure accounting broken, passes restored), and isolated in commit https://github.com/CodexCoder21Organization/UrlProtocol/commit/85aa3448b73222c1cdb3b232674b43b5608910c9 on branch wip/r138-streaming-rebase.
3. Deploy TinySite main now, or wait? https://github.com/CodexCoder21Organization/TinySiteWui/pull/24 was merged at 2026-10-22 01:54 UTC by the codexcoder21 account through the merge queue — not by this effort's lanes (none had merge permission and no lane log shows it). kotlin.build (remote) never concluded on it. It pins foundation.url:resolver:0.0.1329, which contains the leak fix but was built from a tree missing one later main commit (https://github.com/CodexCoder21Organization/UrlResolver/pull/1351) and through a non-standard path (its module file is foundation_url_resolver_sandbox.kotlin_module). Option A (recommended): wait for a properly built resolver from https://github.com/CodexCoder21Organization/UrlResolver/pull/1382 and a new TinySite PR. Option B: deploy TinySite main now. If the owner merged PR 24 deliberately, treat that as a deploy go-ahead.
4. Change an over-strict UrlResolver test, or the code? testGossipNegotiationFailurePreservesPendingRpc fails its first attempt ~1 in 20 runs, on main and on the fix branch alike (main 1/20, fix 2/21; identical assertion at line 115). Diagnosis: after a gossip refusal the test requires a ping to dial a brand-new connection, but pingPeer may validly reuse an existing replacement connection (gossip-idle does not stop peer exchange). Forcing that valid state failed the assertion 3/3 while the held RPC still completed exactly once. Option A (recommended): narrow the assertion to "either a new dial or reuse of an existing replacement, and the held RPC completes exactly once". Option B: make the resolver always dial fresh here. No code or test was changed.

Mission

TinySite (a static-site publishing service on ContainerNursery at 198.199.106.165) degraded and went down repeatedly on 2026-10-21. Root-cause it, reproduce in a test, fix, merge, deploy. Along the way BuildTest (the remote-CI build service) also latched broken after its disk filled, which blocks every kotlin.build (remote) check and therefore every merge.

What was found and done

TinySite memory leak — root cause proven

• LazyReconnectingMethodDispatcher.runIdleCheck in UrlResolver calls detachCurrentLocked, which adds each closed sandbox connection to the retiredConnections identity set; nothing ever removes it. Every idle-closed sandbox stays reachable until TinySite's 128 MB heap is exhausted (heap dump evidence on branch https://github.com/CodexCoder21Organization/TinySiteWui/tree/evidence/r114-tinysite-heap-20261021).
• Fix: https://github.com/CodexCoder21Organization/UrlResolver/pull/1382 removes the connection by identity under buildLock in a finally once its shutdown finishes, even if shutdown fails part-way. Head ff02bf3d1e2fba09f4245f4fe36764d603ae3232 (rebased onto main after 01:49 UTC).
• An earlier full suite on the previous head (9ec1b5c8) passed 2394/2394; the only retried test was the pre-existing flake in Open question 4 (proved not caused by the fix: no code path from that test reaches the changed dispatcher; production-only revert still passes).
• A second full suite on the rebased head was killed at ~58 minutes (owner instruction to stop lanes running over an hour). Up to then: no test failures; 19 buildMaven() build-rule failures, all Connection refused fetching kompile:build-kotlin-jvm:0.0.33 from https://kotlin.directory while the host was loaded (a direct curl at 03:13 returned 200 five times; one affected test retried green). The run must be repeated from scratch.
• Resolver coordinate: 0.0.1331 is already published (by a main commit), so the fix branch selected 0.0.1332, not yet published (kotlin.directory returns 404).
• TinySite upgrade: because PR 24 already merged, the update lives on branch https://github.com/CodexCoder21Organization/TinySiteWui/tree/wip/r140-tinysite-resolver-handoff-20261022 (local name fix/tinysite-resolver-0.0.1332, 3 commits, unverified; plus a stash snapshot on wip/r140-tinysite-stash-handoff-20261022). It repoints pins to 0.0.1332 and adds managed temp-jar cleanup in buildWebshotServiceClientForTests. No PR opened yet.

BuildTest disk-full latch

• After ENOSPC a failed write left in-memory duration-publication state that never recovered. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1578 keeps pending data and retries with backoff, publishing each result once and in order, with a bounded retry queue. Head d5905a891e588d9cf4d430548f693a207c64cfb9: 9 focused tests first-attempt green; duplicate-publication mutation control reproduced 3/3.
• A fresh-context review of that head found (each reproduced 3/3; full detail with line references in the artifacts branch, r142-findings.md):
  1. HIGH — a stats read during retry backoff permanently drops older all-time history (10 entries, raw journal rotated to 4, read during backoff lost 6 for good).
  2. MEDIUM — recovering an unrelated marker sets the resource-update flag early; the original sample disappears from resource stats.
  3. MEDIUM — a retry-clock failure exits the publisher while intake stays open.
  4. MEDIUM — missing tests: exact byte capacity, later backoff limits, partial-write shutdown, existing startup-repair case.
  5. LOW — journal test lines 96–106 diagnostics incomplete; cleanup stops after the first failed close.
  6. LOW — retry warnings name the marker temp file even when a later write failed.
• A lane is fixing these test-first at write time (left running per owner instruction); its in-flight commits are on https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r143-enospc-review-fixes-handoff-20261022. At 03:26 its new test testDurationPublisherReadDuringBackoffPreservesAllTimeHistory was failing on the unfixed code, as intended. Check its branch / the PR head for the final state. Full suite still required afterwards.
• BuildTest itself is still recovering: every kotlin.build (remote) check since the evening of 2026-10-21 fails at createUploadSession ("Build infrastructure error … createUploadSession failed after 5 attempts"), most recently 2026-10-22 01:52:54 UTC on UrlResolver main. Its startup re-reads 18,544 saved run.json files (~15 GB) at ~2.6/s and then folds the ProjectJournal (7,070 s for 10.24M rows at an earlier restart). Under the owner's fast-startup rule this slowness is itself a bug (next item).

BuildTest startup speed (owner rule: startup must be fast)

• Lane stopped by owner instruction. Current main fails the existing timing budget (800 rows: 3.67 s vs 2.50 s budget). A scoped file session reduced it to 2.09 s on the constructor measurement; the incremental budget still failed. Profile points at per-row persistent AVL page writes and checksums inside LegacyProjectionFold.add; the in-progress design batches rows and applies only surviving changes. Work is on https://github.com/CodexCoder21Organization/BuildTestProjectionJournal/tree/wip/r130-compaction-bounded-delta-handoff-20261022 (2 commits), a stash snapshot on wip/r130-compaction-stash-handoff-20261022, and an uncommitted 24 KB patch in the artifacts branch (r130-projectionjournal-uncommitted.patch); the BuildTestEmbedded pin bump is on https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r130-projectjournal-pin-handoff-20261022. None of it compiles-verified as a whole.

ContainerNursery console-log cap

• Lane stopped by owner instruction. Branch https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/r131-bounded-child-logs-handoff-20261022 (head 85c9d212) routes each child's console output through a relay into size-capped rotating files. Its full suite ran 852 passed / 6 failed. Classification was in progress; one confirmed defect: the relay JVM inherits the route's app-only environment variables (startRelayedChild copies childBuilder.environment() into the relay), shown by /proc/<relay>/environ containing CN_TEST_ROUTE_ENVVAR=FIRST in testContainerNursery_sameRouteRestartReusesBoundedChildLogs.kts:92. Fix plan: serialize the app environment into the relay config, keep the relay on ContainerNursery's own environment. No PR.

UrlProtocol slow-reader buffering

• https://github.com/CodexCoder21Organization/UrlProtocol/pull/765 conflicts with main (Libp2pHostFactory.kt). Rebase in progress on https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r138-streaming-rebase (head 85aa3448), stopped at the two test-contract conflicts in Open question 2. Main's preflight checks (RpcEncodingResources.guard, JSON checks, checkJsonWireStructure, checkRpcAttachmentFrameLimits) must keep running before the streaming producer.

Other paused work

• Reachability-check leak fix: branch https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/r132-bounded-verification-failures (head 91e88b0f), targeted tests and fix-removed check done; full suite and PR remaining.

Relevant PRs / refs

Pull requests (state at 03:28 UTC)

Branches holding unmerged work (every push verified with git ls-remote)

Deployed or published but not merged

• Published: foundation.url:resolver:0.0.1329 is on https://kotlin.directory and contains the leak fix, but it was not built from PR 1382's tree (missing https://github.com/CodexCoder21Organization/UrlResolver/pull/1351) and has the wrong module file name. TinySiteWui main now pins it via the merged PR 24. Reconcile: publish 0.0.1332 from PR 1382, move TinySite to it.
• Live: TinySite production runs /root/ContainerNursery-uploads/jars/tinysite-wui-d56e14dbcd30-20261021T142747Z.jar (built from TinySiteWui d56e14db, before PR 24) — leak still present. BuildTest PRs 1530/1559 are merged but not deployed. This effort made no production deploy in the hours covered here (checked via ContainerNursery containers output at 01:33 UTC).

Next steps

1. Get the owner's answers to Open questions 1–4.
2. When kotlin.build (remote) stops failing at createUploadSession, request remote checks on every final head above.
3. UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1382: rerun the full suite once from the pushed head (one full suite at a time on the host); publish 0.0.1332 through the standard buildMaven path per the publish-maven-artifact skill; verify the jar has foundation_url_resolver.kotlin_module and the retiredConnections removal; update the PR description (keep the opening why; list the known pre-existing flake with main 1/20 vs fix 2/21).
4. TinySite: from TinySiteWui main, apply the wip/r140-tinysite-resolver-handoff-20261022 changes, pin 0.0.1332, run its targeted tests (testWebshotIdleGenerationsReleaseTheirHeap, shutdown test) and full suite, open a new PR whose description opens with the incident and explains why PR 24's pin is being replaced.
5. Merge (only with kotlin.build (remote) green), then deploy TinySite via the ContainerNursery CLI and verify sub-second responses and stable heap over time.
6. BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1578: finish the six review findings test-first (in-flight branch above), full suite, fresh review of the final head, merge, then deploy together with 1530/1559 in one restart.
7. Finish BuildTest startup speed (fast-startup rule), ContainerNursery log cap (fix the relay environment defect, classify remaining failures), the reachability leak PR, and https://github.com/CodexCoder21Organization/UrlProtocol/pull/765 per Open question 2.

Reusable knowledge

• ContainerNursery CLI: cd /tmp && ~/bin/coursier launch containernurserycli:container-nursery-cli:0.0.20 -r https://kotlin.directory -- --url https://api.nursery.wasmserver.com containers (or restart --route-key https:tinysite.wasmserver.com:443). /usr/local/bin/coursier on this host is broken; use ~/bin/coursier.
• A test passing only on retry must be compared against main with ≥20 alternating isolated runs before blame is assigned; 10 runs gave a misleading result here.
• Status-code-only health checks hide this leak: TinySite returned 200 after 12–22 s while degraded. Measure response time.
• The shared GitHub API limit (5000/hour) was exhausted twice during this effort; batch PR reads into one GraphQL query.