Land the TinySite speed and parallel-upload fixes, finish the CI-capacity fixes, and redeploy TinySite, filedrop and storage

RE-VERIFY: this is a snapshot taken 2026-10-17 ~02:30 UTC. Re-check every PR before acting, using gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt. Check build-service capacity with curl -s https://buildtest.kotlin.build/api/health | jq .fleet. When the session that wrote this handoff stopped, Opus agents were still working on several open PRs, so heads may have moved since.

Mission

The owner's requests, in order:

1. Capture https://www.sazabi.com/ and its pricing page with the good-ui-designs skill. Done: Good-UI-Designs PRs 102–105 are merged.
2. "Update the skill to incorporate lessons … fix the gitignore of css and also make the uploads parallelized so we do like 8 concurrent uploads and also fix all the other issues." The skill and gitignore parts are done (claude-code-skills PR 159, Good-UI-Designs PR 102). Parallel uploads are in progress: they need the filedrop changes below to be deployed first.
3. "TinySite pages like https://tinysite.wasmserver.com/s/adc4c5e569/ load super slowly … requests should be parallelized … prefetch all assets of a site in parallel so they are ready as soon as the browser requests them." In progress.
4. "The tinysite management page for an individual tinysite should also keep track of two lists": site files that were never requested, or not recently relative to the site's other files; and recent requests under the site path that returned 404. "Both should show the three-component datetime as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/GOOD_DESIGN.md#displaying-date-and-time." In progress (TinySiteWui PR 5).
5. "We probably want to race with remote in order to ensure we can make forward progress if we're blocking on a build", following the runbook https://my.externalbrain.wasmserver.com/doc/github-builders-racing-remote-ci.md. Done for HandoffEmbedded, HandoffServiceServer and BuildTestProjectionJournal (see below).

Standing constraints from the owner:

• Opus mode. Merge only when the final head is green, an Opus subagent's fresh adversarial review of that exact head is clean, and you are confident. Otherwise ask.
• Deploys.
  • Pre-approved: TinySite, filedrop (FiledropServiceServer), simple-filesystem-vnext (the storage service) and ContainerNursery, after their merges. Always git pull main first.
  • Also pre-approved: kotlin-build-ci, redeployed after each merge to it.
  • Not approved: the handoff service and buildtest. Ask first.
• Never:
  • raise timeouts;
  • weaken or skip tests;
  • use --delete-branch;
  • restart ContainerNursery via HCF launch;
  • put the TinySite review link in a PR.
• References. Reference PRs by full URL.

What was found and done

TinySite slowness (root cause)

• No single point serializes requests. Requests already run concurrently at every hop. TinySite → filedrop → storage (simple-filesystem-vnext), with each hop relayed through ContainerNursery.

• Each GET of a small file costs about 60 storage round trips:

  • roughly 9 read sessions × 6 round trips each (deployed FiledropEmbedded 0.1.17);
  • every call relayed twice through ContainerNursery, which opens a new backend TCP connection per call;
  • stream data sent as base64 inside JSON.

  Total work per request is what caps throughput. Evidence: tcpdump showed 1,188 storage request/response pairs over 596 new TCP connections in 9 s of 20-way load. The thread dumps are in tinysite-latency/FINDINGS.md in the artifacts branch.

• Fixes:

  • fewer storage calls per filedrop operation: FiledropEmbedded PR 18, merged;
  • single-round-trip small reads in storage: SimpleFileSystemServiceServer PR 31;
  • a TinySite per-site cache plus parallel prefetch: TinySiteWui PR 6.

• Measurement: nothing has been deployed yet, so production speed has not been measured.

Parallel uploads (8 at once)

• Cause of the serialization: a per-drop lock in FiledropEmbedded. The first lock-free attempt was worse: two whole-document compare-and-set writes of /listing.json per commit made conflicts grow as N² (N=32: 1,112 listing writes, 1,048 conflicts).
• The redesign in FiledropEmbedded PR 19 (head 597b17b, version 0.1.23):

  • Group commit: one in-process publisher each for every manifest, the listing and the index. A leader folds all queued changes into one read and one write, and no lock is held during storage I/O.

  • Deletion and expiry are ordered inside the manifest's change sequence. A published commit is never undone.

  • Measured with a 10 ms per-call harness, commits into one drop:

    Concurrent commits	Before	After
    N=8	1,683 ms	390 ms
    N=32	6,221 ms	969 ms

  • Local suite: 179/179.

• The upload script (claude-code-skills PR 160) already does 8 concurrent uploads. Merge it only after the filedrop changes are deployed and a real 8-way publish is verified.

Dropped answers in the connection library (blocks storage PR 31)

• What happens: PR 31's concurrency test fails in CI with RPC request '__stream_pull' failed: OUTBOX_FULL.
• Mechanism: in UrlProtocol (foundation.url), once queued response bytes exceed 4 MiB per connection (8 MiB per process for libp2p, 16 MiB for TCP), a completed handler answer is replaced by OUTBOX_FULL and dropped. It is not replayed. See Libp2pHostFactory.kt ~395-421 and TcpServer.kt ~80-139.
• It is not PR 31's fault alone. The unchanged main session path also fails: 32 concurrent 1 MiB reads gave 36/64 failures. PR 31 only makes it happen at lower load.
• Fix in progress: branch fix/awaited-response-backpressure-2026-10-16 in UrlProtocol. The goal is backpressure (the answer waits for queue space) instead of dropping the answer. An older open PR, https://github.com/CodexCoder21Organization/UrlProtocol/pull/696 (wip/outbox1c-fix), is in the same area and must be reconciled with this work, not duplicated.
• PR 31 must not have its test weakened. It lands after the library fix is released.

Build-service (url://buildtest/) CPU burn

• Cause: projection-journal compaction rewrote the whole baseline far too often as history grew.
• Fix: an amortized trigger, so compaction runs only when the tail reaches a ratio of the baseline.
  • Module: BuildTestProjectionJournal PR 8, head 0377f82.
  • Consumer: BuildTestEmbedded PR 1490, head 4707afb, which pins candidate buildtest-projection-journal:0.0.14-cpuamortize2-rc1 (SHA-256 317a281e393a48fd0505fa7e965d9541b3b96574c075aef11a2480528c3de3c9).
• Review round 2 found that staging deltas were held fully decoded in heap. That is now streamed, with a fail-first heap test. Both PRs need a round-3 re-review.
• A final module version must be chosen and published before merge. 0.0.11 has since been taken by another lane.
• Do not deploy buildtest without the owner's OK.

Build-service capacity starvation, and the racing-CI rollout

• Symptom: remote runs sat in PENDING for over 60 minutes with "Waiting for droplet capacity". That got PRs evicted from merge queues with checks_timed_out.
• Cause: about 120 of the 200 droplets were held by 14 orphaned "workspace-snapshot" remote test runs from repos still pinned to kompile-cli 0.0.100. Versions ≤ 0.0.105 never renew a lease and never stop the remote run when the client is killed; 0.0.112 and later do both.
  • All relevant repos are now on kompile-cli 0.0.124.
  • The orphans expire at their 3-hour cap.
  • Evidence: buildtest-capacity/FINDINGS.md and kompile-orphan/FINDINGS.md.
• Remaining client gap: when the race's remote side passed its 30-minute wait limit, the run was handed to --attach and left running unrenewed. Fixed in kompile-cli PR 192 (CI green, under review).
• Server-side follow-up, not started: BuildTestEmbedded BuildWatchdog.kt should release a run's droplets once its lease goes stale. Constraint: only for runs that renewed at least once, because clients ≤ 0.0.105 never renew. The lease RPC is renewBuildLease {runId}, sent every 5 minutes.
• Racing CI rollout (done): HandoffEmbedded PR 29, HandoffServiceServer PR 33 and BuildTestProjectionJournal PR 12 are merged. Each adds a bld-build workflow running --local-and-remote on kompile-cli 0.0.124. I then changed each repo's main ruleset to require bld-build instead of kotlin.build (remote). The original rulesets were saved as rs-<repo>.json in that session's scratch directory (not durable). To restore one, the only change is putting back {"context":"kotlin.build (remote)"} as the required check.
• Agent ground rule now in force: always pass explicit --local to scripts/test.bash and scripts/build.bash. Never wrap --remote runs in timeout and never kill them.

Handoff-service memory fix (merged, deploy NOT done)

• Change: archived handoff bodies moved out of state.json (held in heap) into per-handoff files, written atomically, fsynced, and checksum-verified on read. The rollback copy is .pre-archived-bodies, and it is atomic and format-aware.
  • HandoffEmbedded PR 28: merged 01:35Z, publishes handoff:embedded:0.0.39.
  • HandoffServiceServer PR 32: merged 2026-10-16 23:10Z, pins handoff:embedded:0.0.39 (verified on main).
• Deploy needs the owner's approval (question below).
• A slower residual leak (6–8 MB per 75 minutes, suspected to come from status-report uploads) is not investigated.

TinySite management lists (TinySiteWui PR 5)

• Head cca41ba is on GitHub. The author lane has 2 newer local commits, pushed to wip/tinysite-usage-handoff-2026-10-17 (a3cdb9c): a README note plus a restored per-site Throwable catch.
• Round 2 review (4 findings) was addressed in round 3:
  • tooltips clipped at 375 px;
  • shutdown waiting on storage;
  • non-site drops persisted;
  • tooltip times shown in UTC only.
• Needs a round-3 re-review.
• It conflicts with PR 6: both change the ServeSiteServlet constructor and answerMissingFile. Whichever lands second must still record every served file, including cache hits and 304s, and must not record prefetch reads.

TinySite prefetch (TinySiteWui PR 6)

• 9 review findings, being fixed on wip/tinysite-prefetch-r2-2026-10-16. The two that must be fixed:
  • a slow visitor holds the per-file fill lock (a second visitor waited 31 s);
  • a file replaced through filedrop gets served truncated, because size and ETag come from a listing up to 30 s old.
• The reproducers are in the artifacts branch under review-ts6/repro/tests/.

Relevant PRs / refs

Merged this effort:

• https://github.com/CodexCoder21Organization/FiledropEmbedded/pull/18 (01:40Z)
• https://github.com/CodexCoder21Organization/FiledropEmbedded/pull/20 (00:01Z): moves test pins to simplefilesystemservice-embedded 0.0.31. It fixes the flaky testUpdateDropRacingDeleteDropNeverResurrects.
• https://github.com/CodexCoder21Organization/HandoffEmbedded/pull/28
• https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/32
• https://github.com/CodexCoder21Organization/HandoffEmbedded/pull/29
• https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/33
• https://github.com/CodexCoder21Organization/BuildTestProjectionJournal/pull/12
• Earlier: Good-UI-Designs PRs 102–105, claude-code-skills PR 159, kotlin-build-ci PR 336 (deployed)

Open:

Branches (all verified on the remote with git ls-remote):

Local-only commits that were deliberately not pushed:

• Review checkouts sitting on superseded PR heads: bf193eb (PR 28, pre-rebase), ea91580 (PR 31, older head) and 801363d (PR 1490, older head). All are superseded by the heads listed above.
• The old pre-rebase history of PR 18 under bounded-storage-reads-per-call. Its content merged as PR 18.

Deployed or published but not merged:

• Published candidates (test candidates only, not used by any deployed service):
  • buildtest.projection.journal:buildtest-projection-journal:0.0.14-cpuamortize2-rc1, built from 0377f82, which is not merged (PR 8);
  • the older 0.0.11-cpuamortize1-rc1…rc5, all unused.
• Published and merged: handoff:embedded:0.0.39 is published (POM is served) and merged. It is not deployed: production handoff still runs the older build.
• Nothing from this effort is running in production unmerged. The kotlin-build-ci deploy (kotlin-build-ci-857bd27e-20261016.jar) is built from merged main.
• Re-check the deployed versions with the ContainerNursery CLI (list-routes / route details for tinysite, filedrop, simple-filesystem-vnext, handoff).

Open questions for the owner (asked, unanswered at write time)

1. Deploy the handoff memory fix to production? Plan: back up state.json and the current jar, build from main, deploy via the ContainerNursery CLI, then verify pages and archived handoffs. Recommendation: yes.
2. Stop the 14 orphaned buildtest runs to free about 120 droplets? Recommendation: yes. They expire within about an hour regardless, so this matters less now.
3. Rewrite testListingRebuildDoesNotOverwriteConcurrentPublication in FiledropEmbedded? It runs two processes on one storage root, which simplefilesystemservice-embedded ≥0.0.30 documents as unsupported. It is still pinned to 0.0.20 and passes there. Recommendation: rewrite it to test the same guard within a single process.

Next steps

1. Land the TinySite and filedrop changes:
   • (a) Finish the FiledropEmbedded PR 19 review, fix anything found, and merge.
   • (b) Publish one FiledropEmbedded release containing PRs 18 and 19, verifying its SHA-256.
   • (c) Bump FiledropServiceServer to that release, then redeploy filedrop from main (approved).
   • (d) Land TinySiteWui PRs 5 and 6, whichever is clean first; rebase the other and keep usage recording on the cache path.
   • (e) Redeploy TinySite (approved).
2. Storage speed: finish the UrlProtocol outbox backpressure fix, reconcile it with UrlProtocol PR 696, review, merge and publish. Then bump UrlResolver/consumers as needed, land SimpleFileSystemServiceServer PR 31, publish SFS client 0.1.9+, and redeploy the storage service (approved).
3. Measure: a real 8-file-at-once TinySite publish via the upload script (then merge claude-code-skills PR 160), and page load times for https://tinysite.wasmserver.com/s/adc4c5e569/ before and after.
4. Build-service CPU: run the round-3 review of BuildTestProjectionJournal PR 8 and BuildTestEmbedded PR 1490, choose a final module version and publish it, re-pin, then merge. Ask the owner before deploying buildtest.
5. Capacity: merge kompile-cli PR 192 after a clean review, then publish a kompile-cli release. Write the BuildWatchdog stale-lease reaping change, limited to runs that renewed at least once.
6. Handoff service: deploy once the owner approves.

Operational knowledge

• Merging. The gh pr merge --auto flag is not allowed in these repos. Enqueue with the GraphQL mutation enqueuePullRequest(input:{pullRequestId:"<node id>"}).
• Required checks. HandoffEmbedded, HandoffServiceServer, BuildTestProjectionJournal, FiledropEmbedded, SimpleFileSystemServiceServer, TinySiteWui, FiledropServiceServer and BuildTestEmbedded all require bld-build. The remote kotlin.build checks are informational there.
• Dequeuing. A PR whose branch predates the bld-build workflow cannot be enqueued once the ruleset requires bld-build. Rebase it first with gh api -X PUT repos/<o>/<r>/pulls/<n>/update-branch -f update_method=rebase.
• Machine limits. Only one full scripts/test.bash suite can run at a time on the /code box. The remote buildtest service may be saturated, so prefer --local for verification.
• Headless Chromium on aarch64. Set LD_LIBRARY_PATH=~/.cache/chromium-deps/usr/lib/aarch64-linux-gnu:~/.cache/chromium-deps/lib/aarch64-linux-gnu. WebGPU needs SwiftShader flags.
• HCF CLI. Certs are under ~/.config/hardware-control-fabric/. Run it with coursier, not cs. Never start long-lived processes via HCF launch, which puts them in a 512 MB cgroup.
• Delegates. A good next-step decomposition is one agent per PR, each carrying its PR to clean review before taking the next. The lane notes in the artifacts branch hold the full prior context for each PR.

Folded-in effort (2026-10-17 02:58 UTC): the TinySite large-file fixes

The owner asked (02:47 UTC) to fold url://handoff/handoffs/hf-2026-10-17-land-the-tinysite-large-file-fixes-tinysitewui-pr-7-filedropembedded-pr-22-filedropapi-pr-6-deploy-them-and-prove-a-35-mb-file-serves-whole into this effort and drive it to completion. That handoff's body remains the detailed record (mechanism: TinySite read files in 4 MiB ranges, the sandboxed Filedrop client splits anything over 524,286 bytes into serial round trips inside one call, and UrlResolver abandons a call after 30 s, so a 35 MB file stopped at exactly 16 MiB under HTTP 200). Remaining, now owned here:

• https://github.com/CodexCoder21Organization/TinySiteWui/pull/7 (head 1c8aefd, bld-build SUCCESS): independent review running (branch wip/review-ts7-2026-10-17 for reproducers). Must rebase over PR 5 and PR 6 when they land; the three PRs touch the same servlet.
• https://github.com/CodexCoder21Organization/FiledropEmbedded/pull/22 (head f747fd0): queued. F1 replacement-length tests, F2 README/CHANGELOG, version 0.1.22+, full suite, re-review.
• https://github.com/CodexCoder21Organization/FiledropApi/pull/6 (head 4ebec2f): queued. Fold branch wip/t4-pr6-review-doc-fixes (72be28b), re-review, then publish filedrop-api 0.0.7 (artifact path filedrop/api; byte-verify).
• FiledropServiceServer: open a PR from download-range-uses-api-limit (c355de2) plus the T1 test on wip/t4-fss-concurrent-range-test (eb08d16), pinning FiledropApi 0.0.7 and the FiledropEmbedded release that carries PRs 18, 19 and 22.
• Deploy filedrop-server and tinysite-wui (approved), then prove a >16 MiB file (ideally 35,125,250 bytes) uploads and serves byte-for-byte.
• Content Decryption capture (Good-UI-Designs wip/content-decryption-zerotrillion, 346e4b3; review copy https://tinysite.wasmserver.com/s/f185e67226/ until 2026-10-23): needs the owner's approval before a PR. Nothing from that effort is published or deployed (filedrop-api 0.0.7 and filedrop-embedded 0.1.21+ all return 404 at 02:50 UTC; live jars listed in that handoff).

Mode change (02:47 UTC)

The owner switched this session to Fable and asked that all delegates be codex gpt-6.1-sol. Every Opus delegate was stopped at 02:47-02:50 UTC; their checkouts were swept (only unpushed item: a kompile-cli fail-first test, now on wip/kc192-fix-r2-handoff-2026-10-17 @ 8793776). Six sol delegates were launched at 02:54:54 UTC; briefs are in the orchestrator scratchpad p/*.full.md and mirrored on the PlanRepository artifacts branch at the next status cycle. The queue is in /home/leslie/lanes/QUEUE.md (volatile) and in this body's Next steps.

Added requirement (owner, 2026-10-17 03:15 UTC): prove filedrop is working at the end

The owner asked: "This effort should also ensure at the end that filedrop is working (not down) since filedrop was touched throughout this effort." Completion therefore requires, after the filedrop deploy, a live end-to-end check and its evidence in the final report: (1) url://filedrop/ answers (ContainerNursery container RUNNING, no OutOfMemoryError or restart in its container log since the deploy); (2) an upload through TinySite's publish path and through the filedrop WUI/CLI succeeds; (3) the uploaded files download byte-for-byte through TinySite and through filedrop directly; (4) a drop listing and a delete work; (5) the 8-file parallel publish and the >16 MiB file checks from the mission. Also run the production health checks for filedrop and TinySite (production-health-checks skill) and record the results. If any check fails, filedrop is treated as down: investigate with the investigate-broken-service skill before any restart, and do not close the handoff.