TinySite upload-begin hang (2026-10-24): resolver 0.0.1337 can't be shown to fix it, and the mechanism is still unknown

Written 2026-10-25 ~20:30 UTC. RE-VERIFY: everything below is a write-time snapshot; re-check branch heads with git ls-remote, deployed JARs with the ContainerNursery CLI (routes --json), and whether TinySite upload-begin still answers before acting.

Design decisions in force

• Owner, 2026-10-25 (verbatim): "You should test it, if migrating those to the new url resolver doesn't solve the problem, we need to understand the underlying issues and writeup a detailed handoff."
• Owner, 2026-10-23: bumping a consumer to an already-published UrlResolver/UrlProtocol release that fixes a bug is approved; no deep work inside UrlResolver/UrlProtocol in this effort.
• Production deploys only on the owner's explicit ask.

Why this exists

On 2026-10-24 at 04:29Z a 1 MB TinySite upload failed at its first step, POST /api/upload/begin. After 30.3 s the browser got ContainerNursery's 503 Service Unavailable: Read timed out before any data was sent. The owner's direction (2026-10-25, verbatim): "You should test it, if migrating those to the new url resolver doesn't solve the problem, we need to understand the underlying issues and writeup a detailed handoff."

Bottom line: (1) the failure does not happen in production today, (2) none of three local experiments on the real TinySite -> filedrop -> filesystem chain reproduce it, on the deployed pins or on resolver 0.0.1337, so (3) we cannot show that 0.0.1337 fixes it. The mechanism is still unknown. The evidence narrows the hop and the class of defect, and the next step needs evidence from the next occurrence (section 7).

1. Symptom (from the 2026-10-24 investigation; the original files were lost when the machine was reset)

• TinySite's request thread waited in BeginUploadApiServlet at src/tinysitewui/ApiServlets.kt:392, which calls requireSite(client, siteId) (src/tinysitewui/DeployApi.kt:119). That calls FiledropClient.getDropOrNull, then the sandboxed proxy (SandboxedProxyGenerator.kt:1398), whose worker waited for a url:// RPC reply (UrlResolver.kt:31764, resolver 0.0.1329).
• At the same moment, filedrop was waiting on a pending filesystem RPC (v2_readStart), and every sampled filesystem storage worker was idle. No request-ID correlation proves it was a single chain. The request, or its reply, was apparently lost between filedrop and the filesystem service.
• The timeouts explain why the user saw ContainerNursery's 503 and not an application error:
  • TinySite -> filedrop uses the default sandbox method timeout of 30 s (SandboxedProxyGenerator.kt:1762 in 0.0.1337; :1439 in 0.0.1306).
  • filedrop -> filesystem uses 180 s (FiledropServiceServer src/filedrop/server/Main.kt:313).
  • ContainerNursery cuts the backend off at 30 s.
  • So any filedrop -> filesystem stall over about 30 s surfaces as ContainerNursery's 503.

2. Versions

Candidate: resolver 0.0.1337 with protocol 0.0.1434 (UrlResolver main c7899aeb). JAR SHA-256 is d50db4be...3f83132, matching the brief when fetched into an empty coursier cache. The 0.0.1337 run fetched resolver-0.0.1337.pom, and TinySite's buildMaven() output changed from 85 files / 45,278,304 bytes to 86 files / 47,302,385 bytes (it adds the resolver's cache module).

All three routes have keepWarmSeconds: 300. filedrop uses FILESYSTEM_SERVICE_URL=url://simple-filesystem-vnext/.

3. Production today (exactly what was sent)

Three TinySite production requests were sent in total, the limit:

1. 2026-10-25 19:26:56Z (previous worker): POST https://tinysite.wasmserver.com/api/upload/begin with siteId=l116-readonly-probe-20261025 (a site that does not exist). Answered in 1.6 s: {"error":"Site 'l116-readonly-probe-20261025' does not exist or has expired."}.
2. 2026-10-25 19:48:30Z: POST /api/sites with title=l123-check-20261025&expiresInSeconds=3600. Answered HTTP 200 in 2.16 s: {"siteUrl":"/s/c74cd96261/","pageUrl":"/site/c74cd96261","id":"c74cd96261"}.
3. 2026-10-25 19:53:59Z: POST /api/upload/begin with siteId=c74cd96261&path=index.html&totalSizeBytes=1. Answered HTTP 200 in 7.79 s: {"chunkSizeBytes":2097152,"uploadId":"c74cd96261.1792958039933-d4d6e4ef-a259-45c3-8c47-fc1dcced46e9"}.

So the failing step, upload-begin on an existing site, works today. Site c74cd96261 was not deleted because that would have taken a fourth request. It expires on its own around 20:48Z, and the upload session c74cd96261.1792958039933-d4d6... was left unfinished. 7.79 s is slow but within the limit, and nothing showed where that time went.

Read-only ContainerNursery log reads (CLI 0.0.23, container-logs / logs; no restarts or configuration changes):

• Logs from the incident are gone. A --since filter for 2026-10-24 returns today's lines, both for the containers and for the ContainerNursery server log, so container/server log retention no longer reaches back to 04:29Z.
• The filesystem (vnext) service gets steady read traffic. It logs about 10 read cycles per second: v2_readStart, v2_readChunkStreamWithState, __stream_pull ×3, __stream_close, v2_readClose. The caller was not identified.
• filedrop keeps re-checking peers that have left. Its log is dominated by Reachability verification for peer '…' encountered ~995 failed attempts for many departed peers, plus Pending dial controller failed to construct transport dial, StacklessClosedChannelException on write, and Closing root connection … reason=ADMISSION_RETIREMENT. This looks like the departed-peer pruning defect fixed in resolver 0.0.1326 (UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1318), which filedrop's 0.0.1306 lacks.

4. Local experiments (real components: separate provider JVMs, real libp2p url:// connections, real Jetty TinySite; no mocks, no reflection)

How it works: the filesystem and filedrop "hosts" are small main() wrappers around the real SimpleFileSystemRpcHandler / LazyFiledropRpcHandler + FilesystemServiceConnections. Each is built from its own repository, so it keeps that repository's resolver pin. TinySite runs in the test JVM through createServer and FiledropClient.

Reproduce the setup:

1. Build the hosts: scripts/build.bash --local filedrop.server.buildL118Host <out> and scripts/build.bash --local simplefilesystemservice.server.buildL118Host <out>.
2. Copy them next to TinySiteWui/ as l118-filedrop-host.jar and l118-storage-host.jar.
3. Run scripts/test.bash --local --test <name> --log <xml>.

Cache trap: the buildL118ProviderResources() cache key does not cover the JAR bytes. Edit its comment line in build.kts whenever the hosts change. One run silently reused stale hosts until this was done.

Results are first attempts, read from the XML; none of the passing runs has any attempt history:

These are single runs. They show that none of these conditions reproduces the hang, not that 0.0.1337 removes it.

Side findings (not the production symptom):

• 64 concurrent readers run the TinySite test JVM out of heap. The first serve-amplifier version used 64 concurrent 2 MiB range readers. It failed with OutOfMemoryError in TinySite's persistent RPC reply reader while decoding a JSON response frame (JsonWireEncoding.kt:63 <- Libp2pRpcProtocol.readJsonFrame <- PersistentRpcResponseReader.readResponse). Resolver 0.0.1329 reported this to every pending caller at once ("Persistent RPC connection to service 'filedrop' was closed while requests were still pending. Reader transport failure: java.lang.OutOfMemoryError"). That is prompt failure, not a hang.
• A restarted provider with a new identity fails fast too. When the restarted filesystem provider came back with a new peer ID and port (the stale-host run), filedrop 0.0.1306 failed promptly with Failed to reconnect persistent RPC connection to service 'l118-filesystem'. Original peer … is unreachable … Connection refused. A provider restart with a new identity therefore gives a prompt error locally, not a hang. The local fixture has no discovery node, so it cannot test whether production rediscovers the new provider.
• The 0.0.1337 chain run was about twice as slow (104 s against 49 s). Other builds were running at the same time, so this is not attributed. Measure it alone before relying on it.

5. What the evidence still points at

• The stuck hop is filedrop -> filesystem, not TinySite -> filedrop. filedrop was waiting on v2_readStart while the filesystem service's workers were idle. Upgrading only TinySite's resolver therefore cannot address it: filedrop (caller) and the filesystem service (provider) both run resolver 0.0.1306 (UrlResolver 7e5408898b, 2026-10-13). That is about 120 resolver commits behind 0.0.1337.
• Resolver changes after 0.0.1306 that match "a written request is never answered and the provider looks idle" (none is proven to be the cause):
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/1163 (merged 2026-10-23, after 0.0.1333; in 0.0.1337): handler waits used to occupy the 64-slot transport dispatcher. With parked long-polls such as __stream_pull or effect sessions, a new request is not dispatched while the provider's application workers sit idle. That matches "storage workers idle while the caller waits". Relevant code in 0.0.1306: INBOUND_DISPATCH_MAX_THREADS = 64 with CallerRunsPolicy on the reader thread (UrlResolver.kt ~27801–27820 at 7e5408898b); PROVIDER_RPC_READER_MAX_STREAMS = 64 (~1884–1910).
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/1278: a request rejected before dispatch, or stopped during metadata preparation, left caller idle cleanup cancelled. The caller then keeps a persistent connection past the provider's 120 s stream idle close (RPC_STREAM_IDLE_TIMEOUT_SECONDS = 120, UrlProtocol Libp2pHostFactory.kt:71; the caller idles out after 60 s, idleTimeoutMs = 60_000, ~27652).
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/1353 (replay frames declined before admission) and https://github.com/CodexCoder21Organization/UrlResolver/pull/1349 (provider reader admission returned before close notifications).
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/1314 (persistent RPC relay fallback and transport retirement), https://github.com/CodexCoder21Organization/UrlResolver/pull/1396 (circuit routes), https://github.com/CodexCoder21Organization/UrlResolver/pull/1318 (departed-peer pruning; matches filedrop's current log), https://github.com/CodexCoder21Organization/UrlResolver/pull/1572 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1573 (prompt failure when a provider leaves).
• Ruled out or weakened:
  • UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1185 (one reply reader) does not apply to production. Its fix is only on the injectedNetwork path (openPersistentRpcConnectionWithinDeadline, the if (injectedNetwork != null) branch at UrlResolver.kt ~14395 at c7899aeb), and production builds native default resolvers.
  • Partial reads are not the cause. The filesystem client's chunk reader loops until it has the full length or hits EOF, so 0.0.1337's 8 KiB partial reads do not break it.
  • Not reproduced locally: a lookup of a missing site, plain concurrency, abandoned downloads, and a provider restart with the same or a different identity.
• Neighbouring work: SimpleFileSystemServiceServer has open https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/35 and https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/31. They describe a speculative prefetch that blocks inside Source.read and occupies the server's replay cache during uploads, so a competing reader fails cache admission. That is a filesystem-side read problem during uploads. It fails rather than hangs in their description, but it sits on the same v2_readStart path.

6. The exact remaining unknown

Whether filedrop's v2_readStart frame reached the filesystem process at all. Three things would answer it:

• (a) the filesystem service's URL request: method='v2_readStart' log line with a timestamp, set against filedrop's pending-request start;
• (b) whether the provider's 64 inbound dispatch threads or 64 provider reader admissions were all occupied by parked __stream_pull / long-poll handlers at that moment;
• (c) whether filedrop's persistent RPC transport to the filesystem service was older than 120 s, past the provider's idle close.

Without the incident thread dumps and logs, none of these can be decided now.

7. Next steps

1. Capture the next occurrence. Within seconds, take jstack of all three JVMs and save container-logs for url:filedrop: and url:simple-filesystem-vnext: before log rotation. In the filesystem dump, count threads named rpc-inbound-dispatch and url-resolver-provider-rpc-* and record what each is blocked on. In filedrop's dump, record the pending request's transport generation and age. This decides 6(a)–(c).
2. Build a discriminating reproducer. Extend testL123ConcurrentServeChain so it holds 64 or more concurrent __stream_pull long-polls open on the filesystem provider, for example readers that open a stream and stop pulling, from filedrop 0.0.1306, then issue getDropOrNull. If baseline hangs and 0.0.1337 does not, that confirms the UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1163 mechanism. Also add an idle-gap variant: a request rejected before dispatch (for example an expired deadline), then 130 s of idle, then getDropOrNull. That tests UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/1278.
3. Bump filedrop and the filesystem service, not only TinySite. If a bump goes ahead on its own merits (approved in principle for released versions), FiledropServiceServer and SimpleFileSystemServiceServer are the services on the suspect hop. TinySite at 0.0.1329 -> 0.0.1337 alone cannot change the filedrop -> filesystem behaviour. The pin edits are already prepared on the branches below; full suites were not run for them in this lane.
4. Add an end-to-end bound so a stall surfaces as a named error. No deadline shorter than ContainerNursery's 30 s exists on the TinySite path. Do not raise any timeout. Instead find the stall, so the failure shows up as a named error and not a ContainerNursery 503.

8. Branches (all pushed; diagnostics only, not for merge)

Evidence files (local, volatile), in /tmp/claude-501/-code/b18cd447-2305-423b-ac30-50b4949d9abc/scratchpad/out/:

• test results: L123-.xml / L123-.log
• production response bodies: L123-prod-*.body
• ContainerNursery log excerpts: /tmp/claude-501/-code/b18cd447-2305-423b-ac30-50b4949d9abc/scratchpad/lanes/L123/logs/

Nothing was deployed, restarted, reconfigured, merged or enqueued. No PRs were opened, because the bump could not be shown to fix the symptom.