Original request (2026-06-11, verbatim): *"It looks like the https://dependabot.wasmserver.com/ site is down, let's fix that."*
Moves it out of every priority list and into ArchiveArea.
pageToken='' first-load regressionWritten: 2026-06-16 (authoring date). Thread began 2026-06-11.
⚠️ RE-VERIFY BEFORE ACTING. Every state claim below was true only at write time. PRs merge, CI re-runs, services restart, container logs rotate, and this host's memory/load picture changes hour to hour. Re-check before doing anything:
- PRs:
gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt- Site:
curl -sS -o /tmp/x -w "%{http_code}\n" --max-time 90 https://dependabot.wasmserver.com/ ; grep -c '<tr' /tmp/x(a data page has ~600+<tr>rows; the "Loading - Dependabot"<title>or a 503 means it is not serving data)- Host:
ssh -p 23 root@198.199.106.165then read/root/ContainerNursery/data/container-logs/dependabot-wui-*.stderr.logA fresh query always wins over this document.
Original request (2026-06-11, verbatim): "It looks like the https://dependabot.wasmserver.com/ site is down, let's fix that."
The investigation expanded well beyond the site itself, in this order:
NoClassDefFoundError on every data fetch (missing dependency pins).pageToken='') that keeps the site from serving data even though #1–#3 are addressed.foundation.url:resolver:0.0.497's SandboxedConnection unconditionally loads community.kotlin.observable.DerivedObservable on every RPC dispatch (via RpcObservableManager). The 0.0.497 bump in both dependabot repos never added the resolver's declared compile deps to the flat MavenPrebuilt2 lists, so the fat jars lacked community/kotlin/observable/DerivedObservable.class and every sandboxed fetch threw NCDFE. Broken since the Apr 30 deploy.
First load failed: java.lang.NoClassDefFoundError: community/kotlin/observable/DerivedObservable; unzip -l of the deployed jar showed 0 community/kotlin/observable/ entries.community.kotlin.observable:core-jvm:0.3.6 + community.kotlin.clocks.hierarchical:community-kotlin-clocks-hierarchical:0.0.2 to both build.kts dependency lists. Server bumped dependabot.server:dependabot-server-service 0.0.17→0.0.18; WUI bumped dependabot.wui:dependabot-wui 0.0.22→0.0.23. WUI's DependabotClient also gained a DEPENDABOT_SERVICE_URL env override (default url://dependabot/) for testability./root/ContainerNursery/apps/dependabot-wui.jar, dated Jun 12 16:22 — a redeploy from merged main, not the session's hand-built jar) does bundle DerivedObservable.class. The NCDFE is gone from production logs.Containers enter CN's activeContainers map before start() spawns their OS process, so isAlive() is legitimately false during the whole cold-start. checkInactive() dead-swept STARTING containers ("process is dead, cleaning up" with no kill output = the silent pre-spawn kill no-op), evicted the route's tracking, and start() then ran the JVM unsupervised while the next request cold-started a duplicate — a memory death-spiral (6+ leaked 130–200 MB WUI JVMs exhausted the 4 GB host; load hit 30–70). Both dead-cleanup paths also evicted tracking even when kill() threw (= could not confirm death).
isAlive() at cleanup time, evict only when kill() returns normally. Reproduced red→green with StartingContainerZombieLeakTest (3 JUnit tests) + the rewritten kompile tests/testStartFailure_unconditionalRemoveRemovesReplacement.kts.main.cn-watchdog.log shows gaps of hours) vs. every 25–60 min on 06-11; zero duplicate containers observed since the deploy.Eclipse MAT on a preserved 128 MB heap dump showed the heap dominated by unbounded Netty/libp2p queues (event loops / ChannelOutboundBuffer / queued gossip jobs). Upstream fixes landed: UrlResolver #611 (bounded gossip send queues, resolver 0.0.543, MERGED 06-12) and UrlProtocol #294 (inbound substream admission, CLOSED). The deployed CN still pins protocol 0.0.281 / resolver 0.0.526, which predate all of these — so those fixes are not yet in production. This is why CN still OOM-restarts (just far less often now).
Invalid pageToken: '' on WUI first load — OPEN, this is why the site is down nowhttps://dependabot.wasmserver.com/ serves only its "Loading - Dependabot" interstitial (0 data rows), or a hard 503; it is not serving repository data. Multiple checks: HTTP 200 loading-page, then a 503 with ~55 s TTFB./root/ContainerNursery/data/container-logs/dependabot-wui-d5aed04c.stderr.log this session):
RPC request 'listRepositoriesByRefreshTime' failed: INTERNAL_ERROR
- Invalid pageToken (expected '<millis>:<owner/name>'): ''
cause: java.lang.InternalError: Exception ... foundation/url/sjvm/intrinsics/ServiceBridge:rpc
listRepositoriesByRefreshTime(0, Long.MAX_VALUE, null, 100). The null pageToken is marshaled to '' across the SJVM sandbox boundary, and the server's pageToken validator accepts null but rejects '' as malformed. That container's log showed 0 First-load-complete / 2 First-load-failed.
dependabot-wui-d5aed04c.stderr.log I read has since rotated (a later grep for the string returned empty because the file cycled). Re-confirm by curling the site to force a cold-start container, then reading the newest dependabot-wui-*.stderr.log.listRepositoriesByRefreshTime to getRepository, writing at the time that "the production WUI uses the resolver's own transport, which doesn't have that quirk." That was wrong — production fails with the identical error on the real resolver transport. The 621-row success I verified on 06-11 came from the hand-built jar's warm cache; the Jun-12 redeploy fails this deterministically on every cold cache.'' coercion happens in the SJVM marshaling of a nullable String? argument. Not yet traced to a specific line.| Ref | State (write-time) | Notes |
|---|---|---|
| ContainerNursery #458 — zombie-leak fix | MERGED 06-11 17:38 (merge sha c6476d549be0) | deployed |
| DependabotServerService #26 — observable/clocks pins + e2e | MERGED 06-11 17:25 (391264d9a61d) | 0.0.17→0.0.18 |
| DependabotWui #44 — observable/clocks pins + closure test + env override | MERGED 06-11 17:25 (fe7225087442) | 0.0.22→0.0.23 |
| UrlResolver #611 — bounded gossip queues | MERGED 06-12 05:43 | resolver 0.0.543 |
| UrlProtocol #294 — inbound admission | CLOSED | |
| ContainerNursery #459 — CN self-OOM | CLOSED | superseded by upstream fixes |
| DependabotWui #46 — all-or-nothing first load | OPEN | amplifier (see next steps) |
No uncommitted/unpushed work is outstanding: all fix branches are merged, and the session's local workspace checkouts have been cleaned. There is nothing unsaved to preserve.
dependabot-wui-*.stderr.log still shows Invalid pageToken ... ''.pageToken='' first-load failure (top priority — this is what keeps the site down). Recommended at the layer that owns the defect: in DependabotServerService, make listRepositoriesByRefreshTime's RPC dispatch / validator treat an empty-string pageToken as equivalent to null (first page). One lenient check at the validation boundary. Ship as a new PR (the original #26 is already merged), following the standard bug-fix workflow (reproducer first). Alternative: fix the null→'' coercion in the SJVM marshaling — broader blast radius; prefer the server-side lenient fix unless the marshaling bug affects other services.listRepositoriesByRefreshTime-based probe in DependabotServerService/tests/testSandboxedClientListsRepositoriesFromFatJar.kts (the merged version was switched to getRepository to route around this bug). It fails pre-fix with Invalid pageToken and passes post-fix. Redeploy the server jar after merge.198.199.106.165 is a 1-vCPU/4 GB droplet that ALSO serves kotlin.directory (Maven) and the buildtest CI backend. So when it is loaded, local kompile builds AND the kotlin.build (remote) PR check insta-fail with read-timeouts (a circular dependency). Consequences: (a) stagger CI re-triggers one PR at a time; simultaneous checks self-interfere. (b) The site self-heals within minutes once you stop generating verification load — verification traffic itself keeps the host above the ~load-20 threshold where the WUI first load can't finish.kotlin.build (remote) flake signature: a kompile remote run dies at PROVISIONING with Sandboxed code threw ... ServiceBridge:rpc ... InterruptedException — that is buildtest's own sandboxed RPC to url://digitalocean-droplets/ hitting the 30 s dispatcher watchdog, recorded server-side as the build's failure. Not your test./root/ContainerNursery/data/container-logs/dependabot-{wui,server}-*.{stdout,stderr}.log. They rotate — read the newest by mtime (ls -t ... | head -1)./root/ContainerNursery/{bin,apps}/*.bak-20260611 (CN + both dependabot jars)../gradlew jar --offline (Gradle dep cache primed by an earlier run), then on the host rebuild the fat jar with a python zipfile script (drop all old org/example/**, insert new), mv over bin/container-nursery.jar (new inode), kill CN → /root/cn-watchdog.sh respawns it. Verify the changed .kt file is otherwise unchanged since the deployed build and the build.kts dep pins are identical. Kotlin 2.0.21 Gradle classes link fine against the fat jar's bundled 1.9.22 stdlib.URL_BIND_DOMAIN=localhost.tcp:<port> + PORT, dial with the protocol version bundled in the jar's raw-RPC API (sendServiceRpcRequest(url, method, params) for protocol 0.0.264; openPersistentRpcConnection only exists in 0.0.281+), fetch __bytecode_request, then SandboxedConnection.createWithSjvmDispatch(iface, implJar, implClassName, id, rpcHandler). Gotchas: (1) kompile materializes build-rule artifacts at temp paths with no .jar extension — locate them on the classpath by zip content, not filename; (2) guard child-stdout drain threads with try/catch (IOException) — the testrunner reports uncaught drain-thread exceptions as the test failure; (3) UrlResolver.openSandboxedConnection cannot dial url://localhost.tcp: (P2P-discovery only → "All 0 peers") — a resolver-based consumer (the WUI) can't be round-trip-tested against a local URL-mode server without a CN facade, so assert the consumer artifact's dependency closure (zip-presence of DerivedObservable etc.) and point to the provider repo's behavioral e2e.dependabot-ncdfe-observable-pin, cn-zombie-leak-fix-and-self-oom.