Propagate sandbox exception root causes end-to-end (SJVM ? RPC ? WUI error page) instead of the generic "Exception encountered in implementation" wrapper

Written: 2026-07-19 ~08:5xZ by Claude (supervisor session). RE-VERIFY banner: All state below is a write-time snapshot. Before acting, re-verify against current code (the named classes/lines may have shifted) and re-grep a live incident's logs for the real causes. A fresh query always wins over this document.

UPDATE 2026-07-26 — SJVM end and WUI end both fixed; middle layer re-homed

A scoped campaign worked only the two end layers (SJVM and BuildTestWui); the UrlResolver/UrlProtocol middle layer was explicitly out of scope. The mission below still reads correctly, but three of its factual claims were wrong and are corrected here. Re-verify before acting.

Corrections to the original text (verified live on 2026-07-26)

1. "The real caught Throwable is not attached as the InternalError's cause" — and the proposed fix ("attach the root cause as the cause of the wrapper") is not implementable as written. The wrapper is a guest java.lang.InternalError object living inside the sandbox; a host Throwable cannot be its cause, because the sandbox holds no references to host heap objects. The only channel that legitimately crosses the boundary is that error's message string. The fix landed there instead.
2. The quoted production string — java.lang.InternalError: Exception encountered in implementation with nothing after it — is not a string the code can produce. Since 2023 the message has been "Exception encountered in implementation of <class>:<method> :\n$e cause: ${e.cause}", i.e. it already carried the thrown throwable and one cause level. The handoff quote was abbreviated. What was genuinely missing: cause levels 3 and deeper, and every stack frame.
3. The resolver layer already carries a cause chain, contrary to "the cause chain must be carried across the RPC boundary". UrlResolver src/foundation/url/resolver/sandbox/TypeMarshaling.kt:539-568 (fromJvmResult) reads the guest throwable's getMessage() and walks up to 3 guest cause levels via extractCauseChain (same file, line 154), embedding all of it in the SandboxException message. There is no "INTERNAL_ERROR payload reconstruction" left to build — it is plain string embedding, and it works.

Where the wrapper actually lives (verified)

sandboxjvm (there is no SjvmServer repo), libSJVM/src/commonMain/kotlin/net/javadeploy/sjvm/impl/SJVMThreadImpl.kt, two sites: line 204 (invokeStaticMethodLambda) and line 459 (invokeVirtualLambda). Both wrap a host exception thrown out of a registered method override — which is exactly how UrlResolver implements ServiceBridge.rpc, so every sandboxed url:// call funnels through them.

What now propagates end-to-end

SJVM end — sandboxjvm#97 — MERGED 2026-07-26T04:53:00Z. New describeHostThrowableChain (net.javadeploy.sjvm.util, internal) names every throwable in the host cause chain and appends the deepest described one's frames. Wired into both wrapper sites. Behaviour documented in the repo README under "When an Override Throws".

Proof, from HostExceptionCauseChainTest (15 tests, every one asserting the entire message verbatim; all fail on the old format):

BEFORE  Exception encountered in implementation of test/HostBridge:callHost()I :
        java.lang.RuntimeException: outer RPC dispatch failed cause: java.lang.IllegalStateException: middle transport layer failed
                                                    ? IllegalArgumentException("refCnt: 0") is GONE

AFTER   Exception encountered in implementation of test/HostBridge:callHost()I :
        java.lang.RuntimeException: outer RPC dispatch failed
        Caused by: java.lang.IllegalStateException: middle transport layer failed
        Caused by: java.lang.IllegalArgumentException: refCnt: 0
        Stack trace of the deepest cause described above:
        	at test.HostCode.releaseBuffer(HostCode.kt:42)
        	at test.HostCode.handleRpc(HostCode.kt:17)

Three defects the adversarial review caught in the first draft of this branch — worth knowing about, because two of them are traps anyone rewriting this code would fall into again:

1. Rendering a stack trace aborted the host process on Kotlin/Native. stackTraceToString() there walks the cause chain in a loop that ignores its own already-visited signal, so a self- or mutually-referential cause appends Caused by: [CIRCULAR REFERENCE …] until memory runs out. Reproduced with a native binary built from the repo's own toolchain: Out of memory trying to allocate 994065992 bytes … Aborting., exit 134. Never render a stack trace of an untrusted throwable in commonMain. Frames now come from a hostStackFrames expect/actual reading the throwable's own frames (stackTrace on JVM, getStackTrace() on Native, which needs @OptIn(ExperimentalNativeApi::class)). That also fixed two more text-parsing bugs for free: a multi-line cause message silently deleted the whole stack section (and the deepest cause here is frequently another boundary's multi-line description — the exact shape the feature serves), and suppressed exceptions are indented like frames on both platforms so they were reported as frames of this throwable.
2. The pre-existing e.printStackTrace() on both wrapper paths had to go. It calls toString() on a throwable the sandbox does not control, before any guard, so a throwable with a throwing toString() unwound the whole catch and the guest received a bare UnsupportedOperationException from the caller's frame instead of an InternalError. A test with such a throwable proved it. The description carries the same information, cannot fail, and is logged by createExceptionWithMessage when logging is enabled.
3. Bounding links and frames does not bound size. One throwable may carry an enormous message, and each nested boundary embeds the description it received into the message it raises, so depth multiplies length. The finished string is a guest allocation inside the sandbox heap, once per failed host call (6,713 times during the incident). A 2000-character per-throwable cap closes it.

All three limits — 12 causes, 20 frames, 2000 chars/throwable — state their truncation in the text rather than applying it silently. A cycling chain terminates with a note. An empty stack is reported as none recorded rather than by omitting the section. The whole description is total: every read of cause, message, toString() and the frame list is guarded and a failed read is named in the output.

Local: full :tests:jvmTest green under both the Avian and OpenJDK stdlibs, plus :tests:linuxX64Test green (see the libzip note under "Reusable knowledge").

WUI end — BuildTestWui#145 (error-page-full-cause-chain) — branch CI GREEN, NOT MERGED: repeatedly evicted from the merge queue by buildtest infrastructure, see "BuildTestWui#145 merge-queue status" below. errorPage gained an overload taking the caught throwable; describeFailureChain renders the whole host cause chain inside <pre class="console-output"> so multi-line content keeps its line structure. Applied to the runs list (both views), run detail, cancel, and archive. Three previously-lost things now appear: the cause chain, the throwable class when the message is null (it used to render the literal Unknown error), and line structure. The renderer is total (these throwables arrive across a url:// sandbox proxy whose accessors do throw; a failure while rendering used to replace the page with the container's blank 500), bounded (12 causes, 4000 chars/throwable, truncation stated), and reflection-free (toString() rather than javaClass.name, matching the SJVM side).

Eleven hermetic tests drive a real Jetty server over HTTP against a failing fake API, asserting complete <pre> and <div> elements rather than fragments. They cover both runs views, run detail, cancel and archive (archive had no test of any kind and interpolated the request's run id into an unescaped summary — that hole is closed and pinned), hostile markup in both the message and the run id, a null message, a multi-line description, a cycling chain, and the depth boundary at and over the limit. All six original tests fail against the previous rendering (verified by restoring the old body behind the new signature, purging the kompile artifact cache, and re-running: 0/6). Local: 217/217. Before/after screenshots are embedded in the PR.

The two are independent. The WUI change improves the page against today's deployed backend — it renders whatever chain arrives. The SJVM change makes a richer chain arrive.

BuildTestWui#145 merge-queue status — blocked on infrastructure, not on the change

Branch CI is green. Four merge-queue attempts, each evicted by a different piece of buildtest infrastructure, none implicating this change:

| Queue run | Result | Cause | |---|---|---| | a12e5abf (branch) | 211/212 | goldenScreenshots: "The screenshot service dependency is likely wedged" | | 586ba8da | 216/217 | goldenScreenshots: "Test did not complete during build execution" | | f1b1e383 | 216/217 | goldenScreenshots | | bcba50e6 | 199/217 | "Build failed after service restart: 1 of 8 shard(s) failed during resume: shard 7: Failed to connect to root@147.182.228.187:22 after 12 attempts … Connect timed out" (queue removal reason checks_timed_out) | | 267e4ffd | 0/217 | "8 of 8 shard(s) failed … classification=INFRASTRUCTURE/PROVISIONING … ProvisioningStoppedForTerminalRunException" — no droplet could be provisioned, zero tests ran |

build-watchman auto re-enqueued once and then correctly stopped ("EVICTED from the merge queue AGAIN after an automatic re-enqueue … not re-enqueuing further"). The PR is out of the queue and open.

goldenScreenshots is already failing on BuildTestWui main itself — the 2026-07-24 main run reports "203 passed, 1 failed — goldenScreenshots". Open PR #141 (a different session's) is red for the same reason. So no BuildTestWui PR can pass the merge queue until url://screenshottest/ is healthy again — this is not specific to #145, and it is not something #145 can fix. The buildtest frontend was also 503-flapping throughout (GET /api/test-results timing out) with 12 PENDING / 5 UPLOADING / 3 TESTING runs queued, so the host is saturated as well.

Do not bypass the queue. The next owner should either wait for url://screenshottest/ to recover and re-enqueue, or treat the wedged screenshot service as its own lane. build-watchman auto re-enqueues once per launch, so re-running it is the cheap way to take another attempt: coursier launch buildwatchman:build-watchman:0.0.13 -r https://kotlin.directory -- --repo CodexCoder21Organization/BuildTestWui --pr 145 --to-merged

Re-homed to the resolver chain — NOT done, deliberately out of this campaign's scope

1. SandboxException is thrown with no host cause. TypeMarshaling.kt:568 does throw SandboxException("Sandboxed code threw an exception: $fullMessage") — a string only. So even with both fixes above, a programmatic caller at the resolver boundary must parse text; e.cause is null. (The class does support a cause — SandboxedConnection.kt:~765 uses the 2-arg form for its InterruptedException translation.) Reconstructing a synthetic host cause chain from the guest description is the remaining piece of acceptance criterion (b). Owner: resolver lane.
2. extractCauseChain bounds the guest chain at 3 (TypeMarshaling.kt:561). Harmless today — the guest InternalError is built with the (String) constructor and has no guest causes, so the whole diagnosis rides in the message — but it is a second, inconsistent bound in the same path.
3. ?? LANDMINE for whoever bumps the sandboxjvm pin. UrlResolver pins net.javadeploy.sjvm:*:0.0.49 (build.kts:222-224) and has exact full-text assertions on the old wrapper format, each ending ... cause: null, in: tests/testSandboxRpcResultOneValueOverDefaultLimitIsRejectedBeforeAllocation.kts, tests/testSandboxRpcResultJsonObjectAndArrayAccountingIsCumulative.kts, tests/testSandboxRpcResultNestedListAndMapAccountingIsCumulative.kts. They pass today and are unaffected until the pin moves. The pin bump and those three expected strings must change in the same PR, or that PR is red.
4. Adjacent known middle-layer defect, unchanged by this work: UrlResolver#818 (Any?-typed DTO fields silently dropped over sandboxed connections).

Also noted, not done (BuildTestWui, separate lanes)

• ApiServlet (src/buildtest/wui/ApiServlet.kt:61) and LogStreamServlet (:86) still flatten their error bodies to e.message ?: "Unknown error". Same defect, JSON/SSE surfaces rather than HTML pages.
• The HTML error pages still answer HTTP 200 on a backend failure, while /api/duration-stats answers 500. Changing it would alter behaviour existing tests depend on; left out deliberately.
• The RunDetailServlet swallow issue is a different lane and was not touched.

Release / deployment state

• sandboxjvm keeps version bumps in separate PRs (see #96). #97 does not bump gradle.properties (still 0.0.49), so the SJVM fix is not consumable until someone releases it (? 0.0.50) and bumps the UrlResolver pin (see landmine above), then rebuilds the WUI against it.
• BuildTestWui#145 bumps buildtest.wui:buildtest-wui to 0.0.86. It is not deployed; deploying is a separate, explicit act.
• sandboxjvm CI is unreliable in a way that looks like your fault and is not. Its test job declares timeout-minutes: 50, has a ~20-minute median, and intermittently blows past 50 and is cancelled — including on plain master (2026-07-23T18:53:51Z ? 19:44:12Z, cancelled). Confirm with gh api "repos/CodexCoder21Organization/sandboxjvm/actions/runs?per_page=12" before believing a stall. Written up at challenges/2026-07-26-0428.

Still open in THIS handoff's scope

Acceptance criterion (b) — "the distinctive class+message appears as the client exception's cause" — is not met, and cannot be met at either end layer: it is resolver-layer work (item 1 above). Criteria (a) "in the RPC payload" and (c) "on the rendered WUI error page" are met, verbatim, with chained-cause coverage including the refCnt: 0 shape. This handoff stays OPEN for that residual plus the release chain.

Mission

When sandboxed url:// code throws, the real exception is swallowed behind the generic SJVM message java.lang.InternalError: Exception encountered in implementation, and the user-facing error page shows only that useless wrapper. Fix the whole chain so an exception's root cause is attached and propagated all the way to the error page: the operator/user must see the informative underlying error (class + message, and ideally the sandboxed stack), not a generic wrapper.

Concretely:

1. The root cause must be attached as the cause of any wrapper thrown at the SJVM boundary (not discarded).
2. The cause chain (class names + messages) must be carried across the RPC boundary in the INTERNAL_ERROR payload and reconstructed on the client.
3. The WUI error page must render the root-cause chain, not just the outermost wrapper.

Why this matters (evidence from the 2026-07-19 buildtest incident)

• The error page at https://buildtest.kotlin.build/ showed: Failed to load build runs: Sandboxed code threw an exception: java.lang.InternalError: Exception encountered in implementation — no underlying cause. Diagnosis required SSHing to the host and grepping backend logs.
• Behind the wrapper, the ACTUAL root causes (found only in logs) were:
  • io.netty.util.IllegalReferenceCountException: refCnt: 0 — 6,713 occurrences (a pooled-ByteBuf use-after-free; the real bug — see "Separate follow-up" below).
  • java.lang.InterruptedException — 319x; java.util.concurrent.TimeoutException — a few; url-bind protocol-probe timeouts; and benign business errors (e.g. "Cannot renew build lease — terminal status FAILED").
  • None of these reached the error page. They were all flattened to "Exception encountered in implementation".
• First occurrence in logs: 2026-07-19 00:51:05Z. Affected many calls: listBuildRunsPaginated, getBuildRun, renewBuildLease, WatchmanClient repo invalidation — i.e. all sandboxed url:// calls, not just the WUI.

Where the cause is lost (the chain to fix)

1. SJVM — net.javadeploy.sjvm (the sandboxjvm repo — corrected 2026-07-26). The message java.lang.InternalError: Exception encountered in implementation originates in SJVMThreadImpl.invokeStaticMethodLambda (line 204) and SJVMThreadImpl.invokeVirtualLambda (line 459). FIXED by sandboxjvm#97 — see the update block above for why the original "attach as cause" formulation was not implementable.
2. Resolver — foundation.url.resolver (CodexCoder21Organization/UrlResolver): SandboxedConnection (createWithSjvmDispatch), SandboxedProxyGenerator (InstanceMethodDispatcher.invoke), and the RPC error marshaling that produces UrlResolutionException: RPC request '<m>' failed: INTERNAL_ERROR - Handler error: <msg> and SandboxException: Sandboxed code threw an exception: <msg>. PARTLY ALREADY DONE (the chain text is carried); REMAINING: attach a reconstructed host cause — see "Re-homed to the resolver chain" above.
3. WUI — BuildTestWui (CodexCoder21Organization/BuildTestWui): the error page render (RunListServlet/RunDetailServlet/CancelServlet/ArchiveServlet error paths) showed only the top message. FIXED by BuildTestWui#145. ApiServlet/LogStreamServlet JSON/SSE error bodies still flatten — separate lane.

Acceptance criteria

• Force a sandboxed call to throw an exception with a distinctive message; assert that exact class+message appears (a) in the RPC INTERNAL_ERROR payload, (b) as the client exception's cause, and (c) on the rendered WUI error page — end to end, verbatim. It must NOT be flattened to "Exception encountered in implementation". ? (a) MET and (c) MET (2026-07-26); (b) NOT MET — resolver-layer work, re-homed above.
• Verify with a real chained cause (e.g. an IllegalReferenceCountException): the page shows the netty message, not the wrapper. ? MET — covered by tests at both ends and visible in the PR screenshots.
• Applies to all sandboxed url:// consumers, not just the WUI. ? MET at the SJVM end (the fix is in the shared wrapper both override dispatch paths use), pending the release chain.

Separate follow-up (do NOT conflate)

The dominant actual bug this wrapper was hiding — io.netty.util.IllegalReferenceCountException: refCnt: 0 (a pooled-ByteBuf use-after-free / double-release in the SJVM/RPC path, 6,713x, from 2026-07-19 00:51) — is a distinct defect that deserves its own root-cause + fix. This handoff is ONLY about making errors observable/informative; fixing the refCnt bug is a sibling effort. Link them once both exist.

Reusable knowledge / refs

• Incident context: buildtest.kotlin.build outage 2026-07-19 (gossip storm ? LazyReconnectingMethodDispatcher beginCall lock ? this SJVM wrapper masking). Host 198.199.106.165; logs under /root/ContainerNursery/cn-manual-restart-*.log and /root/buildtest-data/retained-run-forensics/.
• Related handoff: hf-2026-07-19-eliminate-o-n-bookkeeping-... (the Nx gossip redesign).
• Repos: sandboxjvm (net.javadeploy.sjvm), UrlResolver (foundation.url.resolver.sandbox), BuildTestWui.
• Building sandboxjvm locally needs a JDK 8 toolchain that the standard sandbox does not have: curl -sL -o jdk8.tar.gz "https://api.adoptium.net/v3/binary/latest/8/ga/linux/x64/jdk/hotspot/normal/eclipse", extract it, then pass -Porg.gradle.java.installations.paths="$JDK8_HOME" to every ./gradlew invocation. linuxX64Test additionally needs libzip, which apt-get download cannot fetch (no apt lists) and which cannot be installed into /usr/local (not writable). It is reachable unprivileged, and doing so is worth it — it caught a Kotlin/Native opt-in error in 49 seconds that had already cost one 50-minute CI cycle. Fetch the .debs directly (http://archive.ubuntu.com/ubuntu/pool/universe/libz/libzip/libzip-dev_1.7.3-1ubuntu2_amd64.deb and libzip4_...deb), dpkg-deb -x them into a scratch root/, then without committing it prepend -I<abs>/root/usr/include and -L<abs>/root/usr/lib/x86_64-linux-gnu in libSJVM/src/nativeInterop/cinterop/libzip.def, and export LD_LIBRARY_PATH=<abs>/root/usr/lib/x86_64-linux-gnu so the built test.kexe can load the shared object.
• BuildTestWui fail-first proofs need a cache purge. A same-version source edit is invisible to scripts/test.bash: remove both ~/.cache/coursier/v1/bldbinary/repository/buildtest/wui/buildtest-wui/<version> and ~/.aibuildcaches/_code_ws_BuildTestWui between runs, or a reverted fix will appear to still pass. This silently invalidated one fail-first attempt during the 2026-07-26 campaign.
• kompile test-script constraints that cost a round each: a test script's top-level functions may not declare return types, and a test script file may contain only top-level functions — no class declarations (use an anonymous object : expression inside the test instead).

Required engineering rigor (implement it properly — do NOT shortcut like the incident fixes did)

1. Fail-first reproducer/scale test demonstrating the defect (FAILS before the change, PASSES after). Real instances, no mocks; per the Testing Architecture doc.
2. Implement the change at the owning layer.
3. Test-comprehensiveness review + adversarial code review; address findings.
4. Final review (Claude) against the problem, CLAUDE.md, PHILOSOPHY.
5. Drive the PR(s) to green CI and MERGE. This handoff is NOT done until merged.

UPDATE 2026-07-26 ~09:50Z — BuildTestWui#145 is MERGED; the merge-queue blocker was diagnosed and fixed

BuildTestWui#145 merged as a5708fa, merge-queue run d29123cf 217 passed / 0 failed, first attempt. The WUI end of this campaign is landed. BuildTestWui main is green again (run 88013bbb, 206/0).

The "url://screenshottest/ is wedged" diagnosis in the section above is WRONG

The table of five evictions is accurate, but its conclusion — "no BuildTestWui PR can pass the merge queue until url://screenshottest/ is healthy again" — was not. The service was healthy the whole time. That claim traces to the test's own timeout text, which asserted "The screenshot service dependency is likely wedged" whenever a timer expired, whatever the reason. Verified live on 198.199.106.165 while the queue was still failing:

• The service rendered a full 8/8 MATCH that same morning (run ee431c6a, 217/0). A service that renders is not wedged.
• It is an on-demand ContainerNursery route with keepWarmSeconds: 300 — absent from ps between runs by design, and its shutdowns are clean idle timeouts, not crashes.
• /root/screenshottest-data: 20 KB, zero leftover sessions. No leak, no backlog.
• Deployed jar matched ScreenshotTestServerService HEAD; Chromium healthy and matching the committed RENDERER_VERSION; the renderer handshake passed before the upload phase was even reached.

Actual root cause: goldenScreenshots uploaded the entire test JVM classpath to the service on every run so the render worker could launch the fixture server — ~130 MiB over ~237 sequential 1 MiB RPC round-trips, each marshalled client-side by the SJVM bytecode interpreter. That needed ~0.43 MiB/s sustained to fit the upload phase's 300s budget, against a measured 0.4–0.85 MiB/s on the shared host. A coin flip; it passed 1 of 7 attempts on 2026-07-26.

About 90% of those bytes were dependencies only the test process needs in order to reach url://screenshottest/ — the URL resolver and protocol, jvm-libp2p, Netty, Bouncy Castle, the SJVM interpreter, Coursier, and the kompile build tooling. The largest single entry was kotlin-compiler-embeddable at 53 MiB: the Kotlin compiler, uploaded to a worker that only runs a Jetty server. ScreenshotFixtureServerKt.main builds an in-memory fixture API and calls createServer; it never touches BuildTestClient, so nothing in the url:// stack is reachable from it.

Fix: BuildTestWui#148, merged as 3758f29. Uploads only the fixture server's runtime closure — ~15 MiB over ~70 round-trips — and replaces the misleading timeout text with the entries and KiB actually transferred. goldenScreenshots now runs in ~100–125s instead of timing out at 300s. Verified four times locally against the live service before the PR, then in CI: uploaded 56 classpath entries (14746 KiB); skipped 91 harness-only entries (118686 KiB) / All 8 screenshot keys MATCH.

Note for anyone re-deriving the numbers: the service's request log counts uploadFileChunk calls, and a file under 1 MiB is one call — so ~237 chunks was ~130 MiB, not ~237 MiB.

Consequences for this handoff

• The "Do not bypass the queue / wait for the service to recover" advice above is obsolete. The queue is passable now; nothing needs waiting on. build-watchman is not needed for #145.
• The claim that #141 and main are red for a service outage was also this same test-side defect. Another session's pie-minimum-slice-share branch went green (207/0) right after #148 landed.
• Release/deployment state is unchanged and still applies: #145 bumped buildtest.wui:buildtest-wui to 0.0.86 and is NOT deployed — deploying remains a separate, explicit act. The sandboxjvm side is still unreleased at 0.0.49 (#97 merged but no version bump), so the pin-bump landmine documented above (three UrlResolver tests with exact ... cause: null assertions) is entirely unchanged and still live.

Still open in this handoff's scope — unchanged

Acceptance criterion (b) — the distinctive class+message appearing as the client exception's cause — is still not met, and is still resolver-lane work (SandboxException thrown with a string and no host cause, TypeMarshaling.kt:568). Criteria (a) and (c) are met and now both merged. This handoff stays open for (b).

New tracking issue for the screenshot-upload work that was deliberately left undone: https://github.com/CodexCoder21Organization/BuildTestWui/issues/147 — SHA-256 dedup on upload, and the upstream byte-payload throughput work in UrlResolver PRs 813/816/836/845. BuildTestWui no longer depends on any of it.