Handoff: UrlResolver deep-map rejection fix complete, blocked by unrelated remote-CI instability

Written 2026-07-14 UTC.

RE-VERIFY: Everything below is a write-time snapshot. Before acting, run gh pr view 774 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid and inspect the current kotlin.build (remote) details URL. A fresh PR/check query always wins over this document.

Mission summary

Fix flaky testDeeplyNestedMapReturnIsRejectedNotStackOverflow, where the required full depth-limit rejection lost a race against the sandbox's unchanged five-second SJVM invocation deadline on slow runners. The work had to reproduce the timeout under load, profile the owning layer, fix upstream if appropriate, preserve the exact assertion and all timeouts, verify under the reproducing profile, publish/bump any upstream fix, and take the UrlResolver PR through green CI and merge.

The implementation, upstream release, local verification, and PR are complete. The remaining blocker is the repository-wide required remote check: five consecutive full-suite attempts failed with mostly different unrelated networking flakes plus repeated zero-millisecond Coursier classpath-resolution failures. The target regression passed remotely in every attempt.

What was found and done

1. The unchanged test was reproduced 10/10 by pinning every BootstrapRunner thread to one CPU shared with 48 equal-priority spinners. Every failure was SJVM method invocation timed out after 5s: test/DeepMapServiceImpl.getDeepMap(I)Ljava/util/Map;; the proxy call took 5.009–5.323 seconds.
2. Instrumentation and interrupted stacks proved the five seconds were spent in interpreted Avian HashMap construction inside the sandbox. Host-side TypeMarshaling.fromJvmResult only begins after sjvm.invoke returns, so the depth walk did not begin in timeout runs. UrlResolver change Fail loud on nested unmarshalable values instead of silent mkString is on the opposite host-to-sandbox path and is not causal.
3. The upstream SJVM owner was fixed in sandboxjvm PR: loaded-class cache hits bypass the suspending loader mutex, and Avian HashMap(int) construction plus put use exact native implementations with public compatibility/stress coverage. Its full JVM suite passed, the PR merged, and libSJVM-jvm, avianStdlibHelper-jvm, and stdlibHelperCommon-jvm 0.0.47 were published and HTTP-verified on kotlin.directory.
4. Release 0.0.47 materially improved the narrow upstream workload but still did not give the exact downstream test enough margin under the valid 48-spinner profile. An unpublished follow-up cache candidate also timed out and was not released.
5. The consumer fixture was then corrected at the source of the race: it constructs the minimum 1,001-level map in ten bounded addDeepMapLevels(100) setup calls and makes the asserted getDeepMap(0) call return the prepared value. This isolates sandbox-to-host marshaling from unrelated interpreted fixture construction. The five-second deadline and exact full depth-rejection message are unchanged.
6. All 1,063 UrlResolver test-script SJVM annotations were aligned to 0.0.47. A mixed 0.0.40/0.0.47 graph emitted 3,186 artifact-version mismatch warnings, so pinning only the target was not viable. build.kts uses resolver coordinate 0.0.634 and the README names SJVM 0.0.47.
7. The corrected target passed 10/10 under the exact reproducing profile. The asserted-call wall times were 4.801–11.205 seconds; values above five seconds reflect starvation of the timeout scheduler itself, while the required host rejection still won. It passed 3/3 unconstrained after the final rebase. The nested-unmarshalable, deep-list, exact-depth-list, deep-data-class, and cyclic-list siblings all passed.
8. A local isolated-home full run passed 1,089 tests. Twelve tests were not executed because offline Kompile temporary caches lacked unrelated script-only artifacts; online resolution reproduced the Coursier HTTPS-connect stall. The one executed failure was a stale test expecting dedicated SJVM context threads. SJVM 0.0.47 intentionally moved VMs to serialized views of a shared elastic pool, so that test was replaced by a passing end-to-end test which creates, exercises, reuses, and closes 24 real sandboxed connections while asserting no legacy per-VM threads are pinned.
9. The UrlResolver branch is clean and fully pushed. At write time the open PR head is 230ec17d5c2e2bf3801895ad936cbacef4a572fa; bld-build is green.
10. The required remote suite failed five times while the deep-map target stayed green:
    • 993c8d94: 1,090 passed, 12 failed; nine failures were 0ms Coursier non-executions, three were unrelated networking flakes.
    • 23f6d1ca: 1,097 passed, five failed; two were 0ms Coursier non-executions and three were unrelated discovery/timing flakes.
    • 43426389: 1,096 passed, six failed; two were 0ms dependency/build failures and four were different networking/timing flakes.
    • 9603a2b0: 1,097 passed, five failed; two were 0ms build/dependency failures and three were different networking/timing flakes.
    • c76e3ea2: 1,097 passed, five failed; three were 0ms Coursier non-executions and two were different networking flakes. Its test-event counter froze at 0/100 for 20 minutes even though the final result contained 1,102 outcomes.
11. The changing failures include exact-threshold misses such as a 507ms mean against 500ms, a 2.082s sample against 2.000s, and a 422ms response against 100ms, plus peer/service discovery races. No failure path overlaps the deep-map fixture or SJVM marshaling change. The instability is recorded at PlanRepository challenge.
12. Build-watchman required host-level confirmation of both test-events.jsonl and build.log freshness before deleting the fifth stalled run. HardwareControlFabric could not be used because the expected mTLS certificate files were absent from this environment, so the run was correctly left for the buildtest service to terminate itself.

Relevant PRs and references

| Item | Write-time state | Reference | |---|---|---| | UrlResolver fix | OPEN, merge blocked only by failed kotlin.build (remote); head 230ec17d5c2e2bf3801895ad936cbacef4a572fa; branch fix/deep-map-rejection-vs-sjvm-timeout-race | https://github.com/CodexCoder21Organization/UrlResolver/pull/774 | | Upstream sandboxjvm fix | MERGED; 0.0.47 published | https://github.com/CodexCoder21Organization/sandboxjvm/pull/93 | | CI-instability challenge | Enqueued by the challenge workflow at write time | https://github.com/CodexCoder21Organization/PlanRepository/pull/1392 | | Original remote failures | Depth rejection lost to the five-second SJVM invocation deadline | https://buildtest.kotlin.build/run?id=ae34b898, https://buildtest.kotlin.build/run?id=b5568759, https://buildtest.kotlin.build/run?id=83245032 | | Same-code original pass | Demonstrates the old timing race | https://buildtest.kotlin.build/run?id=bc79d16e |

There is no unsaved UrlResolver work: the local checkout, branch, and remote branch all point to the same head, and git status --short is empty. Logs, caches, and stress harnesses live outside the repository under /code/fix-deepmap/ and were not committed.

Next steps

1. Re-verify the UrlResolver PR and current required checks using the command in the banner.
2. Do not bypass CI or branch protection. Do not weaken the full rejection assertion and do not change any timeout.
3. Before another full rerun, verify that the Coursier/buildtest incident has recovered. The durable owning-layer fix is to prewarm/share shard dependency caches or let the remote-build system rerun only failed tests; do not add downstream timeout/retry mitigations to UrlResolver.
4. Once infrastructure is healthy, re-request only the failed kotlin.build (remote) check suite and run exactly one build-watchman through merge: coursier launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 774 --to-merged
5. If a future run fails an actually executed deep-map/SJVM test, investigate that failure with the full bug-fixing workflow. If only unrelated network flakes recur, fix each at its owning layer with a reliable reproducer rather than changing this PR.
6. After the UrlResolver PR is genuinely MERGED, delete this handoff through the create-handoff completion workflow and merge that handoff-deletion PR.

Reusable and operational knowledge

• A valid CPU-pressure harness must pin every BootstrapRunner worker thread, not only the Java main thread. The successful profile used taskset -apc on one CPU plus 48 equal-priority spinners.
• The test fixture's leaf map counts as map depth 1; ten batches of 100 wrappers produce the exact observed depth 1,001.
• Kompile requires every test script participating in the same workspace graph to agree on the three SJVM artifact versions. A single-test pin bump creates a mixed graph and thousands of mismatch warnings.
• Buildtest may show 0/100 while shards are still resolving/compiling. Do not call that a wedge while build.log advances. Build-watchman's rule requires both test-events.jsonl mtime and build.log freshness to be frozen on the host before deletion or re-request.
• All local builds/tests for this effort used HOME=/code/fix-deepmap/home and JAVA_OPTS=-Duser.home=/code/fix-deepmap/home. The warmed Kompile cache is /code/fix-deepmap/home/.aibuildcaches/deep-map-attempt-2-v634.
• Detailed chronological evidence is in /code/fix-deepmap/PROGRESS.md; local full-suite output is /code/fix-deepmap/full-suite-final.log, and watcher output is /code/fix-deepmap/urlresolver-pr-774-watchman.log.