HHandoff
← Priority list

Handoff: UrlResolver #653 (marshaling fail-loud salvage) + non-DO flake findings

Original request: *"Now that #650 merged, is there anything about [UrlResolver #647] worth salvaging (tests, etc.)?"* → salvage it into a PR → *"ensure #653 gets merged"* → then investigate the non-DO CI flakes. **Merging #653 is user-approved** (directive: "ensure 653 gets merged"). 0 reviews required.

hf-2026-07-05-urlresolver-653-marshaling-salvage-and-flake-findings

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver #653 (marshaling fail-loud salvage) + non-DO flake findings

Thread began 2026-07-05; last updated 2026-07-12 ~16:50 UTC. The state snapshots below were verified at update time — re-verify everything before acting. gh pr view 653 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative, not this doc (runs finish, main moves, queues shuffle, flakes get fixed by other sessions).

Mission summary

Original request: "Now that #650 merged, is there anything about [UrlResolver #647] worth salvaging (tests, etc.)?" → salvage it into a PR → "ensure #653 gets merged" → then investigate the non-DO CI flakes. Merging #653 is user-approved (directive: "ensure 653 gets merged"). 0 reviews required.

The salvage is done, correct, rebased, locally verified RED→GREEN, and both required checks are GREEN. The only thing between here and DONE is landing it through the squash merge queue, where it keeps getting evicted for reasons unrelated to its own code (see THE BLOCKER).

Current state — PR #653 (re-verify)

  • State: OPEN, mergeStateStatus=CLEAN, not merged.
  • Branch / head: fix/sandbox-nested-marshal-fail-loud-not-stringify @ b1d63817 (local clone at /code/workspace/UrlResolver-653 is clean, HEAD == remote, nothing unpushed).
  • Head checks: BOTH GREEN. bld-build (GitHub Actions) passed; kotlin.build (remote) passed on buildtest run 069c59c7 in 1h26m (1085/1085 on the second attempt).
  • Rebase onto main: DONE. Rebased off the stale afefcb13-era base onto current main. build.kts resolver coordinate bumped to 0.0.632 (main's build.kts was at 0.0.625 as of 2026-07-12 16:44 UTC, so no collision — RE-CHECK, other version-bumping PRs land constantly). Test @WithArtifact pins aligned to main's canonical post-#734 set (protocol 0.0.337, jvm-libp2p snapshot-10, sjvm 0.0.40, observable-core 0.3.13, etc.).
  • RED→GREEN re-verified locally against current main: with TypeMarshaling.kt reverted to main's version (test kept), testNestedUnmarshalableValueFailsLoudNotStringified FAILS (echoMap() returns normally — the silent stringify); with the fix in place it PASSES. So the residual defect still exists on today's main and the test genuinely reproduces it.

What #653 changes (4 files)

Bug: after #650, TypeMarshaling.hostValueToJvmValue's nested else branch — reached by a value inside a Map/List argument that has neither a registered typed-proxy projection nor a toJson() convention — still did sjvm.mkString(value.toString()). That silent stringify hands the sandboxed callee a String where it expects the typed value → downstream ClassCastException / NoSuchMethodError (the netlab Map<String,SwitchConfig>JString(SwitchConfig(...)) failure class). The top-level toJvmValue already throws here; #653 brings the nested path in line.

  • src/foundation/url/resolver/sandbox/TypeMarshaling.kt — nested else throws a clear SandboxException naming the class instead of mkString(toString).
  • src/foundation/url/resolver/sandbox/testing/TestInterfaces.kt — adds class UnmarshalableValue(val payload: String) (test support).
  • tests/testNestedUnmarshalableValueFailsLoudNotStringified.kts — drives a Map<String,Any> carrying an UnmarshalableValue through MapEchoService.echoMap. RED before fix, GREEN after.
  • build.kts — resolver coordinate bump + comment.

Salvage decision from #647 (why only half): kept the fail-loud half; dropped #647's toJson round-trip helper because it reintroduced Method.invoke reflection (against the global no-reflection rule).

THE BLOCKER: merge-queue evictions (not the PR's code)

The repo uses a SQUASH merge queue. #653 has been enqueued three times this session and evicted each time for reasons unrelated to its own correctness:

  1. Shared build.kts version line. Every version-bumping PR edits the same coordinates = "foundation.url:resolver:0.0.X" line. When another such PR sits ahead of #653 in the queue, the queue's speculative merge of #653 on top of it conflicts → #653's entry shows UNMERGEABLE and is passed over / evicted. Seen repeatedly (behind #749, #671, #632). This is transient — it clears when the conflicting entry leaves the queue, and #653's mergeStateStatus stays CLEAN against real main throughout. At 16:44 UTC #653 was position 3, UNMERGEABLE, behind #749 and #632.
  2. DigitalOcean droplet-API rate-limit (pure infra). One queue verification run (79ce23b8) executed zero tests: the buildtest service restarted mid-run and all 10 shards failed createBuildDroplet with ratelimit-remaining=0 (DO account write-queue exhausted by a ~30-run burst; windowSpend=894/3600, provider reset stamped in the error). The automatic re-enqueue hit the same wall. Recovery was a timed back-off past the DO reset, then re-enqueue.

One CI test failure was seen and diagnosed as an unrelated circulating flake, NOT #653's code: the first full head-check run failed only testBytecodeFetchAggregatesRelayFailureDetails (1084/1085). That test is unstable on main since #733 (2026-07-11) and also failed on unrelated #748's run the same day. Its re-run passed clean. It deserves a proper root-cause fix as separate follow-up (codex gpt-5.6-sol is available for this).

The build-watchman --to-merged watcher (v0.0.12, launched via ~/bin/cs launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 653 --to-merged) was shepherding the queue entry (auto re-enqueues once on eviction, exits on terminal). It died during a ~12h session-teardown gap (04:42→16:44 UTC), so the entry is currently unwatched.

The six flake findings from the original mission — DISPOSITIONED

The original handoff listed six flaky tests to confirm resolved post-#690/snapshot-7. None of the six recur in the ~19 recent completed UrlResolver runs surveyed this session (stressTestJoinNetworkUnderIODispatcherSaturation, stressTestCloseReleasesPortWithInterruptedCallingThread, stressTestHighConcurrencyRegistrationStormDoesNotLeakConnections, testDiscoveredServicesSizeCap, testStaleServiceAnnouncementsNotForwardedToLateJoiners, testNetlabDynamicFailureRecovery). The flakes that DID surface on recent runs are a different, load-correlated set (stressTestConcurrentRegistrationsReliablyPropagateToRelay, stressTestRegistrationAfterWithdrawalRaceUnderContention, and the testBytecodeFetchAggregatesRelayFailureDetails flake above) — all one-per-run under executor load, none blocking on #653's actual diff.

NEXT STEPS (in order)

  1. Re-verify #653 is still CLEAN and both head checks still GREEN (gh pr view 653 ... --json state,mergeStateStatus,statusCheckRollup). If main moved such that 0.0.632 is now taken or the branch went DIRTY, do the one-line rebase: cd /code/workspace/UrlResolver-653 && git fetch origin && git rebase origin/main, resolve the build.kts coordinate to the next free version (inspect main's build.kts), git push --force-with-lease, and re-drive head checks.
  2. Pick a good enqueue window: the local box is fine, but the shared buildtest droplet is the constraint. Prefer a moment when (a) the merge queue has no other version-bumping PR ahead of #653 (check gh api graphql merge-queue entries; a non-version PR ahead — docs/test-only — is fine), and (b) the DO rate window is healthy (buildtest /api/runs responding, not a ~30-run burst in flight).
  3. Enqueue (gh pr merge --auto is disabled in this repo):
    PRID=$(gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){pullRequest(number:653){id}}}' --jq '.data.repository.pullRequest.id')
    gh api graphql -f query="mutation{enqueuePullRequest(input:{pullRequestId:\"$PRID\"}){mergeQueueEntry{position state}}}"
    
    …or just re-run build-watchman --to-merged (command above), which enqueues and shepherds to MERGED, auto re-enqueuing once on eviction. Run it as a tracked background job and re-arm it if it exits non-MERGED.
  4. If evicted again: diagnose before re-running (per the watch-build discipline). Version-line UNMERGEABLE → wait for the conflicting entry to leave, re-enqueue. Infra provisioning failure (droplet rate-limit / shard death) → back off past the reset, re-enqueue. A genuine test failure on #653's own queue run (not the bytecode-aggregate flake) → investigate the diff.
  5. Poll to MERGED, then close out this handoff (delete it via the create-handoff skill's completion path — pre-approved) and decide the fate of the diagnostic-only scratch branch fix/joinnetwork-io-dispatcher-starvation (do not merge — propose deletion if its target flake is confirmed resolved).

Reproducing local verification

cd /code/workspace/UrlResolver-653            # STANDALONE cd first — shared shell cwd is a footgun
git rev-parse --abbrev-ref HEAD               # confirm fix/sandbox-nested-marshal-fail-loud-not-stringify
rm -rf ~/.aibuildcaches/_code_workspace_UrlResolver-653 ~/.cache/coursier/v1/bldbinary/repository/foundation.url/resolver  # clear stale-jar traps
scripts/test.bash --test testNestedUnmarshalableValueFailsLoudNotStringified  # ONE test.bash at a time (-Xmx6g; 2 concurrent → OOM 137)

RED check: git checkout origin/main -- src/foundation/url/resolver/sandbox/TypeMarshaling.kt, purge caches, run the same test → must FAIL. Restore with git checkout HEAD -- <that file>.

Gotchas / procedures learned the hard way

  • Merge-queue mechanics: enqueue via GraphQL enqueuePullRequest; --auto is disabled. Required checks: bld-build (GitHub Actions) AND kotlin.build (remote) — two independent full-suite runs (~1.5–2h each on the shared droplet).
  • buildtest /api/test-results?id=<run> paginates at 100 and uses ?limit=500&page=N (totalPages in the envelope); the bare call returns only page 1 (100 rows) — don't conclude "no failures" from page 1 alone. Under load it flaps 503 / SSL-handshake — that's degraded observability, NOT a dead run; ground-truth on the host (ssh -p 23 root@198.199.106.165 'stat/tail /root/buildtest-data/runs/<id>/{test-events.jsonl,build.log}').
  • ~/bin/cs is the working coursier (/usr/local/bin/coursier is wrong-arch). export GH_TOKEN=$(gh auth token) before launching build-watchman.
  • Do not gamble on CI: version-line UNMERGEABLE and droplet-rate-limit evictions are infra/serialization, not flakes to retry blindly — wait out the specific cause, then re-enqueue.
  • 10 GB / one-JVM local cap: each scripts/test.bash is -Xmx6g; two concurrent → OOM SIGKILL 137. Never run two local builds at once; never broad-pkill java.

🤖 Generated with Claude Code