Handoff: UrlResolver netlab infrastructure flakes blocked at shared-Kotlin-stdlib boundary

Written 2026-07-13 02:18 UTC. RE-VERIFY before acting: this is a point-in-time snapshot. Re-query BuildTest through BuildTestCli, inspect url:netlab-hosted: with container-nursery-cli containers --json, and re-check every linked PR with gh pr view <url> --json state,mergeStateStatus,statusCheckRollup,mergedAt. Fresh service and GitHub state wins over this document.

Mission summary

Investigate and fix the dominant infrastructure root cause behind the testNetlab* flaky-test family on CodexCoder21Organization/UrlResolver CI. The user explicitly prohibited retries, sleeps, and timeout increases in tests or UrlResolver; required roughly ten recent failures classified as acquisition, upload/dial wedge, or post-setup assertion; required a fail-first fix at the owning infrastructure layer; and preferred STUCK + findings if a correct fix crossed a single-repository boundary. Code PRs must not be merged.

What was found and done (the chain)

1. Fresh repositories and ownership were audited. Fresh checkouts were made under /code/netlab-infra/workspace/ for UrlResolver, NetLabManagerServer, NetLabWorkerServer, NetLabEmbedded, UrlProtocol, BuildTestCli, and PlanRepository. No active manager/worker PR owns the artifact mechanism below. Historical upload OOM/session-leak fixes are already merged; the still-open manager issue is NetLabManagerServer issue 40.

2. The requested last-25-main failure names were reconstructed. GitHub check output preserves five isolated netlab failures across three runs: testNetlabResolutionUnderLatencyAndPacketLoss three times, plus testNetlabDynamicFailureRecovery and testNetlabPartitionAndHealAcrossNats once each. Whole-run provisioning/build-lease aborts were excluded. The exact run bodies (86dd9dba, eef37176, 8c59a6d5) have been pruned from the BuildTest service, so their stacks are no longer available.

3. The HTTP evidence path failed, but the authoritative P2P path worked. https://buildtest.kotlin.build/api/test-results became unresponsive and was recorded in this challenge. A fresh BuildTestCli build successfully queried url://buildtest/, returning the recent retained run catalog and full results.

4. Newest ten retained, actually-executed netlab failures:

   | Run | Test | Class | Evidence | |---|---|---|---| | bbd23968 | testNetlabDynamicFailureRecovery | b | Manager-client bytecode frame rejected before upload. | | 13547282 | testNetlabSevereImpairmentScenarios | b | READY, then the first 1 MiB upload timed out on every attempt. | | 13547282 | testNetlabDynamicFailureRecovery | a | Stuck INSTALLING_DEPENDENCIES, then manager health/get timed out. | | 13547282 | testNetlabPartitionAndHealAcrossNats | a | No READY environment. | | 13547282 | testNetlabDiscoveryScenarios | a | Every manager health call timed out. | | 13547282 | testNetlabResolutionUnderLatencyAndPacketLoss | a | Manager health timed out before create. | | 9f9ac054 | testNetlabSevereImpairmentScenarios | b | READY, then classpath upload timed out. | | 9f9ac054 | testNetlabDynamicFailureRecovery | b | Direct and relay manager-bytecode fetch timed out. | | 9f9ac054 | testNetlabPartitionAndHealAcrossNats | a | No READY environment. | | 9f9ac054 | testNetlabDiscoveryScenarios | a | Manager health timed out during every candidate attempt. |

   Totals: 6 acquisition, 4 upload/dial wedge, 0 post-setup assertion. Acquisition is dominant. Runs 13547282 and 9f9ac054 resumed together after a BuildTest restart at 15:02 UTC and started provisioning at 15:04–15:05. Their netlab and unrelated url://performancetest/ requests timed out together, showing shared URL-provider saturation rather than test assertions.

5. The deployed cap is not the old stopgap. The live manager route has 384 MiB memory, MAX_LIVE_ENVIRONMENTS=60, and ENV_IDLE_TTL_MINUTES=30. Recent manager reconciliation retained 26 environments while the DigitalOcean account had 96 live droplets. At write time, container-nursery-cli containers --json showed url:netlab-hosted: RUNNING on host port 40569. The old default cap of 6 is not the current deployed bottleneck.

6. A deterministic artifact reproducer matched production almost exactly. A temporary manager test built buildClientFatJar(), wrapped its bytes in the real legacy bytecode response, and applied UrlProtocol's minimum 8 MiB parse budget with 7x org.json amplification. Unfixed measurements were:

   • client JAR: 2,030,116 bytes
   • framed response: 2,706,923 bytes
   • worst-case parse estimate: 18,948,461 bytes
   • allowed minimum budget: 8,388,608 bytes

   Fresh production run bbd23968 rejected frameBytes=2,706,967, only 44 bytes different. This is a deterministic, 100%-failure reproducer for the fresh bytecode-fetch mechanism. The manager JAR embeds kotlin-stdlib in every sandbox client response.

7. The tempting manager-only fix was disproved and removed. Excluding kotlin-stdlib-* from the fat client made the budget test green, but the existing real manager→worker UrlProtocol/SJVM upload test then failed repeatedly with NoClassDefFoundError: kotlin/collections/MapsKt during healthCheck. AvianStdLibHelper's embedded stdlib.jar is 668,943 bytes and contains Avian's Java runtime, not Kotlin collections. UrlProtocol deliberately drops provider-supplied stdlib bytes as a trust boundary. The manager checkout was restored to a clean state; no invalid source change or code PR remains.

8. Why this is blocked rather than patched: a correct stdlib-sharing fix needs SJVM/UrlResolver to make trusted Kotlin stdlib available once per sandbox runtime, followed by a manager release that stops bundling it and a downstream round-trip verification. That is necessarily multi-repository. The single-repo alternative is a broad rewrite of the manager's 556-line Kotlin TopologyService sandbox client into Java/non-sandbox dispatch, with every API method revalidated. Both exceed the requested narrow single-repo lane.

9. Adjacent active work is not ownership of this mechanism. UrlProtocol PR 360 was OPEN, local CI green, remote BuildTest running at write time. It fixes a RelayPing exception escaping during concurrent close. No observed netlab row contained that exception, and it does not change the 2.7 MB manager artifact or Kotlin stdlib sharing.

Relevant PRs / refs

| Item | Write-time state | Notes | |---|---|---| | UrlResolver issue 689 | Open issue | Family tracking and prior load-sensitive history. | | UrlResolver issue 682 | Open issue | Includes prior ACKed-but-corrupt upload evidence. | | NetLabManagerServer issue 40 | Open, no active fix | Historical upload failures; earlier mechanisms already merged. | | UrlProtocol PR 355 | Merged | Introduced the parse-budget admission that exposes the oversized artifact deterministically. | | UrlProtocol PR 360 | Open; local CI green; remote check running | Adjacent relay-process hardening, not artifact ownership. Head at write time: aba3fe1fc78a549183389352a85d15ebdc041e57. | | NetLabManagerServer branch fix/shared-stdlib-manager-client | Clean, unpushed, based on 8dfddb2 | Experimental changes removed after runtime falsification; nothing to preserve or PR. | | /code/netlab-infra/PROGRESS.md | Local investigation record | Full milestones and operational challenge links. |

Next steps

1. Re-verify current state first. Query BuildTest, live manager route/container/logs, open manager/SJVM/UrlResolver/UrlProtocol PRs, and UrlProtocol PR 360.
2. Decide with the user whether to authorize a multi-repository upstream fix. Do not silently expand scope.
3. If authorized, locate the sandboxjvm/SJVM source repository and first add an upstream failing test: a Kotlin implementation JAR without bundled kotlin-stdlib must execute mapOf/MapsKt through AvianStdLibHelper.addStdlib(builder, null). The baseline must fail deterministically with the observed NoClassDefFoundError.
4. Implement trusted, consumer-side Kotlin stdlib sharing in SJVM/UrlResolver without accepting provider-supplied stdlib bytes. Measure retained heap across many sandboxes; sharing must actually remove per-sandbox copies rather than move them.
5. Publish the upstream artifact, update NetLabManagerServer, restore the manager artifact-budget regression, and rerun testWorkerClientUploadFileReliability plus the complete manager suite. The lean client must both fit the 8 MiB/7x invariant and execute every public TopologyService path.
6. Only then rebase, create code PRs, and watch CI. Do not merge code PRs without explicit user approval.
7. After deployment is separately authorized, replay a controlled single UrlResolver full run, then a controlled concurrent CI-wave reproducer. Do not add retries/sleeps/timeouts to the consumer tests.
8. Delete this handoff once the upstream change, manager update, PR CI, and production validation are genuinely complete.

Reusable / operational knowledge

• BuildTest's direct service can bypass a hung WUI: build BuildTestCli, then run java -jar build-fats/buildtest-cli.jar --json list|results <runId>|status <runId>.
• BuildTest retention differs by layer: GitHub check summaries may retain failed names after the authoritative run body has been pruned.
• ContainerNursery's container --route-key 'url:netlab-hosted:' endpoint returned 404 for the URL-encoded colon-bearing key. Use containers --json and filter .domain == "url:netlab-hosted:" until that CLI/API bug is fixed.
• HardwareControlFabric mTLS credentials were absent in this environment, so read-only ContainerNursery CLI inspection was used; SSH was not used.
• The manager's current buildClientFatJar() dependency is not accidental dead weight: removing Kotlin stdlib without first fixing consumer-side sharing deterministically breaks sandbox execution.
• Full raw evidence and test logs from this session remain under /tmp only (/tmp/urlresolver-results, /tmp/netlab-manager-client-budget-red.xml, /tmp/netlab-manager-worker-upload-green.log) and are not durable. The decisive numeric/error evidence is copied above.