Original user request (verbatim intent): *"Determine why https://buildtest.kotlin.build/run?id=392091d0 claims to have taken 29 minutes of wall clock time and 1 minute of cpu time … Perform a comprehensive forensic analysis."*
Moves it out of every priority list and into ArchiveArea.
Written: 2026-07-15 ~22:45Z
⚠️ RE-VERIFY BANNER — everything below is a write-time snapshot. The incident, PR states, merge-queue positions, host load, and the 401 rate were all true only when written. Re-verify before acting. Authoritative re-checks:
- CN box:
ssh -p 23 root@198.199.106.165thenuptime+ps -eo pid,pcpu,args --sort=-pcpu | grep container-nursery.jar(SSH key~/.ssh/id_ed25519).- Live 401 storm: on the box,
timeout 5 tcpdump -i lo -n -A -s 0 'tcp' | grep -aoE 'HTTP/1\.1 (401|200)' | sort | uniq -c.- PRs:
gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,mergedAt.
Original user request (verbatim intent): "Determine why https://buildtest.kotlin.build/run?id=392091d0 claims to have taken 29 minutes of wall clock time and 1 minute of cpu time … Perform a comprehensive forensic analysis."
It then expanded, in order:
Two decisions already made with the user:
directory.kotlin.www.server) is OUT of scope for the deploy — it has no foundation.url:resolver dependency; user chose "Just CN; maven server needs no change."Build Rules phase = 29m 25s wall / 1m 22s CPU (~4.7% util) = ~95% idle, threads blocked on network I/O resolving deps from kotlin.directory (Connection refused + Unexpected end of file + coursier ResolutionError). The "29 min / 1 min" the user saw is this one phase. Total run 44m. The failing rule buildMaven() only ran 82.8s — so ~28 min was unexplained by the failure itself → pointed at the resolve path, not the rule.
I first blamed (a) yamux ByteBufQueue O(n²), (b) PeerRegistry O(N²) (#776), (c) relay JAR-bootstrap volume. Live profiling disproved all three as the CPU driver. Recorded here so the next person doesn't re-walk them.
container-nursery.jar (PID was 1907532, -Xmx384m) at 498–511% CPU; co-resident maven-server.jar (kotlin.directory backend) at ~86–90%.top -H + /proc/<pid>/task/<tid>/comm) = ForkJoinPool-1 virtual-thread carriers carrying HttpsFacade-handler-* vthreads; ~13.3M vthreads created in 98 min ≈ 2,300/sec.jcmd <pid> JFR.start settings=profile duration=40s filename=/tmp/cn.jfr; then jfr print --events jdk.ExecutionSample). On-CPU profile: 3,884 of 4,224 samples (~92%) in org.example.facade.HttpsFacadeProvider.handleMixedModeTls, and the deep frames are TLS 1.3 handshake crypto — SecureRandom/HashDrbg → XECOperations.generatePrivate (X25519 ephemeral keygen) → ServerHello production. GC was secondary (~7%, G1 running ~3.6 concurrent cycles/sec on the tiny 384M heap).Capturing the plaintext, post-TLS side on the box (tcpdump -i lo -n -A -s 0 'tcp') — CN terminates TLS on :443 and proxies plaintext to backends on loopback:
nstat TcpPassiveOpens), ~59k TIME-WAIT sockets — massive connect→handshake→close churn.Host: kotlin.directory (~2,800 req/s), from 53+ distinct DigitalOcean droplet IPs (X-Forwarded-For = the buildtest/netlab fleet). (An earlier SNI guess of buildtest.kotlin.build was noise from an unreliable cross-packet grep — corrected. Build droplets talking to kotlin.directory for deps is architecturally correct; the 401s are the problem.)HTTP 401 Unauthorized (WWW-Authenticate: Basic realm="Authentication Required"), e.g. 7,403× 401 vs 1,324× 200 in 5s at write time. Requests carry NO Authorization header.okio-jvm, slf4j-parent, ow2 asm, commons-parent, kotlin-stdlib…), and the same ones repeat — not diverse resolution.Mechanism (verified): fleet droplets cold-resolve their full classpath from kotlin.directory (no pre-warmed cache) → every fetch returns 401 → the resolve never completes → it is re-driven in a loop → each attempt opens a fresh TLS connection (no keep-alive) → ~5,000 TLS handshakes/sec → CN burns 5–6 cores on X25519 ephemeral-key crypto → starves the co-resident maven server and every legitimate HTTPS resolve on the box → the original 29-min idle build. Self-amplifying: a 401 returns instantly (no network wait, no backoff), so it hammers far harder than a timeout would.
Checked coursier ArtifactError.scala: coursier already maps 401 → a dedicated terminal ArtifactError.Unauthorized (with realm), distinct from the Recoverable set (Locked, ConcurrentDownload) and from InternalServerError/RetryableHttpError. FileCache.withRetry(n) only retries Recoverable errors. So coursier does not retry a 401 — no fork or error-mapping hook is needed. This corrects an earlier wrong claim ("coursier retries the 401"). The re-drive loop is ABOVE coursier — in kompile's unifiedResolve and/or the buildtest droplet build lifecycle re-invoking resolution on a terminal failure. (HYPOTHESIS — the exact re-drive site is not yet located; see Next steps #3.)
The kotlin.directory backend maven-server.jar --maven-repository /root/maven (PID was 1907696) runs with no auth flags on its cmdline, yet returns 401. grep of /root/ContainerNursery/config.json hits auth/credential strings. So the Basic-auth requirement is configured at the CN route/config level (or a maven-server config file), not the maven-server CLI. Whether requiring auth on public artifact GETs is intended (a regression?) is UNVERIFIED and is the crux of the acute fix. There is also a second maven-server instance (was PID 2074990) with --certificate/--key for maven.directory — a different domain; don't confuse the two.
The CPU is ~92% TLS-handshake crypto driven by the 401 storm — unrelated to PeerRegistry/yamux N². #776 is safe to land but will not fix this incident. The yamux O(n²) fix (jvm-libp2p #23 → snapshot-12) is already in CN main via ContainerNursery #522 and pinned directly at build.kts:279 (jvm-libp2p:1.3.0-codexcoder21-snapshot-12).
| Ref | State @ write | Notes |
|---|---|---|
| UrlResolver #776 | OPEN / CLEAN, in merge queue ~48min, queue:bld-build pending, no buildtest run URL yet | PeerRegistry O(N²) fix; bumps foundation.url:protocol → 0.0.360. Merge may itself be blocked by this incident (buildtest can't run while kotlin.directory 401s). Head branch fix/peerregistry-serviceindex-pin-0.0.360, head SHA 2818468cbc612975b96502e9a0e27670ab99a72d. Watched by build-watchman (see below). |
| jvm-libp2p #23 | MERGED 2026-07-12 | yamux ByteBufQueue O(n²) → ArrayDeque. Released as jvm-libp2p:1.3.0-codexcoder21-snapshot-12. |
| ContainerNursery #522 | MERGED 2026-07-12 | Bumped CN to jvm-libp2p:snapshot-12. Already in CN main. |
| ContainerNursery #524 / #494 / #504 | MERGED (recent) | HTTPS-facade fixes (HEAD parks, handler thread exhaustion, selective hang) — HttpsFacadeProvider is bug-prone + under active work. Relevant to the facade/keep-alive fix. |
| latest published foundation.url:resolver | 0.0.637 (compiled vs protocol:0.0.337) | The N² fix is NOT in any published resolver yet; publishing post-#776-merge main HEAD would be 0.0.638 (first with protocol:0.0.360). |
No code changes were made in this session. ~/workspace/ContainerNursery is a clean clone (no edits). Deploy is fully prepped but not started.
Re-verify the incident is still live (banner commands). If load is normal and 401 rate is low, the storm may have subsided (fleet drained / someone fixed auth) — re-scope before acting.
ACUTE — stop the 401 storm (this is what's melting the box; prioritize over the #776 deploy). Find where kotlin.directory's Basic-auth requirement is configured:
/root/ContainerNursery/config.json for the kotlin.directory route's auth/credential block, and/or any maven-server auth config file. Decide intended vs regression — these are public artifacts every build must fetch, so requiring auth on their GETs is almost certainly wrong.Authorization header on the wire). Fix in droplet image / build config.Find + fix the re-drive loop (owns the "don't retry a terminal 401" behavior). Trace kompile-core's unifiedResolve and the buildtest droplet build lifecycle (BuildTestEmbedded / community.kotlin.kompile.testrunner.bootstrap) to find what re-invokes resolution after coursier returns terminal ArtifactError.Unauthorized. Make it fail fast on Unauthorized instead of re-resolving in a loop. (The re-drive site is a hypothesis — locate it before coding. Reproduce against a 401'ing repo per the bug-fixing workflow.) Coursier itself needs NO change (401 already terminal).
CHRONIC — kill the cold-resolve volume (user's insight). Give droplets a pre-warmed / shared dependency cache (bake the common classpath into the droplet image, or a shared read cache) so a fresh droplet doesn't cold-resolve ~195 public artifacts from kotlin.directory every run. This makes the system resilient to any future kotlin.directory blip.
HARDENING. CN facade should support HTTP keep-alive (so N fetches ≠ N TLS handshakes — see HttpsFacadeProvider.kt, note the Connection: close in rejectAndCloseHttpRequest). Consider a per-client rate-limit / circuit-breaker on :443 as a storm backstop (user suggested ~1 req/10s/droplet — good as a breaker, NOT a cure).
Resume the (separate, pre-approved) deploy once #776 is mergeable. build-watchman is tracking #776 (see below). Then: publish foundation.url:resolver 0.0.638 from post-merge main HEAD (publish-maven-artifact skill), then bump ContainerNursery (~/workspace/ContainerNursery, or re-clone):
build.kts: foundation.url:protocol 0.0.336 → 0.0.360 (line ~167), foundation.url:resolver 0.0.600 → 0.0.638 (line ~244).tests/testFoundationUrlResolverProtocolBinaryCompatibility.kts: update its 4 pinned version strings (the two @file:WithArtifact pins + expectedResolverVersion/expectedProtocolVersion) — its own comment says pins MUST track build.kts.stressTestServerSurvivesMidNegotiationChurnDrop…, …SustainedSubstreamChurn…) pin resolver:0.0.595 / protocol:0.0.336 — bump so the shipping pair is stress-validated.ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_ed25519 -p 23 root@198.199.106.165 (fabric daemon has been down before; SSH is the reliable fallback). JDK 21 (jcmd/jstack/jfr at /usr/bin). ~/workspace is writable, not /code/workspace.top -H -b -n1 -p <pid> → map TID→name via cat /proc/<pid>/task/<tid>/comm (GC/carrier threads DON'T show in jstack; comm never lies).jcmd <pid> JFR.start name=cn settings=profile duration=40s filename=/tmp/cn.jfr; then jfr summary /tmp/cn.jfr and jfr print --events jdk.ExecutionSample --stack-depth 60 /tmp/cn.jfr | grep 'org.example\|foundation' | ... | sort | uniq -c.tcpdump -i lo -n -A -s 0 'tcp' on the box shows CN→backend HTTP after TLS termination — Host:, X-Forwarded-For: (true client), request paths, and response codes (HTTP/1.1 401). This is how the 401 storm was proven.nstat -z; sleep 4; nstat | grep TcpPassiveOpens. Churn: ss -tan '( sport = :443 )' | awk 'NR>1{print $1}' | sort | uniq -c (huge TIME-WAIT = connect/close churn).bin/container-nursery.jar; kotlin.directory backend = maven-server.jar --maven-repository /root/maven (no cert); maven.directory backend = second maven-server.jar with --certificate/--key. CN log: /root/ContainerNursery/container-nursery.log; container backends' stdout under /root/ContainerNursery/container-logs/.ArtifactError.Unauthorized (401), Forbidden (403), NotFound (404) are terminal; .withRetry(n) only retries Recoverable. So a 401 storm is never coursier's own retry — look upstream (kompile/build orchestration).--to-merged, launched via coursier launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 776 --to-merged). It EXITS at merge or failure and does not re-arm itself — relaunch it in the next session (it is not durable across sessions).cn-tls-slowness-relay-cpu-starvation-2026-07-12, buildtest-slow-concurrency-coursier-resolve-on-cn-relay-starvation-2026-07-12, maven-central-429-mirror-seeding-remedy. Those blamed relay CPU / coursier resolve starvation; this handoff supersedes the CPU-attribution: the dominant driver is the 401 TLS-handshake storm, proven by JFR + tcpdump.