HHandoff
← Priority list

Handoff: land kompile-core #213 (entropy CI fix) and UrlResolver #680 (observables resilience tail) — both branch-green, blocked only by shared-CI infra

User directive (this phase, verbatim): **"Merge PR 213, 680 seems red so we need to resolve that and get it green."**

hf-2026-07-11-kompile-core-213-entropy-and-urlresolver-680-observables-tail-merges

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: land kompile-core #213 (entropy CI fix) and UrlResolver #680 (observables resilience tail) — both branch-green, blocked only by shared-CI infra

Written: 2026-07-12 ~17:00 UTC. Workstream began 2026-07-11 (entropy-flake fix surfaced by ContainerNursery #477).

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time. Merge queues shuffle, CI re-runs, branches get force-pushed by parallel sessions (this branch has been force-pushed/rebased by a sibling session more than once), the buildtest droplet fleet and kotlin.directory oscillate between healthy and overloaded, and version coordinates get claimed by other publishes minute-to-minute. Re-verify before you act. Authoritative checks:

  • PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • Merge queue: gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"<repo>"){mergeQueue{entries(first:8){nodes{position pullRequest{number} state}}}}}'
  • Free resolver version: curl -s -o /dev/null -w "%{http_code}" https://kotlin.directory/foundation/url/resolver/0.0.<v>/resolver-0.0.<v>.pom (200 = taken, 401 = free)
  • Host health (198.199.106.165, key ~/.ssh/id_ed25519_new, port 23): ssh ... "cut -d' ' -f1-3 /proc/loadavg"

A fresh query always wins over this document.

1. Mission summary

User directive (this phase, verbatim): "Merge PR 213, 680 seems red so we need to resolve that and get it green."

Two PRs, both with all branch checks green and content fully validated, need to reach MERGED:

  1. kompile-core #213 — the org-wide /dev/random entropy-starvation CI fix. User authorized its merge.
  2. UrlResolver #680 — the final tail of the 2026-07-09 observables close-out (its sibling UrlResolver #707 merged 2026-07-11 19:20Z).

Neither PR's own content is in doubt. The entire remaining problem is passage through contended shared CI infrastructure (kotlin.directory maven server + the DigitalOcean droplet-create window), plus a wedged GHA run and a recurring version-pin collision on #680. This handoff exists because the prior session's automation watchers were torn down (session ended) mid-drive.

2. What was found and done (the chain — evidence, not speculation)

2a. kompile-core #213 — entropy fix, CONTENT DONE, evicted from merge queue 5× on infra

  • The fix (verified, reviewed): kompile-core's executeTestInSeparateJvm (src/Workspace.kt, jvmArgs assembly ~line 1910) now launches every spawned child test JVM with -Djava.security.egd=file:/dev/urandom via a new internal fun entropyConfigurationJvmArgs() in src/EntropyConfiguration.kt (propagating an explicit parent java.security.egd override when set; blank counts as unset). This makes the child JVM's first SecureRandom seed non-blocking, eliminating the ~10 s /dev/random stall that intermittently blew every libp2p/TLS test timeout org-wide. Four nested-workspace tests (tests/testChildTestJvmReceivesNonBlockingEntropyConfiguration.kts, ...InheritsExplicitEntropySourceOverride.kts, plus blank + space-value scenarios) assert the property from inside a real spawned child JVM. Two codex review passes hardened it — including the empirically-confirmed discovery (probe on JDK 21.0.11) that the legacy /dev/./urandom spelling silently switches SecureRandom from NativePRNG to DRBG; the canonical /dev/urandom form used here preserves the provider.
  • Branch checks: all GREEN (verified 16:50Z): bld-all-tests SUCCESS, bld-build-fat-jars SUCCESS, kotlin.build (remote) SUCCESS; mergeStateStatus: CLEAN.
  • Five merge-queue evictions, each root-caused to a DISTINCT infra failure (zero code delta — the orphaned attempt-4 group run even finished All Tests SUCCESS after eviction):
    1. 23:30Z — kotlin.build (remote) test timeouts during a host load-85 storm.
    2. 00:18Z — bld-build-fat-jars MavenFetchFailedException: Failed to fetch Maven artifacts for MavenPrebuilt batch after 3 attempts vs kotlin.directory.
    3. 00:37Z — ten 240–360 s test timeouts stuck in dependency resolution at host load 110.
    4. 02:25Z — kotlin.build (remote) "0 passed, 0 failed, 376 total" = the DigitalOcean droplet-create-429 zero-test signature (global API quota was fine at 4957/5000 — it is the create burst window, not the hourly quota; verified via live DO rate-limit headers read from the CN config.json token).
    5. 03:15Z — MavenFetchFailedException again, while a concurrent 10-shard UrlResolver run (each shard resolving the full dependency set) saturated kotlin.directory.
  • The systemic finding is filed and merged as a challenge: kotlin.directory cannot serve concurrent CI fan-out — full post-mortems, measured ~1-in-5 drop rate under load-85, the working gates, and durable-fix proposals (caching reverse proxy in front of the maven-server container; larger MavenPrebuilt retry budget; staggered buildtest shard resolution).

2b. UrlResolver #680 — resilience fixture, CONTENT DONE, two residual mechanical blockers

  • Head: d98bd3da (branch live-projection-resilience-fixture). Content: the Phase-C live-projection resilience acceptance fixture (server-lineage dead-lineage rebase + review fixes). Its kotlin.build (remote) ran the full 1090-test suite GREEN at 04:41Z on this exact head — content is validated on real infrastructure.
  • The "red" the user saw was TWO separate mechanical issues, both now understood:
    1. A semantic test conflict (FIXED on-branch). A sibling session's commit 999270c7 ("Fail reconnect fast when no discovery channel exists") made connectPersistentRpc exit its reconnect retry loop instantly when no discovery channel exists — which broke the issue-717 regression test tests/testPersistentRpcReconnectDoesNotBlockConcurrentCaller.kts (topology bootstrapPeers = emptyList(), assertion "First caller should still be inside the real reconnect retry loop … Expected <1>, actual <0>"). The same sibling session then pushed the fix (999270c7, "Give issue-717 regression test a live second bootstrap peer") giving the test a live discovery channel so the retry loop is legitimately entered. bld-build (Build and Test) passed GREEN on that fix at 00:47Z.
    2. A recurring version-pin collision (fix keeps getting reverted by rebases). The branch's build.kts publishes foundation.url:resolver, and its pin repeatedly regresses onto an already-published coordinate every time the sibling session rebases. It has been bumped THREE times (0.0.626→629→632→634) and each rebase reverted it. At write time the current head d98bd3da pins 0.0.634, but 0.0.634–0.0.637 are now ALL published (200) — so this pin WILL collide on merge-publish. ⚠️ The pin must be re-bumped to a verified-free version before enqueueing. As of 17:00Z, 0.0.638 is free (401); re-probe upward before using it.

2c. The wedged bld-build GHA run on #680 (BLOCKING)

  • #680's bld-build GitHub-Actions check (Bld - Build and Test, run 29176780753) has been in_progress since 02:27Z — 14+ hours at write time. It is wedged/orphaned. The prior session issued a gh run cancel + gh run rerun on it at ~04:50Z, but that automation did not persist (the session was torn down before it completed), so the run is still stuck. This is why #680 shows mergeStateStatus: BLOCKED despite kotlin.build (remote): SUCCESS.

3. Relevant PRs / refs (write-time state, verified 2026-07-12 16:50Z — RE-VERIFY)

| Ref | State @ write | Link | |---|---|---| | kompile-core #213 (entropy fix) | OPEN, CLEAN, all 3 branch checks SUCCESS, NOT in queue, 5 prior evictions (all infra) | https://github.com/CodexCoder21Organization/kompile-core/pull/213 | | UrlResolver #680 (observables tail) | OPEN, BLOCKED, head d98bd3da, kotlin.build (remote) SUCCESS, bld-build wedged in_progress 14h, pin 0.0.634 (now collided) | https://github.com/CodexCoder21Organization/UrlResolver/pull/680 | | UrlResolver #707 (sibling, idempotent dispatch) | MERGED 2026-07-11 19:20Z | https://github.com/CodexCoder21Organization/UrlResolver/pull/707 | | UrlProtocol #353 (production-side path-B entropy follow-up) | OPEN (deferred) | https://github.com/CodexCoder21Organization/UrlProtocol/issues/353 | | UrlResolver #753 (residual RESURRECTION race, local reproducer) | OPEN | https://github.com/CodexCoder21Organization/UrlResolver/issues/753 | | kotlin.directory concurrent-fan-out challenge | MERGED | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-12-0323-kotlin-directory-cannot-serve-concurrent-ci-fan-out-10.md | | Parent observables handoff (this PR's origin) | — | https://github.com/CodexCoder21Organization/PlanRepository/blob/main/handoffs/2026-07-09-observables-close-16-merges-fleet-ci-fixed-707-680-tail.md |

Unsaved work: none. All code is pushed to the two PR branches. The prior session's watcher scripts were in the session scratchpad only (not code) and are gone with the session — nothing to preserve. Host coordination audit line was written to /root/cn-watchdog.audit at 03:23Z.

4. Next steps (for the next person — re-verify §RE-VERIFY first)

Host must be healthy before touching either queue (load was 2.25 at write time — good; the evictions all happened at load 80–110). Gate on: loadavg < 20 and a 5/5 curl probe of a kotlin.directory .pom returning 200.

A. kompile-core #213 — just needs a clean enqueue (it is CLEAN and branch-green).

  1. Confirm still CLEAN + branch checks green; confirm host healthy; confirm no other 10-shard buildtest run is provisioning/fetching (serialize heavy CI consumers — this is the #1 lesson; enqueuing 213 while 680's suite fetches is what caused eviction #5).
  2. Enqueue via GraphQL (kompile-core uses a merge queue; gh pr merge just enqueues): 
    PRID=$(gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"kompile-core"){pullRequest(number:213){id}}}' --jq '.data.repository.pullRequest.id')
gh api graphql -f query="mutation { enqueuePullRequest(input:{pullRequestId:\"$PRID\"}) { mergeQueueEntry { position state } } }"
  3. Watch to MERGED-or-evicted. If evicted a 6th time, read the merge-group run's failure — do NOT blind-retry. Distinguish infra (Maven fetch / DO-429 zero-test / resolution timeout — wait for a healthy+quiet window and re-enqueue) from a genuine code failure (would be new — investigate). Use the double-confirm watcher pattern (the merge queue briefly shows a PR absent between groups → confirm eviction twice 20 s apart before concluding).

B. UrlResolver #680 — three things, in order:

  1. Re-bump the version pin to a verified-free coordinate. Fresh-clone or git fetch origin pull/680/head, git checkout FETCH_HEAD, edit build.kts (both the // 0.0.NNN: changelog comment line and coordinates = "foundation.url:resolver:0.0.NNN"), probe upward from 0.0.638 for the first 401, commit additively (NO force-push — a sibling session shares this branch), git push origin HEAD:live-projection-resilience-fixture. Leave a PR comment noting the bump and the probe procedure (there are already two such comments).
  2. Un-wedge bld-build. Run 29176780753 has been in_progress 14h. gh run cancel 29176780753 --repo CodexCoder21Organization/UrlResolver, wait for it to reach completed, then gh run rerun 29176780753 (or push in step 1 will trigger a fresh run on the new head, which is cleaner — the new commit gets its own bld-build and kotlin.build). Prefer step 1's push to naturally supersede the wedged run, then just watch the new head's checks.
  3. On the new head: both branch checks green → enqueue #680 (UrlResolver merge queue, same GraphQL mutation with name:"UrlResolver" / number:680). Watch the merge group; the known risk in an UrlResolver merge group is the #753 residual RESURRECTION race (~2%/group, a pre-existing main-side flake, NOT #680's content) and perftest-service RPC-timeout flakes — both are infra/known, re-enqueue if they fire.

C. When both are MERGED: delete this handoff (its own standing pre-approval — see the skill), and the entropy + observables-tail workstream is fully closed. Optionally close #753 only if separately fixed (it is a real open bug, not part of this merge).

5. Reusable / operational knowledge (hard-won this session)

  • Serialize heavy CI consumers. kotlin.directory (a single Jetty maven-server container on the shared 8-core CN droplet) cannot serve two 10-shard fan-outs at once; it drops ~1-in-5 requests under load-85, and kompile's 3-attempt/~18 s MavenPrebuilt retry budget then exhausts → MavenFetchFailedException. Never enqueue a merge group while another 10-shard run is provisioning or fetching. (Filed: the challenge in §2a.)
  • DO droplet-create window ≠ hourly quota. A kotlin.build (remote) result of "0 passed, 0 failed, N total" in ~85 s is the DigitalOcean create-burst 429 signature even when /v2/account shows thousands of requests remaining. Read live headers: extract the token from /root/ContainerNursery/config.json (grep -oE 'dop_v1_[a-f0-9]+'), curl -D- -H "Authorization: Bearer $TOK" https://api.digitalocean.com/v2/account | grep -i ratelimit. Recovery: wait ~25 min for the window to roll, re-fire ONE suite at a time via check-suites/<id>/rerequest.
  • /dev/./urandom is a trap on modern JDKs — it switches SecureRandom to DRBG (probe: java -Djava.security.egd=file:/dev/./urandom → DRBG; file:/dev/urandom → NativePRNG on JDK 21.0.11). Always use canonical file:/dev/urandom.
  • This UrlResolver branch has a live sibling session that rebases it and keeps reverting the version-pin bump onto already-published coordinates. Always additive push, never force-push; always re-probe the pin (.pom 200=taken/401=free) right before enqueue. Versions 0.0.626–637 were all consumed overnight; probe upward.
  • codex launch wedges this session: launching codex exec from a repo cwd, two-at-once, or without < /dev/null reliably wedged (39-byte log, or stuck at the "Reading additional input from stdin…" banner). Launch from the scratchpad cwd with stdin redirected from /dev/null, one at a time, and arm a >200-bytes-after-3-min tripwire.
  • Host/CN: the CN#498 residual log-firehose keeps ContainerNursery pegged at >2 cores on the shared box, amplifying every CI-contention symptom; a coordinated CN restart (audit-logged in /root/cn-watchdog.audit, coordinate with other sessions first) reclaims it if it becomes the bottleneck again.