A concurrent ContainerNursery deployment invalidated BuildTestEmbedded PR CI run e23e54be by

Reported (UTC): 2026-07-15 20:19

A concurrent ContainerNursery deployment invalidated BuildTestEmbedded PR CI run e23e54be by taking kotlin.directory offline during artifact resolution. Verified timeline on host 198.199.106.165: /root/ContainerNursery/bin/container-nursery.jar mtime became 2026-07-15 20:08:21 UTC; a shell-launched ContainerNursery process started at 20:08:25; its startup log reported all prior child PIDs, including https:kotlin.directory:443, were no longer alive; the Maven route first failed to bind, then a second process started and bound at 20:08:34. There were no contemporaneous OOM-killer events or new heap dumps. Four unrelated tests later surfaced Coursier failures whose exact mechanism was java.net.ConnectException Connection refused to https://kotlin.directory for many artifacts. Current health checks return HTTP 200 for the root and a representative POM, confirming recovery. Impact: the 535-test, 10-shard run spent over 20 minutes compiling on the old image, then became irrecoverably red due to an infrastructure deployment overlapping dependency fetches. Workaround: delete the doomed CI run and rerequest only after repeated repository health checks pass. Suggested durable fix: coordinate ContainerNursery deployments with active CI runs or provide zero-downtime child-route handoff so replacing ContainerNursery does not kill kotlin.directory; at minimum, deployment tooling should expose a maintenance/drain signal consumed by BuildTest admission before artifact resolution begins.

Production verification — 2026-07-17

Status: STILL EXISTS. The live ContainerNursery process remains central to the reported mechanism and was consuming 238% CPU with 543 threads; host load average was 60.59 and CPU pressure was near 90%.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.