Resolve the conflicting build-server routes sharing one writer directory
Written 2026-10-27 06:22 UTC by lane hq053.
RE-VERIFY: Snapshot only. Read current /root/cn-watchdog.audit, inspect the router command line and sanitized build-server route definitions through HardwareControlFabric. At 06:14 UTC router PID1273890 ran the restored 61037c23 JAR; no service lifecycle command or route edit was made by hq053.
Direction in force
The assigned lane says: “Never deploy to production, restart services, or change GitHub rulesets/branch protection.” It also prohibits route edits for this investigation. The supervisor makes every merge decision. No source change, publication, merge or deploy was made.
Mission and result
The original request was to explain the duplicate writer after ce4d894a deployment, reproduce a ContainerNursery adoption bug, and prepare a fix. Read-only investigation identifies a configuration conflict between separate route keys. The proposed same-route adoption/cleanup bug is refuted by the observed lifecycle sequence; no source fix or failing adoption test is warranted for that different mechanism.
The mechanism is two independent configured routes launching writer JVMs against the same DATA_DIR. At05:36:34 the router auto-starts persistent url:kompile-remote-build:; at05:36:36 it launches the versioned0.1.30 JAR on port39379. Its process reaper is PID1232622, and the lock acquisition metadata timestamps05:36:40.853. At05:41:22 a request to the legacy HTTPS domain creates https:kompile-build-ci-server.kotlin.build:443; at05:41:23 that separate route launches legacy JAR PID1256928. Its startup exits before readiness because the shared directory is locked by PID1232622. The request is GET /.env; failed-generation cleanup confirms death05:41:28.359.
All three configured server routes share /root/ContainerNursery/data/kompile-build-ci-server: canonical url:kompile-remote-build:, legacy url:kompile-build-ci:, and legacy https:kompile-build-ci-server.kotlin.build:443. Both binaries have entry point kompile.remotebuild.server.MainKt. The canonical versioned JAR hash is4ee3ce700c944c13ee195a69caef05f1b2cfd9be57962190a68917ab58703fc3; the legacy JAR hash isb139edf91b676c7617b78aca1c129c84bcd7f45f781dc2f2ccb04fa4bc036190. Neither build-server route appears in the original27-child adoption census. Configuration and route hashes stayed unchanged across deployment per the saved census. RouteKey.kt and DangerJarFileContainerFactory.kt did not change between61037c23 andce4d894a.
The17 ERROR lines in the deployment watch are lines in a retained stderr snapshot, not17 launches. Two complete lock-error stacks are present: one older holder4048538 and one current holder1232622. Older retained router archives .2/.3/.4 contain zero exact lock-rejection lines; no earlier occurrence is asserted.
Owner decision and next steps
QUESTION: Should the legacy URL and HTTPS server routes be retired (recommended), or maintained through an explicit way to reach the canonical writer? Giving them separate DATA_DIR values creates independent CI state and needs an explicit migration decision. Arbitrarily coalescing JAR routes in ContainerNursery by an application-specific variable changes its route contract and is not the correction.
- Re-verify current configuration and router state before acting.
- Identify consumers of the two legacy server routes, then approve retirement or shared-instance access.
- Apply the owner-authorized route correction and verify only one writer. hq053 is prohibited from doing this.
- Decide whether another main deployment is appropriate after the configuration decision; this investigation does not establish a cleanup-ownership regression.
No product PR exists from hq053. Tests/builds were not run for this evidence-only result. Current main already has racing bld-build with kompile0.0.124 and effective main rule bld-build only. No deploy, publication, restart, ruleset or queue operation occurred.
Relevant references and durable work
| Repository |
Branch |
Remote head SHA |
PR |
Contents/state |
| ContainerNursery |
hq053 evidence |
810a4b3af5b0799ce12fdcb577e1f0a775a438d9 |
No PR |
Sanitized config, exact lifecycle/lock logs, collector scripts, report; no product change; never built |
| ContainerNursery |
hq046 deployment evidence |
181954671735fdaf56a36420e2acce6eb4fe4b13 |
No PR |
Prior deployment/rollback/censuses, read fully for this investigation |
Full hq053 report contains source links and exact verification limits. hq053 deployed or published nothing; restored61037c23 is a merged ancestor of current maince4d894a, not an unmerged fix. Prior candidate and rollback hashes remain in the original snapshot below.
HardwareControlFabric read-file rejects the10.6MB audit at its10MB cap; remote tail works. Large grep stdout can hit the launch API's60s process bound; use the focused read-only Python collectors with bounded output. No timeout was raised. Full raw configuration, unrelated logs and credentials were deliberately not copied.
Original handoff snapshot (retained for provenance; findings above supersede its diagnosis)
Explain the duplicate kompile-build-ci-server start after deploying ContainerNursery main ce4d894a
Written 2026-10-27 ~06:00 UTC by opus-code-2026-10-27.
RE-VERIFY: this is a write-time snapshot. Recheck the live router on 198.199.106.165 (it should be 61037c23, PID 1273890, unit containernursery-hq046-rollback-61037c23-20261027.service), ContainerNursery main, and the tail of /root/cn-watchdog.audit before acting.
What happened
With the owner's authorization, lane hq046 deployed the ContainerNursery router built by CI from exact main ce4d894a (CI run 40430743486, router jar sha256 1fa380fb22d755ee518b968a56e0f31ad54677d70c780a3914716a2aa13dca8c), replacing the live 61037c23.
- 05:35:44Z: the old router stopped. 05:36:44Z: the new router (PID 1229874) adopted all 27 registered app processes with the same PID and start ticks. Config and route hashes were unchanged.
- 05:41:28Z: the router log shows repeated
kompile-remote-build startup failures: the data directory /root/ContainerNursery/data/kompile-build-ci-server was already locked by live holder 1232622@host, acquired at epoch ms 1793079400853 (about 05:36:40Z, i.e. after the new router started). In total there were 17 ERROR lines. Something tried to start a second writer for an app whose instance was already running.
- 05:44:42Z: under the owner's watch condition, the lane rolled back to
61037c23, which is healthy.
The default-handler process changing PID during the watch is not part of this problem. Logs show it was adopted at 05:35:55 and stopped at 05:41:03 after 308 s idle with reason=IDLE_TIMEOUT (keepWarm 300).
What differs between live and the deployed build
61037c23...ce4d894a is exactly two commits:
- e3cbcade "Fence cleanup ownership across blocking lifecycle calls", ContainerNursery PR 742. It changes app lifecycle ownership: it is the only product change and the prime suspect.
- ce4d894a, JAR listener inode boundary tests (PR 755); tests only.
It is not yet established whether the duplicate start comes from PR 742, from adoption itself (also present in 61037c23), or from the app (for example a self-restart on lock contention). Holder PID 1232622 also needs identifying: whose child is it, the adopted instance or one started by the new router?
Mission
Find the mechanism, reproduce it in a ContainerNursery test that fails (router replacement with adoption of a JAR app that holds an exclusive data-directory lock, then whatever makes CN start a second instance), fix it test-first, and get the PR ready. Main must be deployable again: the next deploy is an owner-approved step, so don't deploy.
Evidence
- Branch fix/hq046-cn-router-deploy-evidence at
181954671735fdaf56a36420e2acce6eb4fe4b13: sanitized deploy evidence, the before/after app census, the operator and rollback scripts, and the watch samples. It has no product changes.
- Host logs: router and container logs for the window 05:35–05:45Z on 198.199.106.165,
/root/cn-watchdog.audit entries ROLLBACK_START/ROLLBACK, and the kompile-build-ci-server route's own logs.
- Pre-deploy route failures that already existed (not caused by this deploy): Handoff 503, BuildTest 502 "service is not ready", filedrop 503, TinySite 103 with no body.
Rules
Read /root/cn-watchdog.audit before touching the host and append an entry. Inspect read-only only: no restarts, deploys or route edits. Reproduce test-first through the public API. Don't raise timeouts or cut iterations.