The watch-build workflow flagged BuildTestEmbedded CI run 3f8ac241 at 0/100 after 8 minutes and

Reported (UTC): 2026-07-15 22:37

The watch-build workflow flagged BuildTestEmbedded CI run 3f8ac241 at 0/100 after 8 minutes and prescribed deleting it with the server-side buildtest-cli. Per the server-management hierarchy I attempted the preferred HardwareControlFabric CLI 0.2.6 first, using the exact default certificate paths documented in /home/helena/.codex/skills/hardware-fabric-processes/SKILL.md. It failed with 'Certificate file not found: /home/helena/.config/hardware-control-fabric/fabric-client-cert.pem'; the entire /home/helena/.config/hardware-control-fabric directory is absent, and a search under /home/helena found none of the alternate documented client_cert.pem, fabric_key.pkcs8, or server_cert.pem files. The documented launch form also rejected --args "-jar /root/ContainerNursery/apps/buildtest-cli.jar delete 3f8ac241" with 'Missing argument for option: args' because the argument begins with a dash; parser-safe documentation should use --args=-jar... or another supported encoding. Impact: agents instructed to use HardwareControlFabric cannot perform preferred authenticated server operations and must fall back to raw SSH, defeating the management hierarchy during time-sensitive CI recovery. Durable fix: provision the mTLS bundle in the documented location for Codex environments, add a credential-discovery/preflight command to the skill, and correct the leading-dash launch-argument example/CLI parsing.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.