While remediating a wedged kotlin.build run on production host 198.199.106.165, the mandatory

Reported (UTC): 2026-07-16 03:02

While remediating a wedged kotlin.build run on production host 198.199.106.165, the mandatory HardwareControlFabric-first path in the hardware-fabric-processes skill could not be used for two independent reasons. First, the documented command '... launch --cmd java --args "-jar /root/ContainerNursery/apps/buildtest-cli.jar delete 0252e8a0" --workdir /root --wait' failed exactly with 'Error: Missing argument for option: args'; Apache CLI treated the leading '-jar' value as another option even though it was passed as one quoted argv. The parser-safe form is '--args=-jar /root/ContainerNursery/apps/buildtest-cli.jar delete 0252e8a0', which the skill does not document. Second, that corrected command failed exactly with 'Error: Certificate file not found: /home/helena/.config/hardware-control-fabric/fabric-client-cert.pem' and IllegalArgumentException from CertificateUtils.loadCertificate; '/home/helena/.config/hardware-control-fabric/' contained no files at all, including neither the primary documented names nor the alternative client_cert.pem/fabric_key.pkcs8/server_cert.pem names later in the skill. Impact: the preferred production-management hierarchy was unavailable during an active CI stall, forcing the explicitly last-resort SSH command to run buildtest-cli delete. Workaround used: 'ssh -p 23 root@198.199.106.165 timeout 90 java -jar /root/ContainerNursery/apps/buildtest-cli.jar delete 0252e8a0', which succeeded, followed by a GitHub check-suite rerequest. Suggested durable fixes: provision/rotate the mTLS client credentials into the documented Codex runtime location (and add a startup/preflight that verifies them), update the skill's launch example to use '--args=...' when arguments begin with '-', and ideally register a narrowly scoped fabric function for buildtest run deletion so callers do not need to encode a Java subprocess command.

Production verification — 2026-07-17

Status: STILL EXISTS. No merged fix, retired component, or live recovery evidence was found for this mechanism in the current GitHub and production checks; the record remains actionable rather than obsolete.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.