HardwareControlFabric diagnostics are unusable because the required mTLS credentials are absent

Reported (UTC): 2026-07-15 15:28

HardwareControlFabric diagnostics are unusable because the required mTLS credentials are absent from the Codex environment.

What was being attempted: perform the required read-only forensic investigation of BuildTestEmbedded on production host 198.199.106.165 using the preferred HardwareControlFabric CLI and the documented default certificate paths.

What went wrong: the CLI failed before contacting the daemon with exact error 'Error: Certificate file not found: /home/helena/.config/hardware-control-fabric/fabric-client-cert.pem'. The directory /home/helena/.config/hardware-control-fabric does not exist, and a bounded search under /home, /root, and /code found none of fabric-client-cert.pem, client_cert.pem, fabric-server-cert.pem, server_cert.pem, or fabric_key.pkcs8.

Impact: production diagnostics cannot use the mandated preferred management plane, forcing the documented last-resort SSH path for read-only evidence collection and making investigations more fragile.

Workaround used: use read-only SSH on port 23 only after verifying the HardwareControlFabric credentials are unavailable; do not restart or mutate the service.

Suggested durable fix: provision the HardwareControlFabric client certificate, private key, and CA certificate into the Codex execution environment at a single canonical path, and update the hardware-fabric-processes skill prerequisite check to detect and report all supported filenames before launching the CLI.

---

Production verification — 2026-07-17

Status: STILL EXISTS. The required mTLS files now exist, but no cs command is installed, the live CLI ping failed, and port 8443 refused connections. Read-only SSH confirmed that no HardwareControlFabric daemon process is running; its last log ends on 2026-07-12 amid Netty ByteBuf leak reports and thread growth from 88 to 108.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.