HardwareControlFabric was unavailable during buildtest CI ground-truth inspection.

Reported (UTC): 2026-07-15 13:20

What was being attempted: Use the preferred HardwareControlFabric read-file path on 198.199.106.165 to inspect /root/buildtest-data/runs/829d8d7f/test-events.jsonl and build.log after build-watchman requested host-side confirmation.

What went wrong: community.kotlin.hardwarecontrolfabric:cli:0.2.6 failed immediately against 198.199.106.165:8443 with 'Error: Connection refused' and java.net.ConnectException: Connection refused.

Impact: The preferred authenticated, observable server-management plane could not perform a read-only production diagnostic, forcing use of the explicitly last-resort raw SSH path during active CI monitoring.

Workaround used: Ran only read-only stat and tail commands over ssh -p 23 root@198.199.106.165; no process, file, run, or service state was mutated.

Suggested durable fix: Add supervised automatic restart and health alerting for HardwareControlFabricDaemon on 198.199.106.165, then verify port 8443 and the daemon process whenever the alert fires so operators do not silently fall back to SSH.

Production verification — 2026-07-17

Status: STILL EXISTS. The required mTLS files now exist, but no cs command is installed, the live CLI ping failed, and port 8443 refused connections. Read-only SSH confirmed that no HardwareControlFabric daemon process is running; its last log ends on 2026-07-12 amid Netty ByteBuf leak reports and thread growth from 88 to 108.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.