HardwareControlFabric daemon refused production host connections during BuildTest CI inspection.

Reported (UTC): 2026-07-13 11:14

What was being attempted: use the preferred HardwareControlFabric read-only process launcher to stat /root/buildtest-data/runs/b0c3c224/build.log and test-events.jsonl after build-watchman requested host-side progress confirmation.

What went wrong: two independent CLI invocations against 198.199.106.165:8443 failed with exact error Error: Connection refused and java.net.ConnectException: Connection refused from TlsTransport.executeCommand / FabricApiClient.launchProcess.

Impact: HardwareControlFabric could not provide the mandated preferred production-management path, forcing use of last-resort raw SSH to establish whether the CI run was genuinely progressing.

Workaround used: read-only ssh -p 23 root@198.199.106.165 stat/tail commands confirmed fresh build and test-event files; no remote state was changed.

Suggested durable fix: restore the HardwareControlFabric daemon on port 8443 and add monitored automatic restart/health alerting so operators do not silently fall back to SSH when the daemon is unavailable.

Production verification — 2026-07-17

Status: STILL EXISTS. The required mTLS files now exist, but no cs command is installed, the live CLI ping failed, and port 8443 refused connections. Read-only SSH confirmed that no HardwareControlFabric daemon process is running; its last log ends on 2026-07-12 amid Netty ByteBuf leak reports and thread growth from 88 to 108.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.