HardwareControlFabric daemon refused production-host connections during CI ground-truth inspection.

Reported (UTC): 2026-07-13 17:11

What was being attempted: Use the required HardwareControlFabric-first server-management path to inspect buildtest run 78ad3321 file freshness after build-watchman reported 0/100 tests outside the normal phase budget.

What went wrong: community.kotlin.hardwarecontrolfabric:cli:0.2.6 list-functions against 198.199.106.165:8443 failed with 'Error: Connection refused' and 'java.net.ConnectException: Connection refused'. The skill's documented default certificate filenames were also absent; the installed valid paths are client_cert.pem, fabric_key.pkcs8, and server_cert.pem, but using those reached the host and still got connection refused.

Impact: The preferred audited file/process inspection route is unavailable during a potentially wedged production CI run, forcing operators onto fragile raw SSH for the Layer 3 ground-truth evidence required before any safe rerequest.

Workaround used: Follow the documented hierarchy's last-resort exception and use read-only SSH on port 23 solely to stat and tail the run's test-events.jsonl and build.log files; do not mutate or restart services.

Suggested durable fix: Restore HardwareControlFabricDaemon on port 8443 with an automatic restart supervisor and update the skill's primary certificate examples to the installed client_cert.pem/fabric_key.pkcs8/server_cert.pem names.

Production verification — 2026-07-17

Status: STILL EXISTS. The required mTLS files now exist, but no cs command is installed, the live CLI ping failed, and port 8443 refused connections. Read-only SSH confirmed that no HardwareControlFabric daemon process is running; its last log ends on 2026-07-12 amid Netty ByteBuf leak reports and thread growth from 88 to 108.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.