During CI recovery for UrlResolver PR 770, the required HardwareControlFabric-first host

Reported (UTC): 2026-07-13 21:58

During CI recovery for UrlResolver PR 770, the required HardwareControlFabric-first host verification could not be used. hardware-fabric-processes SKILL.md documents default certificate paths fabric-client-cert.pem/fabric-client-key.pem/fabric-server-cert.pem, but ~/.config/hardware-control-fabric actually contains client_cert.pem, fabric_key.pkcs8, and server_cert.pem; the documented command first failed with 'Certificate file not found'. Retrying with the actual files reached the service but returned 'FAILED - Daemon did not respond'. Impact: the preferred audited method could not verify that buildtest rerequest run 57c68dad materialized, forcing the explicitly permitted last-resort read-only SSH check to 198.199.106.165. Workaround: used the actual cert filenames to confirm the daemon outage, then SSH only for stat/tail of /root/buildtest-data/runs/57c68dad/{build.log,test-events.jsonl}. Durable fix: update the skill's default certificate paths to match installed names (or support discovery), and add supervised auto-restart/health monitoring for HardwareControlFabricDaemon so server operations do not routinely fall back to SSH.

Production verification — 2026-07-17

Status: STILL EXISTS. A live ProductionHealth URL connection succeeded but emitted repeated NothingToCompleteException failures while forwarding gossip; the last HardwareControlFabric daemon log also contains Netty ByteBuf leak reports in libp2p negotiation paths.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.