BuildTest production stores its droplet SSH private key at /root/buildtest-data/ssh/buildtest_key

Reported (UTC): 2026-07-15 16:55

BuildTest production stores its droplet SSH private key at /root/buildtest-data/ssh/buildtest_key with permissions 0644. While diagnosing active CI run bd4dec01 on 2026-07-15, OpenSSH refused the key for every shard with the exact errors 'WARNING: UNPROTECTED PRIVATE KEY FILE!', 'Permissions 0644 for ... are too open', 'This private key will be ignored', and 'Load key ...: bad permissions', preventing direct process inspection on the ten CI droplets. The BuildTest JSch client apparently tolerates this mode, so normal service operation hides both the security hygiene problem and the loss of standard SSH diagnostics. No production mutation was made as a workaround. Durable fix: create the key atomically with owner-only 0600 permissions, chmod existing keys to 0600 during service startup/migration, and add an end-to-end test asserting the persisted private-key mode.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.