ContainerNursery HTTPS access logging silently records nothing due to a route-key mismatch,

Reported (UTC): 2026-07-15 23:25

ContainerNursery HTTPS access logging silently records nothing due to a route-key mismatch, crippling production forensics

What was being attempted: Forensic investigation of the 2026-07-15 kotlin.directory request storm (buildtest run 392091d0 reporting 29m25s wall / 1m22s CPU for its build-rules phase). We needed per-request access logs (source IP, path, status, bytes) for https:kotlin.directory:443 to quantify per-droplet request counts and repeat-download ratios.

What went wrong: No HTTPS access log exists anywhere on the server. Container request-log storage is created under the route key "https:kotlin.directory:443", but HttpsFacadeProvider.kt calls nursery.logRequest("${host}:${port}", ...) ��� i.e. "kotlin.directory:443" without the "https:" prefix ��� so the lookup misses and every HTTPS request is silently dropped from the request log (the only *.requests.log on the box is default_http_80.requests.log). The custom TLS-termination proxy path also never calls eventStore.recordRequest (request_events.db contains zero HTTPS events, only default:http:80), and HTTPS requestTracing defaults to false, so successful-request evidence lives only in a bounded 10k-line in-memory buffer that had long rolled over.

Impact: It was impossible to compute per-droplet request counts, repeat-download ratios, or per-artifact totals for the storm from server records. The investigation had to fall back to short live tcpdump loopback samples of the maven-server port and droplet-side /tmp forensics, which cost hours and cannot reconstruct history.

Workaround used: 5-8 second loopback tcpdump samples of maven-server port 45131, plus deltas of the HttpsFacade monotonically increasing request-ID counter in stdout diagnostics (gives all-HTTPS totals only, not per-route).

Suggested durable fix (ContainerNursery repo): pass the exact route key to logRequest on the HTTPS facade path (or normalize keys at lookup so "host:port" and "https:host:port" match), call eventStore.recordRequest from the TLS proxy path, and add per-route request/byte/status counters. Add an e2e test that makes one HTTPS request through the facade and asserts a line lands in that route's .requests.log ��� the current failure mode is silent and only discoverable during an incident.

---

Production verification — 2026-07-17

Status: STILL EXISTS. The live ContainerNursery process remains central to the reported mechanism and was consuming 238% CPU with 543 threads; host load average was 60.59 and CPU pressure was near 90%.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.