Handoff: Reproduce and fix the HTTPS graceful-shutdown TCP-admission test retry

Written 2026-10-22 11:49 UTC.

RE-VERIFY: This is a write-time snapshot. Re-read https://github.com/CodexCoder21Organization/ContainerNursery/pull/702 with gh pr view ... --json state,headRefOid,statusCheckRollup,mergedAt; inspect the CI run and XML linked below; fetch current main before building. No root cause is established by this record.

Design decisions in force

Owner direction (2026-10-21, quoted in the FIX702b brief): "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

The active lane rules prohibit merging/enqueueing, artifact publication, deployment, service restarts and buildtest deletion/rerequests. Do not increase timeouts, reduce iterations, skip tests, weaken assertions, use mocks/reflection, or expose private members for testing. Local tests only; no JVM tuning flags. Full suites require the machine-wide lock and memory logging if that machine's rules still apply. Read the authoritative TESTING.md and PHILOSOPHY.md before writing tests. A new owner must verify current instructions rather than treating this snapshot as new permission.

Mission and provenance

The remaining work is to reliably reproduce and fix the first-attempt failure of testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer. It was observed while FIX702b completed two test-quality review fixes on the linked PR. Those two requested fixes are complete and the required Actions checks are green; the separate unchanged HTTPS test needed an automatic retry in the full CI XML. This record preserves that observation without claiming a cause or a fix.

What was found and done

1. The exact-head build/test job is https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39767047299/job/119784983763 . Its uploaded bld-test-results XML contains 907 final successes and one global attempt-history, so 906/907 first-attempt successes.
2. containernursery.testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer failed attempt 1 at 6151 ms with full assertion text: A new TCP connection must not be admitted after graceful shutdown starts. Attempt 2 passed at 3122 ms. This is one CI observation plus the runner's automatic retry, not a measured reproduction rate and not proof the defect disappeared.
3. The test first waits for the injected-clock drain deadline to be scheduled, asserts one scheduling call and listenerClosedAtSchedule == true, then opens a new real TCP Socket to the released loopback port and requires connect to fail. That last assertion failed. The earlier assertions did not fail in this recorded attempt. The log includes HTTPS graceful drain started and a later TLS_INCOMPLETE_HEADER ... bytesRead=0 line. These facts do not identify why the new TCP connect succeeded, or whether it reached the old listener or a different listener; investigate the mechanism rather than guessing.
4. git diff origin/main..HEAD -- tests/testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer.kts is empty at the captured baseline. Both main 57a4e8000738a069f55d51be09e6b8f19a39c0b3 and PR head 8fbc2c580f8b7359f84d08bc0182b217f14378f5 contain the same test. Last change reported by git log is https://github.com/CodexCoder21Organization/ContainerNursery/commit/760f3644d95a8bef2893d06f501531917f58fc9b . No local reproducer or fix was attempted for this HTTPS failure.
5. Both changed Direct tests passed attempt 1 in that same CI XML (retry 5906 ms; reconnect 3913 ms), and each passed three ordinary final-head local runs without attempt-history. Their local controls and complete source/test review are preserved separately in the evidence branch. No CI rerun was requested.
6. Searched all current local lane findings for the exact test name and checked the active central handoff list for a matching title/summary. No specific existing handoff for this failure was found; the broad jar-history handoff mentions a different idle HTTP listener test. Re-check ownership before duplicating an investigation.

Relevant PRs and recoverable refs

All branches below were pushed and their remote SHAs read with git ls-remote. No stash, uncommitted source, or local-only source commit remains in the two FIX702b worktrees.

Stable evidence at the CI checkpoint: FIX702b-ci-artifact/test_log_file.xml, FIX702b-ci-summary.json (includes SHA256), FIX702b-ci-retry-full.xml, FIX702b-ci-retry-full.txt, FIX702b.md and FIX702b-review.md. The full retry trace is linked rather than summarized away; download it before investigating.

Deployed/published-but-unmerged: none from FIX702b. The action history contains only local builds/tests, GitHub source/evidence pushes, PR body edits and read-only CI queries; there was no deployment, restart, Maven publication or merge. Build outputs, dependency caches and the locally generated diagnostic-control jar are intentionally omitted from evidence. Its reproducible source driver is preserved; no jar was published.

Next steps

1. Re-verify PR/main/CI and ownership first. Read https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md in full, plus the exact test and HttpsFacadeProvider.shutdownGracefully implementation.
2. Clone fresh into the required workspace directory. Fetch/rebase before each build. Start with scripts/test.bash --local --test testHttpsGracefulShutdownDeadlineClosesIdleTlsPeer --log <owned-path.xml> and count only attempt 1 from XML.
3. Establish a reliable local reproduction through the public API, using genuine narrow-path concurrency for discovery. Once the exact condition is known, force it deterministically. Distinguish an old-listener admission/lifecycle problem from a test observation that can reach another listener after port release; neither is established here. Do not fix based solely on this CI sample, blame host speed, add sleeps, or weaken the TCP-admission assertion.
4. Write the lifecycle ownership/invariant table and public-API failure tests before implementation. Fix the mechanism at the layer that owns it, keep every existing timeout/iteration/assertion, prove red-to-green, review tests/code independently, then open a separate PR and watch required CI without blind reruns. No merge or deployment without new explicit authority.

Operational notes

REST core quota was shared/exhausted twice during FIX702b; GraphQL PR status reads remained available. It recovered at 11:42 UTC and the bounded build-watchman exited GREEN. That read outage is not evidence of a test failure. Informational kotlin-build-ci-test suite 107771872008 had zero dispatched runs; no rerequest was performed. It did not gate the required Actions checks. Use first-attempt XML records rather than final job success when discussing this test.