← ArchiveArea
Completed

Make ContainerNursery's JAR readiness check mean the successor is actually accepting and serving

Completed 2026-10-27: https://github.com/CodexCoder21Organization/ContainerNursery/pull/758 merged as 89f3685f2; JAR readiness now requires the launched process tree to have accepted the nursery's own connection (kernel inode attribution), OSHI removed. Follow-ups tracked by PRs 766 (awaiting owner approval) and 767 (draft); production not yet redeployed.

Handoff document

Markdown

Completed: ContainerNursery JAR readiness now means the launched process tree accepted the nursery's own connection — 2026-10-27 17:00 UTC

RE-VERIFY: everything below is a write-time snapshot taken at 2026-10-27 17:00 UTC. Authoritative re-checks: git log origin/main in https://github.com/CodexCoder21Organization/ContainerNursery (merge commit https://github.com/CodexCoder21Organization/ContainerNursery/commit/89f3685f2587877ee266d5ff5e10996d52040020), and gh pr view <url> --json state,mergedAt for each pull request named here.

Design decisions in force

  • Owner requirement (handoff, 2026-10-27): "Fix so readiness means the successor itself is accepting and serving".
  • Chosen and landed design (2026-10-27): a JAR container on the http, https, tcp and url transports is ready only when the launched process, or a descendant, is alive and has accepted a connection that ContainerNursery itself opened to the backend port. Proof comes from the kernel: the server-side row of that connection in /proc/net/tcp (or /proc/net/tcp6 for an IPv6 probe, which then is a required table) has a nonzero inode, and that inode appears as a socket:[inode] link under /proc/<pid>/fd of a pid in the launched tree. Inode zero (queued in the backlog) is not ready. Accepted outside the tree is not ready; the probe is closed and the next poll opens a fresh connection. EOF, reset or a vanished row before attribution is inconclusive and retried inside the existing startupTimeoutSeconds budget. After attribution the row is re-checked ESTABLISHED. No bytes are written to the app. This proves accepting, not that the app answers requests; the url RPC transport keeps its separate framed-reply probe. Adoption and the Docker provider are unchanged. OSHI is removed from build.kts. Timeouts and polling cadence are unchanged.
  • Standing constraints honoured throughout: no reflection, no mocks, no timeout increases, no iteration cuts, no weakened, skipped or deleted assertions or tests without explicit owner approval; first-attempt failures are real defects; flakes met mid-task are fixed at their cause.

What landed

https://github.com/CodexCoder21Organization/ContainerNursery/pull/758 "Attribute JAR readiness to a connection accepted by the launched process tree" merged into main at 2026-10-27T16:54:47Z as commit https://github.com/CodexCoder21Organization/ContainerNursery/commit/89f3685f2587877ee266d5ff5e10996d52040020 (squash of branch head 54b4b69f3a51f43266788bc9100bec2f1d51d321, 30 commits). Verified on main after the merge: readJarProbeConnection present in src/main/kotlin/org/example/danger_jarfile/DangerJarFileBackedContainer.kt, zero OSHI references in build.kts, the two original reproducers (tests/testDangerJarFile_sharedInheritedListenerDoesNotConfirmForeignServer.kts, tests/testDangerJarFile_boundButNotAcceptingChildIsNotReady.kts) and the three missing-tcp6-table tests present. README.md and docs/ContainerProviders.md carry the accept-only contract, "ContainerNursery sends no bytes", and the IPv6-required-table sentence.

Evidence chain (all recorded in the status reports filed on this handoff): both reproducers red on main before the fix, green after, red again in a reverse control with the old readiness source restored; every adversarial and test-comprehensiveness review finding across heads bfc9b2426 → a044ee705 → 13f55c647 → 54b4b69f3 closed test-first with red and green runs; the final head's required bld-build check green with 1379 of 1379 tests, exactly one of which was retried by the runner (testInheritedHttpsStartupCleanupSettlesExitedApplication, a pre-existing timed window in an untouched fixture; see follow-ups). The optional kotlin.build (remote) check was red on the branch and on the queue ref with a pure infrastructure verdict (the remote build service reported RECOVERING_LOCAL_STATE and ran nothing); it is not a required check under the repository ruleset.

Deployed or published but not merged

This effort deployed and published nothing: no jar was uploaded, no route was changed, no Maven artifact was published. The production ContainerNursery router on 198.199.106.165 was not redeployed and therefore still runs the pre-fix readiness code until someone deploys from main (the owner's explicit ask is required for a production deploy).

Relevant PRs / refs

Repo Branch Remote head SHA PR What is on it State
CodexCoder21Organization/ContainerNursery fix/jar-readiness-accept-attribution 54b4b69f3a51f43266788bc9100bec2f1d51d321 https://github.com/CodexCoder21Organization/ContainerNursery/pull/758 the readiness fix, tests, docs merged to main as 89f3685f2 (squash); branch safe to delete
CodexCoder21Organization/ContainerNursery fix/jar-readiness-ci-fixtures efea2bb0aa1829737ca20756717175b2fad5714c no PR 12 fixture repairs (bind-and-sleep / accept-and-close apps → accept-and-hold) cherry-picked into 758; superseded, safe to delete
CodexCoder21Organization/ContainerNursery fix/jar-readiness-ipv6-review-fixes c18999347908eed0e59aa424787674ba8a327488 no PR absent-tcp6 diagnostic fix and IPv6 tests integrated into 758 as 13f55c647; superseded, safe to delete
CodexCoder21Organization/ContainerNursery fix/jar-readiness-tcp6-worker-timing 13f55c6479108afc107f92783d075f13baafa75a no PR no commits beyond the PR branch (lane found no reproduction) empty, safe to delete
CodexCoder21Organization/ContainerNursery fix/l153-shutdown-settlement ef6a3314880302c2e75b50def86c4126733f0072 https://github.com/CodexCoder21Organization/ContainerNursery/pull/766 L153 test awaits public settlement after bounded shutdown; new contract test OPEN, bld-build green, marked "do not merge" pending owner approval of the test-expectation change (follow-up 1)
CodexCoder21Organization/ContainerNursery fix/inherited-https-exited-status-wait bc144dfa623bae260fbc4e2712ab6d2568768911 https://github.com/CodexCoder21Organization/ContainerNursery/pull/767 inherited-HTTPS scenario observes the adopted app's exit after the successor's startup failure completes; gated regression variant OPEN draft, bld-build in progress; the authoring lane was still running at write time (follow-up 2)

main at write time: 89f3685f2587877ee266d5ff5e10996d52040020.

Follow-ups (tracked by their pull requests, outside this handoff's mission)

  1. Owner decision pending: approve or decline the test-expectation change in https://github.com/CodexCoder21Organization/ContainerNursery/pull/766 (commit https://github.com/CodexCoder21Organization/ContainerNursery/commit/ef6a3314880302c2e75b50def86c4126733f0072). testL153AdoptionClockCannotReplaceCurrentGeneration asserted an active count of zero immediately after a 100 ms-bounded shutdown(); README lines 330–337 document bounded shutdown with cleanup settling afterwards, two sibling L107 tests await settlement, and https://github.com/CodexCoder21Organization/ContainerNursery/pull/742 corrected 35 fixtures for the same assumption. A held-stop control fails the original assertion 3 of 3 times. Recommendation: approve.
  2. Land https://github.com/CodexCoder21Organization/ContainerNursery/pull/767 after review and green CI (test-only; no budget, assertion or iteration change). Its lane also observed two further first-attempt read windows in the same src/test/resources/InheritedHttpsListenerScenarios.java harness (the predecessor's first cold request around line 502 and the restored nursery's first request around line 544), kept out of scope with logs recorded in the lane's findings, and a sibling testInheritedHttpsReplacementWaitsForPersistentApplication failing 3 of 3 locally in its clone with the app stuck "waiting at startup gate" (not yet root-caused at write time; may be caused by the lane's own in-flight edit to InheritedTestSupport.java). Each needs its own small fix-first pull request.
  3. Optional: move the in-test fixture javac (4.8 to 9.0 s) out of the timed body of the two worker-based tcp6 tests (testDangerJarFile_tcp6MetadataFailureNamesTable, testDangerJarFile_missingTcp6TableFailsIpv6Readiness); they once hit their 30 s budget under a load average above 100 and did not reproduce in 18 honest attempts.
  4. Design question for a later slice: an HTTP-level reply probe for http routes so "serving" is proven at the protocol level, at the cost of one app-visible request per cold start.
  5. Coverage notes from review: no scale or soak test exercises many concurrent cold starts through the new attribution loop; several legacy fixtures (testDangerJarFileBackedContainer_urlFacadePortConnectableAfterStart.kts:84, testDangerJarFileBackedContainer_urlFacadeWaitsForTcpReadiness.kts:87) still carry pre-existing timed waits.
  6. Production deploy of ContainerNursery from main, when the owner asks.

Reusable knowledge

  • The kernel row for a connection in /proc/net/tcp shows inode 0 while queued in the listen backlog; after accept() the inode is nonzero and the fd link exists only in the accepting process. This is a reliable, byte-free acceptance proof usable through an okio FileSystem seam.
  • localhost on the CI runner can resolve to ::1; bind fixtures to InetAddress.getAllByName("localhost").first() and treat /proc/net/tcp6 as required only when the probe socket is Inet6Address.
  • The kompile runner retries a failing test silently and prints only "Flaky test: passed on attempt N"; the per-test attempt history exists only in the uploaded test_log_file.xml artifact's <attempt-history> elements. Audit it before enqueueing.
  • gh pr edit fails on this repository; edit descriptions with gh api --method PATCH repos/CodexCoder21Organization/ContainerNursery/pulls/<n> -F body=@file.
  • build-watchman 0.0.21 --to-merged enqueues through GraphQL and follows the queue ref; a failed non-required queue check does not evict by itself here (758 merged with kotlin.build (remote) red on the queue ref).

Model usage

Recorded agent work, including concurrent agents

23 h 4.2 min agent time · 683,472,029 tokens

Aggregate shares of agent-minutes and tokensAgent-minutesgpt-6.1-sol (high): 84.1% (Agent-minutes: 1164.2)84.1%claude-fable-5-1: 15.9% (Agent-minutes: 219.9)15.9%Tokensgpt-6.1-sol (high): 78.4% (Tokens: 535,827,304)78.4%claude-fable-5-1: 21.6% (Tokens: 147,644,725)21.6%

Usage timeline

15-minute columns · empty columns are idle gaps · UTC

Usage timeline: Minutes, By model0%25%50%75%100%2026-10-27 04:45 UTC · gpt-6.1-sol (high): 20.7% of minutes2026-10-27 04:45 UTC · claude-fable-5-1: 79.3% of minutes2026-10-27 05:00 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 05:15 UTC · gpt-6.1-sol (high): 61.9% of minutes2026-10-27 05:15 UTC · claude-fable-5-1: 38.1% of minutes2026-10-27 05:30 UTC · gpt-6.1-sol (high): 90.5% of minutes2026-10-27 05:30 UTC · claude-fable-5-1: 9.5% of minutes2026-10-27 05:45 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 06:00 UTC · gpt-6.1-sol (high): 76.8% of minutes2026-10-27 06:00 UTC · claude-fable-5-1: 23.2% of minutes2026-10-27 06:15 UTC · gpt-6.1-sol (high): 76.7% of minutes2026-10-27 06:15 UTC · claude-fable-5-1: 23.3% of minutes2026-10-27 06:30 UTC · gpt-6.1-sol (high): 82.5% of minutes2026-10-27 06:30 UTC · claude-fable-5-1: 17.5% of minutes2026-10-27 06:45 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 07:00 UTC · gpt-6.1-sol (high): 83.0% of minutes2026-10-27 07:00 UTC · claude-fable-5-1: 17.0% of minutes2026-10-27 07:15 UTC · gpt-6.1-sol (high): 78.9% of minutes2026-10-27 07:15 UTC · claude-fable-5-1: 21.1% of minutes2026-10-27 07:30 UTC · gpt-6.1-sol (high): 94.1% of minutes2026-10-27 07:30 UTC · claude-fable-5-1: 5.9% of minutes2026-10-27 07:45 UTC · gpt-6.1-sol (high): 79.3% of minutes2026-10-27 07:45 UTC · claude-fable-5-1: 20.7% of minutes2026-10-27 08:00 UTC · gpt-6.1-sol (high): 74.7% of minutes2026-10-27 08:00 UTC · claude-fable-5-1: 25.3% of minutes2026-10-27 08:15 UTC · gpt-6.1-sol (high): 90.5% of minutes2026-10-27 08:15 UTC · claude-fable-5-1: 9.5% of minutes2026-10-27 08:30 UTC · gpt-6.1-sol (high): 87.9% of minutes2026-10-27 08:30 UTC · claude-fable-5-1: 12.1% of minutes2026-10-27 09:00 UTC · gpt-6.1-sol (high): 81.7% of minutes2026-10-27 09:00 UTC · claude-fable-5-1: 18.3% of minutes2026-10-27 09:15 UTC · gpt-6.1-sol (high): 71.2% of minutes2026-10-27 09:15 UTC · claude-fable-5-1: 28.8% of minutes2026-10-27 09:30 UTC · gpt-6.1-sol (high): 91.3% of minutes2026-10-27 09:30 UTC · claude-fable-5-1: 8.7% of minutes2026-10-27 09:45 UTC · gpt-6.1-sol (high): 95.9% of minutes2026-10-27 09:45 UTC · claude-fable-5-1: 4.1% of minutes2026-10-27 10:00 UTC · gpt-6.1-sol (high): 93.2% of minutes2026-10-27 10:00 UTC · claude-fable-5-1: 6.8% of minutes2026-10-27 10:15 UTC · gpt-6.1-sol (high): 93.2% of minutes2026-10-27 10:15 UTC · claude-fable-5-1: 6.8% of minutes2026-10-27 10:30 UTC · gpt-6.1-sol (high): 91.2% of minutes2026-10-27 10:30 UTC · claude-fable-5-1: 8.8% of minutes2026-10-27 10:45 UTC · gpt-6.1-sol (high): 82.4% of minutes2026-10-27 10:45 UTC · claude-fable-5-1: 17.6% of minutes2026-10-27 11:00 UTC · gpt-6.1-sol (high): 89.6% of minutes2026-10-27 11:00 UTC · claude-fable-5-1: 10.4% of minutes2026-10-27 11:15 UTC · gpt-6.1-sol (high): 79.8% of minutes2026-10-27 11:15 UTC · claude-fable-5-1: 20.2% of minutes2026-10-27 11:30 UTC · gpt-6.1-sol (high): 90.9% of minutes2026-10-27 11:30 UTC · claude-fable-5-1: 9.1% of minutes2026-10-27 11:45 UTC · gpt-6.1-sol (high): 76.5% of minutes2026-10-27 11:45 UTC · claude-fable-5-1: 23.5% of minutes2026-10-27 12:00 UTC · gpt-6.1-sol (high): 84.1% of minutes2026-10-27 12:00 UTC · claude-fable-5-1: 15.9% of minutes2026-10-27 12:15 UTC · gpt-6.1-sol (high): 91.9% of minutes2026-10-27 12:15 UTC · claude-fable-5-1: 8.1% of minutes2026-10-27 12:30 UTC · gpt-6.1-sol (high): 91.5% of minutes2026-10-27 12:30 UTC · claude-fable-5-1: 8.5% of minutes2026-10-27 12:45 UTC · gpt-6.1-sol (high): 81.3% of minutes2026-10-27 12:45 UTC · claude-fable-5-1: 18.7% of minutes2026-10-27 13:00 UTC · gpt-6.1-sol (high): 75.0% of minutes2026-10-27 13:00 UTC · claude-fable-5-1: 25.0% of minutes2026-10-27 13:15 UTC · gpt-6.1-sol (high): 72.3% of minutes2026-10-27 13:15 UTC · claude-fable-5-1: 27.7% of minutes2026-10-27 13:30 UTC · gpt-6.1-sol (high): 79.0% of minutes2026-10-27 13:30 UTC · claude-fable-5-1: 21.0% of minutes2026-10-27 13:45 UTC · gpt-6.1-sol (high): 84.4% of minutes2026-10-27 13:45 UTC · claude-fable-5-1: 15.6% of minutes2026-10-27 14:00 UTC · gpt-6.1-sol (high): 82.5% of minutes2026-10-27 14:00 UTC · claude-fable-5-1: 17.5% of minutes2026-10-27 14:15 UTC · gpt-6.1-sol (high): 91.2% of minutes2026-10-27 14:15 UTC · claude-fable-5-1: 8.8% of minutes2026-10-27 14:30 UTC · gpt-6.1-sol (high): 80.8% of minutes2026-10-27 14:30 UTC · claude-fable-5-1: 19.2% of minutes2026-10-27 14:45 UTC · gpt-6.1-sol (high): 93.9% of minutes2026-10-27 14:45 UTC · claude-fable-5-1: 6.1% of minutes2026-10-27 15:00 UTC · gpt-6.1-sol (high): 89.1% of minutes2026-10-27 15:00 UTC · claude-fable-5-1: 10.9% of minutes2026-10-27 15:15 UTC · gpt-6.1-sol (high): 74.3% of minutes2026-10-27 15:15 UTC · claude-fable-5-1: 25.7% of minutes2026-10-27 15:30 UTC · gpt-6.1-sol (high): 87.3% of minutes2026-10-27 15:30 UTC · claude-fable-5-1: 12.7% of minutes2026-10-27 15:45 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 16:00 UTC · gpt-6.1-sol (high): 71.8% of minutes2026-10-27 16:00 UTC · claude-fable-5-1: 28.2% of minutes2026-10-27 16:15 UTC · gpt-6.1-sol (high): 82.0% of minutes2026-10-27 16:15 UTC · claude-fable-5-1: 18.0% of minutes2026-10-27 16:30 UTC · gpt-6.1-sol (high): 96.6% of minutes2026-10-27 16:30 UTC · claude-fable-5-1: 3.4% of minutes2026-10-27 16:45 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 17:00 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 17:15 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 17:30 UTC · gpt-6.1-sol (high): 100.0% of minutes2026-10-27 17:45 UTC · gpt-6.1-sol (high): 100.0% of minutesOct 2704:45Oct 2708:00Oct 2711:15Oct 2714:30Oct 2717:45
  • gpt-6.1-sol (high)1,164.2 min · 535,827,304 tokens
  • claude-fable-5-1219.9 min · 147,644,725 tokens
Recorded usage totals by series
SeriesMinutes% minutesTotal tokens% tokensOutput tokensFirst activeLast active
gpt-6.1-sol (high)1,164.284.1%535,827,30478.4%1,428,000 1 day ago 18 hours ago
claude-fable-5-1219.915.9%147,644,72521.6%515,267 1 day ago 19 hours ago

Agent-minutes add across parallel agents. Tokens include input, cached input, cache writes and output. Percentages are rounded to one decimal place in text and may sum to slightly more or less than 100%.

No status reports yet.

Completed Oct 27, 2026 · 17:02 UTC This handoff is read-only in ArchiveArea.