← Handoffs

Repository · handoffs

Decide the remaining production validation for the merged SimpleFileSystem stream fixes

View on GitHub ↗

id: hf-2026-09-24-simplefilesystem-chunk-reads-from-url-simple-filesystem-vnext-run-at-1-2-mb-s-and-intermittently-fail-with-internal-error-stream-closed url: url://handoff/handoffs/hf-2026-09-24-simplefilesystem-chunk-reads-from-url-simple-filesystem-vnext-run-at-1-2-mb-s-and-intermittently-fail-with-internal-error-stream-closed title: Decide production validation for the merged SimpleFileSystem stream fixes summary: All named stream-fix PRs are merged, and ContainerNursery main retains the published resolver 0.0.1259 pins. Decide whether to authorize a separate production deployment and installer-cache validation lane or retire that remaining requirement; currently deployed code and post-deploy behavior were not verified by the 2026-10-06 lane. created: 2026-09-24T07:38:31.559Z completed: null blocked-reason: Operator decision needed: authorize a separate production deployment and installer-cache validation lane, or retire the remaining validation requirement. All named code PRs are merged. dependencies:

Decide the remaining production validation for the merged SimpleFileSystem stream fixes

RE-VERIFY (2026-10-06 05:41:02 UTC): This update is a write-time snapshot. Recheck the PR links below, main ancestry, published resolver POM, and actual production revision before acting. All named code changes are merged. This lane did not verify currently deployed code or rerun the production probe.

2026-10-06 live lane result

hfSFS final report — BLOCKED

Verified 2026-10-06 05:41:02 UTC. Original assignment: drive the SimpleFileSystem chunk-read handoff to LANDABLE, OUTDATED, or BLOCKED; the supervisor owns every merge decision.

BLOCKED: the one missing artifact is an operator decision on the remaining production validation: authorize a separate deployment-and-validation lane, or retire that requirement. This lane may neither deploy nor restart. No open code PR or Maven publication remains in this handoff. The canonical handoff explicitly asks for production deployment followed by the read-only 57,829,910-byte installer-cache probe, so merged code alone does not prove the whole handoff complete.

Live verified code and checks

PR State and merge time Merge commit Final-head checks
https://github.com/CodexCoder21Organization/ContainerNursery/pull/631 MERGED at 2026-09-27T14:19:19Z a82239af44ee8bba9c43485784418bd3c10592ba bld-build-release=SUCCESS, Build and test with Gradle=SUCCESS, kotlin.build (kompile-remote-build)=SUCCESS, kotlin.build (remote)=SUCCESS
https://github.com/CodexCoder21Organization/ContainerNursery/pull/632 MERGED at 2026-09-27T11:20:26Z b3ab5e64a4e2fe5f7d57ac59cfb75e2ad8ab00f4 bld-build-release=SUCCESS, Build and test with Gradle=SUCCESS, kotlin.build (kompile-remote-build)=NEUTRAL, kotlin.build (remote)=SUCCESS
https://github.com/CodexCoder21Organization/ContainerNursery/pull/633 MERGED at 2026-09-27T08:07:56Z 388544a160b6f9912a7006046955958822c1ed3c bld-build-release=SUCCESS, Build and test with Gradle=SUCCESS, kotlin.build (kompile-remote-build)=IN_PROGRESS, kotlin.build (remote)=SUCCESS
https://github.com/CodexCoder21Organization/UrlResolver/pull/1106 MERGED at 2026-09-22T12:58:25Z 78b23e73cc9a6c94e5b3c2c5accb3788eaf1125d bld-build=SUCCESS, kotlin.build (kompile-remote-build)=SUCCESS, kotlin.build (remote)=SUCCESS
https://github.com/CodexCoder21Organization/W3WalletInstallerWebsite/pull/15 MERGED at 2026-09-24T09:05:58Z f2d2501b44d45ce9cc109dfc61dc764d6dff8a28 kotlin.build (kompile-remote-build)=FAILURE, kotlin.build (remote)=SUCCESS

Fresh ContainerNursery clone: origin/main=dab37c7890cb8787c3ea767219ed0ad8bccb1ea5. git merge-base --is-ancestor returned success for all three named ContainerNursery merge commits against this main. Both build files retain resolver 0.0.1259, protocol 0.0.528, and libp2p 1.3.0-codexcoder21-snapshot-27. Resolver POM https://kotlin.directory/foundation/url/resolver/0.0.1259/resolver-0.0.1259.pom returned HTTP 200.

Named remote branches checked with git ls-remote: https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/sweep4-S1-stream-bridge-pulls at eaae50af93c789a4a008d15265db200dfd788ab7; https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/sweep4-S1-stream-close-ownership at f67381b8f22a451a0828c32055ecfc5314794f61; https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/sweep4-cn4-start-hang-kill at 76e9f76583e8926249395a94243f9160ef4cbbd3; original probe artifacts https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/sweep3-w1h-sfs-read-throughput-artifacts at ad6f85784d7ec9781e3482e0f39cf93c4f7b80fb.

The merged stream changes request 1 MiB backend pulls and adopt the upstream EOF-release fix. Their final checks succeeded; earlier red-check descriptions in the historical handoff are stale. Production throughput, backend closure, and recurrence of Stream closed were not measured in this lane, and currently deployed code was not verified.

Scope and verification

The supplied short ID does not exist. The canonical ID is hf-2026-09-24-simplefilesystem-chunk-reads-from-url-simple-filesystem-vnext-run-at-1-2-mb-s-and-intermittently-fail-with-internal-error-stream-closed; its full body was read and claimed as fable-code-2026-10-06. handoff-cli reports returned [], so there were no prior uploaded supervisor reports to read.

Repository README.md and AGENTS.md, the full engineering PHILOSOPHY.md, and the handoff triage README were read. TESTING.md was retrieved, but no new tests or code review were needed for this BLOCKED verdict. The fail-first/five-local-runs/review gates belong to the LANDABLE path; no open implementation PR remains, so no duplicate test batches or new CI runs were started. No source changes, new commits, pushes, deployments, service restarts, publications, queue changes, merges, or completion action occurred. The fresh checkout is clean, has no stashes or local-only commits, and only one worktree.

Plan audit: read/claim/discover/live PR verification DONE; merged-main/pins/publication verification DONE; open-PR LANDABLE gate unnecessary because all named PRs are MERGED; production validation BLOCKED by operator decision. No brief item was silently deferred.

Delegated work: none. Landing sweep: no open PR gates remain to dispatch. Next step: the supervisor obtains the operator decision about the production-validation requirement, then assigns any authorized work to a lane that permits it.

Lessons: check canonical IDs before treating a missing lookup as missing work; recheck final checks because historical failures can be superseded; separate merged-code evidence from deployed-behavior evidence.

Handoff: https://www.handoff.wasmserver.com/handoffs/hf-2026-09-24-simplefilesystem-chunk-reads-from-url-simple-filesystem-vnext-run-at-1-2-mb-s-and-intermittently-fail-with-internal-error-stream-closed

STATUS: BLOCKED Operator decision required for the remaining production deployment and installer-cache validation.

Preserved prior investigation and operational context

The following is the earlier 2026-09-27 snapshot, retained as history. Its historical red-check descriptions have been superseded by the live final-head checks above. Do not treat its deployment or mechanism claims as freshly verified by this lane.

Deploy the merged stream fixes and validate installer-cache reads after operator authorization

RE-VERIFY (2026-09-27 14:56 UTC): This body is a write-time snapshot; query GitHub and kotlin.directory again before acting. Pull requests were rechecked with gh pr view <number> --repo CodexCoder21Organization/<repo> --json state,mergedAt,mergeCommit; named branches were checked with git ls-remote. ContainerNursery PRs 631, 632, and 633 have merged. ContainerNursery PR 638 also merged today, but its UDP listener startup change is unrelated to this SimpleFileSystem stream work. No production change was made by this lane.

Remaining work first

  1. After the operator explicitly authorizes a production deployment of the merged ContainerNursery stream changes, rerun the read-only 57,829,910-byte W3 Wallet installer-cache probe. Record per-chunk latency, byte equality, backend connection closure at EOF, and whether Stream closed recurs. No production deployment or restart has been performed by this lane.
  2. If a Stream closed failure recurs in that probe, trace that occurrence to its first close event rather than assuming the EOF ownership fix explains every historical error.

RESULT

  • OBSERVED: ContainerNursery PR 632 merged at 2026-09-27T11:20:26Z as b3ab5e64a4e2fe5f7d57ac59cfb75e2ad8ab00f4; PR 631 merged at 2026-09-27T14:19:19Z as a82239af44ee8bba9c43485784418bd3c10592ba; PR 633 merged at 2026-09-27T08:07:56Z as 388544a160b6f9912a7006046955958822c1ed3c. UrlResolver PR 1106 merged at 2026-09-22T12:58:25Z as 78b23e73cc9a6c94e5b3c2c5accb3788eaf1125d; W3WalletInstallerWebsite PR 15 merged at 2026-09-24T09:05:58Z as f2d2501b44d45ce9cc109dfc61dc764d6dff8a28.
  • OBSERVED: ContainerNursery PR 638 merged at 2026-09-27T14:37:53Z as d93022fa4f56aa50ad35f0cf752db32206b2b99a. Its title and body concern binding UDP listeners before startup returns; it is unrelated to the URL stream read fixes and is not named by this handoff.
  • OBSERVED: ContainerNursery main at d93022fa4f56aa50ad35f0cf752db32206b2b99a contains resolver 0.0.1259, protocol 0.0.528, and libp2p 1.3.0-codexcoder21-snapshot-27 in both build.kts and build.gradle.kts. The published resolver 0.0.1259 POM returned HTTP 200.
  • OBSERVED: The named remote branches still exist: bridge-pulls eaae50af93c789a4a008d15265db200dfd788ab7, EOF ownership f67381b8f22a451a0828c32055ecfc5314794f61, and PlanRepository probe artifacts ad6f85784d7ec9781e3482e0f39cf93c4f7b80fb.
  • INFER: The named code changes and dependency pin integration are complete. The only remaining step is the production deploy and post-deploy read validation, which require explicit operator authorization. No publication remains; resolver 0.0.1259 is already available.

Mission

A production read of the 57,829,910-byte W3 Wallet installer cache through url://simple-filesystem-vnext/ took 44-52 seconds and sometimes failed with v2_readChunkStream INTERNAL_ERROR Stream closed. The original 2026-09-24 probe and logs are at https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/sweep3-w1h-sfs-read-throughput-artifacts/handoffs/artifacts/sfs-read-throughput-2026-09-24. The website cache-lifecycle change in https://github.com/CodexCoder21Organization/W3WalletInstallerWebsite/pull/15 is merged and separate from these transport symptoms.

Verified mechanisms and limits

  1. Slow reads: ContainerNursery main's StreamBridgeInputStream asks the backend for 262144 bytes per pull while the outside reader asks for 1048576. A 3 MiB file therefore needs twelve data pulls instead of three. Production measurements cited by PR 631 observed exactly 262144-byte outside responses and thirteen backend calls including EOF. https://github.com/CodexCoder21Organization/ContainerNursery/pull/631 changes the backend pull size to 1048576. Its fail-first real TCP URL-facade test checks three full 1 MiB pulls, a 123-byte tail, EOF, and content equality. This is the measured throughput mechanism; faster production throughput still needs validation after merge and authorized deploy.
  2. One observed stream-closed failure: In UrlResolver before merged commit 78b23e73c, StreamAwareServiceHandler removed a response stream from its registry at EOF but did not close the owned InputStream. The URL facade bridge owns a backend TCP connection, and a later __stream_close cannot find a removed registration. The source now calls releasePullStream on EOF. https://github.com/CodexCoder21Organization/UrlResolver/pull/1106 is merged; resolver 0.0.1259 is published at https://kotlin.directory/foundation/url/resolver/0.0.1259/resolver-0.0.1259.pom (HTTP 200 checked 2026-09-27). https://github.com/CodexCoder21Organization/ContainerNursery/pull/632 pins this version and has a fail-first 3 MiB real TCP test that observes the backend connection closing at EOF. Its author recorded 34 readChunkStream and 34 readClose calls but zero backend __stream_close calls across two production probes, followed by a backend heap error and child restart during the failed probe. This evidence supports the leak mechanism for that observed failure. Earlier log entries with the same error text have not all been traced individually.
  3. Test coverage caveat for PR 632: its new EOF test invokes the URL facade with empty metadata. The facade uses a shared backend push bridge only for requests with pushSupported=true on a live outside connection. In shared mode, StreamBridgeInputStream.close sends backend __stream_close but intentionally leaves the shared TCP connection open. Thus the current test proves unshared connection closure, not stream release on a shared bridge. The SimpleFileSystem SJVM client calls ServiceBridge.rpc for v2_readChunkStream without pushSupported metadata, and current UrlResolver and UrlProtocol source do not add it, so the installer workload likely uses the tested unshared mode. Shared mode remains a neighboring edge case for final review.
  4. The old handoff suggested a ConcurrentModificationException in a website concurrent-cold-request test might be the same bug. That was only a candidate, and the two URL facade mechanisms above do not require that explanation. Do not modify SimpleFileSystem read-session code on that guess.

PRs and remote branches at the live recheck

Purpose PR or branch Head at verification Live state
Reduce short backend pulls https://github.com/CodexCoder21Organization/ContainerNursery/pull/631 ; https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/sweep4-S1-stream-bridge-pulls eaae50af93c789a4a008d15265db200dfd788ab7 MERGED 2026-09-27T14:19:19Z; merge commit a82239af44ee8bba9c43485784418bd3c10592ba
Close backend connection on EOF using resolver 0.0.1259 https://github.com/CodexCoder21Organization/ContainerNursery/pull/632 ; https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/sweep4-S1-stream-close-ownership f67381b8f22a451a0828c32055ecfc5314794f61 MERGED 2026-09-27T11:20:26Z; merge commit b3ab5e64a4e2fe5f7d57ac59cfb75e2ad8ab00f4
Fix start-timeout test blocking Gradle CI on both PRs https://github.com/CodexCoder21Organization/ContainerNursery/pull/633 76e9f76583e8926249395a94243f9160ef4cbbd3 MERGED 2026-09-27T08:07:56Z; merge commit 388544a160b6f9912a7006046955958822c1ed3c
Original read-only probe and logs https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/sweep3-w1h-sfs-read-throughput-artifacts ad6f85784d7ec9781e3482e0f39cf93c4f7b80fb Branch, no PR
EOF library fix https://github.com/CodexCoder21Organization/UrlResolver/pull/1106 78b23e73cc9a6c94e5b3c2c5accb3788eaf1125d MERGED 2026-09-22T12:58:25Z; resolver 0.0.1259 POM returns HTTP 200

Both ContainerNursery fixes and the CI gate PR are merged. The current main build files contain resolver 0.0.1259, protocol 0.0.528, and libp2p snapshot-27. This lane made no production change. The two stream branches remain on the remote at the heads listed above.

Historical CI diagnosis (snapshot written 2026-09-27 04:57 UTC)

PR 631's Gradle job https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/36286398835/job/108527887320 ran 931 tests with one failure: startHangIsKilled asserts the kill count before tracked cleanup finishes. PR 632's Gradle job https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/36287643252/job/108531331363 ran 931 tests with two failures: the same start test and RollingFileLogCaptureTest.closeCannotOvertakeACompletedTimestampRecoveryResult, which expected a waiting writer but saw RUNNABLE. PR 633 addresses the first test and passed Gradle CI. The kotlin.build remote checks for PRs 631 and 632 are red; their run pages report a persistent connection to url://buildtest/ closing while getBuildLogChunk was pending. Do not treat those checks as proof of a filesystem code failure or simply rerun them. The prior handoff blocker wording about allocation failures is incomplete and was cleared.

Historical next steps (snapshot written 2026-09-27 04:57 UTC)

  1. Coordinate the active ContainerNursery PR owners and the orchestrator. Bring PR 633 and any independently diagnosed CI gate fixes to green, then rebase PRs 631 and 632 on latest main and confirm their fail-first regressions and neighbors still pass. Keep their changes in their existing PRs; do not duplicate them.
  2. Obtain green required checks, including kotlin.build (remote), for both URL facade PRs. The orchestrator alone decides merge or enqueue. Reconcile the two dependency/build pin changes when integrating them.
  3. Production deployment requires a separate explicit user request. After an authorized deploy, repeat the read-only 57.8 MB probe and verify per-chunk latency, byte equality, backend connection closure, and whether any Stream closed failures recur. If historical errors persist, trace each to its first close event rather than assuming the EOF leak was their only cause.

Next steps

After explicit operator authorization to deploy, run the read-only 57,829,910-byte W3 Wallet installer-cache probe against the deployed ContainerNursery URL facade. Record per-chunk latency, verify byte equality and backend connection closure at EOF, and record whether Stream closed recurs. If it recurs, trace that occurrence to its first close event before considering another code change.

No artifact publication or production action was performed by this lane. Production deployment and the post-deploy verification remain pending operator authorization.