Finish the unresolved reconnect investigation and review the prepared PRs

RE-VERIFY: This is a write-time snapshot from 2026-10-21 09:30 UTC. Check each PR's state, head SHA and named check before acting; no merge or deployment is authorized. All requested p104 actions are complete. The broader natural reconnect investigation remains open. This body replaces stale statements that the assigned branch and port were not yet submitted.

Design decisions in force

Original request: “We need to investigate why https://my.externalbrain.wasmserver.com/ is failing to load and fix the underlying problem.”

Owner decision, 2026-10-21, verbatim: “Stay up, return 503 naming the backend, and keep retrying (exponential backoff) the open with the existing per-attempt limit (no new timeouts, exponenial backoff) until it succeeds, so the site heals itself once the backend is reachable.” This supersedes terminal startup failure after ordinary errors or three timeouts.

The p104 brief explicitly directs proceeding on the prior exact-b7ef230 proof: full suite 124/124 first-attempt, both lazy reconnect cases 20/20 each, both owner reproducers 3/3 each. It separates the unexplained fixture handshake into an upstream observation issue. Later passing runs do not explain earlier failures. No merge, enqueue, artifact publication, deployment, new product timeout/sleep, raised limit or reduced iteration count is authorized. Leave the source port PR open. Source commits use the required co-author footer. Local heavy JVM work is serial; check cgroup memory below 8.5 GB and disk above 8 GB. Remote buildtest was unavailable during this execution.

Work completed in p104

All six existing-test edits passed a per-file audit; none needed reverting. Two terminal-failure expectations now require the owner-decided retries. Two clock tests distinguish a legal search-backend retry from a Markdown deadline or request deadline while preserving the original budgets. The shutdown-recovery fixture counts initial reconciliation separately from recovery, with exact counts for both. The capped retry test adds exact latest-error GET/PUT refusals and unchanged storage checks. No assertions were weakened and no bounds or iteration counts changed. The complete table links raw failure-first and mutation evidence: https://github.com/CodexCoder21Organization/ExternalBrainWui/blob/wip/r104-evidence/handoff-artifacts/r104/r104-six-test-audit.md .

The assigned PR was checked OPEN, rebased with --rebase-merges on main a0638f7784f5f951f706e3af59e7c62aecd8582b, and normally fast-forward pushed to exact b7ef230fead71cf4db3b3cf99ceb4724e0dc3ec8. No force push. Its why-first description preserves both user quotes and honest verification limits. Its required bld-build passed: https://github.com/CodexCoder21Organization/ExternalBrainWui/actions/runs/39574069597/job/119135113055 . Build-watchman 0.0.21 plain --pr ended GREEN/exit 0. The complete CI log reports 124/124 with no retry warning. A separate old zero-run non-required kotlin-build-ci-test suite remains informational; it was not rerequested.

The stacked follow-up contains exactly two servlet files and six tests over that exact parent. It ports ordinary read/list errors to full text/plain 500 responses, preserves DependencyStartingException handling, and adds both requested cold-start guards. Copying all old source files would have undone current startup handling, so the prepared narrow semantic port was used. Targeted tests passed 6/6 first-attempt. Separate test and source review found no blocking issue in this eight-file port. Four local screenshot states were inspected and current uploaded bytes verified, with all 22 production source hashes matching the screenshot build. Screenshots are embedded in the PR and remain at https://github.com/CodexCoder21Organization/PrScreenshots/tree/main/ExternalBrainWui/r92-port-green-b7ef230 .

Follow-up CI: https://github.com/CodexCoder21Organization/ExternalBrainWui/actions/runs/39575689902 on exact 2a931d4f9aaacc88c53b36143f0ca92a1fae5675 . It was dispatched once because the workflow's automatic pull-request filter only targets main, while this PR has a stacked base. The named bld-build completed successfully. Its complete CI log reports 130/130 passed with no retry warning; this is separate from the qualified local result. No manual rerun was made.

Observations that remain unresolved

The upstream issue https://github.com/CodexCoder21Organization/UrlProtocol/issues/762 is OPEN. It contains the exact TotalTimeoutException and full stack, mediated resolver 600 / protocol 337 / libp2p snapshot 9 graph, fixture path, load observations and precise mixed counts. The failure path is MarkdownServiceClient.borrowing -> getAllFiles -> LazyReconnectingMethodDispatcher -> UrlProtocol2 -> libp2p, before startApplication. The two old 20-case groups had 0/20 and 1/20 passes, with nine outer-bound exits each; only three surviving logs contain the exact handshake exception, so do not label every nonpass a handshake failure. Load average 140–240 was an observed condition, not an established cause. No deterministic reproducer or fix exists. Closest main fixture passed 10/10 normal and 10/10 prior flags; that does not rule out the intermittent failure. Issue and handoff link all raw evidence.

The earlier normal-flags post-start success-reconnect case failed once at its unchanged reconnecting.get(5, SECONDS), after releasing the real bytecode gate; the companion recovery case passed 3/3. Later 20/20 groups and observation-only diagnostics passed. Three deliberate public-adapter gate controls each failed at the same five-second assertion while gate-off controls passed, proving that assertion alone does not identify a broken connection. They do not explain or waive the natural failure. No failed-window stack or owning-code fix has been established for that natural event. Common 113 JAR hash prefixes matched earlier runtime, so a version-graph or same-version republish explanation was unsupported. Preserve all unchanged lazy assertions/bounds.

Fresh p104 local validation must not be described as first-attempt clean:

• Primary exact b7ef230: final 124/124, 121 first-attempt and three automatic second-attempt passes. testNewDocumentBrowser first failed waiting for .cm-editor; testPermanentMarkdownServiceFailureIsReported failed its initial shell GET with SocketTimeoutException before any changed assertion; unchanged testReviewShutdownCancelsPendingAttemptWithoutAdvancingClock recorded ConcurrentModificationException in ArrayList.sort -> ManualClock.schedule line 70 -> openStartingDependency before the factory entered. Raw first stacks are preserved. Browser/shell mechanisms remain unknown.
• The WUI uses clock 0.0.3. Upstream already has the thread-safety fix https://github.com/CodexCoder21Organization/community.kotlin.clocks.simple/commit/beb423631b4a99636a6d9778647c5ffabb31180a (published 0.0.5). A migration must also account for removed shutdown/hierarchicalClock APIs; no guessed consumer lock or broad migration was made here.
• Follow-up exact 2a931d4: final 130/130, 129 first-attempt and one automatic second-attempt pass in unchanged testWikiPageReferencesAreFollowableInEditorBrowser. The first attempt displayed wiki/Markdown syntax that should have been hidden; all six new tests passed first attempt. Its mechanism is unestablished. No manual rerun, source change or weakened assertion was used to obtain the final verdict.

Native XML stores attempt-history as root-level siblings of test entries, and contains numeric ESC references. Preserve raw XML; sanitize only a separate parser copy. Complete XML, full console logs, extracted first-failure stacks and parser/proof JSON are at https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r104-evidence/handoff-artifacts/r104 .

Durable branches and PRs

All refs must be rechecked on pickup. The evidence branch is moving; use its current ref rather than a stale checkpoint SHA.

Legacy fallback https://github.com/CodexCoder21Organization/PlanRepository/pull/11111 remains OPEN and unmerged, head 93a908fd8d54c2b5745b52a3e65aa303c9d10fd7. Use this central handoff as the source of truth; do not merge the fallback.

Deployed or published but not merged: none by this effort. All product changes are source PRs only. No production restart or routing change occurred. Local test artifacts are not publications.

Resume instructions and known tooling details

Read the complete p104 brief, r104-findings.md, six-test audit and raw histories in the r104 evidence tree. Read inherited p92.md, r92-findings.md, r92-final.md, specialist findings and contract controls at https://github.com/CodexCoder21Organization/ExternalBrainWui/tree/wip/r92-evidence/handoff-artifacts/r92 . The previous full handoff snapshot is retained as r104-handoff-start.json. Testing and engineering documents actually read are retained as r104-TESTING.md and r104-PHILOSOPHY.md; repository README/AGENTS were read in each fresh clone.

Recheck with gh pr view <full-PR-URL> --json state,headRefOid,mergedAt,baseRefName,statusCheckRollup and git ls-remote <repo-URL> refs/heads/<branch>. GitHub PR status reads must stay at least 60 seconds apart. Never infer green from monitor exit alone: verify actual completed/success state on the exact SHA.

A fresh clone belongs under workspace/. Read README/AGENTS, fetch/rebase before building, freeze source while a runner reads it, and use server/scripts/test.bash --local --test <scenario> --log <absolute.xml> --result-file <absolute.json>. The two relevant lazy scenarios are testLazyIdleMarkdownReconnectCompletesHttpOperation and testLazyIdleReconnectFailureStartsBackgroundRecovery. Their repeat helpers and exact runtime manifests are inherited in jobs/r92-lazy-repeats and the specialist findings. Full suite uses --test .; do not rerun unchanged CI to diagnose a failure.

Browser launches used process-local LD_LIBRARY_PATH for existing extracted Chromium libraries under /home/helena/.cache/chromium-deps and out/r77-chromium-deps. Shared dependencies were not modified. Recreate prerequisites through the screenshot skill if these paths no longer exist. Runtime classpaths are ephemeral; reconstruct from source/manifests rather than expecting local JARs after cleanup. Literal test dependency headers differ from mediated runtime; retain the actual graph in every analysis.

On a stacked base with no required rules, build-watchman falls back to ALL checks and --check adds rather than restricts. This would await an old informational remote check, so the follow-up's named workflow is watched separately. Redirected gh run watch exited 0 with no output while REST still showed in_progress. The bounded terminal workaround is script -q -e -c 'timeout 4800 gh run watch 39575689902 --repo CodexCoder21Organization/ExternalBrainWui --interval 60 --exit-status' <log>. No run was rerequested. The challenge record is retained in r104-monitor-challenge.md; report-challenge CLI was not invoked because it automatically enqueues/merges, contrary to this brief.

Next engineering work is to capture and deterministically force the natural failing reconnect condition, locate the owning code, then verify a fix there without changing existing bounds. The handshake issue, post-start five-second event, shell/browser observations and old clock migration remain distinct until evidence connects them. The prepared PRs can be reviewed independently under the explicit p104 proceed decision. No new owner question is pending.

Cleanup and scoped completion

Cleanup completed at 09:29:59 UTC. The two owned clones were clean, had no stashes or extra worktrees, and exactly matched their pushed source heads before deletion; the proof is r104-precleanup.json. Final deletion/process verification is recorded in r104-cleanup.json and r104-closeout.md. Both owned clone trees, both task-specific caches and four reconstructed screenshot PNGs were removed. No owned build, test or watcher process remains. Shared caches, unrelated clones, other lanes' monitors and existing browser libraries must remain untouched. The central handoff remains open for the unresolved engineering observations and unmerged reviews; it must not be archived as though those were fixed.