← Priority list

ExternalBrainWui: duplicate-document bug fixed and merged; Cloud Run deploy of the merged code is blocked

The reported duplicate baby-sleep.md in the left nav is fully resolved in production (storage duplicate repaired; the shadowed file was the user's real 4885-char document, the visible one a 37-char stub). Both code fixes are merged: the name-addressed listing now reports each name once, and the fat JAR now bundles its full dependency closure (it had been missing 28 transitive artifacts since PR #49, making every built JAR fail to start). Blocked on deploying that merged code: newly built Cloud Run revisions fail the startup TCP probe emitting zero application logs, while the same JAR binds its port in 15s locally. Traffic is safely on healthy revision 00005 and users are unaffected.

hf-2026-07-29-externalbrainwui-duplicate-document-bug-fixed-and-merged-cloud-run-deploy-of-the-merged-code-is-blocked

Edit handoff

Title Summary Markdown body

# ExternalBrainWui: duplicate-document bug fixed and merged; Cloud Run deploy of the merged code is blocked

**Written 2026-07-29 ~16:00 UTC.**

**RE-VERIFY: everything below is a write-time snapshot.** Before acting, re-check:
- Live listing: `curl -s https://my.externalbrain.wasmserver.com/api/documents`
- Serving revision vs newest built: the `latestReadyRevision` / `latestCreatedRevision` check in "Operational knowledge" below
- PRs: `gh pr view <n> --repo CodexCoder21Organization/ExternalBrainWui --json state,mergedAt`

## Mission

A user reported that `baby-sleep.md` appeared twice in the left nav of https://my.externalbrain.wasmserver.com/, with both entries showing as selected when either was clicked. Fix it, merge the fix, deploy it, and repair the duplicate in storage.

## Current state

**The user-facing problem is fully resolved in production.** No user-visible work remains.

- Live listing is `["baby-sleep.md","untitled-1.md","untitled-2.md","untitled.md"]` — one entry.
- `baby-sleep.md` serves the user's real 4885-character document.
- Both code fixes are merged to `main`.

**What is blocked:** getting the merged code onto the *running* Cloud Run revision. The site currently runs revision `externalbrain-wasmserver-00005-jb9` (built 2026-07-29 10:15, pre-dating both fixes). Users are unaffected because the deployed old code produces the correct output now that storage holds only one `baby-sleep.md`. The merged listing fix is defence-in-depth against recurrence, not something production currently depends on.

## What was found and done

### 1. The listing bug (fixed, merged)

`/api/documents` itself returned the name twice: `["baby-sleep.md","baby-sleep.md",...]`.

The storage service behind `url://markdown/` addresses files by UUID and can hold several files under one name. The WUI addresses documents purely by name — `/api/documents/:name`, `/doc/:name`, `/edit/:name` all resolve through `getFileByName`, which returns exactly one file (most recently modified). So a repeated name was a sidebar entry denoting no document of its own: both entries opened, highlighted and edited the same document, and the extra one could not be saved or deleted independently.

`MarkdownServiceClient.listDocuments()` now reports each name once, and `MarkdownStorage` documents that contract. The collision is logged, but only when the colliding *set changes* — the method runs on every sidebar render, so warning per request about an unchanging property of stored data would bury the warning and grow the log without bound.

Merged: https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/51 (merge commit `6878292`).

### 2. The storage duplicate (repaired)

Two files shared the name `baby-sleep.md`:

| UUID | lastModified | size | content |
|---|---|---|---|
| `4fbb6c65-ae2b-439e-891e-f24a85502a06` | 2026-02-17 (newer) | 37 chars | `# Baby Sleep Tips\n\nSome content here.` |
| `c0b643e1-1bcc-4de8-8cf8-85f12c54db3a` | 2026-02-02 (older) | 4885 chars | the user's full 10-step baby sleeping procedure + FAQ |

Because `getFileByName` returns the most recently modified, the site had been showing the **37-char placeholder**, and the user's real notes had been shadowed and unreachable since 2026-02-17. "Delete the newer duplicate" and "delete the older duplicate" were therefore opposite actions, and the naive reading would have destroyed the real document.

The stub was judged obsolete (it carries no information) and deleted; the real document remains under `baby-sleep.md`. **Both files were backed up before the delete** and remain recoverable:
- On the droplet: `/root/markdown-service-data/backup-20260729-dupfix/` (both UUID files)
- md5 of backups: `5423baeb05badc3baf07ca80bd0f0c4c` (stub), `ddda494aa098fe2231172709cbd7f55a` (real document)

Deleting the file directly on disk is safe for this service: `MarkdownServiceImpl.getAllFiles()` re-lists the directory on every call and `MarkdownFileImpl` re-reads the file on every property access, so there is no in-memory index to invalidate.

### 3. The fat JAR could not start at all (fixed, merged)

The first deploy attempt created Cloud Run revision `00006-w5p`, which never became READY, so traffic silently stayed on `00005` and the site kept serving old code. Running the built JAR locally gave the cause immediately:

java.lang.NoClassDefFoundError: io/netty/handler/codec/ByteToMessageDecoder
        at io.libp2p.multistream.Negotiator$GenericHandler.<init>(Negotiator.kt:80)

`buildFatJar` bundles `dependencies.map { it.jar }` — exactly the coordinates named in `resolveDependencies`, **with no transitive closure**. Netty 4.2 (added in PR #49) split `netty-codec` into `netty-codec-base`/`-compression`/`-protobuf`/`-marshalling`/`-classes-quic`/`-native-quic` plus `netty-tcnative-*`; only `netty-codec` was named. Fixing netty then surfaced the same gap in ASM, the SJVM libs and the algebraic-effects runtime — **28 artifacts** total.

`main` had therefore been un-deployable since PR #49, undetected because (a) the kompile *test* classpath resolves the full graph, so all 8 tests pass against a classpath the shipped artifact does not have, and (b) a Cloud Run revision that fails to start does not take the service down — traffic stays on the last healthy revision, so a deploy looks successful.

Merged: https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/52 (merge commit `428c202`).

## The blocker

A JAR built from merged `main` (`428c202`) **starts correctly locally**:

RESULT: PORT BOUND after 15s
    LISTING: ["baby-sleep.md","untitled-1.md","untitled-2.md","untitled.md"]

run as `java -Xmx200m -jar externalbrain-wui-final.jar --port 18083`. It opens the markdown connection in ~10s and serves real production data, so the artifact itself is good.

Deployed to Cloud Run, it fails. Revision `externalbrain-wasmserver-00007-t4h` (created 15:37:35 from this JAR via `cloudrun-deploy-cli`) failed with:

Default STARTUP TCP probe failed 1 time consecutively for container
    "externalbrain-wasmserver-1" on port 8080. The instance was not started.
    Connection failed with status DEADLINE_EXCEEDED.

**The revision emitted zero application log lines.** `Main.kt` prints `Starting ExternalBrain WUI Server...` as the first statement of `main()`, and that line never appeared — so the JVM appears not to have reached `main()` at all. This points at the container image or its entrypoint rather than at application code. The same was true of revision `00006`.

Container specs of the working and failing revisions are effectively identical (both rely on the image's own entrypoint, no `command`/`args`/`env` overrides, same service account and 300s timeout); the failing one even has more memory (512Mi vs 256Mi). So the difference lies inside the image.

Note the two revisions were built by **different** paths, which is the most promising lead:
- `00005` (working, serving) — built by ContainerNursery's `cloudrunjar` provider.
- `00007` (failing) — built by `cloudrun-deploy-cli` (Jib), because CN stopped producing new revisions (see below).

## Next steps

1. **Compare the two images' entrypoints directly.** Pull or inspect the manifests of the working `00005` image and the failing `00007` image from `us-west1-docker.pkg.dev/project-bf27c031-2b31-4eb6-939/cloudrunapps/externalbrain-wasmserver`. A Jib "exploded" layout (`java -cp /app/resources:/app/classes:/app/libs/* <MainClass>`) behaves differently from a plain `java -jar` of a fat JAR — in particular, a fat JAR's merged `META-INF/services` entries and its embedded static resources may not survive explosion intact. This is the single highest-value check and does not touch production.
2. **Get ContainerNursery to rebuild instead.** CN's image path is the one known to produce working revisions, and CN already has the *fixed* JAR on disk at `/root/ContainerNursery-uploads/jars/externalbrain-wui.jar` (sha256 `4542ff745f1d1b59ebe6c9e38920e7e4a13908cefeb5ae0ed00ebe82b66a4b20`, uploaded 15:24). Determine why `upload-jar` triggered a Cloud Run deploy on the first upload (13:38, revision `00006`) but not on the second, and re-trigger it. If CN then builds from the fixed JAR, the deploy likely succeeds.
3. **Verify the deploy properly.** `upload-jar` reports success regardless. Always confirm `latestReadyRevision == latestCreatedRevision` *and* that the serving image tag matches the new JAR's hash before calling a deploy done.
4. Optional follow-ups already noted in PR #52: migrate to `MavenPrebuilt2`/`resolveDependencies2` (resolves transitively — needs kompile-cli >= 0.0.63 in **both** `scripts/build.bash`, which currently pins **0.0.2**, and `scripts/test.bash`, which pins 0.0.65, plus the remote builder); and add a packaging smoke test that launches the built JAR and asserts it binds its port (needs a network-free design, since the real startup path dials `url://markdown/`).

## Relevant PRs / refs

- https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/51 — MERGED, listing fix + e2e test (`6878292`)
- https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/52 — MERGED, fat JAR dependency closure (`428c202`)
- Challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-29-1405-fat-jar-omits-transitive-dependencies-leaving.md
- Challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-29-1043-wrong-architecture-coursier-makes-build-watchman-exit-0.md

**All work is on `main`; nothing is unpushed.** Verified with `git status --porcelain --untracked-files=all`, `git stash list`, and `git log --branches --not --remotes` — all empty. No WIP branches exist. Backups of the two storage files live on the droplet path given above rather than in git, because they are user document content.

## Reusable / operational knowledge

**Reading the service's Cloud Run state.** There is no `gcloud` on the /code box, and `google.auth`'s transport is broken on the droplet (pyOpenSSL raises `AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'`). A working path is a manual JWT exchange with the service-account key at `/root/ContainerNursery/gcp-credentials.json` (project `project-bf27c031-2b31-4eb6-939`); the helper used is at `/root/gcp.py` on the droplet, exposing `token()` and `api(method, url, payload)`.

Deploy verification that actually catches a failed deploy:

```
created = latestCreatedRevision, ready = latestReadyRevision
if created != ready:  the newest revision never went healthy and traffic is on an OLDER one
also compare the serving image tag (jar-<first 12 of jar sha256>) against the JAR you built
```

**Do not probe startup with an API endpoint.** The port binds ~15s in, but the *first* `/api/documents` request blocks on SJVM sandbox spin-up far longer, so a short `curl -m 5` on an API path reads as "never started" and overstates startup time by minutes. Probe the TCP port. Cloud Run's own probe is a `tcpSocket` check, so only the bind matters.

**Kompile caches mask source edits.** After editing this module, purge **both** `~/.cache/coursier/v1/bldbinary/repository/externalbrain` and `~/.aibuildcaches/_code_workspace_ExternalBrainWui_server`, and bump the artifact version. Purging one alone silently serves the old jar; `build rule executed: ...buildMaven() (succeeded in 1ms)` is the tell.

**Local suite is 7/8 at best on the /code box.** `testTitlePasteBrowser` cannot run — Chromium fails with `libglib-2.0.so.0: cannot open shared object file`, and `npx playwright install-deps` needs root (no sudo, uid 1000). It passes in CI. Do not chase it as a code failure.

**CI may report failure for a run that passed.** PR #52's first run reported `Build timed out after 42 minutes` at 15:01:16, while the buildtest run itself finished at 15:02:26 with `8 passed, 0 failed` and `kompile-cli exit code: 0` — the service had restarted mid-run (`resumed after restart`), pushing it past the poller's budget. Read the run's own `build.log`/`test-events.jsonl` on the host before deciding a failure is real; a re-request then succeeded.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

ExternalBrainWui: duplicate-document bug fixed and merged; Cloud Run deploy of the merged code is blocked

Written 2026-07-29 ~16:00 UTC.

RE-VERIFY: everything below is a write-time snapshot. Before acting, re-check:

Live listing: curl -s https://my.externalbrain.wasmserver.com/api/documents
Serving revision vs newest built: the latestReadyRevision / latestCreatedRevision check in "Operational knowledge" below
PRs: gh pr view <n> --repo CodexCoder21Organization/ExternalBrainWui --json state,mergedAt

Mission

A user reported that baby-sleep.md appeared twice in the left nav of https://my.externalbrain.wasmserver.com/, with both entries showing as selected when either was clicked. Fix it, merge the fix, deploy it, and repair the duplicate in storage.

Current state

The user-facing problem is fully resolved in production. No user-visible work remains.

Live listing is ["baby-sleep.md","untitled-1.md","untitled-2.md","untitled.md"] — one entry.
baby-sleep.md serves the user's real 4885-character document.
Both code fixes are merged to main.

What is blocked: getting the merged code onto the running Cloud Run revision. The site currently runs revision externalbrain-wasmserver-00005-jb9 (built 2026-07-29 10:15, pre-dating both fixes). Users are unaffected because the deployed old code produces the correct output now that storage holds only one baby-sleep.md. The merged listing fix is defence-in-depth against recurrence, not something production currently depends on.

What was found and done

1. The listing bug (fixed, merged)

/api/documents itself returned the name twice: ["baby-sleep.md","baby-sleep.md",...].

The storage service behind url://markdown/ addresses files by UUID and can hold several files under one name. The WUI addresses documents purely by name — /api/documents/:name, /doc/:name, /edit/:name all resolve through getFileByName, which returns exactly one file (most recently modified). So a repeated name was a sidebar entry denoting no document of its own: both entries opened, highlighted and edited the same document, and the extra one could not be saved or deleted independently.

MarkdownServiceClient.listDocuments() now reports each name once, and MarkdownStorage documents that contract. The collision is logged, but only when the colliding set changes — the method runs on every sidebar render, so warning per request about an unchanging property of stored data would bury the warning and grow the log without bound.

Merged: https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/51 (merge commit 6878292).

2. The storage duplicate (repaired)

Two files shared the name baby-sleep.md:

| UUID | lastModified | size | content | |---|---|---|---| | 4fbb6c65-ae2b-439e-891e-f24a85502a06 | 2026-02-17 (newer) | 37 chars | # Baby Sleep Tips\n\nSome content here. | | c0b643e1-1bcc-4de8-8cf8-85f12c54db3a | 2026-02-02 (older) | 4885 chars | the user's full 10-step baby sleeping procedure + FAQ |

Because getFileByName returns the most recently modified, the site had been showing the 37-char placeholder, and the user's real notes had been shadowed and unreachable since 2026-02-17. "Delete the newer duplicate" and "delete the older duplicate" were therefore opposite actions, and the naive reading would have destroyed the real document.

The stub was judged obsolete (it carries no information) and deleted; the real document remains under baby-sleep.md. Both files were backed up before the delete and remain recoverable:

On the droplet: /root/markdown-service-data/backup-20260729-dupfix/ (both UUID files)
md5 of backups: 5423baeb05badc3baf07ca80bd0f0c4c (stub), ddda494aa098fe2231172709cbd7f55a (real document)

Deleting the file directly on disk is safe for this service: MarkdownServiceImpl.getAllFiles() re-lists the directory on every call and MarkdownFileImpl re-reads the file on every property access, so there is no in-memory index to invalidate.

3. The fat JAR could not start at all (fixed, merged)

The first deploy attempt created Cloud Run revision 00006-w5p, which never became READY, so traffic silently stayed on 00005 and the site kept serving old code. Running the built JAR locally gave the cause immediately:

java.lang.NoClassDefFoundError: io/netty/handler/codec/ByteToMessageDecoder
    at io.libp2p.multistream.Negotiator$GenericHandler.<init>(Negotiator.kt:80)

buildFatJar bundles dependencies.map { it.jar } — exactly the coordinates named in resolveDependencies, with no transitive closure. Netty 4.2 (added in PR #49) split netty-codec into netty-codec-base/-compression/-protobuf/-marshalling/-classes-quic/-native-quic plus netty-tcnative-*; only netty-codec was named. Fixing netty then surfaced the same gap in ASM, the SJVM libs and the algebraic-effects runtime — 28 artifacts total.

main had therefore been un-deployable since PR #49, undetected because (a) the kompile test classpath resolves the full graph, so all 8 tests pass against a classpath the shipped artifact does not have, and (b) a Cloud Run revision that fails to start does not take the service down — traffic stays on the last healthy revision, so a deploy looks successful.

Merged: https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/52 (merge commit 428c202).

The blocker

A JAR built from merged main (428c202) starts correctly locally:

RESULT: PORT BOUND after 15s
LISTING: ["baby-sleep.md","untitled-1.md","untitled-2.md","untitled.md"]

run as java -Xmx200m -jar externalbrain-wui-final.jar --port 18083. It opens the markdown connection in ~10s and serves real production data, so the artifact itself is good.

Deployed to Cloud Run, it fails. Revision externalbrain-wasmserver-00007-t4h (created 15:37:35 from this JAR via cloudrun-deploy-cli) failed with:

Default STARTUP TCP probe failed 1 time consecutively for container
"externalbrain-wasmserver-1" on port 8080. The instance was not started.
Connection failed with status DEADLINE_EXCEEDED.

The revision emitted zero application log lines. Main.kt prints Starting ExternalBrain WUI Server... as the first statement of main(), and that line never appeared — so the JVM appears not to have reached main() at all. This points at the container image or its entrypoint rather than at application code. The same was true of revision 00006.

Container specs of the working and failing revisions are effectively identical (both rely on the image's own entrypoint, no command/args/env overrides, same service account and 300s timeout); the failing one even has more memory (512Mi vs 256Mi). So the difference lies inside the image.

Note the two revisions were built by different paths, which is the most promising lead:

00005 (working, serving) — built by ContainerNursery's cloudrunjar provider.
00007 (failing) — built by cloudrun-deploy-cli (Jib), because CN stopped producing new revisions (see below).

Next steps

Compare the two images' entrypoints directly. Pull or inspect the manifests of the working 00005 image and the failing 00007 image from us-west1-docker.pkg.dev/project-bf27c031-2b31-4eb6-939/cloudrunapps/externalbrain-wasmserver. A Jib "exploded" layout (java -cp /app/resources:/app/classes:/app/libs/* <MainClass>) behaves differently from a plain java -jar of a fat JAR — in particular, a fat JAR's merged META-INF/services entries and its embedded static resources may not survive explosion intact. This is the single highest-value check and does not touch production.
Get ContainerNursery to rebuild instead. CN's image path is the one known to produce working revisions, and CN already has the fixed JAR on disk at /root/ContainerNursery-uploads/jars/externalbrain-wui.jar (sha256 4542ff745f1d1b59ebe6c9e38920e7e4a13908cefeb5ae0ed00ebe82b66a4b20, uploaded 15:24). Determine why upload-jar triggered a Cloud Run deploy on the first upload (13:38, revision 00006) but not on the second, and re-trigger it. If CN then builds from the fixed JAR, the deploy likely succeeds.
Verify the deploy properly. upload-jar reports success regardless. Always confirm latestReadyRevision == latestCreatedRevision and that the serving image tag matches the new JAR's hash before calling a deploy done.
Optional follow-ups already noted in PR #52: migrate to MavenPrebuilt2/resolveDependencies2 (resolves transitively — needs kompile-cli >= 0.0.63 in both scripts/build.bash, which currently pins 0.0.2, and scripts/test.bash, which pins 0.0.65, plus the remote builder); and add a packaging smoke test that launches the built JAR and asserts it binds its port (needs a network-free design, since the real startup path dials url://markdown/).

Relevant PRs / refs

https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/51 — MERGED, listing fix + e2e test (6878292)
https://github.com/CodexCoder21Organization/ExternalBrainWui/pull/52 — MERGED, fat JAR dependency closure (428c202)
Challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-29-1405-fat-jar-omits-transitive-dependencies-leaving.md
Challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-29-1043-wrong-architecture-coursier-makes-build-watchman-exit-0.md

All work is on main; nothing is unpushed. Verified with git status --porcelain --untracked-files=all, git stash list, and git log --branches --not --remotes — all empty. No WIP branches exist. Backups of the two storage files live on the droplet path given above rather than in git, because they are user document content.

Reusable / operational knowledge

Reading the service's Cloud Run state. There is no gcloud on the /code box, and google.auth's transport is broken on the droplet (pyOpenSSL raises AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'). A working path is a manual JWT exchange with the service-account key at /root/ContainerNursery/gcp-credentials.json (project project-bf27c031-2b31-4eb6-939); the helper used is at /root/gcp.py on the droplet, exposing token() and api(method, url, payload).

Deploy verification that actually catches a failed deploy:

created = latestCreatedRevision, ready = latestReadyRevision
if created != ready:  the newest revision never went healthy and traffic is on an OLDER one
also compare the serving image tag (jar-<first 12 of jar sha256>) against the JAR you built

Do not probe startup with an API endpoint. The port binds ~15s in, but the first /api/documents request blocks on SJVM sandbox spin-up far longer, so a short curl -m 5 on an API path reads as "never started" and overstates startup time by minutes. Probe the TCP port. Cloud Run's own probe is a tcpSocket check, so only the bind matters.

Kompile caches mask source edits. After editing this module, purge both ~/.cache/coursier/v1/bldbinary/repository/externalbrain and ~/.aibuildcaches/_code_workspace_ExternalBrainWui_server, and bump the artifact version. Purging one alone silently serves the old jar; build rule executed: ...buildMaven() (succeeded in 1ms) is the tell.

Local suite is 7/8 at best on the /code box. testTitlePasteBrowser cannot run — Chromium fails with libglib-2.0.so.0: cannot open shared object file, and npx playwright install-deps needs root (no sudo, uid 1000). It passes in CI. Do not chase it as a code failure.

CI may report failure for a run that passed. PR #52's first run reported Build timed out after 42 minutes at 15:01:16, while the buildtest run itself finished at 15:02:26 with 8 passed, 0 failed and kompile-cli exit code: 0 — the service had restarted mid-run (resumed after restart), pushing it past the poller's budget. Read the run's own build.log/test-events.jsonl on the host before deciding a failure is real; a re-request then succeeded.