Finish, merge and deploy url://webshot/ and TinySite home-page thumbnails

Written 2026-10-17 ~02:40 UTC. RE-VERIFY: everything below is a write-time snapshot. Before acting, re-check each PR with gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,headRefOid,statusCheckRollup, each branch with git ls-remote https://github.com/CodexCoder21Organization/<repo>.git, and each version with curl -s -o /dev/null -w '%{http_code}' https://kotlin.directory/<group-path>/<artifact>/<version>/ (200 = published, 404 = not).

Mission

The owner asked on 2026-10-08: "The tinysite homepage has a list of recent sites, each site should include a preview/thumbnail generated by playwright or something. I think we have a screenshot service, right? Does the screenshot service support web screenshots? Should it support web screenshots or should that be a separate service?"

Decisions the owner made:

• Separate service. url://webshot/, not an extension of url://screenshottest/ (which only compares golden screenshots of an uploaded WUI bundle and renders 127.0.0.1).
• One fresh DigitalOcean droplet per capture.
• "Build and deploy." The owner's authorization covers deploying url://webshot/ and redeploying TinySite with the thumbnail pass enabled, for this effort only.
• Catalog. List TinySite in ServiceAtlas. Done in https://github.com/CodexCoder21Organization/ServiceAtlas/pull/54. Webshot still needs its own entry, added at deploy time.

Later the owner said "Drive this to completion." The session was paused on 2026-10-17 at the owner's request ("stop the delegates, create a handoff").

The binding design and contract are in https://github.com/CodexCoder21Organization/PlanRepository/blob/main/workstreams/Webshot.md, landed by https://github.com/CodexCoder21Organization/PlanRepository/pull/10763 and https://github.com/CodexCoder21Organization/PlanRepository/pull/10781. Read it first.

Architecture (what each repo is)

Lockstep rule. The runner's request parser rejects unknown fields. The runner and WebshotEmbedded must therefore be built on compatible webshot-api versions; WebshotEmbedded checks this via RUNNER_WEBSHOT_API_VERSIONS and REQUEST_COMPATIBLE_WEBSHOT_API_VERSIONS. The target set is webshot-api 0.0.9 + screenshottest-embedded 0.0.19 + runner 0.0.4 + embedded 0.0.6 + server 0.0.3.

Merge order and gate

Merge in dependency order: api (done) → ScreenshotTestEmbedded → runner → embedded → server → TinySite.

Each PR needs all of the following before merging:

1. A green required check on its final head.
2. A clean adversarial review of that exact head by an independent reviewer. A review is void once the head moves.
3. The merger's own final review.

CI change made in this effort. The shared kotlin.build (remote) check sat in_progress for hours with no build attached (recorded in https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-16-2157-kotlin-build-remote-checks-stay-in-progress-with-no-build.md). Following the owner's 2026-10-16 "GitHub runner required, remote CI optional" rollout, every repo in this effort now requires the GitHub Actions job bld-build, which runs scripts/build.bash and scripts/test.bash with --local-and-remote and kompile-cli 0.0.124. kotlin.build (remote) is informational in these repos.

• WebshotApi, WebshotRunner, WebshotEmbedded, WebshotServerService: the existing rulesets (ids 24716086, 24716185, 24717264, 25275694) had kotlin.build (remote) swapped for bld-build, strict.
• ScreenshotTestEmbedded: it had no ruleset. New ruleset 25286167 copies SimpleFileSystemServiceServer's: required bld-build strict, merge queue (squash), PR required, no deletion, no force-push.
• Restoring the old rulesets: the pre-edit ruleset JSON was saved only in the session scratchpad. To restore, put kotlin.build (remote) back as the required check (strict) in the four rulesets, and delete 25286167.
• Runner and embedded PRs: they do not yet carry the bld-build workflow (next steps).
• Enqueueing: use the GraphQL enqueuePullRequest mutation. Auto-merge is disabled in these repos.

Relevant PRs / refs

Merged

Open PRs and branches (every unmerged piece of work)

Not pushed, on purpose: reviewer probe programs (kotlinc and javap probes, hostile-PNG generators, deep-tree generators) and the CI lane's pre-edit ruleset JSON. They lived only in the session scratchpad and are reproducible from the descriptions here.

Published artifacts (all byte-verified against local builds when published)

Deployed

Nothing from this effort is deployed (verified 2026-10-17 ~02:45Z with container-nursery-cli:0.0.20 ... routes). There is no webshot route. The tinysite.wasmserver.com:443 route (danger_jarfile, /root/ContainerNursery/apps/tinysite-wui.jar) has no TINYSITE_THUMBNAIL_PASS variable and depends only on url://filedrop/. No lane deployed anything.

What was found and done

• WebshotApi (merged).

  • Reached 0.0.9 over nine review rounds. They tightened url rules (no userinfo, ports 1–65535, punycode, no zone id), strict JSON, Unicode Cc/Cf/Zl/Zp refusal, monotonic timestamps, a retention hard cap of 500, and an error cap of 2000 with ... [truncated].
  • Added RecoveryState, with FAILED getting its own refusal sentence.
  • 0.0.8 restored the 0.0.6 binary signatures of WorkerPoolStatus; 0.0.9 hides those overloads from Kotlin source with @Deprecated(level = HIDDEN).
  • The README documents how errors look to remote callers (below).

• The transport drops exception types — upstream, not fixed here. Through UrlResolver, every remote refusal reaches the caller as foundation.url.protocol.sandbox.SandboxException with the message Sandboxed code threw an exception: java.lang.InternalError: … UrlResolutionException: RPC request '<method>' failed: INVALID_PARAMS - <service message> cause: null. Callers must match on the documented message text. WebshotServerService tests/testRemoteClientErrorTypes.kts pins this. Recorded at https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-17-0210-url-rpc-transport-drops-exception-types-every-remote.md, with the four code locations where the type is lost; the fix belongs in UrlProtocol/UrlResolver.

• ScreenshotTestEmbedded.

  • PR 7 adds namePrefix-scoped droplets.
  • PR 10 adds WorkerRecoveryOperation, so a host can cancel worker recovery and wait for it. Its docs were corrected three times until they matched the code exactly, including that only an interrupt observed by a covered call counts as cancellation.
  • PR 12 fixes a recovery record that could never be resolved. It is an upstream fix: WebshotEmbedded cannot reach that record without depending on ScreenshotTestEmbedded internals.

• WebshotRunner 0.0.3 → 0.0.4.

  • 0.0.3 contains the browser in a per-capture cgroup (pids.max 2048, cgroup.kill, verified empty), uses descriptor-relative link-safe deletion, and decodes and scales PNGs row by row (peak about 8 MiB).
  • The 0.0.3 security re-review found containment sound but raised three findings, all fixed in 0.0.4 and listed in the branch table. 0.0.4 is not yet re-reviewed.
  • The reviewer also recommended memory.max. On a real host this showed memory.swap.max 0 is needed, otherwise the browser swaps instead of being killed.

• WebshotEmbedded 0.0.6 candidate (head 402ad88).

  • Pins api 0.0.9 and screenshottest-embedded 0.0.19. DEFAULT_RUNNER_COORDINATES is still runner 0.0.3; move it to 0.0.4 and add its row.
  • Hosted recovery runs inside WorkerRecoveryOperation. Failed teardown is retried in-process after 60 s, doubling up to 30 min.
  • shutdown() waits at most 60 s and returns captures still releasing a worker, which keep cleanupPending for the next start.
  • A close() versus recovery hang was found by test and fixed.
  • CaptureListener events, plus test-quality rewrites: no sleep-polling, no reading data-directory files.
  • New refusal types in package webshot.embedded: WebshotRefusalException (extends IllegalStateException), with subtypes CaptureQueueFullException, CaptureNotCompletedException, CaptureNotCancellableException, CaptureNotDeletableException, WebshotServiceClosedException and WebshotRecoveryPendingException.
  • Faults use WebshotServiceFault (a RuntimeException that is not an IllegalStateException; no paths in the message). Messages are byte-identical to the api 0.0.9 docs.
  • testUnknownCreationRequestRecovered passes on screenshottest-embedded 0.0.19. Nobody has confirmed it fails on 0.0.18.

• WebshotServerService review (at 2e4e01b). Previous deploy blockers (crash-loop, image-slice type, bytecode encoding, URL-mode codes, unbounded listing) are fixed. Items from this review:

  1. Map only WebshotRefusalException and IllegalArgumentException to INVALID_PARAMS; everything else becomes INTERNAL_ERROR with a logged stack trace. This waits on embedded 0.0.6. CancellationException pass-through is already done.
  2. Health becomes DEGRADED or UNHEALTHY. Done.
  3. README section on remote errors. Done.
  4. The backoff test is tied to the injected Clock. Done.
  5. Close latency with an in-flight deletion: re-check on embedded 0.0.6.
  6. README uses --local. Done.

• TinySite round 4 review (at 0794f0b). Round-3 fixes hold. Three problems remain:

  • (1) Medium-high. isWebshotRefusal (SiteThumbnails.kt ~837) does not recognise the remote SandboxException shape. A refused capture is then never counted as an attempt, so there is no backoff, a new intent and a createCapture call are written every pass, and in-flight slots fill up. The reviewer reproduced 4 refusals in 4 passes. All existing tests call WebshotEmbedded in-process, which hid this.
  • (2) README.md:75 misstates when a refused write holds a site.
  • (3) requireCapturableBase checks syntax only, while Webshot also refuses hosts that resolve to non-public addresses; the README overclaims. The test's "good" example 203.0.113.9 is a documentation-range address.

  Commits 796e982 and d035320 on wip/tinysite-thumbnails address all three but are unverified.

Next steps

1. ScreenshotTestEmbedded PR 7. Look over the main-merge commit 5fcfe4c, confirm bld-build is green on the head, then enqueue.
2. ScreenshotTestEmbedded PR 12. Get an adversarial review of the fix and of the changed testRestartUnknownWorkerRequest, then merge. It may need a rebase after PR 7 lands.
3. WebshotRunner PR 1.
   • Security re-review of 6b8d261..d5ba12d: width cap, iterative deletion and depth cap, cleanup Throwable handling, truncated-PNG messages and last IDAT CRC, memory.max, swap.max 0, oom.group.
   • Add the bld-build workflow. Copy it from https://github.com/CodexCoder21Organization/WebshotApi/commit/b33f82a with webshot.runner.buildMaven, and add TinySiteWui's Node and Playwright install step, because runner tests launch real Chromium. Bump kompile-cli to 0.0.124.
   • Update the PR description for 0.0.4, get bld-build green, then merge.
4. WebshotEmbedded PR 1.
   • Set the default runner to 0.0.4 and add its RUNNER_WEBSHOT_API_VERSIONS row (api 0.0.9).
   • Check that testUnknownCreationRequestRecovered fails on screenshottest-embedded 0.0.18.
   • Run the full suite with --local.
   • Add the bld-build workflow (webshot.embedded.buildMaven; no Playwright needed).
   • Publish 0.0.6 after a 404 check, byte-verified.
   • Update the PR description, get an adversarial review of 0.0.5..head, then merge.
5. WebshotServerService PR 1.
   • Move to embedded 0.0.6 and screenshottest-embedded 0.0.19.
   • Finish item 1 (match WebshotRefusalException) with tests, and re-check item 5.
   • Run the full suite, publish 0.0.3 byte-verified, and get bld-build green.
   • Get a re-review of 2e4e01b..head, then merge.
6. TinySiteWui PR 2.
   • Verify the two round-5 commits on wip/tinysite-thumbnails: run the full suite with --local, then move them onto thumbnails.
   • Move the pins to api 0.0.9 and embedded 0.0.6 together.
   • Get a round-5 re-review that specifically checks classification under the real remote error shape, then merge.
   • Separately, testBrowserLeavesCommitAloneWhenResponseBodyIsLost is a known older flake (a cold Playwright install lock). Do not raise its timeout.
7. Deploy (authorized by the owner's "build and deploy").
   • Add the WebshotServerService fat jar to ContainerNursery as route webshot (url://webshot/) using the CN CLI ≥ 0.0.20 add-route.
   • Route settings: keepWarmSeconds -1, 128MB, dependency url://digitalocean-droplets/, no JAVA_TOOL_OPTIONS. add-route has no flag for startupTimeoutSeconds (the README wants 120), so it must be set in the route config.
   • Verify health with details shows RECOVERED, then do one real capture of a public page.
   • Redeploy TinySite from merged main with TINYSITE_THUMBNAIL_PASS=enabled and the url://webshot/ dependency. Run git pull before building.
   • Check https://tinysite.wasmserver.com shows thumbnails, and that its first screen has no errors.
   • Add Webshot to ServiceAtlas, noting that access control is deferred (any peer can call every method).
8. Housekeeping.
   • buildtest run d2c55190 (label WebshotServerService), created 2026-10-17 02:03Z, was PENDING with no droplet. It is most likely orphaned by a stopped --remote client. Do not archive or cancel buildtest runs by hand; report it if it is still stuck.
   • Delete the throwaway branches listed above.
   • Upstream follow-ups:
     • FiledropApi should expose its per-drop limit. TinySite currently hard-codes 1 GiB, guarded by testPerDropLimitMatchesFiledrop.
     • The UrlProtocol/UrlResolver typed-error fix from the challenge record.

Reusable knowledge

• Clone setup. Private clones need gh auth setup-git first, and commits need the git identity codexcoder21 / codexcoder21@gmail.com.
• Use --local, not --remote. Pass --local to scripts/test.bash and scripts/build.bash when iterating. A killed --remote client can leave buildtest runs holding droplets.
• Byte-verify every publish. Check the version returns 404 before publishing, then download it back and compare SHA-256. Never reuse a version number.
• Changing only KDoc can still change the jar. Kotlin line-number tables move when lines shift inside a file with code. Keep line counts constant (reflow in place) and confirm with a hash comparison.
• Test remote error handling through the real transport, or with the exact SandboxException message shape. In-process tests hide the type loss.
• A sweep found nothing deployed and no unpushed commits in any lane checkout. Verified with git status, git log --branches --not --remotes and git ls-remote per repo at write time.