Handoff: LambdaServer MVP — restructure COMPLETE; 2 of ~5 feature PRs merged (W3Wallet auth + exec-limits); metering/observability/e2e remain

Written: 2026-07-06. Updated: 2026-07-12 (W3Wallet-auth + execution-limits feature PRs driven to green/merge — see the 2026-07-12 update block below). Effort began: 2026-06-11 (the date the implementation PRs were opened; the vision doc it implements is older).

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time (2026-07-06). PRs merge, new PRs open, artifacts get published, CI re-runs, and the LambdaServer repo has other in-flight work. A fresh query always wins over this document. Before doing anything, re-verify:

• PRs: gh pr view <num> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Open work in the server repo: gh pr list --repo CodexCoder21Organization/LambdaServer --state open
• Published artifacts: curl -s https://kotlin.directory/api/list-artifacts | grep -i lambda
• #62 merge-queue entry: gh api graphql -f query='query { repository(owner:"CodexCoder21Organization", name:"LambdaServer"){ pullRequest(number:62){ mergeQueueEntry{ state position } } } }'
• buildtest fleet health: curl -s "https://buildtest.kotlin.build/api/runs?limit=25" (count statuses; provisioning failing fleet-wide ⇒ DO-outage, back off).

---

⚡ 2026-07-12 UPDATE — 2 MVP feature PRs done (W3Wallet auth + exec-limits); metering/observability/e2e remain

The MVP feature work of §5 step 3 is now partially done. Verified live at 2026-07-12 ~16:43 UTC (RE-VERIFY — the #62 merge state below is the volatile one):

MERGED this session:

• LambdaServer #58 — url-stack bump — MERGED 2026-07-12T00:28Z. Landed on the newest mutually-consistent triple: foundation.url:resolver:0.0.624 / protocol:0.0.337 / community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-9. Do not naively bump resolver to "latest": resolver 0.0.625–0.0.630 POMs pin protocol 0.0.337 alongside jvm-libp2p snapshot-10, but protocol 0.0.337 was built against snapshot-9 — an inconsistent triple (challenge 2026-07-11-2341). 0.0.624 is the newest consistent one. lambda-server bumped 0.0.31→0.0.32 here.
• LambdaEmbedded #3 — execution isolation & limits — MERGED 2026-07-12T03:51Z (LambdaEmbedded main now dd98ebdb). Per-invocation wall-clock timeout (dedicated thread + daemon watchdog interrupt → InvocationTimeoutException), per-function + per-executor concurrency caps (semaphores, reject-at-submit → ConcurrencyLimitExceededException, async rejections recorded FAILED), allocated-bytes budget (com.sun.management.ThreadMXBean watchdog → AllocationBudgetExceededException), CPU-time + allocated-bytes metering exposed via getInvocationMetrics(id), OrphanedInvocationThreadWarning algebraic effect, honest sandboxing docs. ExecutionLimits.UNLIMITED preserves legacy behavior; enforced identically in FunctionManager and FunctionExecutor via shared src/ExecutionLimits.kt. 95/95 tests, twice-reviewed (an independent review caught + fixed an unbounded FunctionExecutor metrics-map leak → now a bounded LRU). These metrics (cpuTimeMs, allocatedBytes) are the intended billing inputs for the metering PR.

OPEN — the one still to land:

• LambdaServer #62 — opt-in W3Wallet authn/authz gating registration + invocation RPCs — OPEN, branch CI FULLY GREEN (bld-all-tests, bld-build-fat-jar, kotlin.build (remote) all SUCCESS), mergeable CLEAN, 65/65 tests, twice-reviewed. Built on com.w3wallet.gate:w3wallet-gate:0.0.1 (newest published; README's 0.0.2 is unpublished). Wraps mutating/invoking RPCs in Gate.protect; registration ops (createFunction/updateFunction/deleteFunction/syncWorkspace/buildWorkspace/runTests) require a registerFunction capability, invocation (invokeFunction + path invokeAsync) requires invokeFunction; server-minted single-use expiring challenge nonce as channel binding (getW3WalletChallenge); enforcement default-OFF (production unchanged when merged); lambda-server bumped to 0.0.33. Presentation wire: "w3wallet": {"tokenChain","presentation","challenge"}. Adversarial review caught + fixed a HIGH defect: the CapabilityGranted/DeniedAuditEffect handlers were installed via Effective{} on the parked main thread while Gate.protect tosses effects on RPC worker threads (AlgebraicEffectsV1 is ThreadLocal) → audit never fired; fixed by wrapping the per-request path in a shared withW3WalletAuditHandlers { } in both URL-bind and P2P serving modes.
  • ⚠️ #62's ONLY obstacle is buildtest infra, NOT the code. It was enqueued and dequeued twice without merging — each time the merge queue sharded the suite and one shard PASSED 65/65 (build=True) while a sibling shard hit CRITICAL EMERGENCY: Whole-build timeout — exceeded 62 minutes (build=False, testsTotal=0, never finished compiling) on a slow/overloaded droplet during/after a fleet-wide DigitalOcean provisioning outage (challenge 2026-07-12-0301). Diagnose a dequeue via the buildtest runs API: curl -s "https://buildtest.kotlin.build/api/runs?limit=50" → find runIds whose notes contain LambdaServer + /62, read buildSucceeded/testsPassed/testsTotal/errorMessage. A build=False, tests 0/0, whole-build-timeout shard is infra; a testsFailed>0 shard is real. It was re-enqueued at ~16:43 UTC as this handoff was written (position 1, QUEUED) — RE-VERIFY its state first.
  • IMMEDIATE NEXT ACTION: confirm #62's queue state; if dequeued-again-on-infra, re-enqueue when the fleet is healthy (provisioning succeeding, not a wall of FAILED/INFRASTRUCTURE/PROVISIONING). Do not re-request into a saturated/ratelimit-remaining=0 fleet — back off and let the burst drain. The code is proven; never weaken/disable a test or "fix" the diff in response to a build-timeout. Standing merge approval applies ("ensure these changes get merged").

PENDING publish: the new lambda-embedded (isolation/limits, from #3 / main dd98ebdb) is NOT yet published — kotlin.directory still shows lambda-embedded:0.0.2 (pre-isolation). Publish the next version from merged main before anything downstream consumes the limits (see §5 step 1 commands; probe the target version pom URL first — parallel sessions collide). lambda-server published is still 0.0.30 (main is 0.0.32; #62 will take it to 0.0.33) — publish at close-out.

Remaining MVP feature PRs (dependency order, per workstreams/LambdaServer.md — delegate to codex gpt-5.6-sol, available again after a quota reset, or Opus subagents):

1. Metered invocation + rate limiting/quota — caller presents a W3Wallet spend capability; meter using #3's cpuTimeMs/allocatedBytes; enforce via W3WalletGate's SpendEnforcerPort over W3WalletAssetPools (RegisterSpendResult = Approved | Tentative | Declined), settling against the placeholder ledger. Depends on #62's gate being merged.
2. Baseline observability + cold-start budget — centralized invocation logs/history with durations+failures into ProductionHealth; measure+document time-to-first-invocation under ContainerNursery lazy-start.
3. Three-source end-to-end registration tests — register from Maven coordinate / uploaded JAR / built-from-git, invoke sync+async, assert invocation records (per TESTING.md).

Then close-out: publish lambda-server, add a DocumentationRepository project page + ALL_PROJECTS entry (workstream graduation), and delete this handoff.

Also filed this session (context, not blockers): challenge 2026-07-12-0107 — community.kotlin.clocks.simple Clock.schedule takes an absolute deadline (not a relative delay) and SystemClock's scheduler uses non-daemon threads without removeOnCancelPolicy, which hangs the JVM; #3 works around it with a dedicated daemon watchdog scheduler. Issue LambdaEmbedded #4 — FunctionExecutor.createFunction uploads the JAR but never links it into the definition's jarFileIds (pre-existing; executor tests work around it with addJarToFunction).

Nothing unsaved: all session work is merged or pushed to its branch (#62 branch w3wallet-authn-authz, head bb0ce6b). Local checkouts were in the session scratchpad (/tmp/claude-1000/-code/1ecf3dbd-.../scratchpad/workspace) and are transient — re-clone as needed. Agent-memory lambdaserver-mvp-pr-state.md tracks this state too.

---

⚡ 2026-07-11 UPDATE — restructure is COMPLETE; only the MVP feature PRs remain

Everything in §2/§5 about publishing lambda-embedded and doing PR C is done (verified at update time):

• lambda-embedded:0.0.1 was published 2026-07-11 but found ABI-incompatible with LambdaServer main — lambda-api 0.0.16 (via LambdaServer #59) added a 9th requiredCapabilities parameter to createFunction while LambdaEmbedded still built against 0.0.15. See challenge 2026-07-11-0854.
• Fixed upstream: LambdaEmbedded #2 (MERGED 2026-07-11 11:27Z) syncs to lambda-api 0.0.16 + lambda-store-api 0.0.3, persists requiredCapabilities (legacy metadata defaults to empty list), preserves parameterNames/requiredCapabilities across rename/update, 77/77 tests. LambdaEmbedded:lambda-embedded:0.0.2 is published — use 0.0.2, never 0.0.1.
• PR C is DONE: LambdaServer #60 (MERGED 2026-07-11 12:50Z) slims LambdaServer to the ServiceServer shell over lambda-embedded:0.0.2 — removed src/FunctionManager.kt, src/FunctionExecutor.kt, the dead local src/FunctionService.kt, and 76 migrated tests (each verified to have a LambdaEmbedded equivalent); 49 server-layer tests remain green; lambda-server bumped to 0.0.31 (artifact not yet published from the new main).
• LambdaServer #58 (url-stack bump) was still OPEN at update time and now needs a rebase over the slim-down.

Remaining work = §5 step 3 only: the MVP feature PRs in dependency order (W3Wallet authn/authz → execution isolation & limits → metering/quota → observability + cold-start budget → three-source registration e2e), per workstreams/LambdaServer.md.

---

1. Mission summary

Original user request, verbatim, in order:

1. "Let's start implementing all of this." — Implement the LambdaServer MVP described in the plan at PlanRepository/workstreams/LambdaServer.md. The MVP's first item is restructuring the LambdaServer family onto the standard layered architecture (Api / Embedded / ServiceServer); the later items are W3Wallet auth gating, execution isolation/limits, metering + quota (caller-pays), and observability.
2. "ensure these changes get merged." — Drive the resulting PRs to merged.

Where it stands: Phase 1 — the Api / Embedded split — is complete, merged, and published. The remaining work is the ServiceServer slim-down (PR C) and then the MVP feature PRs.

---

2. Bottom line (status at handoff, 2026-07-06)

• Phase 1 done & merged (all 2026-06-16): lambda-api stripped to interfaces-only, all client implementations consolidated into one repo, and a new LambdaEmbedded repo holding the Embedded layer. Two artifacts published: lambda-api:0.0.15, lambda-api-client:0.0.3. LambdaEmbedded validated 75/75 tests locally.
• The one loose end from phase 1: lambda-embedded:0.0.1 is NOT yet published (the repo is merged, the artifact was never pushed). This blocks PR C, which needs to depend on it.
• lambda-server is still 0.0.13 and still the full monolith — PR C (the slim-down) has not been started.
• Change since the last working session: pre-existing LambdaServer #51 ("Fix __bytecode_request OOM by sending JARs as binary attachments") — which was open and flagged as a reconcile risk — merged on 2026-06-20. PR C now builds on top of it rather than colliding with it.
• New in-flight work in the server repo: LambdaServer #58 ("Bump url stack: protocol 0.0.325, resolver 0.0.585, jvm-libp2p snapshot-9") is open — PR C will need to coordinate/rebase around it.
• Nothing is uncommitted or unpushed. All feature branches merged; local workspace clones are transient and were reset between sessions. There is no lost work.

---

3. What was found and done (the chain)

1. Explored the family (verified). It is 10 repos, not the 6 first assumed: LambdaServerApi (Api, but was polluted with client impls), LambdaServerApiClientImplementation, LambdaEmbedded (new, created this effort), LambdaServer (ServiceServer + legacy workspace surface), LambdaStoreApi/LambdaStore (+ its client impl) for storage, and three production *RefreshCallback consumers. Two facts shaped the design: the server already had a proto-Api interface FunctionService with two implementations (FunctionManager local, FunctionExecutor LambdaStore-backed), so the split follows existing seams; and there is zero W3Wallet / quota / limit / timeout code anywhere in the family today (execution is in-process reflection with a child-first classloader).

2. LambdaServerApi #30 — MERGED 2026-06-16 → lambda-api:0.0.15 PUBLISHED. Stripped lambda-api to interfaces + data types only (package lambdaapi): FunctionDefinition, FunctionDefinitionData, InvocationResult, LambdaService, plus the server-side contract moved in from the LambdaServer repo — FunctionService, FunctionInfo, InvocationInfo (the interface the Embedded implements and the ServiceServer hosts). Dependencies cut to the Kotlin stdlib only. During merge this branch had diverged — LambdaServerApi #31 had merged an assertNoBuildToolingInDependencies regression guard into the same build.kts; I rebased and git's 3-way merge cleanly combined both (my stdlib-only strip trivially satisfies that guard). Verified: LambdaApi.buildMaven compiles; interfaces-only module carries no tests (per TESTING.md).

3. LambdaServerApiClientImplementation #3 — MERGED 2026-06-16 → lambda-api-client:0.0.3 PUBLISHED. Made this repo the single home of every LambdaService client implementation (package lambdaapiclient): HttpLambdaService (synced up to the current contract — the copy here had been frozen at the 0.0.4 contract and no longer compiled), UrlLambdaService, WebSocketRpcClient, RemoteLambdaWorkspace, LambdaClientMain, poll helpers, and the 34 end-to-end client tests + test-function-src moved out of the Api repo. Verified: 34/34 pass locally (each boots a real Jetty LambdaServer).

4. LambdaEmbedded #1 — new repo, MERGED 2026-06-16. Extracted FunctionManager (local okio-filesystem storage) + FunctionExecutor (remote LambdaStore-backed, injected FunctionStore/JarStore/InvocationStore) from the server repo, package lambdaserver → lambdaembedded, now implementing lambdaapi.FunctionService. This is the embeddable-library tier of the deployment spectrum in the plan. build.kts depends on lambda-api:0.0.15 + lambda-store-api:0.0.2. 75 pure-embedded-layer end-to-end tests moved with it. Verified: 75/75 pass against the published lambda-api:0.0.15. (The user merged this PR themselves after I left it at the merge gate.)

5. Merge mechanics + a CI outage (verified). All three repos use merge queues with allow_auto_merge=false, so merges went via the GraphQL enqueuePullRequest mutation, not gh pr merge. The first CI attempts failed on shared buildtest infrastructure, not the code: the droplet-service was refusing to create build droplets ("WebCron service is not available" — the known startup-latch bug) amid a host cold-start avalanche. Resolved by waiting for the avalanche to drain and re-triggering; both then passed. (This is buildtest infra, not a LambdaServer defect — see the many buildtest CI-outage memories.)

---

4. Relevant PRs / refs (write-time state, 2026-07-06)

| Ref | Purpose | State | |-----|---------|-------| | LambdaServerApi #30 | Strip lambda-api to interfaces-only; add FunctionService contract | ✅ MERGED 2026-06-16 | | LambdaServerApiClientImplementation #3 | Single home of all LambdaService client impls | ✅ MERGED 2026-06-16 | | LambdaEmbedded #1 | New repo: extract Embedded layer (75 tests) | ✅ MERGED 2026-06-16 | | LambdaServer #51 | __bytecode_request OOM → binary attachments | ✅ MERGED 2026-06-20 (was open; PR C now builds on it) | | LambdaServer #58 | Bump url stack (protocol 0.0.325 / resolver 0.0.585 / libp2p snapshot-9) | ⏳ OPEN — coordinate PR C around it | | LambdaApi:lambda-api:0.0.15 | interfaces-only artifact | ✅ published | | LambdaApiClient:lambda-api-client:0.0.3 | client impls artifact | ✅ published | | LambdaEmbedded:lambda-embedded:0.0.1 | Embedded-layer artifact | ❌ NOT published (next step) | | LambdaServer:lambda-server:0.0.13 | ServiceServer (still full monolith) | unchanged |

Plan this implements: workstreams/LambdaServer.md. Superseded interim note: a memory-store draft (lambdaserver-mvp-implementation-workstream.md) was written before this canonical handoff existed — this document supersedes it.

---

5. Next steps (ordered — re-verify current state first)

0. Re-verify all of §4 with the commands in the RE-VERIFY banner. In particular confirm lambda-embedded is still unpublished and check whether #58 merged (it changes PR C's base).

1. Publish lambda-embedded:0.0.1 (unblocks PR C). The publisher tool cannot compile the modern build.kts (see §6), so build locally then publish --artifact-dir:

   gh repo clone CodexCoder21Organization/LambdaEmbedded /code/workspace/LambdaEmbedded
   cd /code/workspace/LambdaEmbedded
   # its scripts/build.bash is the modern kompile-cli launcher (good)
   scripts/build.bash LambdaEmbedded.buildMaven /tmp/pub-lambda-embedded
   coursier launch community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.4 \
     -r https://kotlin.directory -- --artifact-dir /tmp/pub-lambda-embedded \
     --api-url https://api.kotlin.directory/upload --force
   

2. PR C — slim the LambdaServer repo to the ServiceServer shell. Base it on current main (which now includes the #51 binary-attachment bytecode fix; rebase around open #58 if still in flight). Remove src/FunctionManager.kt + src/FunctionExecutor.kt (now in lambda-embedded) and the 75 migrated tests; add a dependency on lambda-embedded:0.0.1; keep LambdaServerRpcHandler, LambdaManager (workspace sync/build/test surface), BytecodeGenerator, LambdaServer.kt (main), src-client/, and the 46 remaining tests (RPC / workspace / bytecode). Run the full server suite locally before opening the PR; bump lambda-server to 0.0.14.

3. Then the MVP features, each its own PR, in dependency order (per workstreams/LambdaServer.md): W3Wallet authn/authz on register + invoke → execution isolation & limits (mem/CPU/wall/concurrency — none exists today) → metered invocation + rate limiting/quota (caller-pays via W3Wallet) → baseline observability into ProductionHealth + cold-start budget → three-source end-to-end registration tests (Maven / uploaded JAR / built-from-git).

Decision for the user: none blocking — #51 already merged, removing the only reconcile question. Confirm the lambda-server version bump number if a convention beyond "next patch" applies.

---

6. Reusable / operational knowledge (so you don't rediscover it)

• The Maven publisher tool (community.kotlin.maven.artifact.publishing:0.0.4) bundles an OLD kompile runtime that CANNOT compile a modern build.kts using resolveDependencies2 / MavenPrebuilt2 / { it.jar } — it fails with unresolved reference: it. Workaround: build the artifact locally with the repo's scripts/build.bash <rule> <outdir>, then publish with --artifact-dir <outdir> --api-url https://api.kotlin.directory/upload --force (NOT --workspace/--buildrule).
• LambdaServerApiClientImplementation's scripts/build.bash on main is a broken old stub (a bare kompile call → "command not found"); only test.bash was modernized in #3. For local builds there, copy the modern kompile-cli launcher build.bash from LambdaServer (or LambdaEmbedded). Worth a tiny follow-up PR to fix in-repo.
• All the Lambda repos + PlanRepository use merge queues with allow_auto_merge=false. gh pr merge is rejected ("Auto merge is not allowed"). Enqueue via gh api graphql enqueuePullRequest(input:{pullRequestId}); a PR must be MERGEABLE/CLEAN with green checks first; if it's behind_by>0 under strict required-status-checks, rebase + force-push + let CI re-run before enqueuing.
• CI is the external kotlin.build (remote) check-run (not a legacy commit status; poll statusCheckRollup / check-runs). Its failures are frequently shared buildtest-infra flakes (WebCron/droplet startup latch, host cold-start avalanche) that red innocent PRs — wait for the host to drain and re-trigger (empty-commit push or check-run rerequest); do not "fix" your code in response.
• Embedded-extraction test-split heuristic used: a LambdaServer test is "pure embedded" (→ moved to LambdaEmbedded) iff it constructs FunctionManager/FunctionExecutor AND references none of {LambdaServerRpcHandler, LambdaManager, startServer, BytecodeGenerator}. That split was 75 of 121; the other 46 stay with the ServiceServer.
• Target layering: lambda-api (interfaces) ← lambda-embedded (FunctionService impls) ← lambda-server (thin ServiceServer shell over url://lambdaserver/). Separately, lambda-api-client is the remote client path and LambdaStoreApi/LambdaStore is the storage service FunctionExecutor consumes.