The request was to add HardwareControlFabric ServiceServer end-to-end tests under `tests/` as `testE2e<Scenario>.kts`, following the NetLab (in-process network-simulator) brief patterns, covering three scenarios:
Moves it out of every priority list and into ArchiveArea.
Written: 2026-07-07 UTC (original). Substantially rewritten 2026-07-12 UTC — the original blocker (kotlin.directory / coursier infrastructure outage) is resolved and the work has largely landed; the remaining state is entirely different from the 2026-07-07 snapshot. Dated 2026-07-07 to keep the thread grouped.
RE-VERIFY before acting — everything here is a write-time (2026-07-12 ~04:40 UTC) snapshot:
gh pr view 8 --repo CodexCoder21Organization/HardwareControlFabricServiceServer --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOidgit ls-remote https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer workstream/netlab-e2e-full-assertions (write-time head 67b246bb29cb55777c75233efc367b2145ca292b)curl -I --max-time 20 https://kotlin.directory/community/kotlin/hardwarecontrolfabric/daemon/0.3.0/daemon-0.3.0.pom (write-time: HTTP 200, recovered)The request was to add HardwareControlFabric ServiceServer end-to-end tests under tests/ as testE2e<Scenario>.kts, following the NetLab (in-process network-simulator) brief patterns, covering three scenarios:
listPeers, getHealthDashboard, listFunctions, and invokeFunction (all via the public HardwareControlFabricService API, through the NetLab mesh).The 2026-07-07 effort was blocked on infrastructure (kotlin.directory Maven repo timing out inside coursier dependency resolution), not on any scenario assertion.
Branch workstream/netlab-e2e-full-assertions was created off an earlier scaffolding commit, the three test scripts + NetlabE2eSupport.kt support harness were drafted, but validation was blocked by the kotlin.directory/coursier outage. No PR existed at that time.
Between 2026-07-07 and 2026-07-11 the original scaffolding evolved (through a peer-scoped daemon dial-back registration model) and merged to main as:
But the merged tests asserted LESS than the mission required: flat-LAN and partition scenarios ran only ONE daemon; only peer-registration (listPeers) was asserted; no getHealthDashboard, no listFunctions, no invokeFunction through the mesh; echo-function registration in the daemon support app had been deliberately disabled (a stage literally named after-skip-register-echo-function). Function invocation was proven only in a localhost (non-NetLab) test.
A fresh branch workstream/netlab-e2e-full-assertions off latest origin/main closes those gaps. All of the following is VERIFIED (local 7/7 pass + remote CI green), except where marked.
Extended the three scenarios to the mission spec: flat-LAN now launches two daemons and asserts, for BOTH, exact-set listPeers, online getHealthDashboard, exact listFunctions == [netlab-e2e-echo], and full invokeFunction result (echo:<value>); partition now runs two daemons (partitioned one goes offline while sibling stays online, then heals to both-online) with exact-set health assertions; NAT retains registration coverage (see the documented scope limit in item 4).
Root-caused a real product defect that the merged tests had silently sidestepped — VERIFIED with thread dumps + decompiled bytecode. The public invokeFunction call hung forever in the NetLab mesh: HardwareControlFabricApi 0.3.16's UrlFabricTransport eagerly opened one persistent RPC connection per daemon client, which implicitly enabled UrlResolver 0.0.622's relay-backed path for ordinary health/function commands. On a flat LAN that writer blocks in RelayBackedRpcController.output.flush -> UrlProtocol2.callServiceViaPeerRelay -> RelayService.sendViaRelay before the daemon ever receives the request, and timeout cleanup then blocks in UrlProtocol2.close. Direct daemon invocation (bypassing the ServiceServer entirely) reproduced the same hang, proving the defect is below the ServiceServer.
registrationFabricClientFactory with allowRelayFallback=true, while ordinary health/function dial-back uses allowRelayFallback=false); UrlResolver bumped to 0.0.628. These are the same pins as the (separate, author-not-us) PR #6; our branch carries the equivalent pin + factory-split changes directly.Fixed a test-teardown thread-pool leak (in OUR test support code, not the product) — VERIFIED. The NetLab harness shuts an app down by interrupting its parked main thread; the daemon support app then ran daemon.stop() with the interrupt flag still set, and Jetty's QueuedThreadPool.joinThreads() aborts early on a pending interrupt without reaping its non-daemon workers, so the test JVM (BootstrapRunner) never exited. Additional cause: the success path deleted tempRoot (which hosts the echo-support jar on every role's classpath) while daemon teardown still ran asynchronously, so Jetty's Server.doStop lazily loaded a lambda class mid-shutdown, hit the deleted jar (BootstrapMethodError: NoSuchFileException), and aborted before the pool was reaped. Fix: clear the interrupt flag before daemon.stop(); and the tests now assertEquals(0, daemonApp.awaitExit(120_000)) BEFORE scope exit/deletion so teardown completes while the classpath is intact.
NAT scenario health/invoke is intentionally NOT asserted — documented, not weakened (VERIFIED unsatisfiable against current product design). Ordinary dial-back is direct-only by design (Embedded 0.0.20 default factory and FabricApiClient.overUrl both default allowRelayFallback=false, confirmed in bytecode; src-main uses these defaults), and a full-cone-NAT'd daemon has no dialable direct address (it announces a private IP; netsim full-cone maps to an ephemeral external port nothing announces). Relay reachability works only for registration. The NAT test keeps its registration/listPeers assertion plus an in-code NOTE pointing to the fix owners. Closing this needs a product decision (relay-capable dial-back, observed-address announcement, or NAT traversal) — see Next Steps.
Declared a measured 512 MB memory quota on the three topology tests via @BillingQuotaExecutionLimit(quantity=512, resource_name="memory_megabytes") — VERIFIED needed. With the default 128 MB heap the multi-JVM-role topology GC-thrashed (Metaspace ~99%, a RelayService listener thread died under pressure, 30 s bytecode-fetch timeouts). This is a genuinely larger concurrent working set (multiple real libp2p nodes + Jetty servers + SJVM sandboxes co-hosted in one JVM), not a leak — heap sawtooths back down as apps close. No timeouts were inflated and no assertions weakened.
Two independent reviews addressed before CI reliance — a test-comprehensiveness review (per TESTING.md) and an adversarial code review (per PHILOSOPHY.md). Accepted fixes applied in commit 67b246b: exact-set health-assertion equality (was subset containment — a ghost peer could have satisfied "both online"); atomic temp-sibling+rename write for the echo-support jar (was a TOCTOU truncation race); full stack traces in waitUntil timeout messages. The adversarial review found no blocking issues (it decompiled the harness/library jars to verify the teardown-await, interrupt-clearing, and factory-split semantics and confirmed no production src//src-main/ behavior changed).
Infrastructure remediation during the CI drive (VERIFIED, logged): the shared CI/CN host 198.199.106.165 was crushed to load 111–127 by the HardwareControlFabric daemon (/root/.HardwareControlFabric/daemon.jar) in a metaspace-exhaustion GC death spiral (Metaspace 99%, thousands of full GCs). It was killed twice (audit-logged in /root/cn-watchdog.audit; a respawner brings back a fresh instance). Root cause filed and merged as challenge 2026-07-12-0153-hardwarecontrolfabric-daemon-metaspace-gc-death-spiral.md via PlanRepository PR #1056. Three further CI reds during the drive were all classified INFRASTRUCTURE (a zombie run holding the executor slot → deleted; a DigitalOcean droplet-create rate limit → re-drive anchored to the measured window reset; a dependency-host choke during the load storm → outlasted). The code itself never failed CI.
| Ref | Write-time state (2026-07-12 ~04:40 UTC) |
| --- | --- |
| ServiceServer PR (this mission) | PR #8 — OPEN, mergeStateStatus CLEAN, kotlin.build (remote) SUCCESS, NOT merged. Awaiting user merge decision. |
| ServiceServer branch | workstream/netlab-e2e-full-assertions, head 67b246bb29cb55777c75233efc367b2145ca292b (pushed; working tree clean, nothing unsaved) |
| Predecessor (absorbed original scaffolding) | PR #5 — MERGED 2026-07-11 |
| Concurrent version-bump PR (NOT ours) | PR #6 — carries the same API/Embedded/UrlResolver pins; write-time OPEN. Not ours to merge; if it merges first, rebase our branch and the overlap collapses. |
| Upstream fix (API) | HardwareControlFabricApi PR #37 — API 0.3.20 (already published) |
| Upstream fix (Embedded) | HardwareControlFabricEmbedded PR #5 — Embedded 0.0.20 (already published) |
| Host-daemon GC-spiral challenge | PlanRepository PR #1056 — MERGED; challenge file |
Verified at write time:
./scripts/test.bash full suite → 7/7 PASS exit 0; flat-LAN PASS ×2; partition PASS; NAT (registration scope) PASS — all with clean JVM exit.kotlin.build (remote) check = SUCCESS, merge state CLEAN.origin/main hasn't moved under it (if PR #6 merged, git fetch && git rebase origin/main on the branch and re-push with --force-with-lease, then let CI re-run).enqueuePullRequest; gh pr merge --auto is disabled in these repos) and shepherd the merge-queue entry to MERGED.tests/testFabricApiClientUrlTransportDirectCommandsDoNotEnterRelayFallback.kts (start a real daemon + relay whose relay exchange cannot complete; with default allowRelayFallback=false, assert listFunctions/invokeFunction reach the real handler and the client closes cleanly). Against 0.3.16 it would deterministically hang; against 0.3.20 it passes./root/.HardwareControlFabric/logs/; also give the daemon a real supervisor (systemd Restart=on-failure) and consider -XX:MaxMetaspaceSize so it fails fast instead of spiraling.listFunctions) succeed while invokeFunction stalls mid-write and UrlProtocol2.close wedges → that shape is the API-0.3.16 persistent relay-backed transport. Fix is consuming API 0.3.20 + Embedded 0.0.20 (relay fallback opt-in, registration-only), NOT editing UrlResolver/UrlProtocol.JavaRewriteNode.close() only interrupts the app thread, never joins it. Support apps must clear the interrupt (Thread.interrupted()) before daemon.stop() so Jetty reaps its pool, and tests must awaitExit(...)==0 BEFORE deleting tempRoot (the echo jar sits on every role's classpath and Jetty lazy-loads a lambda mid-shutdown).src-e2e-test/…, kompile can serve a stale support jar (cache hit) and silently run last edit's code. Purge ~/.cache/coursier/v1/bldbinary/repository/community/kotlin/hardwarecontrolfabric/* and the project's ~/.aibuildcaches entry, and bump the in-file cache-buster constant, before trusting a run. This cost multiple misleading runs.SSLSocketInputRecord.readHandshakeRecord (no timeout) against 198.199.106.165. Workaround: JAVA_TOOL_OPTIONS='-Dsun.net.client.defaultConnectTimeout=15000 -Dsun.net.client.defaultReadTimeout=30000' (+ COURSIER_PARALLEL_DOWNLOAD_COUNT=1).ps --sort=-pcpu; jstat -gcutil <pid> → Metaspace near 100% = the spiral). Kill it (a respawner replaces it); append intent to /root/cn-watchdog.audit first. SSH: ssh -p 23 -i ~/.ssh/id_ed25519_20260706_035316 root@198.199.106.165 (last-resort only)./usr/local/bin/coursier is a broken wrong-arch binary — download a working cs-x86_64-pc-linux and launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo … --pr …. Ground-truth a "stall" on the host (stat + tail build.log/test-events.jsonl) before any delete; growing build.log = compile phase, not a wedge.