Handoff: NetLab worker transport parity stopped because the historical manager transport cannot be rebuilt

Written 2026-07-28 08:55 UTC.

RE-VERIFY: This is a write-time snapshot. Before acting, re-check the production jar checksums and class CRCs over SSH, re-check worker and manager main, inspect open PRs with gh pr list, and read current Maven metadata from https://kotlin.directory/. Fresh live evidence takes precedence over this document.

Mission summary

Rebuild /root/ContainerNursery/apps/netlab-worker-server.jar from the same transport source vintage as the running Jul-27 NetLab manager, add bounded worker JVM and heap-dump flags through the owning launcher, atomically deploy only the worker jar without restarting the manager, then measure locally closed stream failures for at least 45 minutes after new workers use the replacement.

The task explicitly required a stop without source edits if worker main could not supply the manager transport.

What was found and done

1. Fresh checkouts verified the supplied source revisions:
   • NetLabManagerServer main: 022724742a3cbf103f1359f4a6a24f9458f58c7d.
   • NetLabWorkerServer main: 7e02b058ed094cccbc996a7f9e18ec233bc16c3c.
2. Production was inspected read-only. The running manager jar remains SHA-256 9c0ca9e12a72196ced1917fca649d6aea72991261c4f1220e5fab55b38783296; its UrlProtocol2.class is 581869 bytes with CRC 3cacd644, and it contains 750 resolver-package classes. The live worker jar remains SHA-256 846cadc5ae02374a4addd6d1e6ebefc3024caf09fa97105bcca96c2fba195a5b; its corresponding class is 483373 / 79255c71.
3. A clean isolated build proved the running manager application source is exactly 0227247: all 23 ProvisioningService*.class entries match the running jar in name, size, and CRC, including the class introduced by that final commit.
4. The manager source pins resolver 0.0.670, protocol 0.0.401, and libp2p snapshot-19. A clean rebuild of those nominal inputs now embeds UrlProtocol2.class 542842 / 5b043c4b and 669 resolver-package classes, not the running 581869 / 3cacd644 and 750 classes. The coordinate no longer reproduces the historical Jul-27 bytes.
5. Worker main pins resolver 0.0.641, protocol 0.0.330, and libp2p snapshot-9. The only open worker PR, https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/62, pins 0.0.650/0.0.384/snapshot-19; published worker 0.0.36 pins 0.0.599/0.0.332/snapshot-9. None matches the manager set.
6. The worker launch command belongs to the manager repository at src/netlabmanager/SshProvisioner.kt, function buildWorkerStartCommand. The proposed 4 GB droplet flags are -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/netlab-worker/heapdumps, with the directory created before launch. The worker jar contains no launcher script, so a jar-only swap cannot add these flags.
7. The explicit STOP condition was applied. No source edits, new source PRs, host backup, host file swap, manager restart, worker provisioning, or measurement occurred.

Relevant PRs and refs

| Ref | Write-time state | Purpose | |---|---|---| | https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/62 | OPEN, CLEAN; both checks successful | Existing worker transport bump, but it does not match the manager transport | | https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/netlab-workerjar-blocked-handoff-2026-07-28 | Remote head 468b6d1eae1ee79045c684b18ca89f91c4cc4cdf | Preserves the full requested report under handoffs/artifacts/2026-07-28-netlab-workerjar-report.md | | https://github.com/CodexCoder21Organization/NetLabManagerServer/commit/022724742a3cbf103f1359f4a6a24f9458f58c7d | manager main and running application source | Source revision proven by class CRC comparison | | https://github.com/CodexCoder21Organization/NetLabWorkerServer/commit/7e02b058ed094cccbc996a7f9e18ec233bc16c3c | worker main | Latest worker source; transport pins remain behind the manager |

All implementation repositories are clean. There are no local-only source commits, stashes, or worktrees.

Next steps

1. Re-verify all current state first.
2. Decide one of two authorized paths:
   • Recover or publish an immutable resolver/protocol/libp2p artifact set whose UrlProtocol2.class is exactly 581869 / 3cacd644, then build the worker against that set.
   • Explicitly authorize a coordinated source change that selects a new common manager/worker transport set. This is not a same-vintage rebuild and requires tests at both ends.
3. If source changes are authorized, read the testing architecture guidance before adding tests. Keep the worker dependency change and manager JVM launcher change separate.
4. For the launcher PR, create /opt/netlab-worker/heapdumps and launch the worker with -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/netlab-worker/heapdumps.
5. Before any deployment, verify the built worker's UrlProtocol2.class size/CRC and resolver-package class count against the chosen manager ground truth.
6. Follow the rollback procedure in the preserved report, atomically replace only the worker jar, do not restart the manager, and wait for naturally provisioned new workers.
7. Start the 45-minute measurement only after a new worker log proves it runs the replacement. Record before/after 10-minute buckets and attribute failures by worker UUID.

Reusable and operational knowledge

• The current worker repository is named NetLabWorkerServer; older NetLab skill text calls it NetLabServiceServer.
• The normal manager build scans sibling tests and can emit unhandled dependency-pin warnings without producing the requested jar. An isolated build workspace containing only build.kts, src, and src-client produced the comparison jar.
• HardwareControlFabric client certificates were absent in this environment, so the authorized SSH fallback was used for read-only production inspection.
• Matching an application source commit is not enough when dependency coordinates can resolve to changed bytes. Compare the embedded class CRC and class count before treating two fat jars as the same transport vintage.
• The pre-registered baseline is 3.02% uploadFile failures, with closed-stream events in 10-minute buckets of 76, 304, 503, 778, 736, 439, and 565.