Handoff: PeerObservationLog release built but ContainerNursery redeploy is blocked by protocol/resolver binary incompatibility

Written 2026-07-11 UTC.

RE-VERIFY BEFORE ACTING: every state below is a write-time snapshot. Re-check Maven availability with curl -fsSL https://kotlin.directory/api/list-artifacts, re-check the UrlProtocol change with gh pr view 352 --repo CodexCoder21Organization/UrlProtocol --json state,mergeStateStatus,statusCheckRollup,mergedAt, and rerun the full ContainerNursery suite against the exact protocol/resolver pair before treating any fat jar as deployable. At write time kotlin.directory:443 was refusing connections, although artifact publication and download had been independently verified earlier in the session.

Mission summary

The production ContainerNursery router was OOM-cycling roughly every 25 minutes because PeerObservationLog retained observations for evicted peers. The request was to break the CI/ContainerNursery deadlock by publishing the already-reviewed UrlProtocol fix from commit 0516dcd, build a fresh ContainerNursery fat jar pinned to the new artifact, and stop before any production upload, restart, or redeploy.

The protocol publication and fat-jar build succeeded. The required downstream compatibility check then proved that the jar is unsafe: UrlResolver 0.0.600 calls a constructor descriptor removed from protocol 0.0.341, so ContainerNursery's URL facade would fail at runtime.

What was found and done

1. /code/UrlProtocol was verified clean on branch fix-peerobservationlog-unbounded-retention at 0516dcda853d2fb6d5276b5b7cc487d0475daaf3, exactly one commit ahead of then-current origin/main. Its foundation.url.protocol.buildMaven rule declared foundation.url:protocol:0.0.341, while the repository initially listed 0.0.337 as latest published.

2. The exact requested publisher command succeeded:

   coursier launch community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.5 \
     -r https://kotlin.directory -- \
     --workspace /code/UrlProtocol \
     --buildrule foundation.url.protocol.buildMaven \
     --api-url https://api.kotlin.directory/upload \
     --force
   

3. Publication was independently verified before kotlin.directory degraded again:

   • https://kotlin.directory/api/list-artifacts returned foundation.url:protocol:0.0.341.
   • The published POM declared group foundation.url, artifact protocol, version 0.0.341.
   • The published JAR was 1,070,281 bytes with SHA-256 1aadea3d600ad4cc5f8b1375c3fc9e59f5856c179a25326a3612e597f7620424.
   • javap -c -p foundation.url.protocol.PeerRegistry showed PeerObservationLog.removePeer(peerId) at bytecode offset 23 before the absent-peer early return at offset 30. The published artifact therefore contains the requested leak cleanup.

4. The existing UrlProtocol review https://github.com/CodexCoder21Organization/UrlProtocol/pull/352 merged at 2026-07-11T10:45:46Z; its bld-build, bld-all-tests, and kotlin.build (remote) checks were all successful at the last live query.

5. ContainerNursery was cloned fresh into /code/ContainerNursery at 7c337e9a55b29f15e75d005ab2fd0ad0635c0733. A temporary branch incident/peer-observation-log-0.0.341 changed only build.kts, replacing direct protocol pin 0.0.336 with 0.0.341; resolver remained 0.0.600.

6. The exact ContainerNursery build rule was containernursery.buildFatJar, invoked as:

   ./scripts/build.bash containernursery.buildFatJar \
     /code/container-nursery-protocol-0.0.341.jar
   

   The build completed successfully and explicitly resolved protocol-0.0.341.jar with resolver-0.0.600.jar. The output is 128,678,608 bytes with SHA-256 f8954a4459ff65af54a3e34ae14b6059d05d51b436005481262cbdf937c9791e.

7. The built fat jar contains exactly one foundation/url/protocol/PeerRegistry.class. Disassembly again proved that its removePeer implementation calls PeerObservationLog.removePeer before early return.

8. The full local ContainerNursery suite was run with:

   scripts/test.bash --test . \
     --log /tmp/container-nursery-protocol-0.0.341-tests.xml
   

   It failed deterministically: TESTS FAILED (282/327 tests completed successfully, 45 failed). URL-facade tests consistently reported:

   java.lang.NoSuchMethodError: 'void foundation.url.protocol.UrlProtocol.<init>(foundation.url.protocol.PeerRegistry, foundation.url.protocol.RelayConfig)'
       at foundation.url.resolver.UrlProtocol2.<init>(UrlResolver.kt:975)
   

9. Bytecode comparison proves the mechanism end to end:

   • UrlResolver 0.0.600 invokes UrlProtocol.<init>(PeerRegistry, RelayConfig) at UrlProtocol2 bytecode offset 936.
   • Protocol 0.0.336 exposes that exact two-argument constructor.
   • Protocol 0.0.341 exposes PeerRegistry constructors that also include NameResolutionConfig (plus larger primary/synthetic forms), but no method with the exact two-argument JVM descriptor.
   • Compilation succeeds because ContainerNursery compiles against current classes; runtime fails when the already-compiled resolver links its older constructor call.

10. The unsafe ContainerNursery pin was reverted. /code/ContainerNursery is clean on its temporary local branch, no ContainerNursery commit or PR was created, and the fat jar remains only as a quarantined diagnostic artifact at the requested path. Do not deploy it.

11. The ecosystem challenge and durable-fix recommendation were recorded at the merged PlanRepository challenge.

Relevant PRs and references

| Item | Write-time state | | --- | --- | | UrlProtocol fix | MERGED at 2026-07-11T10:45:46Z; head 0516dcda853d2fb6d5276b5b7cc487d0475daaf3; all three checks successful | | foundation.url:protocol:0.0.341 | Published and bytecode-verified earlier; live repository endpoint refusing connections at handoff write time | | Compatibility challenge | Merged into PlanRepository main | | /code/container-nursery-protocol-0.0.341.jar | Built and bytecode-verified, but QUARANTINED / DO NOT DEPLOY because 45 tests prove runtime ABI failure | | /code/ContainerNursery, branch incident/peer-observation-log-0.0.341 | Clean, no source delta, no commit, no PR; temporary unsafe pin was reverted |

Next steps

1. Re-verify current state first using the commands in the banner, especially kotlin.directory availability and whether a newer compatible protocol or resolver has since been published.
2. Fix the incompatibility in the owning upstream layer rather than hiding it in ContainerNursery. The lowest-risk option is to restore the exact public JVM constructor UrlProtocol(PeerRegistry, RelayConfig) as a delegating compatibility constructor, add an end-to-end test that loads/exercises published UrlResolver 0.0.600, bump protocol to a fresh version greater than 0.0.341, and publish it. A separately published resolver compiled against the new ABI is another valid route, but it must be tested as an exact pair.
3. Do not overwrite 0.0.341; publish a new Maven version and independently inspect its POM/JAR bytecode after upload.
4. Fresh-clone or update ContainerNursery from current main, pin it to the chosen compatible protocol/resolver pair, and run both containernursery.buildFatJar and scripts/test.bash --test .. Zero URL-facade failures are required.
5. Inspect the rebuilt fat jar for a single PeerRegistry.class, prove the cleanup call with javap, record size/SHA-256, and only then hand it to the human operator.
6. Per the original safety boundary, a human must review and perform any production ContainerNursery upload/redeploy/restart. Do not perform those operations automatically.

Reusable and operational knowledge

• A successful fat-jar compilation does not prove binary compatibility with already-compiled Maven dependencies. Exercise the runtime constructor/call path or inspect both sides' JVM descriptors.
• Resolver 0.0.600's POM transitively requests protocol 0.0.337, but a higher direct protocol version wins dependency resolution. That version override is unsafe when the higher artifact removes an older JVM descriptor.
• The quarantined jar's hash is stable and was rechecked after the full suite: f8954a4459ff65af54a3e34ae14b6059d05d51b436005481262cbdf937c9791e.
• During this incident kotlin.directory alternated between successful artifact access and immediate curl: (7) Failed to connect to kotlin.directory port 443. Follow the requested 60–120 second retry cadence for fetch failures, but do not mistake a deterministic NoSuchMethodError for transient infrastructure.