← Priority list

Handoff git mirror: five of six defects fixed and deployed; last fix blocked by red UrlResolver main

Six defects found in the Handoff service's GitHub mirror. Five are fixed, merged and deployed (four deterministic drop paths plus a retry-recovery defect deployed 2026-07-30). The sixth �� sandboxed connections being closed while child-object reads are still in flight �� is fixed and open at UrlResolver pull request 897, isolated onto current main and green locally, but cannot merge because UrlResolver's main is itself failing four network-lab/relay tests that provision real droplets on a host sitting at load 54 with a 92%-full disk. Production currently does not reconcile: the sync is stuck in its startup seed phase with the exact connection-closed signature that pull request fixes.

hf-2026-07-28-handoff-git-mirror-four-drop-defects-fixed-and-deployed-two-transport-fixes-blocked-on-build-capacity

Edit handoff

Title Summary Markdown body

# Handoff git mirror: five of six defects fixed and deployed; last fix blocked by red UrlResolver main

**Written 2026-07-30T04:30Z. RE-VERIFY everything below before acting** — it is a write-time
snapshot. Authoritative re-checks: `gh pr view <n> --repo CodexCoder21Organization/<repo> --json
state,mergeStateStatus,statusCheckRollup,mergedAt` for each pull request; `uptime` and the
`url_handoff_.stdout.log` container log on 198.199.106.165 for live mirror behavior; and
`gh run list --repo CodexCoder21Organization/UrlResolver --branch main` for whether that
repository's main branch is still failing.

## Mission

Handoffs created in the Handoff service (`url://handoff/`) sometimes never appeared in its GitHub
mirror at https://github.com/CodexCoder21Organization/PlanRepository/tree/main/handoffs. Find out
whether mirroring is buggy or flaky, fix it, verify in production.

Answer: six defects. Four were deterministic ways a handoff could be dropped forever. Two are
transport-reliability defects that make sync attempts fail. **Five are fixed, merged and deployed.
The sixth is fixed and open as a pull request but cannot land, because UrlResolver's `main` is red
on infrastructure-dependent tests.**

## Current production state (the important part)

The Handoff service currently runs jar SHA-256
`302d64b1116629204e3262b1504e600b85d695115a9becc2f5ec4156434fabc7`, deployed 2026-07-30T03:54Z from
`HandoffServiceServer` main, verified both from the upload response and independently over SSH.
Health is `OK`.

**The mirror is not reconciling.** Since that deploy the sync has been stuck in its startup seed
phase, which must succeed before any reconciliation runs. Two seed failures at 03:59 and 04:03, and
no reconcile attempts at all — so a naive "zero sync failures" reading is misleading; there are zero
*attempts*. No mirror commit has landed since 02:30Z.

Every failure carries the same signature, which is exactly the remaining unfixed defect:

```
UrlResolutionException: RPC request 'apiRequest' to service 'githubproxy' failed:
  Persistent RPC connection to service 'githubproxy' was closed while requests were still pending.
kotlinx.coroutines.JobCancellationException: Job was cancelled
```

Historical rate for context, measured from log timestamps rather than raw totals: 24 sync failures
in the 24h before the deploy, only 2 in the final 6h — the mirror was "mostly working, occasionally
failing", which matches the original report of handoffs going missing *sometimes*. The 391-failure
total in the log spans five days from 2026-07-25 and should not be read as a current rate.

## The six defects

Fixed, merged, deployed (four deterministic drop paths):

1. **Non-ASCII identifiers were skipped forever** — the mirror's filename validator accepted only
   ASCII while handoff ids keep Unicode letters.
2. **A restart erased pending changes** — startup recorded service and git content as already
   synchronized without comparing them.
3. **A handoff on no priority list was invisible** — discovery walked priority lists, so a handoff
   removed from its last list before its first sync was never enumerated again, while the sync still
   reported success.
4. **A dequeued mirror pull request stalled everything** — when the sync's own pull request lost its
   merge-queue place, later attempts only polled it and never re-queued.

Fixed, merged, deployed 2026-07-30 (transport defect 1 of 2):

5. **A replacement connection session that failed immediately killed the whole attempt** — recovery
   opened a fresh session then performed the retried read *outside* the retry loop, so an unhealthy
   replacement ended the sync with 39 of 40 retries unused.

Fixed, pull request open, **NOT merged** (transport defect 2 of 2 — the one currently stopping
everything):

6. **Connections are closed while child-object work is still in flight** — a sandboxed connection's
   idle cleanup counted activity only on the root proxy object, so reads proceeding through child
   proxies were judged idle and the connection closed mid-read.

## The blocker

`UrlResolver` main is failing its own required checks, independently of our change:

- main (`086927a2c`, run of 2026-07-29T08:40Z): `bld-build` failure, and `kotlin.build (remote)`
  1224 passed / 4 failed — `testNetlabDynamicFailureRecovery`, `testNetlabLargeSwarmGossip`,
  `testNetlabSevereImpairmentScenarios`, `testUnverifiedPeerAnnouncementDoesNotReplayDiscoveredServices`.
- our isolated branch: 1219 passed / 4 failed — `testNetlabDynamicFailureRecovery`,
  `testNetlabNatToNatRelayOutageRecovery`, `stressTestHealthySameTierRelayWinsUnderContention`,
  `testUnverifiedPeerAnnouncementDoesNotReplayDiscoveredServices`. Two are identical to main's; all
  four are in the network-lab / relay class. None touch the sandbox lifecycle code this change
  modifies.

The network-lab tests provision real droplets, and the shared host 198.199.106.165 has been at load
average ~54 on 8 cores with the root filesystem 92% full throughout. Per the standing guidance that
host overload is a code defect rather than under-provisioning, that saturation deserves its own
investigation; it was out of scope here.

**Do not bypass the required check to land this.** The merge gate is doing its job; main needs to be
green first.

## Relevant pull requests, branches and refs

Merged:

- https://github.com/CodexCoder21Organization/HandoffApi/pull/5 (2026-07-28T13:43:58Z) — adds
  `listAllOpenHandoffsJson()`; published `handoff:api:0.0.6`.
- https://github.com/CodexCoder21Organization/HandoffEmbedded/pull/5 (2026-07-28T13:52:48Z) —
  implements it; published `handoff:embedded:0.0.7`.
- https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/13 (2026-07-28T14:04:21Z) —
  defects 1-4; published server/client `0.0.10`; deployed.
- https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/14 (2026-07-29T13:18:29Z) —
  defect 5; **deployed 2026-07-30T03:54Z** (it had been merged but never deployed for a day).

Open:

- https://github.com/CodexCoder21Organization/UrlResolver/pull/897 — branch
  `sandbox-child-proxy-activity`, remote head `b2fb6535a4316a5e598cba906d45a6f6ed0889f8`,
  https://github.com/CodexCoder21Organization/UrlResolver/tree/sandbox-child-proxy-activity
  Defect 6, isolated onto current main. Three regression tests, each demonstrated failing first;
  all three pass locally (3/3). Blocked by the red main described above.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/885 — the original branch for this
  fix, which a different session has since extended with three unrelated stabilization efforts
  (harness subscriber cleanup, NetLab route state checks, same-tier relay determinism). Left alone;
  a comment there points at the isolated pull request.

Artifacts (investigation working notes, `OBSERVED:`/`INFER:` format):

- https://github.com/CodexCoder21Organization/PlanRepository/tree/handoff-artifacts-mirror-reliability-20260728/handoffs/artifacts/2026-07-28-handoff-mirror-reliability
  remote head `f92120eaa87f269efff7cb94c98aa92ab6a89435`. Start with that directory's `README.md`.

All work is pushed; local and remote heads were compared and matched for each branch.

## Next steps

1. **Get UrlResolver main green.** Until then nothing merges there. The four failing tests are
   network-lab and relay scenarios that provision real droplets; check whether the droplet host is
   still saturated (`uptime` on 198.199.106.165, and its 92%-full root filesystem) before assuming
   the tests themselves are at fault.
2. **Merge https://github.com/CodexCoder21Organization/UrlResolver/pull/897** once its required
   check can pass. No code change should be needed — it is green locally and its failures are shared
   with main.
3. **Publish the resolver artifact** from merged main; verify the published jar contains
   `SandboxActivityTracker` and resolves from an empty Coursier cache before relying on it.
4. **Bump the Handoff service's resolver dependency, rebuild, redeploy.** Deploy is
   `container-nursery-cli upload-jar --file <jar> --route "url:handoff:" --url
   https://api.nursery.wasmserver.com`, which restarts the route itself and reports the server-side
   SHA-256 — compare that against the local build rather than trusting the success message.
5. **Verify against the original bug, asserting the negative too.** The startup seed must complete,
   a reconcile must succeed, and a probe handoff with a non-ASCII title must appear in the mirror
   with its escaped filename and then disappear when completed. Confirm the seed/sync failure lines
   stop appearing in `url_handoff_.stdout.log`.

## Reusable knowledge

- **Zero errors can mean zero attempts.** The post-deploy metric showed no sync failures, which
  looked like success; in fact nothing was syncing because an earlier phase never completed. Confirm
  the operation actually ran before crediting a fix.
- **A raw error count is not a rate.** Divide by time against the expected attempt rate before
  calling something broken — a five-day tally read as a current rate badly overstated severity here.
- **A merged fix is not a delivered fix.** Defect 5 sat merged on main for a day while production
  ran the older binary.
- **Verify a deploy by artifact identity from two directions** — the upload tool's reported digest
  and the host's own hash, both compared to the local build.
- **When a third party extends your branch, isolate rather than untangle.** Cherry-picking the two
  relevant commits onto current main produced a pull request whose CI signal means something.
- **Compare a failing pull request against its base before blaming the change.** Main was failing
  the same class of tests, which is what established the change was not at fault.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff git mirror: five of six defects fixed and deployed; last fix blocked by red UrlResolver main

Written 2026-07-30T04:30Z. RE-VERIFY everything below before acting — it is a write-time snapshot. Authoritative re-checks: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt for each pull request; uptime and the url_handoff_.stdout.log container log on 198.199.106.165 for live mirror behavior; and gh run list --repo CodexCoder21Organization/UrlResolver --branch main for whether that repository's main branch is still failing.

Mission

Handoffs created in the Handoff service (url://handoff/) sometimes never appeared in its GitHub mirror at https://github.com/CodexCoder21Organization/PlanRepository/tree/main/handoffs. Find out whether mirroring is buggy or flaky, fix it, verify in production.

Answer: six defects. Four were deterministic ways a handoff could be dropped forever. Two are transport-reliability defects that make sync attempts fail. Five are fixed, merged and deployed. The sixth is fixed and open as a pull request but cannot land, because UrlResolver's main is red on infrastructure-dependent tests.

Current production state (the important part)

The Handoff service currently runs jar SHA-256 302d64b1116629204e3262b1504e600b85d695115a9becc2f5ec4156434fabc7, deployed 2026-07-30T03:54Z from HandoffServiceServer main, verified both from the upload response and independently over SSH. Health is OK.

The mirror is not reconciling. Since that deploy the sync has been stuck in its startup seed phase, which must succeed before any reconciliation runs. Two seed failures at 03:59 and 04:03, and no reconcile attempts at all — so a naive "zero sync failures" reading is misleading; there are zero attempts. No mirror commit has landed since 02:30Z.

Every failure carries the same signature, which is exactly the remaining unfixed defect:

UrlResolutionException: RPC request 'apiRequest' to service 'githubproxy' failed:
  Persistent RPC connection to service 'githubproxy' was closed while requests were still pending.
kotlinx.coroutines.JobCancellationException: Job was cancelled

Historical rate for context, measured from log timestamps rather than raw totals: 24 sync failures in the 24h before the deploy, only 2 in the final 6h — the mirror was "mostly working, occasionally failing", which matches the original report of handoffs going missing sometimes. The 391-failure total in the log spans five days from 2026-07-25 and should not be read as a current rate.

The six defects

Fixed, merged, deployed (four deterministic drop paths):

Non-ASCII identifiers were skipped forever — the mirror's filename validator accepted only ASCII while handoff ids keep Unicode letters.
A restart erased pending changes — startup recorded service and git content as already synchronized without comparing them.
A handoff on no priority list was invisible — discovery walked priority lists, so a handoff removed from its last list before its first sync was never enumerated again, while the sync still reported success.
A dequeued mirror pull request stalled everything — when the sync's own pull request lost its merge-queue place, later attempts only polled it and never re-queued.

Fixed, merged, deployed 2026-07-30 (transport defect 1 of 2):

A replacement connection session that failed immediately killed the whole attempt — recovery opened a fresh session then performed the retried read outside the retry loop, so an unhealthy replacement ended the sync with 39 of 40 retries unused.

Fixed, pull request open, NOT merged (transport defect 2 of 2 — the one currently stopping everything):

Connections are closed while child-object work is still in flight — a sandboxed connection's idle cleanup counted activity only on the root proxy object, so reads proceeding through child proxies were judged idle and the connection closed mid-read.

The blocker

UrlResolver main is failing its own required checks, independently of our change:

main (086927a2c, run of 2026-07-29T08:40Z): bld-build failure, and kotlin.build (remote) 1224 passed / 4 failed — testNetlabDynamicFailureRecovery, testNetlabLargeSwarmGossip, testNetlabSevereImpairmentScenarios, testUnverifiedPeerAnnouncementDoesNotReplayDiscoveredServices.
our isolated branch: 1219 passed / 4 failed — testNetlabDynamicFailureRecovery, testNetlabNatToNatRelayOutageRecovery, stressTestHealthySameTierRelayWinsUnderContention, testUnverifiedPeerAnnouncementDoesNotReplayDiscoveredServices. Two are identical to main's; all four are in the network-lab / relay class. None touch the sandbox lifecycle code this change modifies.

The network-lab tests provision real droplets, and the shared host 198.199.106.165 has been at load average ~54 on 8 cores with the root filesystem 92% full throughout. Per the standing guidance that host overload is a code defect rather than under-provisioning, that saturation deserves its own investigation; it was out of scope here.

Do not bypass the required check to land this. The merge gate is doing its job; main needs to be green first.

Relevant pull requests, branches and refs

Merged:

https://github.com/CodexCoder21Organization/HandoffApi/pull/5 (2026-07-28T13:43:58Z) — adds listAllOpenHandoffsJson(); published handoff:api:0.0.6.
https://github.com/CodexCoder21Organization/HandoffEmbedded/pull/5 (2026-07-28T13:52:48Z) — implements it; published handoff:embedded:0.0.7.
https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/13 (2026-07-28T14:04:21Z) — defects 1-4; published server/client 0.0.10; deployed.
https://github.com/CodexCoder21Organization/HandoffServiceServer/pull/14 (2026-07-29T13:18:29Z) — defect 5; deployed 2026-07-30T03:54Z (it had been merged but never deployed for a day).

Open:

https://github.com/CodexCoder21Organization/UrlResolver/pull/897 — branch sandbox-child-proxy-activity, remote head b2fb6535a4316a5e598cba906d45a6f6ed0889f8, https://github.com/CodexCoder21Organization/UrlResolver/tree/sandbox-child-proxy-activity Defect 6, isolated onto current main. Three regression tests, each demonstrated failing first; all three pass locally (3/3). Blocked by the red main described above.
https://github.com/CodexCoder21Organization/UrlResolver/pull/885 — the original branch for this fix, which a different session has since extended with three unrelated stabilization efforts (harness subscriber cleanup, NetLab route state checks, same-tier relay determinism). Left alone; a comment there points at the isolated pull request.

Artifacts (investigation working notes, OBSERVED:/INFER: format):

https://github.com/CodexCoder21Organization/PlanRepository/tree/handoff-artifacts-mirror-reliability-20260728/handoffs/artifacts/2026-07-28-handoff-mirror-reliability remote head f92120eaa87f269efff7cb94c98aa92ab6a89435. Start with that directory's README.md.

All work is pushed; local and remote heads were compared and matched for each branch.

Next steps

Get UrlResolver main green. Until then nothing merges there. The four failing tests are network-lab and relay scenarios that provision real droplets; check whether the droplet host is still saturated (uptime on 198.199.106.165, and its 92%-full root filesystem) before assuming the tests themselves are at fault.
Merge https://github.com/CodexCoder21Organization/UrlResolver/pull/897 once its required check can pass. No code change should be needed — it is green locally and its failures are shared with main.
Publish the resolver artifact from merged main; verify the published jar contains SandboxActivityTracker and resolves from an empty Coursier cache before relying on it.
Bump the Handoff service's resolver dependency, rebuild, redeploy. Deploy is container-nursery-cli upload-jar --file <jar> --route "url:handoff:" --url https://api.nursery.wasmserver.com, which restarts the route itself and reports the server-side SHA-256 — compare that against the local build rather than trusting the success message.
Verify against the original bug, asserting the negative too. The startup seed must complete, a reconcile must succeed, and a probe handoff with a non-ASCII title must appear in the mirror with its escaped filename and then disappear when completed. Confirm the seed/sync failure lines stop appearing in url_handoff_.stdout.log.

Reusable knowledge

Zero errors can mean zero attempts. The post-deploy metric showed no sync failures, which looked like success; in fact nothing was syncing because an earlier phase never completed. Confirm the operation actually ran before crediting a fix.
A raw error count is not a rate. Divide by time against the expected attempt rate before calling something broken — a five-day tally read as a current rate badly overstated severity here.
A merged fix is not a delivered fix. Defect 5 sat merged on main for a day while production ran the older binary.
Verify a deploy by artifact identity from two directions — the upload tool's reported digest and the host's own hash, both compared to the local build.
When a third party extends your branch, isolate rather than untangle. Cherry-picking the two relevant commits onto current main produced a pull request whose CI signal means something.
Compare a failing pull request against its base before blaming the change. Main was failing the same class of tests, which is what established the change was not at fault.