2 commits

Author	SHA1	Message	Date
Danny Avila	8c14f03432	🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056) ... * 🗂️ feat: Scope Scheduled Chats to Chat Projects Adds an optional chat-project destination to a schedule, plus the operator config to require one — or to pin every scheduled run to a specific project. Feature: - `chatProjectId` on the schedule row, accepted on create/update, projected on the wire, and carried into the run's conversation through the durable trigger envelope's `run` context. - `interface.schedules.requireProject` refuses schedules that are not filed under a project; `interface.schedules.projectId` pins every run to one project and implies the requirement. - Dialog gains a project picker (required when configured, a read-only row when pinned); the card shows the destination and the new disabled reasons. Invariants: - ONE resolver (`resolveScheduleProjectId`) decides the destination for the write handler, the fire path, and the wire projection alike, and an operator pin OUTRANKS the stored id in all three. Tightening the config therefore redirects — or stops — existing schedules instead of grandfathering where their runs land. A pin implies `requireProject` for the same reason: without it, a row created before the pin would keep firing with no project at all. - Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write this handler accepts is one the next fire also accepts. Any edit leaving a schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips the requirement, or a schedule auto-disabled for `project_required` could never be turned off. - Fire-time enforcement auto-disables rather than filing runs loose, matching agent_deleted: new `project_required` (requirement raised after creation) and `project_deleted` (gone, or pinned to a project this owner does not have) reasons, both refused BEFORE a billed generation is dispatched, and both advancing so a schedule can never wedge on the occurrence. - `computeCreateDigest` appends the field only when present, so a payload without a project digests byte-identically to one from before this change — a create retried across the upgrade still matches its own row instead of reading as key reuse. - Project reads are scoped to the owner, so ownership and existence are the same lookup; a read ERROR propagates instead of failing closed, so a Mongo blip retries the fire rather than auto-disabling the schedule. The trigger idempotency key hashes principal/event/target and never `envelope.run`, so the added run field cannot destabilize delivery identity. * 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget The project picker landed as a new ROW in the schedule dialog, which broke the e2e edit spec: `md:overflow-visible` turns off the template's scrolling from `md` up, so the dialog's content must fit the viewport. The extra row pushed the footer's Save button below a 720x1280 window, where Playwright reported a visible, enabled button it could never click — 226 scroll-into-view retries and a 2-minute timeout, on all three attempts. Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720): dev 688 (32px slack) project row ~790 (off-screen, CI failure) 3-column row 704 (16px slack — half the budget spent) this commit 686 (34px slack, 2px better than dev) The identity row is now three columns — name, agent, project — and its caption moved out of the agent cell to sit full width beneath the row: at a third of the dialog that sentence wraps an extra line, and the row is the tallest thing competing for the budget. The caption is grouped with the row rather than left to the form's own 4-unit rhythm, which spent more height on the gap than the caption occupies. The e2e spec now asserts the button is in the viewport before clicking it, so the next field that overflows this dialog says so in one line instead of a two-minute timeout on a visible element. FOLLOWUPS.md records what the planned dialog controls (multi-day weekly, timezone, attachments) need first: give `ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the popovers into the dialog content, and let the form scroll again. * 🧹 fix: Address Codex Review on Scheduled Chat Project Scope Four P2 findings, all real. Unreachable clearing path (ScheduleDialog). The picker only held live projects, so `com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a schedule had a project the owner could never take it away, leaving the server's `chatProjectId: null` path reachable only by API. The picker now carries a real "No project" option whenever a project is optional, and omits it when one is required, where there is nothing valid to select. Placeholder shown for a real project (ScheduleDialog). A stored or pinned project outside the first loaded page had no name in the paged map, and the combobox renders its placeholder for an empty display value — telling the owner a scoped schedule had no project. That one project is now read by id, with the raw id as a last resort: a poor label, but an honest one. Project policy skipped at the resume boundary (service.ts). `claimScheduleResume` re-applied the schedules gate, the revision fence, the kill switch and SCHEDULES:USE, but not the project policy this PR added. Approving a paused run whose project was deleted — or whose owner now sits under a requirement or a pin it no longer satisfies — billed a continuation the very next scheduled fire would refuse and auto-disable the schedule for. The effective-project resolution now runs there too, refused before the lease and the capacity slot so a policy refusal costs nothing and leaves no state to unwind. NOTE: agent access and balance are still not rechecked on resume; that gap predates this PR and is left alone. Per-card project derivation (ScheduleCard). Every card ran the projects hook and rebuilt the full option array, name map, and one icon element per project, to use a single name — O(schedules x projects) per render and per project-list refresh. The hook is split: `useChatProjectNames` (map only, for the panel, which resolves every card's name once and passes it down) and `useChatProjectPicker` (options and pagination, for the dialog's one combobox). The panel skips the query entirely until some schedule actually has a scope. Tests: four at the resume boundary (verified failing without the gate) and four in the dialog spec. The picker selection in one existing test now goes through the search field — the popover's VIRTUALIZED renderer sizes its window from a scroll height jsdom always reports as 0, so with three options it materialized only two. Full schedules e2e re-run green against the rebuilt client. * 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent Second Codex round, four P2s. Three were consequences of the resume gate added in the previous commit, which was half-built: it admitted where it should not and stranded the run where it refused. Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy` branch. Both entry points consult that branch FIRST, and both already route its refusal through abort-and-settle — so a policy stop now settles the occurrence instead of answering a bare 409 while the job stays `requires_action`, the card keeps reading "Needs approval", and every retry repeats the same 409 until expiry. No change to resume.js: the existing branch does the work. The rule is deliberately NARROW. It refuses only where no valid destination is left — the requirement is on with nothing satisfying it, or the schedule's own project is gone (which also unset it on the conversation). It does NOT refuse because an operator's pin moved: the paused conversation cannot be rebound (`chatProjectId` is excluded from the resume context and the continuation reuses the same conversationId), so refusing would strand a pending approval over a destination it can never reach, for a pin that governs only where the NEXT run lands — which the fire path already redirects. Create retries are idempotent again. Project policy had been applied BEFORE the `clientRequestId` replay lookup, so a raised requirement, a deleted project, or a moved pin could answer 400 for a create that already committed — pushing the client to rotate its key and create a DUPLICATE schedule, the exact failure the key exists to prevent. Policy now applies only to a genuinely new insert, and the digest is computed from the CLIENT's payload rather than the resolved destination, so today's policy can no longer re-digest a genuine retry into a mismatch. An explicit `chatProjectId: null` under a pin is refused rather than silently resolved to the pin. The payload contract defines `null` as clearing the scope; answering 201 while filing under the pin reported success for the opposite of what was asked. Only an OMITTED field takes the pin silently. Tests: five on the policy branch (both refusals verified failing without it, plus guards that a moved pin and a live project still admit) and two on the handlers (the pinned explicit clear, and a committed create recovered by retry after the policy tightened — verified answering 400 without the reordering). * 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved Third Codex round, four P2s. The root confusion behind the resume findings: an operator pin outranks the stored id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row keeps its old value. The row therefore LIED about where that occurrence's conversation went, and every later re-validation — the resume boundary above all — checked a project the conversation was never filed under. A schedule storing A, pinned to B, with B later deleted and A still live, was admitted for resume into a conversation that had just been unscoped. Fixed at the source rather than at each reader: a fire that resolves a destination different from the stored one writes it back, claim-token fenced like every other worker-side write and deliberately WITHOUT a configRevision bump — this is the server reconciling itself to policy, not an owner edit, and a bump would fence an in-flight occurrence off its own run. Written only AFTER the destination validates, so an unusable pin never lands in the row, and best-effort: the envelope already carries the right destination, so a failed write costs accuracy on a later recheck, never the run itself. The wire projection already reported the pin, so this also stops the row and the UI disagreeing. An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit path too. The pin check and the requirement are independent rules, and folding them together let `{enabled: false, chatProjectId: null}` skip the pin check entirely, unset the row, and answer with a wire projection still naming the pin. Only the requirement is waived for a disabling edit. The dialog no longer requires a project for an edit that leaves a schedule DISABLED. The server waives the requirement there precisely so a row auto-disabled for `project_required` can still be renamed or tidied up; requiring it in the form made that unreachable, and an owner with no projects could not edit the stopped schedule at all. FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second deletion race inside the resume claim window (which needs the effective project persisted per OCCURRENCE plus a distinct policy conflict routed through abort-and-settle), and the fact that convergence happens only when a schedule fires. Tests: four on convergence (pin written, no write when unchanged, no write for a destination that failed validation, fire survives a failed write), two on the disabling-edit pin rules, two in the dialog. Schedules e2e re-run green. * 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest `computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and an explicit `null` produced the same digest. Because the replay lookup and `matchesCreateIntent` deliberately run before project policy, a request could reuse a pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and receive 201 describing the pinned row — success reported for the opposite of what it asked, and the pinned-clear refusal the normal create path applies never reached. Now `!== undefined`: an omitted field still digests byte-identically to a payload from before project scope existed, so a create in flight across the upgrade still matches its own row, while an explicit clear is a distinct intent and digests differently. A pre-scope client never sent the field at all, so nothing legacy can carry an explicit null. * 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used The schedule-wide convergence from the previous commit was not enough, and the reason is the single-active run index: it covers `status: 'started'` only, so a PAUSED run does not block the next occurrence. While run 1 sat paused in project A, a pin move plus one later fire rewrote the schedule row to B — and the resume policy then validated B while run 1's conversation was still filed under A. Delete A and the continuation was admitted into a conversation that had just been unscoped. That window lasts as long as the pause, not the sub-second race the previous commit documented. The reservation now records the destination THIS occurrence used, and `isScheduleLive` validates that record when given the occurrence's `scheduledFor` — which `resume.js` already reads two lines above the call. No new conflict type and no settlement-path surgery: the refusal rides the abort-and-settle branch that check already has. An ABSENT record falls back to the schedule-level resolution rather than reading as unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as they do today. Schedule-wide convergence stays: it keeps the row honest for the UI and for every check that has no occurrence in hand. FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the claim window, which needs a distinct policy conflict routed through abort-and-settle rather than the bare 409 an `inactive` conflict produces. Tests: three on occurrence-vs-row precedence (the decisive one verified failing without the lookup), two on what the reservation records, and the resume controller spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green. * 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One The occurrence fallback added in the previous commit conflated two different absences. A post-upgrade run that deliberately went unscoped omitted the field exactly like a row written before the field existed, so both took the fallback — and a paused unscoped run was then validated against the schedule's CURRENT project. Under a requirement or a pin added while it sat paused, that admitted a billed continuation into a conversation satisfying no present policy. The reservation now ALWAYS records its decision, `null` for unscoped, and the read reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record — a pre-scope row, or no row at all — falls back to the schedule-level resolution; a recorded null is the genuinely unscoped occurrence it says it is, and is refused once a project becomes required. The distinction rests entirely on a stored `null` surviving as a present key while a never-written field stays absent, so that is asserted against real Mongo rather than assumed: if it ever stopped holding, unscoped runs would silently start being validated against the schedule's current project again. Tests: two against mongodb-memory-server (recorded null vs never-written vs missing row), plus refusal of a recorded-unscoped occurrence under a new requirement and the preserved fallback for a pre-scope one. Schedules e2e green. * 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence The initial-start policy check in `request.js` called `isScheduleLive(..., { policy: true })` without `scheduledFor`, so it fell back to the schedule-level resolution even though the run row already carries the occurrence's recorded scope. An occurrence reserved unscoped, with a pin introduced while its loopback request sat queued, was therefore admitted against the new pin — producing a billed unscoped conversation under a requirement it does not satisfy, from an envelope already built without a project. `scheduledFor` was already in scope there. Passing it makes the initial start and the resume validate the same way: against the destination the occurrence itself recorded.	2026-08-21 11:09:04 -04:00
Danny Avila	c5276fc63d	⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939) ... * feat: Scheduled Chats — agent-centric scheduled runs creating real conversations Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto latest dev, preserving the exact verified tree. History prior to this commit lived on the pre-squash branch; every invariant below survived 25 Codex review rounds plus two external audit rounds (R26) with regression tests that fail without their fixes. Feature: - Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/ permissions (SCHEDULES:USE), interface.schedules availability, per-user limits and capacity slots, timezone-aware cadence with DST-conservative floors and misfire grace. - Engine: single-process claim/fire loop with leases, loopback POST dispatch (signed schedule-fire JWT claims, per-occurrence idempotency key inside the route's clientRequestId charset), overlap/balance/capacity/duplicate skip policies with auto-disable streaks (too_many_failures, insufficient_balance), reconciliation from retained terminal-job evidence, erasure sweep. - Scheduled runs create real conversations through the resumable agents chat path: HITL pauses surface on the card (requires_action), resumes re-apply the fire boundary's admission policy (revision fence, enabled, global kill switch, SCHEDULES:USE, availability) before continuing a billed generation. Correctness invariants (the audit surface): - Settlement discipline: every persistence-producing write happens-before a run's terminal outcome write; Stop/complete/pause race through single-winner terminal CAS claims (dev's TerminalJobClaim substrate) with retained, completedAt-less evidence for scheduled fires plus an owner-intended outcome stamp (scheduleOutcome) the reconciler prefers over re-derived success — round-tripped through the Redis hash mapper. - Swallowed generation failures (client error content parts) classify to error/skipped_balance instead of success on both initial and resumed paths; stale stamps are refreshed evidence-first when persistence plus the Mongo outcome write both fail. - Abort honesty: delivery judged by generation ownership and the CAS's actual from-status; republication escalates to the transport's acknowledged variant; the Stop route settles only genuinely paused runs. - Account deletion: one-way barrier (deletionRequestedAt) with auth-cache tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce of scheduled + interactive work with durable per-stream abort fences (positive-evidence acknowledgement only), owner-side finalization markers for post-terminal billed writes, deferred-deletion sweep (explicitly ensured partial index) that completes cascades autonomously. Remote OpenAI-compatible/ Responses requests are documented as outside the quiesce and tracked in issue 14594. - Store compatibility: finalization markers optional on the legacy IJobStore contract with coherent degradation (registration fails -> synchronous title fallback; count reads 0). * fix: fence terminal response persistence from deletion; retain stale-pause evidence Two blockers from the third external review round. Terminal persistence visible to account deletion: - The finalization marker was registered only when post-terminal TITLE work was possible, but every persistence-owning terminal CAS opens the same window: the claim drops the job out of the active set BEFORE the response save (and the background user-message/convo saves), so a deletion quiesce landing there saw neither an active job nor a marker and could cascade while the admitted request could still recreate messages. Both controllers now register the marker before every persistence-owning claim — the fresh-turn path and the HITL resume — and release it only after their pending saves (and any post-terminal title) have landed, on success and failure paths alike. The TTL bounds a crash. - settleAbortFence no longer clears a complete/error fence while `terminalPersistencePending` is set: terminal at the CAS is not settled while the owner is still persisting. The job facade now surfaces the flag. - The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2 refuses a store without it at configure time, keeping the failure loud and deterministic) while remaining optional on the legacy public IJobStore type for source compatibility. The silent degrade path from the previous round is gone — it was not deletion-safe. Stale-pause recovery retains scheduled evidence: - Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence and the InMemory/Redis stale-pause cleanups — unconditionally stamped `completedAt`, putting a scheduled fire's failed-pause error terminal on the short completed TTL. A Mongo outage longer than that TTL erased the evidence and the reconciler recovered the run as `interrupted` instead of `error`. All three now follow the controller-observed path from the previous round: scheduled jobs omit `completedAt` (retained-evidence TTL) and stamp the error outcome. The PR description now explicitly narrows the deletion guarantee for the remote OpenAI-compatible/Responses paths (tracked in issue 14594). * fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths One consolidated pass over the finalization-marker mechanism, per the fourth external review round. The invariant it establishes: NO persistence-owning terminal CAS runs without a durable, generation-qualified marker covering the window it opens, and every consumer treats a pending terminal as unsettled. - Generation-scoped markers. Entries were keyed (userId, streamId), so a Stop-superseded generation finishing late could clear the marker its replacement registered on the same conversation. Marker fields are now qualified by the generation's createdAt; clears must present the same identity, and an unqualified legacy clear cannot drop a qualified entry. - Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the user's existing hash expires between the two commands (or the process dies there); registration is now a single Lua script carrying both. - Fail closed everywhere. Registration failure (after one retry) now REFUSES the terminal CAS instead of proceeding uncovered: the completion claim throws into the error path, the error path skips completeJob and leaves the job ACTIVE — deletion-visible by itself, recovered by the stale-running reaper — and abortJob returns a new retryable `fence_unavailable` failure the Stop route answers with 503 and the deletion quiesce treats as an unacknowledged stop (fence kept). The previous round's log-and-proceed is gone. - Every terminal path enrolled. abortJob now owns its window (register before the abort CAS, clear in its finally — the Stop route's checkpoint prune and partial save run inside beforePublish, between CAS and publication); the interactive and background generation-error paths register before their completeJob; the resume controller's error finalization registers before its completeJob. A lost or thrown claim releases the marker after pending saves flush instead of holding the user's deletion behind the TTL. - Every pending terminal unsettled. settleAbortFence defers on terminalPersistencePending for ALL statuses — including `aborted`, whose route-side persistence the previous guard missed. Also: a direct Redis regression for scheduled stale-pause retention (the P2 test gap), and the PR description no longer claims to carry every commit. * fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment Fifth external review round; four P1s, handled as the requested consolidated lifecycle-fence pass. - Lease tokens. Marker fields were (streamId, createdAt), shared by every contender on the same generation — completion and Stop, or two racing Stops — so a losing contender's cleanup erased the winner's still-live marker. Every registrant now carries a unique lease token in the field and may only ever clear its own lease; unqualified legacy clears cannot touch qualified entries. - Admission fence. Authentication can pass before the deletion barrier goes up, and the durable createJob is several async steps later — a deletion quiesce in that window saw neither an active job nor a marker and could cascade before the admitted request created its job. The controller now registers an admission lease and THEN rereads the deletion barrier: the ordering guarantees either this request observes the barrier (403, lease released, slot/claim cleanup) or the quiesce observes the lease and defers. Held until createJob is durable; released on every refusal and initialization-error path. Fail closed when the lease itself cannot be registered (503 retryable). - Undelivered-Stop retention. abortJob released its lease in a finally even when delivery AND publication had provably failed — the job reads terminal (invisible to active-set scans), a user Stop writes no durable abort fence, and the remote owner keeps generating and will persist its abort-catch writes whenever the signal finally lands. The lease is now retained in exactly that case, and each resignal attempt heartbeats a fresh lease so the fence outlives the TTL for as long as delivery is still being driven. The abort-winning turn's own loser-side pending saves are additionally fenced in the controller catch (best-effort — those writes are already in flight). - Legacy abort enrollment. abortMiddleware (assistants abort route fallback for non-assistants endpoints) awaited abortJob and then spent usage and saved the stopped response AFTER the abort's lease was released. Both writes now run inside `beforePublish`, between the abort CAS and publication, covered by the same lease as every other abort. Barrier tests, each verified to fail without its fix: same-generation lease isolation (store), racing two-Stop loser cleanup (manager, stale-read forced CAS race), undelivered-Stop lease retention, resignal heartbeat, admission refusal with lease-before-reread ordering plus release-on-durable-create, and legacy-abort persistence inside beforePublish. * fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races Sixth external review round: the remaining P1 interleavings were one structural problem — lease handoffs that were not atomic — resolved as the requested consolidated lifecycle-lease pass. - Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable -job handoff is only atomic against a reader in the OPPOSITE order of the writer: writers hold the lease strictly until the job is active-set visible, so leases-first shows every interleaving either the lease or the job. Jobs-first allowed a request to create its job after the scan and release its lease before the count — hiding both, cascading, and letting the new generation persist into a deleted account. - The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed the moment the owner's AbortController tripped; the stopping side released its lease on that ACK while the owner's asynchronous abort-catch persistence was still ahead. The transport now awaits a manager-installed pre-ACK hook that registers a DETERMINISTIC owner lease (exactly one owner exists per generation, and determinism is what lets the signal-time registrant and the owner's catch-side release agree across processes) before the acknowledgement is persisted or published; an owned same-replica abort bridges to the same lease before tripping its local controller. Both generation-owner catches (fresh turn, resume) release it once their writes land. - A failed replacement handoff no longer orphans the predecessor. The atomic replacement removes it from active storage, and an unconfirmed handoff terminalizes the replacement too — leaving nothing a quiesce could discover while the predecessor's provider may still be generating. Its owner lease is now retained at the point the receipt fails delivery; the owner replica renews it through the pre-ACK fence when the signal finally lands. - Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed holder; live persistence — a stalled save, a long deferred title — must never outlive its own fence. holdUserFinalization registers and renews every minute until released; the controllers' completion/error/admission leases all hold. The undelivered-Stop retention moved to a deterministic `stop` lease that every resignal attempt renews and the first successful one clears (no more opaque leases accumulating to TTL), and a THROWN abort transition releases the contender lease instead of leaking it. - The user-document abort-fence mutations now invalidate the auth user-doc cache, matching every other user-doc write. Barrier tests, each verified to fail without its fix: quiesce lease-scan ordering (plus the observed-lease defer), pre-ACK fence ordering at the transport, owned-abort owner-lease bridging, replacement-handoff predecessor retention, held-lease heartbeat past the TTL, and failed-then-successful resignal reaping the retained stop lease. * fix: one manager-owned owner-lease span across every abort delivery path Seventh external review round; four lifecycle-fence gaps, closed by making the owner-lifecycle lease a single manager-owned, heartbeat-held span. - Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and the transport ACKed even when it failed; a same-replica owned abort likewise swallowed registration failure. The hook now acquires a HELD owner lease (heartbeat until the owner's catch releases it via releaseOwnerLease) and a rejection SUPPRESSES the acknowledgement — the stopping side stays retryable behind its retention lease, and every resignal re-drives the handler. The hook also stopped gating on `job.createdAt === generationId`: during a replacement handoff the store holds the replacement while the abort targets the predecessor, and that gate silently skipped exactly the generation being acknowledged (the store job is owner identity, never a generation gate). A local owned abort acquires the same held lease before tripping its provider; post-CAS the trip cannot be withheld, so acquisition failure downgrades delivery and the retention handoff keeps the user fenced. - Committed-but-lost-reply disambiguation. A thrown abort transition released the contender lease as if nothing had happened, but a Lua CAS can commit and lose its reply — an aborted job invisible to active-set scans whose provider was never signalled, with no fence left. The throw path now re-reads the exact generation: only a job still live under the caller's identity proves no commit; committed or ambiguous outcomes hand the fence to the deterministic stop lease (kept on the contender lease if even that fails) before rethrowing. - Replacement handoff covered end to end. A LOCAL replacement abort acquires the predecessor's held owner lease before the trip (failure reports the receipt undelivered, engaging retention). Failed-handoff retention is no longer a swallowed one-shot: it heartbeats with the durable acknowledgement proof as its renewal predicate — acquisition failures keep retrying for as long as the fence is needed, and the retainer stands down (without clearing the shared field) once the remote owner ACKs and thereby holds its own lease. - A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the retained stop lease, and keeps the stop lease when that handoff fails. Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply retention with its provably-uncommitted counterpart, local-resignal owner handoff ordering, pre-ACK owner lease held past the store TTL until release (and provably stopped after), and failed-handoff retention retrying on its heartbeat — verified fail-before/pass-after by stashing the fixes. * fix: finish scheduled chat lifecycle hardening * fix: close scheduled chat review follow-ups * fix: generation-fence abort recovery evidence * test: wait for settled approval tool output * test: preserve scheduled init reconciliation option * refactor: rebuild scheduled chats on durable agent triggers * test: reset MCP cache mock between cases * test: isolate scheduler startup in server specs * fix: harden scheduled run lifecycle * fix: normalize schedule capacity conflicts * test: type schedule collision fixture * fix: re-fence scheduled resume and expiry * fix: fence scheduled resume handoffs * fix: release superseded manual schedule leases * fix: release failed run-now claims * fix: release superseded engine claims * fix: repair schedule dialog interaction and rework its form The agent picker was unusable: ControlCombobox portals its popover to the body by default, which lands it outside the dialog's Radix focus trap. Clicks passed through it, it could not be tabbed into, and the trap fighting Ariakit for focus locked the page up on selection. The prop is documented for exactly this case — pass `portal={false}` and give the dialog `overflow-visible`, as ProjectButton already does. The time and day dropdowns defaulted the same way. Alongside that: - Extract the agent builder's instructions editor (special-variable menu plus expand-to-fullscreen) into a controlled `VariableEditor` and use it for the schedule prompt. Insertions now route through `onChange`, so react-hook-form sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue` that skipped dirty tracking would have dropped an inserted variable silently. - Replace the hand-rolled frequency buttons with the shared `Radio`. They marked the selection with `bg-surface-hover` on an outline button whose hover is the same token, so the selected option was indistinguishable from a hovered one; `Radio` is also a real radiogroup rather than four `aria-pressed` toggles. - Wrap the fields in a real `<form>` and associate the footer button by id, so Enter submits. Group the frequency, day and time controls in fieldsets. - Add placeholders for name and prompt, match the textarea fill to the other fields, and label the hourly case as minutes past the hour. - Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits without scrolling on desktop. - Move scheduled chats below skills and above prompts in the side nav. The new dialog spec fails when the portal fix is reverted. * test: cover scheduled and subagent deletion drains * refactor: own the form-control appearance in the client primitives Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS` restated the `Input` primitive's border, radius, height and background so it could be pasted onto the schedule dropdowns, leaving those controls with no connection to the primitive they were imitating. Move that appearance into `packages/client/src/components/Field.ts` as the single source `Input` and `Textarea` now compose, and give `Dropdown` and `ControlCombobox` a `variant="field"` that applies it. The schedule dialog passes the variant and carries no class strings of its own. This also repairs a break the dev merge would otherwise have introduced: the newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the old pasted classes would have landed on the wrapper and left the triggers unstyled. The semantic-token guard now watches the shared module and asserts each primitive still composes it, which covers more than the two files it read before. * fix: keep an explicit schedules disable from becoming an opt-in `use` is two things at once for a dual-purpose runtime interface field: a permission bit, which DB overrides strip, and the runtime disable signal that `getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an object, and `getLimits` treats any object without `use: false` as enabled — so an admin override written to stop scheduled billing for a role or user started it instead. Collapse an explicit disable to the boolean form before the strip, on both paths that accept it: the `interface.schedules` field patch, which admitted the object wholesale because bare runtime paths deliberately bypass the permission gate, and the overrides merge, which reached the composite-field branch and kept `maxPerUser`. Objects that only narrow limits are untouched, so a principal can still be given a smaller cap. Both regressions fail without the normalizer. * fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep Slice 1 (thread r3804518381): route the existing-null balance initialization through a { user, tokenCredits: null } compare-and-set instead of a blind $set, so a concurrent initializer/charge landing between the preflight read and the write is never handed back its spent starting balance. On a CAS miss the preflight re-reads the winner. The absent-record $setOnInsert path and the credited-record refill-config sync are unchanged. Adds the initializeNullBalance adapter (no upsert) and regression coverage for winner/miss/sync cases. Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action` lastRun projection atomically with the configRevision bump. Any pause present at edit time was projected under the pre-edit revision and can never be replaced by its own revision-fenced terminal outcome — a disabling edit would strand the card on "Needs approval" forever. Implemented as classic-operator CAS branches (DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL being the pause so a terminal outcome or newer occurrence that races in is preserved. Terminal history survives untouched. Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the schedule runtime facade and start it in every clustered (experimental) worker after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so a hidden prompt cannot outlive its drain when the delete/erase-on-settle attempts miss. It arms nothing else and never infers owner death from a process-local missing job (isTopologySafeToArm gates that). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): Wave B.1 — reversible account-deletion schedule suspension Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`, disabled it, and cleared nextRunAt destructively. When a later cascade step (or the drain itself) failed, the controller cancelled the user-deletion fence — restoring the user — but the schedules stayed `deleting` and were erased by the sweep, silently losing all of a live user's scheduled prompts. Replace the destructive marking with a REVERSIBLE, token-fenced suspension: - suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is not erasure-eligible. Snapshotting reads then bulkWrites (a classic update cannot copy field values under DocumentDB), fenced per row so an already- suspended/soft-deleted/edited row is left alone; idempotent per token. - restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and re-arming only rows still carrying the exact attempt token and not independently deleted — so an owner-deleted or newer-attempt-suspended schedule is never resurrected. - deleteUserController generates the attempt token, passes it to quiesce, and on any failure that cancels the user-deletion fence restores the suspended rows. A successful deletion hard-deletes them (and their snapshots) via the existing cascade and never restores. Adds a `deletionSuspension` embedded field (excluded from the wire schedule), data-method regression tests (suspend/restore/fence/idempotency/no-resurrect), and controller tests (restore on drain-false and post-quiesce cascade failure, no restore on success). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop', getScheduleRunAbortState, markRunAbortPersisted) already existed but production only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job to `aborted` and then persisted its partial message + checkpoint inside `beforePublish`, without ever stamping the schedule Stop or acknowledging it — so reconciliation, the generation owner, or a concurrent schedule/account deletion could terminalize the run and release its capacity (and erase data) mid-write. Wire the request -> persist -> acknowledge -> settle barrier through the schedule runtime (the route never touches raw Mongo): - Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service. - The abort route stamps the Stop BEFORE signalling abortJob (a serialized 'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort), acknowledges only after beforePublish persistence succeeds, and on a persistence failure leaves the barrier unresolved so the run stays preserved (client retries; stale-owner timeout is the bounded recovery). A failed abort releases the stamp it placed so a replacement/retry is never blocked through its predecessor. - recordScheduleOutcome (the owner settlement path) now waits, bounded, for the Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker proceeds immediately. The paused Stop settles only after its own ack. Adds service-layer barrier tests and route-level ordering/persistence-failure/ in-progress tests; the data-layer serialization is already covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the independently-found long-Retry-After race. fireSchedule reserves a `started` run and a global capacity slot BEFORE the durable trigger delivery reaches the chat route, where an interactive limiter (manual Run Now), PII, or moderation can reject it before any generation job exists — dead-lettering the delivery while the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted. And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled and have its capacity released, then fire anyway. Translate durable delivery state into the schedule outcome: - Store the deterministic trigger deliveryKey on the ScheduleRun reservation (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it). - Add a getTriggerDelivery engine dep (wired to the merged trigger service's getDelivery) that reads the durable delivery by key. - Schedule reconciliation, for a jobless `started` run: staging/pending/leased → admission is live, never orphan; dead → record `error` from the durable lastError and release capacity promptly (no 30-minute wait), through the ordinary outcome/auto-disable path; succeeded or no record → the existing legacy orphan policy (interrupted only past the cutoff); a delivery lookup failure defers rather than orphaning a possibly-live delivery. Limiter/PII/ moderation middleware writes no schedule state. Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/ lookup-failure) and a fire test that the reservation's deliveryKey equals the enqueued delivery's idempotency key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a failure, and because later sweeps enumerate only `requires_action` jobs, the now- aborted job was never offered again. In the clustered entrypoint (no schedule reconciler) the ScheduleRun stayed `requires_action` and its retained job persisted indefinitely. Make the host lifecycle work durable rather than schedule-specific: - Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host adapter is installed. - Retain and index such jobs: both stores keep them out of terminal reaping and expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded) TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak. - The manager clears the marker only after the adapter acknowledges success, fenced by generation identity (clearTerminalHostAction), so a replacement generation can neither clear nor execute its predecessor's action. - cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions across restarts and replicas and retries the idempotent hook; the relay only re-invokes while the marker is unacknowledged, so a successful ack prevents duplicate work. Store-won expiry marks it too, so a loser-replica relay still crosses the hook. - Terminal SSE notification continues regardless of host-hook outcome. Covers in-memory behavior (retry after failure, restart/other-replica retry, ack prevents duplicates, identity fence, terminal notification on failure, no marker accumulation for non-scheduled jobs) and updates the Redis cluster-membership contract test for the new index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * style(schedules): satisfy import sorting in fire.ts and fire.spec.ts CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey). Applied scripts/sort-imports.mts to exactly those files — imports-only reordering, no behavior change. Other files reported by a repo-wide check are pre-existing on dev and deliberately left untouched so this PR is not widened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): repair the deletion CLI and order restore before the fence release Addresses three findings from the fresh Codex review. P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which Wave B removed in favor of suspendUserSchedulesForDeletion. The file is @ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests caught it; the real CLI would throw a TypeError before deleting anything and then only unwind the fence. The CLI now uses the tokenized protocol: it mints a suspension token, suspends with it, and restores that exact attempt's rows in its finally block when the deletion does not commit. Its spec mocks the real methods, so the breakage can no longer hide. P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE restoring schedules. That fence is what refuses new schedule writes/claims, so the gap let an owner PATCH edit a still-suspended row and have its enabled/next-run state overwritten by the older snapshot, and let a second deletion attempt re-suspend under a new token — making the first restore a no-op and stranding the disabled snapshot permanently. Restore now runs first, while writes are still fenced. Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by an earlier attempt, instead of re-capturing the row's current (already-suspended) state. Without this, an attempt that died before restoring would have its successor snapshot "disabled, no next run" and permanently strand the schedule. Tests: CLI restore-before-fence ordering and no-restore-on-success; the same ordering assertion on both controller post-quiesce failure paths; a data-method regression that a second attempt adopts the abandoned snapshot and restores the original enabled/next-run state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): converge dead deliveries in topology-safe maintenance Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it runs erasure-only maintenance. A delivery queued before a restart into clustered mode and then rejected before generation creation (interactive limiter, PII, moderation) dead-letters while its ScheduleRun stays `started`, holding a global capacity slot indefinitely for an ordinary non-deleting schedule. Add a dead-delivery convergence pass to the erasure sweep, so every topology that runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is therefore safe where absence-based reconciliation is not: a `dead` delivery is durable shared state proving no generation owns the reservation. It settles only when the job is confirmed absent or identity-mismatched (an identity-matched job still owns the run), defers on an unknown job lookup, on an in-flight abort, and on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey, and applies a short grace so an accepted delivery still creating its generation is never settled mid-handoff. Auto-disable policy is deliberately left to the armed engine; this path records the failure and frees the slot. Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint already starts this sweep, so the convergence arrives through the existing initializer and the shared-file footprint stays as-is. Tests: settles a dead delivery as error under an explicitly UNSAFE topology, leaves live deliveries alone, never settles under an identity-matched running generation (delivery is not even consulted), defers an in-flight abort, and ignores a reservation with no deliveryKey. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(stream): refresh host-action retention on each retry attempt Codex finding: unacknowledged terminal host-action evidence was capped at the 24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let the Redis key — and its pending marker — expire with no generation-fenced acknowledgement, stranding the ScheduleRun where no reconciler is armed. Measure retention from the LAST retry rather than from the terminal transition: enumerating a pending host action IS the retry attempt, so both stores refresh its retention as they hand it to the hook (Redis re-EXPIREs the job key; the in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence therefore survives as long as some replica is still actively retrying, while a deployment that stops sweeping entirely still lets it age out — so this does not reintroduce the unbounded leak the cap existed to prevent. Test: after a failed hook, a later cleanup pass keeps the marker pending and moves its retention basis forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): defer settlement when the Stop barrier times out Codex finding: waitForStopPersistence returned after its 5s poll budget even when the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went straight on to terminalize the run — releasing its capacity, deletion, and erasure barriers while beforePublish may still have been writing. Slow checkpoint cleanup is indistinguishable from a dead route on that signal, so the timeout was being treated as if the barrier had been satisfied. The poll budget now means "undecided", not "clear". On timeout with a fresh, unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false without recording, leaving the run active/preserved. Settlement then happens either when the route acknowledges, or once the existing stale-owner cutoff (ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop already treats as clear-to-settle. Callers with durable retry (the approval-expiry host action, reconciliation) re-drive it, so a deferral converges rather than stranding the run. Test: a fresh Stop that never acknowledges within the budget reports not-settled and records no outcome. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): require definite delivery failure, extract its message, converge deferred Stops Third Codex round. Three findings in code from this closeout, plus one pre-existing P1 that is a one-line operator-safety fix. P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a request was rejected: the trigger host marks response timeouts and invalid success responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries are exhausted. The erasure sweep treated every dead letter as positive evidence, so an ambiguous one sitting over a generation a peer had accepted could terminalize the run and release its capacity mid-flight. It now settles only on a DEFINITE rejection, unless job absence is deployment-authoritative (safe topology), where the confirmed-absent job is itself the evidence. P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate- types rule), so both the sweep and the engine reconciler passed the object into the String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row catch would swallow it, and the run would keep its global capacity slot. The dep type now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass `.message`. Re-typing immediately surfaced a stale test that had asserted a string. P1 (pre-existing) — the base-config global stop is honored in `getLimits` via `isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and deepMerge turns base `{ use: false }` plus a principal override of `true` into `{ use: true }`, so the literal check reported the feature enabled and Run Now dispatched straight through fireSchedule, bypassing the operator's emergency stop. Now the same predicate the engine gate already uses. P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is match-guarded and idempotent, so an owner that already settled makes it a no-op. The abort route passes it for a running generation; a paused job still settles explicitly. Tests: ambiguous dead letters refused under unsafe topology but settled when absence is authoritative, definite rejections settled either way, the failure message carried through, and the abort route's re-drive present for running / absent for paused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): converge terminal runs in clustered workers, retry suspension restore Fourth Codex round on 74f7af8. Both findings are in code from this closeout. P2 - a clustered worker never settled a run whose generation finished but whose outcome write failed. `recordScheduleOutcome` retries three times and then returns false; the owner honors that by PRESERVING the terminal job as the only surviving evidence, and the armed engine's reconciler replays exactly that. The clustered entrypoint arms no engine, and its sweep only covered deleting schedules (settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL capacity slot, and kept a preserved job that carries no `completedAt` and is therefore invisible to the store's finished-job sweep: both leaked until store expiry. The live-schedule pass now also converges from a retained terminal job, mirroring the reconciler branch it stands in for: honor the owner's stamped outcome over the generic status (so a balance refusal still walks its streak rather than resetting it), clear the reserved conversationId when the generation never emitted its created event, and delete the retained job only AFTER the outcome write is durable. This stays positive-evidence-only and safe in every topology. Presence, not absence, is the evidence: an identity-matched job is authoritative wherever it is observed - a shared store shows the real generation, a process-local store can only be showing this process's own - which is why it needs no canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run whose owner may still be persisting. Both cases now share one pass over one window rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the engine's mapping instead of duplicating it (and stays independent of the engine). P2 - a failed restore stranded a live account's schedules. Cancelling an account deletion restores the suspended rows while the deletion fence is still armed and then releases that fence; nothing re-drives the restore afterwards, so one transient write failure left the user with silently disabled, next-run-less schedules. The restore is now retried at the single choke point both the HTTP controller and the CLI share. Retrying is safe because each attempt re-reads only the rows STILL carrying the token: a partially-applied unordered write converges on exactly the stragglers, and a fully-applied one finds nothing. The fence is still released when every attempt fails, deliberately: retaining it would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion report `in_progress` forever, blocking the retry that is the convergence path (a later attempt adopts this snapshot, so its cancel restores these exact rows). Both callers now log the user id and suspension token so the state stays recoverable by hand if that never happens. Tests: retained terminal job settled and its evidence released, stamped outcome preferred over the generic status, stamped failure reason carried, reserved conversationId cleared for a never-created conversation, identity-mismatched terminal job ignored, abort-in-flight deferred; restore retried past a transient write failure and converging on a partially applied restore. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs Two of the four findings I flagged as pre-existing to my closeout commits. Both are in code this PR introduces, so merging would have shipped them; both are the same capacity/evidence-leak class the last four rounds have been closing. The remaining two (slotless rows escaping the per-user cap, and the non-rotating `started` reconciliation bucket) are genuinely latent and deliberately left alone. An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is what moves a run row off `started`; `recordScheduleOutcome` already retries it, but the request controller discarded the result, so three failed attempts were dropped silently. The armed engine's reconciler replays that state, but the clustered sweep did not: a paused job is not terminal, so the retained-job path returned without settling, and the dead-delivery path never inspects an identity-matched job at all. The row stayed `started` with no cutoff that would ever clear it. The call site now surfaces the failure, and the sweep converges it, mirroring the reconciler's pause branch: project `requires_action` (which frees the slot) but do NOT release the job's evidence — unlike a terminal job it is still live, awaiting an approval. The resume hand-off fence still defers, so re-projecting cannot release a slot a continuation just claimed. The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only because its owner crashed before bookkeeping — which is also before it could release the job it retained for exactly this recovery. The active-run pass clears its own; this one never did, and a preserved job is deliberately kept WITHOUT `completedAt` so the store's finished-job sweep cannot reap it early, so nothing else ever would. It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when no job is retained, and skipped entirely when the replay itself failed, since the retained job is the only surviving evidence in that case. Tests: pause projected and the slot freed while the live job's evidence is kept, paused job deferred during a resume hand-off, retained job released once replayed bookkeeping is durable, and retained job kept when the replay fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): fence the clustered pause replay against a concurrent resume claim Codex round 5, on code I added in f735341. The finding is correct and the exposure is one I introduced. The pause replay lives in a sweep that runs in EVERY clustered replica, so several sweepers can observe the same unprojected pause. Its guards were all derived from an in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in `started`/`requires_action` with no fence of its own. The race: sweeper A projects the pause and frees the slot; the owner's approval then claims a fresh one (markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write); sweeper B, still holding the pre-projection snapshot, passes its hand-off check and replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is already running. The run reverts to `requires_action` while its generation proceeds outside global capacity. The engine's reconciler makes the same call and has the same snapshot-derived guard, but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first thing to run this transition in parallel. Left the engine alone rather than widening the change: its `requires_action` re-affirmation is deliberate and single-writer. Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional `requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause filter, and the sweep sets it. Because the stamp is written in the SAME update that moves the row to `started`, its absence is atomic proof no resume owns the row. The flag is deliberately NOT set by the generation owner: its own re-pause legitimately clears the stamp as the hand-off's completion signal. The fence cannot block the recovery it exists to enable: markRunResumeClaimed only matches `requires_action`, so a genuinely stuck `started` row can carry no resume claim — it is in fact blocking its own approval until this replay frees it. Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and claim stamp intact, while a stuck row with no claim still recovers; and the sweep is asserted to send the fence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq * fix(schedules): fence stale resume claims by age, not existence Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is the mirror image of the one it fixed. The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is abandoned: a worker that dies after markRunResumeClaimed takes the row to `started` and stamps resumeClaimedAt — but before the continuation resumes or releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound the sweep correctly stops deferring and tries to recover the row, but the write rejected it purely because the field still existed. The row stayed `started` holding its global capacity slot, and its approval was unresumable for good, since markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state this replay exists to clear, so the fence had reintroduced it for the crashed-resume case. `requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the filter matches a row with no claim OR a claim older than that cutoff. The sweep passes the SAME bound its in-flight check uses, so the two can no longer disagree. A genuinely racing claim is by construction fresh — it is created after the sweeper's snapshot — so the race from round 5 stays closed. Tests: a row whose resume claim was abandoned by a dead worker now recovers (status, slot and stamp all cleared), alongside the existing two — a fresh claim still repels a stale-snapshot replay, and an unclaimed stuck row still recovers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-20 11:51:30 -04:00