fix: Codex round 6 — index-gate engine, global capacity, misfire skip, ambiguous reconcilable

- P1: do NOT arm the engine if ensureScheduleIndexes fails (fires need the unique
  idempotency index); leave scheduler disabled instead of firing without it
- Count capacity in system (global) scope, not the owner's tenant, so multiple
  tenants can't collectively exceed fireConcurrency
- Record schedule error on the accepted-then-init-failure path (before completeJob)
- Reconcile a retained 'complete' job for a still-'started' run as success
- Misfire skip-forward: occurrences overdue past a 15m grace advance without
  firing (no stale/bursty chats on restart)
- Ambiguous loopback failures stay reconcilable ('started'), not terminalized
  'interrupted' (so a late real completion can still finalize them)
- Record scheduled success BEFORE completeJob deletes the job (no success→interrupted
  mislabel on a crash in the window)
Note: e2e nav-id finding already fixed in f8b27693b (role-based selector).
This commit is contained in:
Danny Avila 2026-07-21 20:41:17 -04:00
parent f8b27693b6
commit 1e61adebb6
5 changed files with 77 additions and 15 deletions

View file

@ -938,7 +938,10 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
});
await GenerationJobManager.emitDone(streamId, finalEvent);
GenerationJobManager.completeJob(streamId);
// Record the schedule success BEFORE completeJob: the default job
// manager deletes completed jobs immediately, so if this write ran
// after and the process died in between, reconciliation would see no
// job for the still-`started` run and mislabel a success as interrupted.
if (scheduleId) {
await recordScheduleOutcome({
scheduleId,
@ -947,6 +950,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
conversationId: conversation?.conversationId,
});
}
GenerationJobManager.completeJob(streamId);
await finishResumableRequest(req, userId);
} else {
const finalEvent = {
@ -1105,6 +1109,19 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
// JSON already sent, emit error to stream so client can receive it
await GenerationJobManager.emitError(streamId, error.message || 'Failed to start generation');
}
// A scheduled fire whose run row is already `started` (inserted before the
// early 200) must be terminalized here — an init failure after acceptance
// would otherwise leave it running until orphan reconciliation. Before
// completeJob deletes the job (same ordering rationale as the success path).
if (scheduleId) {
await recordScheduleOutcome({
scheduleId,
scheduledFor,
status: 'error',
error: error.message,
conversationId: streamId,
});
}
// Finalize THIS failed job before releasing the idempotency claim. Releasing first would
// let the client's retry win the same key and createJob() the same streamId while we are
// still here — and completeJob() is not guarded by the original createdAt, so it would