LibreChat/api/server/services/Schedules/index.js
Danny Avila 696bcb6c1e
🏦 fix: Reserve Credits for In-Flight Requests (#15879)
* 💳 fix: Reserve Balance for In-Flight Requests

Admit each balance-checked request against the credits that other
in-flight requests have not already reserved, and hold its prompt cost
on the balance record until the turn settles. Apply a due auto-refill in
the same fenced write so one refill window credits exactly once.

- data-schemas: reserveBalance / releaseBalanceReservation replace
  createAutoRefillTransaction; the write is a compare-and-swap on
  tokenCredits, reservations and lastRefill (no pipeline updates)
- balance.reservations is select:false so existing reads never see it
- checkBalance returns a reservation handle; BaseClient and the
  assistants controllers release it once the turn settles
- balance.reservationTtlMs bounds how long an unreleased reservation
  counts (default 30 minutes)

* 🧮 refactor: Admit Reservations Without Contention and Record Refills Durably

- admission is a conditional $push/$inc against a reservedCredits total,
  so concurrent admissions of a funded balance commit on their first write
- expired reservations are pruned per element with their credits
- auto-refill is fenced on every setting its decision reads and leaves a
  pendingRefill ledger marker in the same write; any later read records the
  transaction under the fixed id and clears the marker
- a missing record is created with $setOnInsert inside reserveBalance
- reservation lifecycle moves to packages/api (createBalanceReservations,
  withBalanceReservations); BaseClient releases once usage is recorded

* 🪪 fix: Resolve Each User to One Balance Record

- balance records a creator inserts are keyed by the user id, so creators
  racing on a missing record converge on one document; every balance read
  and write resolves a user to their oldest record, so existing duplicates
  stay inert
- upsertBalanceFields takes insert-only fields; the balance-config
  middleware, schedules and lazy initialization pass the starting credit
  through them so it never overwrites a record another writer created
- the auto-refill write is also fenced on reservedCredits
- the scheduled-chat balance pre-skip counts reserved credits as unavailable

* ⏳ fix: Count Only Unexpired Reservations in Scheduled Balance Checks

findBalanceByUser with includeReservedCredits now totals the reservations
that have not expired, so a reservation left by a crashed request stops
pre-skipping scheduled chats at its expiry instead of when a later
reservation write prunes it.

* 🔁 fix: Renew Balance Reservations While Their Request Runs

A reservation is renewed every half TTL until it is released, so only a
reservation whose process stopped expires. Pruning removes a reservation
only while it is still expired at the write, so one renewed after the
pruning read survives.

* 🛟 fix: Keep Reservations Alive Through Renewal Failures and Lost Acks

- renewal reschedules itself: every half TTL, capped to the timer range,
  and a failed renewal retries within a tenth of that window
- an admission write that errors removes its possibly committed
  reservation before rethrowing

* 🛑 fix: Hold a Stopped Turn's Reservation Until Its Charge Lands

- an aborted turn lets its reservations lapse after a minute (or the TTL)
  instead of releasing them, so the Stop request's charge lands before the
  credits are admitted again
- balance.reservationTtlMs has a 10 second minimum, enforced in the schema
  and at runtime, so renewal stays bounded

* ↩️ revert: Release a Stopped Turn's Reservation When the Turn Settles

An aborted turn releases its reservations when it settles, the same as
every other exit, instead of holding them for a fixed lapse window that
refused funded requests after every Stop. The minimum reservation TTL
stays.

* ⏸️ fix: Hold Assistants Reservations Through Background Run Retries

An in-progress Assistants run continues in the background after the
handler responds; the turn's reservations now stay held until that run
settles (createBalanceReservations().holdUntil).

* 🔁 fix: Apply an Auto-Refill Once per Admission

A refill interval that is due again the moment it is applied, such as a zero
interval, let one admission refill repeatedly until the retry bound. Each
admission now applies at most one refill, as the previous check did.

* 🧹 fix: Prune Expired Reservations in One Write

An admission that found expired reservations issued one write per hold, so a
backlog left by a stopped process fanned out into as many concurrent writes.
The prune is now a single write fenced on every hold it removes; a hold renewed
after the read makes the write miss and the admission re-read.

* ⏱️ fix: Renew Reservations From Their Stored Expiry

The next renewal was scheduled half a TTL after the previous write settled,
and the first one half a TTL after the admission returned, so a slow write
could let a live hold expire before its renewal. Renewals are now scheduled
from the expiry the last write stored.

Expired holds are pruned as one bulk write of per-hold fenced updates, so a
hold that changed after the read is skipped while the rest are removed, and
the admission no longer spends a retry on the whole prune.
2026-09-13 12:53:19 -04:00

107 lines
4.7 KiB
JavaScript

let service;
/**
* Build the schedule service on first use. Controllers import this facade in many
* focused test/runtime paths that never use scheduling; eagerly loading Config and
* the trigger worker made those paths initialize two unrelated service graphs just
* by requiring a controller.
*/
function getService() {
if (service != null) {
return service;
}
const mongoose = require('mongoose');
const { createSchedulesService } = require('@librechat/api');
const { getAppConfig } = require('~/server/services/Config/app');
const {
enqueueAgentTrigger,
getAgentTriggerDelivery,
} = require('~/server/services/Agents/triggers');
const { resolveAgentFireAccess } = require('./access');
const methods = require('~/models');
const isUserDeleting = async (userId) => !(await methods.isAgentTriggerPrincipalActive(userId));
service = createSchedulesService({
preflightMCP: require('./mcp'),
methods,
getAppConfig,
findUserById: (userId) =>
mongoose.models.User.findById(userId).select('_id tenantId role').lean(),
findBalance: (userId) => methods.findBalanceByUser(userId, { includeReservedCredits: true }),
upsertBalance: (userId, { set, setOnInsert }) =>
methods.upsertBalanceFields(userId, set ?? {}, setOnInsert ?? {}),
// Compare-and-set: only initialize an existing record while its credit is still null.
// No upsert — a CAS miss must re-read the winner, not insert a fresh balance. `null`
// in the filter also matches a legacy record whose `tokenCredits` field is absent.
initializeNullBalance: (userId, { tokenCredits, sync }) =>
mongoose.models.Balance.findOneAndUpdate(
{ user: userId, tokenCredits: null },
{ $set: { tokenCredits, ...(sync && Object.keys(sync).length > 0 ? sync : {}) } },
{ new: true, sort: { _id: 1 } },
).lean(),
enqueueAgentTrigger,
// Reconciliation reads the durable delivery to tell a still-live admission (a
// deferred Retry-After) or a dead-letter apart from a genuinely orphaned run.
getTriggerDelivery: getAgentTriggerDelivery,
resolveAgentFireAccess,
// Chat projects are user-owned, so this scoped read is both the existence and the
// authorization check the fire-time destination precheck needs.
getChatProject: (userId, projectId) => methods.getChatProject(userId, projectId),
// Reuse the merged trigger/deletion admission fence. A schedule fire and every
// generic trigger now fail closed on the same durable principal state.
isUserDeleting,
});
return service;
}
const invoke =
(method) =>
(...args) =>
getService()[method](...args);
/** Host adapter for the generic generation runtime's approval-expiry seam. The
* callback is intentionally idempotent: a replica relay or schedule reconciliation
* may re-drive the same retained terminal evidence after a transient failure. */
async function recordExpiredScheduleApproval(streamId, job) {
if (!job?.scheduleId || !job?.scheduledFor) {
return;
}
const recorded = await getService().recordScheduleOutcome({
scheduleId: job.scheduleId,
scheduledFor: job.scheduledFor,
streamId,
jobCreatedAt: job.createdAt,
status: 'interrupted',
conversationId: job.conversationId ?? streamId,
error: 'Approval expired before a decision was made',
});
if (!recorded) {
throw new Error(`Failed to settle expired scheduled approval ${job.scheduleId}`);
}
}
module.exports = {
getLimits: invoke('getLimits'),
fireScheduleNow: invoke('fireScheduleNow'),
recordScheduleOutcome: invoke('recordScheduleOutcome'),
beginScheduledStop: invoke('beginScheduledStop'),
acknowledgeScheduledStopPersistence: invoke('acknowledgeScheduledStopPersistence'),
claimScheduleResume: invoke('claimScheduleResume'),
releaseScheduleResumeClaim: invoke('releaseScheduleResumeClaim'),
finalizeScheduleResumeClaim: invoke('finalizeScheduleResumeClaim'),
releaseScheduleResumeFence: invoke('releaseScheduleResumeFence'),
isScheduleLive: invoke('isScheduleLive'),
deleteScheduleForOwner: invoke('deleteScheduleForOwner'),
quiesceUserSchedules: invoke('quiesceUserSchedules'),
restoreUserSchedulesFromDeletion: invoke('restoreUserSchedulesFromDeletion'),
initializeScheduleEngine: invoke('initializeScheduleEngine'),
initializeScheduleErasureSweep: invoke('initializeScheduleErasureSweep'),
recordExpiredScheduleApproval,
isUserDeleting: async (userId) => {
const methods = require('~/models');
return !(await methods.isAgentTriggerPrincipalActive(userId));
},
// Identity-fenced delete for a retained job whose run has since been settled
// inline; reconciliation only reaches jobs whose run is still active.
clearScheduledJob: (...args) => getService().engineDeps.clearReconciledJob(...args),
};