mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-18 22:14:27 +00:00
* 💳 fix: Reserve Balance for In-Flight Requests Admit each balance-checked request against the credits that other in-flight requests have not already reserved, and hold its prompt cost on the balance record until the turn settles. Apply a due auto-refill in the same fenced write so one refill window credits exactly once. - data-schemas: reserveBalance / releaseBalanceReservation replace createAutoRefillTransaction; the write is a compare-and-swap on tokenCredits, reservations and lastRefill (no pipeline updates) - balance.reservations is select:false so existing reads never see it - checkBalance returns a reservation handle; BaseClient and the assistants controllers release it once the turn settles - balance.reservationTtlMs bounds how long an unreleased reservation counts (default 30 minutes) * 🧮 refactor: Admit Reservations Without Contention and Record Refills Durably - admission is a conditional $push/$inc against a reservedCredits total, so concurrent admissions of a funded balance commit on their first write - expired reservations are pruned per element with their credits - auto-refill is fenced on every setting its decision reads and leaves a pendingRefill ledger marker in the same write; any later read records the transaction under the fixed id and clears the marker - a missing record is created with $setOnInsert inside reserveBalance - reservation lifecycle moves to packages/api (createBalanceReservations, withBalanceReservations); BaseClient releases once usage is recorded * 🪪 fix: Resolve Each User to One Balance Record - balance records a creator inserts are keyed by the user id, so creators racing on a missing record converge on one document; every balance read and write resolves a user to their oldest record, so existing duplicates stay inert - upsertBalanceFields takes insert-only fields; the balance-config middleware, schedules and lazy initialization pass the starting credit through them so it never overwrites a record another writer created - the auto-refill write is also fenced on reservedCredits - the scheduled-chat balance pre-skip counts reserved credits as unavailable * ⏳ fix: Count Only Unexpired Reservations in Scheduled Balance Checks findBalanceByUser with includeReservedCredits now totals the reservations that have not expired, so a reservation left by a crashed request stops pre-skipping scheduled chats at its expiry instead of when a later reservation write prunes it. * 🔁 fix: Renew Balance Reservations While Their Request Runs A reservation is renewed every half TTL until it is released, so only a reservation whose process stopped expires. Pruning removes a reservation only while it is still expired at the write, so one renewed after the pruning read survives. * 🛟 fix: Keep Reservations Alive Through Renewal Failures and Lost Acks - renewal reschedules itself: every half TTL, capped to the timer range, and a failed renewal retries within a tenth of that window - an admission write that errors removes its possibly committed reservation before rethrowing * 🛑 fix: Hold a Stopped Turn's Reservation Until Its Charge Lands - an aborted turn lets its reservations lapse after a minute (or the TTL) instead of releasing them, so the Stop request's charge lands before the credits are admitted again - balance.reservationTtlMs has a 10 second minimum, enforced in the schema and at runtime, so renewal stays bounded * ↩️ revert: Release a Stopped Turn's Reservation When the Turn Settles An aborted turn releases its reservations when it settles, the same as every other exit, instead of holding them for a fixed lapse window that refused funded requests after every Stop. The minimum reservation TTL stays. * ⏸️ fix: Hold Assistants Reservations Through Background Run Retries An in-progress Assistants run continues in the background after the handler responds; the turn's reservations now stay held until that run settles (createBalanceReservations().holdUntil). * 🔁 fix: Apply an Auto-Refill Once per Admission A refill interval that is due again the moment it is applied, such as a zero interval, let one admission refill repeatedly until the retry bound. Each admission now applies at most one refill, as the previous check did. * 🧹 fix: Prune Expired Reservations in One Write An admission that found expired reservations issued one write per hold, so a backlog left by a stopped process fanned out into as many concurrent writes. The prune is now a single write fenced on every hold it removes; a hold renewed after the read makes the write miss and the admission re-read. * ⏱️ fix: Renew Reservations From Their Stored Expiry The next renewal was scheduled half a TTL after the previous write settled, and the first one half a TTL after the admission returned, so a slow write could let a live hold expire before its renewal. Renewals are now scheduled from the expiry the last write stored. Expired holds are pruned as one bulk write of per-hold fenced updates, so a hold that changed after the read is skipped while the rest are removed, and the admission no longer spends a retry on the whole prune.
107 lines
4.7 KiB
JavaScript
107 lines
4.7 KiB
JavaScript
let service;
|
|
|
|
/**
|
|
* Build the schedule service on first use. Controllers import this facade in many
|
|
* focused test/runtime paths that never use scheduling; eagerly loading Config and
|
|
* the trigger worker made those paths initialize two unrelated service graphs just
|
|
* by requiring a controller.
|
|
*/
|
|
function getService() {
|
|
if (service != null) {
|
|
return service;
|
|
}
|
|
const mongoose = require('mongoose');
|
|
const { createSchedulesService } = require('@librechat/api');
|
|
const { getAppConfig } = require('~/server/services/Config/app');
|
|
const {
|
|
enqueueAgentTrigger,
|
|
getAgentTriggerDelivery,
|
|
} = require('~/server/services/Agents/triggers');
|
|
const { resolveAgentFireAccess } = require('./access');
|
|
const methods = require('~/models');
|
|
const isUserDeleting = async (userId) => !(await methods.isAgentTriggerPrincipalActive(userId));
|
|
|
|
service = createSchedulesService({
|
|
preflightMCP: require('./mcp'),
|
|
methods,
|
|
getAppConfig,
|
|
findUserById: (userId) =>
|
|
mongoose.models.User.findById(userId).select('_id tenantId role').lean(),
|
|
findBalance: (userId) => methods.findBalanceByUser(userId, { includeReservedCredits: true }),
|
|
upsertBalance: (userId, { set, setOnInsert }) =>
|
|
methods.upsertBalanceFields(userId, set ?? {}, setOnInsert ?? {}),
|
|
// Compare-and-set: only initialize an existing record while its credit is still null.
|
|
// No upsert — a CAS miss must re-read the winner, not insert a fresh balance. `null`
|
|
// in the filter also matches a legacy record whose `tokenCredits` field is absent.
|
|
initializeNullBalance: (userId, { tokenCredits, sync }) =>
|
|
mongoose.models.Balance.findOneAndUpdate(
|
|
{ user: userId, tokenCredits: null },
|
|
{ $set: { tokenCredits, ...(sync && Object.keys(sync).length > 0 ? sync : {}) } },
|
|
{ new: true, sort: { _id: 1 } },
|
|
).lean(),
|
|
enqueueAgentTrigger,
|
|
// Reconciliation reads the durable delivery to tell a still-live admission (a
|
|
// deferred Retry-After) or a dead-letter apart from a genuinely orphaned run.
|
|
getTriggerDelivery: getAgentTriggerDelivery,
|
|
resolveAgentFireAccess,
|
|
// Chat projects are user-owned, so this scoped read is both the existence and the
|
|
// authorization check the fire-time destination precheck needs.
|
|
getChatProject: (userId, projectId) => methods.getChatProject(userId, projectId),
|
|
// Reuse the merged trigger/deletion admission fence. A schedule fire and every
|
|
// generic trigger now fail closed on the same durable principal state.
|
|
isUserDeleting,
|
|
});
|
|
return service;
|
|
}
|
|
|
|
const invoke =
|
|
(method) =>
|
|
(...args) =>
|
|
getService()[method](...args);
|
|
|
|
/** Host adapter for the generic generation runtime's approval-expiry seam. The
|
|
* callback is intentionally idempotent: a replica relay or schedule reconciliation
|
|
* may re-drive the same retained terminal evidence after a transient failure. */
|
|
async function recordExpiredScheduleApproval(streamId, job) {
|
|
if (!job?.scheduleId || !job?.scheduledFor) {
|
|
return;
|
|
}
|
|
const recorded = await getService().recordScheduleOutcome({
|
|
scheduleId: job.scheduleId,
|
|
scheduledFor: job.scheduledFor,
|
|
streamId,
|
|
jobCreatedAt: job.createdAt,
|
|
status: 'interrupted',
|
|
conversationId: job.conversationId ?? streamId,
|
|
error: 'Approval expired before a decision was made',
|
|
});
|
|
if (!recorded) {
|
|
throw new Error(`Failed to settle expired scheduled approval ${job.scheduleId}`);
|
|
}
|
|
}
|
|
|
|
module.exports = {
|
|
getLimits: invoke('getLimits'),
|
|
fireScheduleNow: invoke('fireScheduleNow'),
|
|
recordScheduleOutcome: invoke('recordScheduleOutcome'),
|
|
beginScheduledStop: invoke('beginScheduledStop'),
|
|
acknowledgeScheduledStopPersistence: invoke('acknowledgeScheduledStopPersistence'),
|
|
claimScheduleResume: invoke('claimScheduleResume'),
|
|
releaseScheduleResumeClaim: invoke('releaseScheduleResumeClaim'),
|
|
finalizeScheduleResumeClaim: invoke('finalizeScheduleResumeClaim'),
|
|
releaseScheduleResumeFence: invoke('releaseScheduleResumeFence'),
|
|
isScheduleLive: invoke('isScheduleLive'),
|
|
deleteScheduleForOwner: invoke('deleteScheduleForOwner'),
|
|
quiesceUserSchedules: invoke('quiesceUserSchedules'),
|
|
restoreUserSchedulesFromDeletion: invoke('restoreUserSchedulesFromDeletion'),
|
|
initializeScheduleEngine: invoke('initializeScheduleEngine'),
|
|
initializeScheduleErasureSweep: invoke('initializeScheduleErasureSweep'),
|
|
recordExpiredScheduleApproval,
|
|
isUserDeleting: async (userId) => {
|
|
const methods = require('~/models');
|
|
return !(await methods.isAgentTriggerPrincipalActive(userId));
|
|
},
|
|
// Identity-fenced delete for a retained job whose run has since been settled
|
|
// inline; reconciliation only reaches jobs whose run is still active.
|
|
clearScheduledJob: (...args) => getService().engineDeps.clearReconciledJob(...args),
|
|
};
|