mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-04 14:57:42 +00:00
728fc1276e
* 🔒 fix: Bound `/files/usage` TTL Hold Instead of Clearing It `POST /files/usage` marks queued attachments so the 1-hour upload-window TTL cannot reap them before the client queue drains. It did this by calling `updateFilesUsage`, which unsets `expiresAt` outright, turning every touched upload into a permanently retained file. The client queue is ephemeral browser state, so this also leaks in normal use: a closed tab or cleared queue leaves nothing referencing the files, but their TTL is already gone. The same mechanism let an authenticated user pin arbitrary owned uploads indefinitely, and the route was excluded from the file limiters, so the touch was entirely unmetered. Make the operation match its intent, a renewable hold rather than a release: - Add `extendFilesTTL`, which pushes `expiresAt` forward by a bounded window in a single owner-scoped `updateMany`. Two filter guards keep it safe under client-supplied ids: `$exists: true` so an already-released file never has a TTL re-added (that would schedule a live file for deletion), and `$lt` so a hold only ever moves the deadline later. The owner scope is a required argument, so an unscoped call is a no-op rather than a cross-user update. - `handleFilesUsageRequest` now holds for 24h instead of clearing, and no longer increments `usage`, since a queue touch is not a send. The real release still happens at drain, where `updateFilesUsage` marks the files used against an actual message. - Give `/usage` its own per-user limiter. Keeping it off the upload quota was intentional, leaving it unmetered was not. Abandoned queues are now reaped on schedule, and a replayed touch can only ever re-assert the same bounded window. * 🔒 fix: Anchor the `/files/usage` hold to upload time Codex review onb687922. The hold derived each new deadline from `Date.now()`, so a caller touching once a day advanced it by another 24h every time, far below the rate limit. That left indefinite preservation reachable and made the PR's replay claim wrong: the window was bounded per call but not in aggregate. Anchor the deadline to the file's immutable `createdAt` instead of the request clock. `extendFilesTTL` now takes a lifetime and sets `expiresAt = max(expiresAt, createdAt + holdMs)` in an aggregation pipeline, so the target is a fixed point per file and replay is inert rather than merely bounded. `$max` keeps the widen-only property and the `expiresAt: {$exists: true}` filter still refuses to resurrect a released TTL; `createdAt: {$exists: true}` fail-closes when the anchor is absent. The update runs with `timestamps: false`: a hold is TTL bookkeeping, not a content write, and bumping `updatedAt` also made every re-touch count as a modification, hiding whether the deadline actually moved. Also drop four `.node_modules-*` symlinks that `git add -A` swept in from an npm install. They pointed at absolute paths on one machine, so every other checkout got dangling entries. Added the pattern to .gitignore so a workspace install cannot reintroduce them. * 🔒 fix: Track the configured approval window in the `/files/usage` hold Codex review on9277620. `endpoints.agents.checkpointer.ttl` is a positive int with no upper bound, and its docs invite raising it for longer review windows. It drives the pending-action expiry, so a run can legitimately stay paused past 24h. The fixed 24h lifetime would then let Mongo reap an attachment while its approval was still live, and the later queue drain would send a file that no longer exists. Replace the fixed constant with `resolveFilesUsageHoldMs`, which adds the configured approval window to a 24h baseline covering upload, enqueue, and the run reaching its pause. The route reads the window from the same `getApprovalTtlMs(checkpointerCfg)` the pending action uses, so the two stay in lockstep. The replay bound is unaffected: the window is a per-deployment constant and the deadline is still `createdAt + holdMs`, so a replayed touch re-asserts the same instant and `$max` skips the write. Only an operator config change moves it, never a client. * 🔒 fix: Renew the `/files/usage` hold across queued runs, under a ceiling Codex review on2bd3c52. The drain sends one queued item per run completion, and each item starts a run that may itself pause for the full approval window. Since the hold was taken once at enqueue and pinned to the upload time, an item several places back could sit through multiple approval windows and lose its attachment while its chip and the live approval were still there. Another regression from this PR: the old `$unset` made retention permanent, so deep queues happened to work. The queue is unbounded, so no fixed lifetime covers it. Split the hold into a renewable window and a ceiling: expiresAt = max(expiresAt, min(now + renewMs, createdAt + maxLifetimeMs)) `renewMs` covers one run's wait and is granted from now, so a queue that is still draining re-asserts it at each transition; `useQueueDrain` now marks the remaining items' files whenever it pops one. `maxLifetimeMs` is measured from the immutable upload time and clamps every renewal, so repeated touches converge on a ceiling instead of advancing per call, which keeps the replay bound from the previous round intact. This also tightens abandonment: a queue nobody drains now lapses one `renewMs` after its last touch instead of surviving to the ceiling. `useQueueDrain`'s spec gained a QueryClientProvider, since the renewal goes through react-query. * 🔒 fix: Renew queued holds on a heartbeat, and stop dropping batches Codex review onf616bed. Three gaps in the renewal added last commit: - `collectQueuedFileIds` returned early at the server's 10-id cap, so a remainder holding more than one batch renewed only its first message and left the rest on their enqueue-time hold. Collect everything and split into capped requests instead of truncating. - A refused `ask()` restores the popped item, but renewal ran before the send and covered only the pre-existing remainder. Since the run-end signal is already consumed, nothing would touch that item again. Renewal now runs after `ask` and includes the restored item. - A single run can interrupt for approval more than once, each pause running to the configured window, so renewing only at drain transitions leaves a gap longer than `renewMs` with no renewal in it. The ceiling cannot help when nothing renews. The third is the same structural gap as the previous round along a new axis: renewal tied to discrete events loses the file whenever two events are further apart than the hold. Rather than hook each transition, renew on a 30 minute heartbeat while anything is queued, which is far below the smallest hold (24h) and so covers any single gap regardless of cause. Still bounded: every renewal is clamped against the file's upload time, so the ceiling is unchanged. A queue nobody has open emits no heartbeat and lapses one `renewMs` after its last touch, preserving the abandonment behaviour. * 🔒 fix: Cover the pre-migration queue, first tick, and `/usage/` Codex review on892a27d. - The heartbeat watched only the active conversation id, but `drainNext` merges in the `NEW_CONVO` queue, which outlives the URL update: items queued during the first turn stay keyed there until that run ends. It now renews the union of both, deduped since they are the same atom before migration. - The interval installed without firing, so returning to a conversation whose hold was nearly up waited out a full period before the first renewal. It now renews immediately, then on each tick. - Express's non-strict routing sends `POST /files/usage/` to the same handler with `req.path === '/usage/'`, so the exact comparison pushed it onto both upload limiters. A trailing-slash client would have spent its upload quota, and collected file-upload violations, on metadata heartbeats. Matching now tolerates the trailing slash. Firing on effect start also made the drain-time renewal redundant: popping an item changes the held set, so the renewal effect re-runs on its own. The one case it cannot see is a refused send, where restoring the item leaves the set identical, so that branch keeps an explicit renewal and the rest is removed. Net one request per transition instead of two.
81 lines
2.7 KiB
JavaScript
81 lines
2.7 KiB
JavaScript
const express = require('express');
|
|
const {
|
|
createFileUsageLimiter,
|
|
createFileLimiters,
|
|
configMiddleware,
|
|
requireJwtAuth,
|
|
uaParser,
|
|
checkBan,
|
|
} = require('~/server/middleware');
|
|
const { restoreTenantContextFromReq } = require('@librechat/api');
|
|
const { avatar: asstAvatarRouter } = require('~/server/routes/assistants/v1');
|
|
const { avatar: agentAvatarRouter } = require('~/server/routes/agents/v1');
|
|
const { createMulterInstance } = require('./multer');
|
|
|
|
const files = require('./files');
|
|
const images = require('./images');
|
|
const avatar = require('./avatar');
|
|
const speech = require('./speech');
|
|
|
|
const initialize = async () => {
|
|
const router = express.Router();
|
|
router.use(requireJwtAuth);
|
|
router.use(configMiddleware);
|
|
router.use(checkBan);
|
|
router.use(uaParser);
|
|
|
|
const upload = await createMulterInstance();
|
|
router.post('/speech/stt', upload.single('audio'), restoreTenantContextFromReq);
|
|
|
|
/* Important: speech route must be added before the upload limiters */
|
|
router.use('/speech', speech);
|
|
|
|
const { fileUploadIpLimiter, fileUploadUserLimiter } = createFileLimiters();
|
|
const fileUsageLimiter = createFileUsageLimiter();
|
|
|
|
/** Non-strict routing means `/usage/` reaches the same handler, so match the
|
|
* route the way Express does. An exact comparison would push a
|
|
* trailing-slash request onto the upload quota instead. */
|
|
const isUsagePath = (path) => path.replace(/\/+$/, '') === '/usage';
|
|
|
|
/** Apply rate limiters to all POST routes (excluding /speech which is handled
|
|
* above). `/usage` is a metadata touch, so it gets its own limiter rather
|
|
* than consuming upload quota, but it is never left unmetered. */
|
|
router.use((req, res, next) => {
|
|
if (req.method !== 'POST' || req.path.startsWith('/speech')) {
|
|
return next();
|
|
}
|
|
if (isUsagePath(req.path)) {
|
|
return fileUsageLimiter(req, res, next);
|
|
}
|
|
return fileUploadIpLimiter(req, res, (err) => {
|
|
if (err) {
|
|
return next(err);
|
|
}
|
|
return fileUploadUserLimiter(req, res, next);
|
|
});
|
|
});
|
|
|
|
router.post('/', upload.single('file'), restoreTenantContextFromReq);
|
|
router.post('/images', upload.single('file'), restoreTenantContextFromReq);
|
|
router.post('/images/avatar', upload.single('file'), restoreTenantContextFromReq);
|
|
router.post(
|
|
'/images/agents/:agent_id/avatar',
|
|
upload.single('file'),
|
|
restoreTenantContextFromReq,
|
|
);
|
|
router.post(
|
|
'/images/assistants/:assistant_id/avatar',
|
|
upload.single('file'),
|
|
restoreTenantContextFromReq,
|
|
);
|
|
|
|
router.use('/', files);
|
|
router.use('/images', images);
|
|
router.use('/images/avatar', avatar);
|
|
router.use('/images/agents', agentAvatarRouter);
|
|
router.use('/images/assistants', asstAvatarRouter);
|
|
return router;
|
|
};
|
|
|
|
module.exports = { initialize };
|