🧯 fix: Harden Data Retention Semantics (#13049)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions

* feat: support data retention for normal chats

Add retentionMode config variable supporting "all" and "temporary" values.
When "all" is set, data retention applies to all chats, not just temporary ones.
Adds isTemporary field to conversations for proper filtering.

Adapted to new TS method files in packages/data-schemas since upstream
moved models out of api/models/.

Based on danny-avila/LibreChat#10532

Co-Authored-By: WhammyLeaf <233105313+WhammyLeaf@users.noreply.github.com>
(cherry picked from commit 30109e90b0)

* feat: extend data retention to files, tool calls, and shared links

Add expiredAt field and TTL indexes to file, toolCall, and share schemas.
Set expiredAt on tool calls, shared links, and file uploads when
retentionMode is "all" or chat is temporary.

(cherry picked from commit 48973752d3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: lint/test

(cherry picked from commit 310c514e6a)

* fix: address code review feedback for data retention PR

Critical:
- Fix BookmarkMenu crash: restore optional chaining on conversation
- Fix migration hazard: backward-compatible sidebar filter that also
  checks expiredAt for documents without isTemporary field

Major:
- Add logging to getRetentionExpiry error path, align with tools.js
- Add tests for retentionMode: ALL in saveConvo and saveMessage
- Fix share route: apply expiredAt for temporary chats too by
  querying the conversation's isTemporary flag server-side
- Add assertions for getRetentionExpiry mocks in process tests

Minor:
- Fix ChatRoute isTemporaryChat to be strictly boolean via Boolean()
- Fix stale test description (expired -> temporary)
- Comment out retentionMode default in example yaml
- Simplify verbose if/else to isTemporary === true
- Add compound index on { user: 1, isTemporary: 1 }
- Remove narrating comment from process.spec.js

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
(cherry picked from commit 6bad535f90)

* chore: fix typescript

(cherry picked from commit 826527a46b)

* fix: lint

(cherry picked from commit 77817e80ea)

* fix: use mockSanitizeArtifactPath in retention test

The 'getRetentionExpiry is called with the request object' test
referenced an undefined `mockSanitizeFilename` identifier, breaking
both lint (no-undef) and the test suite. Use the existing
`mockSanitizeArtifactPath` mock that the surrounding tests already
use, since `processCodeOutput` calls `sanitizeArtifactPath` (not
`sanitizeFilename`) before invoking `getRetentionExpiry`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 52ea2da66d)

* fix: forward isTemporary from client for retention on file uploads and tool calls

Server-side `getRetentionExpiry` (file uploads) and the tool-call
controller both read `req.body.isTemporary`, but the file upload
multipart form and the tool-call payload did not include that field.
In `retentionMode: temporary` (default), files uploaded and tool
calls created from temporary chats were therefore retained
indefinitely.

Forward the Recoil `isTemporary` flag in both client paths so the
existing server checks can fire correctly. `ToolParams` gains an
optional `isTemporary` field.

Addresses Codex P1 review feedback on PR #29.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 7e937df05a)

* test: stub store.isTemporary in useFileHandling test mocks

Previous commit added `useRecoilValue(store.isTemporary)` to the
hook. The test file mocks `~/store` with only `ephemeralAgentByConvoId`
and does not stub `useRecoilValue`, so all 7 cases threw
"Invalid argument to useRecoilValue: expected an atom or selector but
got undefined". Add a stub default export with `isTemporary` and a
`useRecoilValue` mock returning `false`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit eb1609537d)

* fix: harden data retention semantics

* fix: provide sweep request context for expired files

* fix: preserve temporary flags in all-retention updates

* fix: honor assistant versions in retention sweeps

* fix: retain non-temporary flags in all mode

* fix: hide expired retained records

* fix: propagate retained conversation expiry

* fix: refresh meili retention cutoff

* fix: prevent overlapping file sweeps

* fix: show legacy retained conversations

* fix: index legacy retained records

* fix: harden retention cleanup edge cases

* fix: count failed file storage sweeps

* fix: preserve legacy temporary retention

* fix: assign retention sweep worker deterministically

* fix: hide expired shared links on reads

* fix: prevent retention refresh after parent expiry

* fix: break code output retention import cycle

* fix: harden retention review findings

* fix: ignore expired share duplicates

* fix: reject expired retained share creation

* fix: harden retention review edge cases

* fix: address retention audit findings

* fix: enforce expired conversation shares in all retention

* fix: scope temporary upload flag to chat files

* fix: address retention review findings

* fix: address codex retention review findings

* fix: tighten missing storage detection

* test: remove unused file process spec bindings

---------

Co-authored-by: WhammyLeaf <233105313+WhammyLeaf@users.noreply.github.com>
Co-authored-by: Aron Gates <aron@muonspace.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Danny Avila 2026-05-19 21:58:42 -04:00 committed by GitHub
parent 2418f854df
commit 9dd062e42e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
67 changed files with 3299 additions and 171 deletions

View file

@ -4,7 +4,11 @@ const { v4: uuidv4 } = require('uuid');
const { ProxyAgent, fetch } = require('undici');
const { logger } = require('@librechat/data-schemas');
const { Tool } = require('@librechat/agents/langchain/tools');
const { getImageBasename, extractBaseURL } = require('@librechat/api');
const {
getImageBasename,
extractBaseURL,
createMinimalRetentionRequest,
} = require('@librechat/api');
const { FileContext, ContentTypes } = require('librechat-data-provider');
const dalle3JsonSchema = {
@ -49,6 +53,7 @@ class DALLE3 extends Tool {
this.userId = fields.userId;
this.tenantId = fields.req?.user?.tenantId;
this.retentionRequest = createMinimalRetentionRequest(fields.req);
this.fileStrategy = fields.fileStrategy;
/** @type {boolean} */
this.isAgent = fields.isAgent;
@ -230,6 +235,7 @@ Error Message: ${error.message}`);
fileStrategy: this.fileStrategy,
context: FileContext.image_generation,
tenantId: this.tenantId,
req: this.retentionRequest,
});
if (this.returnMetadata) {

View file

@ -4,6 +4,7 @@ const { v4: uuidv4 } = require('uuid');
const { logger } = require('@librechat/data-schemas');
const { HttpsProxyAgent } = require('https-proxy-agent');
const { Tool } = require('@librechat/agents/langchain/tools');
const { createMinimalRetentionRequest } = require('@librechat/api');
const { FileContext, ContentTypes } = require('librechat-data-provider');
const fluxApiJsonSchema = {
@ -110,6 +111,7 @@ class FluxAPI extends Tool {
this.userId = fields.userId;
this.tenantId = fields.req?.user?.tenantId;
this.retentionRequest = createMinimalRetentionRequest(fields.req);
this.fileStrategy = fields.fileStrategy;
/** @type {boolean} **/
@ -343,6 +345,7 @@ class FluxAPI extends Tool {
basePath: 'images',
context: FileContext.image_generation,
tenantId: this.tenantId,
req: this.retentionRequest,
});
logger.debug('[FluxAPI] Image saved to path:', result.filepath);
@ -574,6 +577,7 @@ class FluxAPI extends Tool {
basePath: 'images',
context: FileContext.image_generation,
tenantId: this.tenantId,
req: this.retentionRequest,
});
logger.debug('[FluxAPI] Finetuned image saved to path:', result.filepath);

View file

@ -100,11 +100,21 @@ describe('image tools - agent mode ToolMessage format', () => {
});
it('keeps tenant context without retaining the request object', () => {
const req = { user: { tenantId: 'tenant-a' }, socket: {} };
const req = {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
socket: {},
};
const dalle = new DALLE3({ isAgent: false, processFileURL: jest.fn(), req });
expect(dalle.tenantId).toBe('tenant-a');
expect(dalle.req).toBeUndefined();
expect(dalle.retentionRequest).toEqual({
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
});
});
it('invoke() returns ToolMessage with base64 in artifact, not serialized in content', async () => {
@ -181,11 +191,90 @@ describe('image tools - agent mode ToolMessage format', () => {
});
it('keeps tenant context without retaining the request object', () => {
const req = { user: { tenantId: 'tenant-a' }, socket: {} };
const req = {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
socket: {},
};
const flux = new FluxAPI({ isAgent: false, processFileURL: jest.fn(), req });
expect(flux.tenantId).toBe('tenant-a');
expect(flux.req).toBeUndefined();
expect(flux.retentionRequest).toEqual({
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
});
});
it('passes minimal retention context when saving generated images', async () => {
const processFileURL = jest.fn().mockResolvedValue({ filepath: '/images/generated.png' });
const req = {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
socket: {},
};
const flux = new FluxAPI({
isAgent: false,
processFileURL,
req,
userId: 'user-1',
fileStrategy: 'local',
});
const invokePromise = flux.invoke(
makeToolCall('flux', { prompt: 'a box', endpoint: '/v1/flux-dev' }),
);
await jest.runAllTimersAsync();
await invokePromise;
expect(processFileURL).toHaveBeenCalledWith(
expect.objectContaining({
req: {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
},
}),
);
});
it('passes minimal retention context when saving finetuned generated images', async () => {
const processFileURL = jest.fn().mockResolvedValue({ filepath: '/images/generated.png' });
const req = {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
socket: {},
};
const flux = new FluxAPI({
isAgent: false,
processFileURL,
req,
userId: 'user-1',
fileStrategy: 'local',
});
const invokePromise = flux.invoke(
makeToolCall('flux', {
action: 'generate_finetuned',
prompt: 'a box',
finetune_id: 'ft-abc123',
endpoint: '/v1/flux-pro-finetuned',
}),
);
await jest.runAllTimersAsync();
await invokePromise;
expect(processFileURL).toHaveBeenCalledWith(
expect.objectContaining({
req: {
user: { id: 'user-1', tenantId: 'tenant-a' },
body: { conversationId: 'convo-1', isTemporary: 'true' },
config: { interfaceConfig: { retentionMode: 'all' } },
},
}),
);
});
it('invoke() returns ToolMessage with base64 in artifact, not serialized in content', async () => {

View file

@ -1,4 +1,4 @@
const { logger } = require('@librechat/data-schemas');
const { logger, buildRetentionVisibilityFilter } = require('@librechat/data-schemas');
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
@ -26,7 +26,10 @@ async function batchResetMeiliFlags(collection) {
try {
while (hasMore) {
const docs = await collection
.find({ expiredAt: null, _meiliIndex: { $ne: false } }, { projection: { _id: 1 } })
.find(
{ ...buildRetentionVisibilityFilter(), _meiliIndex: { $ne: false } },
{ projection: { _id: 1 } },
)
.limit(BATCH_SIZE)
.toArray();

View file

@ -83,6 +83,60 @@ describe('batchResetMeiliFlags', () => {
expect(expiredDoc._meiliIndex).toBe(true);
});
it('should reset active non-temporary documents with expiredAt set for all-data retention', async () => {
const retentionDate = new Date(Date.now() + 60 * 60 * 1000);
await testCollection.insertMany([
{
_id: new mongoose.Types.ObjectId(),
isTemporary: false,
expiredAt: retentionDate,
_meiliIndex: true,
},
{
_id: new mongoose.Types.ObjectId(),
isTemporary: true,
expiredAt: retentionDate,
_meiliIndex: true,
},
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
const retainedDoc = await testCollection.findOne({ isTemporary: false });
const temporaryDoc = await testCollection.findOne({ isTemporary: true });
expect(retainedDoc._meiliIndex).toBe(false);
expect(temporaryDoc._meiliIndex).toBe(true);
});
it('should not reset expired non-temporary documents with expiredAt set for all-data retention', async () => {
const retentionDate = new Date(Date.now() - 60 * 60 * 1000);
await testCollection.insertMany([
{
_id: new mongoose.Types.ObjectId(),
isTemporary: false,
expiredAt: retentionDate,
_meiliIndex: true,
},
{
_id: new mongoose.Types.ObjectId(),
isTemporary: false,
expiredAt: null,
_meiliIndex: true,
},
]);
const result = await batchResetMeiliFlags(testCollection);
expect(result).toBe(1);
const expiredDoc = await testCollection.findOne({ expiredAt: retentionDate });
const permanentDoc = await testCollection.findOne({ expiredAt: null });
expect(expiredDoc._meiliIndex).toBe(true);
expect(permanentDoc._meiliIndex).toBe(false);
});
it('should not modify documents with _meiliIndex: false', async () => {
await testCollection.insertMany([
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },

View file

@ -75,7 +75,7 @@ jest.mock('@librechat/api', () => ({
}));
jest.mock('~/server/services/Files/process', () => ({
processDeleteRequest: jest.fn().mockResolvedValue(undefined),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
}));
jest.mock('~/server/services/Config', () => ({

View file

@ -67,7 +67,7 @@ jest.mock('~/server/services/Config/getCachedTools', () => ({
}));
jest.mock('~/server/services/Files/process', () => ({
processDeleteRequest: jest.fn(),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
}));
jest.mock('~/server/services/Config', () => ({

View file

@ -127,7 +127,7 @@ function stubDeletionMocks() {
mockDeleteUserById.mockResolvedValue();
mockDeleteAllSharedLinks.mockResolvedValue();
mockGetFiles.mockResolvedValue([]);
mockProcessDeleteRequest.mockResolvedValue();
mockProcessDeleteRequest.mockResolvedValue({ deletedFileIds: [], failedFileIds: [] });
mockDeleteFiles.mockResolvedValue();
mockDeleteToolCalls.mockResolvedValue();
mockDeleteUserAgents.mockResolvedValue();

View file

@ -81,7 +81,7 @@ jest.mock('~/server/services/Config/getCachedTools', () => ({
}));
jest.mock('~/server/services/Files/process', () => ({
processDeleteRequest: jest.fn(),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
}));
jest.mock('~/server/services/Config', () => ({

View file

@ -10,6 +10,7 @@ const {
} = require('librechat-data-provider');
const { getRoleByName, createToolCall, getToolCallsByConvo, getMessage } = require('~/models');
const { processFileURL, uploadImageBuffer } = require('~/server/services/Files/process');
const { getRetentionExpiry } = require('~/server/services/Files/retention');
const { processCodeOutput, runPreviewFinalize } = require('~/server/services/Files/Code/process');
const { loadAuthValues } = require('~/server/services/Tools/credentials');
const { loadTools } = require('~/app/clients/tools/util');
@ -167,6 +168,7 @@ const callTool = async (req, res) => {
conversationId,
result: content,
user: req.user.id,
...(await getRetentionExpiry(req)),
};
if (!artifact || !artifact.files || toolId !== Tools.execute_code) {

View file

@ -24,6 +24,7 @@ const {
const { connectDb, indexSync } = require('~/db');
const initializeOAuthReconnectManager = require('./services/initializeOAuthReconnectManager');
const createValidateImageRequest = require('./middleware/validateImageRequest');
const { startExpiredFileSweep } = require('./services/Files/process');
const { jwtLogin, ldapLogin, passportLogin } = require('~/strategies');
const { updateInterfacePermissions: updateInterfacePerms } = require('@librechat/api');
const {
@ -139,8 +140,32 @@ if (cluster.isMaster) {
logger.info(`Spawning ${workers} workers to simulate multi-pod environment`);
let activeWorkers = 0;
const listeningWorkers = new Set();
let retentionSweepWorkerId = null;
const startTime = Date.now();
const assignRetentionSweepWorker = () => {
if (retentionSweepWorkerId && cluster.workers[retentionSweepWorkerId]) {
return;
}
const connectedWorkers = Object.values(cluster.workers).filter(
(worker) => worker && worker.isConnected(),
);
const availableWorkers = connectedWorkers.filter((worker) => listeningWorkers.has(worker.id));
const workerPool = availableWorkers.length > 0 ? availableWorkers : connectedWorkers;
const retentionSweepWorker = workerPool[workerPool.length - 1];
if (!retentionSweepWorker) {
return;
}
retentionSweepWorkerId = retentionSweepWorker.id;
logger.info(
wrapLogMessage(`Worker ${retentionSweepWorker.process.pid} assigned to file-retention sweep`),
);
retentionSweepWorker.send({ type: 'file-retention-sweep-worker' });
};
/** Flush Redis cache before starting workers */
flushRedisCache()
.then(() => {
@ -162,19 +187,29 @@ if (cluster.isMaster) {
`Worker ${worker.process.pid} is online (${activeWorkers}/${workers}) after ${uptime}s`,
);
/** Notify the last worker to perform one-time initialization tasks */
/** Assign one worker for process-wide background jobs */
if (activeWorkers === workers) {
const allWorkers = Object.values(cluster.workers);
const lastWorker = allWorkers[allWorkers.length - 1];
if (lastWorker) {
logger.info(wrapLogMessage(`All ${workers} workers are online`));
lastWorker.send({ type: 'last-worker' });
}
logger.info(wrapLogMessage(`All ${workers} workers are online`));
}
});
cluster.on('listening', (worker) => {
listeningWorkers.add(worker.id);
if (
listeningWorkers.size === workers ||
(!retentionSweepWorkerId && activeWorkers >= workers)
) {
assignRetentionSweepWorker();
}
});
cluster.on('exit', (worker, code, signal) => {
activeWorkers--;
listeningWorkers.delete(worker.id);
if (worker.id === retentionSweepWorkerId) {
retentionSweepWorkerId = null;
assignRetentionSweepWorker();
}
logger.error(
`Worker ${worker.process.pid} died (${activeWorkers}/${workers}). Code: ${code}, Signal: ${signal}`,
);
@ -202,6 +237,32 @@ if (cluster.isMaster) {
* Each worker runs a full Express server instance
*/
const app = express();
/**
* The master may assign the sweep worker before or after this worker has
* loaded app config. These flags join the IPC assignment with config
* availability and ensure the background sweep starts only once.
*/
let shouldStartExpiredFileSweep = false;
let expiredFileSweepOptions = null;
let expiredFileSweepStarted = false;
const startExpiredFileSweepOnce = () => {
if (!shouldStartExpiredFileSweep || expiredFileSweepStarted || !expiredFileSweepOptions) {
return;
}
expiredFileSweepStarted = true;
startExpiredFileSweep(expiredFileSweepOptions);
};
/** Handle inter-process messages from master */
process.on('message', (msg) => {
if (msg.type === 'file-retention-sweep-worker') {
shouldStartExpiredFileSweep = true;
logger.info(wrapLogMessage(`Worker ${process.pid} is assigned file-retention sweep`));
startExpiredFileSweepOnce();
}
});
const startServer = async () => {
logger.info(`Worker ${process.pid} initializing...`);
@ -233,6 +294,8 @@ if (cluster.isMaster) {
/** Initialize app configuration */
const appConfig = await getAppConfig();
initializeFileStorage(appConfig);
expiredFileSweepOptions = { appConfig, loadAppConfig: getAppConfig };
startExpiredFileSweepOnce();
await performStartupChecks(appConfig);
await updateInterfacePerms({ appConfig, getRoleByName, updateAccessPermissions });
@ -390,19 +453,6 @@ if (cluster.isMaster) {
process.exit(1);
}
});
/** Handle inter-process messages from master */
process.on('message', async (msg) => {
if (msg.type === 'last-worker') {
logger.info(
wrapLogMessage(
`Worker ${process.pid} is the last worker and can perform special initialization tasks`,
),
);
/** Add any one-time initialization tasks here */
/** For example: scheduled jobs, cleanup tasks, etc. */
}
});
};
startServer().catch((err) => {

View file

@ -34,6 +34,7 @@ const {
const initializeOAuthReconnectManager = require('./services/initializeOAuthReconnectManager');
const { capabilityContextMiddleware } = require('./middleware/roles/capabilities');
const createValidateImageRequest = require('./middleware/validateImageRequest');
const { startExpiredFileSweep } = require('./services/Files/process');
const { jwtLogin, ldapLogin, passportLogin } = require('~/strategies');
const { checkMigrations } = require('./services/start/migration');
const optionalJwtAuth = require('./middleware/optionalJwtAuth');
@ -89,6 +90,7 @@ const startServer = async () => {
});
const appConfig = await getAppConfig({ baseOnly: true });
initializeFileStorage(appConfig);
startExpiredFileSweep({ appConfig, loadAppConfig: getAppConfig });
await runAsSystem(async () => {
await performStartupChecks(appConfig);
await updateInterfacePermissions({ appConfig, getRoleByName, updateAccessPermissions });

View file

@ -0,0 +1,262 @@
const express = require('express');
const request = require('supertest');
const mongoose = require('mongoose');
const mockGetSharedLinkExpiration = jest.fn();
jest.mock('@librechat/api', () => ({
isEnabled: jest.fn(() => true),
getSharedLinkExpiration: (...args) => mockGetSharedLinkExpiration(...args),
isActiveExpirationDate: jest.fn((expiredAt) => expiredAt > new Date()),
}));
jest.mock('@librechat/data-schemas', () => ({
logger: { error: jest.fn() },
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
}));
jest.mock('librechat-data-provider', () => ({
RetentionMode: {
ALL: 'all',
TEMPORARY: 'temporary',
},
}));
jest.mock('mongoose', () => ({
models: {
Conversation: {
findOne: jest.fn(),
},
SharedLink: {
findOne: jest.fn(),
},
},
}));
jest.mock('~/models', () => ({
getSharedMessages: jest.fn(),
createSharedLink: jest.fn(),
updateSharedLink: jest.fn(),
deleteSharedLink: jest.fn(),
getSharedLinks: jest.fn(),
getSharedLink: jest.fn(),
}));
jest.mock('~/server/middleware/requireJwtAuth', () => (req, res, next) => next());
const { RetentionMode } = require('librechat-data-provider');
const { createTempChatExpirationDate, logger } = require('@librechat/data-schemas');
const { createSharedLink, updateSharedLink } = require('~/models');
const shareRouter = require('../share');
const activeExpiration = new Date('2030-01-01T00:00:00.000Z');
const expiredExpiration = new Date('2020-01-01T00:00:00.000Z');
const lean = (value) => ({
lean: jest.fn().mockResolvedValue(value),
});
const buildApp = ({ retentionMode = RetentionMode.TEMPORARY } = {}) => {
const app = express();
app.use(express.json());
app.use((req, _res, next) => {
req.user = { id: 'user-123' };
req.config = { interfaceConfig: { retentionMode } };
next();
});
app.use('/api/share', shareRouter);
return app;
};
describe('share routes retention', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('expires new shares for retained non-temporary conversations', async () => {
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
const response = await request(buildApp())
.post('/api/share/convo-123')
.send({ targetMessageId: 'msg-123' });
expect(response.status).toBe(200);
expect(mockGetSharedLinkExpiration).toHaveBeenCalledWith(
expect.objectContaining({
conversationId: 'convo-123',
req: expect.objectContaining({ user: { id: 'user-123' } }),
}),
expect.objectContaining({
getConvo: expect.any(Function),
createExpirationDate: createTempChatExpirationDate,
logger,
}),
);
const [, dependencies] = mockGetSharedLinkExpiration.mock.calls[0];
mongoose.models.Conversation.findOne.mockReturnValue(lean({ expiredAt: activeExpiration }));
await dependencies.getConvo('user-123', 'convo-123');
expect(mongoose.models.Conversation.findOne).toHaveBeenCalledWith(
{ conversationId: 'convo-123', user: 'user-123' },
'isTemporary expiredAt',
);
expect(createSharedLink).toHaveBeenCalledWith(
'user-123',
'convo-123',
'msg-123',
new Date('2030-01-01T00:00:00.000Z'),
);
});
it('rejects new shares when the retained conversation expired', async () => {
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
const response = await request(buildApp())
.post('/api/share/convo-123')
.send({ targetMessageId: 'msg-123' });
expect(response.status).toBe(404);
expect(createSharedLink).not.toHaveBeenCalled();
});
it('rejects new shares for expired conversations in all retention mode', async () => {
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
const response = await request(buildApp({ retentionMode: RetentionMode.ALL }))
.post('/api/share/convo-123')
.send({ targetMessageId: 'msg-123' });
expect(response.status).toBe(404);
expect(createSharedLink).not.toHaveBeenCalled();
});
it('expires updated shares for retained non-temporary conversations', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp()).patch('/api/share/share-123');
expect(response.status).toBe(200);
expect(mongoose.models.SharedLink.findOne).toHaveBeenCalledWith(
{ shareId: 'share-123', user: 'user-123' },
'conversationId',
);
expect(mockGetSharedLinkExpiration).toHaveBeenCalledTimes(1);
expect(mockGetSharedLinkExpiration).toHaveBeenCalledWith(
expect.objectContaining({
conversationId: 'convo-123',
req: expect.objectContaining({ user: { id: 'user-123' } }),
}),
expect.objectContaining({
getConvo: expect.any(Function),
createExpirationDate: createTempChatExpirationDate,
logger,
}),
);
expect(updateSharedLink).toHaveBeenCalledWith(
'user-123',
'share-123',
undefined,
new Date('2030-01-01T00:00:00.000Z'),
);
});
it('rejects updated shares when the retained conversation expired', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp()).patch('/api/share/share-123');
expect(response.status).toBe(404);
expect(updateSharedLink).not.toHaveBeenCalled();
});
it('rejects updated shares for expired conversations in all retention mode', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp({ retentionMode: RetentionMode.ALL })).patch(
'/api/share/share-123',
);
expect(response.status).toBe(404);
expect(mongoose.models.SharedLink.findOne).toHaveBeenCalledWith(
{ shareId: 'share-123', user: 'user-123' },
'conversationId',
);
expect(updateSharedLink).not.toHaveBeenCalled();
});
it('clears updated share expiration when the conversation is no longer retained', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(null);
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp()).patch('/api/share/share-123');
expect(response.status).toBe(200);
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, null);
});
it('preserves updated share expiration when the conversation cannot be found', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(undefined);
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp()).patch('/api/share/share-123');
expect(response.status).toBe(200);
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, undefined);
});
it('clears updated share expiration when creating a new expiration throws', async () => {
const error = new Error('bad config');
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockImplementationOnce(async (_input, dependencies) => {
dependencies.logger.error('[getSharedLinkExpiration] Error creating expiration date:', error);
return null;
});
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
const response = await request(buildApp()).patch('/api/share/share-123');
expect(response.status).toBe(200);
expect(logger.error).toHaveBeenCalledWith(
'[getSharedLinkExpiration] Error creating expiration date:',
error,
);
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, null);
});
it('updates share target message while applying retention expiration', async () => {
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
updateSharedLink.mockResolvedValue({ shareId: 'share-456', targetMessageId: 'msg-456' });
const response = await request(buildApp())
.patch('/api/share/share-123')
.send({ targetMessageId: 'msg-456' });
expect(response.status).toBe(200);
expect(updateSharedLink).toHaveBeenCalledWith(
'user-123',
'share-123',
'msg-456',
new Date('2030-01-01T00:00:00.000Z'),
);
});
it('rejects non-string target message updates', async () => {
const response = await request(buildApp())
.patch('/api/share/share-123')
.send({ targetMessageId: 123 });
expect(response.status).toBe(400);
expect(updateSharedLink).not.toHaveBeenCalled();
});
});

View file

@ -276,6 +276,7 @@ router.post(
filepath: req.file.path,
requestUserId: req.user.id,
userRole: req.user.role,
interfaceConfig: req.config?.interfaceConfig,
});
res.status(201).json({ message: 'Conversation(s) imported successfully' });
} catch (error) {

View file

@ -14,7 +14,7 @@ const { createAgent, createFile } = require('~/models');
// Only mock the external dependencies that we don't want to test
jest.mock('~/server/services/Files/process', () => ({
processDeleteRequest: jest.fn().mockResolvedValue({}),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
filterFile: jest.fn(),
processFileUpload: jest.fn(),
processAgentFileUpload: jest.fn().mockImplementation(async ({ res }) => {

View file

@ -16,7 +16,7 @@ const { createAgent, createFile } = require('~/models');
// Only mock the external dependencies that we don't want to test
jest.mock('~/server/services/Files/process', () => ({
processDeleteRequest: jest.fn().mockResolvedValue({}),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
filterFile: jest.fn(),
processFileUpload: jest.fn(),
processAgentFileUpload: jest.fn(),

View file

@ -36,7 +36,7 @@ jest.mock('~/models', () => ({
jest.mock('~/server/services/Files/process', () => ({
filterFile: jest.fn(),
processFileUpload: jest.fn(),
processDeleteRequest: jest.fn(),
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
processAgentFileUpload: jest.fn(),
}));

View file

@ -1,6 +1,7 @@
const mongoose = require('mongoose');
const express = require('express');
const { isEnabled } = require('@librechat/api');
const { logger } = require('@librechat/data-schemas');
const { isEnabled, isActiveExpirationDate, getSharedLinkExpiration } = require('@librechat/api');
const { logger, createTempChatExpirationDate } = require('@librechat/data-schemas');
const {
getSharedMessages,
createSharedLink,
@ -12,6 +13,22 @@ const {
const requireJwtAuth = require('~/server/middleware/requireJwtAuth');
const router = express.Router();
const resolveSharedLinkExpiration = (req, conversationId) =>
getSharedLinkExpiration(
{ req, conversationId },
{
getConvo: async (userId, sourceConversationId) => {
const Conversation = mongoose.models.Conversation;
return Conversation.findOne(
{ conversationId: sourceConversationId, user: userId },
'isTemporary expiredAt',
).lean();
},
createExpirationDate: createTempChatExpirationDate,
logger,
},
);
/**
* Shared messages
*/
@ -99,7 +116,17 @@ router.get('/link/:conversationId', requireJwtAuth, async (req, res) => {
router.post('/:conversationId', requireJwtAuth, async (req, res) => {
try {
const { targetMessageId } = req.body;
const created = await createSharedLink(req.user.id, req.params.conversationId, targetMessageId);
const expiredAt = await resolveSharedLinkExpiration(req, req.params.conversationId);
if (expiredAt != null && !isActiveExpirationDate(expiredAt)) {
return res.status(404).end();
}
const created = await createSharedLink(
req.user.id,
req.params.conversationId,
targetMessageId,
expiredAt,
);
if (created) {
res.status(200).json(created);
} else {
@ -118,7 +145,25 @@ router.patch('/:shareId', requireJwtAuth, async (req, res) => {
return res.status(400).json({ message: 'targetMessageId must be a string' });
}
const updatedShare = await updateSharedLink(req.user.id, req.params.shareId, targetMessageId);
let expiredAt;
const SharedLink = mongoose.models.SharedLink;
const existing = await SharedLink.findOne(
{ shareId: req.params.shareId, user: req.user.id },
'conversationId',
).lean();
if (existing?.conversationId) {
expiredAt = await resolveSharedLinkExpiration(req, existing.conversationId);
}
if (expiredAt != null && !isActiveExpirationDate(expiredAt)) {
return res.status(404).end();
}
const updatedShare = await updateSharedLink(
req.user.id,
req.params.shareId,
targetMessageId,
expiredAt,
);
if (updatedShare) {
res.status(200).json(updatedShare);
} else {

View file

@ -92,6 +92,11 @@ jest.mock('~/server/utils', () => ({
determineFileType: jest.fn().mockResolvedValue({ mime: 'text/csv' }),
}));
jest.mock('~/server/services/Files/retention', () => ({
getRetentionExpiry: jest.fn(() => ({})),
}));
const { getRetentionExpiry } = require('~/server/services/Files/retention');
const { createFile } = require('~/models');
const { processCodeOutput } = require('../process');
@ -143,6 +148,12 @@ describe('processCodeOutput path traversal protection', () => {
expect(fileArg.tenantId).toBe('tenantA');
});
test('getRetentionExpiry is called with the request object', async () => {
mockSanitizeArtifactPath.mockReturnValueOnce('output.csv');
await processCodeOutput({ ...baseParams, name: 'output.csv' });
expect(getRetentionExpiry).toHaveBeenCalledWith(baseParams.req);
});
test('sanitized name is used for image file records', async () => {
const { convertImage } = require('~/server/services/Files/images/convert');
convertImage.mockResolvedValueOnce({

View file

@ -36,6 +36,7 @@ const { filterFilesByAgentAccess } = require('~/server/services/Files/permission
const { createFile, getFiles, updateFile, claimCodeFile } = require('~/models');
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
const { convertImage } = require('~/server/services/Files/images/convert');
const { getRetentionExpiry } = require('~/server/services/Files/retention');
const { determineFileType } = require('~/server/utils');
const axios = createAxiosInstance();
@ -463,6 +464,7 @@ const processCodeOutput = async ({
source: appConfig.fileStrategy,
context: FileContext.execute_code,
metadata: { codeEnvRef },
...(await getRetentionExpiry(req)),
};
await createFile(file, true);
return { file: Object.assign(file, { messageId, toolCallId }) };
@ -565,6 +567,7 @@ const processCodeOutput = async ({
context: FileContext.execute_code,
usage: isUpdate ? (claimed.usage ?? 0) + 1 : 1,
createdAt: isUpdate ? claimed.createdAt : formattedDate,
...(await getRetentionExpiry(req)),
};
if (expectsPreview) {

View file

@ -137,6 +137,10 @@ jest.mock('~/server/services/Files/images/convert', () => ({
convertImage: jest.fn(),
}));
jest.mock('~/server/services/Files/retention', () => ({
getRetentionExpiry: jest.fn(() => ({})),
}));
// Mock determineFileType
jest.mock('~/server/utils', () => ({
determineFileType: jest.fn(),
@ -145,6 +149,7 @@ jest.mock('~/server/utils', () => ({
const http = require('http');
const https = require('https');
const { createFile, getFiles } = require('~/models');
const { getRetentionExpiry } = require('~/server/services/Files/retention');
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
const { convertImage } = require('~/server/services/Files/images/convert');
const { determineFileType } = require('~/server/utils');
@ -233,6 +238,7 @@ describe('Code Process', () => {
expect(result.file_id).toBe('mock-uuid-1234');
expect(result.usage).toBe(1);
expect(getRetentionExpiry).toHaveBeenCalledWith(baseParams.req);
});
});

View file

@ -18,12 +18,14 @@ const {
getEndpointFileConfig,
documentParserMimeTypes,
} = require('librechat-data-provider');
const { logger } = require('@librechat/data-schemas');
const { logger, runAsSystem } = require('@librechat/data-schemas');
const {
sanitizeFilename,
parseText,
processAudioFile,
getStorageMetadata,
sweepExpiredFiles: sweepExpiredFilesWithDeps,
startExpiredFileSweep: startExpiredFileSweepWithDeps,
} = require('@librechat/api');
const {
convertImage,
@ -36,6 +38,7 @@ const { loadAuthValues } = require('~/server/services/Tools/credentials');
const { getFileStrategy } = require('~/server/utils/getFileStrategy');
const { checkCapability } = require('~/server/services/Config');
const { LB_QueueAsyncCall } = require('~/server/utils/queue');
const { getRetentionExpiry } = require('./retention');
const { getStrategyFunctions } = require('./strategies');
const { determineFileType } = require('~/server/utils');
const { STTService } = require('./Audio/STTService');
@ -64,6 +67,17 @@ const createSanitizedUploadWrapper = (uploadFunction) => {
};
};
const isMissingStorageError = (err) => {
const code = err?.code ?? err?.status ?? err?.statusCode ?? err?.response?.status;
if ([404, '404', 'ENOENT', 'NoSuchKey', 'NotFound', 'ResourceNotFound'].includes(code)) {
return true;
}
return /(?:file|object|blob|key|resource) (?:not found|does not exist)|no such (?:file|key)/i.test(
String(err?.message ?? ''),
);
};
/**
* Enqueues the delete operation to the leaky bucket queue if necessary, or adds it directly to promises.
*
@ -72,10 +86,19 @@ const createSanitizedUploadWrapper = (uploadFunction) => {
* @param {MongoFile} params.file - The file object to delete.
* @param {Function} params.deleteFile - The delete file function.
* @param {Promise[]} params.promises - The array of promises to await.
* @param {string[]} params.resolvedFileIds - The array of promises to await.
* @param {Set<string>} params.resolvedFileIds - File IDs whose storage delete succeeded.
* @param {Set<string>} params.failedFileIds - File IDs whose storage delete failed.
* @param {OpenAI | undefined} [params.openai] - If an OpenAI file, the initialized OpenAI client.
*/
function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileIds, openai }) {
function enqueueDeleteOperation({
req,
file,
deleteFile,
promises,
resolvedFileIds,
failedFileIds,
openai,
}) {
if (checkOpenAIStorage(file.source)) {
// Enqueue to leaky bucket
promises.push(
@ -85,10 +108,17 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
[],
(err, result) => {
if (err) {
if (isMissingStorageError(err)) {
resolvedFileIds.add(file.file_id);
logger.warn('File storage was already missing during delete', err);
resolve(result);
return;
}
failedFileIds.add(file.file_id);
logger.error('Error deleting file from OpenAI source', err);
reject(err);
} else {
resolvedFileIds.push(file.file_id);
resolvedFileIds.add(file.file_id);
resolve(result);
}
},
@ -99,8 +129,14 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
// Add directly to promises
promises.push(
deleteFile(req, file)
.then(() => resolvedFileIds.push(file.file_id))
.then(() => resolvedFileIds.add(file.file_id))
.catch((err) => {
if (isMissingStorageError(err)) {
resolvedFileIds.add(file.file_id);
logger.warn('File storage was already missing during delete', err);
return;
}
failedFileIds.add(file.file_id);
logger.error('Error deleting file', err);
return Promise.reject(err);
}),
@ -121,11 +157,13 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
* @param {string} [params.req.body.assistant_id] - The assistant ID if file uploaded is associated to an assistant.
* @param {string} [params.req.body.tool_resource] - The tool resource if assistant file uploaded is associated to a tool resource.
*
* @returns {Promise<void>}
* @returns {Promise<{ deletedFileIds: string[], failedFileIds: string[] }>}
* @throws {Error} When storage deletion cannot be scheduled or file metadata cleanup fails.
*/
const processDeleteRequest = async ({ req, files }) => {
const appConfig = req.config;
const resolvedFileIds = [];
const resolvedFileIds = new Set();
const failedFileIds = new Set();
const deletionMethods = {};
const promises = [];
@ -167,7 +205,7 @@ const processDeleteRequest = async ({ req, files }) => {
}
if (source === FileSources.text) {
resolvedFileIds.push(file.file_id);
resolvedFileIds.add(file.file_id);
continue;
}
@ -198,6 +236,7 @@ const processDeleteRequest = async ({ req, files }) => {
deleteFile: deletionMethods[source],
promises,
resolvedFileIds,
failedFileIds,
openai,
});
continue;
@ -209,7 +248,15 @@ const processDeleteRequest = async ({ req, files }) => {
}
deletionMethods[source] = deleteFile;
enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileIds, openai });
enqueueDeleteOperation({
req,
file,
deleteFile,
promises,
resolvedFileIds,
failedFileIds,
openai,
});
}
if (agentFiles.length > 0) {
@ -222,17 +269,60 @@ const processDeleteRequest = async ({ req, files }) => {
}
await Promise.allSettled(promises);
await db.deleteFiles(resolvedFileIds);
if (resolvedFileIds.length > 0) {
const deletedFileIds = [...resolvedFileIds];
let metadataDeletedFileIds = deletedFileIds;
if (deletedFileIds.length > 0) {
try {
await db.removeAgentResourceFilesFromAllAgents({ file_ids: resolvedFileIds });
await db.deleteFiles(deletedFileIds);
} catch (error) {
logger.error('Error cleaning up orphaned agent file references', error);
logger.error('Error deleting file metadata after storage deletion', error);
deletedFileIds.forEach((fileId) => failedFileIds.add(fileId));
metadataDeletedFileIds = [];
throw error;
}
if (metadataDeletedFileIds.length > 0) {
try {
await db.removeAgentResourceFilesFromAllAgents({ file_ids: metadataDeletedFileIds });
} catch (error) {
logger.error('Error cleaning up orphaned agent file references', error);
}
}
}
return {
deletedFileIds: metadataDeletedFileIds,
failedFileIds: [...failedFileIds],
};
};
/**
* Deletes expired file storage before removing the corresponding File records.
*
* Mongo TTL indexes delete only the metadata document, so file retention uses
* this application sweep for records with `expiredAt` instead.
*
* @param {object} params
* @param {AppConfig} params.appConfig
* @param {number} [params.limit]
* @param {() => Promise<AppConfig>} [params.loadAppConfig]
* @returns {Promise<{ scanned: number, deleted: number, failed: number }>}
*/
async function sweepExpiredFiles(options = {}) {
return sweepExpiredFilesWithDeps(options, {
getExpiredFiles: db.getExpiredFiles,
processDeleteRequest,
logger,
});
}
function startExpiredFileSweep(options = {}) {
return startExpiredFileSweepWithDeps(options, {
sweepExpiredFiles,
runAsSystem,
logger,
});
}
/**
* Processes a file URL using a specified file handling strategy. This function accepts a strategy name,
* fetches the corresponding file processing functions (for saving and retrieving file URLs), and then
@ -251,6 +341,7 @@ const processDeleteRequest = async ({ req, files }) => {
* @param {string} params.basePath - The base path or directory where the file will be saved or retrieved from.
* @param {FileContext} params.context - The context of the file (e.g., 'avatar', 'image_generation', etc.)
* @param {string} [params.tenantId] - Optional tenant identifier for tenant-prefixed storage paths.
* @param {ServerRequest} [params.req] - Request context used to apply data retention metadata.
* @returns {Promise<MongoFile>} A promise that resolves to the DB representation (MongoFile)
* of the processed file. It throws an error if the file processing fails at any stage.
*/
@ -262,6 +353,7 @@ const processFileURL = async ({
basePath,
context,
tenantId,
req,
}) => {
const { saveURL, getFileURL } = getStrategyFunctions(fileStrategy);
try {
@ -305,6 +397,7 @@ const processFileURL = async ({
source: fileStrategy,
type,
context,
...(await getRetentionExpiry(req)),
tenantId,
width: dimensions.width,
height: dimensions.height,
@ -355,6 +448,7 @@ const processImageFile = async ({ req, res, metadata, returnFile = false }) => {
context: FileContext.message_attachment,
source,
type: `image/${appConfig.imageOutputType}`,
...(await getRetentionExpiry(req)),
width,
height,
tenantId: req.user.tenantId,
@ -415,6 +509,7 @@ const uploadImageBuffer = async ({ req, context, metadata = {}, resize = true })
source,
type,
width,
...(await getRetentionExpiry(req)),
height,
tenantId: req.user.tenantId,
},
@ -517,6 +612,7 @@ const processFileUpload = async ({ req, res, metadata }) => {
context: isAssistantUpload ? FileContext.assistants : FileContext.message_attachment,
model: isAssistantUpload ? req.body.model : undefined,
type: file.mimetype,
...(await getRetentionExpiry(req)),
embedded,
source,
height,
@ -631,20 +727,24 @@ const processAgentFileUpload = async ({ req, res, metadata }) => {
`Extracted text from "${file.originalname}" exceeds the 15MB storage limit (${Math.round(textBytes / megabyte)}MB). Try a shorter document.`,
);
}
const fileInfo = removeNullishValues({
text,
bytes,
file_id,
temp_file_id,
user: req.user.id,
type,
filepath: filepath ?? file.path,
source: FileSources.text,
filename: file.originalname,
model: messageAttachment ? undefined : req.body.model,
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
tenantId: req.user.tenantId,
});
const retentionExpiry = await getRetentionExpiry(req);
const fileInfo = {
...removeNullishValues({
text,
bytes,
file_id,
temp_file_id,
user: req.user.id,
type,
filepath: filepath ?? file.path,
source: FileSources.text,
filename: file.originalname,
model: messageAttachment ? undefined : req.body.model,
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
tenantId: req.user.tenantId,
}),
...retentionExpiry,
};
if (!messageAttachment && tool_resource) {
await db.addAgentResourceFile({
@ -825,24 +925,28 @@ const processAgentFileUpload = async ({ req, res, metadata }) => {
});
}
const fileInfo = removeNullishValues({
user: req.user.id,
file_id,
temp_file_id,
bytes,
filepath,
...storageMetadata,
filename: filename ?? sanitizeFilename(file.originalname),
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
model: messageAttachment ? undefined : req.body.model,
metadata: fileInfoMetadata,
type: file.mimetype,
embedded,
source,
height,
width,
tenantId: req.user.tenantId,
});
const retentionExpiry = await getRetentionExpiry(req);
const fileInfo = {
...removeNullishValues({
user: req.user.id,
file_id,
temp_file_id,
bytes,
filepath,
...storageMetadata,
filename: filename ?? sanitizeFilename(file.originalname),
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
model: messageAttachment ? undefined : req.body.model,
metadata: fileInfoMetadata,
type: file.mimetype,
embedded,
source,
height,
width,
tenantId: req.user.tenantId,
}),
...retentionExpiry,
};
const result = await db.createFile(fileInfo, true);
@ -887,6 +991,7 @@ const processOpenAIFile = async ({
source,
model: openai.req.body.model,
filename: originalName ?? file_id,
...(await getRetentionExpiry(openai.req)),
tenantId: openai.req?.user?.tenantId,
};
@ -931,9 +1036,14 @@ const processOpenAIImageOutput = async ({ req, buffer, file_id, filename, fileEx
context: FileContext.assistants_output,
file_id,
filename,
...(await getRetentionExpiry(req)),
tenantId: req.user.tenantId,
};
db.createFile(file, true);
try {
await db.createFile(file, true);
} catch (error) {
logger.warn('Error saving OpenAI image output file metadata', error);
}
return file;
};
@ -1091,6 +1201,7 @@ async function saveBase64Image(
user: req.user.id,
bytes: image.bytes,
width: image.width,
...(await getRetentionExpiry(req)),
height: image.height,
tenantId: req.user.tenantId,
},
@ -1182,6 +1293,8 @@ module.exports = {
saveBase64Image,
processImageFile,
uploadImageBuffer,
sweepExpiredFiles,
startExpiredFileSweep,
processFileUpload,
processDeleteRequest,
processAgentFileUpload,

View file

@ -1,22 +1,41 @@
jest.mock('uuid', () => ({ v4: jest.fn(() => 'mock-uuid') }));
jest.mock('@librechat/data-schemas', () => ({
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn() },
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn(), info: jest.fn() },
runAsSystem: jest.fn((fn) => fn()),
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
}));
jest.mock('@librechat/agents', () => ({}));
jest.mock('@librechat/api', () => ({
sanitizeFilename: jest.fn((n) => n),
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
processAudioFile: jest.fn(),
getStorageMetadata: jest.fn(() => ({})),
jest.mock('@librechat/agents', () => ({
Providers: {
XAI: 'xai',
DEEPSEEK: 'deepseek',
MOONSHOT: 'moonshot',
OPENROUTER: 'openrouter',
VERTEXAI: 'vertexai',
},
}));
jest.mock('librechat-data-provider', () => ({
...jest.requireActual('librechat-data-provider'),
mergeFileConfig: jest.fn(),
}));
jest.mock('librechat-data-provider', () => {
const actual = jest.requireActual('librechat-data-provider');
return {
...actual,
Providers: actual.Providers,
mergeFileConfig: jest.fn(),
};
});
jest.mock('@librechat/api', () => {
return {
sanitizeFilename: jest.fn((n) => n),
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
processAudioFile: jest.fn(),
getStorageMetadata: jest.fn(() => ({})),
getRetentionExpiry: jest.fn(() => ({})),
sweepExpiredFiles: jest.fn().mockResolvedValue({ scanned: 0, deleted: 0, failed: 0 }),
startExpiredFileSweep: jest.fn().mockReturnValue('sweep-interval'),
};
});
jest.mock('~/server/services/Files/images', () => ({
convertImage: jest.fn(),
@ -41,8 +60,12 @@ jest.mock('~/models', () => ({
createFile: jest.fn().mockResolvedValue({ file_id: 'created-file-id' }),
updateFileUsage: jest.fn(),
deleteFiles: jest.fn(),
findFileById: jest.fn(),
getConvo: jest.fn(),
getExpiredFiles: jest.fn(),
addAgentResourceFile: jest.fn().mockResolvedValue({}),
removeAgentResourceFiles: jest.fn(),
removeAgentResourceFilesFromAllAgents: jest.fn(),
}));
jest.mock('~/server/utils/getFileStrategy', () => ({
@ -69,17 +92,29 @@ jest.mock('~/server/services/Files/Audio/STTService', () => ({
STTService: { getInstance: jest.fn() },
}));
const {
getRetentionExpiry,
sweepExpiredFiles: sweepExpiredFilesWithDeps,
startExpiredFileSweep: startExpiredFileSweepWithDeps,
} = require('@librechat/api');
const {
EToolResources,
FileSources,
FileContext,
RetentionMode,
AgentCapabilities,
} = require('librechat-data-provider');
const { mergeFileConfig } = require('librechat-data-provider');
const { checkCapability } = require('~/server/services/Config');
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
const db = require('~/models');
const { processAgentFileUpload, processFileURL } = require('./process');
const {
processAgentFileUpload,
processDeleteRequest,
processFileURL,
sweepExpiredFiles,
startExpiredFileSweep,
} = require('./process');
const PDF_MIME = 'application/pdf';
const DOCX_MIME = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
@ -534,6 +569,110 @@ describe('processFileURL', () => {
);
});
it('applies retention metadata for generated images when retention mode is all', async () => {
getRetentionExpiry.mockResolvedValueOnce({
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
});
const saveURL = jest.fn().mockResolvedValue({
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
bytes: 512,
type: 'image/png',
});
const getFileURL = jest.fn();
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
await processFileURL({
fileStrategy: FileSources.cloudfront,
userId: 'user-123',
URL: 'https://example.com/image.png',
fileName: 'image.png',
basePath: 'images',
context: FileContext.image_generation,
tenantId: 'tenant-a',
req: {
user: { id: 'user-123', tenantId: 'tenant-a' },
body: {},
config: { interfaceConfig: { retentionMode: 'all' } },
},
});
expect(db.createFile).toHaveBeenCalledWith(
expect.objectContaining({
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
}),
true,
);
});
it('applies retention metadata for retained non-temporary conversations', async () => {
const saveURL = jest.fn().mockResolvedValue({
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
bytes: 512,
type: 'image/png',
});
const getFileURL = jest.fn();
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
getRetentionExpiry.mockResolvedValueOnce({
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
});
await processFileURL({
fileStrategy: FileSources.cloudfront,
userId: 'user-123',
URL: 'https://example.com/image.png',
fileName: 'image.png',
basePath: 'images',
context: FileContext.image_generation,
tenantId: 'tenant-a',
req: {
user: { id: 'user-123', tenantId: 'tenant-a' },
body: { conversationId: 'convo-123' },
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
},
});
expect(db.createFile).toHaveBeenCalledWith(
expect.objectContaining({
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
}),
true,
);
});
it('keeps expired retained conversation files on the parent expiration', async () => {
const parentExpiredAt = new Date('2020-01-01T00:00:00.000Z');
const saveURL = jest.fn().mockResolvedValue({
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
bytes: 512,
type: 'image/png',
});
const getFileURL = jest.fn();
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
getRetentionExpiry.mockResolvedValueOnce({ expiredAt: parentExpiredAt });
await processFileURL({
fileStrategy: FileSources.cloudfront,
userId: 'user-123',
URL: 'https://example.com/image.png',
fileName: 'image.png',
basePath: 'images',
context: FileContext.image_generation,
tenantId: 'tenant-a',
req: {
user: { id: 'user-123', tenantId: 'tenant-a' },
body: { conversationId: 'convo-123' },
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
},
});
expect(db.createFile).toHaveBeenCalledWith(
expect.objectContaining({
expiredAt: parentExpiredAt,
}),
true,
);
});
it('falls back to getFileURL with user and tenant context when metadata lacks filepath', async () => {
const saveURL = jest.fn().mockResolvedValue({
bytes: 256,
@ -602,3 +741,142 @@ describe('processFileURL', () => {
);
});
});
describe('processDeleteRequest', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('removes metadata when backing storage is already missing', async () => {
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
const deleteFile = jest.fn().mockRejectedValue(missingError);
getStrategyFunctions.mockReturnValue({ deleteFile });
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
const result = await processDeleteRequest({
req: {
body: {},
config: {},
user: { id: 'user-123', tenantId: 'tenant-a' },
},
files: [
{
file_id: 'expired-file',
filepath: '/images/user-123/expired.png',
source: FileSources.local,
},
],
});
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
expect(result).toEqual({ deletedFileIds: ['expired-file'], failedFileIds: [] });
});
it('does not treat unrelated not found messages as missing storage', async () => {
const deleteFile = jest.fn().mockRejectedValue(new Error('Configuration not found'));
getStrategyFunctions.mockReturnValue({ deleteFile });
const result = await processDeleteRequest({
req: {
body: {},
config: {},
user: { id: 'user-123', tenantId: 'tenant-a' },
},
files: [
{
file_id: 'expired-file',
filepath: '/images/user-123/expired.png',
source: FileSources.local,
},
],
});
expect(db.deleteFiles).not.toHaveBeenCalled();
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['expired-file'] });
});
it('throws metadata delete failures after storage deletion succeeds', async () => {
const deleteFile = jest.fn().mockResolvedValue(undefined);
const metadataError = new Error('mongo unavailable');
getStrategyFunctions.mockReturnValue({ deleteFile });
db.deleteFiles.mockRejectedValue(metadataError);
await expect(
processDeleteRequest({
req: {
body: {},
config: {},
user: { id: 'user-123', tenantId: 'tenant-a' },
},
files: [
{
file_id: 'expired-file',
filepath: '/images/user-123/expired.png',
source: FileSources.local,
},
],
}),
).rejects.toThrow('mongo unavailable');
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
expect(db.removeAgentResourceFilesFromAllAgents).not.toHaveBeenCalled();
});
});
describe('sweepExpiredFiles', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('delegates expired file sweeping to the shared package with backend dependencies', async () => {
const options = {
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
limit: 1,
};
sweepExpiredFilesWithDeps.mockResolvedValue({ scanned: 1, deleted: 1, failed: 0 });
const result = await sweepExpiredFiles(options);
expect(sweepExpiredFilesWithDeps).toHaveBeenCalledWith(
options,
expect.objectContaining({
getExpiredFiles: db.getExpiredFiles,
processDeleteRequest: expect.any(Function),
logger: expect.objectContaining({
error: expect.any(Function),
info: expect.any(Function),
warn: expect.any(Function),
}),
}),
);
expect(result).toEqual({ scanned: 1, deleted: 1, failed: 0 });
});
});
describe('startExpiredFileSweep', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('delegates background sweep startup to the shared package with system context', () => {
const options = {
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
};
const interval = startExpiredFileSweep(options);
expect(startExpiredFileSweepWithDeps).toHaveBeenCalledWith(
options,
expect.objectContaining({
sweepExpiredFiles: expect.any(Function),
runAsSystem: expect.any(Function),
logger: expect.objectContaining({
error: expect.any(Function),
info: expect.any(Function),
warn: expect.any(Function),
}),
}),
);
expect(interval).toBe('sweep-interval');
});
});

View file

@ -0,0 +1,21 @@
const { getRetentionExpiry: getRetentionExpiryWithDeps } = require('@librechat/api');
const { logger, createTempChatExpirationDate } = require('@librechat/data-schemas');
const db = require('~/models');
/**
* Returns `{ expiredAt }` when the request indicates data retention applies, otherwise `{}`.
* Spread into file data objects before calling createFile.
* @param {ServerRequest} req
* @returns {Promise<{ expiredAt?: Date | null }>}
*/
async function getRetentionExpiry(req) {
return getRetentionExpiryWithDeps(req, {
getConvo: db.getConvoRetention ?? db.getConvo,
createExpirationDate: createTempChatExpirationDate,
logger,
});
}
module.exports = {
getRetentionExpiry,
};

View file

@ -1,16 +1,26 @@
const { v4: uuidv4 } = require('uuid');
const { logger } = require('@librechat/data-schemas');
const { EModelEndpoint, Constants, openAISettings } = require('librechat-data-provider');
const {
logger,
createFallbackRetentionDate,
createTempChatExpirationDate,
} = require('@librechat/data-schemas');
const {
EModelEndpoint,
Constants,
RetentionMode,
openAISettings,
} = require('librechat-data-provider');
const { bulkIncrementTagCounts, bulkSaveConvos, bulkSaveMessages } = require('~/models');
const { FALLBACK_MODEL_BY_ENDPOINT } = require('./defaults');
/**
* Factory function for creating an instance of ImportBatchBuilder.
* @param {string} requestUserId - The ID of the user making the request.
* @param {object} [interfaceConfig] - Runtime interface config for import retention.
* @returns {ImportBatchBuilder} - The newly created ImportBatchBuilder instance.
*/
function createImportBatchBuilder(requestUserId) {
return new ImportBatchBuilder(requestUserId);
function createImportBatchBuilder(requestUserId, interfaceConfig) {
return new ImportBatchBuilder(requestUserId, interfaceConfig);
}
/**
@ -20,11 +30,36 @@ class ImportBatchBuilder {
/**
* Creates an instance of ImportBatchBuilder.
* @param {string} requestUserId - The ID of the user making the import request.
* @param {object} [interfaceConfig] - Runtime interface config for import retention.
*/
constructor(requestUserId) {
constructor(requestUserId, interfaceConfig) {
this.requestUserId = requestUserId;
this.interfaceConfig = interfaceConfig;
this.conversations = [];
this.messages = [];
this.retentionFields = undefined;
}
getRetentionFields() {
if (this.retentionFields !== undefined) {
return this.retentionFields;
}
if (this.interfaceConfig?.retentionMode !== RetentionMode.ALL) {
this.retentionFields = {};
return this.retentionFields;
}
try {
this.retentionFields = {
isTemporary: false,
expiredAt: createTempChatExpirationDate(this.interfaceConfig),
};
} catch (error) {
logger.error('[ImportBatchBuilder] Error creating import expiration date:', error);
this.retentionFields = { isTemporary: false, expiredAt: createFallbackRetentionDate() };
}
return this.retentionFields;
}
/**
@ -89,6 +124,7 @@ class ImportBatchBuilder {
overrideTimestamp: true,
endpoint: this.endpoint,
model: originalConvo.model ?? fallbackModel,
...this.getRetentionFields(),
};
convo._id && delete convo._id;
this.conversations.push(convo);
@ -161,6 +197,7 @@ class ImportBatchBuilder {
error: false,
sender,
text,
...this.getRetentionFields(),
};
message._id && delete message._id;
this.lastMessageId = newMessageId;

View file

@ -2,15 +2,16 @@ const fs = require('fs').promises;
const { resolveImportMaxFileSize } = require('@librechat/api');
const { logger } = require('@librechat/data-schemas');
const { getImporter } = require('./importers');
const { createImportBatchBuilder } = require('./importBatchBuilder');
const maxFileSize = resolveImportMaxFileSize();
/**
* Job definition for importing a conversation.
* @param {{ filepath: string, requestUserId: string, userRole?: string }} job
* @param {{ filepath: string, requestUserId: string, userRole?: string, interfaceConfig?: object }} job
*/
const importConversations = async (job) => {
const { filepath, requestUserId, userRole } = job;
const { filepath, requestUserId, userRole, interfaceConfig } = job;
try {
logger.debug(`user: ${requestUserId} | Importing conversation(s) from file...`);
@ -24,7 +25,12 @@ const importConversations = async (job) => {
const fileData = await fs.readFile(filepath, 'utf8');
const jsonData = JSON.parse(fileData);
const importer = getImporter(jsonData);
await importer(jsonData, requestUserId, undefined, userRole);
await importer(
jsonData,
requestUserId,
(userId) => createImportBatchBuilder(userId, interfaceConfig),
userRole,
);
logger.debug(`user: ${requestUserId} | Finished importing conversations`);
} catch (error) {
logger.error(`user: ${requestUserId} | Failed to import conversation: `, error);

View file

@ -3,6 +3,7 @@ const path = require('path');
const {
EModelEndpoint,
Constants,
RetentionMode,
openAISettings,
anthropicSettings,
} = require('librechat-data-provider');
@ -28,6 +29,7 @@ jest.mock('~/server/controllers/ModelController', () => ({
jest.mock('~/models', () => ({
bulkSaveConvos: jest.fn(),
bulkSaveMessages: jest.fn(),
bulkIncrementTagCounts: jest.fn(),
}));
afterEach(() => {
@ -1046,6 +1048,23 @@ describe('importLibreChatConvo', () => {
expect(result.conversation.endpoint).toBe(EModelEndpoint.openAI);
expect(result.conversation.model).toBe(openAISettings.model.default);
});
it('applies all-data retention to imported conversations and messages', () => {
const requestUserId = 'user-123';
const builder = new ImportBatchBuilder(requestUserId, {
retentionMode: RetentionMode.ALL,
temporaryChatRetention: 24,
});
builder.startConversation(EModelEndpoint.openAI);
const message = builder.addUserMessage('Retained import');
const result = builder.finishConversation('Imported retained chat');
expect(message.isTemporary).toBe(false);
expect(message.expiredAt).toBeInstanceOf(Date);
expect(result.conversation.isTemporary).toBe(false);
expect(result.conversation.expiredAt).toBeInstanceOf(Date);
expect(result.conversation.expiredAt).toBe(message.expiredAt);
});
});
});