mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 04:07:05 +00:00
🧯 fix: Harden Data Retention Semantics (#13049)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: support data retention for normal chats Add retentionMode config variable supporting "all" and "temporary" values. When "all" is set, data retention applies to all chats, not just temporary ones. Adds isTemporary field to conversations for proper filtering. Adapted to new TS method files in packages/data-schemas since upstream moved models out of api/models/. Based on danny-avila/LibreChat#10532 Co-Authored-By: WhammyLeaf <233105313+WhammyLeaf@users.noreply.github.com> (cherry picked from commit30109e90b0) * feat: extend data retention to files, tool calls, and shared links Add expiredAt field and TTL indexes to file, toolCall, and share schemas. Set expiredAt on tool calls, shared links, and file uploads when retentionMode is "all" or chat is temporary. (cherry picked from commit48973752d3) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lint/test (cherry picked from commit310c514e6a) * fix: address code review feedback for data retention PR Critical: - Fix BookmarkMenu crash: restore optional chaining on conversation - Fix migration hazard: backward-compatible sidebar filter that also checks expiredAt for documents without isTemporary field Major: - Add logging to getRetentionExpiry error path, align with tools.js - Add tests for retentionMode: ALL in saveConvo and saveMessage - Fix share route: apply expiredAt for temporary chats too by querying the conversation's isTemporary flag server-side - Add assertions for getRetentionExpiry mocks in process tests Minor: - Fix ChatRoute isTemporaryChat to be strictly boolean via Boolean() - Fix stale test description (expired -> temporary) - Comment out retentionMode default in example yaml - Simplify verbose if/else to isTemporary === true - Add compound index on { user: 1, isTemporary: 1 } - Remove narrating comment from process.spec.js Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> (cherry picked from commit6bad535f90) * chore: fix typescript (cherry picked from commit826527a46b) * fix: lint (cherry picked from commit77817e80ea) * fix: use mockSanitizeArtifactPath in retention test The 'getRetentionExpiry is called with the request object' test referenced an undefined `mockSanitizeFilename` identifier, breaking both lint (no-undef) and the test suite. Use the existing `mockSanitizeArtifactPath` mock that the surrounding tests already use, since `processCodeOutput` calls `sanitizeArtifactPath` (not `sanitizeFilename`) before invoking `getRetentionExpiry`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit52ea2da66d) * fix: forward isTemporary from client for retention on file uploads and tool calls Server-side `getRetentionExpiry` (file uploads) and the tool-call controller both read `req.body.isTemporary`, but the file upload multipart form and the tool-call payload did not include that field. In `retentionMode: temporary` (default), files uploaded and tool calls created from temporary chats were therefore retained indefinitely. Forward the Recoil `isTemporary` flag in both client paths so the existing server checks can fire correctly. `ToolParams` gains an optional `isTemporary` field. Addresses Codex P1 review feedback on PR #29. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit7e937df05a) * test: stub store.isTemporary in useFileHandling test mocks Previous commit added `useRecoilValue(store.isTemporary)` to the hook. The test file mocks `~/store` with only `ephemeralAgentByConvoId` and does not stub `useRecoilValue`, so all 7 cases threw "Invalid argument to useRecoilValue: expected an atom or selector but got undefined". Add a stub default export with `isTemporary` and a `useRecoilValue` mock returning `false`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commiteb1609537d) * fix: harden data retention semantics * fix: provide sweep request context for expired files * fix: preserve temporary flags in all-retention updates * fix: honor assistant versions in retention sweeps * fix: retain non-temporary flags in all mode * fix: hide expired retained records * fix: propagate retained conversation expiry * fix: refresh meili retention cutoff * fix: prevent overlapping file sweeps * fix: show legacy retained conversations * fix: index legacy retained records * fix: harden retention cleanup edge cases * fix: count failed file storage sweeps * fix: preserve legacy temporary retention * fix: assign retention sweep worker deterministically * fix: hide expired shared links on reads * fix: prevent retention refresh after parent expiry * fix: break code output retention import cycle * fix: harden retention review findings * fix: ignore expired share duplicates * fix: reject expired retained share creation * fix: harden retention review edge cases * fix: address retention audit findings * fix: enforce expired conversation shares in all retention * fix: scope temporary upload flag to chat files * fix: address retention review findings * fix: address codex retention review findings * fix: tighten missing storage detection * test: remove unused file process spec bindings --------- Co-authored-by: WhammyLeaf <233105313+WhammyLeaf@users.noreply.github.com> Co-authored-by: Aron Gates <aron@muonspace.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
2418f854df
commit
9dd062e42e
67 changed files with 3299 additions and 171 deletions
|
|
@ -4,7 +4,11 @@ const { v4: uuidv4 } = require('uuid');
|
|||
const { ProxyAgent, fetch } = require('undici');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { Tool } = require('@librechat/agents/langchain/tools');
|
||||
const { getImageBasename, extractBaseURL } = require('@librechat/api');
|
||||
const {
|
||||
getImageBasename,
|
||||
extractBaseURL,
|
||||
createMinimalRetentionRequest,
|
||||
} = require('@librechat/api');
|
||||
const { FileContext, ContentTypes } = require('librechat-data-provider');
|
||||
|
||||
const dalle3JsonSchema = {
|
||||
|
|
@ -49,6 +53,7 @@ class DALLE3 extends Tool {
|
|||
|
||||
this.userId = fields.userId;
|
||||
this.tenantId = fields.req?.user?.tenantId;
|
||||
this.retentionRequest = createMinimalRetentionRequest(fields.req);
|
||||
this.fileStrategy = fields.fileStrategy;
|
||||
/** @type {boolean} */
|
||||
this.isAgent = fields.isAgent;
|
||||
|
|
@ -230,6 +235,7 @@ Error Message: ${error.message}`);
|
|||
fileStrategy: this.fileStrategy,
|
||||
context: FileContext.image_generation,
|
||||
tenantId: this.tenantId,
|
||||
req: this.retentionRequest,
|
||||
});
|
||||
|
||||
if (this.returnMetadata) {
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ const { v4: uuidv4 } = require('uuid');
|
|||
const { logger } = require('@librechat/data-schemas');
|
||||
const { HttpsProxyAgent } = require('https-proxy-agent');
|
||||
const { Tool } = require('@librechat/agents/langchain/tools');
|
||||
const { createMinimalRetentionRequest } = require('@librechat/api');
|
||||
const { FileContext, ContentTypes } = require('librechat-data-provider');
|
||||
|
||||
const fluxApiJsonSchema = {
|
||||
|
|
@ -110,6 +111,7 @@ class FluxAPI extends Tool {
|
|||
|
||||
this.userId = fields.userId;
|
||||
this.tenantId = fields.req?.user?.tenantId;
|
||||
this.retentionRequest = createMinimalRetentionRequest(fields.req);
|
||||
this.fileStrategy = fields.fileStrategy;
|
||||
|
||||
/** @type {boolean} **/
|
||||
|
|
@ -343,6 +345,7 @@ class FluxAPI extends Tool {
|
|||
basePath: 'images',
|
||||
context: FileContext.image_generation,
|
||||
tenantId: this.tenantId,
|
||||
req: this.retentionRequest,
|
||||
});
|
||||
|
||||
logger.debug('[FluxAPI] Image saved to path:', result.filepath);
|
||||
|
|
@ -574,6 +577,7 @@ class FluxAPI extends Tool {
|
|||
basePath: 'images',
|
||||
context: FileContext.image_generation,
|
||||
tenantId: this.tenantId,
|
||||
req: this.retentionRequest,
|
||||
});
|
||||
|
||||
logger.debug('[FluxAPI] Finetuned image saved to path:', result.filepath);
|
||||
|
|
|
|||
|
|
@ -100,11 +100,21 @@ describe('image tools - agent mode ToolMessage format', () => {
|
|||
});
|
||||
|
||||
it('keeps tenant context without retaining the request object', () => {
|
||||
const req = { user: { tenantId: 'tenant-a' }, socket: {} };
|
||||
const req = {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
socket: {},
|
||||
};
|
||||
const dalle = new DALLE3({ isAgent: false, processFileURL: jest.fn(), req });
|
||||
|
||||
expect(dalle.tenantId).toBe('tenant-a');
|
||||
expect(dalle.req).toBeUndefined();
|
||||
expect(dalle.retentionRequest).toEqual({
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
});
|
||||
});
|
||||
|
||||
it('invoke() returns ToolMessage with base64 in artifact, not serialized in content', async () => {
|
||||
|
|
@ -181,11 +191,90 @@ describe('image tools - agent mode ToolMessage format', () => {
|
|||
});
|
||||
|
||||
it('keeps tenant context without retaining the request object', () => {
|
||||
const req = { user: { tenantId: 'tenant-a' }, socket: {} };
|
||||
const req = {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
socket: {},
|
||||
};
|
||||
const flux = new FluxAPI({ isAgent: false, processFileURL: jest.fn(), req });
|
||||
|
||||
expect(flux.tenantId).toBe('tenant-a');
|
||||
expect(flux.req).toBeUndefined();
|
||||
expect(flux.retentionRequest).toEqual({
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
});
|
||||
});
|
||||
|
||||
it('passes minimal retention context when saving generated images', async () => {
|
||||
const processFileURL = jest.fn().mockResolvedValue({ filepath: '/images/generated.png' });
|
||||
const req = {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
socket: {},
|
||||
};
|
||||
const flux = new FluxAPI({
|
||||
isAgent: false,
|
||||
processFileURL,
|
||||
req,
|
||||
userId: 'user-1',
|
||||
fileStrategy: 'local',
|
||||
});
|
||||
const invokePromise = flux.invoke(
|
||||
makeToolCall('flux', { prompt: 'a box', endpoint: '/v1/flux-dev' }),
|
||||
);
|
||||
await jest.runAllTimersAsync();
|
||||
await invokePromise;
|
||||
|
||||
expect(processFileURL).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
req: {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('passes minimal retention context when saving finetuned generated images', async () => {
|
||||
const processFileURL = jest.fn().mockResolvedValue({ filepath: '/images/generated.png' });
|
||||
const req = {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
socket: {},
|
||||
};
|
||||
const flux = new FluxAPI({
|
||||
isAgent: false,
|
||||
processFileURL,
|
||||
req,
|
||||
userId: 'user-1',
|
||||
fileStrategy: 'local',
|
||||
});
|
||||
const invokePromise = flux.invoke(
|
||||
makeToolCall('flux', {
|
||||
action: 'generate_finetuned',
|
||||
prompt: 'a box',
|
||||
finetune_id: 'ft-abc123',
|
||||
endpoint: '/v1/flux-pro-finetuned',
|
||||
}),
|
||||
);
|
||||
await jest.runAllTimersAsync();
|
||||
await invokePromise;
|
||||
|
||||
expect(processFileURL).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
req: {
|
||||
user: { id: 'user-1', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-1', isTemporary: 'true' },
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('invoke() returns ToolMessage with base64 in artifact, not serialized in content', async () => {
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
const { logger } = require('@librechat/data-schemas');
|
||||
const { logger, buildRetentionVisibilityFilter } = require('@librechat/data-schemas');
|
||||
|
||||
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
|
|
@ -26,7 +26,10 @@ async function batchResetMeiliFlags(collection) {
|
|||
try {
|
||||
while (hasMore) {
|
||||
const docs = await collection
|
||||
.find({ expiredAt: null, _meiliIndex: { $ne: false } }, { projection: { _id: 1 } })
|
||||
.find(
|
||||
{ ...buildRetentionVisibilityFilter(), _meiliIndex: { $ne: false } },
|
||||
{ projection: { _id: 1 } },
|
||||
)
|
||||
.limit(BATCH_SIZE)
|
||||
.toArray();
|
||||
|
||||
|
|
|
|||
|
|
@ -83,6 +83,60 @@ describe('batchResetMeiliFlags', () => {
|
|||
expect(expiredDoc._meiliIndex).toBe(true);
|
||||
});
|
||||
|
||||
it('should reset active non-temporary documents with expiredAt set for all-data retention', async () => {
|
||||
const retentionDate = new Date(Date.now() + 60 * 60 * 1000);
|
||||
await testCollection.insertMany([
|
||||
{
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
isTemporary: false,
|
||||
expiredAt: retentionDate,
|
||||
_meiliIndex: true,
|
||||
},
|
||||
{
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
isTemporary: true,
|
||||
expiredAt: retentionDate,
|
||||
_meiliIndex: true,
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
|
||||
const retainedDoc = await testCollection.findOne({ isTemporary: false });
|
||||
const temporaryDoc = await testCollection.findOne({ isTemporary: true });
|
||||
expect(retainedDoc._meiliIndex).toBe(false);
|
||||
expect(temporaryDoc._meiliIndex).toBe(true);
|
||||
});
|
||||
|
||||
it('should not reset expired non-temporary documents with expiredAt set for all-data retention', async () => {
|
||||
const retentionDate = new Date(Date.now() - 60 * 60 * 1000);
|
||||
await testCollection.insertMany([
|
||||
{
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
isTemporary: false,
|
||||
expiredAt: retentionDate,
|
||||
_meiliIndex: true,
|
||||
},
|
||||
{
|
||||
_id: new mongoose.Types.ObjectId(),
|
||||
isTemporary: false,
|
||||
expiredAt: null,
|
||||
_meiliIndex: true,
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await batchResetMeiliFlags(testCollection);
|
||||
|
||||
expect(result).toBe(1);
|
||||
|
||||
const expiredDoc = await testCollection.findOne({ expiredAt: retentionDate });
|
||||
const permanentDoc = await testCollection.findOne({ expiredAt: null });
|
||||
expect(expiredDoc._meiliIndex).toBe(true);
|
||||
expect(permanentDoc._meiliIndex).toBe(false);
|
||||
});
|
||||
|
||||
it('should not modify documents with _meiliIndex: false', async () => {
|
||||
await testCollection.insertMany([
|
||||
{ _id: new mongoose.Types.ObjectId(), expiredAt: null, _meiliIndex: false },
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ jest.mock('@librechat/api', () => ({
|
|||
}));
|
||||
|
||||
jest.mock('~/server/services/Files/process', () => ({
|
||||
processDeleteRequest: jest.fn().mockResolvedValue(undefined),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Config', () => ({
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ jest.mock('~/server/services/Config/getCachedTools', () => ({
|
|||
}));
|
||||
|
||||
jest.mock('~/server/services/Files/process', () => ({
|
||||
processDeleteRequest: jest.fn(),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Config', () => ({
|
||||
|
|
|
|||
|
|
@ -127,7 +127,7 @@ function stubDeletionMocks() {
|
|||
mockDeleteUserById.mockResolvedValue();
|
||||
mockDeleteAllSharedLinks.mockResolvedValue();
|
||||
mockGetFiles.mockResolvedValue([]);
|
||||
mockProcessDeleteRequest.mockResolvedValue();
|
||||
mockProcessDeleteRequest.mockResolvedValue({ deletedFileIds: [], failedFileIds: [] });
|
||||
mockDeleteFiles.mockResolvedValue();
|
||||
mockDeleteToolCalls.mockResolvedValue();
|
||||
mockDeleteUserAgents.mockResolvedValue();
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ jest.mock('~/server/services/Config/getCachedTools', () => ({
|
|||
}));
|
||||
|
||||
jest.mock('~/server/services/Files/process', () => ({
|
||||
processDeleteRequest: jest.fn(),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Config', () => ({
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ const {
|
|||
} = require('librechat-data-provider');
|
||||
const { getRoleByName, createToolCall, getToolCallsByConvo, getMessage } = require('~/models');
|
||||
const { processFileURL, uploadImageBuffer } = require('~/server/services/Files/process');
|
||||
const { getRetentionExpiry } = require('~/server/services/Files/retention');
|
||||
const { processCodeOutput, runPreviewFinalize } = require('~/server/services/Files/Code/process');
|
||||
const { loadAuthValues } = require('~/server/services/Tools/credentials');
|
||||
const { loadTools } = require('~/app/clients/tools/util');
|
||||
|
|
@ -167,6 +168,7 @@ const callTool = async (req, res) => {
|
|||
conversationId,
|
||||
result: content,
|
||||
user: req.user.id,
|
||||
...(await getRetentionExpiry(req)),
|
||||
};
|
||||
|
||||
if (!artifact || !artifact.files || toolId !== Tools.execute_code) {
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ const {
|
|||
const { connectDb, indexSync } = require('~/db');
|
||||
const initializeOAuthReconnectManager = require('./services/initializeOAuthReconnectManager');
|
||||
const createValidateImageRequest = require('./middleware/validateImageRequest');
|
||||
const { startExpiredFileSweep } = require('./services/Files/process');
|
||||
const { jwtLogin, ldapLogin, passportLogin } = require('~/strategies');
|
||||
const { updateInterfacePermissions: updateInterfacePerms } = require('@librechat/api');
|
||||
const {
|
||||
|
|
@ -139,8 +140,32 @@ if (cluster.isMaster) {
|
|||
logger.info(`Spawning ${workers} workers to simulate multi-pod environment`);
|
||||
|
||||
let activeWorkers = 0;
|
||||
const listeningWorkers = new Set();
|
||||
let retentionSweepWorkerId = null;
|
||||
const startTime = Date.now();
|
||||
|
||||
const assignRetentionSweepWorker = () => {
|
||||
if (retentionSweepWorkerId && cluster.workers[retentionSweepWorkerId]) {
|
||||
return;
|
||||
}
|
||||
|
||||
const connectedWorkers = Object.values(cluster.workers).filter(
|
||||
(worker) => worker && worker.isConnected(),
|
||||
);
|
||||
const availableWorkers = connectedWorkers.filter((worker) => listeningWorkers.has(worker.id));
|
||||
const workerPool = availableWorkers.length > 0 ? availableWorkers : connectedWorkers;
|
||||
const retentionSweepWorker = workerPool[workerPool.length - 1];
|
||||
if (!retentionSweepWorker) {
|
||||
return;
|
||||
}
|
||||
|
||||
retentionSweepWorkerId = retentionSweepWorker.id;
|
||||
logger.info(
|
||||
wrapLogMessage(`Worker ${retentionSweepWorker.process.pid} assigned to file-retention sweep`),
|
||||
);
|
||||
retentionSweepWorker.send({ type: 'file-retention-sweep-worker' });
|
||||
};
|
||||
|
||||
/** Flush Redis cache before starting workers */
|
||||
flushRedisCache()
|
||||
.then(() => {
|
||||
|
|
@ -162,19 +187,29 @@ if (cluster.isMaster) {
|
|||
`Worker ${worker.process.pid} is online (${activeWorkers}/${workers}) after ${uptime}s`,
|
||||
);
|
||||
|
||||
/** Notify the last worker to perform one-time initialization tasks */
|
||||
/** Assign one worker for process-wide background jobs */
|
||||
if (activeWorkers === workers) {
|
||||
const allWorkers = Object.values(cluster.workers);
|
||||
const lastWorker = allWorkers[allWorkers.length - 1];
|
||||
if (lastWorker) {
|
||||
logger.info(wrapLogMessage(`All ${workers} workers are online`));
|
||||
lastWorker.send({ type: 'last-worker' });
|
||||
}
|
||||
logger.info(wrapLogMessage(`All ${workers} workers are online`));
|
||||
}
|
||||
});
|
||||
|
||||
cluster.on('listening', (worker) => {
|
||||
listeningWorkers.add(worker.id);
|
||||
if (
|
||||
listeningWorkers.size === workers ||
|
||||
(!retentionSweepWorkerId && activeWorkers >= workers)
|
||||
) {
|
||||
assignRetentionSweepWorker();
|
||||
}
|
||||
});
|
||||
|
||||
cluster.on('exit', (worker, code, signal) => {
|
||||
activeWorkers--;
|
||||
listeningWorkers.delete(worker.id);
|
||||
if (worker.id === retentionSweepWorkerId) {
|
||||
retentionSweepWorkerId = null;
|
||||
assignRetentionSweepWorker();
|
||||
}
|
||||
logger.error(
|
||||
`Worker ${worker.process.pid} died (${activeWorkers}/${workers}). Code: ${code}, Signal: ${signal}`,
|
||||
);
|
||||
|
|
@ -202,6 +237,32 @@ if (cluster.isMaster) {
|
|||
* Each worker runs a full Express server instance
|
||||
*/
|
||||
const app = express();
|
||||
/**
|
||||
* The master may assign the sweep worker before or after this worker has
|
||||
* loaded app config. These flags join the IPC assignment with config
|
||||
* availability and ensure the background sweep starts only once.
|
||||
*/
|
||||
let shouldStartExpiredFileSweep = false;
|
||||
let expiredFileSweepOptions = null;
|
||||
let expiredFileSweepStarted = false;
|
||||
|
||||
const startExpiredFileSweepOnce = () => {
|
||||
if (!shouldStartExpiredFileSweep || expiredFileSweepStarted || !expiredFileSweepOptions) {
|
||||
return;
|
||||
}
|
||||
|
||||
expiredFileSweepStarted = true;
|
||||
startExpiredFileSweep(expiredFileSweepOptions);
|
||||
};
|
||||
|
||||
/** Handle inter-process messages from master */
|
||||
process.on('message', (msg) => {
|
||||
if (msg.type === 'file-retention-sweep-worker') {
|
||||
shouldStartExpiredFileSweep = true;
|
||||
logger.info(wrapLogMessage(`Worker ${process.pid} is assigned file-retention sweep`));
|
||||
startExpiredFileSweepOnce();
|
||||
}
|
||||
});
|
||||
|
||||
const startServer = async () => {
|
||||
logger.info(`Worker ${process.pid} initializing...`);
|
||||
|
|
@ -233,6 +294,8 @@ if (cluster.isMaster) {
|
|||
/** Initialize app configuration */
|
||||
const appConfig = await getAppConfig();
|
||||
initializeFileStorage(appConfig);
|
||||
expiredFileSweepOptions = { appConfig, loadAppConfig: getAppConfig };
|
||||
startExpiredFileSweepOnce();
|
||||
await performStartupChecks(appConfig);
|
||||
await updateInterfacePerms({ appConfig, getRoleByName, updateAccessPermissions });
|
||||
|
||||
|
|
@ -390,19 +453,6 @@ if (cluster.isMaster) {
|
|||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
/** Handle inter-process messages from master */
|
||||
process.on('message', async (msg) => {
|
||||
if (msg.type === 'last-worker') {
|
||||
logger.info(
|
||||
wrapLogMessage(
|
||||
`Worker ${process.pid} is the last worker and can perform special initialization tasks`,
|
||||
),
|
||||
);
|
||||
/** Add any one-time initialization tasks here */
|
||||
/** For example: scheduled jobs, cleanup tasks, etc. */
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
startServer().catch((err) => {
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ const {
|
|||
const initializeOAuthReconnectManager = require('./services/initializeOAuthReconnectManager');
|
||||
const { capabilityContextMiddleware } = require('./middleware/roles/capabilities');
|
||||
const createValidateImageRequest = require('./middleware/validateImageRequest');
|
||||
const { startExpiredFileSweep } = require('./services/Files/process');
|
||||
const { jwtLogin, ldapLogin, passportLogin } = require('~/strategies');
|
||||
const { checkMigrations } = require('./services/start/migration');
|
||||
const optionalJwtAuth = require('./middleware/optionalJwtAuth');
|
||||
|
|
@ -89,6 +90,7 @@ const startServer = async () => {
|
|||
});
|
||||
const appConfig = await getAppConfig({ baseOnly: true });
|
||||
initializeFileStorage(appConfig);
|
||||
startExpiredFileSweep({ appConfig, loadAppConfig: getAppConfig });
|
||||
await runAsSystem(async () => {
|
||||
await performStartupChecks(appConfig);
|
||||
await updateInterfacePermissions({ appConfig, getRoleByName, updateAccessPermissions });
|
||||
|
|
|
|||
262
api/server/routes/__tests__/share.spec.js
Normal file
262
api/server/routes/__tests__/share.spec.js
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
const express = require('express');
|
||||
const request = require('supertest');
|
||||
const mongoose = require('mongoose');
|
||||
|
||||
const mockGetSharedLinkExpiration = jest.fn();
|
||||
|
||||
jest.mock('@librechat/api', () => ({
|
||||
isEnabled: jest.fn(() => true),
|
||||
getSharedLinkExpiration: (...args) => mockGetSharedLinkExpiration(...args),
|
||||
isActiveExpirationDate: jest.fn((expiredAt) => expiredAt > new Date()),
|
||||
}));
|
||||
|
||||
jest.mock('@librechat/data-schemas', () => ({
|
||||
logger: { error: jest.fn() },
|
||||
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
|
||||
}));
|
||||
|
||||
jest.mock('librechat-data-provider', () => ({
|
||||
RetentionMode: {
|
||||
ALL: 'all',
|
||||
TEMPORARY: 'temporary',
|
||||
},
|
||||
}));
|
||||
|
||||
jest.mock('mongoose', () => ({
|
||||
models: {
|
||||
Conversation: {
|
||||
findOne: jest.fn(),
|
||||
},
|
||||
SharedLink: {
|
||||
findOne: jest.fn(),
|
||||
},
|
||||
},
|
||||
}));
|
||||
|
||||
jest.mock('~/models', () => ({
|
||||
getSharedMessages: jest.fn(),
|
||||
createSharedLink: jest.fn(),
|
||||
updateSharedLink: jest.fn(),
|
||||
deleteSharedLink: jest.fn(),
|
||||
getSharedLinks: jest.fn(),
|
||||
getSharedLink: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/middleware/requireJwtAuth', () => (req, res, next) => next());
|
||||
|
||||
const { RetentionMode } = require('librechat-data-provider');
|
||||
const { createTempChatExpirationDate, logger } = require('@librechat/data-schemas');
|
||||
const { createSharedLink, updateSharedLink } = require('~/models');
|
||||
const shareRouter = require('../share');
|
||||
|
||||
const activeExpiration = new Date('2030-01-01T00:00:00.000Z');
|
||||
const expiredExpiration = new Date('2020-01-01T00:00:00.000Z');
|
||||
|
||||
const lean = (value) => ({
|
||||
lean: jest.fn().mockResolvedValue(value),
|
||||
});
|
||||
|
||||
const buildApp = ({ retentionMode = RetentionMode.TEMPORARY } = {}) => {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.user = { id: 'user-123' };
|
||||
req.config = { interfaceConfig: { retentionMode } };
|
||||
next();
|
||||
});
|
||||
app.use('/api/share', shareRouter);
|
||||
return app;
|
||||
};
|
||||
|
||||
describe('share routes retention', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it('expires new shares for retained non-temporary conversations', async () => {
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
|
||||
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
|
||||
|
||||
const response = await request(buildApp())
|
||||
.post('/api/share/convo-123')
|
||||
.send({ targetMessageId: 'msg-123' });
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(mockGetSharedLinkExpiration).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
conversationId: 'convo-123',
|
||||
req: expect.objectContaining({ user: { id: 'user-123' } }),
|
||||
}),
|
||||
expect.objectContaining({
|
||||
getConvo: expect.any(Function),
|
||||
createExpirationDate: createTempChatExpirationDate,
|
||||
logger,
|
||||
}),
|
||||
);
|
||||
const [, dependencies] = mockGetSharedLinkExpiration.mock.calls[0];
|
||||
mongoose.models.Conversation.findOne.mockReturnValue(lean({ expiredAt: activeExpiration }));
|
||||
await dependencies.getConvo('user-123', 'convo-123');
|
||||
expect(mongoose.models.Conversation.findOne).toHaveBeenCalledWith(
|
||||
{ conversationId: 'convo-123', user: 'user-123' },
|
||||
'isTemporary expiredAt',
|
||||
);
|
||||
expect(createSharedLink).toHaveBeenCalledWith(
|
||||
'user-123',
|
||||
'convo-123',
|
||||
'msg-123',
|
||||
new Date('2030-01-01T00:00:00.000Z'),
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects new shares when the retained conversation expired', async () => {
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
|
||||
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
|
||||
|
||||
const response = await request(buildApp())
|
||||
.post('/api/share/convo-123')
|
||||
.send({ targetMessageId: 'msg-123' });
|
||||
|
||||
expect(response.status).toBe(404);
|
||||
expect(createSharedLink).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects new shares for expired conversations in all retention mode', async () => {
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
|
||||
createSharedLink.mockResolvedValue({ shareId: 'share-123' });
|
||||
|
||||
const response = await request(buildApp({ retentionMode: RetentionMode.ALL }))
|
||||
.post('/api/share/convo-123')
|
||||
.send({ targetMessageId: 'msg-123' });
|
||||
|
||||
expect(response.status).toBe(404);
|
||||
expect(createSharedLink).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('expires updated shares for retained non-temporary conversations', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp()).patch('/api/share/share-123');
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(mongoose.models.SharedLink.findOne).toHaveBeenCalledWith(
|
||||
{ shareId: 'share-123', user: 'user-123' },
|
||||
'conversationId',
|
||||
);
|
||||
expect(mockGetSharedLinkExpiration).toHaveBeenCalledTimes(1);
|
||||
expect(mockGetSharedLinkExpiration).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
conversationId: 'convo-123',
|
||||
req: expect.objectContaining({ user: { id: 'user-123' } }),
|
||||
}),
|
||||
expect.objectContaining({
|
||||
getConvo: expect.any(Function),
|
||||
createExpirationDate: createTempChatExpirationDate,
|
||||
logger,
|
||||
}),
|
||||
);
|
||||
expect(updateSharedLink).toHaveBeenCalledWith(
|
||||
'user-123',
|
||||
'share-123',
|
||||
undefined,
|
||||
new Date('2030-01-01T00:00:00.000Z'),
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects updated shares when the retained conversation expired', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp()).patch('/api/share/share-123');
|
||||
|
||||
expect(response.status).toBe(404);
|
||||
expect(updateSharedLink).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects updated shares for expired conversations in all retention mode', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(expiredExpiration);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp({ retentionMode: RetentionMode.ALL })).patch(
|
||||
'/api/share/share-123',
|
||||
);
|
||||
|
||||
expect(response.status).toBe(404);
|
||||
expect(mongoose.models.SharedLink.findOne).toHaveBeenCalledWith(
|
||||
{ shareId: 'share-123', user: 'user-123' },
|
||||
'conversationId',
|
||||
);
|
||||
expect(updateSharedLink).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('clears updated share expiration when the conversation is no longer retained', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(null);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp()).patch('/api/share/share-123');
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, null);
|
||||
});
|
||||
|
||||
it('preserves updated share expiration when the conversation cannot be found', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(undefined);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp()).patch('/api/share/share-123');
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, undefined);
|
||||
});
|
||||
|
||||
it('clears updated share expiration when creating a new expiration throws', async () => {
|
||||
const error = new Error('bad config');
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockImplementationOnce(async (_input, dependencies) => {
|
||||
dependencies.logger.error('[getSharedLinkExpiration] Error creating expiration date:', error);
|
||||
return null;
|
||||
});
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456' });
|
||||
|
||||
const response = await request(buildApp()).patch('/api/share/share-123');
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(logger.error).toHaveBeenCalledWith(
|
||||
'[getSharedLinkExpiration] Error creating expiration date:',
|
||||
error,
|
||||
);
|
||||
expect(updateSharedLink).toHaveBeenCalledWith('user-123', 'share-123', undefined, null);
|
||||
});
|
||||
|
||||
it('updates share target message while applying retention expiration', async () => {
|
||||
mongoose.models.SharedLink.findOne.mockReturnValue(lean({ conversationId: 'convo-123' }));
|
||||
mockGetSharedLinkExpiration.mockResolvedValue(activeExpiration);
|
||||
updateSharedLink.mockResolvedValue({ shareId: 'share-456', targetMessageId: 'msg-456' });
|
||||
|
||||
const response = await request(buildApp())
|
||||
.patch('/api/share/share-123')
|
||||
.send({ targetMessageId: 'msg-456' });
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(updateSharedLink).toHaveBeenCalledWith(
|
||||
'user-123',
|
||||
'share-123',
|
||||
'msg-456',
|
||||
new Date('2030-01-01T00:00:00.000Z'),
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects non-string target message updates', async () => {
|
||||
const response = await request(buildApp())
|
||||
.patch('/api/share/share-123')
|
||||
.send({ targetMessageId: 123 });
|
||||
|
||||
expect(response.status).toBe(400);
|
||||
expect(updateSharedLink).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
|
@ -276,6 +276,7 @@ router.post(
|
|||
filepath: req.file.path,
|
||||
requestUserId: req.user.id,
|
||||
userRole: req.user.role,
|
||||
interfaceConfig: req.config?.interfaceConfig,
|
||||
});
|
||||
res.status(201).json({ message: 'Conversation(s) imported successfully' });
|
||||
} catch (error) {
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ const { createAgent, createFile } = require('~/models');
|
|||
|
||||
// Only mock the external dependencies that we don't want to test
|
||||
jest.mock('~/server/services/Files/process', () => ({
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({}),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
filterFile: jest.fn(),
|
||||
processFileUpload: jest.fn(),
|
||||
processAgentFileUpload: jest.fn().mockImplementation(async ({ res }) => {
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ const { createAgent, createFile } = require('~/models');
|
|||
|
||||
// Only mock the external dependencies that we don't want to test
|
||||
jest.mock('~/server/services/Files/process', () => ({
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({}),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
filterFile: jest.fn(),
|
||||
processFileUpload: jest.fn(),
|
||||
processAgentFileUpload: jest.fn(),
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ jest.mock('~/models', () => ({
|
|||
jest.mock('~/server/services/Files/process', () => ({
|
||||
filterFile: jest.fn(),
|
||||
processFileUpload: jest.fn(),
|
||||
processDeleteRequest: jest.fn(),
|
||||
processDeleteRequest: jest.fn().mockResolvedValue({ deletedFileIds: [], failedFileIds: [] }),
|
||||
processAgentFileUpload: jest.fn(),
|
||||
}));
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
const mongoose = require('mongoose');
|
||||
const express = require('express');
|
||||
const { isEnabled } = require('@librechat/api');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { isEnabled, isActiveExpirationDate, getSharedLinkExpiration } = require('@librechat/api');
|
||||
const { logger, createTempChatExpirationDate } = require('@librechat/data-schemas');
|
||||
const {
|
||||
getSharedMessages,
|
||||
createSharedLink,
|
||||
|
|
@ -12,6 +13,22 @@ const {
|
|||
const requireJwtAuth = require('~/server/middleware/requireJwtAuth');
|
||||
const router = express.Router();
|
||||
|
||||
const resolveSharedLinkExpiration = (req, conversationId) =>
|
||||
getSharedLinkExpiration(
|
||||
{ req, conversationId },
|
||||
{
|
||||
getConvo: async (userId, sourceConversationId) => {
|
||||
const Conversation = mongoose.models.Conversation;
|
||||
return Conversation.findOne(
|
||||
{ conversationId: sourceConversationId, user: userId },
|
||||
'isTemporary expiredAt',
|
||||
).lean();
|
||||
},
|
||||
createExpirationDate: createTempChatExpirationDate,
|
||||
logger,
|
||||
},
|
||||
);
|
||||
|
||||
/**
|
||||
* Shared messages
|
||||
*/
|
||||
|
|
@ -99,7 +116,17 @@ router.get('/link/:conversationId', requireJwtAuth, async (req, res) => {
|
|||
router.post('/:conversationId', requireJwtAuth, async (req, res) => {
|
||||
try {
|
||||
const { targetMessageId } = req.body;
|
||||
const created = await createSharedLink(req.user.id, req.params.conversationId, targetMessageId);
|
||||
const expiredAt = await resolveSharedLinkExpiration(req, req.params.conversationId);
|
||||
if (expiredAt != null && !isActiveExpirationDate(expiredAt)) {
|
||||
return res.status(404).end();
|
||||
}
|
||||
|
||||
const created = await createSharedLink(
|
||||
req.user.id,
|
||||
req.params.conversationId,
|
||||
targetMessageId,
|
||||
expiredAt,
|
||||
);
|
||||
if (created) {
|
||||
res.status(200).json(created);
|
||||
} else {
|
||||
|
|
@ -118,7 +145,25 @@ router.patch('/:shareId', requireJwtAuth, async (req, res) => {
|
|||
return res.status(400).json({ message: 'targetMessageId must be a string' });
|
||||
}
|
||||
|
||||
const updatedShare = await updateSharedLink(req.user.id, req.params.shareId, targetMessageId);
|
||||
let expiredAt;
|
||||
const SharedLink = mongoose.models.SharedLink;
|
||||
const existing = await SharedLink.findOne(
|
||||
{ shareId: req.params.shareId, user: req.user.id },
|
||||
'conversationId',
|
||||
).lean();
|
||||
if (existing?.conversationId) {
|
||||
expiredAt = await resolveSharedLinkExpiration(req, existing.conversationId);
|
||||
}
|
||||
if (expiredAt != null && !isActiveExpirationDate(expiredAt)) {
|
||||
return res.status(404).end();
|
||||
}
|
||||
|
||||
const updatedShare = await updateSharedLink(
|
||||
req.user.id,
|
||||
req.params.shareId,
|
||||
targetMessageId,
|
||||
expiredAt,
|
||||
);
|
||||
if (updatedShare) {
|
||||
res.status(200).json(updatedShare);
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -92,6 +92,11 @@ jest.mock('~/server/utils', () => ({
|
|||
determineFileType: jest.fn().mockResolvedValue({ mime: 'text/csv' }),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Files/retention', () => ({
|
||||
getRetentionExpiry: jest.fn(() => ({})),
|
||||
}));
|
||||
|
||||
const { getRetentionExpiry } = require('~/server/services/Files/retention');
|
||||
const { createFile } = require('~/models');
|
||||
const { processCodeOutput } = require('../process');
|
||||
|
||||
|
|
@ -143,6 +148,12 @@ describe('processCodeOutput path traversal protection', () => {
|
|||
expect(fileArg.tenantId).toBe('tenantA');
|
||||
});
|
||||
|
||||
test('getRetentionExpiry is called with the request object', async () => {
|
||||
mockSanitizeArtifactPath.mockReturnValueOnce('output.csv');
|
||||
await processCodeOutput({ ...baseParams, name: 'output.csv' });
|
||||
expect(getRetentionExpiry).toHaveBeenCalledWith(baseParams.req);
|
||||
});
|
||||
|
||||
test('sanitized name is used for image file records', async () => {
|
||||
const { convertImage } = require('~/server/services/Files/images/convert');
|
||||
convertImage.mockResolvedValueOnce({
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ const { filterFilesByAgentAccess } = require('~/server/services/Files/permission
|
|||
const { createFile, getFiles, updateFile, claimCodeFile } = require('~/models');
|
||||
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
|
||||
const { convertImage } = require('~/server/services/Files/images/convert');
|
||||
const { getRetentionExpiry } = require('~/server/services/Files/retention');
|
||||
const { determineFileType } = require('~/server/utils');
|
||||
|
||||
const axios = createAxiosInstance();
|
||||
|
|
@ -463,6 +464,7 @@ const processCodeOutput = async ({
|
|||
source: appConfig.fileStrategy,
|
||||
context: FileContext.execute_code,
|
||||
metadata: { codeEnvRef },
|
||||
...(await getRetentionExpiry(req)),
|
||||
};
|
||||
await createFile(file, true);
|
||||
return { file: Object.assign(file, { messageId, toolCallId }) };
|
||||
|
|
@ -565,6 +567,7 @@ const processCodeOutput = async ({
|
|||
context: FileContext.execute_code,
|
||||
usage: isUpdate ? (claimed.usage ?? 0) + 1 : 1,
|
||||
createdAt: isUpdate ? claimed.createdAt : formattedDate,
|
||||
...(await getRetentionExpiry(req)),
|
||||
};
|
||||
|
||||
if (expectsPreview) {
|
||||
|
|
|
|||
|
|
@ -137,6 +137,10 @@ jest.mock('~/server/services/Files/images/convert', () => ({
|
|||
convertImage: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Files/retention', () => ({
|
||||
getRetentionExpiry: jest.fn(() => ({})),
|
||||
}));
|
||||
|
||||
// Mock determineFileType
|
||||
jest.mock('~/server/utils', () => ({
|
||||
determineFileType: jest.fn(),
|
||||
|
|
@ -145,6 +149,7 @@ jest.mock('~/server/utils', () => ({
|
|||
const http = require('http');
|
||||
const https = require('https');
|
||||
const { createFile, getFiles } = require('~/models');
|
||||
const { getRetentionExpiry } = require('~/server/services/Files/retention');
|
||||
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
|
||||
const { convertImage } = require('~/server/services/Files/images/convert');
|
||||
const { determineFileType } = require('~/server/utils');
|
||||
|
|
@ -233,6 +238,7 @@ describe('Code Process', () => {
|
|||
|
||||
expect(result.file_id).toBe('mock-uuid-1234');
|
||||
expect(result.usage).toBe(1);
|
||||
expect(getRetentionExpiry).toHaveBeenCalledWith(baseParams.req);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -18,12 +18,14 @@ const {
|
|||
getEndpointFileConfig,
|
||||
documentParserMimeTypes,
|
||||
} = require('librechat-data-provider');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { logger, runAsSystem } = require('@librechat/data-schemas');
|
||||
const {
|
||||
sanitizeFilename,
|
||||
parseText,
|
||||
processAudioFile,
|
||||
getStorageMetadata,
|
||||
sweepExpiredFiles: sweepExpiredFilesWithDeps,
|
||||
startExpiredFileSweep: startExpiredFileSweepWithDeps,
|
||||
} = require('@librechat/api');
|
||||
const {
|
||||
convertImage,
|
||||
|
|
@ -36,6 +38,7 @@ const { loadAuthValues } = require('~/server/services/Tools/credentials');
|
|||
const { getFileStrategy } = require('~/server/utils/getFileStrategy');
|
||||
const { checkCapability } = require('~/server/services/Config');
|
||||
const { LB_QueueAsyncCall } = require('~/server/utils/queue');
|
||||
const { getRetentionExpiry } = require('./retention');
|
||||
const { getStrategyFunctions } = require('./strategies');
|
||||
const { determineFileType } = require('~/server/utils');
|
||||
const { STTService } = require('./Audio/STTService');
|
||||
|
|
@ -64,6 +67,17 @@ const createSanitizedUploadWrapper = (uploadFunction) => {
|
|||
};
|
||||
};
|
||||
|
||||
const isMissingStorageError = (err) => {
|
||||
const code = err?.code ?? err?.status ?? err?.statusCode ?? err?.response?.status;
|
||||
if ([404, '404', 'ENOENT', 'NoSuchKey', 'NotFound', 'ResourceNotFound'].includes(code)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return /(?:file|object|blob|key|resource) (?:not found|does not exist)|no such (?:file|key)/i.test(
|
||||
String(err?.message ?? ''),
|
||||
);
|
||||
};
|
||||
|
||||
/**
|
||||
* Enqueues the delete operation to the leaky bucket queue if necessary, or adds it directly to promises.
|
||||
*
|
||||
|
|
@ -72,10 +86,19 @@ const createSanitizedUploadWrapper = (uploadFunction) => {
|
|||
* @param {MongoFile} params.file - The file object to delete.
|
||||
* @param {Function} params.deleteFile - The delete file function.
|
||||
* @param {Promise[]} params.promises - The array of promises to await.
|
||||
* @param {string[]} params.resolvedFileIds - The array of promises to await.
|
||||
* @param {Set<string>} params.resolvedFileIds - File IDs whose storage delete succeeded.
|
||||
* @param {Set<string>} params.failedFileIds - File IDs whose storage delete failed.
|
||||
* @param {OpenAI | undefined} [params.openai] - If an OpenAI file, the initialized OpenAI client.
|
||||
*/
|
||||
function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileIds, openai }) {
|
||||
function enqueueDeleteOperation({
|
||||
req,
|
||||
file,
|
||||
deleteFile,
|
||||
promises,
|
||||
resolvedFileIds,
|
||||
failedFileIds,
|
||||
openai,
|
||||
}) {
|
||||
if (checkOpenAIStorage(file.source)) {
|
||||
// Enqueue to leaky bucket
|
||||
promises.push(
|
||||
|
|
@ -85,10 +108,17 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
|
|||
[],
|
||||
(err, result) => {
|
||||
if (err) {
|
||||
if (isMissingStorageError(err)) {
|
||||
resolvedFileIds.add(file.file_id);
|
||||
logger.warn('File storage was already missing during delete', err);
|
||||
resolve(result);
|
||||
return;
|
||||
}
|
||||
failedFileIds.add(file.file_id);
|
||||
logger.error('Error deleting file from OpenAI source', err);
|
||||
reject(err);
|
||||
} else {
|
||||
resolvedFileIds.push(file.file_id);
|
||||
resolvedFileIds.add(file.file_id);
|
||||
resolve(result);
|
||||
}
|
||||
},
|
||||
|
|
@ -99,8 +129,14 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
|
|||
// Add directly to promises
|
||||
promises.push(
|
||||
deleteFile(req, file)
|
||||
.then(() => resolvedFileIds.push(file.file_id))
|
||||
.then(() => resolvedFileIds.add(file.file_id))
|
||||
.catch((err) => {
|
||||
if (isMissingStorageError(err)) {
|
||||
resolvedFileIds.add(file.file_id);
|
||||
logger.warn('File storage was already missing during delete', err);
|
||||
return;
|
||||
}
|
||||
failedFileIds.add(file.file_id);
|
||||
logger.error('Error deleting file', err);
|
||||
return Promise.reject(err);
|
||||
}),
|
||||
|
|
@ -121,11 +157,13 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
|
|||
* @param {string} [params.req.body.assistant_id] - The assistant ID if file uploaded is associated to an assistant.
|
||||
* @param {string} [params.req.body.tool_resource] - The tool resource if assistant file uploaded is associated to a tool resource.
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
* @returns {Promise<{ deletedFileIds: string[], failedFileIds: string[] }>}
|
||||
* @throws {Error} When storage deletion cannot be scheduled or file metadata cleanup fails.
|
||||
*/
|
||||
const processDeleteRequest = async ({ req, files }) => {
|
||||
const appConfig = req.config;
|
||||
const resolvedFileIds = [];
|
||||
const resolvedFileIds = new Set();
|
||||
const failedFileIds = new Set();
|
||||
const deletionMethods = {};
|
||||
const promises = [];
|
||||
|
||||
|
|
@ -167,7 +205,7 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
}
|
||||
|
||||
if (source === FileSources.text) {
|
||||
resolvedFileIds.push(file.file_id);
|
||||
resolvedFileIds.add(file.file_id);
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -198,6 +236,7 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
deleteFile: deletionMethods[source],
|
||||
promises,
|
||||
resolvedFileIds,
|
||||
failedFileIds,
|
||||
openai,
|
||||
});
|
||||
continue;
|
||||
|
|
@ -209,7 +248,15 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
}
|
||||
|
||||
deletionMethods[source] = deleteFile;
|
||||
enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileIds, openai });
|
||||
enqueueDeleteOperation({
|
||||
req,
|
||||
file,
|
||||
deleteFile,
|
||||
promises,
|
||||
resolvedFileIds,
|
||||
failedFileIds,
|
||||
openai,
|
||||
});
|
||||
}
|
||||
|
||||
if (agentFiles.length > 0) {
|
||||
|
|
@ -222,17 +269,60 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
}
|
||||
|
||||
await Promise.allSettled(promises);
|
||||
await db.deleteFiles(resolvedFileIds);
|
||||
|
||||
if (resolvedFileIds.length > 0) {
|
||||
const deletedFileIds = [...resolvedFileIds];
|
||||
let metadataDeletedFileIds = deletedFileIds;
|
||||
if (deletedFileIds.length > 0) {
|
||||
try {
|
||||
await db.removeAgentResourceFilesFromAllAgents({ file_ids: resolvedFileIds });
|
||||
await db.deleteFiles(deletedFileIds);
|
||||
} catch (error) {
|
||||
logger.error('Error cleaning up orphaned agent file references', error);
|
||||
logger.error('Error deleting file metadata after storage deletion', error);
|
||||
deletedFileIds.forEach((fileId) => failedFileIds.add(fileId));
|
||||
metadataDeletedFileIds = [];
|
||||
throw error;
|
||||
}
|
||||
if (metadataDeletedFileIds.length > 0) {
|
||||
try {
|
||||
await db.removeAgentResourceFilesFromAllAgents({ file_ids: metadataDeletedFileIds });
|
||||
} catch (error) {
|
||||
logger.error('Error cleaning up orphaned agent file references', error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
deletedFileIds: metadataDeletedFileIds,
|
||||
failedFileIds: [...failedFileIds],
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Deletes expired file storage before removing the corresponding File records.
|
||||
*
|
||||
* Mongo TTL indexes delete only the metadata document, so file retention uses
|
||||
* this application sweep for records with `expiredAt` instead.
|
||||
*
|
||||
* @param {object} params
|
||||
* @param {AppConfig} params.appConfig
|
||||
* @param {number} [params.limit]
|
||||
* @param {() => Promise<AppConfig>} [params.loadAppConfig]
|
||||
* @returns {Promise<{ scanned: number, deleted: number, failed: number }>}
|
||||
*/
|
||||
async function sweepExpiredFiles(options = {}) {
|
||||
return sweepExpiredFilesWithDeps(options, {
|
||||
getExpiredFiles: db.getExpiredFiles,
|
||||
processDeleteRequest,
|
||||
logger,
|
||||
});
|
||||
}
|
||||
|
||||
function startExpiredFileSweep(options = {}) {
|
||||
return startExpiredFileSweepWithDeps(options, {
|
||||
sweepExpiredFiles,
|
||||
runAsSystem,
|
||||
logger,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Processes a file URL using a specified file handling strategy. This function accepts a strategy name,
|
||||
* fetches the corresponding file processing functions (for saving and retrieving file URLs), and then
|
||||
|
|
@ -251,6 +341,7 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
* @param {string} params.basePath - The base path or directory where the file will be saved or retrieved from.
|
||||
* @param {FileContext} params.context - The context of the file (e.g., 'avatar', 'image_generation', etc.)
|
||||
* @param {string} [params.tenantId] - Optional tenant identifier for tenant-prefixed storage paths.
|
||||
* @param {ServerRequest} [params.req] - Request context used to apply data retention metadata.
|
||||
* @returns {Promise<MongoFile>} A promise that resolves to the DB representation (MongoFile)
|
||||
* of the processed file. It throws an error if the file processing fails at any stage.
|
||||
*/
|
||||
|
|
@ -262,6 +353,7 @@ const processFileURL = async ({
|
|||
basePath,
|
||||
context,
|
||||
tenantId,
|
||||
req,
|
||||
}) => {
|
||||
const { saveURL, getFileURL } = getStrategyFunctions(fileStrategy);
|
||||
try {
|
||||
|
|
@ -305,6 +397,7 @@ const processFileURL = async ({
|
|||
source: fileStrategy,
|
||||
type,
|
||||
context,
|
||||
...(await getRetentionExpiry(req)),
|
||||
tenantId,
|
||||
width: dimensions.width,
|
||||
height: dimensions.height,
|
||||
|
|
@ -355,6 +448,7 @@ const processImageFile = async ({ req, res, metadata, returnFile = false }) => {
|
|||
context: FileContext.message_attachment,
|
||||
source,
|
||||
type: `image/${appConfig.imageOutputType}`,
|
||||
...(await getRetentionExpiry(req)),
|
||||
width,
|
||||
height,
|
||||
tenantId: req.user.tenantId,
|
||||
|
|
@ -415,6 +509,7 @@ const uploadImageBuffer = async ({ req, context, metadata = {}, resize = true })
|
|||
source,
|
||||
type,
|
||||
width,
|
||||
...(await getRetentionExpiry(req)),
|
||||
height,
|
||||
tenantId: req.user.tenantId,
|
||||
},
|
||||
|
|
@ -517,6 +612,7 @@ const processFileUpload = async ({ req, res, metadata }) => {
|
|||
context: isAssistantUpload ? FileContext.assistants : FileContext.message_attachment,
|
||||
model: isAssistantUpload ? req.body.model : undefined,
|
||||
type: file.mimetype,
|
||||
...(await getRetentionExpiry(req)),
|
||||
embedded,
|
||||
source,
|
||||
height,
|
||||
|
|
@ -631,20 +727,24 @@ const processAgentFileUpload = async ({ req, res, metadata }) => {
|
|||
`Extracted text from "${file.originalname}" exceeds the 15MB storage limit (${Math.round(textBytes / megabyte)}MB). Try a shorter document.`,
|
||||
);
|
||||
}
|
||||
const fileInfo = removeNullishValues({
|
||||
text,
|
||||
bytes,
|
||||
file_id,
|
||||
temp_file_id,
|
||||
user: req.user.id,
|
||||
type,
|
||||
filepath: filepath ?? file.path,
|
||||
source: FileSources.text,
|
||||
filename: file.originalname,
|
||||
model: messageAttachment ? undefined : req.body.model,
|
||||
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
|
||||
tenantId: req.user.tenantId,
|
||||
});
|
||||
const retentionExpiry = await getRetentionExpiry(req);
|
||||
const fileInfo = {
|
||||
...removeNullishValues({
|
||||
text,
|
||||
bytes,
|
||||
file_id,
|
||||
temp_file_id,
|
||||
user: req.user.id,
|
||||
type,
|
||||
filepath: filepath ?? file.path,
|
||||
source: FileSources.text,
|
||||
filename: file.originalname,
|
||||
model: messageAttachment ? undefined : req.body.model,
|
||||
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
|
||||
tenantId: req.user.tenantId,
|
||||
}),
|
||||
...retentionExpiry,
|
||||
};
|
||||
|
||||
if (!messageAttachment && tool_resource) {
|
||||
await db.addAgentResourceFile({
|
||||
|
|
@ -825,24 +925,28 @@ const processAgentFileUpload = async ({ req, res, metadata }) => {
|
|||
});
|
||||
}
|
||||
|
||||
const fileInfo = removeNullishValues({
|
||||
user: req.user.id,
|
||||
file_id,
|
||||
temp_file_id,
|
||||
bytes,
|
||||
filepath,
|
||||
...storageMetadata,
|
||||
filename: filename ?? sanitizeFilename(file.originalname),
|
||||
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
|
||||
model: messageAttachment ? undefined : req.body.model,
|
||||
metadata: fileInfoMetadata,
|
||||
type: file.mimetype,
|
||||
embedded,
|
||||
source,
|
||||
height,
|
||||
width,
|
||||
tenantId: req.user.tenantId,
|
||||
});
|
||||
const retentionExpiry = await getRetentionExpiry(req);
|
||||
const fileInfo = {
|
||||
...removeNullishValues({
|
||||
user: req.user.id,
|
||||
file_id,
|
||||
temp_file_id,
|
||||
bytes,
|
||||
filepath,
|
||||
...storageMetadata,
|
||||
filename: filename ?? sanitizeFilename(file.originalname),
|
||||
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
|
||||
model: messageAttachment ? undefined : req.body.model,
|
||||
metadata: fileInfoMetadata,
|
||||
type: file.mimetype,
|
||||
embedded,
|
||||
source,
|
||||
height,
|
||||
width,
|
||||
tenantId: req.user.tenantId,
|
||||
}),
|
||||
...retentionExpiry,
|
||||
};
|
||||
|
||||
const result = await db.createFile(fileInfo, true);
|
||||
|
||||
|
|
@ -887,6 +991,7 @@ const processOpenAIFile = async ({
|
|||
source,
|
||||
model: openai.req.body.model,
|
||||
filename: originalName ?? file_id,
|
||||
...(await getRetentionExpiry(openai.req)),
|
||||
tenantId: openai.req?.user?.tenantId,
|
||||
};
|
||||
|
||||
|
|
@ -931,9 +1036,14 @@ const processOpenAIImageOutput = async ({ req, buffer, file_id, filename, fileEx
|
|||
context: FileContext.assistants_output,
|
||||
file_id,
|
||||
filename,
|
||||
...(await getRetentionExpiry(req)),
|
||||
tenantId: req.user.tenantId,
|
||||
};
|
||||
db.createFile(file, true);
|
||||
try {
|
||||
await db.createFile(file, true);
|
||||
} catch (error) {
|
||||
logger.warn('Error saving OpenAI image output file metadata', error);
|
||||
}
|
||||
return file;
|
||||
};
|
||||
|
||||
|
|
@ -1091,6 +1201,7 @@ async function saveBase64Image(
|
|||
user: req.user.id,
|
||||
bytes: image.bytes,
|
||||
width: image.width,
|
||||
...(await getRetentionExpiry(req)),
|
||||
height: image.height,
|
||||
tenantId: req.user.tenantId,
|
||||
},
|
||||
|
|
@ -1182,6 +1293,8 @@ module.exports = {
|
|||
saveBase64Image,
|
||||
processImageFile,
|
||||
uploadImageBuffer,
|
||||
sweepExpiredFiles,
|
||||
startExpiredFileSweep,
|
||||
processFileUpload,
|
||||
processDeleteRequest,
|
||||
processAgentFileUpload,
|
||||
|
|
|
|||
|
|
@ -1,22 +1,41 @@
|
|||
jest.mock('uuid', () => ({ v4: jest.fn(() => 'mock-uuid') }));
|
||||
|
||||
jest.mock('@librechat/data-schemas', () => ({
|
||||
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn() },
|
||||
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn(), info: jest.fn() },
|
||||
runAsSystem: jest.fn((fn) => fn()),
|
||||
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
|
||||
}));
|
||||
|
||||
jest.mock('@librechat/agents', () => ({}));
|
||||
|
||||
jest.mock('@librechat/api', () => ({
|
||||
sanitizeFilename: jest.fn((n) => n),
|
||||
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
|
||||
processAudioFile: jest.fn(),
|
||||
getStorageMetadata: jest.fn(() => ({})),
|
||||
jest.mock('@librechat/agents', () => ({
|
||||
Providers: {
|
||||
XAI: 'xai',
|
||||
DEEPSEEK: 'deepseek',
|
||||
MOONSHOT: 'moonshot',
|
||||
OPENROUTER: 'openrouter',
|
||||
VERTEXAI: 'vertexai',
|
||||
},
|
||||
}));
|
||||
|
||||
jest.mock('librechat-data-provider', () => ({
|
||||
...jest.requireActual('librechat-data-provider'),
|
||||
mergeFileConfig: jest.fn(),
|
||||
}));
|
||||
jest.mock('librechat-data-provider', () => {
|
||||
const actual = jest.requireActual('librechat-data-provider');
|
||||
return {
|
||||
...actual,
|
||||
Providers: actual.Providers,
|
||||
mergeFileConfig: jest.fn(),
|
||||
};
|
||||
});
|
||||
|
||||
jest.mock('@librechat/api', () => {
|
||||
return {
|
||||
sanitizeFilename: jest.fn((n) => n),
|
||||
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
|
||||
processAudioFile: jest.fn(),
|
||||
getStorageMetadata: jest.fn(() => ({})),
|
||||
getRetentionExpiry: jest.fn(() => ({})),
|
||||
sweepExpiredFiles: jest.fn().mockResolvedValue({ scanned: 0, deleted: 0, failed: 0 }),
|
||||
startExpiredFileSweep: jest.fn().mockReturnValue('sweep-interval'),
|
||||
};
|
||||
});
|
||||
|
||||
jest.mock('~/server/services/Files/images', () => ({
|
||||
convertImage: jest.fn(),
|
||||
|
|
@ -41,8 +60,12 @@ jest.mock('~/models', () => ({
|
|||
createFile: jest.fn().mockResolvedValue({ file_id: 'created-file-id' }),
|
||||
updateFileUsage: jest.fn(),
|
||||
deleteFiles: jest.fn(),
|
||||
findFileById: jest.fn(),
|
||||
getConvo: jest.fn(),
|
||||
getExpiredFiles: jest.fn(),
|
||||
addAgentResourceFile: jest.fn().mockResolvedValue({}),
|
||||
removeAgentResourceFiles: jest.fn(),
|
||||
removeAgentResourceFilesFromAllAgents: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/utils/getFileStrategy', () => ({
|
||||
|
|
@ -69,17 +92,29 @@ jest.mock('~/server/services/Files/Audio/STTService', () => ({
|
|||
STTService: { getInstance: jest.fn() },
|
||||
}));
|
||||
|
||||
const {
|
||||
getRetentionExpiry,
|
||||
sweepExpiredFiles: sweepExpiredFilesWithDeps,
|
||||
startExpiredFileSweep: startExpiredFileSweepWithDeps,
|
||||
} = require('@librechat/api');
|
||||
const {
|
||||
EToolResources,
|
||||
FileSources,
|
||||
FileContext,
|
||||
RetentionMode,
|
||||
AgentCapabilities,
|
||||
} = require('librechat-data-provider');
|
||||
const { mergeFileConfig } = require('librechat-data-provider');
|
||||
const { checkCapability } = require('~/server/services/Config');
|
||||
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
|
||||
const db = require('~/models');
|
||||
const { processAgentFileUpload, processFileURL } = require('./process');
|
||||
const {
|
||||
processAgentFileUpload,
|
||||
processDeleteRequest,
|
||||
processFileURL,
|
||||
sweepExpiredFiles,
|
||||
startExpiredFileSweep,
|
||||
} = require('./process');
|
||||
|
||||
const PDF_MIME = 'application/pdf';
|
||||
const DOCX_MIME = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
|
||||
|
|
@ -534,6 +569,110 @@ describe('processFileURL', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('applies retention metadata for generated images when retention mode is all', async () => {
|
||||
getRetentionExpiry.mockResolvedValueOnce({
|
||||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||||
});
|
||||
const saveURL = jest.fn().mockResolvedValue({
|
||||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||||
bytes: 512,
|
||||
type: 'image/png',
|
||||
});
|
||||
const getFileURL = jest.fn();
|
||||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||||
|
||||
await processFileURL({
|
||||
fileStrategy: FileSources.cloudfront,
|
||||
userId: 'user-123',
|
||||
URL: 'https://example.com/image.png',
|
||||
fileName: 'image.png',
|
||||
basePath: 'images',
|
||||
context: FileContext.image_generation,
|
||||
tenantId: 'tenant-a',
|
||||
req: {
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
body: {},
|
||||
config: { interfaceConfig: { retentionMode: 'all' } },
|
||||
},
|
||||
});
|
||||
|
||||
expect(db.createFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||||
}),
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it('applies retention metadata for retained non-temporary conversations', async () => {
|
||||
const saveURL = jest.fn().mockResolvedValue({
|
||||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||||
bytes: 512,
|
||||
type: 'image/png',
|
||||
});
|
||||
const getFileURL = jest.fn();
|
||||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||||
getRetentionExpiry.mockResolvedValueOnce({
|
||||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||||
});
|
||||
|
||||
await processFileURL({
|
||||
fileStrategy: FileSources.cloudfront,
|
||||
userId: 'user-123',
|
||||
URL: 'https://example.com/image.png',
|
||||
fileName: 'image.png',
|
||||
basePath: 'images',
|
||||
context: FileContext.image_generation,
|
||||
tenantId: 'tenant-a',
|
||||
req: {
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-123' },
|
||||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||||
},
|
||||
});
|
||||
|
||||
expect(db.createFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||||
}),
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps expired retained conversation files on the parent expiration', async () => {
|
||||
const parentExpiredAt = new Date('2020-01-01T00:00:00.000Z');
|
||||
const saveURL = jest.fn().mockResolvedValue({
|
||||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||||
bytes: 512,
|
||||
type: 'image/png',
|
||||
});
|
||||
const getFileURL = jest.fn();
|
||||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt: parentExpiredAt });
|
||||
|
||||
await processFileURL({
|
||||
fileStrategy: FileSources.cloudfront,
|
||||
userId: 'user-123',
|
||||
URL: 'https://example.com/image.png',
|
||||
fileName: 'image.png',
|
||||
basePath: 'images',
|
||||
context: FileContext.image_generation,
|
||||
tenantId: 'tenant-a',
|
||||
req: {
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
body: { conversationId: 'convo-123' },
|
||||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||||
},
|
||||
});
|
||||
|
||||
expect(db.createFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
expiredAt: parentExpiredAt,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it('falls back to getFileURL with user and tenant context when metadata lacks filepath', async () => {
|
||||
const saveURL = jest.fn().mockResolvedValue({
|
||||
bytes: 256,
|
||||
|
|
@ -602,3 +741,142 @@ describe('processFileURL', () => {
|
|||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processDeleteRequest', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it('removes metadata when backing storage is already missing', async () => {
|
||||
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
|
||||
const deleteFile = jest.fn().mockRejectedValue(missingError);
|
||||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||||
|
||||
const result = await processDeleteRequest({
|
||||
req: {
|
||||
body: {},
|
||||
config: {},
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
},
|
||||
files: [
|
||||
{
|
||||
file_id: 'expired-file',
|
||||
filepath: '/images/user-123/expired.png',
|
||||
source: FileSources.local,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||||
expect(result).toEqual({ deletedFileIds: ['expired-file'], failedFileIds: [] });
|
||||
});
|
||||
|
||||
it('does not treat unrelated not found messages as missing storage', async () => {
|
||||
const deleteFile = jest.fn().mockRejectedValue(new Error('Configuration not found'));
|
||||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||||
|
||||
const result = await processDeleteRequest({
|
||||
req: {
|
||||
body: {},
|
||||
config: {},
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
},
|
||||
files: [
|
||||
{
|
||||
file_id: 'expired-file',
|
||||
filepath: '/images/user-123/expired.png',
|
||||
source: FileSources.local,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['expired-file'] });
|
||||
});
|
||||
|
||||
it('throws metadata delete failures after storage deletion succeeds', async () => {
|
||||
const deleteFile = jest.fn().mockResolvedValue(undefined);
|
||||
const metadataError = new Error('mongo unavailable');
|
||||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||||
db.deleteFiles.mockRejectedValue(metadataError);
|
||||
|
||||
await expect(
|
||||
processDeleteRequest({
|
||||
req: {
|
||||
body: {},
|
||||
config: {},
|
||||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||||
},
|
||||
files: [
|
||||
{
|
||||
file_id: 'expired-file',
|
||||
filepath: '/images/user-123/expired.png',
|
||||
source: FileSources.local,
|
||||
},
|
||||
],
|
||||
}),
|
||||
).rejects.toThrow('mongo unavailable');
|
||||
|
||||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||||
expect(db.removeAgentResourceFilesFromAllAgents).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('sweepExpiredFiles', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it('delegates expired file sweeping to the shared package with backend dependencies', async () => {
|
||||
const options = {
|
||||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||||
limit: 1,
|
||||
};
|
||||
sweepExpiredFilesWithDeps.mockResolvedValue({ scanned: 1, deleted: 1, failed: 0 });
|
||||
|
||||
const result = await sweepExpiredFiles(options);
|
||||
|
||||
expect(sweepExpiredFilesWithDeps).toHaveBeenCalledWith(
|
||||
options,
|
||||
expect.objectContaining({
|
||||
getExpiredFiles: db.getExpiredFiles,
|
||||
processDeleteRequest: expect.any(Function),
|
||||
logger: expect.objectContaining({
|
||||
error: expect.any(Function),
|
||||
info: expect.any(Function),
|
||||
warn: expect.any(Function),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
expect(result).toEqual({ scanned: 1, deleted: 1, failed: 0 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('startExpiredFileSweep', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it('delegates background sweep startup to the shared package with system context', () => {
|
||||
const options = {
|
||||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||||
};
|
||||
|
||||
const interval = startExpiredFileSweep(options);
|
||||
|
||||
expect(startExpiredFileSweepWithDeps).toHaveBeenCalledWith(
|
||||
options,
|
||||
expect.objectContaining({
|
||||
sweepExpiredFiles: expect.any(Function),
|
||||
runAsSystem: expect.any(Function),
|
||||
logger: expect.objectContaining({
|
||||
error: expect.any(Function),
|
||||
info: expect.any(Function),
|
||||
warn: expect.any(Function),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
expect(interval).toBe('sweep-interval');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
21
api/server/services/Files/retention.js
Normal file
21
api/server/services/Files/retention.js
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
const { getRetentionExpiry: getRetentionExpiryWithDeps } = require('@librechat/api');
|
||||
const { logger, createTempChatExpirationDate } = require('@librechat/data-schemas');
|
||||
const db = require('~/models');
|
||||
|
||||
/**
|
||||
* Returns `{ expiredAt }` when the request indicates data retention applies, otherwise `{}`.
|
||||
* Spread into file data objects before calling createFile.
|
||||
* @param {ServerRequest} req
|
||||
* @returns {Promise<{ expiredAt?: Date | null }>}
|
||||
*/
|
||||
async function getRetentionExpiry(req) {
|
||||
return getRetentionExpiryWithDeps(req, {
|
||||
getConvo: db.getConvoRetention ?? db.getConvo,
|
||||
createExpirationDate: createTempChatExpirationDate,
|
||||
logger,
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
getRetentionExpiry,
|
||||
};
|
||||
|
|
@ -1,16 +1,26 @@
|
|||
const { v4: uuidv4 } = require('uuid');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { EModelEndpoint, Constants, openAISettings } = require('librechat-data-provider');
|
||||
const {
|
||||
logger,
|
||||
createFallbackRetentionDate,
|
||||
createTempChatExpirationDate,
|
||||
} = require('@librechat/data-schemas');
|
||||
const {
|
||||
EModelEndpoint,
|
||||
Constants,
|
||||
RetentionMode,
|
||||
openAISettings,
|
||||
} = require('librechat-data-provider');
|
||||
const { bulkIncrementTagCounts, bulkSaveConvos, bulkSaveMessages } = require('~/models');
|
||||
const { FALLBACK_MODEL_BY_ENDPOINT } = require('./defaults');
|
||||
|
||||
/**
|
||||
* Factory function for creating an instance of ImportBatchBuilder.
|
||||
* @param {string} requestUserId - The ID of the user making the request.
|
||||
* @param {object} [interfaceConfig] - Runtime interface config for import retention.
|
||||
* @returns {ImportBatchBuilder} - The newly created ImportBatchBuilder instance.
|
||||
*/
|
||||
function createImportBatchBuilder(requestUserId) {
|
||||
return new ImportBatchBuilder(requestUserId);
|
||||
function createImportBatchBuilder(requestUserId, interfaceConfig) {
|
||||
return new ImportBatchBuilder(requestUserId, interfaceConfig);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -20,11 +30,36 @@ class ImportBatchBuilder {
|
|||
/**
|
||||
* Creates an instance of ImportBatchBuilder.
|
||||
* @param {string} requestUserId - The ID of the user making the import request.
|
||||
* @param {object} [interfaceConfig] - Runtime interface config for import retention.
|
||||
*/
|
||||
constructor(requestUserId) {
|
||||
constructor(requestUserId, interfaceConfig) {
|
||||
this.requestUserId = requestUserId;
|
||||
this.interfaceConfig = interfaceConfig;
|
||||
this.conversations = [];
|
||||
this.messages = [];
|
||||
this.retentionFields = undefined;
|
||||
}
|
||||
|
||||
getRetentionFields() {
|
||||
if (this.retentionFields !== undefined) {
|
||||
return this.retentionFields;
|
||||
}
|
||||
|
||||
if (this.interfaceConfig?.retentionMode !== RetentionMode.ALL) {
|
||||
this.retentionFields = {};
|
||||
return this.retentionFields;
|
||||
}
|
||||
|
||||
try {
|
||||
this.retentionFields = {
|
||||
isTemporary: false,
|
||||
expiredAt: createTempChatExpirationDate(this.interfaceConfig),
|
||||
};
|
||||
} catch (error) {
|
||||
logger.error('[ImportBatchBuilder] Error creating import expiration date:', error);
|
||||
this.retentionFields = { isTemporary: false, expiredAt: createFallbackRetentionDate() };
|
||||
}
|
||||
return this.retentionFields;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -89,6 +124,7 @@ class ImportBatchBuilder {
|
|||
overrideTimestamp: true,
|
||||
endpoint: this.endpoint,
|
||||
model: originalConvo.model ?? fallbackModel,
|
||||
...this.getRetentionFields(),
|
||||
};
|
||||
convo._id && delete convo._id;
|
||||
this.conversations.push(convo);
|
||||
|
|
@ -161,6 +197,7 @@ class ImportBatchBuilder {
|
|||
error: false,
|
||||
sender,
|
||||
text,
|
||||
...this.getRetentionFields(),
|
||||
};
|
||||
message._id && delete message._id;
|
||||
this.lastMessageId = newMessageId;
|
||||
|
|
|
|||
|
|
@ -2,15 +2,16 @@ const fs = require('fs').promises;
|
|||
const { resolveImportMaxFileSize } = require('@librechat/api');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { getImporter } = require('./importers');
|
||||
const { createImportBatchBuilder } = require('./importBatchBuilder');
|
||||
|
||||
const maxFileSize = resolveImportMaxFileSize();
|
||||
|
||||
/**
|
||||
* Job definition for importing a conversation.
|
||||
* @param {{ filepath: string, requestUserId: string, userRole?: string }} job
|
||||
* @param {{ filepath: string, requestUserId: string, userRole?: string, interfaceConfig?: object }} job
|
||||
*/
|
||||
const importConversations = async (job) => {
|
||||
const { filepath, requestUserId, userRole } = job;
|
||||
const { filepath, requestUserId, userRole, interfaceConfig } = job;
|
||||
try {
|
||||
logger.debug(`user: ${requestUserId} | Importing conversation(s) from file...`);
|
||||
|
||||
|
|
@ -24,7 +25,12 @@ const importConversations = async (job) => {
|
|||
const fileData = await fs.readFile(filepath, 'utf8');
|
||||
const jsonData = JSON.parse(fileData);
|
||||
const importer = getImporter(jsonData);
|
||||
await importer(jsonData, requestUserId, undefined, userRole);
|
||||
await importer(
|
||||
jsonData,
|
||||
requestUserId,
|
||||
(userId) => createImportBatchBuilder(userId, interfaceConfig),
|
||||
userRole,
|
||||
);
|
||||
logger.debug(`user: ${requestUserId} | Finished importing conversations`);
|
||||
} catch (error) {
|
||||
logger.error(`user: ${requestUserId} | Failed to import conversation: `, error);
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ const path = require('path');
|
|||
const {
|
||||
EModelEndpoint,
|
||||
Constants,
|
||||
RetentionMode,
|
||||
openAISettings,
|
||||
anthropicSettings,
|
||||
} = require('librechat-data-provider');
|
||||
|
|
@ -28,6 +29,7 @@ jest.mock('~/server/controllers/ModelController', () => ({
|
|||
jest.mock('~/models', () => ({
|
||||
bulkSaveConvos: jest.fn(),
|
||||
bulkSaveMessages: jest.fn(),
|
||||
bulkIncrementTagCounts: jest.fn(),
|
||||
}));
|
||||
|
||||
afterEach(() => {
|
||||
|
|
@ -1046,6 +1048,23 @@ describe('importLibreChatConvo', () => {
|
|||
expect(result.conversation.endpoint).toBe(EModelEndpoint.openAI);
|
||||
expect(result.conversation.model).toBe(openAISettings.model.default);
|
||||
});
|
||||
|
||||
it('applies all-data retention to imported conversations and messages', () => {
|
||||
const requestUserId = 'user-123';
|
||||
const builder = new ImportBatchBuilder(requestUserId, {
|
||||
retentionMode: RetentionMode.ALL,
|
||||
temporaryChatRetention: 24,
|
||||
});
|
||||
builder.startConversation(EModelEndpoint.openAI);
|
||||
const message = builder.addUserMessage('Retained import');
|
||||
const result = builder.finishConversation('Imported retained chat');
|
||||
|
||||
expect(message.isTemporary).toBe(false);
|
||||
expect(message.expiredAt).toBeInstanceOf(Date);
|
||||
expect(result.conversation.isTemporary).toBe(false);
|
||||
expect(result.conversation.expiredAt).toBeInstanceOf(Date);
|
||||
expect(result.conversation.expiredAt).toBe(message.expiredAt);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue