mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-10-31 19:17:13 +00:00
fe30e3db32
* 🧹 fix: Stop Undeletable Files Starving the Retention Sweep `getExpiredFiles` returns the oldest `expiredAt` first, capped at `limit`, and `processDeleteRequest` leaves the record in place when storage deletion fails. Nothing records the failure, so the same files come back at the head of the next batch an hour later, forever: no backoff, no cap, and — once `limit` of them cannot be deleted — no file that expires afterwards is ever swept again. On the deployment behind #15511 that is ~29k stranded objects permanently occupying a 100-slot queue, which is why fixing the Code Interpreter side alone (LibreChat-AI/code-interpreter#85) would not have resumed deletion there. Failures are now recorded on the file. `deletionRetryAt` holds it back with a backoff doubling from one sweep interval to a day, and `deletionAttempts` retires it from the query entirely once it reaches `FILE_RETENTION_SWEEP_MAX_ATTEMPTS` (10, so roughly five days of retries). Both fields are absent on existing records and absence means "never attempted", so nothing already in the collection changes eligibility. The record itself is kept rather than deleted — the reference is what an operator needs to reconcile a bucket the sweep could not clear, and dropping it would restore the silence that made this leak invisible. Raising `FILE_RETENTION_SWEEP_MAX_ATTEMPTS` re-admits everything previously given up on, which is the supported way to resume once the storage-side failure is fixed; the give-up is logged with that instruction. The counter is incremented server-side with `$inc` so concurrent sweeps on separate nodes cannot overwrite each other's progress toward the cap, and a failure to record a failure is logged and skipped rather than aborting the rest of the batch. * 🧭 fix: Delete Code Environment Files Through the Route That Exists `deleteCodeEnvFile` tried `/sessions/:sid/objects/:fid` before falling back to `/files/:sid/:fid`. Both have been there since #13424, but only the second is mounted by any released codeapi — the first gained DELETE in LibreChat-AI/code-interpreter#85 — so every deletion paid a guaranteed 404 and a wasted round trip, and #15511 read that 404 as the whole bug. Call `/files/:sid/:fid` directly. It is the safe direction to collapse toward: codeapi has mounted it since its first release, so this works against older deployments as well as post-#85 ones, whereas keeping the other path would not. Collapsing the loop tightens two behaviours that only existed to serve it. A 405 now surfaces instead of being swallowed on the way to a second attempt; there is no second route to try, and a service that refuses the method should say so. A 404 is still treated as "already gone" — that is the only thing it can now mean — but it is logged rather than passed over in silence, because a 404 caused by a misconfigured base URL looks identical and this branch drops the file's metadata record either way. * 🔒 fix: Settle sweep retry state from the write, not the read Two findings from the review of0e52924. - `recordFailure` derived the attempt number by adding one to the count the batch had queried. Two nodes sweeping the same file read the same value, so both believed themselves to be the same attempt: each `$inc` landed, the stored count crossed `FILE_RETENTION_SWEEP_MAX_ATTEMPTS`, and neither caller ever saw the threshold. The file drops out of the query — the starvation guard still holds — but the give-up is never reported, and that log line is the operator's only notice, and carries the instruction for resuming. Return the count from the increment itself so every caller gets a distinct attempt number and exactly one observes the cap. The same staleness shortened the backoff, so `deferExpiredFile` now writes with `$max`: a deferral can only move later, and a node that computed a shorter delay cannot pull the file forward past one another node already committed. - The backoff doubled from a hard-coded hour while claiming to start from one sweep interval. At the default they coincide; away from it the schedule stops meaning anything — on a six-hour sweep the first three attempts all land on consecutive passes, and on a five-minute one the first retry skips twelve. Derive the base from `FILE_RETENTION_SWEEP_INTERVAL_MS`, floored at a minute so a pathologically short interval cannot spend the whole give-up budget on a transient outage. * 🧯 fix: Keep the give-up notice and the retry budget honest Four findings from the review ofee1bac5. - The give-up was reported after the deferral write, inside the same catch. Once the increment lands the counter is durable, so `getExpiredFiles` already excludes the file; a deferral that then failed left only the generic recording error and dropped the one line naming the file and saying how to resume. Report it as soon as the increment returns, and let the deferral fail on its own. - Retry deadlines were measured from `Date.now()` after the deletion I/O, but `startExpiredFileSweep` arms its interval before the sweep runs. A one-interval delay therefore expired just *after* the next scheduled pass, which skipped the file and pushed its first retry out by a whole extra interval — and the same drift applied to every delay that is an exact multiple. Anchor deadlines to the sweep's start instead. - The threshold used `>=`, so once two nodes pushed a file past the cap every one of them past it logged the give-up: one error per replica per exhausted file rather than one actionable notice. Only the attempt that lands exactly on the cap reports it. - Retry state outlived the content it described. `processCodeOutput` reuses a record for a repeated `(filename, conversationId)` — new bytes, new storage key, and a fresh `expiredAt` from `getRetentionExpiry` — while `createFile` and `updateFile` set only supplied fields. A record carried to the cap by its previous content stayed excluded from the sweep forever, stranding the new object exactly as this PR set out to prevent; a partially failed one started the new object's budget already spent. Both write paths now clear the fields: the budget belongs to the storage a record currently points at. The retry-state fixtures in `file.spec.ts` were seeded through `createFile`, which now clears them, so they drive the real methods the sweep uses. * 🎯 fix: Clear the retry budget only when a retention lifecycle starts Both findings from the review of8d26f9b, and both are consequences of that commit's reset rather than of the original change. - The reset was unconditional on every `createFile`/`updateFile`, on the reasoning that those paths write content. Two of them do not. `prepareImages{Local,Azure,Firebase}` call `updateFile({ file_id })` with nothing but the id — a TTL touch — every time an existing image is encoded for another chat, and the deferred preview uses the same method to transition `status`. Either handed a stranded record a fresh set of attempts and another give-up notice, on repeat, defeating the cap for the files most likely to be stranded. Gate it on the write actually setting `expiredAt`. The budget belongs to a retention lifecycle: that is the write which starts a new one, it is what `processCodeOutput` supplies when it repurposes a record, and a record with no retention deadline is never swept, so the fields are inert there anyway. - Both retry writes bumped `updatedAt`. `processCodeOutput` falls back to `updatedAt` as the writer-order stamp for records that predate `metadata.sourceDispatchedAt`, so a failed sweep landing mid-harvest read as a newer content writer and the harvest dropped its attachment. Mark them `timestamps: false`, for the reason `claimCodeFile` already does: bookkeeping is not a content write. * 🅿️ refactor: Park exhausted files instead of excluding them Three review rounds in a row found defects in how the give-up cap interacts with record reuse, each in the fix for the last. That is a design error, not a bug list: a permanent exclusion has to be bound precisely to the content lifecycle it was recorded against, and File records outlive their content. `processCodeOutput` repurposes a row for a repeated `(filename, conversationId)`, `createFile`/`updateFile` set only supplied fields, and `getRetentionExpiry` returns `{}` on a lookup failure so the row inherits its old deadline — three separate ways for bookkeeping to survive into a lifecycle it does not describe, each needing its own guard, and the two retry writes needing to be lifecycle-conditional on top. Remove the category instead. `deletionRetryAt` becomes the sweep's only hold, and reaching `FILE_RETENTION_SWEEP_MAX_ATTEMPTS` parks the file for a month rather than excluding it. The bound on the batch is the same — a stranded file costs one slot a month instead of one an hour — but a deadline that outlives its content can only delay the next object, never lose it, so nothing outside the sweep has to reason about this state at all. That deletes more than it adds: - `createFile` and `updateFile` go back to their original form. No reset, so no question of which writes install content, and `prepareImages*` and the deferred preview stop mattering here. - `getExpiredFiles` loses `maxAttempts` and its `$and`; eligibility is one `$or` on the deadline. - The interleaving race between the increment and the deferral degrades from a stranded object to a delayed one. Working through a large backlog is throughput-bound either way: every attempt costs a slot in the bounded batch, so N stranded files need N × `FILE_RETENTION_SWEEP_MAX_ATTEMPTS` passes to settle. Lower that limit when recovering a deployment that has accumulated many.
1884 lines
66 KiB
JavaScript
1884 lines
66 KiB
JavaScript
jest.mock('uuid', () => ({ v4: jest.fn(() => 'mock-uuid') }));
|
||
|
||
jest.mock('@librechat/data-schemas', () => ({
|
||
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn(), info: jest.fn() },
|
||
runAsSystem: jest.fn((fn) => fn()),
|
||
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
|
||
}));
|
||
|
||
jest.mock('@librechat/agents', () => ({
|
||
Providers: {
|
||
XAI: 'xai',
|
||
DEEPSEEK: 'deepseek',
|
||
MOONSHOT: 'moonshot',
|
||
OPENROUTER: 'openrouter',
|
||
VERTEXAI: 'vertexai',
|
||
},
|
||
}));
|
||
|
||
jest.mock('librechat-data-provider', () => {
|
||
const actual = jest.requireActual('librechat-data-provider');
|
||
return {
|
||
...actual,
|
||
Providers: actual.Providers,
|
||
RetentionMode: actual.RetentionMode ?? { ALL: 'all', TEMPORARY: 'temporary' },
|
||
documentParserMimeTypes: actual.documentParserMimeTypes ?? [
|
||
/^application\/pdf$/,
|
||
/^application\/vnd\.openxmlformats-officedocument\./,
|
||
/^application\/vnd\.ms-excel$/,
|
||
/^application\/vnd\.oasis\.opendocument\./,
|
||
/^application\/(?:x-)?msexcel$/,
|
||
],
|
||
mergeFileConfig: jest.fn(),
|
||
};
|
||
});
|
||
|
||
jest.mock('@librechat/api', () => {
|
||
const actualDataProvider = jest.requireActual('librechat-data-provider');
|
||
const RetentionMode = actualDataProvider.RetentionMode ?? { ALL: 'all', TEMPORARY: 'temporary' };
|
||
const getRetentionExpiry = jest.fn(() => ({}));
|
||
const UPLOAD_EXTRACTED_TEXT_PLANS = {
|
||
configuredOCR: 'configured_ocr',
|
||
configuredRAG: 'configured_rag',
|
||
documentParser: 'document_parser',
|
||
};
|
||
const hasActiveFileFieldPolicy = jest.fn((filters, candidates) => {
|
||
const pii = filters?.files?.pii;
|
||
if (pii == null) {
|
||
return false;
|
||
}
|
||
const fieldSelected = candidates.some(
|
||
(field) => pii.fields == null || pii.fields.includes(field),
|
||
);
|
||
const hasPatterns =
|
||
pii.starterPatterns == null ||
|
||
pii.starterPatterns.length > 0 ||
|
||
(pii.customPatterns?.length ?? 0) > 0;
|
||
const failClosed =
|
||
pii.uninspectable === 'block' &&
|
||
candidates.some(
|
||
(field) =>
|
||
['content', 'extracted_text', 'transcript'].includes(field) &&
|
||
(pii.fields == null || pii.fields.includes(field)),
|
||
);
|
||
return (hasPatterns && fieldSelected) || failClosed;
|
||
});
|
||
const hasActiveFilePolicy = jest.fn((filters) =>
|
||
hasActiveFileFieldPolicy(filters, ['name', 'content', 'extracted_text', 'transcript']),
|
||
);
|
||
const getSafeErrorMetadata = jest.fn((error) => ({
|
||
type: error instanceof Error ? 'Error' : 'UnknownError',
|
||
...(Number.isInteger(error?.response?.status) && { status: error.response.status }),
|
||
}));
|
||
const getFileExtractionLogDetails = jest.fn(({ filters, filename, fileId, error }) => {
|
||
const contentProtected = hasActiveFilePolicy(filters);
|
||
return {
|
||
contentProtected,
|
||
fileLabel: contentProtected ? `file_id=${fileId}` : `"${filename}"`,
|
||
errorMetadata: contentProtected ? getSafeErrorMetadata(error) : error,
|
||
};
|
||
});
|
||
const getUploadExtractedTextPlan = jest.fn(
|
||
({ mimeType, fileConfig, ocrConfigured, ragConfigured }) => {
|
||
const checkType = fileConfig.checkType;
|
||
if (ocrConfigured && checkType(mimeType, fileConfig.ocr?.supportedMimeTypes ?? [])) {
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.configuredOCR;
|
||
}
|
||
const isDocumentParserEligible = actualDataProvider.documentParserMimeTypes.some((pattern) =>
|
||
pattern.test(mimeType),
|
||
);
|
||
if (!isDocumentParserEligible) {
|
||
return null;
|
||
}
|
||
if (
|
||
ragConfigured &&
|
||
!actualDataProvider.isPermissiveMimeConfig(fileConfig.text?.supportedMimeTypes) &&
|
||
checkType(mimeType, fileConfig.text?.supportedMimeTypes ?? [])
|
||
) {
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.configuredRAG;
|
||
}
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.documentParser;
|
||
},
|
||
);
|
||
return {
|
||
sanitizeFilename: jest.fn((n) => n),
|
||
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
|
||
processAudioFile: jest.fn(),
|
||
extractInspectableFileText: jest.fn(async ({ extract }) => extract()),
|
||
assertExtractedTextInspectable: jest.fn(),
|
||
getUploadExtractedTextPlan,
|
||
UPLOAD_EXTRACTED_TEXT_PLANS,
|
||
getFileExtractionLogDetails,
|
||
getSafeErrorMetadata,
|
||
hasActiveFilePolicy,
|
||
inspectContent: jest.fn(),
|
||
extractFileContent: jest.fn((input) => [input]),
|
||
hasActiveFileFieldPolicy,
|
||
contentFilterBlockResponse: jest.fn((finding) => ({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: finding.source,
|
||
field: finding.field,
|
||
})),
|
||
sendUploadSuccess: jest.fn((res, sseStream, message, result) => {
|
||
if (sseStream) {
|
||
sseStream.sendData({ message, ...result });
|
||
return;
|
||
}
|
||
res.status(200).json({ message, ...result });
|
||
}),
|
||
getStorageMetadata: jest.fn(() => ({})),
|
||
getRetentionExpiry,
|
||
getAgentFileRetentionExpiry: jest.fn(({ req, messageAttachment, toolResource }) => {
|
||
const interfaceConfig = req?.config?.interfaceConfig;
|
||
if (
|
||
!messageAttachment &&
|
||
!!toolResource &&
|
||
(interfaceConfig?.retentionMode !== RetentionMode.ALL ||
|
||
interfaceConfig?.retainAgentFiles === true)
|
||
) {
|
||
return {};
|
||
}
|
||
return getRetentionExpiry(req);
|
||
}),
|
||
sweepExpiredFiles: jest.fn().mockResolvedValue({ scanned: 0, deleted: 0, failed: 0 }),
|
||
startExpiredFileSweep: jest.fn().mockReturnValue('sweep-interval'),
|
||
};
|
||
});
|
||
|
||
jest.mock('~/server/services/Files/images', () => ({
|
||
convertImage: jest.fn(),
|
||
resizeAndConvert: jest.fn(),
|
||
resizeImageBuffer: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/controllers/assistants/v2', () => ({
|
||
addResourceFileId: jest.fn(),
|
||
deleteResourceFileId: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/controllers/assistants/helpers', () => ({
|
||
getOpenAIClient: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Tools/credentials', () => ({
|
||
loadAuthValues: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/models', () => ({
|
||
createFile: jest.fn().mockResolvedValue({ file_id: 'created-file-id' }),
|
||
updateFileUsage: jest.fn(),
|
||
deleteFiles: jest.fn(),
|
||
findFileById: jest.fn(),
|
||
getConvo: jest.fn(),
|
||
getExpiredFiles: jest.fn(),
|
||
incrementFileDeletionAttempts: jest.fn(),
|
||
deferExpiredFile: jest.fn(),
|
||
addAgentResourceFile: jest.fn().mockResolvedValue({}),
|
||
removeAgentResourceFiles: jest.fn(),
|
||
removeAgentResourceFilesFromAllAgents: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/utils/getFileStrategy', () => ({
|
||
getFileStrategy: jest.fn().mockReturnValue('local'),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Config', () => ({
|
||
checkCapability: jest.fn().mockResolvedValue(true),
|
||
}));
|
||
|
||
jest.mock('~/server/utils/queue', () => ({
|
||
LB_QueueAsyncCall: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Files/strategies', () => ({
|
||
getStrategyFunctions: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('./VectorDB/crud', () => ({
|
||
uploadVectors: jest.fn().mockResolvedValue({
|
||
bytes: 42,
|
||
filename: 'upload.bin',
|
||
filepath: 'vectordb',
|
||
embedded: true,
|
||
}),
|
||
deleteVectors: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/utils', () => ({
|
||
determineFileType: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Files/Audio/STTService', () => ({
|
||
STTService: { getInstance: jest.fn() },
|
||
}));
|
||
|
||
const {
|
||
getRetentionExpiry,
|
||
getAgentFileRetentionExpiry,
|
||
sweepExpiredFiles: sweepExpiredFilesWithDeps,
|
||
startExpiredFileSweep: startExpiredFileSweepWithDeps,
|
||
} = require('@librechat/api');
|
||
const {
|
||
EToolResources,
|
||
FileSources,
|
||
FileContext,
|
||
RetentionMode,
|
||
AgentCapabilities,
|
||
} = require('librechat-data-provider');
|
||
const { mergeFileConfig } = require('librechat-data-provider');
|
||
const { checkCapability } = require('~/server/services/Config');
|
||
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
|
||
const { uploadVectors } = require('./VectorDB/crud');
|
||
const { logger } = require('@librechat/data-schemas');
|
||
const db = require('~/models');
|
||
const {
|
||
processAgentFileUpload,
|
||
processDeleteRequest,
|
||
processFileURL,
|
||
sweepExpiredFiles,
|
||
startExpiredFileSweep,
|
||
} = require('./process');
|
||
const {
|
||
inspectContent,
|
||
extractFileContent,
|
||
processAudioFile,
|
||
extractInspectableFileText,
|
||
assertExtractedTextInspectable,
|
||
contentFilterBlockResponse,
|
||
} = require('@librechat/api');
|
||
|
||
const PDF_MIME = 'application/pdf';
|
||
const DOCX_MIME = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
|
||
const XLSX_MIME = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
|
||
const XLS_MIME = 'application/vnd.ms-excel';
|
||
const ODS_MIME = 'application/vnd.oasis.opendocument.spreadsheet';
|
||
const ODT_MIME = 'application/vnd.oasis.opendocument.text';
|
||
const ODP_MIME = 'application/vnd.oasis.opendocument.presentation';
|
||
const ODG_MIME = 'application/vnd.oasis.opendocument.graphics';
|
||
|
||
const makeUninspectableExtractedTextError = () =>
|
||
Object.assign(new Error('Submitted file content could not be inspected.'), {
|
||
code: 'content_filter_uninspectable',
|
||
statusCode: 400,
|
||
body: {
|
||
error: 'content_filter_uninspectable',
|
||
message: 'Submitted file content could not be inspected before processing.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
},
|
||
});
|
||
|
||
const makeReq = ({
|
||
mimetype = PDF_MIME,
|
||
ocrConfig = null,
|
||
interfaceConfig,
|
||
filters,
|
||
body,
|
||
} = {}) => ({
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
file: {
|
||
path: '/tmp/upload.bin',
|
||
originalname: 'upload.bin',
|
||
filename: 'upload-uuid.bin',
|
||
mimetype,
|
||
},
|
||
body: { model: 'gpt-4o', ...body },
|
||
config: {
|
||
fileConfig: {},
|
||
fileStrategy: 'local',
|
||
ocr: ocrConfig,
|
||
...(filters ? { filters } : {}),
|
||
...(interfaceConfig ? { interfaceConfig } : {}),
|
||
},
|
||
});
|
||
|
||
const makeMetadata = () => ({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
file_id: 'file-uuid-123',
|
||
});
|
||
|
||
const mockRes = {
|
||
status: jest.fn().mockReturnThis(),
|
||
json: jest.fn().mockReturnValue({}),
|
||
};
|
||
|
||
const makeFileConfig = ({
|
||
ocrSupportedMimeTypes = [],
|
||
sttSupportedMimeTypes = [],
|
||
textSupportedMimeTypes = [],
|
||
} = {}) => ({
|
||
checkType: (mime, types) =>
|
||
(types ?? []).some((t) => (typeof t === 'string' ? t === mime : t.test(mime))),
|
||
ocr: { supportedMimeTypes: ocrSupportedMimeTypes },
|
||
stt: { supportedMimeTypes: sttSupportedMimeTypes },
|
||
text: { supportedMimeTypes: textSupportedMimeTypes },
|
||
});
|
||
|
||
const setupStoredFileUpload = (result = {}) => {
|
||
const handleFileUpload = jest.fn().mockResolvedValue({
|
||
bytes: 42,
|
||
filename: 'upload.bin',
|
||
filepath: '/uploads/upload.bin',
|
||
...result,
|
||
});
|
||
getStrategyFunctions.mockReturnValue({ handleFileUpload });
|
||
return handleFileUpload;
|
||
};
|
||
|
||
describe('processAgentFileUpload', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
mockRes.status.mockReturnThis();
|
||
mockRes.json.mockReturnValue({});
|
||
checkCapability.mockResolvedValue(true);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest
|
||
.fn()
|
||
.mockResolvedValue({ text: 'extracted text', bytes: 42, filepath: 'doc://result' }),
|
||
});
|
||
mergeFileConfig.mockReturnValue(makeFileConfig());
|
||
inspectContent.mockReturnValue(null);
|
||
});
|
||
|
||
describe('content filtering for extracted context', () => {
|
||
const filters = { files: { pii: {} } };
|
||
const extractedTextFinding = {
|
||
label: 'protected value',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
};
|
||
|
||
it('blocks extracted document text before agent-resource and file persistence', async () => {
|
||
inspectContent.mockReturnValueOnce(extractedTextFinding);
|
||
const req = makeReq({ filters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(extractFileContent).toHaveBeenCalledWith({ extractedText: 'extracted text' });
|
||
expect(inspectContent).toHaveBeenCalledWith([{ extractedText: 'extracted text' }], {
|
||
filters,
|
||
});
|
||
expect(contentFilterBlockResponse).toHaveBeenCalledWith(extractedTextFinding);
|
||
expect(mockRes.status).toHaveBeenCalledWith(400);
|
||
expect(mockRes.json).toHaveBeenCalledWith({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
});
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('emits a metadata-safe SSE error when extracted text is blocked after streaming starts', async () => {
|
||
inspectContent.mockReturnValueOnce(extractedTextFinding);
|
||
const req = makeReq({ filters });
|
||
const sseStream = { sendError: jest.fn() };
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: makeMetadata(),
|
||
sseStream,
|
||
});
|
||
|
||
expect(sseStream.sendError).toHaveBeenCalledWith({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
code: 400,
|
||
temp_file_id: null,
|
||
tool_resource: EToolResources.context,
|
||
display_to_user: true,
|
||
});
|
||
expect(mockRes.status).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('classifies STT output as a transcript before persistence', async () => {
|
||
const transcriptFinding = {
|
||
label: 'protected value',
|
||
source: 'file',
|
||
field: 'transcript',
|
||
};
|
||
inspectContent.mockReturnValueOnce(transcriptFinding);
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const sttService = {};
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce(sttService);
|
||
processAudioFile.mockResolvedValueOnce({ text: 'submitted transcript', bytes: 20 });
|
||
const req = makeReq({ mimetype: 'audio/webm', filters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(processAudioFile).toHaveBeenCalledWith({
|
||
req,
|
||
file: req.file,
|
||
sttService,
|
||
});
|
||
expect(extractFileContent).toHaveBeenCalledWith({ transcript: 'submitted transcript' });
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('persists the original audio MIME type as transcript provenance', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const sttService = {};
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce(sttService);
|
||
processAudioFile.mockResolvedValueOnce({ text: 'submitted transcript', bytes: 20 });
|
||
const req = makeReq({ mimetype: 'audio/webm' });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
type: 'audio/webm',
|
||
source: FileSources.text,
|
||
text: 'submitted transcript',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('does not persist context audio when STT cannot produce a strict-policy transcript', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce({});
|
||
processAudioFile.mockRejectedValueOnce(new Error('transcription unavailable'));
|
||
const req = makeReq({
|
||
mimetype: 'audio/webm',
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['transcript'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('transcription unavailable');
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('preserves the default-off path without inspecting extracted text', async () => {
|
||
const req = makeReq();
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(inspectContent).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledTimes(1);
|
||
expect(db.createFile).toHaveBeenCalledTimes(1);
|
||
});
|
||
|
||
it('preserves extracted text when the selected file policy has no active patterns', async () => {
|
||
const inactiveFilters = {
|
||
files: {
|
||
pii: { fields: ['extracted_text'], starterPatterns: [], customPatterns: [] },
|
||
},
|
||
};
|
||
const req = makeReq({ filters: inactiveFilters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(inspectContent).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledTimes(1);
|
||
expect(db.createFile).toHaveBeenCalledTimes(1);
|
||
});
|
||
});
|
||
|
||
describe('OCR strategy selection', () => {
|
||
test.each([
|
||
['PDF', PDF_MIME],
|
||
['DOCX', DOCX_MIME],
|
||
['XLSX', XLSX_MIME],
|
||
['XLS', XLS_MIME],
|
||
['ODS', ODS_MIME],
|
||
['Excel variant (msexcel)', 'application/msexcel'],
|
||
['Excel variant (x-msexcel)', 'application/x-msexcel'],
|
||
])('uses document_parser automatically for %s when no OCR is configured', async (_, mime) => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig());
|
||
const req = makeReq({ mimetype: mime, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('does not check OCR capability when using automatic document_parser fallback', async () => {
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('uses the configured OCR strategy when OCR is set up for the file type', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('uses document_parser as default when OCR is configured but no strategy is specified', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { supportedMimeTypes: [PDF_MIME] },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('throws when configured OCR capability is not enabled for the agent', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
checkCapability.mockResolvedValue(false);
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('OCR capability is not enabled for Agents');
|
||
});
|
||
|
||
test('uses document_parser (no capability check) when OCR capability returns false but no OCR config', async () => {
|
||
checkCapability.mockResolvedValue(false);
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('uses document_parser when OCR is configured but the file type is not in OCR supported types', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('does not invoke any OCR strategy for unsupported MIME types without OCR config', async () => {
|
||
const req = makeReq({ mimetype: 'text/plain', ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('File type text/plain is not supported for text parsing.');
|
||
|
||
expect(getStrategyFunctions).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test.each([
|
||
['ODT', ODT_MIME],
|
||
['ODP', ODP_MIME],
|
||
['ODG', ODG_MIME],
|
||
])('routes %s through configured OCR when OCR supports the type', async (_, mime) => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [mime] }));
|
||
const req = makeReq({
|
||
mimetype: mime,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('throws instead of falling back to parseText when document_parser fails for a document MIME type', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('No text found in document')),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
const { parseText } = require('@librechat/api');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow(/image-based and requires an OCR service/);
|
||
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('fails closed without persistence when strict extracted text cannot be produced', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('PRIVATE parser failure')),
|
||
});
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
await extract();
|
||
throw makeUninspectableExtractedTextError();
|
||
});
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
req.file.originalname = 'PRIVATE-report.pdf';
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: {
|
||
error: 'content_filter_uninspectable',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
},
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
expect(logger.error).toHaveBeenCalledWith(expect.stringContaining('file_id=file-uuid-123'), {
|
||
type: 'Error',
|
||
});
|
||
expect(JSON.stringify(logger.error.mock.calls)).not.toContain('PRIVATE');
|
||
});
|
||
|
||
test('fails closed without persistence when strict extracted text is blank', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest
|
||
.fn()
|
||
.mockResolvedValue({ text: ' ', bytes: 3, filepath: 'doc://empty' }),
|
||
});
|
||
assertExtractedTextInspectable.mockImplementationOnce(() => {
|
||
throw makeUninspectableExtractedTextError();
|
||
});
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
ocrConfig: null,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('fails closed when generic configured text extraction throws', async () => {
|
||
const customMime = 'application/x-private-document';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: [customMime] }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('PRIVATE native extraction failure'));
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
try {
|
||
await extract();
|
||
} catch {
|
||
throw makeUninspectableExtractedTextError();
|
||
}
|
||
});
|
||
const req = makeReq({
|
||
mimetype: customMime,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('falls back to document_parser when configured OCR fails for a document MIME type', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const failingUpload = jest.fn().mockRejectedValue(new Error('OCR API returned 500'));
|
||
const fallbackUpload = jest
|
||
.fn()
|
||
.mockResolvedValue({ text: 'parsed text', bytes: 11, filepath: 'doc://result' });
|
||
getStrategyFunctions
|
||
.mockReturnValueOnce({ handleFileUpload: failingUpload })
|
||
.mockReturnValueOnce({ handleFileUpload: fallbackUpload });
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('throws when both configured OCR and document_parser fallback fail', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('failure')),
|
||
});
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
const { parseText } = require('@librechat/api');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow(/image-based and requires an OCR service/);
|
||
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
});
|
||
|
||
describe('configured text (RAG) routing for document MIME types', () => {
|
||
const DOCX_TEXT_REGEX = [
|
||
/^application\/vnd\.openxmlformats-officedocument\.wordprocessingml\.document$/,
|
||
];
|
||
let originalRagUrl;
|
||
|
||
beforeEach(() => {
|
||
originalRagUrl = process.env.RAG_API_URL;
|
||
});
|
||
|
||
afterEach(() => {
|
||
if (originalRagUrl === undefined) {
|
||
delete process.env.RAG_API_URL;
|
||
} else {
|
||
process.env.RAG_API_URL = originalRagUrl;
|
||
}
|
||
});
|
||
|
||
test('routes a document type to RAG /text (no native fallback) when admin narrows text config and RAG is set', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockResolvedValueOnce({ text: 'rag extracted', bytes: 13 });
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(parseText).toHaveBeenCalledWith(
|
||
expect.objectContaining({ allowNativeFallback: false }),
|
||
);
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('keeps the built-in document parser when text config is the permissive default', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(
|
||
makeFileConfig({ textSupportedMimeTypes: [/^[\w.-]+\/[\w.-]+$/] }),
|
||
);
|
||
const { parseText } = require('@librechat/api');
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('keeps the built-in document parser when RAG_API_URL is not configured', async () => {
|
||
delete process.env.RAG_API_URL;
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('falls back to the built-in document parser (not native text) when RAG is unavailable', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('native fallback is disabled'));
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
|
||
expect(parseText).toHaveBeenCalledWith(
|
||
expect.objectContaining({ allowNativeFallback: false }),
|
||
);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('fails closed when RAG and its document-parser fallback cannot extract text', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('PRIVATE RAG failure'));
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('PRIVATE parser failure')),
|
||
});
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
await extract();
|
||
throw makeUninspectableExtractedTextError();
|
||
});
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('surfaces a persistence failure without retrying via the document parser', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockResolvedValueOnce({ text: 'rag extracted', bytes: 13 });
|
||
// RAG extraction succeeds, but persisting the result fails.
|
||
db.createFile.mockRejectedValueOnce(new Error('DB down'));
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('DB down');
|
||
|
||
// The persistence failure must not trigger a second extraction via the built-in parser.
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
});
|
||
|
||
describe('text size guard', () => {
|
||
test('throws before writing to MongoDB when extracted text exceeds 15MB', async () => {
|
||
const oversizedText = 'x'.repeat(15 * 1024 * 1024 + 1);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockResolvedValue({
|
||
text: oversizedText,
|
||
bytes: Buffer.byteLength(oversizedText, 'utf8'),
|
||
filepath: 'doc://result',
|
||
}),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
const { createFile } = require('~/models');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow(/exceeds the 15MB storage limit/);
|
||
|
||
expect(createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('succeeds when extracted text is within the 15MB limit', async () => {
|
||
const okText = 'x'.repeat(1024);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockResolvedValue({
|
||
text: okText,
|
||
bytes: Buffer.byteLength(okText, 'utf8'),
|
||
filepath: 'doc://result',
|
||
}),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
});
|
||
});
|
||
|
||
describe('retention for agent resource uploads', () => {
|
||
test('skips retention metadata for persistent agent context files outside all-data retention when retainAgentFiles is disabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.TEMPORARY, retainAgentFiles: false },
|
||
body: { conversationId: 'temporary-convo', isTemporary: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getAgentFileRetentionExpiry).toHaveBeenCalledWith(
|
||
{
|
||
req,
|
||
messageAttachment: false,
|
||
toolResource: EToolResources.context,
|
||
},
|
||
expect.any(Object),
|
||
);
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('skips retention metadata for persistent agent context files outside all-data retention when retainAgentFiles is enabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.TEMPORARY, retainAgentFiles: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies all-data retention metadata to persistent agent context files when retainAgentFiles is disabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL, retainAgentFiles: false },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('skips all-data retention metadata for persistent agent context files when retainAgentFiles is enabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL, retainAgentFiles: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getAgentFileRetentionExpiry).toHaveBeenCalledWith(
|
||
{
|
||
req,
|
||
messageAttachment: false,
|
||
toolResource: EToolResources.context,
|
||
},
|
||
expect.any(Object),
|
||
);
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies retention metadata to context files uploaded as message attachments', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), message_file: true },
|
||
});
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.message_attachment,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('skips retention metadata for persistent agent file-search files outside all-data retention', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
setupStoredFileUpload();
|
||
const req = makeReq({ mimetype: 'text/plain', ocrConfig: null });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), tool_resource: EToolResources.file_search },
|
||
});
|
||
|
||
expect(uploadVectors).toHaveBeenCalled();
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.file_search,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies all-data retention metadata to persistent agent file-search files', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
setupStoredFileUpload();
|
||
const req = makeReq({
|
||
mimetype: 'text/plain',
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL },
|
||
});
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), tool_resource: EToolResources.file_search },
|
||
});
|
||
|
||
expect(uploadVectors).toHaveBeenCalled();
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.file_search,
|
||
}),
|
||
);
|
||
});
|
||
});
|
||
|
||
/* Phase C / option α regression: the upload must persist its sandbox
|
||
* pointer under `metadata.codeEnvRef` (the post-cutover schema). The
|
||
* legacy `metadata.fileIdentifier` key is silently stripped by mongoose
|
||
* strict mode and downstream readers (`primeFiles`, `getCodeFilesByIds`,
|
||
* `categorizeFileForToolResources`, controller filtering) only check
|
||
* `codeEnvRef`. Storing under the legacy key would orphan the file —
|
||
* priming would skip it on subsequent code-execution turns and the
|
||
* sandbox copy would never re-mount. */
|
||
describe('execute_code uploads persist codeEnvRef metadata', () => {
|
||
const fs = require('fs');
|
||
const { Readable } = require('stream');
|
||
let createReadStreamSpy;
|
||
|
||
beforeEach(() => {
|
||
/* `processAgentFileUpload` opens the multer-staged temp file via
|
||
* `fs.createReadStream`. The test fixture path doesn't exist, so
|
||
* stub it to a tiny in-memory stream. */
|
||
createReadStreamSpy = jest
|
||
.spyOn(fs, 'createReadStream')
|
||
.mockImplementation(() => Readable.from(Buffer.from('')));
|
||
});
|
||
|
||
afterEach(() => {
|
||
createReadStreamSpy.mockRestore();
|
||
});
|
||
|
||
const setupCodeEnvUpload = (uploaded) => {
|
||
/* `processAgentFileUpload` calls `getStrategyFunctions` twice:
|
||
* once with `execute_code` for the codeapi upload, then again with
|
||
* the on-disk strategy (`local`) for the standard storage step that
|
||
* runs in the same flow. Both must return a working
|
||
* `handleFileUpload`. */
|
||
const codeEnvUpload = jest.fn().mockResolvedValue(uploaded);
|
||
const localUpload = jest.fn().mockResolvedValue({
|
||
bytes: 0,
|
||
filename: 'upload.bin',
|
||
filepath: '/uploads/upload.bin',
|
||
});
|
||
getStrategyFunctions.mockImplementation((src) =>
|
||
src === FileSources.execute_code
|
||
? { handleFileUpload: codeEnvUpload }
|
||
: { handleFileUpload: localUpload, saveBuffer: jest.fn() },
|
||
);
|
||
return codeEnvUpload;
|
||
};
|
||
|
||
it('persists kind:user codeEnvRef for chat attachments (messageAttachment=true)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-1', file_id: 'fid-1' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
message_file: true,
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'user',
|
||
id: 'user-123',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'user',
|
||
id: 'user-123',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('persists kind:agent codeEnvRef for agent setup files (messageAttachment=false)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-2', file_id: 'fid-2' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-2',
|
||
file_id: 'fid-2',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-2',
|
||
file_id: 'fid-2',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('skips retention metadata for persistent agent execute_code files outside all-data retention', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-4', file_id: 'fid-4' });
|
||
const req = makeReq();
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
}),
|
||
);
|
||
});
|
||
|
||
it('applies all-data retention metadata to persistent agent execute_code files', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-5', file_id: 'fid-5' });
|
||
const req = makeReq({ interfaceConfig: { retentionMode: RetentionMode.ALL } });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-5',
|
||
file_id: 'fid-5',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-5',
|
||
file_id: 'fid-5',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
}),
|
||
);
|
||
});
|
||
|
||
it('does not persist legacy fileIdentifier key (mongoose strict drops it)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-3', file_id: 'fid-3' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
message_file: true,
|
||
},
|
||
});
|
||
|
||
const persisted = db.createFile.mock.calls[0][0];
|
||
expect(persisted.metadata).not.toHaveProperty('fileIdentifier');
|
||
});
|
||
});
|
||
});
|
||
|
||
describe('processFileURL', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('throws and skips DB persistence when saveURL returns null', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue(null);
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await expect(
|
||
processFileURL({
|
||
fileStrategy: FileSources.local,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
).rejects.toThrow('Strategy "local" did not save "image.png"');
|
||
|
||
expect(getFileURL).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('persists tenantId and strategy-returned filepath metadata', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
dimensions: { width: 32, height: 64 },
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
user: 'user-123',
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
filename: 'image.png',
|
||
source: FileSources.cloudfront,
|
||
type: 'image/png',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
width: 32,
|
||
height: 64,
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('applies retention metadata for generated images when retention mode is all', async () => {
|
||
getRetentionExpiry.mockResolvedValueOnce({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
});
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: {},
|
||
config: { interfaceConfig: { retentionMode: 'all', retainAgentFiles: true } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('applies retention metadata for retained non-temporary conversations', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
getRetentionExpiry.mockResolvedValueOnce({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
});
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: { conversationId: 'convo-123' },
|
||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('keeps expired retained conversation files on the parent expiration', async () => {
|
||
const parentExpiredAt = new Date('2020-01-01T00:00:00.000Z');
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt: parentExpiredAt });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: { conversationId: 'convo-123' },
|
||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: parentExpiredAt,
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('falls back to getFileURL with user and tenant context when metadata lacks filepath', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
bytes: 256,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest
|
||
.fn()
|
||
.mockResolvedValue('https://cdn.example.com/t/tenant-a/images/user-123/image.png');
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).toHaveBeenCalledWith({
|
||
userId: 'user-123',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
tenantId: 'tenant-a',
|
||
});
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('preserves the user path segment for local fallback URLs', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
bytes: 256,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn().mockResolvedValue('/images/user-123/image.png');
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.local,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).toHaveBeenCalledWith({
|
||
userId: 'user-123',
|
||
fileName: 'user-123/image.png',
|
||
basePath: 'images',
|
||
tenantId: 'tenant-a',
|
||
});
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
filepath: '/images/user-123/image.png',
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
});
|
||
|
||
describe('processDeleteRequest', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('removes metadata when backing storage is already missing', async () => {
|
||
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
|
||
const deleteFile = jest.fn().mockRejectedValue(missingError);
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
|
||
const result = await processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
});
|
||
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['expired-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('does not treat unrelated not found messages as missing storage', async () => {
|
||
const deleteFile = jest.fn().mockRejectedValue(new Error('Configuration not found'));
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
|
||
const result = await processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
});
|
||
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['expired-file'] });
|
||
});
|
||
|
||
it('throws metadata delete failures after storage deletion succeeds', async () => {
|
||
const deleteFile = jest.fn().mockResolvedValue(undefined);
|
||
const metadataError = new Error('mongo unavailable');
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
db.deleteFiles.mockRejectedValue(metadataError);
|
||
|
||
await expect(
|
||
processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
}),
|
||
).rejects.toThrow('mongo unavailable');
|
||
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||
expect(db.removeAgentResourceFilesFromAllAgents).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('deletes vector storage before removing embedded file metadata', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['embedded-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['embedded-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('keeps embedded file metadata when vector deletion fails', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const vectorDelete = jest.fn().mockRejectedValue(new Error('rag unavailable'));
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['embedded-file'] });
|
||
});
|
||
|
||
it('does not delete vector storage when primary embedded file deletion fails', async () => {
|
||
const primaryDelete = jest.fn().mockRejectedValue(new Error('permission denied'));
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).not.toHaveBeenCalled();
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['embedded-file'] });
|
||
});
|
||
|
||
it('still deletes vector storage when primary embedded file storage is already missing', async () => {
|
||
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
|
||
const primaryDelete = jest.fn().mockRejectedValue(missingError);
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['embedded-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['embedded-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('deletes code environment storage before removing code resource file metadata', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const codeDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.execute_code
|
||
? { deleteFile: codeDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'code-resource-file',
|
||
filepath: '/uploads/code-resource.txt',
|
||
source: FileSources.local,
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
},
|
||
},
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(codeDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['code-resource-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['code-resource-file'], failedFileIds: [] });
|
||
});
|
||
});
|
||
|
||
describe('sweepExpiredFiles', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('delegates expired file sweeping to the shared package with backend dependencies', async () => {
|
||
const options = {
|
||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||
limit: 1,
|
||
};
|
||
sweepExpiredFilesWithDeps.mockResolvedValue({ scanned: 1, deleted: 1, failed: 0 });
|
||
|
||
const result = await sweepExpiredFiles(options);
|
||
|
||
expect(sweepExpiredFilesWithDeps).toHaveBeenCalledWith(
|
||
options,
|
||
expect.objectContaining({
|
||
getExpiredFiles: db.getExpiredFiles,
|
||
processDeleteRequest: expect.any(Function),
|
||
incrementFileDeletionAttempts: db.incrementFileDeletionAttempts,
|
||
deferExpiredFile: db.deferExpiredFile,
|
||
logger: expect.objectContaining({
|
||
error: expect.any(Function),
|
||
info: expect.any(Function),
|
||
warn: expect.any(Function),
|
||
}),
|
||
}),
|
||
);
|
||
expect(result).toEqual({ scanned: 1, deleted: 1, failed: 0 });
|
||
});
|
||
});
|
||
|
||
describe('startExpiredFileSweep', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('delegates background sweep startup to the shared package with system context', () => {
|
||
const options = {
|
||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||
};
|
||
|
||
const interval = startExpiredFileSweep(options);
|
||
|
||
expect(startExpiredFileSweepWithDeps).toHaveBeenCalledWith(
|
||
options,
|
||
expect.objectContaining({
|
||
sweepExpiredFiles: expect.any(Function),
|
||
runAsSystem: expect.any(Function),
|
||
logger: expect.objectContaining({
|
||
error: expect.any(Function),
|
||
info: expect.any(Function),
|
||
warn: expect.any(Function),
|
||
}),
|
||
}),
|
||
);
|
||
expect(interval).toBe('sweep-interval');
|
||
});
|
||
});
|