mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-16 13:00:55 +00:00
* fix: enforce FILE_SEARCH role permission server-side
The FILE_SEARCH role permission is stored, served by GET /api/roles/:name and
settable through the admin API, but nothing ever checks it.
PermissionTypes.FILE_SEARCH occurs zero times under api/server; the three
occurrences in @librechat/api are all in the interface-to-role sync, which
writes the permission rather than checking it.
Measured on v0.8.8-rc1 and confirmed unchanged in rc2: a user whose role has
FILE_SEARCH.USE = false can still upload a document with
tool_resource=file_search and gets 200 with "embedded": true. The same user
calling execute_code correctly gets 403.
Two gates, both mirroring toolAccessPermType in ~/server/controllers/tools.js:
* Upload (api/server/routes/files/files.js): a tool-resource-to-permission map
checked before any branching, so it also covers the assistants path. Returns
403 with the same log line as the RUN_CODE gate. execute_code is included
because it had the same hole: the tool CALL was gated, the upload was not.
* Tool loading (api/app/clients/tools/util/handleTools.js): checked outside the
lazy loader so a denied user never gets the tool equipped, rather than one
that fails when called. Fails closed if the permission check itself throws.
All four required imports were already present in that file, next to an
existing FILE_CITATIONS check.
Both changes live in /api (JavaScript) rather than /packages/api (TypeScript)
as CLAUDE.md prefers for new backend code: each mirrors an existing pattern in
exactly these files, and a second permission gate in another language and
package would be harder to keep in step with the first.
Tests: api/app/clients/tools/util/handleTools.fileSearchPermission.test.js
covers permit, deny, the log line, and the fail-closed path.
* fixup! fix: enforce FILE_SEARCH role permission server-side
* 🔐 fix: Gate `file_search` and `execute_code` on Role Permissions
The `FILE_SEARCH` gate added in the previous commit lives in
`handleTools.loadTools`, which `loadAgentTools` only reaches when
`definitionsOnly` is false. That default is false on the chat path
(`Endpoints/agents/initialize.js`) but true on the responses and
OpenAI-compatible controllers, so on those paths a denied user still got
the tool definition advertised to the model and their files primed by
`primeSearchFiles`, and the call failed later in `loadToolsForExecution` —
the "gets one that fails when called" outcome the gate set out to avoid.
Move the decision to the capability filters that both loaders already run,
mirroring how `canUseMCP` resolves the `MCP_SERVERS` permission ahead of the
synchronous filter. `AgentCapabilities` stays the instance-wide deployment
switch; the role grant is a second condition a tool has to clear. Because
`hasFileSearch`/`hasExecuteCode` derive from the filtered list, priming is
skipped for a denied user too.
`RUN_CODE` had the same hole: `toolAccessPermType` in
`~/server/controllers/tools.js` gates only `POST /tools/:toolId/call`, not
the agent run, so a role without `RUN_CODE.USE` could still execute code
through an agent. Both tools go through the same map.
Checks use `checkAccessWithRequestCache` and run only for tools whose
capability is already enabled, so a run costs at most one role read. A check
that throws denies the tool.
Also formats the test file added in the previous commit, which Prettier
rejects as-is.
* 🔐 fix: Close the Remaining `RUN_CODE` and `FILE_SEARCH` Bypasses
Codex review of the previous commit found four ways past the gates. All four
share a cause: the permission map was copied per call site, so every boundary
that was not copied into stayed open.
Moves the maps and the check into `packages/api/src/tools/rolePermissions.ts`,
per CLAUDE.md ("all new backend code must be TypeScript in `/packages/api`",
"keep `/api` changes to the absolute minimum"), and points the four `/api`
boundaries at it.
- `handleTools.loadTools` now gates every tool in `toolRolePermissions`, not
just `file_search`. It is the shared boundary the Assistants required-action
flow crosses via `processRequiredActions`, which never passes the agent
capability filter — so a legacy Assistant with a raw function named
`execute_code` ran arbitrary code for a role denied `RUN_CODE`.
- The check is now request-cached (`checkAccessWithRequestCache`). The previous
`checkAccess` call omitted `req`, so the agent path resolved the same grant
twice and issued a second serial `getRoleByName` during chat startup, against
the rule in AGENTS.md.
- `codeExecutionEnabled` is gated by the role result at all three loaders, not
only the `execute_code` entry in the filtered list. It drives tool
classification and the programmatic bash tool, so an agent pairing
`execute_code` with an MCP `code_execution` tool still advertised and
instantiated `run_tools_with_bash` for a denied role.
- The upload map covers `EToolResources.code_interpreter`, which the Assistants
builder posts instead of `execute_code`, and native `code_interpreter` /
`file_search` tools are dropped when an assistant is created or updated.
Those run inside the provider and reach neither loader, so configuration time
is the only place they can be gated.
Renames `handleTools.fileSearchPermission.test.js` to
`handleTools.rolePermissions.test.js` — it now covers both tools.
* 🔐 fix: Gate the Image Upload Route and the v1 Assistant Writers
Second Codex round found the same shape again: boundaries the map had not been
wired into. Adds two shared helpers next to the maps in
`packages/api/src/tools/rolePermissions.ts` so a new upload handler or assistant
writer gets the check by calling one function.
- `/files/images` accepts `tool_resource` and routes agent uploads to
`processAgentFileUpload` on its own. The Code Files UI sends images there, so
every image was a way around the upload boundary for a role denied
`RUN_CODE`. Both upload routes now call
`checkToolResourceUploadPermission`.
- That helper passes `req`, so the role read joins the request cache. The
previous `checkAccess` call omitted it, and an allowed Assistants upload goes
on to `processFileUpload` → `addResourceFileId` → `updateAssistant`, whose own
check does pass `req` — two serial lookups per upload, against AGENTS.md.
- `/assistants/v1` is still mounted and its `createAssistant` / `patchAssistant`
write native tools straight to the provider. Both now filter through
`resolveAssistantToolPermissions`, which v2 also uses in place of its local
copy.
Run-time enforcement for native assistant tools is deliberately not here: it
means overriding `body.tools` at `createRun` in both chat controllers, which
changes run semantics, and is better argued on its own.
* 🔐 fix: Gate Legacy `retrieval` and the Code-Environment File Tools
Third Codex round, same shape twice more.
- The v1 assistant builder submits `{ type: 'retrieval' }` where v2 submits
`file_search` (`AssistantPanel.tsx:203`). Only the v2 spelling was mapped, so
`resolveAssistantToolPermissions` treated legacy retrieval as ungated and both
v1 writers preserved it for a role denied `FILE_SEARCH`. Both spellings now
answer to the same grant.
- `codeEnvAvailable` in the agent initializer was capability-only, and
`initializeAgent` rebuilds `bash_tool`, `read_file` and the workspace file
tools from it — after the tool loader has already dropped `execute_code` for a
denied role. The bash gate stops command execution, but those handlers still
read, search, create and edit files in an attached code environment. The flag
now carries the role grant, and short-circuits before the role read when the
deployment has the capability off.
* 🔐 fix: Resolve Tool Grants Once Per Request and Pair Every Capability Gate
Four review rounds found thirteen issues, all one class: a boundary the
permission check had not been copied into. Rounds 2 and 3 each found gaps
created by the previous round's fix, and a manual sweep of tool constructors
and upload routes still missed the `codeEnvAvailable` family entirely — because
the thing worth enumerating is not "paths that reach a tool" but "reads of the
capability".
There are 28 such reads. This replaces per-site permission checks with one
resolution per request, pairs every read that is a gate, and pins the list so a
new one cannot be added silently.
- `resolveToolRoleGrants` resolves `RUN_CODE` and `FILE_SEARCH` together and
memoizes on the request. Both perf findings disappear by construction: gates
share one role read instead of each issuing its own.
- Newly paired gates: `codeEnvAvailable` in the OpenAI-compatible, Responses and
memory-agent initializers (`initializeAgent` rebuilds `bash_tool`, `read_file`
and the workspace file tools from it, and hands their handlers the code
environment); the capability checks in agent upload processing and in
agent-management upload purposes; and the library chat-completion service,
behind an optional `getRoleByName` dependency so embedders are not broken.
- The v1 Knowledge upload posts `assistant_id` with no `tool_resource`, so there
is no resource to authorize. It now reads the assistant's own native tools and
requires their grants, instead of treating a missing resource as unrestricted.
- The startup role read moved off the critical path into the existing
`Promise.all`, and the direct tool-call check passes `req` so the shared
loader gate reuses its result rather than repeating the lookup.
- `toolCapabilityGates.spec.js` pins every file that reads either capability
with a count and a reason. A new read fails the suite until it is classified,
so the next boundary announces itself instead of waiting for a reviewer.
* 🔐 fix: Abort Assistant Writes on Role-Lookup Failure, Authorize Legacy Uploads First
Round 5 found no missed boundary — the inventory held. All three findings are
about how the gates behave.
- Failing closed is right where a denial blocks the operation, and wrong where a
denial instead filters a payload the caller persists. A transient role-store
outage during an assistant create or patch was silently stripping
`code_interpreter` / `file_search` / `retrieval` and saving the result, turning
a momentary failure into permanent config loss. `checkToolRolePermission`
gains `throwOnError`, and `resolveAssistantToolPermissions` uses it: every one
of its callers either persists the filtered list or rejects the request, so a
lookup failure has to propagate.
- The legacy-assistant upload gate ran after `sanitizedUploadFn` had already
pushed the file to the provider, so a denied role left an untracked remote file
and got a 500 for what is a 403. Moved to the route, next to the tool-resource
gate, before any bytes are sent.
- `resendFiles` hydration derived its resource set from unfiltered `agent.tools`,
so a denied role still paid the thread walk and code-file reads for a tool
about to be dropped. The `execute_code` half now keys off
`effectiveCodeEnvAvailable`, which already carries the grant.
The `file_search` half of that last one needs the grant threaded into
`initializeAgent`, which is plumbing in the most delicate file here for a latency
win on an already-correct path — deliberately left for a follow-up.
* 🧪 fix: Declare `resolveToolRoleGrants` in the Upload Spec's Module Stub
`Tests: api (shard 2/3)` failed on `a0b38c3`: `process.spec.js` stubs
`@librechat/api` wholesale, so the `resolveToolRoleGrants` call added to the
upload gate resolved to undefined and threw. Five failures in
`processAgentFileUpload`, plus one cascading `processFileURL` assertion from the
same suite.
The gate was verified through its own spec and the routes above it, but not
through the spec of the file it lives in — `process.spec.js` was never run
locally. Every spec belonging to a file this PR touches now has been, and the
mongodb-backed ones that still cannot run here were checked statically for the
same stub shape: all but `process.integration.spec.js` spread `requireActual`,
and that one never reaches these paths.
* ⚡ fix: Skip the Role Lookup When Code Execution Is Disabled
The browser initializer already guards this; the OpenAI-compatible and Responses
controllers did not, so every request on those endpoints issued a role read whose
result the capability short-circuit then discarded. Both now start the lookup
only when the deployment has `execute_code` enabled, matching the third.
* ⚡ fix: Drop Three Redundant Reads on the Gated Paths
- The memory initializer resolved grants unconditionally, so a deployment
without `execute_code` paid a role read for a flag that could only be false.
Guarded like the three chat initializers.
- Agent-management upload purposes resolved grants before `checkCapability`, so
a purpose the deployment has switched off paid a role read on the way to being
rejected. Capability first, then the grant.
- The legacy-assistant preflight builds an OpenAI client to read the assistant's
tools, and `processFileUpload` then built a second one — re-reading the user's
key expiry and values. The preflight now hands its client on, and processing
builds one only when it wasn't given one.
---------
Co-authored-by: Paul <200737214+SSIG-IT@users.noreply.github.com>
1892 lines
66 KiB
JavaScript
1892 lines
66 KiB
JavaScript
jest.mock('uuid', () => ({ v4: jest.fn(() => 'mock-uuid') }));
|
||
|
||
jest.mock('@librechat/data-schemas', () => ({
|
||
logger: { warn: jest.fn(), debug: jest.fn(), error: jest.fn(), info: jest.fn() },
|
||
runAsSystem: jest.fn((fn) => fn()),
|
||
createTempChatExpirationDate: jest.fn(() => new Date('2030-01-01T00:00:00.000Z')),
|
||
}));
|
||
|
||
jest.mock('@librechat/agents', () => ({
|
||
Providers: {
|
||
XAI: 'xai',
|
||
DEEPSEEK: 'deepseek',
|
||
MOONSHOT: 'moonshot',
|
||
OPENROUTER: 'openrouter',
|
||
VERTEXAI: 'vertexai',
|
||
},
|
||
}));
|
||
|
||
jest.mock('librechat-data-provider', () => {
|
||
const actual = jest.requireActual('librechat-data-provider');
|
||
return {
|
||
...actual,
|
||
Providers: actual.Providers,
|
||
RetentionMode: actual.RetentionMode ?? { ALL: 'all', TEMPORARY: 'temporary' },
|
||
documentParserMimeTypes: actual.documentParserMimeTypes ?? [
|
||
/^application\/pdf$/,
|
||
/^application\/vnd\.openxmlformats-officedocument\./,
|
||
/^application\/vnd\.ms-excel$/,
|
||
/^application\/vnd\.oasis\.opendocument\./,
|
||
/^application\/(?:x-)?msexcel$/,
|
||
],
|
||
mergeFileConfig: jest.fn(),
|
||
};
|
||
});
|
||
|
||
jest.mock('@librechat/api', () => {
|
||
const actualDataProvider = jest.requireActual('librechat-data-provider');
|
||
const RetentionMode = actualDataProvider.RetentionMode ?? { ALL: 'all', TEMPORARY: 'temporary' };
|
||
const getRetentionExpiry = jest.fn(() => ({}));
|
||
const UPLOAD_EXTRACTED_TEXT_PLANS = {
|
||
configuredOCR: 'configured_ocr',
|
||
configuredRAG: 'configured_rag',
|
||
documentParser: 'document_parser',
|
||
};
|
||
const hasActiveFileFieldPolicy = jest.fn((filters, candidates) => {
|
||
const pii = filters?.files?.pii;
|
||
if (pii == null) {
|
||
return false;
|
||
}
|
||
const fieldSelected = candidates.some(
|
||
(field) => pii.fields == null || pii.fields.includes(field),
|
||
);
|
||
const hasPatterns =
|
||
pii.starterPatterns == null ||
|
||
pii.starterPatterns.length > 0 ||
|
||
(pii.customPatterns?.length ?? 0) > 0;
|
||
const failClosed =
|
||
pii.uninspectable === 'block' &&
|
||
candidates.some(
|
||
(field) =>
|
||
['content', 'extracted_text', 'transcript'].includes(field) &&
|
||
(pii.fields == null || pii.fields.includes(field)),
|
||
);
|
||
return (hasPatterns && fieldSelected) || failClosed;
|
||
});
|
||
const hasActiveFilePolicy = jest.fn((filters) =>
|
||
hasActiveFileFieldPolicy(filters, ['name', 'content', 'extracted_text', 'transcript']),
|
||
);
|
||
const getSafeErrorMetadata = jest.fn((error) => ({
|
||
type: error instanceof Error ? 'Error' : 'UnknownError',
|
||
...(Number.isInteger(error?.response?.status) && { status: error.response.status }),
|
||
}));
|
||
const getFileExtractionLogDetails = jest.fn(({ filters, filename, fileId, error }) => {
|
||
const contentProtected = hasActiveFilePolicy(filters);
|
||
return {
|
||
contentProtected,
|
||
fileLabel: contentProtected ? `file_id=${fileId}` : `"${filename}"`,
|
||
errorMetadata: contentProtected ? getSafeErrorMetadata(error) : error,
|
||
};
|
||
});
|
||
const getUploadExtractedTextPlan = jest.fn(
|
||
({ mimeType, fileConfig, ocrConfigured, ragConfigured }) => {
|
||
const checkType = fileConfig.checkType;
|
||
if (ocrConfigured && checkType(mimeType, fileConfig.ocr?.supportedMimeTypes ?? [])) {
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.configuredOCR;
|
||
}
|
||
const isDocumentParserEligible = actualDataProvider.documentParserMimeTypes.some((pattern) =>
|
||
pattern.test(mimeType),
|
||
);
|
||
if (!isDocumentParserEligible) {
|
||
return null;
|
||
}
|
||
if (
|
||
ragConfigured &&
|
||
!actualDataProvider.isPermissiveMimeConfig(fileConfig.text?.supportedMimeTypes) &&
|
||
checkType(mimeType, fileConfig.text?.supportedMimeTypes ?? [])
|
||
) {
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.configuredRAG;
|
||
}
|
||
return UPLOAD_EXTRACTED_TEXT_PLANS.documentParser;
|
||
},
|
||
);
|
||
return {
|
||
sanitizeFilename: jest.fn((n) => n),
|
||
/** Grants both; these specs vary the capability set, not the role. */
|
||
resolveToolRoleGrants: jest.fn(async () => ({ runCode: true, fileSearch: true })),
|
||
parseText: jest.fn().mockResolvedValue({ text: '', bytes: 0 }),
|
||
processAudioFile: jest.fn(),
|
||
extractInspectableFileText: jest.fn(async ({ extract }) => extract()),
|
||
assertExtractedTextInspectable: jest.fn(),
|
||
getUploadExtractedTextPlan,
|
||
UPLOAD_EXTRACTED_TEXT_PLANS,
|
||
getFileExtractionLogDetails,
|
||
getSafeErrorMetadata,
|
||
hasActiveFilePolicy,
|
||
inspectContent: jest.fn(),
|
||
extractFileContent: jest.fn((input) => [input]),
|
||
hasActiveFileFieldPolicy,
|
||
contentFilterBlockResponse: jest.fn((finding) => ({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: finding.source,
|
||
field: finding.field,
|
||
})),
|
||
sendUploadSuccess: jest.fn((res, sseStream, message, result) => {
|
||
if (sseStream) {
|
||
sseStream.sendData({ message, ...result });
|
||
return;
|
||
}
|
||
res.status(200).json({ message, ...result });
|
||
}),
|
||
getStorageMetadata: jest.fn(() => ({})),
|
||
getRetentionExpiry,
|
||
getAgentFileRetentionExpiry: jest.fn(({ req, messageAttachment, toolResource }) => {
|
||
const interfaceConfig = req?.config?.interfaceConfig;
|
||
if (
|
||
!messageAttachment &&
|
||
!!toolResource &&
|
||
(interfaceConfig?.retentionMode !== RetentionMode.ALL ||
|
||
interfaceConfig?.retainAgentFiles === true)
|
||
) {
|
||
return {};
|
||
}
|
||
return getRetentionExpiry(req);
|
||
}),
|
||
sweepExpiredFiles: jest.fn().mockResolvedValue({ scanned: 0, deleted: 0, failed: 0 }),
|
||
startExpiredFileSweep: jest.fn().mockReturnValue('sweep-interval'),
|
||
};
|
||
});
|
||
|
||
jest.mock('~/server/services/Files/images', () => ({
|
||
convertImage: jest.fn(),
|
||
resizeAndConvert: jest.fn(),
|
||
resizeImageBuffer: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/controllers/assistants/v2', () => ({
|
||
addResourceFileId: jest.fn(),
|
||
deleteResourceFileId: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/controllers/assistants/helpers', () => ({
|
||
getOpenAIClient: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Tools/credentials', () => ({
|
||
loadAuthValues: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/models', () => ({
|
||
createFile: jest.fn().mockResolvedValue({ file_id: 'created-file-id' }),
|
||
updateFileUsage: jest.fn(),
|
||
deleteFiles: jest.fn(),
|
||
findFileById: jest.fn(),
|
||
getConvo: jest.fn(),
|
||
getExpiredFiles: jest.fn(),
|
||
incrementFileDeletionAttempts: jest.fn(),
|
||
deferExpiredFile: jest.fn(),
|
||
addAgentResourceFile: jest.fn().mockResolvedValue({}),
|
||
removeAgentResourceFiles: jest.fn(),
|
||
removeAgentResourceFilesFromAllAgents: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/utils/getFileStrategy', () => ({
|
||
getFileStrategy: jest.fn().mockReturnValue('local'),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Config', () => ({
|
||
checkCapability: jest.fn().mockResolvedValue(true),
|
||
}));
|
||
|
||
jest.mock('~/server/utils/queue', () => ({
|
||
LB_QueueAsyncCall: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Files/strategies', () => ({
|
||
getStrategyFunctions: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('./VectorDB/crud', () => ({
|
||
uploadVectors: jest.fn().mockResolvedValue({
|
||
bytes: 42,
|
||
filename: 'upload.bin',
|
||
filepath: 'vectordb',
|
||
embedded: true,
|
||
}),
|
||
deleteVectors: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/utils', () => ({
|
||
determineFileType: jest.fn(),
|
||
}));
|
||
|
||
jest.mock('~/server/services/Files/Audio/STTService', () => ({
|
||
STTService: { getInstance: jest.fn() },
|
||
}));
|
||
|
||
const {
|
||
getRetentionExpiry,
|
||
getAgentFileRetentionExpiry,
|
||
sweepExpiredFiles: sweepExpiredFilesWithDeps,
|
||
startExpiredFileSweep: startExpiredFileSweepWithDeps,
|
||
} = require('@librechat/api');
|
||
const {
|
||
EToolResources,
|
||
FileSources,
|
||
FileContext,
|
||
RetentionMode,
|
||
AgentCapabilities,
|
||
} = require('librechat-data-provider');
|
||
const { mergeFileConfig } = require('librechat-data-provider');
|
||
const { checkCapability } = require('~/server/services/Config');
|
||
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
|
||
const { uploadVectors } = require('./VectorDB/crud');
|
||
const { logger } = require('@librechat/data-schemas');
|
||
const db = require('~/models');
|
||
const {
|
||
processAgentFileUpload,
|
||
processDeleteRequest,
|
||
processFileURL,
|
||
sweepExpiredFiles,
|
||
startExpiredFileSweep,
|
||
} = require('./process');
|
||
const {
|
||
inspectContent,
|
||
extractFileContent,
|
||
processAudioFile,
|
||
extractInspectableFileText,
|
||
assertExtractedTextInspectable,
|
||
contentFilterBlockResponse,
|
||
} = require('@librechat/api');
|
||
|
||
const PDF_MIME = 'application/pdf';
|
||
const DOCX_MIME = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
|
||
const XLSX_MIME = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
|
||
const XLS_MIME = 'application/vnd.ms-excel';
|
||
const ODS_MIME = 'application/vnd.oasis.opendocument.spreadsheet';
|
||
const ODT_MIME = 'application/vnd.oasis.opendocument.text';
|
||
const ODP_MIME = 'application/vnd.oasis.opendocument.presentation';
|
||
const ODG_MIME = 'application/vnd.oasis.opendocument.graphics';
|
||
|
||
const makeUninspectableExtractedTextError = () =>
|
||
Object.assign(new Error('Submitted file content could not be inspected.'), {
|
||
code: 'content_filter_uninspectable',
|
||
statusCode: 400,
|
||
body: {
|
||
error: 'content_filter_uninspectable',
|
||
message: 'Submitted file content could not be inspected before processing.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
},
|
||
});
|
||
|
||
const makeReq = ({
|
||
mimetype = PDF_MIME,
|
||
ocrConfig = null,
|
||
interfaceConfig,
|
||
filters,
|
||
body,
|
||
} = {}) => ({
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
file: {
|
||
path: '/tmp/upload.bin',
|
||
originalname: 'upload.bin',
|
||
filename: 'upload-uuid.bin',
|
||
mimetype,
|
||
},
|
||
body: { model: 'gpt-4o', ...body },
|
||
config: {
|
||
fileConfig: {},
|
||
fileStrategy: 'local',
|
||
ocr: ocrConfig,
|
||
...(filters ? { filters } : {}),
|
||
...(interfaceConfig ? { interfaceConfig } : {}),
|
||
},
|
||
});
|
||
|
||
const makeMetadata = () => ({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
file_id: 'file-uuid-123',
|
||
});
|
||
|
||
const mockRes = {
|
||
status: jest.fn().mockReturnThis(),
|
||
json: jest.fn().mockReturnValue({}),
|
||
};
|
||
|
||
const makeFileConfig = ({
|
||
ocrSupportedMimeTypes = [],
|
||
sttSupportedMimeTypes = [],
|
||
textSupportedMimeTypes = [],
|
||
} = {}) => ({
|
||
checkType: (mime, types) =>
|
||
(types ?? []).some((t) => (typeof t === 'string' ? t === mime : t.test(mime))),
|
||
ocr: { supportedMimeTypes: ocrSupportedMimeTypes },
|
||
stt: { supportedMimeTypes: sttSupportedMimeTypes },
|
||
text: { supportedMimeTypes: textSupportedMimeTypes },
|
||
});
|
||
|
||
const setupStoredFileUpload = (result = {}) => {
|
||
const handleFileUpload = jest.fn().mockResolvedValue({
|
||
bytes: 42,
|
||
filename: 'upload.bin',
|
||
filepath: '/uploads/upload.bin',
|
||
...result,
|
||
});
|
||
getStrategyFunctions.mockReturnValue({ handleFileUpload });
|
||
return handleFileUpload;
|
||
};
|
||
|
||
describe('processAgentFileUpload', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
mockRes.status.mockReturnThis();
|
||
mockRes.json.mockReturnValue({});
|
||
checkCapability.mockResolvedValue(true);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest
|
||
.fn()
|
||
.mockResolvedValue({ text: 'extracted text', bytes: 42, filepath: 'doc://result' }),
|
||
});
|
||
mergeFileConfig.mockReturnValue(makeFileConfig());
|
||
inspectContent.mockReturnValue(null);
|
||
});
|
||
|
||
describe('content filtering for extracted context', () => {
|
||
const filters = { files: { pii: {} } };
|
||
const extractedTextFinding = {
|
||
label: 'protected value',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
};
|
||
|
||
it('blocks extracted document text before agent-resource and file persistence', async () => {
|
||
inspectContent.mockReturnValueOnce(extractedTextFinding);
|
||
const req = makeReq({ filters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(extractFileContent).toHaveBeenCalledWith({ extractedText: 'extracted text' });
|
||
expect(inspectContent).toHaveBeenCalledWith([{ extractedText: 'extracted text' }], {
|
||
filters,
|
||
});
|
||
expect(contentFilterBlockResponse).toHaveBeenCalledWith(extractedTextFinding);
|
||
expect(mockRes.status).toHaveBeenCalledWith(400);
|
||
expect(mockRes.json).toHaveBeenCalledWith({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
});
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('emits a metadata-safe SSE error when extracted text is blocked after streaming starts', async () => {
|
||
inspectContent.mockReturnValueOnce(extractedTextFinding);
|
||
const req = makeReq({ filters });
|
||
const sseStream = { sendError: jest.fn() };
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: makeMetadata(),
|
||
sseStream,
|
||
});
|
||
|
||
expect(sseStream.sendError).toHaveBeenCalledWith({
|
||
error: 'content_filter_block',
|
||
message: 'Submitted content contains a protected value. Remove it and try again.',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
code: 400,
|
||
temp_file_id: null,
|
||
tool_resource: EToolResources.context,
|
||
display_to_user: true,
|
||
});
|
||
expect(mockRes.status).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('classifies STT output as a transcript before persistence', async () => {
|
||
const transcriptFinding = {
|
||
label: 'protected value',
|
||
source: 'file',
|
||
field: 'transcript',
|
||
};
|
||
inspectContent.mockReturnValueOnce(transcriptFinding);
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const sttService = {};
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce(sttService);
|
||
processAudioFile.mockResolvedValueOnce({ text: 'submitted transcript', bytes: 20 });
|
||
const req = makeReq({ mimetype: 'audio/webm', filters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(processAudioFile).toHaveBeenCalledWith({
|
||
req,
|
||
file: req.file,
|
||
sttService,
|
||
});
|
||
expect(extractFileContent).toHaveBeenCalledWith({ transcript: 'submitted transcript' });
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('persists the original audio MIME type as transcript provenance', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const sttService = {};
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce(sttService);
|
||
processAudioFile.mockResolvedValueOnce({ text: 'submitted transcript', bytes: 20 });
|
||
const req = makeReq({ mimetype: 'audio/webm' });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
type: 'audio/webm',
|
||
source: FileSources.text,
|
||
text: 'submitted transcript',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('does not persist context audio when STT cannot produce a strict-policy transcript', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ sttSupportedMimeTypes: ['audio/webm'] }));
|
||
const { STTService } = require('~/server/services/Files/Audio/STTService');
|
||
STTService.getInstance.mockResolvedValueOnce({});
|
||
processAudioFile.mockRejectedValueOnce(new Error('transcription unavailable'));
|
||
const req = makeReq({
|
||
mimetype: 'audio/webm',
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['transcript'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('transcription unavailable');
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('preserves the default-off path without inspecting extracted text', async () => {
|
||
const req = makeReq();
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(inspectContent).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledTimes(1);
|
||
expect(db.createFile).toHaveBeenCalledTimes(1);
|
||
});
|
||
|
||
it('preserves extracted text when the selected file policy has no active patterns', async () => {
|
||
const inactiveFilters = {
|
||
files: {
|
||
pii: { fields: ['extracted_text'], starterPatterns: [], customPatterns: [] },
|
||
},
|
||
};
|
||
const req = makeReq({ filters: inactiveFilters });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(inspectContent).not.toHaveBeenCalled();
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledTimes(1);
|
||
expect(db.createFile).toHaveBeenCalledTimes(1);
|
||
});
|
||
});
|
||
|
||
describe('OCR strategy selection', () => {
|
||
test.each([
|
||
['PDF', PDF_MIME],
|
||
['DOCX', DOCX_MIME],
|
||
['XLSX', XLSX_MIME],
|
||
['XLS', XLS_MIME],
|
||
['ODS', ODS_MIME],
|
||
['Excel variant (msexcel)', 'application/msexcel'],
|
||
['Excel variant (x-msexcel)', 'application/x-msexcel'],
|
||
])('uses document_parser automatically for %s when no OCR is configured', async (_, mime) => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig());
|
||
const req = makeReq({ mimetype: mime, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('does not check OCR capability when using automatic document_parser fallback', async () => {
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('uses the configured OCR strategy when OCR is set up for the file type', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('uses document_parser as default when OCR is configured but no strategy is specified', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { supportedMimeTypes: [PDF_MIME] },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('throws when configured OCR capability is not enabled for the agent', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
checkCapability.mockResolvedValue(false);
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('OCR capability is not enabled for Agents');
|
||
});
|
||
|
||
test('uses document_parser (no capability check) when OCR capability returns false but no OCR config', async () => {
|
||
checkCapability.mockResolvedValue(false);
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('uses document_parser when OCR is configured but the file type is not in OCR supported types', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).not.toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('does not invoke any OCR strategy for unsupported MIME types without OCR config', async () => {
|
||
const req = makeReq({ mimetype: 'text/plain', ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('File type text/plain is not supported for text parsing.');
|
||
|
||
expect(getStrategyFunctions).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test.each([
|
||
['ODT', ODT_MIME],
|
||
['ODP', ODP_MIME],
|
||
['ODG', ODG_MIME],
|
||
])('routes %s through configured OCR when OCR supports the type', async (_, mime) => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [mime] }));
|
||
const req = makeReq({
|
||
mimetype: mime,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(checkCapability).toHaveBeenCalledWith(expect.anything(), AgentCapabilities.ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
});
|
||
|
||
test('throws instead of falling back to parseText when document_parser fails for a document MIME type', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('No text found in document')),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
const { parseText } = require('@librechat/api');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('No text found in document');
|
||
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('fails closed without persistence when strict extracted text cannot be produced', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('PRIVATE parser failure')),
|
||
});
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
try {
|
||
await extract();
|
||
} catch {
|
||
throw makeUninspectableExtractedTextError();
|
||
}
|
||
});
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
req.file.originalname = 'PRIVATE-report.pdf';
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: {
|
||
error: 'content_filter_uninspectable',
|
||
source: 'file',
|
||
field: 'extracted_text',
|
||
},
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
expect(logger.error).toHaveBeenCalledWith(expect.stringContaining('file_id=file-uuid-123'), {
|
||
type: 'Error',
|
||
});
|
||
expect(JSON.stringify(logger.error.mock.calls)).not.toContain('PRIVATE');
|
||
});
|
||
|
||
test('fails closed without persistence when strict extracted text is blank', async () => {
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest
|
||
.fn()
|
||
.mockResolvedValue({ text: ' ', bytes: 3, filepath: 'doc://empty' }),
|
||
});
|
||
assertExtractedTextInspectable.mockImplementationOnce(() => {
|
||
throw makeUninspectableExtractedTextError();
|
||
});
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
ocrConfig: null,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('fails closed when generic configured text extraction throws', async () => {
|
||
const customMime = 'application/x-private-document';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: [customMime] }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('PRIVATE native extraction failure'));
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
try {
|
||
await extract();
|
||
} catch {
|
||
throw makeUninspectableExtractedTextError();
|
||
}
|
||
});
|
||
const req = makeReq({
|
||
mimetype: customMime,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('falls back to document_parser when configured OCR fails for a document MIME type', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
const failingUpload = jest.fn().mockRejectedValue(new Error('OCR API returned 500'));
|
||
const fallbackUpload = jest
|
||
.fn()
|
||
.mockResolvedValue({ text: 'parsed text', bytes: 11, filepath: 'doc://result' });
|
||
getStrategyFunctions
|
||
.mockReturnValueOnce({ handleFileUpload: failingUpload })
|
||
.mockReturnValueOnce({ handleFileUpload: fallbackUpload });
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.mistral_ocr);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('throws when both configured OCR and document_parser fallback fail', async () => {
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ ocrSupportedMimeTypes: [PDF_MIME] }));
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('failure')),
|
||
});
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: { strategy: FileSources.mistral_ocr },
|
||
});
|
||
const { parseText } = require('@librechat/api');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('failure');
|
||
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
});
|
||
|
||
describe('configured text (RAG) routing for document MIME types', () => {
|
||
const DOCX_TEXT_REGEX = [
|
||
/^application\/vnd\.openxmlformats-officedocument\.wordprocessingml\.document$/,
|
||
];
|
||
let originalRagUrl;
|
||
|
||
beforeEach(() => {
|
||
originalRagUrl = process.env.RAG_API_URL;
|
||
});
|
||
|
||
afterEach(() => {
|
||
if (originalRagUrl === undefined) {
|
||
delete process.env.RAG_API_URL;
|
||
} else {
|
||
process.env.RAG_API_URL = originalRagUrl;
|
||
}
|
||
});
|
||
|
||
test('routes a document type to RAG /text (no native fallback) when admin narrows text config and RAG is set', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockResolvedValueOnce({ text: 'rag extracted', bytes: 13 });
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(parseText).toHaveBeenCalledWith(
|
||
expect.objectContaining({ allowNativeFallback: false }),
|
||
);
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('keeps the built-in document parser when text config is the permissive default', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(
|
||
makeFileConfig({ textSupportedMimeTypes: [/^[\w.-]+\/[\w.-]+$/] }),
|
||
);
|
||
const { parseText } = require('@librechat/api');
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('keeps the built-in document parser when RAG_API_URL is not configured', async () => {
|
||
delete process.env.RAG_API_URL;
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
expect(parseText).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('falls back to the built-in document parser (not native text) when RAG is unavailable', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('native fallback is disabled'));
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
|
||
expect(parseText).toHaveBeenCalledWith(
|
||
expect.objectContaining({ allowNativeFallback: false }),
|
||
);
|
||
expect(getStrategyFunctions).toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
|
||
test('fails closed when RAG and its document-parser fallback cannot extract text', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockRejectedValueOnce(new Error('PRIVATE RAG failure'));
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockRejectedValue(new Error('PRIVATE parser failure')),
|
||
});
|
||
extractInspectableFileText.mockImplementationOnce(async ({ extract }) => {
|
||
try {
|
||
await extract();
|
||
} catch {
|
||
throw makeUninspectableExtractedTextError();
|
||
}
|
||
});
|
||
const req = makeReq({
|
||
mimetype: DOCX_MIME,
|
||
filters: {
|
||
files: {
|
||
pii: {
|
||
fields: ['extracted_text'],
|
||
uninspectable: 'block',
|
||
},
|
||
},
|
||
},
|
||
});
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toMatchObject({
|
||
code: 'content_filter_uninspectable',
|
||
body: { field: 'extracted_text' },
|
||
});
|
||
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('surfaces a persistence failure without retrying via the document parser', async () => {
|
||
process.env.RAG_API_URL = 'http://rag-api.test';
|
||
mergeFileConfig.mockReturnValue(makeFileConfig({ textSupportedMimeTypes: DOCX_TEXT_REGEX }));
|
||
const { parseText } = require('@librechat/api');
|
||
parseText.mockResolvedValueOnce({ text: 'rag extracted', bytes: 13 });
|
||
// RAG extraction succeeds, but persisting the result fails.
|
||
db.createFile.mockRejectedValueOnce(new Error('DB down'));
|
||
const req = makeReq({ mimetype: DOCX_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow('DB down');
|
||
|
||
// The persistence failure must not trigger a second extraction via the built-in parser.
|
||
expect(getStrategyFunctions).not.toHaveBeenCalledWith(FileSources.document_parser);
|
||
});
|
||
});
|
||
|
||
describe('text size guard', () => {
|
||
test('throws before writing to MongoDB when extracted text exceeds 15MB', async () => {
|
||
const oversizedText = 'x'.repeat(15 * 1024 * 1024 + 1);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockResolvedValue({
|
||
text: oversizedText,
|
||
bytes: Buffer.byteLength(oversizedText, 'utf8'),
|
||
filepath: 'doc://result',
|
||
}),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
const { createFile } = require('~/models');
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).rejects.toThrow(/exceeds the 15MB storage limit/);
|
||
|
||
expect(createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('succeeds when extracted text is within the 15MB limit', async () => {
|
||
const okText = 'x'.repeat(1024);
|
||
getStrategyFunctions.mockReturnValue({
|
||
handleFileUpload: jest.fn().mockResolvedValue({
|
||
text: okText,
|
||
bytes: Buffer.byteLength(okText, 'utf8'),
|
||
filepath: 'doc://result',
|
||
}),
|
||
});
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await expect(
|
||
processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() }),
|
||
).resolves.not.toThrow();
|
||
});
|
||
});
|
||
|
||
describe('retention for agent resource uploads', () => {
|
||
test('skips retention metadata for persistent agent context files outside all-data retention when retainAgentFiles is disabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.TEMPORARY, retainAgentFiles: false },
|
||
body: { conversationId: 'temporary-convo', isTemporary: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getAgentFileRetentionExpiry).toHaveBeenCalledWith(
|
||
{
|
||
req,
|
||
messageAttachment: false,
|
||
toolResource: EToolResources.context,
|
||
},
|
||
expect.any(Object),
|
||
);
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('skips retention metadata for persistent agent context files outside all-data retention when retainAgentFiles is enabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.TEMPORARY, retainAgentFiles: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies all-data retention metadata to persistent agent context files when retainAgentFiles is disabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL, retainAgentFiles: false },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('skips all-data retention metadata for persistent agent context files when retainAgentFiles is enabled', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
const req = makeReq({
|
||
mimetype: PDF_MIME,
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL, retainAgentFiles: true },
|
||
});
|
||
|
||
await processAgentFileUpload({ req, res: mockRes, metadata: makeMetadata() });
|
||
|
||
expect(getAgentFileRetentionExpiry).toHaveBeenCalledWith(
|
||
{
|
||
req,
|
||
messageAttachment: false,
|
||
toolResource: EToolResources.context,
|
||
},
|
||
expect.any(Object),
|
||
);
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.context,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies retention metadata to context files uploaded as message attachments', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
const req = makeReq({ mimetype: PDF_MIME, ocrConfig: null });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), message_file: true },
|
||
});
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.message_attachment,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
test('skips retention metadata for persistent agent file-search files outside all-data retention', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
setupStoredFileUpload();
|
||
const req = makeReq({ mimetype: 'text/plain', ocrConfig: null });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), tool_resource: EToolResources.file_search },
|
||
});
|
||
|
||
expect(uploadVectors).toHaveBeenCalled();
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.file_search,
|
||
}),
|
||
);
|
||
});
|
||
|
||
test('applies all-data retention metadata to persistent agent file-search files', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
setupStoredFileUpload();
|
||
const req = makeReq({
|
||
mimetype: 'text/plain',
|
||
ocrConfig: null,
|
||
interfaceConfig: { retentionMode: RetentionMode.ALL },
|
||
});
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: { ...makeMetadata(), tool_resource: EToolResources.file_search },
|
||
});
|
||
|
||
expect(uploadVectors).toHaveBeenCalled();
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.file_search,
|
||
}),
|
||
);
|
||
});
|
||
});
|
||
|
||
/* Phase C / option α regression: the upload must persist its sandbox
|
||
* pointer under `metadata.codeEnvRef` (the post-cutover schema). The
|
||
* legacy `metadata.fileIdentifier` key is silently stripped by mongoose
|
||
* strict mode and downstream readers (`primeFiles`, `getCodeFilesByIds`,
|
||
* `categorizeFileForToolResources`, controller filtering) only check
|
||
* `codeEnvRef`. Storing under the legacy key would orphan the file —
|
||
* priming would skip it on subsequent code-execution turns and the
|
||
* sandbox copy would never re-mount. */
|
||
describe('execute_code uploads persist codeEnvRef metadata', () => {
|
||
const fs = require('fs');
|
||
const { Readable } = require('stream');
|
||
let createReadStreamSpy;
|
||
|
||
beforeEach(() => {
|
||
/* `processAgentFileUpload` opens the multer-staged temp file via
|
||
* `fs.createReadStream`. The test fixture path doesn't exist, so
|
||
* stub it to a tiny in-memory stream. */
|
||
createReadStreamSpy = jest
|
||
.spyOn(fs, 'createReadStream')
|
||
.mockImplementation(() => Readable.from(Buffer.from('')));
|
||
});
|
||
|
||
afterEach(() => {
|
||
createReadStreamSpy.mockRestore();
|
||
});
|
||
|
||
const setupCodeEnvUpload = (uploaded) => {
|
||
/* `processAgentFileUpload` calls `getStrategyFunctions` twice:
|
||
* once with `execute_code` for the codeapi upload, then again with
|
||
* the on-disk strategy (`local`) for the standard storage step that
|
||
* runs in the same flow. Both must return a working
|
||
* `handleFileUpload`. */
|
||
const codeEnvUpload = jest.fn().mockResolvedValue(uploaded);
|
||
const localUpload = jest.fn().mockResolvedValue({
|
||
bytes: 0,
|
||
filename: 'upload.bin',
|
||
filepath: '/uploads/upload.bin',
|
||
});
|
||
getStrategyFunctions.mockImplementation((src) =>
|
||
src === FileSources.execute_code
|
||
? { handleFileUpload: codeEnvUpload }
|
||
: { handleFileUpload: localUpload, saveBuffer: jest.fn() },
|
||
);
|
||
return codeEnvUpload;
|
||
};
|
||
|
||
it('persists kind:user codeEnvRef for chat attachments (messageAttachment=true)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-1', file_id: 'fid-1' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
message_file: true,
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'user',
|
||
id: 'user-123',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'user',
|
||
id: 'user-123',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('persists kind:agent codeEnvRef for agent setup files (messageAttachment=false)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-2', file_id: 'fid-2' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-2',
|
||
file_id: 'fid-2',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-2',
|
||
file_id: 'fid-2',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('skips retention metadata for persistent agent execute_code files outside all-data retention', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-4', file_id: 'fid-4' });
|
||
const req = makeReq();
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(getRetentionExpiry).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(expect.not.objectContaining({ expiredAt }), true);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
}),
|
||
);
|
||
});
|
||
|
||
it('applies all-data retention metadata to persistent agent execute_code files', async () => {
|
||
const expiredAt = new Date('2030-01-01T00:00:00.000Z');
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt });
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-5', file_id: 'fid-5' });
|
||
const req = makeReq({ interfaceConfig: { retentionMode: RetentionMode.ALL } });
|
||
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
},
|
||
});
|
||
|
||
expect(getRetentionExpiry).toHaveBeenCalledTimes(1);
|
||
expect(getRetentionExpiry.mock.calls[0][0]).toBe(req);
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt,
|
||
context: FileContext.agents,
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-5',
|
||
file_id: 'fid-5',
|
||
executionProfile: 'default',
|
||
},
|
||
codeEnvRefs: {
|
||
default: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-5',
|
||
file_id: 'fid-5',
|
||
executionProfile: 'default',
|
||
},
|
||
},
|
||
},
|
||
}),
|
||
true,
|
||
);
|
||
expect(db.addAgentResourceFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
}),
|
||
);
|
||
});
|
||
|
||
it('does not persist legacy fileIdentifier key (mongoose strict drops it)', async () => {
|
||
setupCodeEnvUpload({ storage_session_id: 'sess-3', file_id: 'fid-3' });
|
||
const req = makeReq();
|
||
await processAgentFileUpload({
|
||
req,
|
||
res: mockRes,
|
||
metadata: {
|
||
agent_id: 'agent-abc',
|
||
tool_resource: EToolResources.execute_code,
|
||
file_id: 'file-uuid',
|
||
message_file: true,
|
||
},
|
||
});
|
||
|
||
const persisted = db.createFile.mock.calls[0][0];
|
||
expect(persisted.metadata).not.toHaveProperty('fileIdentifier');
|
||
});
|
||
});
|
||
});
|
||
|
||
describe('processFileURL', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('throws and skips DB persistence when saveURL returns null', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue(null);
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await expect(
|
||
processFileURL({
|
||
fileStrategy: FileSources.local,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
).rejects.toThrow('Strategy "local" did not save "image.png"');
|
||
|
||
expect(getFileURL).not.toHaveBeenCalled();
|
||
expect(db.createFile).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('persists tenantId and strategy-returned filepath metadata', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
dimensions: { width: 32, height: 64 },
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).not.toHaveBeenCalled();
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
user: 'user-123',
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
filename: 'image.png',
|
||
source: FileSources.cloudfront,
|
||
type: 'image/png',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
width: 32,
|
||
height: 64,
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('applies retention metadata for generated images when retention mode is all', async () => {
|
||
getRetentionExpiry.mockResolvedValueOnce({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
});
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: {},
|
||
config: { interfaceConfig: { retentionMode: 'all', retainAgentFiles: true } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('applies retention metadata for retained non-temporary conversations', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
getRetentionExpiry.mockResolvedValueOnce({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
});
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: { conversationId: 'convo-123' },
|
||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: new Date('2030-01-01T00:00:00.000Z'),
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('keeps expired retained conversation files on the parent expiration', async () => {
|
||
const parentExpiredAt = new Date('2020-01-01T00:00:00.000Z');
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
bytes: 512,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn();
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
getRetentionExpiry.mockResolvedValueOnce({ expiredAt: parentExpiredAt });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
req: {
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
body: { conversationId: 'convo-123' },
|
||
config: { interfaceConfig: { retentionMode: RetentionMode.TEMPORARY } },
|
||
},
|
||
});
|
||
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
expiredAt: parentExpiredAt,
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('falls back to getFileURL with user and tenant context when metadata lacks filepath', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
bytes: 256,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest
|
||
.fn()
|
||
.mockResolvedValue('https://cdn.example.com/t/tenant-a/images/user-123/image.png');
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.cloudfront,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).toHaveBeenCalledWith({
|
||
userId: 'user-123',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
tenantId: 'tenant-a',
|
||
});
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
filepath: 'https://cdn.example.com/t/tenant-a/images/user-123/image.png',
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
|
||
it('preserves the user path segment for local fallback URLs', async () => {
|
||
const saveURL = jest.fn().mockResolvedValue({
|
||
bytes: 256,
|
||
type: 'image/png',
|
||
});
|
||
const getFileURL = jest.fn().mockResolvedValue('/images/user-123/image.png');
|
||
getStrategyFunctions.mockReturnValue({ saveURL, getFileURL });
|
||
|
||
await processFileURL({
|
||
fileStrategy: FileSources.local,
|
||
userId: 'user-123',
|
||
URL: 'https://example.com/image.png',
|
||
fileName: 'image.png',
|
||
basePath: 'images',
|
||
context: FileContext.image_generation,
|
||
tenantId: 'tenant-a',
|
||
});
|
||
|
||
expect(getFileURL).toHaveBeenCalledWith({
|
||
userId: 'user-123',
|
||
fileName: 'user-123/image.png',
|
||
basePath: 'images',
|
||
tenantId: 'tenant-a',
|
||
});
|
||
expect(db.createFile).toHaveBeenCalledWith(
|
||
expect.objectContaining({
|
||
filepath: '/images/user-123/image.png',
|
||
tenantId: 'tenant-a',
|
||
}),
|
||
true,
|
||
);
|
||
});
|
||
});
|
||
|
||
describe('processDeleteRequest', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('removes metadata when backing storage is already missing', async () => {
|
||
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
|
||
const deleteFile = jest.fn().mockRejectedValue(missingError);
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
|
||
const result = await processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
});
|
||
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['expired-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('does not treat unrelated not found messages as missing storage', async () => {
|
||
const deleteFile = jest.fn().mockRejectedValue(new Error('Configuration not found'));
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
|
||
const result = await processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
});
|
||
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['expired-file'] });
|
||
});
|
||
|
||
it('throws metadata delete failures after storage deletion succeeds', async () => {
|
||
const deleteFile = jest.fn().mockResolvedValue(undefined);
|
||
const metadataError = new Error('mongo unavailable');
|
||
getStrategyFunctions.mockReturnValue({ deleteFile });
|
||
db.deleteFiles.mockRejectedValue(metadataError);
|
||
|
||
await expect(
|
||
processDeleteRequest({
|
||
req: {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
},
|
||
files: [
|
||
{
|
||
file_id: 'expired-file',
|
||
filepath: '/images/user-123/expired.png',
|
||
source: FileSources.local,
|
||
},
|
||
],
|
||
}),
|
||
).rejects.toThrow('mongo unavailable');
|
||
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['expired-file']);
|
||
expect(db.removeAgentResourceFilesFromAllAgents).not.toHaveBeenCalled();
|
||
});
|
||
|
||
it('deletes vector storage before removing embedded file metadata', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['embedded-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['embedded-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('keeps embedded file metadata when vector deletion fails', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const vectorDelete = jest.fn().mockRejectedValue(new Error('rag unavailable'));
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['embedded-file'] });
|
||
});
|
||
|
||
it('does not delete vector storage when primary embedded file deletion fails', async () => {
|
||
const primaryDelete = jest.fn().mockRejectedValue(new Error('permission denied'));
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).not.toHaveBeenCalled();
|
||
expect(db.deleteFiles).not.toHaveBeenCalled();
|
||
expect(result).toEqual({ deletedFileIds: [], failedFileIds: ['embedded-file'] });
|
||
});
|
||
|
||
it('still deletes vector storage when primary embedded file storage is already missing', async () => {
|
||
const missingError = Object.assign(new Error('no such file'), { code: 'ENOENT' });
|
||
const primaryDelete = jest.fn().mockRejectedValue(missingError);
|
||
const vectorDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.vectordb
|
||
? { deleteFile: vectorDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'embedded-file',
|
||
filepath: '/uploads/embedded.txt',
|
||
source: FileSources.local,
|
||
embedded: true,
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(vectorDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['embedded-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['embedded-file'], failedFileIds: [] });
|
||
});
|
||
|
||
it('deletes code environment storage before removing code resource file metadata', async () => {
|
||
const primaryDelete = jest.fn().mockResolvedValue(undefined);
|
||
const codeDelete = jest.fn().mockResolvedValue(undefined);
|
||
getStrategyFunctions.mockImplementation((source) =>
|
||
source === FileSources.execute_code
|
||
? { deleteFile: codeDelete }
|
||
: { deleteFile: primaryDelete },
|
||
);
|
||
db.deleteFiles.mockResolvedValue({ deletedCount: 1 });
|
||
const req = {
|
||
body: {},
|
||
config: {},
|
||
user: { id: 'user-123', tenantId: 'tenant-a' },
|
||
};
|
||
const file = {
|
||
file_id: 'code-resource-file',
|
||
filepath: '/uploads/code-resource.txt',
|
||
source: FileSources.local,
|
||
metadata: {
|
||
codeEnvRef: {
|
||
kind: 'agent',
|
||
id: 'agent-abc',
|
||
storage_session_id: 'sess-1',
|
||
file_id: 'fid-1',
|
||
},
|
||
},
|
||
};
|
||
|
||
const result = await processDeleteRequest({ req, files: [file] });
|
||
|
||
expect(primaryDelete).toHaveBeenCalledWith(req, file, undefined);
|
||
expect(codeDelete).toHaveBeenCalledWith(req, file);
|
||
expect(db.deleteFiles).toHaveBeenCalledWith(['code-resource-file']);
|
||
expect(result).toEqual({ deletedFileIds: ['code-resource-file'], failedFileIds: [] });
|
||
});
|
||
});
|
||
|
||
describe('sweepExpiredFiles', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('delegates expired file sweeping to the shared package with backend dependencies', async () => {
|
||
const options = {
|
||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||
limit: 1,
|
||
};
|
||
sweepExpiredFilesWithDeps.mockResolvedValue({ scanned: 1, deleted: 1, failed: 0 });
|
||
|
||
const result = await sweepExpiredFiles(options);
|
||
|
||
expect(sweepExpiredFilesWithDeps).toHaveBeenCalledWith(
|
||
options,
|
||
expect.objectContaining({
|
||
getExpiredFiles: db.getExpiredFiles,
|
||
processDeleteRequest: expect.any(Function),
|
||
incrementFileDeletionAttempts: db.incrementFileDeletionAttempts,
|
||
deferExpiredFile: db.deferExpiredFile,
|
||
logger: expect.objectContaining({
|
||
error: expect.any(Function),
|
||
info: expect.any(Function),
|
||
warn: expect.any(Function),
|
||
}),
|
||
}),
|
||
);
|
||
expect(result).toEqual({ scanned: 1, deleted: 1, failed: 0 });
|
||
});
|
||
});
|
||
|
||
describe('startExpiredFileSweep', () => {
|
||
beforeEach(() => {
|
||
jest.clearAllMocks();
|
||
});
|
||
|
||
it('delegates background sweep startup to the shared package with system context', () => {
|
||
const options = {
|
||
appConfig: { paths: { publicPath: '/tmp/public', uploads: '/tmp/uploads' } },
|
||
};
|
||
|
||
const interval = startExpiredFileSweep(options);
|
||
|
||
expect(startExpiredFileSweepWithDeps).toHaveBeenCalledWith(
|
||
options,
|
||
expect.objectContaining({
|
||
sweepExpiredFiles: expect.any(Function),
|
||
runAsSystem: expect.any(Function),
|
||
logger: expect.objectContaining({
|
||
error: expect.any(Function),
|
||
info: expect.any(Function),
|
||
warn: expect.any(Function),
|
||
}),
|
||
}),
|
||
);
|
||
expect(interval).toBe('sweep-interval');
|
||
});
|
||
});
|