LibreChat/api/server/services/Files/process.js
Danny Avila a3fc3b9dbf
🚧 fix: Gate File Search and Run Code by Role (#15691)
* fix: enforce FILE_SEARCH role permission server-side

The FILE_SEARCH role permission is stored, served by GET /api/roles/:name and
settable through the admin API, but nothing ever checks it.
PermissionTypes.FILE_SEARCH occurs zero times under api/server; the three
occurrences in @librechat/api are all in the interface-to-role sync, which
writes the permission rather than checking it.

Measured on v0.8.8-rc1 and confirmed unchanged in rc2: a user whose role has
FILE_SEARCH.USE = false can still upload a document with
tool_resource=file_search and gets 200 with "embedded": true. The same user
calling execute_code correctly gets 403.

Two gates, both mirroring toolAccessPermType in ~/server/controllers/tools.js:

* Upload (api/server/routes/files/files.js): a tool-resource-to-permission map
  checked before any branching, so it also covers the assistants path. Returns
  403 with the same log line as the RUN_CODE gate. execute_code is included
  because it had the same hole: the tool CALL was gated, the upload was not.

* Tool loading (api/app/clients/tools/util/handleTools.js): checked outside the
  lazy loader so a denied user never gets the tool equipped, rather than one
  that fails when called. Fails closed if the permission check itself throws.
  All four required imports were already present in that file, next to an
  existing FILE_CITATIONS check.

Both changes live in /api (JavaScript) rather than /packages/api (TypeScript)
as CLAUDE.md prefers for new backend code: each mirrors an existing pattern in
exactly these files, and a second permission gate in another language and
package would be harder to keep in step with the first.

Tests: api/app/clients/tools/util/handleTools.fileSearchPermission.test.js
covers permit, deny, the log line, and the fail-closed path.

* fixup! fix: enforce FILE_SEARCH role permission server-side

* 🔐 fix: Gate `file_search` and `execute_code` on Role Permissions

The `FILE_SEARCH` gate added in the previous commit lives in
`handleTools.loadTools`, which `loadAgentTools` only reaches when
`definitionsOnly` is false. That default is false on the chat path
(`Endpoints/agents/initialize.js`) but true on the responses and
OpenAI-compatible controllers, so on those paths a denied user still got
the tool definition advertised to the model and their files primed by
`primeSearchFiles`, and the call failed later in `loadToolsForExecution` —
the "gets one that fails when called" outcome the gate set out to avoid.

Move the decision to the capability filters that both loaders already run,
mirroring how `canUseMCP` resolves the `MCP_SERVERS` permission ahead of the
synchronous filter. `AgentCapabilities` stays the instance-wide deployment
switch; the role grant is a second condition a tool has to clear. Because
`hasFileSearch`/`hasExecuteCode` derive from the filtered list, priming is
skipped for a denied user too.

`RUN_CODE` had the same hole: `toolAccessPermType` in
`~/server/controllers/tools.js` gates only `POST /tools/:toolId/call`, not
the agent run, so a role without `RUN_CODE.USE` could still execute code
through an agent. Both tools go through the same map.

Checks use `checkAccessWithRequestCache` and run only for tools whose
capability is already enabled, so a run costs at most one role read. A check
that throws denies the tool.

Also formats the test file added in the previous commit, which Prettier
rejects as-is.

* 🔐 fix: Close the Remaining `RUN_CODE` and `FILE_SEARCH` Bypasses

Codex review of the previous commit found four ways past the gates. All four
share a cause: the permission map was copied per call site, so every boundary
that was not copied into stayed open.

Moves the maps and the check into `packages/api/src/tools/rolePermissions.ts`,
per CLAUDE.md ("all new backend code must be TypeScript in `/packages/api`",
"keep `/api` changes to the absolute minimum"), and points the four `/api`
boundaries at it.

- `handleTools.loadTools` now gates every tool in `toolRolePermissions`, not
  just `file_search`. It is the shared boundary the Assistants required-action
  flow crosses via `processRequiredActions`, which never passes the agent
  capability filter — so a legacy Assistant with a raw function named
  `execute_code` ran arbitrary code for a role denied `RUN_CODE`.

- The check is now request-cached (`checkAccessWithRequestCache`). The previous
  `checkAccess` call omitted `req`, so the agent path resolved the same grant
  twice and issued a second serial `getRoleByName` during chat startup, against
  the rule in AGENTS.md.

- `codeExecutionEnabled` is gated by the role result at all three loaders, not
  only the `execute_code` entry in the filtered list. It drives tool
  classification and the programmatic bash tool, so an agent pairing
  `execute_code` with an MCP `code_execution` tool still advertised and
  instantiated `run_tools_with_bash` for a denied role.

- The upload map covers `EToolResources.code_interpreter`, which the Assistants
  builder posts instead of `execute_code`, and native `code_interpreter` /
  `file_search` tools are dropped when an assistant is created or updated.
  Those run inside the provider and reach neither loader, so configuration time
  is the only place they can be gated.

Renames `handleTools.fileSearchPermission.test.js` to
`handleTools.rolePermissions.test.js` — it now covers both tools.

* 🔐 fix: Gate the Image Upload Route and the v1 Assistant Writers

Second Codex round found the same shape again: boundaries the map had not been
wired into. Adds two shared helpers next to the maps in
`packages/api/src/tools/rolePermissions.ts` so a new upload handler or assistant
writer gets the check by calling one function.

- `/files/images` accepts `tool_resource` and routes agent uploads to
  `processAgentFileUpload` on its own. The Code Files UI sends images there, so
  every image was a way around the upload boundary for a role denied
  `RUN_CODE`. Both upload routes now call
  `checkToolResourceUploadPermission`.

- That helper passes `req`, so the role read joins the request cache. The
  previous `checkAccess` call omitted it, and an allowed Assistants upload goes
  on to `processFileUpload` → `addResourceFileId` → `updateAssistant`, whose own
  check does pass `req` — two serial lookups per upload, against AGENTS.md.

- `/assistants/v1` is still mounted and its `createAssistant` / `patchAssistant`
  write native tools straight to the provider. Both now filter through
  `resolveAssistantToolPermissions`, which v2 also uses in place of its local
  copy.

Run-time enforcement for native assistant tools is deliberately not here: it
means overriding `body.tools` at `createRun` in both chat controllers, which
changes run semantics, and is better argued on its own.

* 🔐 fix: Gate Legacy `retrieval` and the Code-Environment File Tools

Third Codex round, same shape twice more.

- The v1 assistant builder submits `{ type: 'retrieval' }` where v2 submits
  `file_search` (`AssistantPanel.tsx:203`). Only the v2 spelling was mapped, so
  `resolveAssistantToolPermissions` treated legacy retrieval as ungated and both
  v1 writers preserved it for a role denied `FILE_SEARCH`. Both spellings now
  answer to the same grant.

- `codeEnvAvailable` in the agent initializer was capability-only, and
  `initializeAgent` rebuilds `bash_tool`, `read_file` and the workspace file
  tools from it — after the tool loader has already dropped `execute_code` for a
  denied role. The bash gate stops command execution, but those handlers still
  read, search, create and edit files in an attached code environment. The flag
  now carries the role grant, and short-circuits before the role read when the
  deployment has the capability off.

* 🔐 fix: Resolve Tool Grants Once Per Request and Pair Every Capability Gate

Four review rounds found thirteen issues, all one class: a boundary the
permission check had not been copied into. Rounds 2 and 3 each found gaps
created by the previous round's fix, and a manual sweep of tool constructors
and upload routes still missed the `codeEnvAvailable` family entirely — because
the thing worth enumerating is not "paths that reach a tool" but "reads of the
capability".

There are 28 such reads. This replaces per-site permission checks with one
resolution per request, pairs every read that is a gate, and pins the list so a
new one cannot be added silently.

- `resolveToolRoleGrants` resolves `RUN_CODE` and `FILE_SEARCH` together and
  memoizes on the request. Both perf findings disappear by construction: gates
  share one role read instead of each issuing its own.

- Newly paired gates: `codeEnvAvailable` in the OpenAI-compatible, Responses and
  memory-agent initializers (`initializeAgent` rebuilds `bash_tool`, `read_file`
  and the workspace file tools from it, and hands their handlers the code
  environment); the capability checks in agent upload processing and in
  agent-management upload purposes; and the library chat-completion service,
  behind an optional `getRoleByName` dependency so embedders are not broken.

- The v1 Knowledge upload posts `assistant_id` with no `tool_resource`, so there
  is no resource to authorize. It now reads the assistant's own native tools and
  requires their grants, instead of treating a missing resource as unrestricted.

- The startup role read moved off the critical path into the existing
  `Promise.all`, and the direct tool-call check passes `req` so the shared
  loader gate reuses its result rather than repeating the lookup.

- `toolCapabilityGates.spec.js` pins every file that reads either capability
  with a count and a reason. A new read fails the suite until it is classified,
  so the next boundary announces itself instead of waiting for a reviewer.

* 🔐 fix: Abort Assistant Writes on Role-Lookup Failure, Authorize Legacy Uploads First

Round 5 found no missed boundary — the inventory held. All three findings are
about how the gates behave.

- Failing closed is right where a denial blocks the operation, and wrong where a
  denial instead filters a payload the caller persists. A transient role-store
  outage during an assistant create or patch was silently stripping
  `code_interpreter` / `file_search` / `retrieval` and saving the result, turning
  a momentary failure into permanent config loss. `checkToolRolePermission`
  gains `throwOnError`, and `resolveAssistantToolPermissions` uses it: every one
  of its callers either persists the filtered list or rejects the request, so a
  lookup failure has to propagate.

- The legacy-assistant upload gate ran after `sanitizedUploadFn` had already
  pushed the file to the provider, so a denied role left an untracked remote file
  and got a 500 for what is a 403. Moved to the route, next to the tool-resource
  gate, before any bytes are sent.

- `resendFiles` hydration derived its resource set from unfiltered `agent.tools`,
  so a denied role still paid the thread walk and code-file reads for a tool
  about to be dropped. The `execute_code` half now keys off
  `effectiveCodeEnvAvailable`, which already carries the grant.

The `file_search` half of that last one needs the grant threaded into
`initializeAgent`, which is plumbing in the most delicate file here for a latency
win on an already-correct path — deliberately left for a follow-up.

* 🧪 fix: Declare `resolveToolRoleGrants` in the Upload Spec's Module Stub

`Tests: api (shard 2/3)` failed on `a0b38c3`: `process.spec.js` stubs
`@librechat/api` wholesale, so the `resolveToolRoleGrants` call added to the
upload gate resolved to undefined and threw. Five failures in
`processAgentFileUpload`, plus one cascading `processFileURL` assertion from the
same suite.

The gate was verified through its own spec and the routes above it, but not
through the spec of the file it lives in — `process.spec.js` was never run
locally. Every spec belonging to a file this PR touches now has been, and the
mongodb-backed ones that still cannot run here were checked statically for the
same stub shape: all but `process.integration.spec.js` spread `requireActual`,
and that one never reaches these paths.

* ⚡ fix: Skip the Role Lookup When Code Execution Is Disabled

The browser initializer already guards this; the OpenAI-compatible and Responses
controllers did not, so every request on those endpoints issued a role read whose
result the capability short-circuit then discarded. Both now start the lookup
only when the deployment has `execute_code` enabled, matching the third.

* ⚡ fix: Drop Three Redundant Reads on the Gated Paths

- The memory initializer resolved grants unconditionally, so a deployment
  without `execute_code` paid a role read for a flag that could only be false.
  Guarded like the three chat initializers.

- Agent-management upload purposes resolved grants before `checkCapability`, so
  a purpose the deployment has switched off paid a role read on the way to being
  rejected. Capability first, then the grant.

- The legacy-assistant preflight builds an OpenAI client to read the assistant's
  tools, and `processFileUpload` then built a second one — re-reading the user's
  key expiry and values. The preflight now hands its client on, and processing
  builds one only when it wasn't given one.

---------

Co-authored-by: Paul <200737214+SSIG-IT@users.noreply.github.com>
2026-09-07 07:03:22 -04:00

1469 lines
49 KiB
JavaScript

const fs = require('fs');
const path = require('path');
const mime = require('mime');
const { v4 } = require('uuid');
const {
isUUID,
megabyte,
FileContext,
FileSources,
imageExtRegex,
EModelEndpoint,
EToolResources,
mergeCodeEnvRef,
mergeFileConfig,
AgentCapabilities,
checkOpenAIStorage,
removeNullishValues,
isAssistantsEndpoint,
getEndpointFileConfig,
} = require('librechat-data-provider');
const { logger, runAsSystem } = require('@librechat/data-schemas');
const {
sanitizeFilename,
parseText,
processAudioFile,
extractInspectableFileText,
assertExtractedTextInspectable,
getFileExtractionLogDetails,
getUploadExtractedTextPlan,
UPLOAD_EXTRACTED_TEXT_PLANS,
inspectContent,
extractFileContent,
hasActiveFileFieldPolicy,
sendUploadSuccess,
getStorageMetadata,
contentFilterBlockResponse,
sweepExpiredFiles: sweepExpiredFilesWithDeps,
startExpiredFileSweep: startExpiredFileSweepWithDeps,
resolveToolRoleGrants,
} = require('@librechat/api');
const {
convertImage,
resizeAndConvert,
resizeImageBuffer,
} = require('~/server/services/Files/images');
const { addResourceFileId, deleteResourceFileId } = require('~/server/controllers/assistants/v2');
const { getOpenAIClient } = require('~/server/controllers/assistants/helpers');
const { loadAuthValues } = require('~/server/services/Tools/credentials');
const { getFileStrategy } = require('~/server/utils/getFileStrategy');
const { checkCapability } = require('~/server/services/Config');
const { LB_QueueAsyncCall } = require('~/server/utils/queue');
const { getRetentionExpiry, getAgentFileRetentionExpiry } = require('./retention');
const { getStrategyFunctions } = require('./strategies');
const { determineFileType } = require('~/server/utils');
const { STTService } = require('./Audio/STTService');
const db = require('~/models');
/**
* Creates a modular file upload wrapper that ensures filename sanitization
* across all storage strategies. This prevents storage-specific implementations
* from having to handle sanitization individually.
*
* @param {Function} uploadFunction - The storage strategy's upload function
* @returns {Function} - Wrapped upload function with sanitization
*/
const createSanitizedUploadWrapper = (uploadFunction) => {
return async (params) => {
const { req, file, file_id, ...restParams } = params;
// Create a modified file object with sanitized original name
// This ensures consistent filename handling across all storage strategies
const sanitizedFile = {
...file,
originalname: sanitizeFilename(file.originalname),
};
return uploadFunction({ req, file: sanitizedFile, file_id, ...restParams });
};
};
const hasCodeEnvRef = (file) =>
file?.metadata?.codeEnvRef != null || file?.metadata?.codeEnvRefs != null;
const isMissingStorageError = (err) => {
const code = err?.code ?? err?.status ?? err?.statusCode ?? err?.response?.status;
if ([404, '404', 'ENOENT', 'NoSuchKey', 'NotFound', 'ResourceNotFound'].includes(code)) {
return true;
}
return /(?:file|object|blob|key|resource) (?:not found|does not exist)|no such (?:file|key)/i.test(
String(err?.message ?? ''),
);
};
/**
* Enqueues the delete operation to the leaky bucket queue if necessary, or adds it directly to promises.
*
* @param {object} params - The passed parameters.
* @param {ServerRequest} params.req - The express request object.
* @param {MongoFile} params.file - The file object to delete.
* @param {Function} params.deleteFile - The delete file function.
* @param {Promise[]} params.promises - The array of promises to await.
* @param {Set<string>} params.resolvedFileIds - File IDs whose storage delete succeeded.
* @param {Set<string>} params.failedFileIds - File IDs whose storage delete failed.
* @param {OpenAI | undefined} [params.openai] - If an OpenAI file, the initialized OpenAI client.
*/
function enqueueDeleteOperation({
req,
file,
deleteFile,
promises,
resolvedFileIds,
failedFileIds,
openai,
}) {
if (checkOpenAIStorage(file.source)) {
// Enqueue to leaky bucket
promises.push(
new Promise((resolve, reject) => {
LB_QueueAsyncCall(
() => deleteFile(req, file, openai),
[],
(err, result) => {
if (err) {
if (isMissingStorageError(err)) {
resolvedFileIds.add(file.file_id);
logger.warn('File storage was already missing during delete', err);
resolve(result);
return;
}
failedFileIds.add(file.file_id);
logger.error('Error deleting file from OpenAI source', err);
reject(err);
} else {
resolvedFileIds.add(file.file_id);
resolve(result);
}
},
);
}),
);
} else {
// Add directly to promises
promises.push(
deleteFile(req, file)
.then(() => resolvedFileIds.add(file.file_id))
.catch((err) => {
if (isMissingStorageError(err)) {
resolvedFileIds.add(file.file_id);
logger.warn('File storage was already missing during delete', err);
return;
}
failedFileIds.add(file.file_id);
logger.error('Error deleting file', err);
return Promise.reject(err);
}),
);
}
}
const getDeleteMethod = ({ source, deletionMethods }) => {
if (deletionMethods[source]) {
return deletionMethods[source];
}
const { deleteFile } = getStrategyFunctions(source);
if (!deleteFile) {
throw new Error(`Delete function not implemented for ${source}`);
}
deletionMethods[source] = deleteFile;
return deleteFile;
};
const createDeleteFileWithSecondaryStorage = ({ source, deleteFile, deletionMethods }) => {
return async (req, file, openai) => {
const secondaryDeleteMethods = [];
if (file.embedded === true && source !== FileSources.vectordb) {
secondaryDeleteMethods.push(
getDeleteMethod({ source: FileSources.vectordb, deletionMethods }),
);
}
if (hasCodeEnvRef(file) && source !== FileSources.execute_code) {
secondaryDeleteMethods.push(
getDeleteMethod({ source: FileSources.execute_code, deletionMethods }),
);
}
try {
await deleteFile(req, file, openai);
} catch (err) {
if (!isMissingStorageError(err)) {
throw err;
}
logger.warn('Primary file storage was already missing during delete', err);
}
await Promise.all(
secondaryDeleteMethods.map((secondaryDeleteFile) => secondaryDeleteFile(req, file)),
);
};
};
// TODO: refactor as currently only image files can be deleted this way
// as other filetypes will not reside in public path
/**
* Deletes a list of files from the server filesystem and the database.
*
* @param {Object} params - The params object.
* @param {MongoFile[]} params.files - The file objects to delete.
* @param {ServerRequest} params.req - The express request object.
* @param {DeleteFilesBody} params.req.body - The request body.
* @param {string} [params.req.body.agent_id] - The agent ID if file uploaded is associated to an agent.
* @param {string} [params.req.body.assistant_id] - The assistant ID if file uploaded is associated to an assistant.
* @param {string} [params.req.body.tool_resource] - The tool resource if assistant file uploaded is associated to a tool resource.
*
* @returns {Promise<{ deletedFileIds: string[], failedFileIds: string[] }>}
* @throws {Error} When storage deletion cannot be scheduled or file metadata cleanup fails.
*/
const processDeleteRequest = async ({ req, files }) => {
const appConfig = req.config;
const resolvedFileIds = new Set();
const failedFileIds = new Set();
const deletionMethods = {};
const promises = [];
/** @type {Record<string, OpenAI | undefined>} */
const client = { [FileSources.openai]: undefined, [FileSources.azure]: undefined };
const initializeClients = async () => {
if (appConfig.endpoints?.[EModelEndpoint.assistants]) {
const openAIClient = await getOpenAIClient({
req,
overrideEndpoint: EModelEndpoint.assistants,
});
client[FileSources.openai] = openAIClient.openai;
}
if (!appConfig.endpoints?.[EModelEndpoint.azureOpenAI]?.assistants) {
return;
}
const azureClient = await getOpenAIClient({
req,
overrideEndpoint: EModelEndpoint.azureAssistants,
});
client[FileSources.azure] = azureClient.openai;
};
if (req.body.assistant_id !== undefined) {
await initializeClients();
}
const agentFiles = [];
for (const file of files) {
const source = file.source ?? FileSources.local;
if (req.body.agent_id && req.body.tool_resource) {
agentFiles.push({
tool_resource: req.body.tool_resource,
file_id: file.file_id,
});
}
if (source === FileSources.text) {
resolvedFileIds.add(file.file_id);
continue;
}
if (checkOpenAIStorage(source) && !client[source]) {
await initializeClients();
}
const openai = client[source];
if (req.body.assistant_id && req.body.tool_resource) {
promises.push(
deleteResourceFileId({
req,
openai,
file_id: file.file_id,
assistant_id: req.body.assistant_id,
tool_resource: req.body.tool_resource,
}),
);
} else if (req.body.assistant_id) {
promises.push(openai.beta.assistants.files.del(req.body.assistant_id, file.file_id));
}
const deleteFile = getDeleteMethod({ source, deletionMethods });
enqueueDeleteOperation({
req,
file,
deleteFile: createDeleteFileWithSecondaryStorage({ source, deleteFile, deletionMethods }),
promises,
resolvedFileIds,
failedFileIds,
openai,
});
}
if (agentFiles.length > 0) {
promises.push(
db.removeAgentResourceFiles({
agent_id: req.body.agent_id,
files: agentFiles,
}),
);
}
await Promise.allSettled(promises);
const deletedFileIds = [...resolvedFileIds];
let metadataDeletedFileIds = deletedFileIds;
if (deletedFileIds.length > 0) {
try {
await db.deleteFiles(deletedFileIds);
} catch (error) {
logger.error('Error deleting file metadata after storage deletion', error);
deletedFileIds.forEach((fileId) => failedFileIds.add(fileId));
metadataDeletedFileIds = [];
throw error;
}
if (metadataDeletedFileIds.length > 0) {
try {
await db.removeAgentResourceFilesFromAllAgents({ file_ids: metadataDeletedFileIds });
} catch (error) {
logger.error('Error cleaning up orphaned agent file references', error);
}
}
}
return {
deletedFileIds: metadataDeletedFileIds,
failedFileIds: [...failedFileIds],
};
};
/**
* Deletes expired file storage before removing the corresponding File records.
*
* Mongo TTL indexes delete only the metadata document, so file retention uses
* this application sweep for records with `expiredAt` instead.
*
* @param {object} params
* @param {AppConfig} params.appConfig
* @param {number} [params.limit]
* @param {() => Promise<AppConfig>} [params.loadAppConfig]
* @returns {Promise<{ scanned: number, deleted: number, failed: number }>}
*/
async function sweepExpiredFiles(options = {}) {
return sweepExpiredFilesWithDeps(options, {
getExpiredFiles: db.getExpiredFiles,
processDeleteRequest,
incrementFileDeletionAttempts: db.incrementFileDeletionAttempts,
deferExpiredFile: db.deferExpiredFile,
logger,
});
}
function startExpiredFileSweep(options = {}) {
return startExpiredFileSweepWithDeps(options, {
sweepExpiredFiles,
runAsSystem,
logger,
});
}
/**
* Processes a file URL using a specified file handling strategy. This function accepts a strategy name,
* fetches the corresponding file processing functions (for saving and retrieving file URLs), and then
* executes these functions in sequence. It first saves the file using the provided URL and then retrieves
* the URL of the saved file. If any error occurs during this process, it logs the error and throws an
* exception with an appropriate message.
*
* @param {Object} params - The parameters object.
* @param {FileSources} params.fileStrategy - The file handling strategy to use.
* Must be a value from the `FileSources` enum, which defines different file
* handling strategies (like saving to Firebase, local storage, etc.).
* @param {string} params.userId - The user's unique identifier. Used for creating user-specific paths or
* references in the file handling process.
* @param {string} params.URL - The URL of the file to be processed.
* @param {string} params.fileName - The name that will be used to save the file (including extension)
* @param {string} params.basePath - The base path or directory where the file will be saved or retrieved from.
* @param {FileContext} params.context - The context of the file (e.g., 'avatar', 'image_generation', etc.)
* @param {string} [params.tenantId] - Optional tenant identifier for tenant-prefixed storage paths.
* @param {ServerRequest} [params.req] - Request context used to apply data retention metadata.
* @returns {Promise<MongoFile>} A promise that resolves to the DB representation (MongoFile)
* of the processed file. It throws an error if the file processing fails at any stage.
*/
const processFileURL = async ({
fileStrategy,
userId,
URL,
fileName,
basePath,
context,
tenantId,
req,
}) => {
const { saveURL, getFileURL } = getStrategyFunctions(fileStrategy);
try {
const savedFile = await saveURL({ userId, URL, fileName, basePath, tenantId });
if (!savedFile) {
throw new Error(`Strategy "${fileStrategy}" did not save "${fileName}"`);
}
const {
bytes = 0,
type = '',
dimensions = {},
} = typeof savedFile === 'string' ? {} : savedFile;
const fallbackFileName =
fileStrategy === FileSources.local || fileStrategy === FileSources.firebase
? `${userId}/${fileName}`
: fileName;
const filepath =
typeof savedFile === 'string'
? savedFile
: (savedFile.filepath ??
(await getFileURL({ userId, fileName: fallbackFileName, basePath, tenantId })));
if (!filepath) {
throw new Error(`Strategy "${fileStrategy}" did not return a file URL for "${fileName}"`);
}
const storageMetadata = getStorageMetadata({
filepath,
source: fileStrategy,
storageKey: typeof savedFile === 'string' ? undefined : savedFile.storageKey,
storageRegion: typeof savedFile === 'string' ? undefined : savedFile.storageRegion,
});
return await db.createFile(
{
user: userId,
file_id: v4(),
bytes,
filepath,
...storageMetadata,
filename: fileName,
source: fileStrategy,
type,
context,
...(await getRetentionExpiry(req)),
tenantId,
width: dimensions.width,
height: dimensions.height,
},
true,
);
} catch (error) {
logger.error(`Error while processing the image with ${fileStrategy}:`, error);
throw new Error(`Failed to process the image with ${fileStrategy}. ${error.message}`);
}
};
/**
* Applies the current strategy for image uploads.
* Saves file metadata to the database with an expiry TTL.
*
* @param {Object} params - The parameters object.
* @param {ServerRequest} params.req - The Express request object.
* @param {Express.Response} [params.res] - The Express response object.
* @param {ImageMetadata} params.metadata - Additional metadata for the file.
* @param {boolean} params.returnFile - Whether to return the file metadata or return response as normal.
* @param {import('@librechat/api').UploadSseStream | null} [params.sseStream] - Active upload SSE stream, if enabled.
* @returns {Promise<void>}
*/
const processImageFile = async ({ req, res, metadata, returnFile = false, sseStream }) => {
const { file } = req;
const appConfig = req.config;
const source = getFileStrategy(appConfig, { isImage: true });
const { handleImageUpload } = getStrategyFunctions(source);
const { file_id, temp_file_id, endpoint } = metadata;
const { filepath, bytes, width, height, storageKey, storageRegion } = await handleImageUpload({
req,
file,
file_id,
endpoint,
});
const storageMetadata = getStorageMetadata({ filepath, source, storageKey, storageRegion });
const result = await db.createFile(
{
user: req.user.id,
file_id,
temp_file_id,
bytes,
filepath,
...storageMetadata,
filename: file.originalname,
context: FileContext.message_attachment,
source,
type: `image/${appConfig.imageOutputType}`,
...(await getRetentionExpiry(req)),
width,
height,
tenantId: req.user.tenantId,
},
true,
);
if (returnFile) {
return result;
}
sendUploadSuccess(res, sseStream, 'File uploaded and processed successfully', result);
};
/**
* Applies the current strategy for image uploads and
* returns minimal file metadata, without saving to the database.
*
* @param {Object} params - The parameters object.
* @param {ServerRequest} params.req - The Express request object.
* @param {FileContext} params.context - The context of the file (e.g., 'avatar', 'image_generation', etc.)
* @param {boolean} [params.resize=true] - Whether to resize and convert the image to target format. Default is `true`.
* @param {{ buffer: Buffer, width: number, height: number, bytes: number, filename: string, type: string, file_id: string }} [params.metadata] - Required metadata for the file if resize is false.
* @returns {Promise<{ filepath: string, filename: string, source: string, type: string}>}
*/
const uploadImageBuffer = async ({ req, context, metadata = {}, resize = true }) => {
const appConfig = req.config;
const source = getFileStrategy(appConfig, { isImage: true });
const { saveBuffer } = getStrategyFunctions(source);
let { buffer, width, height, bytes, filename, file_id, type } = metadata;
if (resize) {
file_id = v4();
type = `image/${appConfig.imageOutputType}`;
({ buffer, width, height, bytes } = await resizeAndConvert({
inputBuffer: buffer,
desiredFormat: appConfig.imageOutputType,
}));
filename = `${path.basename(req.file.originalname, path.extname(req.file.originalname))}.${
appConfig.imageOutputType
}`;
}
const fileName = `${file_id}-${filename}`;
const filepath = await saveBuffer({
userId: req.user.id,
fileName,
buffer,
tenantId: req.user.tenantId,
});
const storageMetadata = getStorageMetadata({ filepath, source });
return await db.createFile(
{
user: req.user.id,
file_id,
bytes,
filepath,
...storageMetadata,
filename,
context,
source,
type,
width,
...(await getRetentionExpiry(req)),
height,
tenantId: req.user.tenantId,
},
true,
);
};
/**
* Applies the current strategy for file uploads.
* Saves file metadata to the database with an expiry TTL.
* Files must be deleted from the server filesystem manually.
*
* @param {Object} params - The parameters object.
* @param {ServerRequest} params.req - The Express request object.
* @param {Express.Response} params.res - The Express response object.
* @param {FileMetadata} params.metadata - Additional metadata for the file.
* @param {import('@librechat/api').UploadSseStream | null} [params.sseStream] - Active upload SSE stream, if enabled.
* @returns {Promise<void>}
*/
/**
* @param {OpenAI} [params.openai] - Client the caller already built (the legacy
* assistant preflight needs one to authorize). Reused rather than rebuilt, since
* constructing it re-reads the user's key.
*/
const processFileUpload = async ({ req, res, metadata, sseStream, openai: providedOpenAI }) => {
const appConfig = req.config;
const isAssistantUpload = isAssistantsEndpoint(metadata.endpoint);
const assistantSource =
metadata.endpoint === EModelEndpoint.azureAssistants ? FileSources.azure : FileSources.openai;
// Use the configured file strategy for regular file uploads (not vectordb)
const source = isAssistantUpload ? assistantSource : appConfig.fileStrategy;
const { handleFileUpload } = getStrategyFunctions(source);
const { file_id, temp_file_id = null } = metadata;
/** @type {OpenAI | undefined} */
let openai = providedOpenAI;
if (openai == null && checkOpenAIStorage(source)) {
({ openai } = await getOpenAIClient({ req }));
}
const { file } = req;
const sanitizedUploadFn = createSanitizedUploadWrapper(handleFileUpload);
const {
id,
bytes,
filename,
filepath: _filepath,
storageKey: _storageKey,
storageRegion: _storageRegion,
embedded,
height,
width,
} = await sanitizedUploadFn({
req,
file,
file_id,
openai,
});
if (isAssistantUpload && !metadata.message_file && !metadata.tool_resource) {
/** Authorized at the route before any bytes are sent — see
* `assertLegacyAssistantUploadAllowed` in `~/server/routes/files/files`. */
await openai.beta.assistants.files.create(metadata.assistant_id, {
file_id: id,
});
} else if (isAssistantUpload && !metadata.message_file) {
await addResourceFileId({
req,
openai,
file_id: id,
assistant_id: metadata.assistant_id,
tool_resource: metadata.tool_resource,
});
}
let filepath = isAssistantUpload ? `${openai.baseURL}/files/${id}` : _filepath;
let storageMetadata = getStorageMetadata({
filepath,
source,
storageKey: _storageKey,
storageRegion: _storageRegion,
});
if (isAssistantUpload && file.mimetype.startsWith('image')) {
const result = await processImageFile({
req,
file,
metadata: { file_id: v4() },
returnFile: true,
});
filepath = result.filepath;
storageMetadata = getStorageMetadata({
filepath,
source: result.source,
storageKey: result.storageKey,
storageRegion: result.storageRegion,
});
}
const result = await db.createFile(
{
user: req.user.id,
file_id: id ?? file_id,
temp_file_id,
bytes,
filepath,
...storageMetadata,
filename: filename ?? sanitizeFilename(file.originalname),
context: isAssistantUpload ? FileContext.assistants : FileContext.message_attachment,
model: isAssistantUpload ? req.body.model : undefined,
type: file.mimetype,
...(await getRetentionExpiry(req)),
embedded,
source,
height,
width,
tenantId: req.user.tenantId,
},
true,
);
sendUploadSuccess(res, sseStream, 'File uploaded and processed successfully', result);
};
/**
* Applies the current strategy for file uploads.
* Saves file metadata to the database with an expiry TTL.
* Files must be deleted from the server filesystem manually.
*
* @param {Object} params - The parameters object.
* @param {ServerRequest} params.req - The Express request object.
* @param {Express.Response} params.res - The Express response object.
* @param {FileMetadata} params.metadata - Additional metadata for the file.
* @param {import('@librechat/api').UploadSseStream | null} [params.sseStream] - Active upload SSE stream, if enabled.
* @returns {Promise<void>}
*/
const processAgentFileUpload = async ({ req, res, metadata, sseStream }) => {
const { file } = req;
const appConfig = req.config;
const { agent_id, tool_resource, file_id, temp_file_id = null } = metadata;
let messageAttachment = !!metadata.message_file;
if (agent_id && !tool_resource && !messageAttachment) {
throw new Error('No tool resource provided for agent file upload');
}
if (tool_resource === EToolResources.file_search && file.mimetype.startsWith('image')) {
throw new Error('Image uploads are not supported for file search tool resources');
}
if (!messageAttachment && !agent_id) {
throw new Error('No agent ID provided for agent file upload');
}
const isImage = file.mimetype.startsWith('image');
let fileInfoMetadata;
const entity_id = messageAttachment === true ? undefined : agent_id;
const basePath = mime.getType(file.originalname)?.startsWith('image') ? 'images' : 'uploads';
if (tool_resource === EToolResources.execute_code) {
const isCodeEnabled =
(await checkCapability(req, AgentCapabilities.execute_code)) &&
(await resolveToolRoleGrants({ req, getRoleByName: db.getRoleByName, context: 'fileUpload' }))
.runCode;
if (!isCodeEnabled) {
throw new Error('Code execution is not enabled for Agents');
}
const { handleFileUpload: uploadCodeEnvFile } = getStrategyFunctions(FileSources.execute_code);
const stream = fs.createReadStream(file.path);
/* Resource identity for codeapi's sessionKey:
* - chat attachments (messageAttachment=true): `kind: 'user'`, codeapi
* buckets under `<tenant>:user:<authContext.userId>` regardless of `id`.
* - agent setup files (messageAttachment=false): `kind: 'agent'`, shared
* per agent identity. `id` carries the agent id. */
const codeKind = messageAttachment === true ? 'user' : 'agent';
const codeId = messageAttachment === true ? req.user.id : agent_id;
/* Upload under the same sanitized filename LC stores in its DB
* (`fileInfo.filename` below uses `sanitizeFilename(originalname)`).
* Codeapi/file_server use this as the on-disk name in the sandbox
* — `/mnt/data/<filename>` — and `primeFiles`'s `toolContext` text
* + `_injected_files.name` both reference `file.filename`. Sending
* the unsanitized `file.originalname` here makes the sandbox path
* (with spaces / special chars) drift from what LC tells the model
* is available, causing FileNotFoundError on the first reference. */
const sandboxFilename = sanitizeFilename(file.originalname);
const uploaded = await uploadCodeEnvFile({
req,
stream,
filename: sandboxFilename,
kind: codeKind,
id: codeId,
});
/* Persist under the structured `codeEnvRef` shape — the only key the
* post-cutover schema (`metadata.codeEnvRef`) and downstream readers
* (`primeFiles`, `getCodeFilesByIds`, `categorizeFileForToolResources`,
* controller filtering) accept. Storing under the legacy
* `fileIdentifier` key would be silently dropped by mongoose strict
* mode and the file would lose its sandbox reference on subsequent
* priming turns. */
fileInfoMetadata = mergeCodeEnvRef(undefined, {
kind: codeKind,
id: codeId,
storage_session_id: uploaded.storage_session_id,
file_id: uploaded.file_id,
executionProfile: 'default',
});
} else if (tool_resource === EToolResources.file_search) {
const isFileSearchEnabled =
(await checkCapability(req, AgentCapabilities.file_search)) &&
(await resolveToolRoleGrants({ req, getRoleByName: db.getRoleByName, context: 'fileUpload' }))
.fileSearch;
if (!isFileSearchEnabled) {
throw new Error('File search is not enabled for Agents');
}
// Note: File search processing continues to dual storage logic below
} else if (tool_resource === EToolResources.context) {
const { file_id, temp_file_id = null } = metadata;
const getExtractionLogDetails = (error) =>
getFileExtractionLogDetails({
filters: appConfig?.filters,
filename: file.originalname,
fileId: file_id,
error,
});
const { fileLabel: extractionFileLabel } = getExtractionLogDetails(undefined);
/**
* @param {object} params
* @param {string} params.text
* @param {number} params.bytes
* @param {string} params.filepath
* @param {string} params.type
* @param {boolean} params.isTranscript
* @return {Promise<void>}
*/
const createTextFile = async ({
text,
bytes,
filepath,
type = 'text/plain',
isTranscript = false,
}) => {
if (!isTranscript) {
assertExtractedTextInspectable({
filters: appConfig?.filters,
text,
});
}
const textBytes = Buffer.byteLength(text, 'utf8');
if (textBytes > 15 * megabyte) {
throw new Error(
`Extracted text from "${file.originalname}" exceeds the 15MB storage limit (${Math.round(textBytes / megabyte)}MB). Try a shorter document.`,
);
}
if (
hasActiveFileFieldPolicy(appConfig?.filters, [
isTranscript ? 'transcript' : 'extracted_text',
])
) {
const content = isTranscript ? { transcript: text } : { extractedText: text };
const finding = inspectContent(extractFileContent(content), {
filters: appConfig.filters,
});
if (finding != null) {
const blockResponse = contentFilterBlockResponse(finding);
if (sseStream) {
sseStream.sendError({
...blockResponse,
code: 400,
temp_file_id,
tool_resource,
display_to_user: true,
});
} else {
res.status(400).json(blockResponse);
}
return;
}
}
const retentionExpiry = await getAgentFileRetentionExpiry({
req,
messageAttachment,
tool_resource,
});
const fileInfo = {
...removeNullishValues({
text,
bytes,
file_id,
temp_file_id,
user: req.user.id,
type,
filepath: filepath ?? file.path,
source: FileSources.text,
filename: file.originalname,
model: messageAttachment ? undefined : req.body.model,
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
tenantId: req.user.tenantId,
}),
...retentionExpiry,
};
if (!messageAttachment && tool_resource) {
await db.addAgentResourceFile({
file_id,
agent_id,
tool_resource,
updatingUserId: req?.user?.id,
});
}
const result = await db.createFile(fileInfo, true);
sendUploadSuccess(res, sseStream, 'Agent file uploaded and processed successfully', result);
};
const fileConfig = mergeFileConfig(appConfig.fileConfig);
const extractedTextPlan = getUploadExtractedTextPlan({
endpoint: metadata.endpoint,
toolResource: tool_resource,
mimeType: file.mimetype,
fileConfig,
ocrConfigured: appConfig?.ocr != null,
ragConfigured: !!process.env.RAG_API_URL,
});
const shouldUseConfiguredOCR = extractedTextPlan === UPLOAD_EXTRACTED_TEXT_PLANS.configuredOCR;
const shouldUseConfiguredText = extractedTextPlan === UPLOAD_EXTRACTED_TEXT_PLANS.configuredRAG;
const shouldUseDocumentParser =
extractedTextPlan === UPLOAD_EXTRACTED_TEXT_PLANS.documentParser;
const shouldUseOCR = shouldUseConfiguredOCR || shouldUseDocumentParser;
const resolveDocumentText = async () => {
if (shouldUseConfiguredOCR) {
try {
const ocrStrategy = appConfig?.ocr?.strategy ?? FileSources.document_parser;
const { handleFileUpload } = getStrategyFunctions(ocrStrategy);
return await handleFileUpload({ req, file, loadAuthValues });
} catch (err) {
const { errorMetadata } = getExtractionLogDetails(err);
logger.error(
`[processAgentFileUpload] Configured OCR failed for ${extractionFileLabel}, falling back to document_parser:`,
errorMetadata,
);
}
}
try {
const { handleFileUpload } = getStrategyFunctions(FileSources.document_parser);
return await handleFileUpload({ req, file, loadAuthValues });
} catch (err) {
const { errorMetadata } = getExtractionLogDetails(err);
logger.error(
`[processAgentFileUpload] Document parser failed for ${extractionFileLabel}:`,
errorMetadata,
);
throw err;
}
};
if (shouldUseConfiguredOCR && !(await checkCapability(req, AgentCapabilities.ocr))) {
throw new Error('OCR capability is not enabled for Agents');
}
if (shouldUseOCR) {
const ocrResult = await extractInspectableFileText({
filters: appConfig?.filters,
extract: resolveDocumentText,
});
if (ocrResult) {
const { text, bytes, filepath: ocrFileURL } = ocrResult;
return await createTextFile({ text, bytes, filepath: ocrFileURL });
}
throw new Error(
`Unable to extract text from "${file.originalname}". The document may be image-based and requires an OCR service to process.`,
);
}
const shouldUseSTT = fileConfig.checkType(
file.mimetype,
fileConfig.stt?.supportedMimeTypes || [],
);
if (shouldUseSTT) {
const sttService = await STTService.getInstance();
const { text, bytes } = await processAudioFile({ req, file, sttService });
return await createTextFile({ text, bytes, type: file.mimetype, isTranscript: true });
}
const shouldUseText = fileConfig.checkType(
file.mimetype,
fileConfig.text?.supportedMimeTypes || [],
);
if (!shouldUseText) {
throw new Error(`File type ${file.mimetype} is not supported for text parsing.`);
}
/**
* A document type the admin routed to configured text extraction: prefer RAG `/text`, but fall
* back to the built-in document parser (not raw native text) when RAG is unavailable, so a
* transient outage doesn't degrade a docx/pdf to unreadable bytes. Only the RAG extraction is
* inside the fallback catch: a downstream persistence failure (size guard, DB, agent-resource
* mutation) must surface as itself, not trigger a second extraction attempt.
*/
if (shouldUseConfiguredText) {
let configuredText;
try {
configuredText = await parseText({ req, file, file_id, allowNativeFallback: false });
} catch (err) {
const { errorMetadata } = getExtractionLogDetails(err);
logger.warn(
`[processAgentFileUpload] Configured RAG text extraction unavailable for ${extractionFileLabel}, using built-in document parser:`,
errorMetadata,
);
const documentText = await extractInspectableFileText({
filters: appConfig?.filters,
extract: resolveDocumentText,
});
if (!documentText) {
throw new Error(
`Unable to extract text from "${file.originalname}". RAG text extraction was unavailable and the built-in parser produced no result.`,
);
}
const { text, bytes, filepath: docFileURL } = documentText;
return await createTextFile({ text, bytes, filepath: docFileURL });
}
return await createTextFile({
text: configuredText.text,
bytes: configuredText.bytes,
type: file.mimetype,
});
}
const { text, bytes } = await extractInspectableFileText({
filters: appConfig?.filters,
extract: () => parseText({ req, file, file_id }),
});
return await createTextFile({ text, bytes, type: file.mimetype });
}
// Dual storage pattern for RAG files: Storage + Vector DB
let storageResult, embeddingResult;
const isImageFile = file.mimetype.startsWith('image');
const source = getFileStrategy(appConfig, { isImage: isImageFile });
if (tool_resource === EToolResources.file_search) {
// FIRST: Upload to Storage for permanent backup (S3/local/etc.)
const { handleFileUpload } = getStrategyFunctions(source);
const sanitizedUploadFn = createSanitizedUploadWrapper(handleFileUpload);
storageResult = await sanitizedUploadFn({
req,
file,
file_id,
basePath,
entity_id,
});
// SECOND: Upload to Vector DB
const { uploadVectors } = require('./VectorDB/crud');
embeddingResult = await uploadVectors({
req,
file,
file_id,
entity_id,
});
// Vector status will be stored at root level, no need for metadata
fileInfoMetadata = {};
} else {
// Standard single storage for non-RAG files
const { handleFileUpload } = getStrategyFunctions(source);
const sanitizedUploadFn = createSanitizedUploadWrapper(handleFileUpload);
storageResult = await sanitizedUploadFn({
req,
file,
file_id,
basePath,
entity_id,
});
}
let {
bytes,
filename,
filepath: _filepath,
storageKey: _storageKey,
storageRegion: _storageRegion,
height,
width,
} = storageResult;
// For RAG files, use embedding result; for others, use storage result
let embedded = storageResult.embedded;
if (tool_resource === EToolResources.file_search) {
embedded = embeddingResult?.embedded;
filename = embeddingResult?.filename || filename;
}
let filepath = _filepath;
let storageMetadata = getStorageMetadata({
filepath,
source,
storageKey: _storageKey,
storageRegion: _storageRegion,
});
if (!messageAttachment && tool_resource) {
await db.addAgentResourceFile({
file_id,
agent_id,
tool_resource,
updatingUserId: req?.user?.id,
});
}
if (isImage) {
const result = await processImageFile({
req,
file,
metadata: { file_id: v4() },
returnFile: true,
});
filepath = result.filepath;
storageMetadata = getStorageMetadata({
filepath,
source: result.source,
storageKey: result.storageKey,
storageRegion: result.storageRegion,
});
}
const retentionExpiry = await getAgentFileRetentionExpiry({
req,
messageAttachment,
tool_resource,
});
const fileInfo = {
...removeNullishValues({
user: req.user.id,
file_id,
temp_file_id,
bytes,
filepath,
...storageMetadata,
filename: filename ?? sanitizeFilename(file.originalname),
context: messageAttachment ? FileContext.message_attachment : FileContext.agents,
model: messageAttachment ? undefined : req.body.model,
metadata: fileInfoMetadata,
type: file.mimetype,
embedded,
source,
height,
width,
tenantId: req.user.tenantId,
}),
...retentionExpiry,
};
const result = await db.createFile(fileInfo, true);
sendUploadSuccess(res, sseStream, 'Agent file uploaded and processed successfully', result);
};
/**
* @param {object} params - The params object.
* @param {OpenAI} params.openai - The OpenAI client instance.
* @param {string} params.file_id - The ID of the file to retrieve.
* @param {string} params.userId - The user ID.
* @param {string} [params.filename] - The name of the file. `undefined` for `file_citation` annotations.
* @param {boolean} [params.saveFile=false] - Whether to save the file metadata to the database.
* @param {boolean} [params.updateUsage=false] - Whether to update file usage in database.
*/
const processOpenAIFile = async ({
openai,
file_id,
userId,
filename,
saveFile = false,
updateUsage = false,
}) => {
const _file = await openai.files.retrieve(file_id);
const originalName = filename ?? (_file.filename ? path.basename(_file.filename) : undefined);
const filepath = `${openai.baseURL}/files/${userId}/${file_id}${
originalName ? `/${originalName}` : ''
}`;
const type = mime.getType(originalName ?? file_id);
const source =
openai.req.body.endpoint === EModelEndpoint.azureAssistants
? FileSources.azure
: FileSources.openai;
const file = {
..._file,
type,
file_id,
filepath,
usage: 1,
user: userId,
context: _file.purpose,
source,
model: openai.req.body.model,
filename: originalName ?? file_id,
...(await getRetentionExpiry(openai.req)),
tenantId: openai.req?.user?.tenantId,
};
if (saveFile) {
await db.createFile(file, true);
} else if (updateUsage) {
try {
await db.updateFileUsage({
file_id,
user: userId,
tenantId: openai.req?.user?.tenantId,
});
} catch (error) {
logger.error('Error updating file usage', error);
}
}
return file;
};
/**
* Process OpenAI image files, convert to target format, save and return file metadata.
* @param {object} params - The params object.
* @param {ServerRequest} params.req - The Express request object.
* @param {Buffer} params.buffer - The image buffer.
* @param {string} params.file_id - The file ID.
* @param {string} params.filename - The filename.
* @param {string} params.fileExt - The file extension.
* @returns {Promise<MongoFile>} The file metadata.
*/
const processOpenAIImageOutput = async ({ req, buffer, file_id, filename, fileExt }) => {
const currentDate = new Date();
const formattedDate = currentDate.toISOString();
const appConfig = req.config;
const _file = await convertImage(req, buffer, undefined, `${file_id}${fileExt}`);
// Create only one file record with the correct information
const file = {
..._file,
usage: 1,
user: req.user.id,
type: mime.getType(fileExt),
createdAt: formattedDate,
updatedAt: formattedDate,
source: getFileStrategy(appConfig, { isImage: true }),
context: FileContext.assistants_output,
file_id,
filename,
...(await getRetentionExpiry(req)),
tenantId: req.user.tenantId,
};
try {
await db.createFile(file, true);
} catch (error) {
logger.warn('Error saving OpenAI image output file metadata', error);
}
return file;
};
/**
* Retrieves and processes an OpenAI file based on its type.
*
* @param {Object} params - The params passed to the function.
* @param {OpenAIClient} params.openai - The OpenAI client instance.
* @param {RunClient} params.client - The LibreChat client instance: either refers to `openai` or `streamRunManager`.
* @param {string} params.file_id - The ID of the file to retrieve.
* @param {string} [params.basename] - The basename of the file (if image); e.g., 'image.jpg'. `undefined` for `file_citation` annotations.
* @param {boolean} [params.unknownType] - Whether the file type is unknown.
* @returns {Promise<{file_id: string, filepath: string, source: string, bytes?: number, width?: number, height?: number} | null>}
* - Returns null if `file_id` is not defined; else, the file metadata if successfully retrieved and processed.
*/
async function retrieveAndProcessFile({
openai,
client,
file_id,
basename: _basename,
unknownType,
}) {
if (!file_id) {
return null;
}
let basename = _basename;
const processArgs = { openai, file_id, filename: basename, userId: client.req.user.id };
// If no basename provided, return only the file metadata
if (!basename) {
return await processOpenAIFile({ ...processArgs, saveFile: true });
}
const fileExt = path.extname(basename);
if (client.attachedFileIds?.has(file_id) || client.processedFileIds?.has(file_id)) {
return processOpenAIFile({ ...processArgs, updateUsage: true });
}
/**
* @returns {Promise<Buffer>} The file data buffer.
*/
const getDataBuffer = async () => {
const response = await openai.files.content(file_id);
const arrayBuffer = await response.arrayBuffer();
return Buffer.from(arrayBuffer);
};
let dataBuffer;
if (unknownType || !fileExt || imageExtRegex.test(basename)) {
try {
dataBuffer = await getDataBuffer();
} catch (error) {
logger.error('Error downloading file from OpenAI:', error);
dataBuffer = null;
}
}
if (!dataBuffer) {
return await processOpenAIFile({ ...processArgs, saveFile: true });
}
// If the filetype is unknown, inspect the file
if (dataBuffer && (unknownType || !fileExt)) {
const detectedExt = await determineFileType(dataBuffer);
const isImageOutput = detectedExt && imageExtRegex.test('.' + detectedExt);
if (!isImageOutput) {
return await processOpenAIFile({ ...processArgs, saveFile: true });
}
return await processOpenAIImageOutput({
file_id,
req: client.req,
buffer: dataBuffer,
filename: basename,
fileExt: detectedExt,
});
} else if (dataBuffer && imageExtRegex.test(basename)) {
return await processOpenAIImageOutput({
file_id,
req: client.req,
buffer: dataBuffer,
filename: basename,
fileExt,
});
} else {
logger.debug(`[retrieveAndProcessFile] Non-image file type detected: ${basename}`);
return await processOpenAIFile({ ...processArgs, saveFile: true });
}
}
/**
* Converts a base64 string to a buffer.
* @param {string} base64String
* @returns {Buffer<ArrayBufferLike>}
*/
function base64ToBuffer(base64String) {
try {
const typeMatch = base64String.match(/^data:([A-Za-z-+/]+);base64,/);
const type = typeMatch ? typeMatch[1] : '';
const base64Data = base64String.replace(/^data:([A-Za-z-+/]+);base64,/, '');
if (!base64Data) {
throw new Error('Invalid base64 string');
}
return {
buffer: Buffer.from(base64Data, 'base64'),
type,
};
} catch (error) {
throw new Error(`Failed to convert base64 to buffer: ${error.message}`);
}
}
async function saveBase64Image(
url,
{ req, file_id: _file_id, filename: _filename, endpoint, context, resolution },
) {
const appConfig = req.config;
const effectiveResolution = resolution ?? appConfig.fileConfig?.imageGeneration ?? 'high';
const file_id = _file_id ?? v4();
let filename = `${file_id}-${_filename}`;
const { buffer: inputBuffer, type: declaredType } = base64ToBuffer(url);
const image = await resizeImageBuffer(inputBuffer, effectiveResolution, endpoint);
/** Sharp re-encodes what it resizes, so the bytes being saved are not necessarily in the
* format the data URL declared — an SVG arrives here and is rasterized to PNG. The record has
* to describe the bytes, because `file.type` is handed to providers verbatim as `media_type`
* (Anthropic), `inlineData.mimeType` (Google), and the `data:` prefix (OpenAI). */
const type = image.type ?? declaredType;
if (!path.extname(_filename)) {
const extension = mime.getExtension(type);
if (extension) {
filename += `.${extension}`;
} else {
throw new Error(`Could not determine file extension from MIME type: ${type}`);
}
}
const source = getFileStrategy(appConfig, { isImage: true });
const { saveBuffer } = getStrategyFunctions(source);
const filepath = await saveBuffer({
userId: req.user.id,
fileName: filename,
buffer: image.buffer,
tenantId: req.user.tenantId,
});
const storageMetadata = getStorageMetadata({ filepath, source });
return await db.createFile(
{
type,
source,
context,
file_id,
filepath,
...storageMetadata,
filename,
user: req.user.id,
bytes: image.bytes,
width: image.width,
...(await getRetentionExpiry(req)),
height: image.height,
tenantId: req.user.tenantId,
},
true,
);
}
/**
* Filters a file based on its size and the endpoint origin.
*
* @param {Object} params - The parameters for the function.
* @param {ServerRequest} params.req - The request object from Express.
* @param {string} [params.req.endpoint]
* @param {string} [params.req.file_id]
* @param {number} [params.req.width]
* @param {number} [params.req.height]
* @param {number} [params.req.version]
* @param {boolean} [params.image] - Whether the file expected is an image.
* @param {boolean} [params.isAvatar] - Whether the file expected is a user or entity avatar.
* @returns {void}
*
* @throws {Error} If a file exception is caught (invalid file size or type, lack of metadata).
*/
function filterFile({ req, image, isAvatar }) {
const { file } = req;
const { endpoint, endpointType, file_id, width, height } = req.body;
if (!file_id && !isAvatar) {
throw new Error('No file_id provided');
}
if (file.size === 0) {
throw new Error('Empty file uploaded');
}
/* parse to validate api call, throws error on fail */
if (!isAvatar) {
isUUID.parse(file_id);
}
if (!endpoint && !isAvatar) {
throw new Error('No endpoint provided');
}
const appConfig = req.config;
const fileConfig = mergeFileConfig(appConfig.fileConfig);
const endpointFileConfig = getEndpointFileConfig({
endpoint,
fileConfig,
endpointType,
});
const fileSizeLimit =
isAvatar === true ? fileConfig.avatarSizeLimit : endpointFileConfig.fileSizeLimit;
if (file.size > fileSizeLimit) {
throw new Error(
`File size limit of ${fileSizeLimit / megabyte} MB exceeded for ${
isAvatar ? 'avatar upload' : `${endpoint} endpoint`
}`,
);
}
const isSupportedMimeType = fileConfig.checkType(
file.mimetype,
endpointFileConfig.supportedMimeTypes,
);
if (!isSupportedMimeType) {
throw new Error('Unsupported file type');
}
if (!image || isAvatar === true) {
return;
}
if (!width) {
throw new Error('No width provided');
}
if (!height) {
throw new Error('No height provided');
}
}
module.exports = {
filterFile,
processFileURL,
saveBase64Image,
processImageFile,
uploadImageBuffer,
sweepExpiredFiles,
startExpiredFileSweep,
processFileUpload,
processDeleteRequest,
processAgentFileUpload,
retrieveAndProcessFile,
};