LibreChat/api/server/services/Endpoints/agents/skillDeps.js
Danny Avila 2ffee7e143
🪂 feat: Opt-In Text Fallback for Files No Tool Can Read (#15940)
* 🪂 feat: Opt-In Text Fallback for Files No Tool Can Read

An upload routed to tools (`llmDeliveryPath: none`) reaches nothing when the
agent handling the turn runs neither Run Code nor File Search, or runs one that
cannot read the type. With `fileConfig.endpoints.<endpoint>.textFallbackWithoutTools`
(also accepted at the top level and inherited from `default`), such a turn
delivers the text extracted at upload instead. The setting is off by default,
which keeps today's behavior.

- Upload: when the setting is on and an inferred `none` route finds no reading
  tool, the built-in document parser or native text reader stores the text on
  the file, best effort, under the extracted-text size cap and content policy.
- Turn: `resolveTurnLLMDeliveryPath` re-resolves each inferred route against
  the tools the agent runs, and `applyTurnTextFallback` marks copies of the
  records a turn loads, so endpoint filtering, limits, inspection, history
  replay and steer media all see the text the turn delivers. The stored route
  is never rewritten.
- Explicit destinations, records predating routing, and turns whose tools are
  unknown are left as they are.

* 🧪 test: Resolve Custom Endpoint Fallback Under the Endpoint Name

Uploads route a custom endpoint agent by its saved provider, the endpoint name. initializeAgent marks turn copies before getProviderConfig swaps the provider for the backing client, and the endpoint keeps the name afterward, so the client and child encoders resolve the opt-in under the same endpoint. Cover both the initialization path and the post-initialization agent shape.

* ♻️ refactor: Share One Extracted-Text Storage Cap Across Uploads

The fallback defined its own copy of the 15 MiB cap context uploads already apply to extracted text. Export MAX_STORED_EXTRACTED_TEXT_BYTES from files/extract and use it in both places, documenting that it bounds what a MongoDB file document can hold rather than operator policy.

* 🧭 fix: Keep Fallback Text Wherever a Later Turn Can Deliver It

Extraction skipped uploads filed under a reading tool, so a handoff agent
without that tool, or the same agent after its tools or grants changed, had
no text to fall back to. Store fallback text for every inferred route on a
message attachment; files kept on an agent's tool resources never reach a
prompt and still skip it.

Turn marking only ran while the fallback was enabled, so a stored tool-routed
record whose type the endpoint now routes to text stayed marked `none` and
`extractFileContext` skipped it. applyTurnTextDelivery (renamed from
applyTurnTextFallback) now marks any stored `none` record with text that the
turn resolves to text, and returns early when no such record is loaded.

* 🚦 fix: Admit Every Attachment by the Route Its Turn Delivers

A record upload routed to tools that the current endpoint sends to the
provider stayed marked `none` on the turn copy, so admission skipped it while
BaseClient encoded its bytes, bypassing attachment count and size limits.
applyTurnDelivery (renamed from applyTurnTextDelivery) now gives every record
whose route upload inferred the route the turn resolves, before endpoint
filtering, limits and inspection, matching the run-file encoder.
hasInferredLLMDeliveryPath shares the inferred-route rule with the resolver.

Fallback extraction selected the document parser inside the CJS upload path.
resolveUploadFallbackText now takes the upload and picks parseDocument or
parseTextNative itself, with the extractors injectable, so process.js passes
request data only. Its tests run the real parser against the xlsx fixture and
the real native reader against temporary files.

* 🎚️ fix: Read Custom Endpoint Dialect From Config on Both Sides of Init

initializeAgent materializes turn routes before getProviderConfig swaps the
saved custom endpoint name in agent.provider for its backing client, so
routing read that name as a non-OpenAI dialect. Media a custom endpoint opted
into could then be materialized off the provider path, skipping admission,
while the finalized client still sent the bytes. resolveAgentDeliveryRouting
now reads the dialect with getCustomEndpointProvider, as upload does, which
holds before and after the swap.

The upload fallback plan now keys on the chosen-destination marker the turn
resolver reads, so an upload recorded as chosen, including every legacy
chooser upload, never pays for text a turn cannot deliver.

* 🛡️ fix: Apply Turn Routes Only While the Record Stays Admitted

Resolution while initializing and at delivery can disagree: delivery reads a
Responses API choice only the finished client config holds. Applying a route
that takes a record out of admission, off the model path or to text it never
stored, would then skip the limits for a file the client still sends.
applyTurnDelivery now applies a route only while the record stays model-bound;
admitting a record the turn leaves out cannot slip past a limit.

* 🕹️ feat: Run Declared Project Actions on Attached Workers (#15943)

* feat: use declared attached project actions

* Complete named action definitions and approval previews

* Sort named environment imports

* Sort native environment fixture imports

* 🔓 fix: Keep Code Approval Mode Selectable Mid-Run (#15938)

* 🔓 fix: Keep Code Approval Mode Selectable Mid-Run

The composer's code approval mode selector was disabled while a run streamed, and a mode picked during a run was reverted when the final or abort event merged the server conversation back. The selector now stays usable in flight and both merges retain a mid-run selection, which the next send carries.

* 🔓 fix: Retain Mid-Run Approval Mode Across Cache and Recovery

Stamp the live approval mode onto the submission conversation at send so the retention baseline matches what was read, keep a retained mode in the conversation query cache on final and cancel, and rebuild failed or aborted conversations from a preset that carries the mid-run pick.

* 🔓 fix: Scope Retained Approval Mode to the Submitted Conversation

Retention now requires the live conversation to be the one the run submitted, or the id the server assigned to a new chat, so navigating elsewhere mid-run cannot write that conversation's mode back. A single recovery helper rebuilds failed or aborted conversations from the retained preset and patches the detail cache for every recovery site.

* 🔓 refactor: Keep the Local Approval Mode on Every Server Merge

Replace per-path retention with one rule: a locally picked code approval mode is newer than any server copy of the same conversation, so the final and abort merges and the error recovery preset keep it. The selector writes the pick into the conversation's detail cache, which navigation rebuilds from, so the pick is conversation-scoped without reading the index-global atom from the SSE handlers.

* 🔓 fix: Prefer the Live Conversation When Caching the Final Merge

The final handler's detail-cache merge now takes the local approval mode from the open conversation when it is the one that finished, falling back to the cached record only for a conversation that is no longer on screen.

* 🔓 fix: Keep the Local Approval Mode Through Settled-Start Reconciliation

The resumable transport's settled-start reconciliation and replacement handoff wrote a fetched conversation straight into conversation state and the detail cache. Both now apply the same local-mode rule as the event handlers, treating a new chat's pending id as the settled conversation's own.

* 🔓 fix: Seed the Detail Record When a Mode Is Picked

A conversation that has its id but no detail record yet now gets one from the live conversation when a mode is picked, the same key the resumable transport seeds optimistically, so recovery and navigation find the pick. A chat with no id yet still keeps the pick in conversation state alone.

* ⬆️ chore: bump agents to 3.8.7 (#15947)

* 🧭 refactor: Settle Turn Delivery Routing Once in Agent Initialization

`initializeAgent` resolved the provider and its client options only after the
turn's files were loaded and admitted, and the two delivery readers each rebuilt
the attachment routing from the agent object at their own moment: `BaseClient`
took the custom-endpoint dialect from the already-swapped `agent.provider`, the
run-file encoder from whatever child config it was handed.

Move `getProviderConfig`/`getOptions` ahead of file discovery, where nothing in
between fed them, and settle one `deliveryRouting` value with every input final:
the file policy under the endpoint's own name, the dialect its config declares,
the Responses API decision the model call uses, and the transcription setting.
`InitializedAgent`, the child encoder and `BaseClient` consume that value;
`resolveTurnLLMDeliveryPath` is the one place a stored route is resolved again.

* fix: Preserve attachment reachability across handoffs and late steers

* test: Type the untrusted attachment reference fixture

---------

Co-authored-by: Ravi Kumar L <upman@users.noreply.github.com>
2026-09-14 21:02:54 -04:00

438 lines
14 KiB
JavaScript

const crypto = require('crypto');
const { getStrategyFunctions } = require('~/server/services/Files/strategies');
const { batchUploadCodeEnvFiles } = require('~/server/services/Files/Code/crud');
const {
getSessionInfo,
checkIfActive,
readWorkspaceFile,
searchWorkspace,
listWorkspaceFiles,
writeWorkspaceFile,
previewWorkspaceEdit,
editWorkspaceFile,
readSandboxFile,
readSandboxImage,
writeSandboxFile,
} = require('~/server/services/Files/Code/process');
const {
checkAccess,
isMemoryEnabled,
getStorageMetadata,
resolveRequestTenantId,
enrichWithSkillConfigurable,
mergeDeploymentSkillIds,
createDeploymentSkillMethods,
isDeploymentSkillFileSource,
getDeploymentSkillDownloadStream,
} = require('@librechat/api');
const {
Permissions,
FileContext,
ResourceType,
PermissionBits,
AccessRoleIds,
PrincipalType,
PermissionTypes,
AgentCapabilities,
isEphemeralAgentId,
} = require('librechat-data-provider');
const { checkPermission, grantPermission } = require('~/server/services/PermissionService');
const { getFileStrategy } = require('~/server/utils/getFileStrategy');
const db = require('~/models');
const deploymentSkillMethods = createDeploymentSkillMethods({
getSkillById: db.getSkillById,
getSkillByName: db.getSkillByName,
listSkillsByAccess: db.listSkillsByAccess,
listAlwaysApplySkills: db.listAlwaysApplySkills,
listSkillFiles: db.listSkillFiles,
getSkillFileByPath: db.getSkillFileByPath,
updateSkillFileContent: db.updateSkillFileContent,
updateSkillFileCodeEnvIds: db.updateSkillFileCodeEnvIds,
});
function getSkillDbMethods() {
return deploymentSkillMethods;
}
function withDeploymentSkillIds(ids = []) {
return mergeDeploymentSkillIds(ids);
}
function getSkillStrategyFunctions(source) {
if (isDeploymentSkillFileSource(source)) {
return {
getDownloadStream: (_req, filepath) => getDeploymentSkillDownloadStream(filepath),
};
}
return getStrategyFunctions(source);
}
function resolveSkillStorage(req, { isImage = false } = {}) {
const source = getFileStrategy(req.config, { context: FileContext.skill_file, isImage });
const strategy = getStrategyFunctions(source);
if (!strategy.saveBuffer) {
throw new Error(`Storage backend "${source}" does not support file writes`);
}
return { saveBuffer: strategy.saveBuffer, source };
}
function basename(relativePath) {
const slash = relativePath.lastIndexOf('/');
return slash === -1 ? relativePath : relativePath.slice(slash + 1);
}
async function saveSkillFileContent({ req, skillId, relativePath, content, mimeType }) {
const existingFile = await db.getSkillFileByPath(skillId, relativePath);
const tenantId = resolveRequestTenantId(req);
const fileId = crypto.randomUUID();
const filename = basename(relativePath);
const storageFileName = `${fileId}__${filename}`;
const buffer = Buffer.from(content, 'utf8');
const storage = resolveSkillStorage(req, { isImage: mimeType.startsWith('image/') });
const filepath = await storage.saveBuffer({
userId: req.user.id,
buffer,
fileName: storageFileName,
basePath: 'uploads',
tenantId,
});
const storageMetadata = getStorageMetadata({ filepath, source: storage.source });
let result;
try {
result = await db.upsertSkillFile({
skillId,
relativePath,
file_id: fileId,
filename,
filepath,
...storageMetadata,
source: storage.source,
mimeType,
bytes: buffer.length,
isExecutable: false,
author: req.user._id ?? req.user.id,
tenantId,
});
if (!result) {
const error = new Error('Skill file save failed to persist metadata');
error.code = 'SKILL_FILE_UPSERT_NOT_FOUND';
throw error;
}
} catch (error) {
const { deleteFile } = getStrategyFunctions(storage.source);
if (deleteFile) {
await deleteFile(req, { filepath, user: req.user.id, tenantId }).catch(() => undefined);
}
throw error;
}
if (existingFile && existingFile.filepath !== filepath) {
const { deleteFile } = getStrategyFunctions(existingFile.source);
if (deleteFile) {
deleteFile(req, {
filepath: existingFile.filepath,
storageKey: existingFile.storageKey,
storageRegion: existingFile.storageRegion,
user: existingFile.author ?? req.user.id,
tenantId: existingFile.tenantId ?? tenantId,
}).catch(() => undefined);
}
}
return { bytes: result.bytes, relativePath: result.relativePath };
}
function canCreateSkill({ req }) {
return checkAccess({
req,
user: req.user,
permissionType: PermissionTypes.SKILLS,
permissions: [Permissions.USE, Permissions.CREATE],
getRoleByName: db.getRoleByName,
});
}
function canEditSkill({ req, skillId }) {
return checkPermission({
userId: req.user.id,
role: req.user.role,
resourceType: ResourceType.SKILL,
resourceId: skillId,
requiredPermission: PermissionBits.EDIT,
});
}
function isAgentSkillAuthoringEnabledForRun({
agent,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}) {
if (!skillsCapabilityEnabled) {
return false;
}
if (isEphemeralAgentId(agent.id)) {
if (agent.skills_enabled === false) {
return false;
}
if (agent.skills_enabled === true) {
return true;
}
return ephemeralSkillsToggle === true;
}
return agent.skills_enabled === true || agent.skill_authoring_enabled === true;
}
function canAuthorSkillFiles({
agent,
scopedEditableSkillIds = [],
skillCreateAllowed,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}) {
return (
isAgentSkillAuthoringEnabledForRun({
agent,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}) &&
(scopedEditableSkillIds.length > 0 || skillCreateAllowed === true)
);
}
function grantSkillOwner({ req, skillId }) {
return grantPermission({
principalType: PrincipalType.USER,
principalId: req.user.id,
resourceType: ResourceType.SKILL,
resourceId: skillId,
accessRoleId: AccessRoleIds.SKILL_OWNER,
grantedBy: req.user.id,
});
}
function getAuthorSkillByName({ req, name }) {
const author = req.user?._id ?? req.user?.id;
if (!author) {
return null;
}
return db.getAuthorSkillByName({
name,
author,
tenantId: resolveRequestTenantId(req),
});
}
/**
* Builds the `skillPrimedIdsByName` map threaded through
* `buildAgentToolContext`. Centralized here so every runtime route shares
* one source of truth — if `ResolvedManualSkill` ever renames `_id` or
* gains new identifying fields, only this helper changes.
*
* Combines both manual (`$`-popover) primes AND always-apply primes so
* `read_file` can:
* - Relax the `disable-model-invocation: true` gate for either source
* (the body is already in context; blocking its own files would be
* nonsensical).
* - Pin same-name collision lookups to the exact `_id` the resolver
* primed (otherwise a newer same-name duplicate could shadow the
* body/file pair within a single turn).
*
* On the rare overlap (a name appears in both arrays because upstream
* dedup was skipped), manual wins — manual invocation is explicit user
* intent and carries the authoritative `_id` for this turn.
*
* Returns `undefined` (not `{}`) when both arrays are empty, so the
* downstream `enrichWithSkillConfigurable` cleanly omits the field from
* `mergedConfigurable` rather than threading an empty object.
*
* @param {Array<{ name: string, _id: { toString(): string } }> | undefined} manualSkillPrimes
* @param {Array<{ name: string, _id: { toString(): string } }> | undefined} alwaysApplySkillPrimes
* @returns {Record<string, string> | undefined}
*/
function buildSkillPrimedIdsByName(manualSkillPrimes, alwaysApplySkillPrimes) {
const manualCount = manualSkillPrimes?.length ?? 0;
const alwaysApplyCount = alwaysApplySkillPrimes?.length ?? 0;
if (manualCount === 0 && alwaysApplyCount === 0) {
return undefined;
}
const out = {};
/* Order matters on the edge case where the same name appears in both
lists: always-apply goes in first, then manual overwrites — manual
wins because it's explicit user intent for this turn. */
if (alwaysApplyCount > 0) {
for (const p of alwaysApplySkillPrimes) {
out[p.name] = p._id.toString();
}
}
if (manualCount > 0) {
for (const p of manualSkillPrimes) {
out[p.name] = p._id.toString();
}
}
return out;
}
/**
* Builds the per-agent context consumed by ON_TOOL_EXECUTE. Keeping this
* shape in one Adapter gives every runtime path the same configurable
* fields and the same primed-skill pinning behavior.
*
* @param {object} params
* @param {object} params.agent
* @param {object} params.config
* @param {Record<string, import('@librechat/api').LCAvailableTools>} [params.config.mcpAvailableTools]
* @param {import('@librechat/api').RequestScopedMCPConnectionStore} [params.config.requestScopedConnections]
* @returns {object}
*/
function buildAgentToolContext({ agent, config }) {
return {
agent,
fileEncodingAgent: {
provider: config.provider,
model_parameters: config.model_parameters,
imageDetail: config.imageDetail,
agentContextAttachments: config.agentContextAttachments,
fileConsumers: config.fileConsumers,
deliveryRouting: config.deliveryRouting,
},
/** Per-agent resolved endpoint token/pricing config. Retained here because
* `agentToolContexts` is the one map that holds every agent — including
* pure subagents pruned from `agentConfigs` — so usage can be priced with
* the producing agent's config in multi-endpoint graphs. */
endpointTokenConfig: config.endpointTokenConfig,
toolRegistry: config.toolRegistry,
backgroundToolNames: config.backgroundToolNames,
intentToolNames: config.intentToolNames,
mcpAvailableTools: config.mcpAvailableTools,
requestScopedConnections: config.requestScopedConnections,
userMCPAuthMap: config.userMCPAuthMap,
tool_resources: config.tool_resources,
actionsEnabled: config.actionsEnabled,
accessibleMcpServerNames: config.accessibleMcpServerNames,
accessibleSkillIds: config.accessibleSkillIds,
activeSkillNames: config.activeSkillNames,
codeEnvAvailable: config.codeEnvAvailable,
codeExecutionContext: config.codeExecutionContext,
skillAuthoringAvailable: config.skillAuthoringAvailable,
fileAuthoringToolNames: config.fileAuthoringToolNames,
skillPrimedIdsByName:
buildSkillPrimedIdsByName(config.manualSkillPrimes, config.alwaysApplySkillPrimes) ?? {},
provisionState: config.provisionState,
};
}
/** Resolves the full run-level gate used to expose inline memory tools. */
function resolveMemoryAvailability({ enabledCapabilities, memoryConfig, user, getRoleByName }) {
if (
!enabledCapabilities.has(AgentCapabilities.memory) ||
!isMemoryEnabled(memoryConfig) ||
user?.personalization?.memories === false
) {
return false;
}
return checkAccess({
user,
permissionType: PermissionTypes.MEMORIES,
permissions: [Permissions.USE, Permissions.CREATE, Permissions.UPDATE],
getRoleByName,
});
}
function hasOwn(value, key) {
return Object.prototype.hasOwnProperty.call(value ?? {}, key);
}
/**
* Applies per-agent runtime context to a loadToolsForExecution result.
*
* @param {object} params
* @param {{ loadedTools: unknown[], configurable?: Record<string, unknown> }} params.result
* @param {object} params.req
* @param {object | undefined} params.ctx
* @param {object | undefined} [params.fallback]
* @returns {{ loadedTools: unknown[], configurable: Record<string, unknown> }}
*/
function enrichLoadedToolsWithAgentContext({ result, req, ctx = {}, fallback = {} }) {
const codeEnvAvailable = hasOwn(ctx, 'codeEnvAvailable')
? ctx.codeEnvAvailable === true
: fallback.codeEnvAvailable === true;
const skillAuthoringAvailable = hasOwn(ctx, 'skillAuthoringAvailable')
? ctx.skillAuthoringAvailable === true
: fallback.skillAuthoringAvailable === true;
return enrichWithSkillConfigurable({
result,
context: {
req,
codeEnvAvailable,
accessibleSkillIds: ctx.accessibleSkillIds ?? fallback.accessibleSkillIds,
skillPrimedIdsByName: ctx.skillPrimedIdsByName ?? fallback.skillPrimedIdsByName,
activeSkillNames: ctx.activeSkillNames ?? fallback.activeSkillNames,
skillAuthoringAvailable,
fileAuthoringToolNames: ctx.fileAuthoringToolNames ?? fallback.fileAuthoringToolNames,
},
});
}
/** Skill-related properties for ToolExecuteOptions (stable references, allocated once). */
const skillToolDeps = {
getSkillByName: deploymentSkillMethods.getSkillByName,
getAuthorSkillByName,
createSkill: db.createSkill,
updateSkill: db.updateSkill,
deleteSkill: db.deleteSkill,
canCreateSkill,
canEditSkill,
grantSkillOwner,
saveSkillFileContent,
listSkillFiles: deploymentSkillMethods.listSkillFiles,
getStrategyFunctions: getSkillStrategyFunctions,
batchUploadCodeEnvFiles,
getSessionInfo,
checkIfActive,
updateSkillFileCodeEnvIds: deploymentSkillMethods.updateSkillFileCodeEnvIds,
getSkillFileByPath: deploymentSkillMethods.getSkillFileByPath,
updateSkillFileContent: deploymentSkillMethods.updateSkillFileContent,
readWorkspaceFile,
searchWorkspace,
listWorkspaceFiles,
writeWorkspaceFile,
previewWorkspaceEdit,
editWorkspaceFile,
/**
* `read_file` falls back to a sandbox `cat` for `/mnt/data/...` paths
* and for `{firstSegment}/...` paths whose first segment isn't a known
* skill name. The handler routes through this when the agent has code
* execution enabled; the codeapi base URL comes from
* `LIBRECHAT_CODE_BASEURL` and the sandbox session id is forwarded by
* the agents-side `ToolNode` via `tc.codeSessionContext`.
*/
readSandboxFile,
/**
* Companion to `readSandboxFile` for the raster-image case: pulls the
* bytes base64-encoded (size-guarded in-sandbox) so `read_file` can
* return an image the model can see instead of refusing it as binary.
*/
readSandboxImage,
writeSandboxFile,
};
function getSkillToolDeps() {
return skillToolDeps;
}
module.exports = {
getSkillToolDeps,
canAuthorSkillFiles,
isAgentSkillAuthoringEnabledForRun,
getSkillDbMethods,
withDeploymentSkillIds,
getSkillStrategyFunctions,
enrichWithSkillConfigurable,
buildSkillPrimedIdsByName,
buildAgentToolContext,
resolveMemoryAvailability,
enrichLoadedToolsWithAgentContext,
};