LibreChat/api/server/services/Endpoints/agents/initialize.js
Danny Avila 259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00

1423 lines
52 KiB
JavaScript

const { logger } = require('@librechat/data-schemas');
const { createContentAggregator, GraphNodeKeys } = require('@librechat/agents');
const {
resolveSender,
createConcurrencyLimiter,
loadSkillStates,
initializeAgent,
primeInvokedSkillsForProfiles,
validateAgentModel,
extractManualSkills,
GenerationJobManager,
getCustomEndpointConfig,
getProviderConfig,
discoverConnectedAgents,
resolveAgentTokenConfig,
resolveAgentScopedSkillIds,
resolveModelSpecSkillIds,
getAgentStartupTelemetry,
buildAgentContextAttachmentsByAgentId,
collectCodeExecutionProfileRoutes,
getLazySubagentConfigId,
createStatefulCodeEnvironmentPolicyError,
buildSubagentThreadTaskConfig,
} = require('@librechat/api');
const {
ResourceType,
EModelEndpoint,
PermissionBits,
MAX_SUBAGENT_DEPTH,
isAgentsEndpoint,
AgentCapabilities,
Tools,
MAX_SUBAGENT_GRAPH_NODES,
MAX_SUBAGENT_RUN_CONFIGS,
isEphemeralAgentId,
resolveAllowedStatefulCodeEnvironments,
} = require('librechat-data-provider');
const {
createToolEndCallback,
createAttachmentEmitter,
createBackgroundCodeResultHandler,
getDefaultHandlers,
} = require('~/server/controllers/agents/callbacks');
const {
loadAgentTools,
loadToolsForExecution,
getAccessibleMcpServerNames,
isFatalAgentInitializationError,
} = require('~/server/services/ToolService');
const { filterFilesByAgentAccess } = require('~/server/services/Files/permissions');
const {
getSkillToolDeps,
getSkillDbMethods,
canAuthorSkillFiles,
withDeploymentSkillIds,
buildAgentToolContext,
resolveMemoryAvailability,
enrichLoadedToolsWithAgentContext,
} = require('./skillDeps');
const { getModelsConfig } = require('~/server/controllers/ModelController');
const { checkPermission, findAccessibleResources } = require('~/server/services/PermissionService');
const AgentClient = require('~/server/controllers/agents/client');
const { processAddedConvo } = require('./addedConvo');
const subagentThreadTaskStore = require('./subagentThreadStore');
const { logViolation } = require('~/cache');
const db = require('~/models');
const SUBAGENT_GRAPH_LOAD_CONCURRENCY = 4;
/**
* Creates a tool loader function for the agent.
* @param {AbortSignal} signal - The abort signal
* @param {string | null} [streamId] - The stream ID for resumable mode
* @param {boolean} [definitionsOnly=false] - When true, returns only serializable
* tool definitions without creating full tool instances (for event-driven mode)
* @param {number} [jobCreatedAt] - The generation epoch that owns emitted tool events
*/
function createToolLoader(signal, streamId = null, definitionsOnly = false, jobCreatedAt) {
/**
* @param {object} params
* @param {ServerRequest} params.req
* @param {ServerResponse} params.res
* @param {string} params.agentId
* @param {string[]} params.tools
* @param {string} params.provider
* @param {string} params.model
* @param {AgentToolResources} params.tool_resources
* @returns {Promise<{
* tools?: StructuredTool[],
* toolContextMap: Record<string, unknown>,
* toolDefinitions?: import('@librechat/agents').LCTool[],
* userMCPAuthMap?: Record<string, Record<string, string>>,
* toolRegistry?: import('@librechat/agents').LCToolRegistry
* } | undefined>}
*/
return async function loadTools({
req,
res,
tools,
model,
agentId,
provider,
tool_options,
tool_resources,
codeExecutionContext,
accessibleMcpServerNames,
}) {
const agent = { id: agentId, tools, provider, model, tool_options };
try {
return await loadAgentTools({
req,
res,
agent,
signal,
streamId,
jobCreatedAt,
tool_resources,
codeExecutionContext,
definitionsOnly,
accessibleMcpServerNames,
});
} catch (error) {
if (isFatalAgentInitializationError(error)) {
throw error;
}
logger.error('Error loading tools for agent ' + agentId, error);
}
};
}
/**
* Initializes the AgentClient for a given request/response cycle.
* @param {Object} params
* @param {Express.Request} params.req
* @param {Express.Response} params.res
* @param {AbortSignal} params.signal
* @param {Object} params.endpointOption
* @param {number} [params.jobCreatedAt]
* @param {string} [params.checkpointNamespace] Immutable saver-level generation scope
*/
const initializeClient = async ({
req,
res,
signal,
endpointOption,
jobCreatedAt,
checkpointNamespace,
}) => {
if (!endpointOption) {
throw new Error('Endpoint option not provided');
}
const appConfig = req.config;
/** The normal controller resolves this once for timestamp anchoring. Reuse
* that trusted document for child-thread execution policy; resume and direct
* callers fall back to the same owner-scoped lookup. */
const conversationId = req.body?.conversationId;
let requestConversationPromise = Promise.resolve(null);
if (Object.prototype.hasOwnProperty.call(req, 'resolvedConversation')) {
requestConversationPromise = Promise.resolve(req.resolvedConversation);
} else if (typeof conversationId === 'string' && conversationId !== '') {
requestConversationPromise = db.getConvo(req.user.id, conversationId);
}
const startupTelemetry = getAgentStartupTelemetry(req);
/** @type {string | null} */
const streamId = req._resumableStreamId || null;
/** @type {Array<UsageMetadata>} */
const collectedUsage = [];
/**
* Vertex Gemini 3 thought signatures captured from `chat_model_end` events,
* keyed by `tool_call_id`. Persisted on
* `responseMessage.metadata.thoughtSignatures` so subsequent conversation
* turns can restore each signature onto the right reconstructed AIMessage's
* `additional_kwargs.signatures` and avoid 400s when resuming after a tool
* round-trip without a final text reply. Always allocated; capture path
* is a no-op for providers that don't emit signatures (OpenAI, Anthropic,
* Bedrock, etc.).
* @type {Record<string, string>}
*/
const collectedThoughtSignatures = {};
/** @type {ArtifactPromises} */
const artifactPromises = [];
/** @type {Map<string, import('@librechat/api').ToolInputValidationError>} */
const toolInputValidationErrors = new Map();
const { contentParts, aggregateContent, stepMap } = createContentAggregator();
const artifactToolEndCallback = createToolEndCallback({
req,
res,
artifactPromises,
streamId,
jobCreatedAt,
});
/** Query accessible skill IDs once per run (shared across all agents).
* Skills activate under strict opt-in semantics — see
* `resolveAgentScopedSkillIds` for the per-agent activation predicate:
* - Ephemeral agent → model-spec `skills` config first, otherwise the
* per-conversation skills badge toggle (full catalog).
* - Persisted agent → `agent.skills_enabled === true`. Optional
* `agent.skills` allowlist narrows the catalog; empty/undefined
* allowlist with the toggle on = full accessible catalog. */
const enabledCapabilities = new Set(appConfig?.endpoints?.[EModelEndpoint.agents]?.capabilities);
const skillsCapabilityEnabled = enabledCapabilities.has(AgentCapabilities.skills);
const codeEnvAvailable = enabledCapabilities.has(AgentCapabilities.execute_code);
const backgroundToolsAvailable = enabledCapabilities.has(AgentCapabilities.run_in_background);
const toolIntentsAvailable = enabledCapabilities.has(AgentCapabilities.tool_intents);
const statefulSessionsAvailable = enabledCapabilities.has(
AgentCapabilities.stateful_code_sessions,
);
const allowedStatefulCodeEnvironments = resolveAllowedStatefulCodeEnvironments(
appConfig?.endpoints?.[EModelEndpoint.agents]?.statefulCodeSessions?.allowedEnvironments,
);
const ephemeralSkillsToggle = req.body?.ephemeralAgent?.skills === true;
const skillDbMethods = getSkillDbMethods();
if (!endpointOption.agent) {
throw new Error('No agent promise provided');
}
/** Run-level gate for inline memory tools: the `memory` capability must be
* enabled, memory must be configured, and the user must not have opted out.
* Requires the memory WRITE permissions (CREATE + UPDATE) — both inline tools
* mutate memory — so the tools aren't registered (and shown to the model) for
* read-only-memory roles that the runtime loader would then refuse to build.
* Agents (or the ephemeral memory badge) opt in per-agent via the `memory`
* marker on `tools`. */
const memoryAvailablePromise = resolveMemoryAvailability({
enabledCapabilities,
memoryConfig: appConfig?.memory,
user: req.user,
getRoleByName: db.getRoleByName,
});
const accessibleSkillIdsPromise = skillsCapabilityEnabled
? findAccessibleResources({
userId: req.user.id,
role: req.user.role,
resourceType: ResourceType.SKILL,
requiredPermissions: PermissionBits.VIEW,
}).then(withDeploymentSkillIds)
: Promise.resolve([]);
const editableSkillIdsPromise = skillsCapabilityEnabled
? findAccessibleResources({
userId: req.user.id,
role: req.user.role,
resourceType: ResourceType.SKILL,
requiredPermissions: PermissionBits.EDIT,
})
: Promise.resolve([]);
const skillCreateAllowedPromise = skillsCapabilityEnabled
? getSkillToolDeps().canCreateSkill({ req })
: Promise.resolve(false);
const skillStatesPromise = accessibleSkillIdsPromise.then((accessibleSkillIds) =>
loadSkillStates({
userId: req.user.id,
appConfig,
getUserById: db.getUserById,
accessibleSkillIds,
}),
);
const primaryAgentPromise = endpointOption.agent;
const modelsConfigPromise = getModelsConfig(req);
const validatedPrimaryAgentPromise = Promise.all([primaryAgentPromise, modelsConfigPromise]).then(
async ([primaryAgent, modelsConfig]) => {
if (!primaryAgent) {
throw new Error('Agent not found');
}
const validationResult = await validateAgentModel({
req,
res,
modelsConfig,
logViolation,
agent: primaryAgent,
});
if (!validationResult.isValid) {
throw new Error(validationResult.error?.message);
}
return { primaryAgent, modelsConfig };
},
);
/**
* Agent context store - populated after initialization, accessed by callback via closure.
* Maps agentId -> { userMCPAuthMap, agent, tool_resources, toolRegistry, openAIApiKey }
* @type {Map<string, {
* userMCPAuthMap?: Record<string, Record<string, string>>,
* agent?: object,
* tool_resources?: object,
* toolRegistry?: import('@librechat/agents').LCToolRegistry,
* requestScopedConnections?: import('@librechat/api').RequestScopedMCPConnectionStore,
* openAIApiKey?: string
* }>}
*/
const agentToolContexts = new Map();
/** Attach only the host-resolved route for the actually executing agent.
* Runnable metadata is transport data and may contain caller-controlled
* keys, so discard any incoming route context before resolving from the
* server-owned per-agent map. This covers both traditional TOOL_END events
* and event-driven ON_TOOL_EXECUTE callbacks. */
const toolEndCallback = async (data, metadata = {}) => {
const node = typeof metadata.langgraph_node === 'string' ? metadata.langgraph_node : '';
const nodeAgentId = node.startsWith(GraphNodeKeys.TOOLS)
? node.slice(GraphNodeKeys.TOOLS.length)
: undefined;
const executingAgentId =
metadata.executingAgentId ?? metadata.agentId ?? metadata.agent_id ?? nodeAgentId;
const soleContext =
agentToolContexts.size === 1 ? agentToolContexts.values().next().value : null;
const trustedContext =
(typeof executingAgentId === 'string' ? agentToolContexts.get(executingAgentId) : null) ??
soleContext;
const callbackMetadata = { ...metadata };
delete callbackMetadata.codeExecutionContext;
if (trustedContext?.codeExecutionContext) {
callbackMetadata.codeExecutionContext = trustedContext.codeExecutionContext;
}
return artifactToolEndCallback(data, callbackMetadata);
};
/** @type {Map<string, import('@librechat/api').EndpointTokenConfig | undefined>} */
const endpointTokenConfigByAgentId = new Map();
const toolExecuteOptions = {
loadTools: async (toolNames, agentId) => {
const ctx = agentToolContexts.get(agentId) ?? {};
logger.debug(`[ON_TOOL_EXECUTE] ctx found: ${!!ctx.userMCPAuthMap}, agent: ${ctx.agent?.id}`);
logger.debug(`[ON_TOOL_EXECUTE] toolRegistry size: ${ctx.toolRegistry?.size ?? 'undefined'}`);
const result = await loadToolsForExecution({
req,
res,
signal,
streamId,
conversationId,
toolNames,
agent: ctx.agent,
toolRegistry: ctx.toolRegistry,
backgroundToolNames: ctx.backgroundToolNames,
intentToolNames: ctx.intentToolNames,
mcpAvailableTools: ctx.mcpAvailableTools,
requestScopedConnections: ctx.requestScopedConnections,
userMCPAuthMap: ctx.userMCPAuthMap,
tool_resources: ctx.tool_resources,
actionsEnabled: ctx.actionsEnabled,
accessibleMcpServerNames: ctx.accessibleMcpServerNames,
jobCreatedAt,
});
logger.debug(`[ON_TOOL_EXECUTE] loaded ${result.loadedTools?.length ?? 0} tools`);
/** Per-agent narrowed flag (admin capability AND agent.tools
* includes execute_code), captured in `agentToolContexts` when
* the agent initialized. Falls back to `false` on any stray
* ctx miss so a skills-only agent never gains sandbox access
* even if capability lookup somehow skips. */
return enrichLoadedToolsWithAgentContext({
result,
req,
ctx,
});
},
toolEndCallback,
persistBackgroundCodeResult: createBackgroundCodeResultHandler({
req,
updateToolCallResult: db.updateToolCallResult,
}),
emitAttachment: createAttachmentEmitter({ res, streamId, jobCreatedAt }),
...getSkillToolDeps(),
};
const summarizationOptions =
appConfig?.summarization?.enabled === false ? { enabled: false } : { enabled: true };
/**
* Per-request map of per-subagent `createContentAggregator` instances
* keyed by the parent's `tool_call_id`. The handler in `callbacks.js`
* lazily creates an aggregator for each distinct `parentToolCallId`
* and folds every `ON_SUBAGENT_UPDATE` event into it as they stream
* in. `AgentClient` pulls each aggregator's `contentParts` at message
* save time and attaches them to the matching `subagent` tool_call so
* the child's reasoning / tool calls / final text survive a page
* refresh — the client-side Recoil atom is best-effort live-only.
*/
const subagentAggregatorsByToolCallId = new Map();
/** Backend prices each model call authoritatively (premium tiers, cache
* rates) and emits the cost on on_token_usage when contextCost is on, so
* the gauge sums real costs instead of re-deriving from base rates.
* `endpointTokenConfig` is filled in once `primaryConfig` resolves below so
* custom-endpoint agents price with their configured rates, not defaults. */
const usageCost = {
enabled: appConfig?.interfaceConfig?.contextCost === true,
pricing: { getMultiplier: db.getMultiplier, getCacheMultiplier: db.getCacheMultiplier },
};
/** Latest visible context snapshot + every emitted usage payload for this
* response, captured by the handlers and persisted on the response message's
* metadata so the breakdown and branch/total cost survive a reload.
* @type {{ latest: import('librechat-data-provider').TContextUsageEvent | null, count: number }} */
const contextUsageSink = { latest: null, count: 0 };
/** @type {Array<import('librechat-data-provider').TTokenUsageEvent>} */
const usageEmitSink = [];
const [
memoryAvailable,
accessibleSkillIds,
editableSkillIds,
skillCreateAllowed,
{ skillStates, defaultActiveOnShare },
{ primaryAgent, modelsConfig },
requestConversation,
] = await Promise.all([
memoryAvailablePromise,
accessibleSkillIdsPromise,
editableSkillIdsPromise,
skillCreateAllowedPromise,
skillStatesPromise,
validatedPrimaryAgentPromise,
requestConversationPromise,
]);
delete endpointOption.agent;
const agentConfigs = new Map();
const allowedProviders = new Set(appConfig?.endpoints?.[EModelEndpoint.agents]?.allowedProviders);
/** Event-driven mode: only load tool definitions, not full instances */
const loadTools = createToolLoader(signal, streamId, true, jobCreatedAt);
/** @type {Array<MongoFile>} */
const requestFiles = req.body.files ?? [];
/** @type {string | undefined} */
const parentMessageId = req.body.parentMessageId;
/**
* Skill names the user invoked via the `$` popover for this turn. Only flows
* to the primary agent — handoff agents are follow-up turns that don't see
* the user's per-submission `$` selections. `extractManualSkills` also
* drops non-string / empty elements so a crafted payload can't reach the
* `getSkillByName` DB query with nonsense values.
* @type {string[] | undefined}
*/
const manualSkills = extractManualSkills(req.body);
const selectedModelSpec =
endpointOption.spec && Array.isArray(appConfig?.modelSpecs?.list)
? appConfig.modelSpecs.list.find((modelSpec) => modelSpec.name === endpointOption.spec)
: null;
if (
primaryAgent &&
isEphemeralAgentId(primaryAgent.id) &&
selectedModelSpec &&
Object.hasOwn(selectedModelSpec, 'skills')
) {
if (selectedModelSpec.skills === true) {
primaryAgent.skills_enabled = true;
delete primaryAgent.skills;
} else if (selectedModelSpec.skills === false) {
primaryAgent.skills_enabled = false;
primaryAgent.skills = [];
} else if (Array.isArray(selectedModelSpec.skills)) {
const resolvedSkillIds = await resolveModelSpecSkillIds({
names: selectedModelSpec.skills,
accessibleSkillIds,
getSkillByName: skillDbMethods.getSkillByName,
});
primaryAgent.skills_enabled = true;
primaryAgent.skills = resolvedSkillIds.map((id) => id.toString());
}
}
const primaryScopedSkillIds = resolveAgentScopedSkillIds({
agent: primaryAgent,
accessibleSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
});
const primaryScopedEditableSkillIds = resolveAgentScopedSkillIds({
agent: primaryAgent,
accessibleSkillIds: editableSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
});
const primarySkillAuthoringAvailable = canAuthorSkillFiles({
agent: primaryAgent,
scopedEditableSkillIds: primaryScopedEditableSkillIds,
skillCreateAllowed,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
});
const primaryConfig = await initializeAgent(
{
req,
res,
loadTools,
requestFiles,
conversationId,
parentMessageId,
agent: primaryAgent,
endpointOption,
allowedProviders,
isInitialAgent: true,
accessibleSkillIds: primaryScopedSkillIds,
skillAuthoringAvailable: primarySkillAuthoringAvailable,
codeEnvAvailable,
backgroundToolsAvailable,
toolIntentsAvailable,
statefulSessionsAvailable,
allowedStatefulCodeEnvironments,
memoryAvailable,
skillStates,
defaultActiveOnShare,
manualSkills,
},
{
getFiles: db.getFiles,
getUserKey: db.getUserKey,
getMessages: db.getMessages,
getConvoFiles: db.getConvoFiles,
getAccessibleMcpServerNames,
updateFilesUsage: db.updateFilesUsage,
getUserKeyValues: db.getUserKeyValues,
getUserCodeFiles: db.getUserCodeFiles,
getToolFilesByIds: db.getToolFilesByIds,
getCodeGeneratedFiles: db.getCodeGeneratedFiles,
filterFilesByAgentAccess,
listSkillsByAccess: skillDbMethods.listSkillsByAccess,
listAlwaysApplySkills: skillDbMethods.listAlwaysApplySkills,
getSkillByName: skillDbMethods.getSkillByName,
},
);
/** Price emitted usage with the primary agent's resolved endpoint config so
* custom-endpoint agents reflect configured rates (mirrors the AgentClient
* spending path, which reads the same config). */
usageCost.endpointTokenConfig = primaryConfig.endpointTokenConfig;
logger.debug(
`[initializeClient] Storing tool context for ${primaryConfig.id}: ${primaryConfig.toolDefinitions?.length ?? 0} tools, registry size: ${primaryConfig.toolRegistry?.size ?? '0'}`,
);
agentToolContexts.set(
primaryConfig.id,
buildAgentToolContext({ agent: primaryAgent, config: primaryConfig }),
);
const {
agentConfigs: discoveredConfigs,
edges: discoveredEdges,
userMCPAuthMap: discoveredMCPAuthMap,
skippedAgentIds: discoveredSkippedIds,
} = await discoverConnectedAgents(
{
req,
res,
primaryConfig,
agent_ids: primaryConfig.agent_ids,
endpointOption,
allowedProviders,
modelsConfig,
loadTools,
requestFiles,
conversationId,
parentMessageId,
computeAccessibleSkillIds: (agent) =>
resolveAgentScopedSkillIds({
agent,
accessibleSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}),
computeSkillAuthoringAvailable: (agent) =>
canAuthorSkillFiles({
agent,
scopedEditableSkillIds: resolveAgentScopedSkillIds({
agent,
accessibleSkillIds: editableSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}),
skillCreateAllowed,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}),
skillStates,
defaultActiveOnShare,
codeEnvAvailable,
backgroundToolsAvailable,
toolIntentsAvailable,
statefulSessionsAvailable,
allowedStatefulCodeEnvironments,
memoryAvailable,
},
{
getAgent: db.getAgent,
checkPermission,
logViolation,
db: {
getFiles: db.getFiles,
getUserKey: db.getUserKey,
getMessages: db.getMessages,
getConvoFiles: db.getConvoFiles,
getAccessibleMcpServerNames,
updateFilesUsage: db.updateFilesUsage,
getUserKeyValues: db.getUserKeyValues,
getUserCodeFiles: db.getUserCodeFiles,
getToolFilesByIds: db.getToolFilesByIds,
getCodeGeneratedFiles: db.getCodeGeneratedFiles,
filterFilesByAgentAccess,
listSkillsByAccess: skillDbMethods.listSkillsByAccess,
listAlwaysApplySkills: skillDbMethods.listAlwaysApplySkills,
getSkillByName: skillDbMethods.getSkillByName,
},
// The callback fires during BFS, before the helper prunes agents
// whose edges end up filtered. Don't populate `agentConfigs` here —
// `discoveredConfigs` (returned below) is the authoritative pruned
// set. The per-agent tool context map is OK to keep populated even
// for pruned ids: it's only read by closure in ON_TOOL_EXECUTE,
// stale entries are unreachable at runtime.
onAgentInitialized: (agentId, agent, config) => {
agentToolContexts.set(agentId, buildAgentToolContext({ agent, config }));
},
// Pass through the `@librechat/api` exports so that tests which
// `jest.mock('@librechat/api')` can override the initializer/validator.
initializeAgent,
validateAgentModel,
},
);
// Copy the pruned discovery result into the outer map. Anything the
// helper dropped (skipped or unreachable after edge filtering) is
// intentionally absent. `processAddedConvo` below may still add more
// entries for parallel multi-convo execution.
for (const [agentId, config] of discoveredConfigs) {
agentConfigs.set(agentId, config);
}
let userMCPAuthMap = discoveredMCPAuthMap;
let edges = discoveredEdges;
/** Multi-Convo: Process addedConvo for parallel agent execution */
const { userMCPAuthMap: updatedMCPAuthMap } = await processAddedConvo({
req,
res,
loadTools,
logViolation,
modelsConfig,
requestFiles,
agentConfigs,
primaryAgent,
endpointOption,
userMCPAuthMap,
conversationId,
parentMessageId,
allowedProviders,
primaryAgentId: primaryConfig.id,
accessibleSkillIds,
editableSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
skillCreateAllowed,
skillStates,
defaultActiveOnShare,
codeEnvAvailable,
backgroundToolsAvailable,
toolIntentsAvailable,
statefulSessionsAvailable,
memoryAvailable,
});
if (updatedMCPAuthMap) {
userMCPAuthMap = updatedMCPAuthMap;
}
userMCPAuthMap ??= {};
for (const [agentId, config] of agentConfigs) {
if (agentToolContexts.has(agentId)) {
continue;
}
agentToolContexts.set(agentId, buildAgentToolContext({ agent: config, config }));
}
// `discoverConnectedAgents` always returns a concrete array, so no
// further normalization is needed before handing this to `createRun`.
primaryConfig.edges = edges;
// Subagents run in isolated context windows and are invoked via a dedicated
// spawn tool, not handoff edges. Explicit children are advertised as inert,
// VIEW-checked descriptors; model, tool, MCP, file, and skill initialization
// happens only when the SDK selects one.
const atSubagentThreadDepthLimit = !subagentThreadTaskStore.canCreateChildThread(
requestConversation?.subagentThread?.depth ?? 0,
);
const subagentsCapabilityEnabled = enabledCapabilities.has(AgentCapabilities.subagents);
const subagentsAvailableForRun = subagentsCapabilityEnabled && !atSubagentThreadDepthLimit;
/** Track skipped ids locally so repeated failures short-circuit within
* the subagent loading loop. Seeded from the discovery helper's skip
* list so agents that already failed handoff loading don't get retried. */
const skippedAgentIds = new Set(discoveredSkippedIds ?? []);
const lazyMetadataByAgentId = new Map();
const subagentGraphIds = new Set();
const expandedSubagentDescriptorState = { configCount: 0, rootAgentIds: [] };
const assertSubagentGraphRoom = (agentId) => {
if (subagentGraphIds.has(agentId)) {
return;
}
if (subagentGraphIds.size >= MAX_SUBAGENT_GRAPH_NODES) {
logger.warn('[initializeClient] Subagent graph node limit exceeded', {
agentId,
primaryAgentId: primaryConfig.id,
loadedSubagentCount: subagentGraphIds.size,
maxSubagentGraphNodes: MAX_SUBAGENT_GRAPH_NODES,
});
throw new Error(
`Subagent graph exceeds the maximum of ${MAX_SUBAGENT_GRAPH_NODES} unique agents.`,
);
}
};
const countExpandedSubagentDescriptor = (agentId) => {
expandedSubagentDescriptorState.configCount += 1;
if (expandedSubagentDescriptorState.configCount <= MAX_SUBAGENT_RUN_CONFIGS) {
return;
}
logger.warn('[initializeClient] Subagent run configuration limit exceeded', {
agentId,
expandedConfigCount: expandedSubagentDescriptorState.configCount,
maxSubagentRunConfigs: MAX_SUBAGENT_RUN_CONFIGS,
rootAgentIds: expandedSubagentDescriptorState.rootAgentIds,
});
throw new Error(
`Subagent run configuration exceeds the maximum of ${MAX_SUBAGENT_RUN_CONFIGS} expanded entries.`,
);
};
const userId = req.user?.id;
const userRole = req.user?.role;
const throwIfAborted = (abortSignal) => {
if (!abortSignal?.aborted) return;
throw abortSignal.reason instanceof Error
? abortSignal.reason
: new Error('Subagent resolution was aborted.');
};
const waitForAbort = (promise, abortSignal) => {
throwIfAborted(abortSignal);
if (!abortSignal) return promise;
return new Promise((resolve, reject) => {
const onAbort = () => {
reject(
abortSignal.reason instanceof Error
? abortSignal.reason
: new Error('Subagent resolution was aborted.'),
);
};
abortSignal.addEventListener('abort', onAbort, { once: true });
if (abortSignal.aborted) {
onAbort();
}
promise.then(resolve, reject).finally(() => {
abortSignal.removeEventListener('abort', onAbort);
});
});
};
const hasSubagentViewAccess = async (agent, agentId, abortSignal) => {
throwIfAborted(abortSignal);
if (!userId) return false;
const hasAccess = await waitForAbort(
checkPermission({
userId,
role: userRole,
resourceType: ResourceType.AGENT,
resourceId: agent._id,
requiredPermission: PermissionBits.VIEW,
}),
abortSignal,
);
throwIfAborted(abortSignal);
if (!hasAccess) {
logger.warn(
`[processAgent] User ${userId} lacks VIEW access to subagent ${agentId}, skipping`,
);
}
return hasAccess;
};
const getIncludeReasoningHistory = (agent) => {
if (!agent.provider) return undefined;
try {
return getProviderConfig({ provider: agent.provider, appConfig }).customEndpointConfig
?.customParams?.includeReasoningHistory;
} catch {
return undefined;
}
};
const getExplicitSubagentIds = (agent) =>
Array.from(
new Set(
Array.isArray(agent.subagents?.agent_ids)
? agent.subagents.agent_ids.filter(
(id) => typeof id === 'string' && id && id !== agent.id,
)
: [],
),
);
const toLazySubagentMetadata = (agent) => {
const statefulCodeSessions =
statefulSessionsAvailable === true &&
codeEnvAvailable === true &&
agent.stateful_code_sessions === true &&
agent.tools?.includes(Tools.execute_code) === true;
const statefulCodeEnvironment = agent.stateful_code_environment ?? 'user';
if (
statefulCodeSessions &&
!allowedStatefulCodeEnvironments.includes(statefulCodeEnvironment)
) {
throw createStatefulCodeEnvironmentPolicyError(statefulCodeEnvironment);
}
return {
id: agent.id,
name: agent.name,
description: agent.description,
provider: agent.provider,
model: agent.model,
model_parameters: { model: agent.model_parameters?.model },
recursion_limit: agent.recursion_limit,
subagents: agent.subagents,
configId: getLazySubagentConfigId(agent),
codeEnvAvailable:
codeEnvAvailable === true && agent.tools?.includes(Tools.execute_code) === true,
statefulCodeSessions,
statefulCodeEnvironment,
includeReasoningHistory: getIncludeReasoningHistory(agent),
};
};
const loadSubagentMetadata = async (agentId) => {
if (skippedAgentIds.has(agentId)) return null;
const cached = lazyMetadataByAgentId.get(agentId);
if (cached) return cached;
try {
const agent = await db.getAgentWithVersionCount({ id: agentId });
if (!agent || !(await hasSubagentViewAccess(agent, agentId))) {
skippedAgentIds.add(agentId);
return null;
}
const metadata = toLazySubagentMetadata(agent);
lazyMetadataByAgentId.set(agentId, metadata);
return metadata;
} catch (error) {
if (isFatalAgentInitializationError(error)) {
throw error;
}
logger.error(`[initializeClient] Error loading subagent metadata ${agentId}:`, error);
skippedAgentIds.add(agentId);
return null;
}
};
const loadGraphMemberCapabilityMetadata = async (agent) => {
if (!agent.subagents?.enabled) return [];
const memberIds = Array.from(
new Set((agent.subagents.graphs ?? []).flatMap((graph) => graph.agent_ids ?? [])),
).filter(
(memberId) =>
memberId !== agent.id && memberId !== primaryConfig.id && !agentConfigs.has(memberId),
);
const stagedMemberIds = memberIds.filter((memberId) => !subagentGraphIds.has(memberId));
if (subagentGraphIds.size + stagedMemberIds.length > MAX_SUBAGENT_GRAPH_NODES) {
logger.warn('[initializeClient] Subagent graph node limit exceeded', {
agentId: stagedMemberIds[0],
primaryAgentId: primaryConfig.id,
loadedSubagentCount: subagentGraphIds.size,
stagedSubagentCount: stagedMemberIds.length,
maxSubagentGraphNodes: MAX_SUBAGENT_GRAPH_NODES,
});
throw new Error(
`Subagent graph exceeds the maximum of ${MAX_SUBAGENT_GRAPH_NODES} unique agents.`,
);
}
for (const memberId of stagedMemberIds) {
subagentGraphIds.add(memberId);
}
const memberMetadata = await Promise.all(memberIds.map(loadSubagentMetadata));
return memberMetadata.filter(Boolean);
};
/**
* Resolves the selected descriptor inside the foreground request. The
* legacy initializer requires request/response objects for tool and MCP
* setup, so this intentionally remains request-scoped until AI-1597 gives
* child execution a durable runtime context.
*/
const initializeLoadedSubagent = async ({
agent,
agentId,
configId,
context,
lazyChildren,
viewAccessChecked = false,
}) => {
throwIfAborted(context.signal);
if (!agent || getLazySubagentConfigId(agent) !== configId) {
throw new Error(`Subagent ${agentId} changed before it could be initialized.`);
}
if (!viewAccessChecked && !(await hasSubagentViewAccess(agent, agentId, context.signal))) {
throw new Error(`You no longer have access to subagent ${agentId}.`);
}
const validation = await waitForAbort(
validateAgentModel({ req, res, agent, modelsConfig, logViolation }),
context.signal,
);
throwIfAborted(context.signal);
if (!validation.isValid) {
throw new Error(validation.error?.message ?? `Subagent ${agentId} failed model validation.`);
}
const scopedSkillIds = resolveAgentScopedSkillIds({
agent,
accessibleSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
});
const scopedEditableSkillIds = resolveAgentScopedSkillIds({
agent,
accessibleSkillIds: editableSkillIds,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
});
const config = await waitForAbort(
initializeAgent(
{
req,
res,
agent,
loadTools: createToolLoader(context.signal, streamId, true, jobCreatedAt),
requestFiles,
conversationId,
parentMessageId,
endpointOption: { ...endpointOption, endpoint: EModelEndpoint.agents },
allowedProviders,
accessibleSkillIds: scopedSkillIds,
skillAuthoringAvailable: canAuthorSkillFiles({
agent,
scopedEditableSkillIds,
skillCreateAllowed,
skillsCapabilityEnabled,
ephemeralSkillsToggle,
}),
codeEnvAvailable,
backgroundToolsAvailable,
toolIntentsAvailable,
statefulSessionsAvailable,
allowedStatefulCodeEnvironments,
memoryAvailable,
skillStates,
defaultActiveOnShare,
},
{
getFiles: db.getFiles,
getUserKey: db.getUserKey,
getMessages: db.getMessages,
getConvoFiles: db.getConvoFiles,
getAccessibleMcpServerNames,
updateFilesUsage: db.updateFilesUsage,
getUserKeyValues: db.getUserKeyValues,
getUserCodeFiles: db.getUserCodeFiles,
getToolFilesByIds: db.getToolFilesByIds,
getCodeGeneratedFiles: db.getCodeGeneratedFiles,
filterFilesByAgentAccess,
listSkillsByAccess: skillDbMethods.listSkillsByAccess,
listAlwaysApplySkills: skillDbMethods.listAlwaysApplySkills,
getSkillByName: skillDbMethods.getSkillByName,
},
),
context.signal,
);
throwIfAborted(context.signal);
config.lazySubagentConfigs = lazyChildren;
if (config.userMCPAuthMap) {
Object.assign(userMCPAuthMap, config.userMCPAuthMap);
}
agentToolContexts.set(agentId, buildAgentToolContext({ agent, config }));
endpointTokenConfigByAgentId.set(agentId, config.endpointTokenConfig);
return config;
};
const initializeLazySubagent = async ({ agentId, configId, context, lazyChildren }) => {
throwIfAborted(context.signal);
const agent = await waitForAbort(db.getAgentWithVersionCount({ id: agentId }), context.signal);
return initializeLoadedSubagent({ agent, agentId, configId, context, lazyChildren });
};
const buildLazySubagentDescriptors = async (agent, depth = 0, ancestors = new Set()) => {
if (!subagentsAvailableForRun || !agent.subagents?.enabled) {
return [];
}
if (agent.subagents.allowSelf !== false) {
countExpandedSubagentDescriptor(agent.id);
}
const subagentIds = getExplicitSubagentIds(agent);
if (subagentIds.length > 0 && depth >= MAX_SUBAGENT_DEPTH) {
logger.warn('[initializeClient] Subagent graph depth limit exceeded', {
agentId: agent.id,
primaryAgentId: primaryConfig.id,
depth,
maxSubagentDepth: MAX_SUBAGENT_DEPTH,
});
throw new Error(
`Subagent graph exceeds the maximum depth of ${MAX_SUBAGENT_DEPTH} at agent ${agent.id}.`,
);
}
const nextAncestors = new Set(ancestors);
nextAncestors.add(agent.id);
const descriptors = [];
for (const subagentId of subagentIds) {
if (skippedAgentIds.has(subagentId) || nextAncestors.has(subagentId)) continue;
if (subagentId !== primaryConfig.id) {
assertSubagentGraphRoom(subagentId);
}
const existing =
subagentId === primaryConfig.id ? primaryConfig : agentConfigs.get(subagentId);
if (existing) {
countExpandedSubagentDescriptor(subagentId);
if (subagentId !== primaryConfig.id) {
subagentGraphIds.add(subagentId);
}
const existingChildren = await buildLazySubagentDescriptors(
existing,
depth + 1,
nextAncestors,
);
existing.lazySubagentConfigs = existingChildren.filter((child) => child.configId);
existing.subagentAgentConfigs = existingChildren.filter((child) => !child.configId);
descriptors.push(existing);
continue;
}
const metadata = await loadSubagentMetadata(subagentId);
if (!metadata) continue;
countExpandedSubagentDescriptor(subagentId);
subagentGraphIds.add(subagentId);
const childDescriptors = await buildLazySubagentDescriptors(
metadata,
depth + 1,
nextAncestors,
);
const lazyChildren = childDescriptors.filter((child) => child.configId);
const eagerChildren = childDescriptors.filter((child) => !child.configId);
const subagentGraphMemberMetadata = await loadGraphMemberCapabilityMetadata(metadata);
descriptors.push({
id: metadata.id,
name: metadata.name,
description: metadata.description,
provider: metadata.provider,
model: metadata.model,
model_parameters: metadata.model_parameters,
recursion_limit: metadata.recursion_limit,
subagents: metadata.subagents,
configId: metadata.configId,
codeEnvAvailable: metadata.codeEnvAvailable,
statefulCodeSessions: metadata.statefulCodeSessions,
statefulCodeEnvironment: metadata.statefulCodeEnvironment,
includeReasoningHistory: metadata.includeReasoningHistory,
lazySubagentConfigs: lazyChildren,
subagentAgentConfigs: eagerChildren,
subagentGraphMemberMetadata,
resolve: async (context) =>
initializeLazySubagent({
agentId: metadata.id,
configId: metadata.configId,
context,
lazyChildren,
}).then(async (config) => {
config.subagentAgentConfigs = eagerChildren;
graphMemberConfigsById.set(config.id, config);
await resolveGraphSubagentsFor(config, context.signal);
return config;
}),
});
}
return descriptors;
};
const resolveSubagentTrees = async (rootConfigs) => {
expandedSubagentDescriptorState.rootAgentIds = rootConfigs
.filter((config) => config?.id)
.map((config) => config.id);
for (const config of rootConfigs) {
if (!config?.id) continue;
const descriptors = await buildLazySubagentDescriptors(config);
config.lazySubagentConfigs = descriptors.filter((child) => child.configId);
config.subagentAgentConfigs = descriptors.filter((child) => !child.configId);
}
};
const rootSubagentConfigs = [primaryConfig, ...agentConfigs.values()];
await resolveSubagentTrees(rootSubagentConfigs);
const graphMemberConfigsById = new Map(
rootSubagentConfigs.filter((config) => config?.id).map((config) => [config.id, config]),
);
const graphMemberLoadsById = new Map();
const initializeGraphMember = createConcurrencyLimiter(SUBAGENT_GRAPH_LOAD_CONCURRENCY);
const loadGraphMemberOnce = async (memberId) => {
throwIfAborted(signal);
const cached = graphMemberConfigsById.get(memberId);
if (cached) return cached;
if (skippedAgentIds.has(memberId)) return null;
assertSubagentGraphRoom(memberId);
subagentGraphIds.add(memberId);
const agent = await waitForAbort(db.getAgentWithVersionCount({ id: memberId }), signal);
if (!agent || !(await hasSubagentViewAccess(agent, memberId, signal))) {
skippedAgentIds.add(memberId);
return null;
}
try {
const config = await initializeLoadedSubagent({
agent,
agentId: memberId,
configId: getLazySubagentConfigId(agent),
context: { signal },
lazyChildren: [],
viewAccessChecked: true,
});
graphMemberConfigsById.set(memberId, config);
return config;
} catch (error) {
if (isFatalAgentInitializationError(error)) {
throw error;
}
logger.error(`[initializeClient] Error initializing graph member ${memberId}:`, error);
skippedAgentIds.add(memberId);
return null;
}
};
const loadGraphMember = async (memberId, graphSignal = signal) => {
throwIfAborted(graphSignal);
const cached = graphMemberConfigsById.get(memberId);
if (cached) return cached;
let pending = graphMemberLoadsById.get(memberId);
if (!pending) {
pending = loadGraphMemberOnce(memberId);
graphMemberLoadsById.set(memberId, pending);
pending.then(
() => {
if (graphMemberLoadsById.get(memberId) === pending) {
graphMemberLoadsById.delete(memberId);
}
},
() => {
if (graphMemberLoadsById.get(memberId) === pending) {
graphMemberLoadsById.delete(memberId);
}
},
);
}
return waitForAbort(pending, graphSignal);
};
async function resolveGraphSubagentsFor(config, graphSignal = signal) {
throwIfAborted(graphSignal);
const definitions =
subagentsAvailableForRun && config.subagents?.enabled === true
? (config.subagents.graphs ?? [])
: [];
const resolvedGraphs = [];
for (const definition of definitions) {
const memberIds = [...new Set(definition.agent_ids ?? [])];
const unloadedMemberIds = memberIds.filter(
(memberId) => !graphMemberConfigsById.has(memberId) && !subagentGraphIds.has(memberId),
);
if (subagentGraphIds.size + unloadedMemberIds.length > MAX_SUBAGENT_GRAPH_NODES) {
const overflowIndex = MAX_SUBAGENT_GRAPH_NODES - subagentGraphIds.size;
logger.warn('[initializeClient] Subagent graph node limit exceeded', {
agentId: unloadedMemberIds[Math.max(overflowIndex, 0)],
primaryAgentId: primaryConfig.id,
loadedSubagentCount: subagentGraphIds.size,
stagedSubagentCount: unloadedMemberIds.length,
maxSubagentGraphNodes: MAX_SUBAGENT_GRAPH_NODES,
});
continue;
}
for (const memberId of unloadedMemberIds) {
subagentGraphIds.add(memberId);
}
const memberConfigs = await Promise.all(
memberIds.map((memberId) =>
initializeGraphMember(() => loadGraphMember(memberId, graphSignal)),
),
);
throwIfAborted(graphSignal);
if (memberConfigs.some((member) => member == null)) {
logger.warn('[initializeClient] Skipping incomplete graph subagent', {
parentAgentId: config.id,
graphType: definition.type,
expectedMemberCount: memberIds.length,
resolvedMemberCount: memberConfigs.filter(Boolean).length,
});
continue;
}
resolvedGraphs.push({ definition, memberConfigs });
}
config.subagentGraphConfigs = resolvedGraphs;
}
for (const config of rootSubagentConfigs) {
await resolveGraphSubagentsFor(config);
}
/** Build detached execution only for an attributable owner/thread. New
* tasks still require a spawnable child, while an existing registered live
* task keeps its poll/control seam after agent configuration changes. The
* SDK receives only this trusted host scope; models can select a child
* `threadId`, never the owner or parent-thread namespace. */
const hasSpawnableSubagent = rootSubagentConfigs.some(
(config) =>
config.subagents?.enabled === true &&
(config.subagents.allowSelf !== false ||
(config.subagentAgentConfigs?.length ?? 0) > 0 ||
(config.lazySubagentConfigs?.length ?? 0) > 0 ||
(config.subagentGraphConfigs?.length ?? 0) > 0),
);
const trustedSubagentTasks =
backgroundToolsAvailable &&
typeof req.user?.id === 'string' &&
req.user.id !== '' &&
typeof conversationId === 'string' &&
conversationId !== ''
? buildSubagentThreadTaskConfig(subagentThreadTaskStore, {
userId: req.user.id,
parentConversationId: conversationId,
...(typeof req.user.tenantId === 'string' && req.user.tenantId !== ''
? { tenantId: req.user.tenantId }
: {}),
})
: undefined;
let hasExistingSubagentTask = false;
if (trustedSubagentTasks != null && !(subagentsAvailableForRun && hasSpawnableSubagent)) {
try {
hasExistingSubagentTask = await subagentThreadTaskStore.hasTasks(
trustedSubagentTasks.scopeId,
);
} catch (error) {
/** Keep the poll/control tool visible when the owner directory is briefly
* unavailable. The tool then returns an honest `unavailable` status
* instead of making a live task look nonexistent. */
logger.warn('[initializeClient] Failed to inspect routed subagent tasks', error);
hasExistingSubagentTask = true;
}
}
const subagentTasks =
trustedSubagentTasks != null &&
((subagentsAvailableForRun && hasSpawnableSubagent) || hasExistingSubagentTask)
? trustedSubagentTasks
: undefined;
if (subagentTasks != null) {
toolExecuteOptions.subagentTasks = subagentTasks;
}
primaryConfig.subagents = subagentsAvailableForRun ? primaryConfig.subagents : undefined;
/** If the capability is off or this durable child is at the depth limit,
* strip `subagents` on every loaded config — not just the primary. `run.ts` calls
* `buildSubagentConfigs` for every agent in the array, so a handoff
* agent with `subagents.enabled: true` persisted on its document would
* otherwise still expose self-spawn at runtime. */
if (!subagentsAvailableForRun) {
primaryConfig.lazySubagentConfigs = undefined;
primaryConfig.subagentGraphConfigs = undefined;
for (const config of agentConfigs.values()) {
config.subagents = undefined;
config.subagentAgentConfigs = undefined;
config.lazySubagentConfigs = undefined;
config.subagentGraphConfigs = undefined;
}
}
const agentContextAttachmentsByAgentId = buildAgentContextAttachmentsByAgentId([
primaryConfig,
...agentConfigs.values(),
]);
let endpointConfig = appConfig.endpoints?.[primaryConfig.endpoint];
if (!isAgentsEndpoint(primaryConfig.endpoint) && !endpointConfig) {
try {
endpointConfig = getCustomEndpointConfig({
endpoint: primaryConfig.endpoint,
appConfig,
});
} catch (err) {
logger.error(
'[/api/server/services/Endpoints/agents/initialize.js] Error getting custom endpoint config',
err,
);
}
}
const sender = resolveSender({
agent: primaryConfig,
specLabel: selectedModelSpec?.label,
endpointOption: {
...endpointOption,
model: endpointOption.model_parameters.model,
modelDisplayLabel: endpointConfig?.modelDisplayLabel,
modelLabel: endpointOption.model_parameters.modelLabel,
},
});
/** History priming uses the user's full ACL-accessible skill set (not
* per-agent scoped) because prior turns may reference skills no longer
* in any active agent's scope; the ACL check is the security gate. Each
* selected Code API deployment receives its own upload, and only session
* partitions routed to that deployment receive those storage pointers. */
const codeExecutionProfiles = collectCodeExecutionProfileRoutes(
[primaryConfig, ...agentConfigs.values()],
{
userId: req.user.id,
conversationId,
},
);
const handlePrimeInvokedSkills = skillsCapabilityEnabled
? (payload) =>
primeInvokedSkillsForProfiles({
req,
payload,
accessibleSkillIds,
executionProfiles: codeExecutionProfiles,
...getSkillToolDeps(),
})
: undefined;
/** Per-agent resolved endpoint token config, keyed by agent id. Built from
* `agentToolContexts` (the one map holding every agent, including pure
* subagents pruned from `agentConfigs`) so usage billed/emitted for a
* connected or subagent on a different custom endpoint is priced with THAT
* agent's configured rates instead of the primary's. Every known agent is
* recorded — even with an `undefined` config — so the resolver can tell a
* known non-custom agent (built-in pricing) from an untagged/unknown one
* (primary fallback).
* @type {Map<string, import('@librechat/api').EndpointTokenConfig | undefined>} */
for (const [agentId, ctx] of agentToolContexts) {
endpointTokenConfigByAgentId.set(agentId, ctx?.endpointTokenConfig);
}
/** Price emitted usage per producing agent too, so the streamed/persisted
* `metadata.usage.cost` matches the per-agent balance transaction. */
usageCost.resolveEndpointTokenConfig = (usage) =>
resolveAgentTokenConfig({
agentId: usage?.agentId,
byAgentId: endpointTokenConfigByAgentId,
fallback: usageCost.endpointTokenConfig,
});
const eventHandlers = getDefaultHandlers({
res,
contentParts,
stepMap,
toolInputValidationErrors,
toolExecuteOptions,
summarizationOptions,
aggregateContent,
toolEndCallback,
collectedUsage,
collectedThoughtSignatures,
streamId,
jobCreatedAt,
subagentAggregatorsByToolCallId,
usageCost,
contextUsageSink,
usageEmitSink,
});
const client = new AgentClient({
req,
res,
sender,
contentParts,
stepMap,
agentConfigs,
eventHandlers,
collectedUsage,
collectedThoughtSignatures,
aggregateContent,
artifactPromises,
primeInvokedSkills: handlePrimeInvokedSkills,
agent: primaryConfig,
spec: endpointOption.spec,
iconURL: endpointOption.iconURL,
chatProjectId: endpointOption.chatProjectId,
attachments: primaryConfig.requestAttachments ?? primaryConfig.attachments,
agentContextAttachmentsByAgentId,
endpointType: endpointOption.endpointType,
resendFiles: primaryConfig.resendFiles ?? true,
maxContextTokens: primaryConfig.maxContextTokens,
endpoint: isEphemeralAgentId(primaryConfig.id) ? primaryConfig.endpoint : EModelEndpoint.agents,
subagentAggregatorsByToolCallId,
subagentTasks,
/** Resolved endpoint token/pricing config so spending and cost reflect
* configured rates for custom-endpoint agents instead of defaults. */
endpointTokenConfig: primaryConfig.endpointTokenConfig,
/** Per-agent override of the above for multi-endpoint graphs (connected
* agents + subagents); falls back to the primary config when an agent
* isn't present or has no configured rates. */
endpointTokenConfigByAgentId,
/** Capture sinks the handlers fill during the run; `sendCompletion` reads
* them to persist the breakdown + usage rollup on the response message. */
contextUsageSink,
usageEmitSink,
startupTelemetry,
toolInputValidationErrors,
jobCreatedAt,
checkpointNamespace,
});
if (streamId) {
GenerationJobManager.setCollectedUsage(streamId, collectedUsage, jobCreatedAt);
}
return { client, userMCPAuthMap };
};
module.exports = { initializeClient };