LibreChat/api/server/services/Tools/mcp.js
Danny Avila ddd2dd8bf4
🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap (#15867)
* 🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap

A server stored as an inspection-failed stub is recovered by the first
request that inspects it successfully, but every request that read the
stub before that write inspected again and wrote again. Each later write
bumped `updatedAt` past the connection made after the earlier one, so the
connection was replaced and a caller still holding it listed no tools. A
request that reached `reinspectServer` after the write hit the
"not in a failed state" guard and reported the server as still
unreachable.

- Share one in-flight reinspection per stored entry and effective
  allowlists in each process; DB entries are keyed per user and tenant.
- Replace the stub only while it is still the inspected stub: a new
  optional `replaceStub` compare-and-set on the in-memory and Redis
  aggregate-key stores (one Lua script), so replicas never overwrite a
  recovery another replica stored.
- Resolve a reinspection that finds the server already recovered to the
  stored config, and move the reinitialize decision into
  `MCPServersRegistry.recoverServerConfig` so `reinitMCPServer` connects
  with the recovered config instead of the stub it read.
- Pin the interleavings in `MCPReinitRecovery.integration.test.ts` against
  a real in-process MCP server, and require every concurrent reinitialize
  flow there to recover with a complete tool list.

* 🪢 fix: Key Shared Reinspections by the Stub They Inspect

A registry re-initialization can replace a failed stub while an older
reinspection of it is still running. The shared flight was keyed by
entry and allowlists only, so a request that read the newer stub joined
the older inspection and inherited its outcome without the newer config
ever being inspected.

`reinspectServer` now reads the entry before joining and keys the flight
by the stub's `updatedAt`. A flight whose compare-and-set loses to a
newer failed stub joins the inspection of that stub instead of reporting
the server unreachable, and a store that leaves the inspected stub in
place rejects rather than joining its own flight.

* 🪢 fix: Settle a Reinspection Against the Entry Stored After It

A reinspection that did not replace its stub only looked at storage when
its compare-and-set lost. Two paths reached a stale outcome:

- An inspection that failed rejected as unreachable even when another
  request had already stored a recovery, for example after capturing the
  stub before a slow allowlist resolution. Allowlists now resolve before
  the entry is read, and a failed inspection settles against the stored
  entry the same way a lost compare-and-set does.
- Returning a recovery that another replica stored left this replica's
  process-local `getAllServerConfigs` memo serving the stub until its TTL.
  Settling on a stored recovery now invalidates this replica's registry
  read caches, as the writing path does.

* 🪢 fix: Decide Reinspection Outcomes From Current Shared State

The Redis aggregate-key store answers `get` from a process-local
snapshot that can lag another replica's write by its TTL. A replica
whose inspection failed read that snapshot while settling, still saw the
stub, and reported the server unreachable although Redis held another
replica's recovery. The same stale read also sent a replica into a
needless inspection of a server another replica had already recovered.

- Add an optional `getCurrent` to the repository interface, implemented
  by the aggregate-key store as a read past its snapshot, and use it for
  the reinspection's first read and for settling.
- Inspect a newer stub within the flight that found it instead of
  joining another flight, so no flight ever waits on a second one.

* 🪢 fix: Join the Flight Already Inspecting a Newer Stub

A flight that lost to a newer failed stub inspected that stub itself even
when another request was already inspecting it. If that redundant
inspection failed transiently, the flight settled against the unchanged
newer stub and rejected its callers while the other flight went on to
store the recovery.

A flight now joins the flight for the newer stub, starting it when none
is pending, but only when the stub is strictly newer by `updatedAt`, so
every wait points forward and flights stay acyclic. A replacement that
is not newer, which only clock skew can produce, is still inspected
within the flight.

* 🪢 fix: Track Reinspection Waits Instead of Ordering Them by Clock

Joining only flights for strictly newer stubs kept waits acyclic by
relying on `updatedAt` order, which replicas with skewed clocks do not
guarantee. A replacement stored with an older timestamp was inspected
redundantly instead of joining the flight already inspecting it, and a
transient failure of that redundant inspection rejected callers the
other flight went on to recover.

Settling flights now record the flight they wait on and join any
in-flight reinspection for the stub they found, unless that flight
already waits on them directly or through others. Only a join that
would close a cycle falls back to inspecting within the settling flight.

* 🪢 refactor: Resolve the Reinitialize Config in packages/api

`reinitMCPServer` still decided in CJS what a failed-inspection config
recovers to and built the `unreachable` response itself. That decision
now lives in `resolveMCPReinitializeConfig` in packages/api, which takes
the registry as a dependency and returns either the config to connect
with or the typed result that ends the reinitialization. The CJS service
only passes request data in and returns that result.
2026-09-12 19:50:14 -04:00

472 lines
17 KiB
JavaScript

const { logger } = require('@librechat/data-schemas');
const {
formatMCPServerTools,
getUserMCPAuthMap,
getMissingCustomUserVars,
loadMCPServerCatalogs: loadCatalogs,
resolveMCPReinitializeConfig,
requiresEphemeralUserConnection,
getMissingRuntimeBodyPlaceholderFields,
MCPAuthenticationRejectedError,
MCPAuthenticationRefreshError,
OpenIDReauthRequiredError,
prepareMCPAuthorizationMutation,
} = require('@librechat/api');
const { CacheKeys, Constants } = require('librechat-data-provider');
const { getMCPManager, getMCPServersRegistry, getFlowStateManager } = require('~/config');
const {
findToken,
createToken,
updateToken,
deleteTokens,
findPluginAuthsByKeys,
} = require('~/models');
const { getGraphApiToken } = require('~/server/services/GraphTokenService');
const { exchangeOboToken } = require('~/server/services/OboTokenService');
const { createOboTrustChecker } = require('~/server/services/OboPolicyService');
const {
getMCPServerTools,
cacheMCPServerTools,
getMCPToolsCacheGeneration,
updateMCPServerTools,
invalidateCachedTools,
} = require('~/server/services/Config');
const { getLogStores } = require('~/cache');
const {
clearMCPAuthorizationFenceRetry,
persistMCPAuthorizationFenceRetry,
} = require('~/server/services/MCPAuthorizationFenceRetry');
const MCP_REINITIALIZE_FAILURE_REASONS = {
MISSING_CUSTOM_USER_VARS: 'missing_custom_user_vars',
OAUTH_REQUIRED: 'oauth_required',
INITIALIZATION_FAILED: 'initialization_failed',
};
const isMCPReauthenticationError = (error) =>
error instanceof MCPAuthenticationRejectedError ||
error instanceof MCPAuthenticationRefreshError ||
error instanceof OpenIDReauthRequiredError;
/** Wires application dependencies into the passive, request-local catalog recovery service.
* @param {Object} params
* @param {IUser} params.user
* @param {Array<{ serverName: string, serverConfig: object }>} params.servers
* @param {import('@librechat/api').UpstreamTokenProvider} [params.upstreamTokenProvider] - Live upstream-token closure for OBO discovery, built at the request boundary so this layer never receives the raw Express request.
* @param {import('@librechat/api').AuthIdentityContext} [params.oboIdentityContext] - Non-template-visible OBO identity context built from the real request user.
* @param {AbortSignal} [params.signal] - Cancels queued and in-flight catalog reads when the request ends.
* @param {import('@librechat/api').MCPServerCatalogRecoveryPolicy} [params.recoveryPolicy]
*/
async function loadMCPServerCatalogs({
user,
servers,
upstreamTokenProvider,
oboIdentityContext,
signal,
recoveryPolicy,
}) {
const flowManager = getFlowStateManager(getLogStores(CacheKeys.FLOWS));
const tokenMethods = { findToken, updateToken, createToken, deleteTokens };
const mcpManager = getMCPManager();
const onOAuthCredentialsChanging = (scope) =>
prepareMCPAuthorizationMutation(scope, {
invalidateRecoveryGeneration: invalidateCachedTools,
persistPublicationRetry: persistMCPAuthorizationFenceRetry,
clearPublicationRetry: clearMCPAuthorizationFenceRetry,
clearLocalRecovery: (userId, serverName) =>
mcpManager.clearCatalogRecoveryState?.(userId, serverName),
retryDelaysMs: recoveryPolicy?.authorizationFenceRetryMs,
attemptTimeoutMs: recoveryPolicy?.authorizationFenceTimeoutMs,
});
return loadCatalogs(
{ user, servers, signal, recoveryPolicy },
{
loadUserMCPAuthMap: (userId, serverNames) =>
getUserMCPAuthMap({
userId,
servers: serverNames,
findPluginAuthsByKeys,
}),
discoverServerTools: (options) =>
mcpManager.discoverServerTools({
...options,
flowManager,
tokenMethods,
graphTokenResolver: getGraphApiToken,
oboTokenResolver: exchangeOboToken,
oboTrustChecker: createOboTrustChecker(),
upstreamTokenProvider,
oboIdentityContext,
onOAuthCredentialsChanging,
}),
formatServerTools: formatMCPServerTools,
recoveryTracker: mcpManager.getCatalogRecoveryTracker?.(),
getRecoveryGeneration: getMCPToolsCacheGeneration,
getCachedServerTools: getMCPServerTools,
getServerToolFunctionsSnapshot: (userId, serverName, serverConfig, options) =>
mcpManager.getServerToolFunctionsSnapshot(userId, serverName, serverConfig, options),
cacheServerTools: cacheMCPServerTools,
},
);
}
/**
* Reinitializes an MCP server connection and discovers available tools.
* When OAuth is required, uses discovery mode to list tools without full authentication
* (per MCP spec, tool listing should be possible without auth).
* @param {Object} params
* @param {IUser} params.user - The user from the request object.
* @param {import('@librechat/api').UpstreamTokenProvider} [params.upstreamTokenProvider] - Live upstream-token closure for OBO connection establishment, built at the request boundary so this layer never receives the raw Express request.
* @param {import('@librechat/api').AuthIdentityContext} [params.oboIdentityContext] - Non-template-visible OBO identity context built from the real request user.
* @param {string} params.serverName - The name of the MCP server
* @param {boolean} params.returnOnOAuth - Whether to initiate OAuth and return, or wait for OAuth flow to finish
* @param {AbortSignal} [params.signal] - The abort signal to handle cancellation.
* @param {boolean} [params.forceNew]
* @param {number} [params.connectionTimeout]
* @param {FlowStateManager<any>} [params.flowManager]
* @param {(authURL: string, options?: { expiresAt?: number }) => Promise<void>} [params.oauthStart]
* @param {() => Promise<void>} [params.oauthEnd]
* @param {import('@librechat/api').RequestBody} [params.requestBody]
* @param {import('@librechat/api').RequestScopedMCPConnectionStore} [params.requestScopedConnections]
* @param {Record<string, Record<string, string>>} [params.userMCPAuthMap]
* @param {import('@librechat/api').MCPServerCatalogRecoveryPolicy} [params.recoveryPolicy]
*/
async function reinitMCPServer({
user,
signal,
forceNew,
serverName,
configServers,
userMCPAuthMap,
connectionTimeout,
returnOnOAuth = true,
oauthStart: _oauthStart,
flowManager: _flowManager,
serverConfig: providedConfig,
requestBody,
requestScopedConnections,
upstreamTokenProvider,
oboIdentityContext,
oauthEnd,
recoveryPolicy,
}) {
/** @type {MCPConnection | null} */
let connection = null;
let serverConfig = providedConfig;
/** @type {LCAvailableTools | null} */
let availableTools = null;
/** @type {ReturnType<MCPConnection['fetchTools']> | null} */
let tools = null;
let oauthRequired = false;
let oauthUrl = null;
let oauthExpiresAt;
let ephemeralServer = false;
let publicationGeneration;
let publicationRevision;
try {
const registry = getMCPServersRegistry();
const resolution = await resolveMCPReinitializeConfig(
registry,
serverName,
serverConfig ?? (await registry.getServerConfig(serverName, user?.id, configServers)),
user?.id,
);
if (resolution.result) {
return resolution.result;
}
serverConfig = resolution.serverConfig;
ephemeralServer = serverConfig ? requiresEphemeralUserConnection(serverConfig) : false;
const customUserVars = userMCPAuthMap?.[`${Constants.mcp_prefix}${serverName}`];
const missingUserVars = getMissingCustomUserVars(serverConfig ?? {}, customUserVars);
if (missingUserVars.length > 0) {
logger.warn('[MCP Reinitialize] Skipping server with missing user configuration', {
missingVariableCount: missingUserVars.length,
});
return {
availableTools: null,
success: false,
message: `MCP server '${serverName}' requires user-provided variable(s) [${missingUserVars.join(
', ',
)}] which are not set`,
failureReason: MCP_REINITIALIZE_FAILURE_REASONS.MISSING_CUSTOM_USER_VARS,
missingUserVars,
oauthRequired: false,
serverName,
oauthUrl: null,
tools: null,
};
}
/** `{{LIBRECHAT_BODY_*}}` placeholders only resolve during a chat turn; connecting
* without them would fail, so defer the connection instead of reporting a failure. */
const missingBodyFields = serverConfig
? getMissingRuntimeBodyPlaceholderFields(serverConfig, requestBody)
: [];
if (missingBodyFields.length > 0) {
logger.info(
'[MCP Reinitialize] Runtime placeholders unresolved; connection deferred to first use',
{ missingBodyFieldCount: missingBodyFields.length },
);
return {
availableTools: null,
success: true,
/** Lets clients distinguish "connection deferred to a chat turn" from a
* plain success with no tools, e.g. to attach the server at the server
* level instead of waiting for a tool list that never arrives. */
connectionDeferred: true,
message: `MCP server '${serverName}' uses request-scoped placeholders; connection will be established on first use in a chat turn`,
oauthRequired: false,
serverName,
oauthUrl: null,
tools: null,
};
}
const flowManager = _flowManager ?? getFlowStateManager(getLogStores(CacheKeys.FLOWS));
const mcpManager = getMCPManager();
const onOAuthCredentialsChanging = (scope) =>
prepareMCPAuthorizationMutation(scope, {
invalidateRecoveryGeneration: invalidateCachedTools,
persistPublicationRetry: persistMCPAuthorizationFenceRetry,
clearPublicationRetry: clearMCPAuthorizationFenceRetry,
clearLocalRecovery: (userId, changedServerName) =>
mcpManager.clearCatalogRecoveryState?.(userId, changedServerName),
retryDelaysMs: recoveryPolicy?.authorizationFenceRetryMs,
attemptTimeoutMs: recoveryPolicy?.authorizationFenceTimeoutMs,
});
const tokenMethods = { findToken, updateToken, createToken, deleteTokens };
if (!ephemeralServer) {
publicationGeneration = await getMCPToolsCacheGeneration({
userId: user.id,
serverName,
});
}
const oauthStart =
_oauthStart ??
(async (authURL, options) => {
logger.info('[MCP Reinitialize] OAuth URL received');
if (authURL !== oauthUrl) {
oauthExpiresAt = undefined;
}
oauthUrl = authURL;
if (typeof options?.expiresAt === 'number' && Number.isFinite(options.expiresAt)) {
oauthExpiresAt = options.expiresAt;
}
oauthRequired = true;
});
try {
connection = await mcpManager.getConnection({
user,
signal,
forceNew,
oauthStart,
serverName,
flowManager,
tokenMethods,
onOAuthCredentialsChanging,
returnOnOAuth,
oauthEnd,
customUserVars,
requestBody,
requestScopedConnections,
connectionTimeout,
serverConfig,
graphTokenResolver: getGraphApiToken,
oboTokenResolver: exchangeOboToken,
oboTrustChecker: createOboTrustChecker(),
upstreamTokenProvider,
oboIdentityContext,
});
logger.info('[MCP Reinitialize] Successfully established connection');
} catch (err) {
if (isMCPReauthenticationError(err)) {
throw err;
}
logger.info('[MCP Reinitialize] Connection attempt failed');
logger.info(
`[MCP Reinitialize] OAuth state - oauthRequired: ${oauthRequired}, oauthUrl: ${oauthUrl ? 'present' : 'null'}`,
);
const isOAuthError =
err.message?.includes('OAuth') ||
err.message?.includes('authentication') ||
err.message?.includes('401');
const isOAuthFlowInitiated = err.message === 'OAuth flow initiated - return early';
if (isOAuthError || oauthRequired || isOAuthFlowInitiated) {
logger.info('[MCP Reinitialize] OAuth required; attempting tool discovery without auth');
oauthRequired = true;
try {
const discoveryResult = await mcpManager.discoverServerTools({
user,
signal,
serverName,
flowManager,
tokenMethods,
onOAuthCredentialsChanging,
oauthStart,
customUserVars,
requestBody,
connectionTimeout,
configServers,
graphTokenResolver: getGraphApiToken,
oboTokenResolver: exchangeOboToken,
oboTrustChecker: createOboTrustChecker(),
upstreamTokenProvider,
oboIdentityContext,
});
if (discoveryResult.tools && discoveryResult.tools.length > 0) {
tools = discoveryResult.tools;
logger.info(
`[MCP Reinitialize] Discovered ${tools.length} tools without full authentication`,
);
}
} catch (error) {
if (isMCPReauthenticationError(error)) {
throw error;
}
logger.debug('[MCP Reinitialize] Tool discovery failed');
}
} else {
logger.error('[MCP Reinitialize] Error initializing MCP server');
}
}
if (connection && !oauthRequired) {
publicationGeneration =
mcpManager.getToolPublicationGeneration(connection) ?? publicationGeneration;
let snapshot;
if (typeof connection.fetchOrderedToolsSnapshot === 'function') {
snapshot = await connection.fetchOrderedToolsSnapshot();
} else if (typeof connection.fetchToolsSnapshot === 'function') {
snapshot = await connection.fetchToolsSnapshot();
} else {
snapshot = { tools: await connection.fetchTools(), complete: true };
}
if (snapshot.complete) {
tools = snapshot.tools;
/** Reserved before this snapshot's tools/list; an app-level catalog cannot publish
* without it, and allocating a later one here would outrank fresher tools. */
publicationRevision = snapshot.publicationRevision;
if (snapshot.orderingUnavailable && typeof connection.refreshToolList === 'function') {
/** These tools still serve this request; the connection republishes the shared
* catalog under backoff rather than leaving it cold until the next reinitialize. */
connection
.refreshToolList()
.catch((err) =>
logger.debug(
`[MCP Reinitialize] Could not schedule a catalog republish for ${serverName}: ${err?.message ?? String(err)}`,
),
);
}
} else {
logger.warn(
`[MCP Reinitialize] Preserving cached tools for ${serverName} because tools/list returned an incomplete snapshot`,
);
}
}
if (tools && !ephemeralServer && publicationGeneration) {
const currentGeneration = await getMCPToolsCacheGeneration({
userId: user.id,
serverName,
});
if (currentGeneration !== publicationGeneration) {
logger.warn(
`[MCP Reinitialize] Discarding stale tools for ${serverName} because its publication generation changed during discovery`,
);
tools = null;
}
}
if (tools) {
availableTools = await updateMCPServerTools({
userId: user.id,
serverName,
tools,
serverConfig,
...(publicationGeneration && { publicationGeneration }),
...(publicationRevision && { publicationRevision }),
});
if (availableTools == null) {
tools = null;
}
}
logger.debug('[MCP Reinitialize] Sending response', {
oauthRequired,
hasOauthUrl: Boolean(oauthUrl),
});
const getResponseMessage = () => {
if (oauthRequired && tools && tools.length > 0) {
return `MCP server '${serverName}' tools discovered, OAuth required for execution`;
}
if (oauthRequired) {
return `MCP server '${serverName}' ready for OAuth authentication`;
}
if (connection) {
return `MCP server '${serverName}' reinitialized successfully`;
}
return `Failed to reinitialize MCP server '${serverName}'`;
};
const success = Boolean(
(connection && !oauthRequired) || (oauthRequired && oauthUrl) || (tools && tools.length > 0),
);
let failureReason;
if (!success) {
failureReason = oauthRequired
? MCP_REINITIALIZE_FAILURE_REASONS.OAUTH_REQUIRED
: MCP_REINITIALIZE_FAILURE_REASONS.INITIALIZATION_FAILED;
}
const result = {
availableTools,
success,
message: getResponseMessage(),
failureReason,
oauthRequired,
serverName,
oauthUrl,
oauthExpiresAt,
tools,
};
logger.debug('[MCP Reinitialize] Response ready', {
success: result.success,
oauthRequired: result.oauthRequired,
hasOauthUrl: Boolean(result.oauthUrl),
toolsCount: tools?.length ?? 0,
});
return result;
} catch (error) {
if (isMCPReauthenticationError(error)) {
throw error;
}
logger.error('[MCP Reinitialize] Error loading MCP tools; servers may still be initializing');
} finally {
if (connection && ephemeralServer && !requestScopedConnections) {
try {
await connection.dispose();
} catch {
logger.warn('[MCP Reinitialize] Failed to dispose ephemeral server');
}
}
}
}
module.exports = {
reinitMCPServer,
loadMCPServerCatalogs,
};