mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-21 15:33:45 +00:00
🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap (#15867)
* 🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap A server stored as an inspection-failed stub is recovered by the first request that inspects it successfully, but every request that read the stub before that write inspected again and wrote again. Each later write bumped `updatedAt` past the connection made after the earlier one, so the connection was replaced and a caller still holding it listed no tools. A request that reached `reinspectServer` after the write hit the "not in a failed state" guard and reported the server as still unreachable. - Share one in-flight reinspection per stored entry and effective allowlists in each process; DB entries are keyed per user and tenant. - Replace the stub only while it is still the inspected stub: a new optional `replaceStub` compare-and-set on the in-memory and Redis aggregate-key stores (one Lua script), so replicas never overwrite a recovery another replica stored. - Resolve a reinspection that finds the server already recovered to the stored config, and move the reinitialize decision into `MCPServersRegistry.recoverServerConfig` so `reinitMCPServer` connects with the recovered config instead of the stub it read. - Pin the interleavings in `MCPReinitRecovery.integration.test.ts` against a real in-process MCP server, and require every concurrent reinitialize flow there to recover with a complete tool list. * 🪢 fix: Key Shared Reinspections by the Stub They Inspect A registry re-initialization can replace a failed stub while an older reinspection of it is still running. The shared flight was keyed by entry and allowlists only, so a request that read the newer stub joined the older inspection and inherited its outcome without the newer config ever being inspected. `reinspectServer` now reads the entry before joining and keys the flight by the stub's `updatedAt`. A flight whose compare-and-set loses to a newer failed stub joins the inspection of that stub instead of reporting the server unreachable, and a store that leaves the inspected stub in place rejects rather than joining its own flight. * 🪢 fix: Settle a Reinspection Against the Entry Stored After It A reinspection that did not replace its stub only looked at storage when its compare-and-set lost. Two paths reached a stale outcome: - An inspection that failed rejected as unreachable even when another request had already stored a recovery, for example after capturing the stub before a slow allowlist resolution. Allowlists now resolve before the entry is read, and a failed inspection settles against the stored entry the same way a lost compare-and-set does. - Returning a recovery that another replica stored left this replica's process-local `getAllServerConfigs` memo serving the stub until its TTL. Settling on a stored recovery now invalidates this replica's registry read caches, as the writing path does. * 🪢 fix: Decide Reinspection Outcomes From Current Shared State The Redis aggregate-key store answers `get` from a process-local snapshot that can lag another replica's write by its TTL. A replica whose inspection failed read that snapshot while settling, still saw the stub, and reported the server unreachable although Redis held another replica's recovery. The same stale read also sent a replica into a needless inspection of a server another replica had already recovered. - Add an optional `getCurrent` to the repository interface, implemented by the aggregate-key store as a read past its snapshot, and use it for the reinspection's first read and for settling. - Inspect a newer stub within the flight that found it instead of joining another flight, so no flight ever waits on a second one. * 🪢 fix: Join the Flight Already Inspecting a Newer Stub A flight that lost to a newer failed stub inspected that stub itself even when another request was already inspecting it. If that redundant inspection failed transiently, the flight settled against the unchanged newer stub and rejected its callers while the other flight went on to store the recovery. A flight now joins the flight for the newer stub, starting it when none is pending, but only when the stub is strictly newer by `updatedAt`, so every wait points forward and flights stay acyclic. A replacement that is not newer, which only clock skew can produce, is still inspected within the flight. * 🪢 fix: Track Reinspection Waits Instead of Ordering Them by Clock Joining only flights for strictly newer stubs kept waits acyclic by relying on `updatedAt` order, which replicas with skewed clocks do not guarantee. A replacement stored with an older timestamp was inspected redundantly instead of joining the flight already inspecting it, and a transient failure of that redundant inspection rejected callers the other flight went on to recover. Settling flights now record the flight they wait on and join any in-flight reinspection for the stub they found, unless that flight already waits on them directly or through others. Only a join that would close a cycle falls back to inspecting within the settling flight. * 🪢 refactor: Resolve the Reinitialize Config in packages/api `reinitMCPServer` still decided in CJS what a failed-inspection config recovers to and built the `unreachable` response itself. That decision now lives in `resolveMCPReinitializeConfig` in packages/api, which takes the registry as a dependency and returns either the config to connect with or the typed result that ends the reinitialization. The CJS service only passes request data in and returns that result.
This commit is contained in:
parent
fb1322401d
commit
ddd2dd8bf4
14 changed files with 1435 additions and 122 deletions
|
|
@ -4,6 +4,7 @@ const {
|
|||
getUserMCPAuthMap,
|
||||
getMissingCustomUserVars,
|
||||
loadMCPServerCatalogs: loadCatalogs,
|
||||
resolveMCPReinitializeConfig,
|
||||
requiresEphemeralUserConnection,
|
||||
getMissingRuntimeBodyPlaceholderFields,
|
||||
MCPAuthenticationRejectedError,
|
||||
|
|
@ -37,7 +38,6 @@ const {
|
|||
} = require('~/server/services/MCPAuthorizationFenceRetry');
|
||||
|
||||
const MCP_REINITIALIZE_FAILURE_REASONS = {
|
||||
UNREACHABLE: 'unreachable',
|
||||
MISSING_CUSTOM_USER_VARS: 'missing_custom_user_vars',
|
||||
OAUTH_REQUIRED: 'oauth_required',
|
||||
INITIALIZATION_FAILED: 'initialization_failed',
|
||||
|
|
@ -166,45 +166,17 @@ async function reinitMCPServer({
|
|||
|
||||
try {
|
||||
const registry = getMCPServersRegistry();
|
||||
serverConfig =
|
||||
serverConfig ?? (await registry.getServerConfig(serverName, user?.id, configServers));
|
||||
ephemeralServer = serverConfig ? requiresEphemeralUserConnection(serverConfig) : false;
|
||||
if (serverConfig?.inspectionFailed) {
|
||||
if (serverConfig.source === 'config') {
|
||||
logger.info(
|
||||
'[MCP Reinitialize] Config-source server inspection failed; retry handled by config cache',
|
||||
);
|
||||
return {
|
||||
availableTools: null,
|
||||
success: false,
|
||||
message: `MCP server '${serverName}' is still unreachable`,
|
||||
failureReason: MCP_REINITIALIZE_FAILURE_REASONS.UNREACHABLE,
|
||||
oauthRequired: false,
|
||||
serverName,
|
||||
oauthUrl: null,
|
||||
tools: null,
|
||||
};
|
||||
} else {
|
||||
logger.info('[MCP Reinitialize] Server inspection failed; attempting reinspection');
|
||||
try {
|
||||
const storageLocation = serverConfig.source === 'user' ? 'DB' : 'CACHE';
|
||||
await registry.reinspectServer(serverName, storageLocation, user?.id);
|
||||
logger.info('[MCP Reinitialize] Server reinspection succeeded');
|
||||
} catch {
|
||||
logger.error('[MCP Reinitialize] Server reinspection failed');
|
||||
return {
|
||||
availableTools: null,
|
||||
success: false,
|
||||
message: `MCP server '${serverName}' is still unreachable`,
|
||||
failureReason: MCP_REINITIALIZE_FAILURE_REASONS.UNREACHABLE,
|
||||
oauthRequired: false,
|
||||
serverName,
|
||||
oauthUrl: null,
|
||||
tools: null,
|
||||
};
|
||||
}
|
||||
}
|
||||
const resolution = await resolveMCPReinitializeConfig(
|
||||
registry,
|
||||
serverName,
|
||||
serverConfig ?? (await registry.getServerConfig(serverName, user?.id, configServers)),
|
||||
user?.id,
|
||||
);
|
||||
if (resolution.result) {
|
||||
return resolution.result;
|
||||
}
|
||||
serverConfig = resolution.serverConfig;
|
||||
ephemeralServer = serverConfig ? requiresEphemeralUserConnection(serverConfig) : false;
|
||||
|
||||
const customUserVars = userMCPAuthMap?.[`${Constants.mcp_prefix}${serverName}`];
|
||||
|
||||
|
|
|
|||
|
|
@ -445,6 +445,58 @@ describe('reinitMCPServer — customUserVars gating (issue #10969)', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('reinitMCPServer — recovery of a server that failed inspection', () => {
|
||||
const user = { id: 'user-123' };
|
||||
const serverName = 'Recovering';
|
||||
const stub = {
|
||||
type: 'streamable-http',
|
||||
url: 'https://recovering.example.com/mcp',
|
||||
source: 'yaml',
|
||||
inspectionFailed: true,
|
||||
};
|
||||
const { getMCPServersRegistry } = require('~/config');
|
||||
|
||||
beforeEach(() => {
|
||||
mockUpdateMCPServerTools.mockResolvedValue({});
|
||||
});
|
||||
|
||||
it('connects with the recovered config instead of the stub it read', async () => {
|
||||
const recovered = {
|
||||
type: 'streamable-http',
|
||||
url: 'https://recovering.example.com/mcp',
|
||||
source: 'yaml',
|
||||
requiresOAuth: false,
|
||||
};
|
||||
const recoverServerConfig = jest.fn().mockResolvedValue(recovered);
|
||||
getMCPServersRegistry.mockReturnValueOnce({ recoverServerConfig });
|
||||
mockGetConnection.mockResolvedValue({ fetchTools: jest.fn().mockResolvedValue([]) });
|
||||
|
||||
const result = await reinitMCPServer({ user, serverName, serverConfig: stub });
|
||||
|
||||
expect(recoverServerConfig).toHaveBeenCalledWith(serverName, stub, user.id);
|
||||
expect(mockGetConnection).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ serverName, serverConfig: recovered }),
|
||||
);
|
||||
expect(result).toMatchObject({ success: true, serverName });
|
||||
});
|
||||
|
||||
it('reports the server unreachable without connecting while it cannot be recovered', async () => {
|
||||
const recoverServerConfig = jest.fn().mockResolvedValue(undefined);
|
||||
getMCPServersRegistry.mockReturnValueOnce({ recoverServerConfig });
|
||||
|
||||
const result = await reinitMCPServer({ user, serverName, serverConfig: stub });
|
||||
|
||||
expect(mockGetConnection).not.toHaveBeenCalled();
|
||||
expect(result).toMatchObject({
|
||||
availableTools: null,
|
||||
success: false,
|
||||
message: `MCP server '${serverName}' is still unreachable`,
|
||||
failureReason: 'unreachable',
|
||||
tools: null,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('reinitMCPServer — direct bearer authentication outcomes', () => {
|
||||
it('preserves a typed rejection instead of reducing it to a generic result', async () => {
|
||||
const { MCPAuthenticationRejectedError } = require('@librechat/api');
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue