LibreChat/api/server
Danny Avila ddd2dd8bf4
🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap (#15867)
* 🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap

A server stored as an inspection-failed stub is recovered by the first
request that inspects it successfully, but every request that read the
stub before that write inspected again and wrote again. Each later write
bumped `updatedAt` past the connection made after the earlier one, so the
connection was replaced and a caller still holding it listed no tools. A
request that reached `reinspectServer` after the write hit the
"not in a failed state" guard and reported the server as still
unreachable.

- Share one in-flight reinspection per stored entry and effective
  allowlists in each process; DB entries are keyed per user and tenant.
- Replace the stub only while it is still the inspected stub: a new
  optional `replaceStub` compare-and-set on the in-memory and Redis
  aggregate-key stores (one Lua script), so replicas never overwrite a
  recovery another replica stored.
- Resolve a reinspection that finds the server already recovered to the
  stored config, and move the reinitialize decision into
  `MCPServersRegistry.recoverServerConfig` so `reinitMCPServer` connects
  with the recovered config instead of the stub it read.
- Pin the interleavings in `MCPReinitRecovery.integration.test.ts` against
  a real in-process MCP server, and require every concurrent reinitialize
  flow there to recover with a complete tool list.

* 🪢 fix: Key Shared Reinspections by the Stub They Inspect

A registry re-initialization can replace a failed stub while an older
reinspection of it is still running. The shared flight was keyed by
entry and allowlists only, so a request that read the newer stub joined
the older inspection and inherited its outcome without the newer config
ever being inspected.

`reinspectServer` now reads the entry before joining and keys the flight
by the stub's `updatedAt`. A flight whose compare-and-set loses to a
newer failed stub joins the inspection of that stub instead of reporting
the server unreachable, and a store that leaves the inspected stub in
place rejects rather than joining its own flight.

* 🪢 fix: Settle a Reinspection Against the Entry Stored After It

A reinspection that did not replace its stub only looked at storage when
its compare-and-set lost. Two paths reached a stale outcome:

- An inspection that failed rejected as unreachable even when another
  request had already stored a recovery, for example after capturing the
  stub before a slow allowlist resolution. Allowlists now resolve before
  the entry is read, and a failed inspection settles against the stored
  entry the same way a lost compare-and-set does.
- Returning a recovery that another replica stored left this replica's
  process-local `getAllServerConfigs` memo serving the stub until its TTL.
  Settling on a stored recovery now invalidates this replica's registry
  read caches, as the writing path does.

* 🪢 fix: Decide Reinspection Outcomes From Current Shared State

The Redis aggregate-key store answers `get` from a process-local
snapshot that can lag another replica's write by its TTL. A replica
whose inspection failed read that snapshot while settling, still saw the
stub, and reported the server unreachable although Redis held another
replica's recovery. The same stale read also sent a replica into a
needless inspection of a server another replica had already recovered.

- Add an optional `getCurrent` to the repository interface, implemented
  by the aggregate-key store as a read past its snapshot, and use it for
  the reinspection's first read and for settling.
- Inspect a newer stub within the flight that found it instead of
  joining another flight, so no flight ever waits on a second one.

* 🪢 fix: Join the Flight Already Inspecting a Newer Stub

A flight that lost to a newer failed stub inspected that stub itself even
when another request was already inspecting it. If that redundant
inspection failed transiently, the flight settled against the unchanged
newer stub and rejected its callers while the other flight went on to
store the recovery.

A flight now joins the flight for the newer stub, starting it when none
is pending, but only when the stub is strictly newer by `updatedAt`, so
every wait points forward and flights stay acyclic. A replacement that
is not newer, which only clock skew can produce, is still inspected
within the flight.

* 🪢 fix: Track Reinspection Waits Instead of Ordering Them by Clock

Joining only flights for strictly newer stubs kept waits acyclic by
relying on `updatedAt` order, which replicas with skewed clocks do not
guarantee. A replacement stored with an older timestamp was inspected
redundantly instead of joining the flight already inspecting it, and a
transient failure of that redundant inspection rejected callers the
other flight went on to recover.

Settling flights now record the flight they wait on and join any
in-flight reinspection for the stub they found, unless that flight
already waits on them directly or through others. Only a join that
would close a cycle falls back to inspecting within the settling flight.

* 🪢 refactor: Resolve the Reinitialize Config in packages/api

`reinitMCPServer` still decided in CJS what a failed-inspection config
recovers to and built the `unreachable` response itself. That decision
now lives in `resolveMCPReinitializeConfig` in packages/api, which takes
the registry as a dependency and returns either the config to connect
with or the typed result that ends the reinitialization. The CJS service
only passes request data in and returns that result.
2026-09-12 19:50:14 -04:00
..
controllers 🔣 fix: Encode Images From Storage Keys (#15851) 2026-09-12 12:39:26 -04:00
middleware ⏳ fix: Separate Temporary and General Chat Retention (#15734) 2026-09-08 11:22:54 -04:00
routes 🔎 feat: Filter and Sort the Sidebar Chats List (#15842) 2026-09-12 15:56:10 -04:00
services 🪢 fix: Recover a Failed MCP Server Once When Reinspections Overlap (#15867) 2026-09-12 19:50:14 -04:00
utils 📿 fix: Inspect Stored File Locators Per Message (#15841) 2026-09-11 16:47:48 -04:00
cleanup.js
csp.spec.js 🔌 fix: Close Leaked HTTP Server in Server Specs (#15821) 2026-09-10 15:17:06 -04:00
experimental.js 🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784) 2026-09-09 15:48:11 -04:00
experimental.spec.js 🛎️ fix: Start Code Environment Reconciliation After Mongo Connects (#15620) 2026-09-04 19:08:44 -04:00
index.js 🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784) 2026-09-09 15:48:11 -04:00
index.metrics.spec.js 🔌 fix: Close Leaked HTTP Server in Server Specs (#15821) 2026-09-10 15:17:06 -04:00
index.spec.js 🔌 fix: Close Leaked HTTP Server in Server Specs (#15821) 2026-09-10 15:17:06 -04:00
socialLogins.js ⏳ feat: Make OpenID Token Reuse Window Configurable (#13546) 2026-06-06 15:15:58 -04:00
socialLogins.spec.js ⏳ feat: Make OpenID Token Reuse Window Configurable (#13546) 2026-06-06 15:15:58 -04:00
telemetry.js 🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565) 2026-09-03 21:16:09 -04:00
telemetry.spec.js 🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565) 2026-09-03 21:16:09 -04:00