Commit graph

2130 commits

Author SHA1 Message Date
Danny Avila
7347cfc195
🍡 feat: Batched User Questions With A Single Bounded Answer Form (#14737)
* feat: support batched user questions

* test: align batched question fixtures

* fix: harden batched question lifecycle

* test: submit batched HITL answers in e2e

* fix: address batched question review findings

* fix: preserve invoke return typing
2026-08-11 01:06:16 -04:00
Danny Avila
09cbd54f48
🪆 fix: Rebase Activity Phase Bounds over Sparse Content (#14729)
The aggregator writes content parts at provider-source indexes, which can
skip slots and leave holes in contentParts. Array.prototype.map preserves
those holes and the Map constructor iterates them as undefined, so
rebaseActivityPhaseBounds threw "Iterator value undefined is not an entry
object" at the end of every run with sparse content — deterministic with
parent phase summaries enabled, on both the completion and resume paths.

Build the identity map with an index loop that skips nullish slots. Holes
must stay out of the map: one undefined key would falsely match every hole
in previousParts as a retained part and corrupt the rebased bound.
2026-08-10 15:05:10 -04:00
Danny Avila
a3cec67e08
🪆 feat: Add Parent Activity Phase Summaries (#14721)
* feat: add activity phase summaries

* fix: preserve activity phase lifecycle semantics

* fix: satisfy activity phase type checks

* fix: simplify activity phase status mapping

* style: format activity phase changes

* fix: rebase activity phase bounds after shaping

* fix: link activity phase trace ancestry

* fix: reconcile activity phase bounds

* style: format activity phase reconciliation test

* style: align activity phase assertion

* fix: retain reasoning across commentary

* fix: preserve activity phase boundary state

* fix: detect renderable phase children

* test: type parallel phase assertion

* chore: bump agents SDK for activity phases

* fix: retain unphased lane reasoning

* fix: preserve tool group expansion across phases

* style: format phase expansion regression

* fix: preserve phase interaction state efficiently

* perf: skip sparse phase segment holes

* perf: partition phase segments with offsets

* fix: preserve phase boundaries and cursor state

* test: align activity phase regressions with CI

* test: keep phase context mock hoist-safe
2026-08-10 13:41:37 -04:00
Danny Avila
7fc62023eb
🧷 fix: Safely Recover Runtime MCP OAuth Rejections (#14684)
* fix runtime MCP OAuth recovery

* style: sort LC-008 imports

* fix: single-flight runtime OAuth handlers

* fix: retain transport OAuth failures for recovery

* fix(mcp): preserve OAuth recovery connections

* test(mcp): type request-scoped config fixture

* fix(mcp): harden shared OAuth recovery

* fix(mcp): bound OAuth recovery escalation

* style(mcp): sort OAuth integration imports

* fix(mcp): harden OAuth recovery boundaries

* fix(mcp): abort shared recovery waiters

* fix(mcp): bound request OAuth recovery phases

* fix(mcp): close OAuth recovery ownership gaps

* fix(mcp): retry borrowers closed by OAuth recovery

* fix(mcp): drain borrowers before OAuth reconnect

* fix(mcp): preserve eviction across OAuth recovery

* fix(mcp): unify OAuth recovery leases

* fix(mcp): serialize cache reuse with recovery

* fix(mcp): make recovery checkout atomic

* test(mcp): use numeric config timestamp

* fix(mcp): reacquire recovery checkouts

* fix(mcp): retain shared recovery disposal

* fix(mcp): restart checkout after recovery takeover

* fix(mcp): close recovery lifecycle gaps

* refactor(mcp): deepen OAuth recovery lifecycle

* fix(mcp): harden OAuth lifecycle disposal

* style(mcp): sort OAuth lifecycle imports

* fix: lease MCP OAuth lifecycle edges

* fix(mcp): isolate shared OAuth flows from aborts

---------

Co-authored-by: Dennis Schenk <dennis@gridonic.ch>
2026-08-10 10:38:34 -04:00
Dustin Healy
87a8b9aa12
📡 feat: Route Web Search and Scrape Egress through the SSRF-safe Agent (#14606)
* feat(web-search): route outbound search and scrape requests through the SSRF-safe agent

Build the SSRF-safe agents at the web-search tool-assembly site and pass them into the
search tool config so outbound search and scrape connections are validated at connect
time against their resolved IP, on every hop including redirects, consistent with the
other outbound clients.

Add allowedAddresses to webSearchSchema, reusing allowedAddressesSchema, so self-hosters
can permit a deliberately-private search or scrape endpoint (for example a private
SearXNG instance). The field is resolved directly from the webSearch config at the
createSearchTool call site, not through loadWebSearchAuth, because it is config and not
an auth credential. webSearchSchema is flat (providers are chosen by enums, not by
counting keys), so the field is inert with respect to provider selection.

Document the field and its operator warning in librechat.example.yaml, and assert the
wiring in handleTools.test.js: the SSRF-safe agents are threaded into the search tool
config, allowedAddresses is passed through when set, and omitting it still threads the
agents with no exemptions.

TODO awaits @librechat/agents release with the httpAgent hook: this consumes optional
httpAgent/httpsAgent fields on the search-tool config that are not yet in a published
@librechat/agents. package.json is intentionally left at the current version; bump it to
the release that ships the hook before this lands. Validated locally against a revendored
@librechat/agents build, not a published release.

* 🛡️ fix: Apply allowedAddresses to the Web Search SSRF Preflight

The connect-time SSRF agent already honors webSearch.allowedAddresses, but
loadWebSearchAuth ran the isSSRFUrl preflight without it, so an admin-permitted
private search or scrape URL was stripped before the agent could ever use it.

Thread allowedAddresses and the URL's effective port through isSSRFTarget and
resolveHostnameSSRF so the exemption is consistent across both SSRF layers.

* 🛡️ fix: Validate Web Search Destinations and Defer to Configured Proxies

Handing agents to createSearchTool covered only the connect-time DNS lookup,
which Node skips for IP-literal hosts, and a configured proxy connects on our
behalf without running that check. A literal private target such as
http://169.254.169.254 could therefore reach the network.

Route every resolved web-search destination through the existing
applySSRFSafeAgentIfDirect contract so a blocked literal target throws before
any request is made, and withhold the agents when a proxy owns egress, since one
agent pair is shared by every provider and a direct-connect agent on a proxied
connection would break the request while asserting protection the proxy's
network context cannot provide.

* 🛡️ fix: Keep Web Search SSRF Agents Under a Proxy and Restore Pooling

Withholding the agents whenever a proxy was configured removed protection from
every direct and NO_PROXY destination in exchange for preventing a failure that
cannot occur: for an https target Axios substitutes its own CONNECT tunnel, so
the injected agent is never used for the proxy connection. Only a plaintext http
target keeps our agent and repoints it at the proxy, and only a proxy whose
hostname resolves private then trips the connect-time check.

Always pass the agents and exempt the proxy endpoint instead, deriving host:port
from the same PROXY, HTTP_PROXY, and HTTPS_PROXY resolution the rest of LibreChat
uses so the proxy hop stays reachable while destinations remain guarded. Axios
already applies NO_PROXY per request, so bypassed routes keep enforcement with no
extra logic.

Drop the load-time destination validation. It duplicated the isSSRFTarget
preflight for user-provided URLs, rejected admin values that were previously
legal, and threw from inside loadTools, where both loader wrappers swallow the
error and drop every tool for the turn rather than degrading web search alone.

Build the agents with keepAlive and cache them per exemption list. A bare
http.Agent does not pool, so the previous code replaced the pooled global agents
for every search, scrape, and rerank call and allocated a fresh pair per turn.

* 🛡️ fix: Reject IP-Literal Private Targets on Web Search Connections

Node resolves nothing for a literal host, so the connect-time lookup never saw
one: a destination or a redirect target given as http://169.254.169.254 reached
the network. Redirect hops pass through the same createConnection, so checking the
literal there covers both cases and removes the need for a maxRedirects control
that createSearchTool cannot accept.

Gate it behind blockLiteralHosts so only web search opts in. A caller that
reaches a proxy or a deliberate private service by literal address must exempt it
first, and the merged consumers of createSSRFSafeAgents have no such exemption, so
enabling this everywhere would break configurations that work today.

* 🛡️ fix: Keep IPv6 Brackets on Derived Proxy Exemptions

The exemption parser accepts an IPv6 entry only as [ipv6]:port, so stripping the
brackets produced fd00::1:3128, which carries three colons and is dropped as
malformed. An IPv6 proxy therefore stayed unexempted and the connect-time check
rejected it, failing every web-search request routed through it.

Use the URL hostname as parsed, which already carries the brackets.

* 🛡️ fix: Exempt Proxies Configured Through ALL_PROXY

Axios resolves a proxy through proxy-from-env, which falls back to all_proxy in
either case after <protocol>_proxy, so ALL_PROXY on its own is enough to route a
request through a proxy. Exemptions were derived from PROXY, HTTP_PROXY, and
HTTPS_PROXY only, leaving such a proxy unexempted and rejected with ESSRF.

Derive the exemptions from the full set of variables that can put a proxy in
front of these requests instead. The installed proxy-from-env 2.1.0 reads no
npm_config variables, so those are deliberately not included.

* 🛡️ fix: Drop the Unearned PROXY Exemption and Harden the Web Search Guard

Nothing on this path consumes PROXY: Axios resolves proxies through
proxy-from-env, which reads only <protocol>_proxy and all_proxy, and web search
never calls applyAxiosProxyConfig. Exempting it therefore granted a bypass rather
than preserving a working route, and a user-settable search URL that redirects to
that address reached it and returned the body. Remove PROXY and proxy, and skip a
socks endpoint for the same reason, since Axios cannot proxy through one.

Tolerate a non-array allowedAddresses instead of spreading it, which threw out of
loadTools and dropped every tool for the turn. The YAML path is schema-validated
but the admin override path merges without parsing, so the value is reachable.

Separate cache keys with NUL rather than a newline, so an entry containing a
newline cannot collide with two separate entries, and bound the cache. Give the
agents the idle timeout the global agents carry, which keepAlive alone did not
restore. Reject a unix socket, which carries no host to validate. Also treat
fec0::/10 site-local as private, matching the fe80::/10 handling beside it.

Exercise the real resolver in handleTools.test.js rather than mocking it, so the
wiring test now fails if the agents it threads do not actually block a private
target.

* 🛡️ fix: Derive Proxy Exemptions Through Axios's Own Resolver

Unioning every populated proxy variable exempted addresses that never carry a
request. proxy-from-env picks a protocol-specific variable before all_proxy and
lowercase before uppercase, so an ignored value became a trusted host:port that a
redirect onto a direct route could reach. It also normalizes a scheme-less value
such as proxy.internal:3128 to an http URL, where parsing the raw string yielded
an empty hostname and no exemption at all, breaking the proxy hop.

Resolve through getProxyForUrl, the entry point Axios itself calls, so precedence,
scheme normalization, and NO_PROXY match exactly and cannot drift. NO_PROXY
covering everything now yields no exemption, since nothing is proxied. Declared
locally rather than adding a types package, alongside the existing declaration in
the same directory.

Also revert the fec0::/10 site-local change. domain.spec asserts that boundary
deliberately to prove the fe80::/10 mask does not over-reach, and the shared
address schema still classifies fec0 as public, so a runtime block there would
leave operators unable to configure the exemption. It belongs with those two
together, not in this PR.

* 🛡️ fix: Resolve Proxy Exemptions Against the Real Destinations

Resolving against placeholder probe hosts applied destination-specific NO_PROXY
rules to a host nobody dials. With NO_PROXY matching the probe domain but not a
real provider, no exemption was derived even though Axios still proxied the actual
request, so the agent rejected the private proxy hop with ESSRF.

Resolve per configured destination instead, passing the values loadWebSearchAuth
already resolved. Only plaintext http destinations are considered, since for an
https destination Axios substitutes its own CONNECT tunnel and never uses the
injected agent for the proxy connection, which is also why provider defaults need
no exemption: every one of them is https.

* 🛡️ fix: Accept Embedded-IPv4 IPv6 Forms in the Address Exemption Schema

The runtime guard blocks 6to4, NAT64, and Teredo addresses whose embedded IPv4 is
private, but the schema's local copy recognized only ULA, link-local, and the
dotted IPv4-mapped form, so an entry such as [64:ff9b::a00:1]:8080 was dropped as
a public literal. An operator reaching a private endpoint that way could not
configure the exemption at all.

Mirror hasPrivateEmbeddedIPv4 in the schema helper, which the surrounding comment
already asks to keep in sync. Public embedded addresses stay rejected, since an
exemption there has no defensive purpose.
2026-08-10 10:33:54 -04:00
Danny Avila
54d7f04d71
🪶 feat: Resolve Explicit Subagents Lazily (#14714)
* feat: resolve explicit subagents lazily

* fix: satisfy lazy subagent type checks

* test: persist lazy subagent mutation through model API

* style: format lazy subagent persistence test

* fix: log lazy subagent depth limit failures

* fix: harden lazy subagent resolution

* fix: Yield during lazy cancellation test

* test: Synchronize lazy cancellation setup

* style: Format lazy cancellation test
2026-08-09 19:23:45 -04:00
Marco Beretta
152dcf4721
🔗 feat: Shared Conversation Badge and Stable Share Links (#14712)
* feat: improve shared conversation links

* test: Cover Shared Link Lifecycle

* test: Cover Shared File Snapshots

* fix: address review findings on shared links

Stop double-decoding the conversation search term. Express already decodes
req.query, so the route's extra decodeURIComponent threw URIError on any term
containing a bare percent sign and mangled percent-escape-looking text. The
sidebar already sent the term raw, so this failed there too.

Advance a share's stored target to its branch tail when an update omits one.
Updating from the conversation list could not resolve the tail and reused the
stored target verbatim, silently republishing the same snapshot instead of the
turns added since.

Require revalidation on shared files. Updates now keep the shareId, so the file
URL no longer changes and a cached response could outlive a revoked share-files
choice; an ETag over the pinned snapshot fields keeps unchanged files on 304.

* fix: keep the shared badge across conversation cache replacements

isShared is derived per list request and absent from single-conversation
payloads, so rename, pin, and the SSE conversation updates dropped it when they
swapped a server response into the sidebar cache, hiding the badge until an
unrelated list refetch. Carry the cached value forward in updateConvoInAllQueries
so every replacing caller is covered, while an explicit value still wins.

* test: mock syncStaticTools in server boot specs

initializeMCPs now calls syncStaticTools when no MCP servers are configured, but the server boot specs stub ~/server/services/Config without it. Post-listen initialization threw, hit process.exit(1), and took the jest worker down until it exceeded the retry limit.

* fix: address codex findings on the shared DataTable and file ETag

Keep the published DataTable export bound to the legacy component and ship the design-system table as VirtualizedDataTable, so external consumers of @librechat/client keep the props they compile against.

Fold the snapshot's stored location into the shared-file ETag, so a re-published output that keeps its size and revision but moves its object no longer revalidates to a stale 304.

Auto-fill the table when a first page is too short to overflow its container, since pagination is otherwise only reachable through the scroll handler.

* fix: re-scope share grants before publishing and retry stalled auto-fill

Move the shared-link ACL expiration write ahead of the content update. The shareId survives an update, so a failed ACL write after the write-through left the new messages and file snapshot readable at the same URL while the owner saw a 500.

Retry a rejected auto-fill fetch up to three times: an unscrollable table has no scroll event to fall back on, and the sentinel alone would strand it on the first page.

* fix: follow regenerated branches and pin forks to the payload they saw

advanceTargetToBranchTail only walked descendants, so a target replaced by a regeneration (a sibling, not a child) left the update parked on the obsolete branch and published none of the turns that followed. A childless target now hops once to the newest sibling the conversation continued under.

A shareId survives an update, so an owner republishing between a viewer's load and their Continue click would resolve targetMessageIndex against different messages. The fork request now carries the payload's updatedAt and is rejected with 409 when it no longer matches; the viewer gets the current version pulled in and can retry.

* fix: keep table sorting and legacy backfills from breaking share flows

Restore the union formatting a local lint-staged prettier collapsed in data-provider types, which broke the CI lint run.

Header clicks now toggle direction instead of cycling through an unsorted state, which the controlled tables translated straight back into the default and made one direction unreachable.

Re-arm the auto-fill guard on a sort change: a re-sorted first page arrives with the same row count, and the guard would otherwise suppress paging on a container that still cannot scroll.

Lazy fileSnapshots backfills no longer touch updatedAt. That timestamp is the revision a viewer's fork is validated against, so a legacy share's first read would have made the Continue click that followed it fail with a 409.

* fix: break pagination ties by id and reset share state per conversation

Both list cursors marked a page boundary with values that repeat: conversations by (sort field, updatedAt) and shared links by the sort field alone. Imported chats share a title and a timestamp, so every row tied with the boundary was skipped. Both now carry the boundary row's _id and sort by it last, and the shared-links cursor is an opaque composite the route still validates before querying.

The share dialog outlives a switch between conversations, so a link with files disabled left the next conversation's dialog showing the switch off and quietly published without files. The stored choice now falls back to the enabled default, and a stale link no longer sits in the copy field.

* fix: keep titleless shared links in the paginated list

A share has no title default, and BSON orders a missing title before every string, so encoding the boundary as an empty string skipped the remaining untitled links when sorting Name ascending and re-admitted all of them descending.

The cursor now carries the boundary's null rather than flattening it, and because $lt and $gt are type-bracketed against a string, descending adds an explicit clause for the untitled tail that a string comparison can never reach.

* style: sort share method imports

* fix: fail closed on orphaned share targets and guard snapshot backfills

getMessagesUpToTarget walked levels from the roots and returned everything it had accumulated when the target was never reached. An imported or partially deleted branch whose parent is missing therefore published the whole conversation instead of the selected branch; the walk now returns nothing unless it actually reaches the target.

A lazy backfill wrote fileSnapshots unconditionally, so a viewer's first read of a legacy link could land after a republish and restore the snapshot it replaced, re-authorizing the stable URL of a file the owner had just removed. The write is now conditional on the link still having no snapshot, and the stored one wins any race.

Regenerating a message above the shared tail leaves the whole stored branch childless, so the target walk now climbs to the closest ancestor the conversation continued under instead of stopping at the stored tail's own siblings.

Changing the search or sort also returns the table's viewport to the top, since the query holds the previous rows while it refetches.

* fix: page through titleless rows on both sides of the cursor

The route validator still required a string primary, so the composite cursor the data layer issues for a titleless boundary came back as a 400 and the shared-links table stopped at that page.

Conversations had the same type-bracketing gap the shared links just closed: a name-sorted page could not reach conversations with no title, since a comparison against a string never matches a missing field. The cursor now carries the null and the filter spells out the titleless clauses for both directions.

* fix: keep the share badge read-only and refresh rows on cell changes

ensureLinkPermissions re-granted the owner ACL entry on every call, so rendering the header's shared badge turned ordinary navigation into a permission write. It now checks for the grant first and only migrates a link that still lacks one.

A fork carrying a positional target but no revision falls back to the whole share, since nothing proves which payload the index was counted against.

The memoized table row compared row data and selection only, so a cell rendering external state (the archived list's pending Restore, for one) kept its stale rendering until the row object itself moved; rows now also compare a marker that moves with the column definitions.

* fix: keep the shared badge honest when a delete fails or a link remains

A failed delete left the conversation looking unshared: the optimistic snapshot covered only the shared-link queries, not the conversation caches the badge reads. The cleared conversations are now restored with the rest.

A conversation can hold one link per target message, so clearing the badge on delete is a guess. The conversation list is invalidated once the mutation settles, letting the server decide from the links that are actually left.

* fix: refetch every cached conversation page after deleting a link

The invalidation was pinned to page zero, so a conversation cached further down the sidebar kept the badge the optimistic update had already cleared even when another targeted link survived.

* fix: treat a failed page fetch as a failed auto-fill

React Query resolves fetchNextPage with an error result instead of rejecting, so the rejection handler never ran: the guard stayed armed on the unchanged row count and an unscrollable table could never reach the next page.

* refactor: move the share request helpers into the typed backend

Cursor validation, page-size clamping and the shared-file cache validator were plain backend logic sitting in the legacy JS route. They now live in packages/api with their own tests, and the route keeps only the Express-side wiring: reading query params, mapping domain error codes to status codes, and writing the response.

Also carries the requested file choice into the cache entry the create and update mutations synthesize, since the response never echoes it and the dialog reads a resolved entry with no choice as the enabled default.

* fix: hold auto-fill while the replacement page is in flight

A search or sort swap keeps the previous rows and hasNextPage on screen while the new first page loads, so the re-armed auto-fill asked for page two against a query that was still fetching its first. An infinite query runs one fetch at a time, so that request could cancel or interfere with the one already out. Both tables now pass their fetching state and the guard waits for it.

* fix: stop advertising links a deployment no longer serves

The sidebar badge rendered from the derived flag alone, so a deployment that turned shared links off still told owners a link was live while the public routes were unregistered.

The share dialog called the link a snapshot, but the payload populates the referenced message documents on every request: an edit to an already-shared message is visible immediately, and Update only adds newly referenced ones. The copy now says that.

Scroll pagination inspects a resolved error the way auto-fill already does, since React Query reports a failed page that way instead of rejecting.

* a11y: gate the shared conversation label on the feature flag

The icon stopped rendering when a deployment turns shared links off, but the row still announced the conversation as shared to screen readers. Both now read the same condition.

* fix: accept long title cursors and stop badge work the feature disables

The cursor cap was tight enough that a Name-sorted page ending on a long title produced a nextCursor the next request rejected, stranding the rest of the list. It now sits well clear of anything the server can issue.

The conversation list skipped straight into the shared-link lookup even where ALLOW_SHARED_LINKS is off, paying a round trip on the sidebar's first page for a badge that is never rendered.

A regeneration is newer than what it replaced, so only a newer sibling counts: an older one that still has follow-ups is the branch the target was regenerated away from, and resuming there published turns the target had excluded.

* fix: hold scroll pagination while a replacement page loads

Resetting the viewport to the top after a search or sort change fires a scroll event, and the retained previous rows still report another page, so the handler asked for page two of a query that was still loading page one.

* fix: keep the legacy share migration ahead of the owner-grant shortcut

A legacy row keeps its marker until every grant it needs exists, so an owner grant on its own is not proof the migration finished. Reading the marker first means a half-migrated public link still gets its public grant, while a fully migrated one keeps the read-only settled path the badge lookup depends on.
2026-08-09 08:14:54 -04:00
Danny Avila
5c939d129b
🔌 feat: Add Agent Plugins (Experimental) (#14704)
* 🔌 feat: Add Agent Plugins v1.0.0 Support

Implements the Agent Plugins 1.0.0 specification so LibreChat can load
portable plugin packages: a `plugin.json` manifest, `skills/` holding Agent
Skills, `mcp.json` describing MCP servers, and reverse-domain extension
directories.

- Validate the closed `plugin.json` schema, selecting rules from `$schema`
  without retrieving it. Unknown top-level fields and a non-object
  `extensions` field are reported and ignored; every other violation rejects
  the plugin.
- Enforce plugin-root containment through realpath, including for paths whose
  leaf does not exist, and apply the narrowest failure boundary per component.
- Map `mcp.json` onto LibreChat MCP options across stdio, Streamable HTTP, and
  legacy HTTP+SSE, bypassing the config loader's `${VAR}` process-env
  expansion so plugin values never resolve against the server environment.
- Expand only `${PLUGIN_ROOT}` and `${PLUGIN_DATA}`, once and non-recursively,
  in `args`, `env` values, and `cwd`; supply both variables to the subprocess
  after configured `env`, and reject entries that declare them.
- Discover skills from the immediate children of `skills/` only, reusing the
  deployment skill loader so plugin skills are ordinary deployment skills with
  a distinct id namespace.
- Read LibreChat's `ai.librechat` extension directory and hand
  `hooks/hooks.json` to the Claude hook compatibility layer.
- Load operator-installed plugins from `DEPLOYMENT_PLUGINS_DIR` at startup,
  merging their skills into the deployment skill registry and their MCP
  servers into the app config. Plugins never displace a configured server or
  deployment skill.
- Add `cwd` to the stdio MCP transport, which the specification requires and
  LibreChat did not previously support.

Component failures stay isolated: a malformed `mcp.json`, an invalid skill, or
a bad hooks document never prevents the rest of a plugin from loading.

* 🔒 fix: Contain Agent Plugins config at the runtime boundary

Review of #14704 surfaced that every real finding sat where the loader's
output crosses into LibreChat's existing runtime, not in the specification
logic. The loader deliberately left plugin placeholders literal, but
downstream layers re-processed the same fields and undid it.

- Mark plugin MCP configuration with `source: 'plugin'` and return it verbatim
  from `processMCPEnv`. Without this a remote plugin could declare
  `Authorization: Bearer ${OPENAI_API_KEY}` and receive host credentials at its
  own origin. The gate reads the configuration rather than a caller-supplied
  flag, so no future call site can reintroduce the leak by omitting it.
- Skip `preProcessGraphTokens` for plugin configuration as well; it resolves
  placeholders into headers, url, and args on the same path.
- Reject plugin server names that change under `normalizeServerName`. Tool keys
  embed the normalized name while request-time resolution uses the raw name, so
  an unstable name published tools that nothing could resolve.
- Reject `__proto__`, `constructor`, and `prototype` as server names, and merge
  plugin servers with `Object.defineProperty` and an own-property conflict
  check, so a package cannot reach a prototype setter or collide with an
  inherited member.
- Enforce manifest-name uniqueness before components are accepted; two packages
  sharing a name would share one `PLUGIN_DATA` directory.
- Isolate a failed data-directory creation to the single plugin instead of
  rejecting the whole scan.
- Prefix rejected-plugin diagnostics with the directory, which is the only
  identifier a package without a valid manifest has.
- Type extension namespace contents as JSON rather than `unknown`, and correct
  the header field-value comment to name obs-text.

Verified end to end from the built package: a plugin declaring an environment
placeholder in a header reaches the transport with the placeholder intact while
operator-authored configuration still resolves normally.

* 🔇 fix: Report Agent Plugin hooks that will not run

The loader reads `ai.librechat/hooks/hooks.json`, but nothing registers the
resulting plan, and startup supplies no hook capabilities. A package declaring
hooks was therefore accepted in silence, leaving an operator to believe the
hooks ran.

Detect the document when no capabilities are registered and report it as
unsupported, so the limitation is visible in startup diagnostics rather than
inferred from behavior that never happens.

* 🧯 test: Restore MCP startup test mocks

Carries the two mock additions from #14711 so this branch can prove itself
green. `initializeMCPs` now calls `syncStaticTools`, which the server startup
specs do not stub, so they fail on every branch that has not picked this up.
Drops out of the rebase once #14711 lands.

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-09 08:10:22 -04:00
Danny Avila
1bd4455c2d
🧭 fix: Make MCP Catalog Redis Cluster-Safe (#14717)
* fix: make MCP catalog Redis startup cluster-safe

* fix: stabilize Redis readiness gate

* fix: type Redis readiness export

* style: apply canonical import order
2026-08-09 06:59:29 -04:00
Danny Avila
ccb43bf32b
test(mcp): mock static tool startup sync (#14711) 2026-08-08 23:12:07 -04:00
Danny Avila
1bccc2bc18
📡 fix: Refresh MCP Tools After List-Changed Notifications (#14686)
* fix(mcp): handle dynamic tool list changes

Co-authored-by: Pascal Garber <pascal@artandcode.studio>

* test(mcp): fix CI validation

* fix(mcp): keep dynamic tool catalogs live

* fix(mcp): harden dynamic catalog lifecycle

* test(mcp): use typed startup connection

* test(mcp): isolate dynamic e2e fixtures

* fix(mcp): refresh tools after reconnect

* fix(mcp): close dynamic catalog cache gaps

* test(mcp): update OAuth connection mocks

* fix(mcp): preserve app snapshot ownership

* style(mcp): sort connection imports

* fix(mcp): close review race conditions

* fix(mcp): preserve cache ownership edges

* fix(mcp): harden recovery lifecycle

* fix(mcp): guard tool-less app refresh

* fix(mcp): fence distributed cache races

* fix(mcp): retire stale connection state

* fix(mcp): keep tool snapshots authoritative

* fix(mcp): fence stale app tool publications

* style(mcp): sort repository test imports

* test(mcp): mock empty startup publication

* fix(mcp): preserve app publication generations

* fix(mcp): harden publication recovery races

* fix(mcp): address tool catalogs by runtime config

* fix(mcp): load scoped catalogs for assistant writes

* fix(mcp): harden catalog publication recovery

* fix(mcp): serialize forced connection replacement

* fix(mcp): serialize ordinary creation with replacements

* fix(mcp): harden catalog fallback boundaries

* fix(mcp): close lifecycle fencing gaps

* fix(mcp): preserve catalog authority on failures

* fix(mcp): compensate failed catalog mutations

* fix(mcp): fence catalog refresh ordering

* style(mcp): sort agent loader imports

* fix(mcp): cancel stale connection creation

* fix(mcp): fence catalog coordination

* fix(mcp): close catalog race windows

* fix(mcp): harden cross-pod catalog fencing

* fix(mcp): close catalog lifecycle edges

* style(mcp): sort assistant imports

* fix(mcp): reject stale recovery authority

* fix(mcp): restore static catalog on every startup

* fix(mcp): order app catalog publications

* style(mcp): sort catalog revision imports

* fix(mcp): separate catalog allocation and commit fences

---------

Co-authored-by: Pascal Garber <pascal@artandcode.studio>
2026-08-08 13:50:21 -04:00
Danny Avila
ef38f362ec
📦 chore: bump @librechat/agents to v3.4.2 and npm audit (#14702)
* 📦 chore: bump `@librechat/agents` to version 3.4.2

* 📦 chore: bump `mermaid` to version 11.16.1 and update related dependencies

* 📦 chore: bump `js-yaml` to version 4.3.1 in package-lock and data-provider

* 📦 chore: bump `nanoid` to version 3.3.18 in package.json and package-lock.json across multiple packages

* 🔧 fix: Remove stray `api/tsconfig.json` breaking e2e `~` alias

An empty `api/tsconfig.json` was accidentally committed with the agents bump.
Playwright's require hook resolves path aliases from the nearest path-config,
checking `tsconfig.json` before `jsconfig.json` in each folder, so the empty
file shadowed `api/jsconfig.json` — the only place `"~/*": ["./*"]` is defined.

Every e2e spec that calls `cleanupUser` then failed on
`Cannot find module '~/cache/getLogStores'` from `api/models/index.js`.

- delete the stray file and gitignore it so tooling can't re-commit it
- register `module-alias` in `cleanupUser` so backend requires resolve
  regardless of which path-config Playwright happens to find

* 📦 chore: bump `@librechat/agents` to version 3.4.3 in package.json and package-lock.json
2026-08-08 12:24:06 -04:00
Marco Beretta
39f5f9d846
perf: Agent List and Model Selector at Scale (#14601)
* perf: cut serial round trips from the agent list query path

The agent list was the slowest path on first page load. Three separate
problems compounded:

- `getListAgentsHandler` chained its reads: two ACL lookups, the avatar
  refresh cache probe and the viewer skill scope all resolved serially
  ahead of the list query, and `attachOwnerContacts` added two more hops
  after it. The four independent reads now resolve together, and the
  avatar refresh runs alongside the list query instead of before it -
  refreshed paths reach the response through `urlCache`, not through
  whatever the list query happened to read. Serial hops per request drop
  from 7 to 4 on a warm cache.

- The avatar refresh loaded the user's whole accessible agent set (up to
  MAX_AVATAR_REFRESH_AGENTS) to discover which entries were S3-backed.
  Scoping the query to `avatar.source` means deployments on any other
  file strategy match nothing instead of walking the full set.

- `fetchAllAgentPages` walked cursor pages at the server's default size
  of 100, and callers consume the flattened result, so every extra page
  was a serial round trip for no benefit. It now requests the server
  maximum. Measured over a 2,860 agent account: 29 requests / 1.65s
  before, 3 requests / 0.29s after.

Also parallelizes the conversation file reads in `initializeAgent`. The
convo file refs and the execute_code thread walk share no inputs, and the
two code-file lookups depend only on `threadFileIds`, so the chain of six
serial reads on every turn collapses to two. This one is time to first
token the user waits through.

* perf: virtualize the model selector agent list

Opening the agents submenu with a large agent set froze the tab and could
kill it outright. With ~10k accessible agents the submenu blocked for over
15 seconds and took the heap from 96MB to 911MB. Four per-row costs were
being multiplied by the full list, which rendered unwindowed:

- `useIsActiveItem` allocated a MutationObserver per row (10,016 of them
  for one dropdown). Replaced with an Ariakit store subscription, which
  needs no observer at all and returns a boolean so a row only re-renders
  when its own active state flips.

- `useFavorites` ran per row, opening a jotai subscription, a query
  subscription and a mutation each time. Hoisted to one call per endpoint.

- Each row rescanned `endpoint.models` to recover `isGlobal`, a field the
  parent had already discarded from the array it was mapping. The parent
  now passes it down from a lookup map.

- The list itself is now windowed above 100 rows. Ariakit's composite only
  knows about mounted rows, so arrow-keying to the window edge previously
  found no next item and let focus escape the nested menu, closing it;
  `handleBoundaryNavigation` scrolls the next index in, waits for it to
  mount, then moves the composite onto it. Navigation inside the window is
  left to Ariakit.

Open drops from >15s to 96ms, mounted rows from 10,028 to ~18, DOM nodes
from 123,346 to ~1,000, and the heap no longer grows. Verified in browser:
arrow keys track 1:1 to index 238 and back, and click selection works.

* perf: serve the model selector from the shared VIEW agent query

The model selector asked for EDIT-scoped agents whenever the marketplace
is enabled, while `useAgentsMap` and `useMentions` asked for VIEW. Since
the cache key includes the params, that was two distinct entries, so first
page load ran the paginated walk twice and held two copies of the whole
agent list in memory. Measured against a 10k agent account: 22 list handler
invocations per page load, now 11.

Collapsing the two by asking for the same permission everywhere would have
changed what the selector shows - under the marketplace the EDIT scope is
what makes it "My Agents", with discovery handled by the marketplace entry.
So the list endpoint now marks each row with `isEditable`, resolved from an
ACL read folded into the existing parallel batch (no extra serial hop), and
the selector filters the shared VIEW response instead of refetching. A
VIEW-scoped list for a user with 2861 visible / 361 editable agents returns
exactly 360 rows flagged editable, matching what the EDIT query returned.

`AgentSelect` deliberately keeps its own EDIT query: it reads `skills` and
`skills_enabled`, which `sanitizeViewerSkillScope` strips from VIEW-scoped
responses. It also only mounts when the builder panel is open, so it is not
part of the first-load cost.

The field is set unconditionally rather than omitted when false so that a
client talking to an older server sees `undefined`, keeps every agent, and
degrades to showing too many rather than none.

* fix: address review findings on the agent list at scale

Three issues from review, all confirmed against the code before fixing.

Avatar refresh no longer runs alongside the list query. `updateAgent` writes
through `findOneAndUpdate` on a `timestamps: true` schema, so refreshing an
avatar advances `updatedAt` — the field `getListAgentsByAccess` sorts and
cursors on. A write landing after the first page's snapshot moved that agent
ahead of the returned cursor, dropping it from every later page and silently
truncating the caller's flattened list. This was a regression introduced when
the two were parallelized; serializing them costs nothing on the common path,
because a cache hit returns without issuing any query, so only the
once-per-30-minutes miss pays for the ordering. The new test asserts the write
lands before the list snapshot and fails against the parallel version.

The virtualized list no longer inserts a focusable grid into the combobox.
`List` spreads its props onto `Grid`, whose defaults are `role="grid"`,
`containerRole="row"` and `tabIndex={0}`; inside Ariakit's listbox that added a
tab stop ahead of any row and put grid/row semantics between the listbox and its
options. All three are now neutralized so focus and ARIA stay with the combobox
items.

The list also resets to the top when the filter changes. `Grid` keeps its scroll
offset across prop changes and clamps an out-of-range offset to
`totalRowsHeight - height`, the end of the shorter list. Scrolling deep and then
searching landed on the tail: measured at row 626 of 667 matches, with only
those rows mounted and reachable by keyboard. Keying the list on the search
value restores row 0.

* fix: declare option position and set size for the virtualized model list

Once the model list is windowed, only the mounted slice exists in the listbox,
so a screen reader infers position and total from ~19 elements instead of the
real set — announcing "3 of 19" partway through 10,014 agents.

Model rows now carry aria-posinset and aria-setsize. The marketplace entry and
any model specs share the same numbering, because they are options in the same
listbox: declaring the values on some options while leaving others to be
inferred from the DOM would make the set internally inconsistent. Both are
omitted entirely when the list is short enough to render unwindowed, where the
DOM holds every option and the implicit values are already correct.

Verified against a 10,014 agent account: the marketplace entry reports 1 of
10015, the first models 2 and 3, and after scrolling to row 4999 the leading
mounted model reports 5001 of 10015 with 19 options in the DOM.

* 🩹 fix: Address Follow-Ups on the Agent List at Scale

Corrects residual issues in the agent-list perf work, all inside its own scope.

- Forward `idOnTheSource` through `PermissionService.findAccessibleResources`
  so `getUserPrincipals` skips the user-document read. The list handler resolves
  three permission sets per request and each was paying its own `User.findById`;
  the auth strategies already normalize the field to a value or null.
- Gate the editable-set lookup on its own predicate instead of borrowing
  `canReturnSkillConfig`. The two answer unrelated questions and only coincide
  today, so redefining the skill flag would have marked every agent editable.
- Log mapping failures in the list response instead of swallowing them.
- Apply the walk page size after the caller's params in `fetchAllAgentPages`.
  A caller limit only changed page size, never what the flattened walk returned,
  so `defaultAgentParams`' `limit: 10` would have turned one request into 301.
- Carry `isEditable` on the agent rows the create and update mutations write
  into the list cache. Mutation responses omit the field, so those rows lost it.
- Document `isEditable` as list-only, ACL-derived, and fail-open on absence.
- Restore the truthiness guard on the thread walk in `initializeAgent`. Widening
  it to `!= null` made an empty `parentMessageId` issue a full-conversation read
  against an anchor that can never match.
- Await `getConvoFiles` directly rather than calling `.then()` on it, restoring
  tolerance for synchronous test doubles.
- Correct the avatar-refresh comment: the projection was never full documents,
  and the real reason to filter is that an unfiltered budget is self-reinforcing.

Tests: both new `initialize` tests and both new backend tests are
mutation-verified; the concurrency test fails under either serialization order.

* fix: preserve ACL isEditable when merging agent mutation responses

Mutation responses omit list-only isEditable. Inferring true from write
success promoted VIEW-only rows into the editable subset for MANAGE_AGENTS
callers who can PATCH agents their ACL marks non-editable.

* fix: sort imports in agent mutations test

ESLint import-order check failed on the isEditable cache-preservation test.

* 🧷 fix: Carry isEditable Onto Duplicated Agent List Rows

`useDuplicateAgentMutation` prepended the raw duplicate response to the cached
list, and mutation responses omit the list-only `isEditable` field. The row
survived the "My Agents" filter only by failing open on `undefined`, so it would
disappear the moment a consumer read the flag strictly.

Duplicating grants the caller ownership, so the new row is editable outright;
this is the create case rather than the merge case `mergeAgentListRow` handles.
Last cache write on this path that did not carry the field.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-07 21:04:55 -04:00
Danny Avila
1596df724a
🫆 chore: Remove Published Credential Defaults (#14680) 2026-08-07 07:25:05 -04:00
Danny Avila
5ff46d8c67
🛟 fix: Stop Agents When Code Resources Cannot Recover (#14651)
* fix: Block Agents When Code Resources Cannot Recover

* fix: Preserve Resource Recovery Failures Across Agent Paths

* fix: Centralize Fatal Agent Initialization

* chore: sort agent imports
2026-08-07 00:33:28 -04:00
Dustin Healy
6d2f29266c
🔑 feat: Refresh-Capable Google Admin OAuth Sessions (#13832)
* 🔑 feat: Refresh-Capable Google Admin OAuth Sessions

Google admin sessions cannot be refreshed today. Three gaps add up to that:
passport.authenticate('googleAdmin', ...) in api/server/routes/admin/auth.js
never sets access_type=offline, so Google omits the refresh_token from its
token response; createOAuthHandler in api/server/controllers/auth/oauth.js
only forwards a refresh token into the admin exchange payload when the user's
provider is 'openid' AND OPENID_REUSE_TOKENS is enabled; and
/api/admin/oauth/refresh is openid-only, calling openid-client.refreshTokenGrant
against the configured OIDC issuer. OpenID admins refresh transparently
because all three are in place for them.

This PR closes all three. The googleAdmin authenticate call now passes
accessType: 'offline' and prompt: 'consent' so Google issues a refresh token
on consent; the chat-side googleLogin is untouched. The shared socialLogin
verify callback now passes the IdP refreshToken through as passport's third
argument (info), landing on req.authInfo, with the two-argument call shape
preserved when no refresh token is present so existing strategy tests stay
valid. createOAuthHandler reads req.authInfo?.refreshToken for non-OpenID
admin providers and forwards it into the exchange code; the OpenID branch
and its OPENID_REUSE_TOKENS gate are unchanged. /api/admin/oauth/refresh
now accepts an optional provider field ('openid' | 'google', default 'openid').
The new Google branch POSTs grant_type=refresh_token to
https://oauth2.googleapis.com/token, decodes the returned id_token for the sub
claim, looks up the admin user by googleId, enforces tenant scope and
ACCESS_ADMIN, and mints a fresh LibreChat JWT in the same response shape
/oauth/exchange returns. It is gated on GOOGLE_CLIENT_ID and
GOOGLE_CLIENT_SECRET being set (returns 503 GOOGLE_NOT_CONFIGURED otherwise);
unknown provider values return 400 INVALID_PROVIDER.

* 🔁 fix: Harden Google admin refresh against bot review findings

Five validated findings from the initial bot pass:

socialLogin.js: mirror the OpenID migrate-or-reject pattern on the email
fallback. When an existing user is found by email and the stored provider
id is empty, persist the refreshed sub so the refresh path can later bind
to it. When the stored id is present and differs, reject as AUTH_FAILED
to prevent identity-swap, matching the existing OpenID behavior in
packages/api/src/auth/openid.ts.

oauth.js: scope the non-OpenID admin refresh-token forwarding to
provider === 'google'. The previous else branch would have forwarded a
Discord refresh token (passport-discord supplies one) into the admin
exchange payload even though /api/admin/oauth/refresh only accepts
openid or google, leaving the admin client with a token it could not
refresh.

admin/auth.js (refreshGoogleAdminSession): drop id_token from the
mandatory-fields check. Google's OAuth refresh response is documented to
include id_token only conditionally, so the previous mandatory check
broke refresh whenever Google omitted it. Decode id_token when present
(fast path); when absent, call Google's userinfo endpoint with the
access token to read sub. Wrap tokenResponse.json() in try/catch and
return IDP_INCOMPLETE on parse failure instead of a generic 500.
Tighten access_token to a typeof string check.

admin/auth.js (refreshGoogleAdminSession): reuse serializeUserForExchange
for the response user so the Google refresh shape matches /oauth/exchange
and the OpenID branch exactly (full _id, id, email, name, username, role,
avatar, provider, openidId). The previous Google-specific subset dropped
fields the admin client relies on for later provider-specific refreshes
and disambiguation.

Tests cover each fix: socialLogin's migration and rejection cases, the
oauth.js Discord-gating case, the userinfo fallback path on missing
id_token, CLAIMS_INCOMPLETE when both id_token and userinfo are absent,
IDP_INCOMPLETE on a non-JSON token body, and the full response shape on
the happy path.

* 🧪 fix: Add updateUser to appleStrategy test mock for socialLogin migration

The shared socialLogin verify callback now invokes `updateUser` when the
email-fallback path discovers a same-provider user with an empty provider
id, persisting the refreshed sub. The Apple strategy test's `~/models`
mock did not stub `updateUser`, so the migration path hit
`TypeError: updateUser is not a function` and failed the
`should handle existing user and update avatarUrl` case in CI shard 1/3.

* 🧹 refactor: Move Google admin refresh into TypeScript @librechat/api helper

Per repo guidance (CLAUDE.md): all new backend code must be TypeScript in
/packages/api, and /api is a thin JS wrapper. The previous commit landed the
Google admin refresh flow as ~120 lines of new JS inside
api/server/routes/admin/auth.js, which violates that. This commit extracts
the flow into a new TS helper at packages/api/src/auth/googleRefresh.ts and
reduces the route handler to a thin dep-wiring wrapper.

The helper exports applyGoogleAdminRefresh(deps, options) with the same
shape as the OpenID applyAdminRefresh: callers pass findUsers, getUserById,
canAccessAdmin, and mintToken as deps so the package stays free of /api
model imports and capability/session helpers. The route handler now builds
those deps from the existing model + capability + token modules and calls
the helper, mapping AdminRefreshError to the documented HTTP responses.

While moving the code, the helper now guards getUserById with
Types.ObjectId.isValid before the direct-lookup branch, matching the
OpenID admin path at packages/api/src/auth/refresh.ts. Without this guard
a malformed user_id from the admin client would hit Mongoose findById's
CastError and surface as a 500 INTERNAL_ERROR instead of falling through
to the documented sub-based lookup.

Tests move with the code: packages/api/src/auth/googleRefresh.spec.ts now
owns the helper's behavior (token endpoint, userinfo fallback, ObjectId
guard, USER_ID_MISMATCH/TENANT_MISMATCH/USER_NOT_FOUND/FORBIDDEN, rotated
refresh-token pass-through, GOOGLE_NOT_CONFIGURED, IDP_INCOMPLETE on
non-JSON body, CLAIMS_INCOMPLETE when both id_token and userinfo miss).
The route-level api/server/routes/admin/auth.refresh.test.js drops the
duplicated end-to-end Google cases and keeps a smaller surface: route
delegates to applyGoogleAdminRefresh with the right deps + options, maps
AdminRefreshError to HTTP status/code, falls through to 500 for unknown
errors, and rejects unknown providers with INVALID_PROVIDER.

* 🔁 fix: Tighten Google admin refresh and limit social-login changes

Brutal-review findings on top of the upstream feature work.

socialLogin.js: the migrate-or-reject pattern from the previous commit
applied to every provider's chat-side verify callback, not just the admin
flow. Gate both branches on `options.existingUsersOnly` so the chat-side
googleLogin / facebookLogin / etc. keep their pre-existing email-fallback
behavior unchanged. Tests follow: restore the original `should fallback to
finding user by email` chat-side case and re-add the migration and
mismatch-reject cases as admin-only by passing `{ existingUsersOnly: true }`
to socialLogin in those tests.

googleRefresh.ts: add a defense-in-depth `isEmailAllowed(user)` dep that
the helper invokes before `canAccessAdmin`. Mirrors the
`isEmailDomainAllowed` check the initial Google admin login already runs,
so a deployment that removes a domain from `registration.allowedDomains`
after issuance can no longer mint fresh JWTs for that admin via refresh.
The route handler wires it up with `resolveAppConfigForUser` +
`isEmailDomainAllowed`, falling back to `baseOnly` config for users
without a tenantId.

googleRefresh.ts: drop the unreachable `?? ''` defensive coalescing in
`fetchGoogleTokenset`. The `GOOGLE_NOT_CONFIGURED` guard upstream already
narrows `clientId`/`clientSecret` to non-empty strings; the function
takes a narrowed `GoogleAdminRefreshConfiguredOptions` shape and
`applyGoogleAdminRefresh` constructs that shape after the guard.

* 🔒 fix: Apply brutal-review hardening to Google admin refresh

Tighten the Google OAuth refresh flow against all outstanding code review
findings: enforce JWT aud claim verification against the configured clientId
(ISSUER_MISMATCH on mismatch), reject ambiguous googleId matches (limit:2 in
findUsers, USER_ID_MISMATCH when multiple rows match), scope the authInfo
refresh-token carrier to the Google provider only, add TOCTOU re-read defense
after the admin googleId migration write in socialLogin, deduplicate
canAccessAdmin/mintToken closures via buildAdminRefreshClosures shared by both
OpenID and Google refresh paths, document rotation semantics on
AdminExchangeResponse.refreshToken, standardise all log prefixes to
[admin/oauth/refresh], and expand test coverage for all new paths.

* 🔒 fix: Reject refresh for users migrated off the Google provider

The interactive Google admin login path in socialLogin.js already rejects
a user whose provider field is not 'google', returning AUTH_FAILED. Without
a matching guard in the refresh path, a user migrated to OpenID could use
an unexpired Google refresh token to keep minting admin JWTs indefinitely.

Add a PROVIDER_MISMATCH check after resolving the user in both the direct
getUserById branch and the findUsers fallback branch of resolveAdminUser,
mirroring the provider gate the interactive path enforces.

* 🔒 fix: Add ban check and fix domain allowlist on admin OAuth refresh

Two gaps in the /api/admin/oauth/refresh route:

Add middleware.checkBan to the route chain before preAuthTenantMiddleware,
matching the gate that /login/local and createOAuthHandler already apply.
Without it a banned admin could keep minting JWTs until their IdP refresh
token expired.

Replace getAppConfig({ baseOnly: true }) in the non-tenant isEmailAllowed
closure with getAppConfig({ role: user.role }), which includes DB-layer
overrides from the admin panel. baseOnly returns only YAML-derived config,
so any allowedDomains list maintained entirely through the admin panel was
silently inert on this path. Extract isEmailAllowedForUser as a shared
helper, move it into buildAdminRefreshClosures so both Google and OpenID
refresh paths enforce domain policy consistently, and add isEmailAllowed
to AdminRefreshDeps in the TS package so applyAdminRefresh can invoke it.

* 🔒 fix: Harden admin OAuth refresh against user bans, tenant scope gaps, and cross-tenant migration

Post-identity-resolution ban check: the initial checkBan middleware fires before the
refresh token is exchanged and req.user is populated, so it can only evaluate IP bans.
After applyGoogleAdminRefresh/applyAdminRefresh resolves the user identity, we now
synthesize req.user and re-run checkBan against the resolved user's id before emitting
the JWT, so a user-level ban is enforced even from a fresh IP.

Domain allowlist now includes userId: the getAppConfig call in isEmailAllowedForUser
was passing only role, missing user and group-level allowedDomains overrides that the
initial OAuth callback's checkDomainAllowed enforces via userId. Both branches now
pass userId so buildPrincipals takes the full user+group+role resolution path. The
tenant branch is also inlined (replacing resolveAppConfigForUser) to accept userId,
wrapped in tenantStorage.run for correct Mongoose scoping and cache-key resolution.

Cross-tenant email-fallback migration: the Passport verify callback fires before
tenantContextMiddleware, so findUser({email}) is unscoped and can return a same-email
user from another tenant. Writing googleId onto that document permanently corrupts
the other tenant's account. Migration is now blocked for users with a tenantId;
single-tenant users are unaffected.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-07 00:10:34 -04:00
Dustin Healy
c6bb77325a
🔗 feat: Admin Panel Link in Settings for Admins (#14662)
Expose ADMIN_PANEL_URL through the startup config for users holding the
access:admin capability, and render an Admin section in Settings > General
with an external link to the admin panel. The URL is omitted server-side
for unauthenticated requests and users without admin access.
2026-08-06 12:40:44 -04:00
Dustin Healy
96404bfc73
🛡️ fix: SSRF-Guard Speech (STT/TTS) and OCR Outbound Requests at Connect Time (#14560)
* fix: SSRF-guard speech (STT/TTS) and OCR outbound requests at connect time

Speech (STT/TTS) and OCR issued outbound HTTP to operator-provided target URLs
with only proxy config attached, so a target that resolves to a private, loopback,
link-local, or cloud-metadata address was reachable by the server. This is the same
class already guarded for the custom models fetch, Actions, avatar, MCP, and the
OpenAI/Anthropic endpoint clients.

Add one helper applySSRFSafeAgentIfDirect(config, url, allowedAddresses) that rejects
non-http(s) or unparseable target URLs, sets maxRedirects to 0 so a redirect cannot
bypass the connect-time check, and attaches createSSRFSafeAgents when no proxy or agent
is already set (proxy precedence preserved). Wire it into the six speech/OCR call sites
and thread each section's allowedAddresses exemption from pr-01.

Scope is private/internal SSRF only. It does not restrict forwarding a credential to a
public host, which is a separate egress-allowlist concern. Absent an allowedAddresses
entry, private targets now fail closed, matching endpoints, actions, and MCP. Document
the new fields in librechat.example.yaml with the operator warning that allowedAddresses
hostnames are trusted before the private-IP check.

* fix: block literal private-IP hosts and thread STT exemptions through generic uploads

applySSRFSafeAgentIfDirect only attached the DNS-lookup agents, but Node
skips the custom lookup for IP-literal hosts, so a literal private IP such
as http://127.0.0.1 connected unchecked. Reject literal private IPs
synchronously in the helper, reusing the same allowedAddresses exemption
logic as the lookup path.

The generic audio-upload path did not forward the section-level
allowedAddresses to sttRequest, so a private STT endpoint permitted via
speech.stt.allowedAddresses failed with ESSRF outside the speech route.
Thread the exemption through files/audio.ts and widen the STTService type.

* fix: derive effective SSRF port for literal IPs and validate OCR target before opening its stream

The literal-IP precheck normalized an empty URL.port to '', so
allowedAddresses exemptions on a default port (127.0.0.1:80, [::1]:443)
never matched. Derive 80/443 from the scheme when the port is omitted.

uploadDocumentToMistral opened the upload file stream before the SSRF
check, so a blocked or malformed target threw with the descriptor still
open. Run the proxy/SSRF setup before fs.createReadStream.

Reword the librechat.example.yaml allowedAddresses guidance to prefer a
private IP literal over a hostname, and reconcile the speech/OCR SSRF
specs to assert the synchronous literal-IP block instead of driving the
connect-time lookup that Node skips for IP literals.

* fix: block literal private IPs before the proxy return, destroy OCR stream on failure, canonicalize IPv6 exemptions

Move the literal-IP check above the proxy/agent early return so a literal
private IP is rejected even when a proxy is configured; document that a
forward proxy must be SSRF-enforcing.

Wrap the Mistral upload post in try/finally and destroy the file stream,
so an async connect-time block does not leak the descriptor.

Canonicalize IP literals in normalizeAddressCandidate through the same URL
serialization targets use, so IPv4-mapped and expanded IPv6 exemptions match.
2026-08-06 09:06:04 -04:00
Dustin Healy
0e14d91ed9
⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine (ReDoS) (#14555)
* ⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine

convertStringsToRegex compiled admin-configured supportedMimeTypes with the native RegExp engine, and checkType runs those patterns against an uploaded file's Content-Type on the server event loop, so a catastrophic-backtracking pattern in fileConfig could ReDoS the whole process on upload.

The MIME-pattern compiler is now swappable. It defaults to native RegExp, which browser builds keep so no engine is added to the client bundle, and the server injects a linear-time engine (RE2JS) at startup. Only test is ever called on these matchers, so the shared type widens to a structural RegexLike with no behavior change for valid patterns. The browser stays on native because a client-side stall would only affect that one tab.

* ⏱️ fix: Wire the linear MIME compiler in the experimental entry point

api/server/experimental.js mounts the same upload routes and calls mergeFileConfig but never set the linear-time compiler, so admin MIME patterns still compiled with native RegExp there. Mirror the setup, and widen the client-side supportedMimeTypes type to the shared RegexLike so the browser typechecks against the same structural matcher.

* 🧹 refactor: Configure the file-config linear engine from a shared helper

Move the RE2 wiring out of both JS server entry points into a single
configureFileConfigRegexEngine helper exported from @librechat/api, so /api stays a thin
caller and the setup no longer has to be kept in sync across index.js and experimental.js.

Also warn loudly when compiling an endpoint's supportedMimeTypes drops every pattern (an
empty allowlist would reject all uploads), and correct the isMimeTypeSupported docstring to
say RegexLike rather than RegExp.

* fix: fail closed when every MIME pattern fails to compile

convertStringsToRegex returned [] when all configured patterns failed to
compile, and filter.ts reads an empty allowlist as no restriction, so a
restrictive config whose patterns all fail allowed every attachment.
Return a single reject-all matcher instead so every consumer fails closed.
2026-08-06 09:05:42 -04:00
Danny Avila
dd159c4566
🔐 fix: Preserve Structured JWT Auth Context (#14652)
* 🔐 fix: Preserve structured JWT auth context

* fix: Omit identity from auth correlation logs

* fix: Isolate pre-auth request context

* style: Sort auth context imports

* fix: Preserve structured auth metadata

* fix: Narrow structured log formatter types

* fix: Annotate structured log context keys

* test: Fix request fixture typing

* fix: Namespace request path log context

* fix: Namespace request method log context

* fix: Preserve captured tenant error paths

* style: Sort tenant error imports

* fix: Classify bulk tenant isolation failures

* fix: Enforce safe request correlation invariants
2026-08-06 08:12:00 -04:00
Danny Avila
45cc53c40b
🛰️ chore: bump @librechat/agents to v3.4.0, cover streamed subagent results e2e (#14647)
* test: cover streamed subagent results end to end

* test: assert real e2e conversation id

* test: harden streamed subagent e2e

* test: stop incompatible subagent fixtures

* chore: update @librechat/agents to version 3.4.0 in package.json and package-lock.json
2026-08-05 22:25:03 -04:00
Danny Avila
56175af0b5
🎟️ fix: Reconcile MCP OAuth Readiness Across Pods (#14629)
* fix: stabilize MCP OAuth readiness across pods

* fix: harden MCP readiness review findings

* fix: resolve CI type check and terminal OAuth polling

* fix: address MCP OAuth readiness review

* fix: align MCP OAuth readiness state

* test: stabilize MCP OAuth readiness assertion

* fix: reject stale MCP OAuth callbacks

* fix: close distributed MCP OAuth readiness gaps

* style: sort Redis MCP test imports

* fix: preserve MCP OAuth polling across rolling pods

* fix: finalize distributed MCP OAuth readiness

* fix: preserve runtime-detected MCP OAuth

* fix: report runtime MCP OAuth readiness

* fix: preserve live MCP OAuth classification

* style: sort MCP connection imports
2026-08-05 19:42:26 -04:00
Danny Avila
489bc02d4a
🧭 fix: Fail Closed When Expected MCP Tools Are Unavailable (#14646)
* fix: fail closed when expected mcp tools are unavailable

* test: strengthen MCP handoff coverage

* fix: clarify unavailable MCP tool guidance

* fix: preserve MCP discovery for empty catalogs
2026-08-05 17:30:57 -04:00
Ravi Kumar L
7775f25b0d
🪢 fix: disable central fanout media uploads for langfuse (#14642)
* fix(langfuse): disable central fanout media uploads

* test(langfuse): cover fanout media policy in run config

* chore(deps): bump agents for Langfuse media policy

* chore(deps): bump agents to 3.3.13

* fix(langfuse): gate central fanout media uploads
2026-08-05 17:23:54 -04:00
Dustin Healy
4cec1a675f
🔒 feat: Add allowedAddresses Exemption to Speech (STT/TTS) and OCR Config Schemas (#14559)
* feat: add allowedAddresses exemption to speech (STT/TTS) and OCR config schemas

Add the existing allowedAddressesSchema as an optional field on sttSchema,
ttsSchema, and ocrSchema, reusing the schema already attached to endpoints,
mcpSettings, and actions so port scoping and normalization stay identical.

STT and TTS resolve a single provider by counting non-empty section keys, so
exclude the allowedAddresses key from that scan. Without the exclusion a
configured exemption list would be counted as a second provider and trip the
"Multiple providers are set" guard. The field is inert on its own: nothing
reads it for SSRF yet, and provider detection now ignores it.

* fix: preserve allowedAddresses through the OCR config loaders

loadOCRConfig rebuilt the ocr config with only apiKey, baseURL,
mistralModel, and strategy, dropping allowedAddresses before it reached
req.config.ocr. Pass it through in both the AppService loader
(packages/data-schemas/src/app/ocr.ts) and the duplicate at
packages/api/src/files/ocr.ts so the exemption survives config load.
2026-08-05 13:42:43 -04:00
Dustin Healy
3f0a1ec8d9
🛡️ fix: Run message-filter PII patterns on a linear-time regex engine (ReDoS) (#14554)
* 🛡️ fix: Run message-filter PII patterns on a linear-time regex engine

The messageFilter.pii middleware compiled admin-configured customPatterns with the native RegExp engine and ran them synchronously against every message on the shared event loop, so a catastrophic-backtracking pattern such as (a+)+$ could stall the entire process (native RegExp takes tens of seconds at roughly 32 characters) and take the instance down for every user.

Compile these patterns with RE2JS, a linear-time RE2 port with no native addon, so catastrophic backtracking is impossible regardless of the pattern rather than something the code tries to detect. Patterns using features RE2 does not support, such as backreferences, fail to compile and are dropped and logged exactly as an invalid pattern already is. The filter only tests for a match, so this is a drop-in engine swap with no behavior change for valid patterns.

* 🛡️ fix: Reject RE2-incompatible messageFilter patterns at config load

The customPatterns regex was validated with native RegExp at config load, but the runtime now compiles it with a linear-time engine (RE2) that does not support backreferences or lookaround. Such a pattern passed validation, then failed to compile and was silently dropped at request time, quietly removing PII protection after upgrade.

Reject backreferences and lookaround during config validation with an explicit message, and document RE2 syntax in the example config instead of "JavaScript-flavor". The runtime engine remains the authoritative boundary and still drops-and-logs anything this load-time check misses.

* 🧹 test: Use direct MessageFilterPiiConfig annotations in the PII specs

The added ReDoS cases satisfy the exported MessageFilterPiiConfig type directly, so the `as unknown as` assertions were unnecessary. Annotate the config objects directly, matching the repo's type-safety guidance.

* 🧹 fix: Reject named backreferences in messageFilter patterns at config load

Extend the config-load check to also reject named backreferences (\k<name>), which are valid JavaScript regex but unsupported by the linear-time runtime engine, so they surface at load rather than being dropped at request time. Together with the existing numeric-backreference and lookaround checks this covers the RE2-incompatible construct set; the runtime engine remains authoritative.

* 🛡️ fix: Preserve Unicode whitespace matching in messageFilter starter patterns

RE2's \s is ASCII-only, so after the engine swap the built-in api-key and Bearer starters no
longer matched a secret separated by non-ASCII whitespace (e.g. a non-breaking space), which
native RegExp did match. Broaden the whitespace classes to [\s\p{Zs}] so those patterns keep
their original coverage, and add a regression test for a non-breaking-space separator.

* 🛡️ fix: Validate messageFilter patterns with the RE2 engine at config load

Replace the syntax blacklist (numeric/named backreferences, lookaround) with authoritative
validation: config load now compiles each custom pattern with the same linear-time engine the
runtime uses, so any RE2-incompatible construct (including control escapes like \cA) is rejected
at load with a clear error instead of being silently dropped at request time.

The validator is swappable and defaults to native RegExp so browser builds add no engine; the
server wires the RE2-backed check at startup via configureMessageFilterRegexValidator in both
entry points.

* 🛡️ fix: Match the full whitespace set in messageFilter starter patterns

RE2's `\s` omits the vertical tab and `\p{Zs}` omits U+2028, U+2029, and
U+FEFF, so a separator built from one of those characters slipped past the
`api-key` and `Bearer` starter patterns and reached the model. Broaden the
starter whitespace class to the full JavaScript whitespace set so those
separators are covered again.

* fix: fail closed when messageFilter.pii compiles to zero patterns

DB and admin config overrides bypass the RE2 schema validation (it only
runs at YAML load), so an override whose only pattern is RE2-incompatible
was dropped at compile time, left zero patterns, and let the request
through. compile() now returns a failClosed flag when a config declared
patterns but every one failed to compile; the middleware returns 400 and
findPiiMatchInMessages returns a distinct misconfigured match that the
OpenAI and Responses controllers surface with an admin-facing message.

* 🛡️ fix: Fail closed when any messageFilter.pii custom pattern drops

compile() previously set failClosed only when every pattern dropped (patterns.length === 0 && dropped > 0). With the default starters present, a single RE2-incompatible custom override incremented dropped but left patterns.length > 0, so the filter silently enforced only the surviving subset and text matching only the dropped rule passed.

failClosed now keys off dropped > 0, so any dropped custom pattern blocks with the misconfigured 400. YAML patterns are RE2-validated at load, so dropped stays 0 for valid configs and only unvalidated DB or admin overrides can trip it. Reframed the two keeps-others-active specs to assert fail-closed and added a default-starters partial-drop regression.

* 🧹 fix: Correct the misconfigured JSDoc and drop redundant casts in the PII specs

The misconfigured flag now means any configured custom pattern failed to compile, not that every pattern failed, so its JSDoc on PiiMatch is updated to match. The partial-drop regressions now use direct MessageFilterPiiConfig annotations instead of as-unknown-as casts, keeping the specs type-checked, consistent with the rest of the suite.
2026-08-05 13:42:18 -04:00
Danny Avila
1bbe7dfe83
📦 chore: npm audit (#14640)
* chore: Update undici dependency to version 8.10.0

* chore: npm audit fix

* chore: downgrade undici dependency to version 7.29.0
2026-08-05 13:33:46 -04:00
Danny Avila
22642df40c
🧹 fix: Exclude Mongo ID From Conversation Updates (#14631)
* fix: Exclude Mongo ID from conversation updates

* fix: Limit conversation sync to conversation ID

* fix: Preserve explicit conversation metadata
2026-08-05 11:24:44 -04:00
Danny Avila
ccf4301093
🚦 feat: Configurable Circuit Breakers for Runaway Streamed Tool Args (#14613)
* 🚦 feat: Configurable Circuit Breakers for Runaway Streamed Tool Args

* docs: forewarn create_file about the streamed tool-argument limit

The breaker failing a near-limit write should not be the model's first
exposure to the bound. Both create_file variants now state the default
64 KB per-call limit and the incremental pattern (create the first
section, extend with edit_file) in the tool description and the content
parameter description.

* fix: keep skill create_file description under the provider advisory cap

The limit-guidance paragraph pushed the skill-aware description to 1169
chars, past the 1024-char advisory bound where providers may truncate.
The skill variant now carries the guidance only in its content parameter
description, which sits closest to the generated payload and is not at
truncation risk; the shorter code-sandbox variant keeps the full
paragraph.

* 🚦 feat: per-tool streamed-arg limits with a create_file default

Thirty days of production data show create_file is the only tool class
with legitimate near-limit arguments (p99 80.6 KiB; every other tool
p99 under 10 KiB). Rather than loosening the global 64 KiB cap for all
tools, the yaml gains maxToolCallArgBytesByTool (per-tool overrides,
keyed by model-facing tool name, 0 disables that tool's guard) and
LibreChat ships { create_file: 131072 } by default; yaml entries merge
over and can replace it. Pairs with maxToolCallArgBytesByTool support
in the agents SDK and stays inert until the dependency bump.

* test: pass per-tool spec configs as plain Partial literals

The as-TAgentsEndpoint casts fail TS2352 for object-valued fields:
comparability does not grant nested literals the implicit index
signature that plain assignability does, so casts carrying
maxToolCallArgBytesByTool never sufficiently overlap. The mapper
already accepts Partial<TAgentsEndpoint>, so the new cases pass
uncast literals instead.

* chore(deps): bump @librechat/agents to 3.3.12
2026-08-04 23:05:05 -04:00
Danny Avila
b11978017d
🧱 fix: Enforce Agent Runtime File Trust Boundaries (#14577)
* fix: secure agent runtime file metadata

* chore: sort agent resource test imports

* fix: Align Agent Tool Resource Types

* fix: Rehydrate Agent Image Resources

* fix: preserve remote agent file authorization
2026-08-02 14:18:28 -04:00
Danny Avila
db6ba5392a
🪢 fix: Bind MCP OAuth Secrets to Trusted Endpoints (#14578)
* fix: bind MCP OAuth secrets to trusted endpoints

* fix: bind stored MCP OAuth clients during refresh

* fix: address MCP OAuth review findings

* fix: bind stored MCP OAuth credentials

* fix: make MCP OAuth credentials generation-safe

* test: update MCP OAuth uninstall binding fixtures

* fix: harden MCP OAuth credential persistence

* fix: scope MCP OAuth refresh single-flight

* style: sort MCP OAuth token imports
2026-08-02 13:38:58 -04:00
Danny Avila
7e74f8eb8c
🪪 fix: Strip Unresolved Header Placeholders at Final Resolution (#14595)
Unresolved {{LIBRECHAT_USER_*}} header templates leaked literally to
upstream providers when user context was missing at resolution time
(e.g. async title generation racing client disposal), letting a gateway
trust LibreChat's own template syntax as an account identity.

resolveHeaders now takes an opt-in stripUnresolved flag that blanks any
resolvable-but-unresolved LIBRECHAT_USER/BODY/OPENID placeholder, enabled
at every final resolution boundary (resolveConfigHeaders, model fetches,
Google init, summarization overrides, azureAssistants init). Staged
passes that resolve again later with more context are left untouched, as
is the async-resolved {{LIBRECHAT_GRAPH_ACCESS_TOKEN}} and unknown names.

titleConvo now resolves headers from the req captured at entry instead of
re-reading this.options.req, which disposeClient nulls concurrently.

Fixes #14580
2026-08-02 06:38:06 -04:00
Danny Avila
59395a6bf0
🪢 refactor: Move Agent Execution Seam Before Initialization (#14581)
* refactor: move agent execution seam before initialization

* refactor: type librechat agent request extensions

* refactor: read envelope values from descriptors

* fix: preserve envelope types and validation errors

* fix: bound agent envelope traversal
2026-08-01 18:25:34 -04:00
Rayan Salhab
3551c1ba8e
🪧 fix: Guard Admin OAuth Routes When Providers Are Not Configured (#14507)
* fix: guard admin OpenID routes without config

* fix: guard remaining admin SSO routes without registered strategy

---------

Co-authored-by: cyphercodes <cyphercodes@users.noreply.github.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-01 14:46:17 -04:00
Danny Avila
b253b623fe
📦 chore: update sanitize-html to latest (#14573)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 📦 chore: update `sanitize-html` to latest

* chore: add additional modules to esModules for Jest configuration
2026-08-01 09:35:57 -04:00
Danny Avila
3191f6975a
🏷️ fix: Skip Title Generation for Preempt-Incomplete Turns (#14571)
Some checks failed
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
2026-08-01 09:01:13 -04:00
Danny Avila
3dc5532111
🏢 fix: Preserve Tenant Context for Partial Response Saves on Disconnect (#14567) 2026-08-01 08:18:32 -04:00
Danny Avila
6f45a9e32e
🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases (#14553)
* 🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases

Tool keys had two spellings that could diverge for any server whose name
contains characters outside [a-zA-Z0-9_.-]: the tool cache (and registry
inspector) built keys with the RAW server name, while runtime instances
are named with normalizeServerName(serverName). Three code comments
already asserted "tool keys embed the normalized server name" - no
producer honored it. For a special-character server that meant:

- definitions-only mode shipped raw def names the model echoed back,
  but the executor's tool map held the normalized instance name, so
  every call failed with "Tool not found";
- per-tool tool_options (defer_loading / allowed_callers /
  run_in_background / describe_intent) were persisted under raw keys
  that never matched the definition names the option passes resolve
  against, so builder settings were silently inert;
- tool-key parsing against normalized candidate lists silently fell
  back to last-delimiter splitting, which mis-parses delimiter-bearing
  tool names.

The reconciliation is one contract enforced in three moves:

1. PRODUCERS NORMALIZE. The tool cache (packages/api/src/mcp/tools.ts)
   and the registry inspector build keys with the normalized server
   name, matching the instance names MCP.js has always assigned. The
   builder's tool ids, agent.tools entries, tool_options keys, and
   definition names all flow from these keys, so every model-facing
   name now agrees. The cache STORE stays keyed by the raw config name.

2. CONFIG LOOKUPS RESOLVE ALIASES. New shared helpers in data-provider
   (buildServerNameAliases, normalizeMCPToolKey) map a parsed
   normalized name back to the raw config name that the registry,
   config maps, tool cache, and plugin-auth rows are keyed by. Applied
   in the definitions loader closure, handleTools grouping,
   createMCPTool's parsing fallback, getUserMCPAuthMap, and the MCP
   tools endpoint - matching both spellings so legacy raw keys keep
   resolving.

3. LEGACY DATA HEALS AT ONE BOUNDARY. initializeAgent rewrites
   raw-keyed agent.tools entries and tool_options keys to the
   normalized form (normalizeAgentToolKeys) before anything consumes
   them, so agents persisted under the old convention load their tools
   AND have all four per-tool options honored. Placeholder and
   server-pin tokens stay raw - they are config-identity references,
   not model-facing names.

Servers whose names are already in the safe character set (the common
case) produce byte-identical keys before and after; the fast path
allocates nothing. Stale Redis-cached raw keys self-heal via the
existing reconnect-on-missing path within one cache cycle.

* 🧯 fix: Deterministic Alias Collisions + Raw Names in Definition Metadata

Two review findings on the normalization contract:

- Two configured server names that normalize to the same segment (e.g.
  'Sales Force' and 'Sales:Force' -> 'Sales_Force') produce inherently
  ambiguous tool keys; the alias map silently resolved last-wins, so a
  tool selected from one server could execute against the other's
  config. buildServerNameAliases now resolves collisions to the FIRST
  configured name deterministically, and resolveMCPServerContext warns
  once per colliding pair per process so the operator can rename one
  server. A collision-resistant identifier would change every existing
  tool key, so detection + stable routing is the right treatment here;
  startup-time config validation can follow separately.

- The definitions loader resolved parsed (normalized) server names to
  raw only inside the ToolService closure, while the definition
  metadata (serverName -> mcpRawServerName) kept the normalized value.
  Server instructions are keyed by raw config names, so a
  special-character server's instructions were silently omitted in
  definitions-only mode. loadToolDefinitions now takes rawServerNames,
  resolves the boundary against both spellings, and stores the RAW
  name in definition metadata - consistent with the instance path.

* 🧯 fix: Heal Stale Caches, Skill Allowed-Tools, and Builder Selectors

Three review findings on the normalization rollout, all in the
transition class:

- Stale cache entries (P1): the definitions-only loader treats the
  per-server tool map as authoritative and never reconnects on a
  per-key miss, so a pre-change raw-keyed Redis entry would make a
  special-character server's tools vanish for up to the cache TTL.
  getMCPServerTools now heals legacy raw-keyed entries to the
  normalized format at read time (keys and function names), covering
  every consumer with no coordinated invalidation; safe names return
  the map untouched.

- Skill allowed-tools: a skill declaring a raw MCP key in
  allowed-tools bypassed the initialize-boundary heal (the union runs
  after it) and would neither dedupe against healed agent tools nor
  match the normalized tool map. The primes' allowedTools now pass
  through the same normalizeAgentToolKeys heal before unioning.

- Builder selectors: matchesMcpServer and useVisibleTools parsed tool
  ids against raw server names only, so an attached special-character
  server rendered as an unselected orphan card. Both now accept the
  normalized spelling and resolve it back to the raw map key, keeping
  legacy raw ids working.

* 🧯 fix: Fail Closed on Normalized Server-Name Collisions

Escalation of the collision finding: a deterministic first-wins alias
plus a warning still let the tools listing publish BOTH colliding
servers, so a tool selected under the shadowed second server would
silently execute against the first server's configuration (their
model-facing keys are identical, so routing cannot ever distinguish
them).

- findShadowedServerNames identifies later-configured names whose
  normalized form an earlier different name claimed.
- getMCPTools excludes shadowed servers from the published listing
  entirely (with a warn naming the collision), so their tools are
  never selectable - nothing ambiguous can be picked.
- Server creation reserves both spellings: a generated slug may not
  collide with a raw config name OR the normalized form its tool keys
  would carry.

Collision-resistant model-facing IDs remain out of scope: changing
normalizeServerName's output would rewrite every existing tool key
(agent documents, caches, instance names) for ALL servers to handle a
misconfiguration that is now blocked from exposure instead.

*  fix: Dedupe Reserved Server-Name Spellings at Creation

The reservation list appended normalized forms unconditionally, which
duplicated every safe name (raw === normalized) and broke the
route-level contract test pinning the exact list. Dedupe via a Set so
safe names contribute one entry, while special-character names still
reserve both spellings; adds the special-character reservation case.

* 🧯 fix: Never Heal a Shadowed Server's Keys; Align Authorization Tie-Break

Persisted references were the remaining collision vector: an agent or
skill saved with the shadowed later server's raw key was HEALED into
the shared normalized key, authorized through a last-wins map, and
routed first-wins - authorized as one server, executed as another.

- normalizeAgentToolKeys now refuses to rewrite keys of shadowed
  servers (findShadowedServerNames): rewriting would produce exactly
  the first server's key. Left raw, the key cannot match the
  normalized-keyed tool map and the tool fails visibly - broken beats
  misrouted. Covers agent.tools, tool_options, and skill
  allowed-tools through the shared heal.

- filterAuthorizedTools (agents/v1.js) builds its normalized-to-raw
  map via the shared buildServerNameAliases instead of a last-wins
  Map constructor, so authorization resolves a colliding key to the
  SAME first server execution routes to.

* 🧯 fix: Direct Identity Wins Over Aliases; Heal Client Forms and Degraded Contexts

Four review findings on the normalization edges:

- Alias hijack (P1): a user-DB server named exactly like an operator
  server's normalized form ('foo' vs YAML 'foo!') had its tools
  rerouted to the operator server by unconditional alias resolution.
  Resolution is now DIRECT-FIRST everywhere: the parsed name is tried
  as-is, and only when nothing resolves is it treated as a normalized
  spelling (definitions loader, handleTools grouping, createMCPTool
  fallback). buildServerNameAliases seats identity entries before
  derived ones so a literal name owns its slot regardless of config
  order, findShadowedServerNames and the collision warning derive from
  the same construction, and getUserMCPAuthMap fetches auth under both
  spellings so either owner finds its rows.

- Builder double-match: a normalized name containing the delimiter
  ('foo mcp bar' -> 'foo_mcp_bar') also suffix-matched a server named
  'bar', selecting both cards and making removal strip the wrong tool.
  matchesMcpServer now resolves the token ONCE against the full
  configured list (longest boundary, both spellings) when the caller
  supplies it; selection and removal share the resolution.

- Builder legacy ids: an agent saved with raw-keyed ids showed its
  tools unchecked while the runtime heal kept them active, and
  selection updates never replaced the legacy entries. McpSection maps
  legacy raw ids to their current normalized ids when deriving and
  rewriting this server's selection.

- Degraded context: a transient ensureConfigServers failure returned
  an entirely empty context, leaving normalized keys unresolvable for
  the request. resolveMCPServerContext now keeps the name lists (they
  derive from the config snapshot alone) and degrades only the
  lazy-init configs.

* 🧯 fix: Collision Detection Sees Accessible Servers; Shadowed Refs Fail Closed End to End

Round follow-ups on the collision design, all in the
DB-server-visibility class:

- The legacy-key heal detected collisions against operator-config
  names only, so healing could still produce a key that direct-first
  resolution routes to an invisible user-DB server. initializeAgent
  gains an optional getAccessibleMcpServerNames dep (wired through
  ToolService for controllers that mock it, directly elsewhere),
  consulted ONLY when a configured name needs normalization - zero
  cost for safe-name deployments. The heal then sees the full
  accessible set and skips shadowed servers' keys.

- Wildcard and legacy raw tokens bypassed catalog filtering, letting a
  shadowed server's instances join a run under the same normalized
  names as the winner's. filterAuthorizedTools rejects tools of
  shadowed servers at authorization (its merged map sees DB + config),
  and handleTools skips them at execution.

- The builder migrated only tool selection, not tool_options: legacy
  raw option keys showed disabled while the runtime honored them, and
  toggles could not clear them. McpSection now migrates option keys to
  the current normalized ids (existing normalized entries win).

- A transient ensureConfigServers failure degraded to an EMPTY server
  context, leaving normalized keys unresolvable for the request.
  resolveMCPServerContext keeps the name lists (derived from the
  config snapshot alone) and degrades only the lazy-init configs.

* 🧯 fix: Complete the Collision Audit at Every Gate; Safer Heal Semantics

Round follow-ups hardening the collision audit:

- Execution guards now consult the FULL accessible set: the caller's
  heal threads its already-fetched names through loadTools, and
  handleTools fetches them itself when a configured name needs
  normalization (never for safe-name deployments) - so a cross-tier
  collision (user-DB 'foo' vs operator 'foo!') fails closed at eager
  execution instead of joining the run under one normalized name.

- Healing is SKIPPED when the collision audit cannot complete
  (transient lookup failure, or no dep): un-healed raw keys still
  resolve through the direct-first candidates, so skipping is safe
  while rewriting against an incomplete audit is not.

- The audit lookup is gated on the agent actually carrying
  delimiter-bearing keys (tools, tool_options, or skill
  allowed-tools), so non-MCP agents never pay a registry round-trip
  even on specially named deployments.

- normalizeAgentToolKeys gives the CURRENT (normalized) entry
  precedence when both spellings carry options, matching the builder's
  migration semantics instead of letting insertion order decide.

- The builder's toCurrentToolId resolves entries boundary-exactly
  against every configured server (longest match, both spellings), so
  a raw suffix shared with a LONGER server name can no longer reassign
  that server's selection or options while another dialog is open.

* 🧯 fix: Shared Collision Audit for Definitions Loading; Fail Closed on Audit Failure

Round follow-ups closing the remaining audit gaps:

- The definitions-only loader now consumes the same collision audit as
  eager loading: shadowed servers' entries (wildcards included) are
  dropped before definitions are emitted, so the default execution
  path can never resolve a shadowed server's normalized function name
  to another server. The audit names thread from initializeAgent's
  heal; the loader self-fetches only when a configured name needs
  normalization.

- resolveCollisionAuditNames centralizes the audit-resolution policy
  (threaded set > self-fetch when needed > incomplete on failure), and
  BOTH loaders now fail closed under an incomplete audit: any
  normalization-sensitive reference (its own name needs normalizing,
  or it equals the normalized form of a configured special-character
  name) is skipped with a warning instead of being audited against
  operator names alone. isNormalizationSensitiveName lives in
  packages/api as a pure helper so test mocks use the real predicate.

- normalizeAgentToolKeys collapses duplicate ids after healing
  (order-preserving): a document carrying both spellings converges on
  one key, never two instances with the same function name.

* 🧯 fix: Thread the Audit Everywhere; Identity-Aware Alias Fallback

Round follow-ups on audit plumbing:

- The OpenAI-compatible and Responses tool loaders now forward the
  already-resolved accessibleMcpServerNames instead of discarding it,
  so the definitions loader neither repeats the registry lookup nor
  fails closed on a transient second lookup after the first succeeded.

- The skill-only path threads its audit: when the baseline agent has
  no MCP keys but a primed skill's allowed-tools fetched the complete
  set, that set (not the operator-only list) reaches the loader, so
  the collision remains visible and the shadowed reference stays
  rejected end to end.

- OAuth discovery iterates the collision-FILTERED tool list, so a
  request can no longer emit an OAuth prompt, wait out the connection
  timeout, and reconnect a server whose definitions were deliberately
  rejected.

- The definitions loader's alias fallback is identity-aware: when the
  parsed name IS a known accessible server, a null tool fetch means
  temporarily unavailable (OAuth pending, missing user variables,
  disconnected) and no longer reroutes to the raw alias - previously
  the aliased operator server's definitions could be emitted under the
  unavailable DB server's names.

* 🧯 fix: Legacy-Key Definition Lookup; Retain Audit for Deferred Execution

- createMCPTool resolves tool definitions by BOTH spellings: the key as
  persisted plus the canonical normalized key built from the resolved
  server name. Assistants and direct tool calls persisted before the
  rollout bypass the agent-boundary heal and arrive with raw keys, while
  availableTools is now indexed canonically - previously every such call
  missed the index, burned a reconnect, and returned the unavailable
  stub permanently via the negative cache.

- The initialized agent retains accessibleMcpServerNames (the COMPLETE
  collision audit this initialization resolved), buildAgentToolContext
  copies it into every per-agent tool context, and loadToolsForExecution
  threads it into the eager loader as bare options. Deferred/event-driven
  execution therefore reuses the snapshot instead of repeating the merged
  registry read - a transient failure there could fail-closed a tool the
  same turn already advertised from the successful first audit.

- MCP.spec.js keeps @librechat/api pure helpers REAL (requireActual
  spread) so normalization paths are exercised rather than mirrored.

* 🧯 fix: Parse Legacy Keys Against Both Server-Name Spellings

createMCPTool's boundary candidates were normalized-only, so a legacy
raw key whose server name contains the delimiter (foo_mcp_bar!) missed
the suffix match and fell to the generic last-delimiter split - the
canonical rebuild then produced a key that could never hit the index
and the persisted call stubbed out. The candidate list now carries the
RAW resolved name (and raw config names on the parse-only path) next
to the normalized spellings.

* 🧯 fix: Honest Audit Completeness; Shadowed-Server Form-Key Guard

- resolveAllMcpConfigs tolerates ensureConfigServers failures, so the
  merged registry read can silently omit config-only servers while the
  audit still reported complete: true - a foo/foo! collision would go
  unseen and a persisted key could route to the wrong server. Both
  audit consumers now union the snapshot-derived raw config names back
  in (resolveCollisionAuditNames unions the caller's rawServerNames;
  the initializeAgent heal unions configRawServerNames), keeping the
  completeness label honest without an extra read: operator names come
  from the registry-independent config snapshot, user-DB names from the
  merged read that fails loudly into the existing incomplete path.

- The client tool_options migration now mirrors the runtime heal's
  fail-closed rule for SHADOWED servers: when the dialog's server has
  lost its normalized slot to another catalog name, legacy raw keys
  stay raw instead of being rewritten onto the winning server's key,
  where a later save would apply the wrong server's per-tool settings.
  The dialog's own server joins the alias construction so a stale
  catalog map can't misread as a collision.

* 🧯 fix: Heal Legacy Assistant MCP Tool Names on Save

The assistants create/update controllers look tools up in the cached
definitions by exact key, and the cache is now normalized-keyed - an
assistant saved before the convention resubmits its raw-suffixed MCP
name on every edit, so any save silently removed the tool.

healMcpToolNames pre-heals the payload's tool list: a delimiter-bearing
string that misses the cache resolves through the configured raw names
(longest-suffix, boundary-exact) and rewrites to the normalized key
only when that key actually exists in the cache. SHADOWED raw names
stay raw and fail closed, mirroring the runtime heal; the config read
happens only when a delimiter-bearing name actually misses, and read
failures propagate (write path) rather than silently dropping tools.
v2's update loop also stops re-reading the tool cache per iteration.

* 🧯 fix: Full-Audit Shadow Set + Dedupe in the Assistant Key Heal

- The assistant-save heal built its shadow set from operator config
  names alone, so a cross-tier collision (user-DB `foo` owning the
  normalized slot of operator `foo!`) looked unshadowed and the legacy
  key healed into the shared normalized name - which direct-first
  execution then binds to the DB server. The shadow set now comes from
  resolveCollisionAuditNames' full accessible audit, and an incomplete
  audit skips healing outright (every rewrite candidate is
  normalization-sensitive by construction, so raw-and-fail-closed is
  the only safe answer).

- Healed string entries dedupe order-preserving: a payload carrying
  both spellings of the same tool collapses to one entry instead of
  expanding into duplicate function definitions the provider rejects.
2026-08-01 07:39:24 -04:00
Danny Avila
b9ca391b84
📦 chore: bump @librechat/agents to v3.3.11 (#14562) 2026-08-01 02:39:24 -04:00
Danny Avila
e7f1838515
feat: Reliable Interrupt & Steer Escalation and Recovery (#14558)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: surface interrupt-steer escalation on waiting messages

The interrupt & steer feature shipped reachable only through the
composer chord, the send-button hovercard, and the composer button; a
message already waiting (queued for after the run, or steered and
parked at the next tool boundary) had no path to it. Both waiting
surfaces now carry one:

- Queued rows get an icon-only ZapOff escalation button beside the
  existing Steer primary. It routes through sendQueuedNow, which now
  takes a preempt option on its live-run path. The tooltip teaches the
  composer chord, derived through resolveComposerKeyDown so a rebound
  or yielded chord is never advertised.
- In-flight steer bubbles get an "Interrupt now" overflow entry with
  the same race rules as Edit: reclaim first, and only a `reclaimed`
  outcome resubmits (via retrySteer with preempt, swapping the chip
  for an interrupting one). `applied` and run-ended-mid-reclaim
  outcomes stop at the existing informational toasts, so the words can
  never land twice. Not offered on a steer already preempting.
- Every during-run overflow menu gains an "Always interrupt instead"
  toggle for steerInterruptsByDefault, next to the existing steer/queue
  default toggle. MenuEntry supports disabled for the new entries.

Only one interrupt can be unresolved at a time: while one preempt is
pending (or the run is paused on approval, where the server 409s),
every escalation control disables instead of racing the same seal.

Ten new tests across both surfaces; 381 green in the affected suites.

* fix: lock escalation across its reclaim window, keep the paused control visible, label as steer

Codex round 1, all three findings.

P2, escalation race. The single-interrupt invariant had a window between
clicking "Interrupt now" and the reclaim resolving, where no preempt
chip existed for the chip-derived gate to see: two bubbles escalated
back-to-back could both resubmit. A shared escalating flag (Jotai,
per-conversation) now covers the window and disables every escalation
control on both surfaces, and a fresh recheck before resubmitting
catches an interrupt armed elsewhere meanwhile (composer chord, queued
row); those words re-home to the queue with an informational toast
instead of breaking the invariant.

P2, unreachable paused state. canSteer is defined as
hasRealConvoId && !pausedOnApproval, so gating the button on canSteer
removed it exactly when it was meant to render disabled; the test only
passed on an impossible stub combination. The render gate is now
duringRunActive && (canSteer || pausedOnApproval), and the test uses the
real invariant.

P2, label semantics. "Interrupt & send now" borrowed the name of the
hard-abort action; this one preserves the partial answer and steers.
Renamed to "Interrupt & steer now" (com_ui_interrupt_steer_now).

Both behavior fixes counterfactually verified; 384 tests green across
the affected suites.

* fix: disable bubble escalation while the run cannot accept a steer

Codex round 2, one P2. Answer mode (ask_user_question) sets
duringRunActive false while pausedOnApproval stays false, since that
flag only detects approval-bearing tool calls. The bubble's escalation
entry stayed enabled there, so clicking it cancelled a healthy waiting
steer and the preempt resubmission bounced off RUN_PAUSED, degrading
the words to the queue. The entry now also disables on
!duringRunActive, matching the queued-row control's gate.

Counterfactually verified: reverting the gate fails the new
answer-mode test.

* fix: recheck live run state after the reclaim, not just at the click

Codex round 3, one P2, and it is the round-1 recheck principle applied
one level deeper: the entry-time disable cannot see a run that pauses
(tool approval, answer mode) while the reclaim round-trip is in flight,
and the .then closure held the render's stale steering controls, so the
resubmit would fire into a RUN_PAUSED rejection after the reclaim had
already surrendered the steer's boundary slot.

The escalation continuation now reads the LIVE controls through a
latest-ref: if the run can no longer accept a steer, the words re-home
to the queue with an informational toast instead of resubmitting, and
the resubmit itself also goes through the live controls.

Counterfactually verified: reading the stale closure instead of the ref
fails the new mid-reclaim pause test.

* refactor: make escalation one atomic server-side arm, in place

Codex round 4: four P2s, every one an interleaving of the same window —
escalation as reclaim-then-repost is a compound, non-atomic operation
whose continuation must revalidate the world (FIFO position lost, ref
assigned too late, no run fence, competing bubble actions). Rounds 1-3
patched that window with a lock and rechecks; round 4 shows the window
itself is the defect, so this removes it instead of guarding it again.

Escalation is now POST /chat/steer/arm: the server flips preempt on the
EXISTING queued item in one atomic store op (new IJobStore.armSteer; a
decode-patch-encode LSET Lua on Redis, an in-place mutation in memory),
fenced to the validated generation and refused once the queue closes.
The handler mirrors the steer POST's preempt contract exactly: durable
flag gated on the owner's recorded capability, volatile requestPreempt
fire-and-forget because the durable flag is the truth resume/handover
re-arm from.

By construction this resolves all four findings: FIFO survives (the
item never moves; the whole queue still drains in instruction order at
the seal), there is no continuation to hold stale controls, the store
op is fenced to the original run, and a competing Edit/Queue/Cancel
either beats the arm (armed:false, chip untouched) or operates on the
armed item, whose cancel already disarms.

The client escalation entry becomes one mutation: armed:true relabels
the chip in place (same steerId, same position), PREEMPT_UNSUPPORTED
and lost races toast honestly, and the round 1-3 machinery — the
escalating lock atom, the latest-ref, the post-reclaim rechecks and
their two toast strings — is deleted rather than extended.

Verified: 7 new handler tests on the real in-memory manager (including
FIFO preservation and the stale-generation fence), 2 Redis integration
tests against real Redis (in-place arm keeps order and every field;
missing/stale/closed all refuse), client suites 396 green.

* fix: decide capability inside the atomic arm, neutralize the lost-race toast

Codex round 5, both findings, both edges of the new arm design rather
than its mechanism.

P2, capability TOCTOU. A HITL resume on a rolling deploy rewrites
preemptCapable for the SAME generation, so the handler's read could go
stale between validation and the flag flip, arming a steer the live
owner cannot seal. armSteer now returns armed | missing | incapable,
with the owner's live capability part of the same atomic predicate as
the generation fence (HGET preemptCapable inside the Lua; the flat job
field, not a metadata blob — the in-memory store reads the same field).
The handler's pre-check is deleted rather than kept alongside; the
store predicate is the single source. New handler test rewrites the
capability after queueing and expects PREEMPT_UNSUPPORTED with the item
left unflagged; the Redis guards test now asserts the incapable refusal
against real Redis.

P2, ambiguous toast. armed:false covers injected, cancelled, re-homed,
and run-over alike, so telling the user the message "already reached
the agent" claimed one specific outcome. The lost-race branch now uses
a neutral message (com_ui_steer_arm_lost_race) and defers to the events
for what actually happened.

* fix: flip the escalation lock synchronously before the arm request

Codex round 6, one P2. Round 4 deleted the escalating flag along with
the reclaim continuation it guarded, but that left the one-interrupt
gate blind during the arm request's own round trip: the chip-derived
check cannot see an arm until its response relabels the chip, so on a
slow connection two bubbles could both arm before either response
landed. Double-arm is harmless server-side now (the run seals once and
drains the whole queue in order), but every escalation control
advertises "one interrupt at a time" by disabling, and the controls
must tell the truth.

The per-conversation escalating flag returns as a pure UX gate: set
synchronously at click, before the mutation, cleared on settlement, and
folded into interruptPending on both surfaces. Unlike its round 1-3
ancestor there is no continuation behind it to guard and no recheck to
pair with it.

Counterfactually verified: without the synchronous set, the two-bubble
race test arms twice. 207 tests green across the Chat Input suites.

* test(e2e): cover escalation of waiting messages through the real seal

Three mock-harness tests on E2E_SLOW_REPLY, a 160-chunk stream with no
tool boundary, so an in-thread steer part can ONLY come from a genuine
mid-stream seal — which makes each test a behavioral proof rather than
a UI check:

- Queued row escalation: the ZapOff button turns a waiting queued
  message into a preempt-armed steer (202 echoes preempt: true) that
  seals and injects, where the sibling steering.spec test proves the
  unescalated path waits for run end instead.
- Bubble in-place arm: an ordinary steer (202 with no preempt echo)
  waits as a bubble, POST /chat/steer/arm answers armed: true, the
  bubble relabels in place (same single bubble, same text, escalation
  no longer offered on reopen), and the armed steer seals mid-stream.
- Always-interrupt toggle: flipped from a waiting row's overflow menu,
  plain Enter now produces a preempt: true steer that seals in the SAME
  run, and the menu offers the way back. An afterEach clears the
  localStorage preference so a mid-test failure cannot leak
  preempt-by-default into the rest of the serial suite.

All three verified locally through the full harness (real backend, mock
LLM, seeded DB): 3 passed in 27s.

* feat: dedicated escalation arrow + shortcut, menu split into actions and preferences

The escalation was still half-hidden: the bubble only offered it inside
the overflow menu, and the tooltip taught the composer chord, which does
a different thing (interrupts with typed text, not this chip). Three
changes make it a first-class command:

- A shared EscalateNowButton (circular arrow, ghost-bordered like the
  composer's interrupt control) is always visible on BOTH surfaces:
  beside each queued row's Steer primary and on every waiting steer
  bubble next to its menu. It disappears once a steer is interrupting.
- A dedicated registry shortcut, escalateSteer (Cmd/Ctrl+Shift+.),
  editing-allowed and rebindable like every other action. Deliberately
  NOT an Enter chord: the composer owns every Enter chord, and the
  yield design rests on no default binding using Enter besides submit.
  Its handler clicks the newest enabled arrow control (bubbles beat
  queued rows), so the shortcut can never diverge from the button, and
  the arrow's tooltip teaches THIS command via the registry display.
- The overflow menus separate one-off actions from sticky behavior
  changes: Edit, Cancel, Queue, then a smaller "Preferences" section
  holding the queueing and always-interrupt toggles, each with the
  standard InfoHoverCard reusing the Settings panel's descriptions.
  "Interrupt & steer now" leaves the menu entirely.

386 client tests green, including a menu-structure test locking the
order and the absence of the escalation entry; bubble escalation tests
drive the visible arrow. The e2e spec's bubble test now clicks the
arrow, and a fourth test drives the dedicated shortcut end to end
through a real mid-stream seal.

* style: bind the escalation arrow to its message (variant A anatomy)

Two same-weight circles in a row read as one control group, leaving the
arrow's ownership ambiguous, and a floating arrow stops meaning anything
once several messages stack. The shared control now carries variant A's
anatomy: a thin divider binds a small SOLID arrow (filled, inverted) to
the message region on its left, and the menu ellipsis stays a bare
glyph, so the two affordances can no longer blur together — and the
divider+arrow pairing repeats cleanly per chip at N messages.

* chore: drop the unused within import CI lint caught

* fix: advertise the escalation shortcut only while the control is live

Codex on the e2e head, one P2: the tooltip appended the chord hint even
while the button was disabled, advertising a shortcut that does nothing
during an approval pause. The flagged control (InterruptNowButton) was
since replaced by the shared EscalateNowButton, which inherited the
pattern; the successor now omits the chord whenever the control is
disabled, matching the rule the during-run hovercard already follows.

* fix: harden steer escalation lifecycle and recovery

* test(e2e): disambiguate accessible steer preferences

* test: align abort persistence coverage with prerequisites

* chore(i18n): remove obsolete steer race message

* chore: normalize imports across steering changes

* test: exercise stream integration on Redis Cluster

* test: scope HITL checkpoints to generation

* test: fix cluster cleanup and locale policy

* fix: keep escalation visible during ask pauses

* fix: fence recovery downgrade and stale predecessors

* fix: require generation owner abort acknowledgement

* fix: validate delayed preempt arms

* test: align final escalation fixtures

* fix: preserve in-memory predecessor abort handoff

* fix: restore controls for recovered queued messages

* test: cover recovered queue controls

* fix: close final steering review gaps
2026-07-31 20:07:56 -04:00
Danny Avila
60ca751a7f
🧠 fix: Preserve Deferred Tool Schemas Across HITL Resume (#14552)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🧠 fix: Preserve deferred tool schemas across HITL resume

* 🧪 test: Harden deferred tool resume regression

* 📦 chore: bump @librechat/agents to v3.3.10
2026-07-31 14:06:13 -04:00
Danny Avila
78ec1940a2
🧹 fix: Clean Up MCP OAuth State Mappings on Uninstall + Reject Superseded Callbacks (#14549)
* 🧹 fix: Clean Up MCP OAuth State Mappings on Uninstall + Reject Superseded Callbacks

Disconnecting an OAuth MCP server deleted its mcp_oauth flows but left the
mcp_oauth_state:{state} mappings behind for the full TTL. Because flow ids
are deterministic (userId:serverName) and the CSRF token is HMAC(flowId), a
stale browser tab's callback could resolve its orphaned state to the NEXT
flow for the same server, pass CSRF, burn the fresh flow's one-shot CSRF
cookie, and fail the PKCE exchange, sabotaging the legitimate retry.

- Add MCPOAuthHandler.deleteFlowAndStateMapping: reads the flow's stored
  state and deletes the mapping before the flow (mapping-first so a crash
  between deletes fails closed instead of recreating the orphan)
- Route mcp_oauth deletions in clearStoredMCPOAuthState through the helper
  for both tenant-scoped and legacy flow ids
- Reject callbacks whose state does not match the resolved flow's stored
  state: the only control distinguishing a superseded attempt from the
  current one on a deterministic flow id

Fixes #14534

* fix: gate failFlow on state match in the OAuth error branch (Codex P1)

The provider-error branch failed the resolved flow on CSRF/session alone,
so a superseded error callback resolved through an orphaned mapping could
mark the current flow FAILED. Apply the same stored-state equality gate
before failFlow.

* fix: leave the flow in place when the state-mapping delete fails (Codex P2)

deleteFlow swallows storage errors and returns false, and
deleteStateMapping discarded that result, so a failed mapping delete
followed by a successful flow delete would silently recreate the orphan.
Surface the boolean from deleteStateMapping and throw from
deleteFlowAndStateMapping before touching the flow, so the caller's
allSettled warn branch fires and the next replacement retries both.

* fix: restore the state mapping when the flow delete fails (Codex P2)

The inverse partial failure of the round-3 fix: a successful mapping
delete followed by a silently failed flow delete left a PENDING flow
whose reused authorization URL could never resolve, dead-ending every
callback in invalid_state until the flow went stale. Check deleteFlow's
result, re-store the mapping on failure, and throw so the caller's
allSettled warn branch fires.

* fix: never leave a callback-capable flow behind on uninstall (Codex round 6)

Teardown runs after the server's tokens are deleted, so a preserved
flow+mapping pair (the round-3 early-throw path) let a lingering consent
tab complete the callback and recreate credentials post-uninstall. Now
that both callback branches gate on stored-state equality, an orphaned
mapping is the benign failure mode, so invert the order: delete the flow
first, attempt the mapping delete regardless, and reject when either
reports a storage failure. This supersedes the round-4 mapping restore,
which also preserved a callback-capable pair.

* fix: delete the flow even when its metadata read fails (Codex round 7)

A storage error on the initial getFlowState aborted teardown before any
delete ran, preserving the callback-capable flow after token deletion.
Tolerate the read failure, delete the flow blindly, skip the mapping it
could not identify (the callback gates neutralize the possible orphan),
and reject so the caller's warn branch fires.
2026-07-31 12:11:36 -04:00
Danny Avila
f5e8feba80
📦 chore: bump @librechat/agents to v3.3.9 (#14548) 2026-07-31 10:43:00 -04:00
Danny Avila
1e1de6eff9
🎯 fix: Exact Ask-Question Attribution via Interrupt tool_call_id (#14539)
The ask_user_question pause/answer stamps (server pause-time args stamp,
resume-time answer stamp, and the client mirror) targeted the newest
unanswered ask part by position. When a model emits several ask calls in
one turn, the interrupt's question and the user's answer land on the
wrong card.

@librechat/agents > 3.3.8 surfaces the interrupting call's tool_call_id
on the ask interrupt payload. All three stamps now target that id
exactly when present, keeping the positional fallback for older
payloads. The tool body passes config.toolCall.id through to
askUserQuestion via a typed alias that is a no-op on the pinned SDK and
lights up on the next dependency bump.

Companion to danny-avila/agents#366, which also fixes the underlying
dangling tool_use 400 (INVALID_TOOL_RESULTS) when one of the parallel
ask calls streams malformed args.
2026-07-31 10:11:08 -04:00
Danny Avila
8e165eb451
🔒 fix: Remove Owner Email from Agent owner_contact Fallback (#14541)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🔒 fix: Remove Owner Email from Agent `owner_contact` Fallback

The owner-contact fallback for agents without an explicit support_contact
exposed the owner's private account email to any VIEW-level caller via
GET /agents/:id and GET /agents. The fallback now resolves a display name
only (name/username/authorName): the User query no longer projects email,
the resolver never returns one, and the shared AgentOwnerContact type drops
the field. Emails are only served when the owner opts in via support_contact.

* 🔒 fix: Reject Email-Shaped Owner Display Names in Contact Fallback

OpenID and SAML strategies fall back to the account email for the user's
name and username when no display-name claims exist, so the name-only
owner fallback could still surface the email through those fields. The
resolver now rejects email-shaped display-name candidates entirely.

* 🔒 fix: Treat Any @-Containing Display Name as Email-Derived

RFC-5321 quoted local parts may contain whitespace and the User schema
email validator is an unanchored substring match, so such addresses can
reach the name/username fields via SSO fallbacks. Rejecting on '@'
presence covers every legal email form without re-fetching the account
email.
2026-07-30 23:46:22 -04:00
Jens Schumann
ad74a282d1
🪃 fix: Resolve User Vars Before the First Post-OAuth Reconnect (#14538)
* fix: resolve customUserVars before first post-OAuth-callback MCP reconnect

The OAuth callback route reconnects the user's MCP connection immediately
after storing new tokens, but never resolves customUserVars before doing
so - unlike the /reinitialize route a few hundred lines below, which does.
As a result, headers/oauth_headers templates like `{{MY_KEY}}` are sent
to the MCP server literally, unsubstituted, on this first connection
attempt, even though the user's value is already saved. The upstream
server rejects it as an invalid credential.

Fixes #14537

* refactor: share getServerCustomUserVars reader from @librechat/api

The mcp_-prefixed key shape was built by getUserMCPAuthMap but re-derived
by hand at each read site (a private helper in services/MCP.js, and the
new callback-route extraction). Export a reader from the same module that
owns the writer and reuse it at both sites, so the key shape has a single
source of truth.

* chore: sort destructured require members in routes/mcp.js

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-30 22:17:06 -04:00
Danny Avila
8af6414e13
🪟 fix: Surface MCP Initialization Errors (#14529) 2026-07-30 13:22:11 -04:00
Danny Avila
7bb6651883
🛑 feat: Preemptive Steer - Backend Interrupt & Steer (#14518)
* 🛑 feat: Preemptive Steer — server half (Interrupt & Steer, PR 2 of 3)

Lets the steer route ask the generating replica to seal its live model
stream at the next provider-safe boundary instead of waiting for a tool
step. The run is never aborted, job status never changes, the partial
answer is kept, and generation resumes in the same assistant message
after the injected steer. Consumes the SDK seam in @librechat/agents
(danny-avila/agents#335, #346).

Transport: IEventTransport gains a fenced emitPreempt/onPreempt pair
beside abort. RedisEventTransport fans PREEMPT out on the SAME events
channel and subscription (no new connection, key, or subscribe call);
onPreempt returns a registration-scoped unsubscribe with the same
replacement-safe state-identity guard onAbort uses. InMemory implements
neither — single-process preempt lives entirely in the runtime set.

Runtime state: RuntimeJobState carries the per-generation request set,
createdAt-fenced and capped at STEER_QUEUE_MAX_DEPTH, plus a bounded
`cleared` tombstone so a late cross-replica arm cannot resurrect a
request whose steer already drained. registerPreemptSubscription
mirrors the abort registration's double fence (runtime identity +
generation createdAt); releaseAbortSubscription retires BOTH listeners
and the armed set, so every terminal path drops preempt state for free.
Public surface: requestPreempt (arm + fenced publish, never a rejection
surface, never touches job status), isPreemptRequested (O(1)
level-triggered poll), noteSteersRemoved (drain/cancel bookkeeping +
fenced clear), clearPreemptRequests (empty-boundary disarm).

One drain body, two boundaries: createSteerDrainHook (PostToolBatch)
and createSteerPreemptBoundaryHook (PreemptBoundary) share
drainAndBuildInjections, so the two injection sites cannot drift — the
SDK's provider-safety argument rests on identical HumanMessage shapes.
The shared body builds injections incrementally under a swallow-all
catch (a mid-loop throw still injects what was applied — those parts
are already persisted), clears preempt requests in finally, and
disarms the generation when a boundary drains nothing.

Request path: POST /chat/steer accepts preempt: true. The guard ladder
is unchanged in order and in every status code. A preempt request is
NEVER a rejection reason — without the capability the steer still
enqueues and the 202 echoes preempt: false. Armed strictly after a
successful enqueue; cancel disarms. The capability is read from the
OWNING replica's recorded `preemptCapable` rather than the route
replica's own SDK probe, so a rolling deploy cannot label a steer
"interrupting" that the older owner will only inject at a tool step.

Durable label: SteerQueueItem.preempt → TPendingSteer.preempt, so a
parked/claimed/replayed chip keeps its wording.

Run wiring: createRun registers the PreemptBoundary hook and threads
RunConfig.preemption, both gated on isSteerPreemptSupported() — a
separate probe from isSteeringSupported(), so the client affordance can
never arm against an SDK that only injects at tool boundaries.
buildSteerWiring builds both hooks from one shared closures object, so
preemption survives HITL pause/resume for free.

Honest finalization: an empty preempt boundary persists and emits with
unfinished: true — the same contract an abort gets — re-marked
explicitly because BaseClient has already saved the row as
unfinished: false by that point.

Not changed: no new job status, store method, Lua, SSE event type,
endpoint, or authorization surface. abortJob, completeJob,
transitionStatus, closeAndDrainSteers, getResumeState, emitChunk,
applySteerPart and the whole abort path are untouched.

Tests: 120 packages/api steering specs (preempt lifecycle, tombstone,
fences, caps, terminal release, both-boundary drain parity,
level-triggered poll, request/cancel arming, owner-capability
degradation) plus 5 in api for buildSteerWiring gating, and 2
Redis-gated cross-replica transport specs.

* 🔒 fix: Codex round 2 — evict tombstones, scope the empty-boundary disarm, honest resumes

All four server findings were fresh consequences of the round-1 fixes,
which is the review doing exactly what it should.

- Tombstone cap refused new entries instead of evicting. Every drained
  or cancelled steer is tombstoned, not just preempting ones, so a
  generation that processed 20 steers exhausted the set and the
  late-arm race resurfaced silently. Now evicts oldest-first (Set
  iteration is insertion-ordered), with the budget named
  PREEMPT_TOMBSTONE_MAX rather than an inline expression.

- The empty-boundary disarm I added in round 1 wiped the generation's
  ENTIRE armed set. A second steer can enqueue and arm between the
  atomic drain returning empty and the disarm running — that arm is
  backed by a live, uninjected queue item and must survive. The drain
  now snapshots the armed ids BEFORE draining
  (getArmedPreemptIds) and clearPreemptRequests takes an explicit id
  list instead of clearing everything.

- HITL resume finalized with a hardcoded unfinished: false. The
  boundary hook is re-registered on resume via buildSteerWiring, so a
  resumed segment can end on an empty preempt boundary exactly like a
  fresh one; finalizeResumedTurn now reads getPreemptStats() and the
  halt reason, matching the normal request path.

- Ownership moves on resume, so the job's recorded preemptCapable must
  describe the replica that will actually generate. Refreshed before
  resumeCompletion; a job created on a capable replica that resumes on
  an older one during a rolling deploy no longer acknowledges steers as
  interrupting.

Tests: +3 (scoped disarm sparing a post-snapshot arm, oldest-first
tombstone eviction, id-list disarm). 122 packages/api steering specs
green.

* 🚨 fix: Codex round 3 — deserialize preemptCapable from Redis (feature was dead under Redis)

The P1 here is the most consequential defect in the whole feature, and
it was introduced by round 1's own capability fix.

- `RedisJobStore.serializeJob` writes booleans generically, so
  `preemptCapable` reached Redis — but `deserializeJob` is an EXPLICIT
  field map and had no line for it. Every `getJob()` therefore dropped
  the flag, `job.metadata.preemptCapable` was always undefined, and
  `handleSteerRequest` computed `preemptArmed: false` unconditionally.
  Interrupt & steer would have silently degraded to ordinary
  tool-boundary steering in EVERY Redis deployment — i.e. the feature
  shipping as a no-op in production while passing every in-memory test.
  Now deserialized, with a round-trip assertion in the metadata spec
  that fails (`Received: undefined`) against the unfixed store.

- The resume capability refresh moved from just-before
  `resumeCompletion` to immediately after `approvals.resolve` claims
  the run. That call already flips the job back to `running`, so the
  steer route accepts requests from that instant; leaving the refresh
  135 lines later (across the whole client reconstruction) left a real
  window where a steer read the PREVIOUS owner's capability. Not the
  fully atomic transition Codex suggested — that reaches into the
  approvals Lua — but it shrinks the window from seconds to one await,
  which is proportionate for a label-accuracy issue.

Refuted: "avoid triggering preemption inside subagents". The premise —
that the run-wide poll can seal a subagent stream — does not hold
against the shipped SDK. Child graphs are constructed with
`subagentScope: true` (SubagentExecutor) and `preemption` is NOT
propagated into child inputs, while `canClaimPreemptSeal()` requires
`!subagentScope && preemption != null`. Both conditions fail
independently, so a subagent can never claim a seal and the boundary
cannot fire with `agentId` set. The `input.agentId != null` guard in
the hook is defensive depth, not the thing standing between us and the
described failure.

140 packages/api specs green.

* 🔁 fix: Codex round 4 — re-arm durable interrupt steers when resume moves owners

- An arm lives only in the owning replica's runtime plus a transient
  pub/sub message, while the steer's `preempt` flag is durable on the
  queue item. A HITL resume landing on a different replica therefore
  started with an empty armed set and a poll stuck false, so an
  interrupt the user had already been ACKed for silently waited for an
  ordinary tool boundary. New `GenerationJobManager.rearmQueuedPreempts`
  rebuilds the armed set by peeking the durable queue (fenced on the
  generation) and re-arming every item flagged `preempt`; resume calls
  it right after claiming. Safe by construction: every item peeked is
  still queued, so no drained steer can be resurrected.

- Capability-refresh failure now logs at error rather than warn, but
  deliberately does NOT fail the resume — see the reply on that thread.

Tests: +2 (rebuild from queue arms only the flagged item and reports
the count; a stale generation arms nothing). 124 packages/api steering
specs green.

* 📡 fix: Codex round 5 — acknowledge only what was actually armed

- A cross-replica arm was fire-and-forget: `emitPreempt` logged its own
  publish failure and `requestPreempt` returned void, so the route
  answered `preempt: true` even when the owner never armed a poll. The
  steer still injected at the next tool boundary, but the chip claimed
  an interrupt that could not happen — and unlike HITL resume, an
  ordinary running generation had no durable reconciliation to recover
  it.

  `emitPreempt` now resolves to the subscriber count and rejects on
  failure; `requestPreempt` is async and returns whether the arm truly
  landed (owned locally, or delivered to at least one subscriber). The
  202 reports THAT rather than what was asked for, so the chip relabels
  to ordinary steering exactly as it does for a capability-degraded
  deployment. Errors are swallowed into `false` — an unarmed interrupt
  is a downgrade, never a failed steer.

- The owner capability is re-read immediately before enqueue rather
  than reused from the top of the guard ladder. `checkAgentAccess` and
  file resolution are awaits, so a request can span an entire HITL
  pause/resume that moves ownership to a replica with different
  capability and rewrites that very flag. Only paid for by requests
  that actually asked to interrupt.

Tests: +3 (not-armed when the publish reaches nobody; armed when this
replica owns the generation; a throwing publish downgrades instead of
propagating). 127 packages/api steering specs green.

* 🎯 fix: Codex round 6 — real ownership, confirmed disarms, and a CI regression of my own

Three review findings plus three CI failures the round-5 commit caused.

Review:
- Ownership came from `runtimeState`, which a cross-replica `getJob`
  populates with a FACADE runtime on any replica that merely read the
  job. Matching `createdAt` therefore proved only "we looked at this
  job", so a non-owner could arm nothing and report success. Ownership
  now comes from `ownedJobs`, the actual owner map.
- `armPreemptIds` returns how many ids it accepted, and a local arm is
  only reported as armed when one was. A tombstoned id (its steer
  drained at an ordinary boundary mid-request) no longer answers
  `preempt: true` for an interrupt that cannot happen.
- The cancel disarm is awaited. A dropped clear is worse than a dropped
  arm: the owner keeps a level-triggered request for a steer that no
  longer exists, seals its next chunk and truncates an unrelated
  answer. The boundary drain's own call stays non-blocking — there the
  owner is local, so the disarm is already effective and awaiting the
  informational publish would only delay injection.
- Subscriber count is NOT read as proof of owner receipt: the count
  includes this replica's own facade subscription. A successful publish
  reports armed, a rejected one does not. Documented rather than
  papered over — see the acknowledgement-semantics note on the PR.

CI regressions from round 5, all mine:
- `registerPreemptSubscription` was AWAITED at both runtime-init sites,
  so job creation blocked on a second Redis channel subscription and
  hung when that subscribe was slow. Abort is awaited because a missed
  abort strands a run; a missed preempt only degrades that steer to the
  next tool boundary, so it now registers without gating createJob.
- Two api specs mocked `@librechat/api` without the newly imported
  `isSteerPreemptSupported`, so the call threw before createJob; and one
  exact-match assertion needed the new `preemptCapable` metadata field.
- My own Redis integration spec asserted arm-before-clear ordering,
  which two publishes carry no guarantee of — the receiving tombstone
  exists precisely because of that. Now asserts delivery and payload
  fidelity, order-independent.

158 packages/api specs, 27 api specs green.

* 🧭 fix: Codex round 7 — settle the acknowledgement semantics (Option A)

Round 7's second finding is the incoherence I flagged on the PR: the
route persisted `preempt: true` on the durable queue item while
returning `preempt: false` when delivery could not be confirmed. Those
two then disagreed, and `rearmQueuedPreempts` trusts the DURABLE one —
so a resumed owner would honour an interrupt the client had explicitly
been told degraded to ordinary steering.

Rather than patch the disagreement, this settles the meaning:

`preempt` in the 202 means "queued as an interrupt request", NOT "a
seal is guaranteed". It mirrors `SteerQueueItem.preempt` exactly, so
the response, the durable record, and the resume-time re-arm can never
disagree. The gates that ARE knowable stay — the owner's recorded
capability and a successful enqueue. Everything past that degrades to
the documented fallback of injecting at the next tool boundary.

A route cannot synchronously know whether another replica will seal:
proving it needs a correlated request/response over pub-sub, and even
that only proves the owner heard, not that it is still streaming when
the arm lands. Four rounds of tightening this boolean each surfaced a
narrower case; the sequence does not converge, so the invariant is now
"the flag describes the durable decision" and an unconfirmed arm logs a
warning instead of rewriting the answer.

Also from this round: a failed disarm publish is retried once and its
outcome reported. `handleSteerCancel` keeps `removed: true` — the steer
really did leave the queue, and saying otherwise would make the client
re-show a chip for a steer that can never arrive — and adds
`disarmed: false` so the residual risk is visible rather than swallowed.
Damage stays bounded regardless: the empty-boundary self-clear disarms
the generation after a single seal.

Tests: +1 pinning the response/durable-flag invariant. 159
packages/api specs green.

* 🧹 fix: Codex round 8 — remove the unverifiable disarm signal

Round 8 found the same over-promise on the disarm side that round 7
corrected on the arm side, so this applies the same answer rather than
patching around it.

The `disarmed: false` field added in round 7 was both unreliable and
unused: a resolved publish is not proof the owner heard it (the
delivery count includes this replica's own facade subscription), and it
was never threaded into `CancelSteerResponse` or read by any client. A
signal that claims a certainty the transport cannot provide is worse
than no signal — it invites callers to trust it.

Removed from the response. The retry stays, because it genuinely
reduces the failure rate, and `noteSteersRemoved` still returns whether
the publish succeeded FOR LOGGING, now documented explicitly as
"published without error", not "the owner disarmed".

Disarm is best effort with a bounded, self-healing failure: if the
clear is lost the owner seals once, the empty-boundary self-clear
disarms the generation, and the turn is persisted `unfinished: true`
rather than silently truncated. Tightening that further needs a
correlated request/response over pub-sub with a timeout — noted on the
PR as the deliberate boundary of this design rather than an oversight.

130 packages/api steering specs green.

* 🧽 fix: Codex round 9 — spend snapshot arms on nonempty drains too

The round-6 scoping fix only cleared the pre-drain snapshot when the
drain came back EMPTY. On a nonempty drain the `finally` cleared just
the drained ids, so a stale arm — typically a cancel whose
cross-replica clear was lost — survived the boundary. It would then
immediately seal the continuation meant to answer the steer that had
just been injected, and land on an empty boundary as
`preempt_incomplete`: the interrupt appears to work, and the answer to
it is truncated.

A boundary that runs has spent its seal, so everything armed at
snapshot time is spent whether or not it came back from the drain. The
`finally` now clears the union of the snapshot and the drained ids.
Arms that land AFTER the snapshot are still spared — their queue items
are live and uninjected, which is the property round 6 added.

Also fixes an api-workspace CI failure of mine: `resume.spec.js` stubs
`GenerationJobManager` wholesale, and the round-3/4 resume work added
two calls (`updateMetadata`, `rearmQueuedPreempts`) the stub did not
define, so 34 specs threw. Stub extended.

Tests: +2 (a nonempty drain clears a stale snapshot arm; a nonempty
drain spares an arm that landed mid-drain). Counterfactually verified —
the stale-arm spec fails against the unfixed drain. 132 packages/api
specs, 60 resume specs green.

* fix: never let a failed preempt subscription reject into the void

registerPreemptSubscription is called detached at both sites, so a
rejected Redis SUBSCRIBE became an unhandled rejection — process-fatal
under Node's default --unhandled-rejections=throw. The comment already
promised this path merely degrades steering; it now does.

Swallowed and logged inside the registration rather than at each call
site, so a future third caller cannot reintroduce the trap. Losing the
channel costs this generation's cross-replica preempts, not the server:
same-replica arming is runtime state and still works, and remote arms
fall back to the next tool boundary.

Verified counterfactually — the new spec surfaces SUBSCRIBE failed as an
unhandled rejection against the unfixed registration.

* docs: state the real blast radius of a failed preempt subscription

LibreChat's own entrypoints install a global unhandledRejection handler
that logs and keeps serving, so the escaping rejection this guards was
never fatal to this server — only to another consumer of @librechat/api
that installs no handler. The fix stands either way; the comment just
should not overstate what it prevents.

* test: cover the cross-replica preempt hop with two manager instances

Every other preempt test runs against a single manager, so the hop that
actually carries an interrupt in production had no coverage: the steer POST
lands on whichever replica the balancer picks, which is usually not the one
generating. Non-owner publishes, owner arms, owner's level-triggered poll
flips — none of that was exercised end to end.

Two GenerationJobManagerClass instances are a faithful replica pair here.
runtimeState and ownedJobs are private instance fields, there is no
module-level mutable state between them, and createStreamServices duplicates
a dedicated subscriber connection per call, so separate OS processes would
exercise the same objects over the same Redis.

Both assertions verified counterfactually against real Redis:
- Deleting the preemptCapable deserialization in RedisJobStore fails this
  with 'Expected: true, Received: undefined' — the exact P1 that shipped past
  every in-memory test and would have made the feature a silent no-op on
  every Redis deployment.
- Dropping the non-owner arm publish fails it with 'Received: false'.

* test: remove the fixed sleeps and vacuity from the cross-replica preempt test

Codex round 11, both findings, both on the test I added last commit.

P2 — the 300ms waits were load-bearing. Redis pub/sub never replays and the
owner's SUBSCRIBE is detached, so on a slow CI worker the publish could land
before anyone was listening and the test would fail against correct code.
Now it republishes until the owner's state converges, which is safe because
arms and clears are idempotent set writes keyed by steerId. Side effect: the
tests got ~10x faster (85ms/57ms vs 929ms/606ms) since they finish on
delivery rather than on a timer.

P3 — afterEach destroyed only the transports, leaving each manager alive in
its own cleanup-interval closure, still working against a dead transport.
Now tracks the managers and awaits destroy(), which disposes the job store
and its timer too. Matches how the rest of this file cleans up.

Fixing the sleeps exposed a third problem codex did not flag: the stale-arm
test could pass vacuously, because an undelivered arm and a fenced one look
identical. It now brackets the stale publish between two control arms — the
first proves the owner is listening before the stale one is sent, the second
proves it has had its chance to arrive.

Verified counterfactually against real Redis, and stable over 5 runs:
- dropping the preemptCapable deserialization fails with 'Received: undefined'
- dropping the non-owner arm publish times out both tests
- removing the generation fence fails the stale test with
  ["control-before", "steer-stale", "control-after"] — which also confirms
  the bracketing orders as intended rather than by luck

* fix: gate interrupt on the OWNER's capability alone, not the route's

Codex round 12. The comment above this gate already said 'the OWNER's
recorded capability, not this replica's probe' — and then the code ANDed in
isSteerPreemptSupported(), which is exactly this replica's probe. The
contradiction dates to the original commit; round 6 made the gate
owner-scoped and wrote that comment without removing the local conjunct.

The route never seals. It enqueues and publishes an arm, neither of which
touches the SDK, so during a rolling deploy a steer landing on an
un-upgraded replica silently lost its interrupt even though the owner could
seal. When the route IS the owner the probe is redundant anyway: the flag it
would consult is the one this process wrote at createJob.

The real degradation path is unchanged and still tested — an owner that
recorded no capability relabels to an ordinary steer. The test that pinned
the local probe asserted an impossible same-replica state (capable metadata
plus an incapable local SDK, when the metadata is written from that probe);
it now pins the mixed-SDK direction instead, and fails with
'Expected: true, Received: false' if the probe is put back.

* fix: reconcile arms at handover, and stop holding the 202 on a publish

Codex round 13, two of three findings.

P2 — rearmQueuedPreempts only ever ADDED. A replica that merely read the job
still installs a facade runtime and subscribes, so it can accept an arm and
then miss the best-effort clear that follows the drain. HITL resume promotes
that facade to owner, the union keeps the orphan, and the first resumed
stream seals on a steer no longer in the queue, drains nothing, and
truncates the resumed answer as preempt_incomplete. acquireResumedJobOwnership
only sets ownedJobs, so nothing else was clearing it. The durable queue is
the sole authority at a handover: arms it does not back are now disarmed and
tombstoned, so an in-flight publish cannot revive them either.

Worth recording that my own independent review raised this and my verifier
refuted it. Codex found it separately; two reviewers converging should have
outweighed one refutation.

P2 — the route awaited the arm publish before answering. The 202 reports
capability, not delivery, so the await could not change the response; it only
exposed the caller to Redis latency after the queue item was already durable.
A client that times out and retries mints a second steer while the first
stays queued, injecting the same instruction twice, whereas a lost publish
merely takes the tool-boundary fallback. Detached, with both outcomes logged.

All three tests verified counterfactually: union-only rearm fails the two new
handover specs, and re-awaiting the publish hangs the stalled-publish spec
until jest kills it.

* fix: snapshot arms before reading the queue at handover

Codex round 14 — a regression from my own round-13 fix, and a worse failure
than the one it corrected.

Round 13 read the durable queue first, then tombstoned any armed id the
snapshot did not back. But approvals.resolve reopens steering before
reconciliation runs, so another replica can commit a preempt steer and
publish its arm while the peek is in flight. That arm is then present locally
but absent from a snapshot taken before the steer existed, so a LIVE
interrupt the route already acknowledged got dropped — and tombstoned, which
blocks the re-arm, making it unrecoverable rather than merely late.

Fixed by inverting the two reads rather than by locking or paying a second
round trip. A steer is durably enqueued BEFORE its arm is published, so any
id in an arms-first snapshot was already queued when it was armed, and the
later peek must observe it unless it has since drained — which is exactly the
orphan this reconciliation exists to drop. Arms landing after the snapshot
are simply not candidates.

Also re-checks runtime identity across the await, since the generation can be
replaced while the queue read is in flight.

New spec injects a steer + arm during the peek and verifies it survives;
against the round-13 ordering it fails with Received array: [].

* fix: bound the cancel disarm wait and fence enqueue to its generation

Codex round 15.

P2 — the cancel awaited its disarm publish unbounded. ioredis queues
commands during an outage rather than rejecting, so that await could hang for
the length of the outage with the steer ALREADY durably cancelled; a client
that gives up then treats the cancel as failed and restores a chip for a
steer that can never produce an applied event. Every successful cancel
publishes, so ordinary steers were exposed too, not only preemptive ones.
Now bounded at 1s, with the publish continuing behind it — its retry and
logging are unchanged, it is just no longer in front of the response. This is
the sibling of round 13's arm-publish finding; I fixed one path and left this
one.

P3 — enqueue was not fenced to the generation the capability decision was
made against. The access checks, file resolution and owner re-read are all
awaits, so the run can be replaced before the enqueue: the item then lands on
the REPLACEMENT queue while the durable preempt flag and the arm still name
the previous epoch, the arm is fenced out at the owner, and the 202 promises
an interrupt that cannot happen. enqueueSteer now takes an expected
generation, mirroring drain/peek, and the Redis path enforces it inside
STEER_ENQUEUE_LUA so the check is atomic with the push rather than racing it.

All three new specs verified counterfactually, including the Lua guard
against real Redis (removing it returns 1 where -1 is required).

* fix: fence the steer to its authorized generation, bound resume setup, keep preempt when Redis parks

Codex round 16, all three findings.

P2 — round 15 fenced the enqueue to owner.createdAt, the RE-READ job. Every
guard above it (ownership, tenant, paused-state, agent ACL) ran against the
job read at the top, so if the run was replaced during those awaits the fence
happily accepted the steer into a generation the request was never authorized
against, carrying the wrong agent's metadata. Now rejects on any mismatch
between the validated job and the re-read.

P2 — resume awaited its steering bookkeeping unbounded, after
approvals.resolve had consumed the action and flipped the job to running, and
outside the resume lifecycle's own try/finally. .catch does not fire on a
promise that never settles, which is what ioredis produces during an outage,
so the client times out, its retry gets a 409 for a spent action, and no
cleanup runs. Bounded at 1s with the writes finishing in the background.

P3 — Redis parks leftover steers inside its terminal-transition Lua, which
projects item fields one by one, so preempt was silently dropped and a steer
recovered from /chat/status lost its interrupting label. Added to both
projections.

All three verified counterfactually, two against real Redis. Worth recording
that my first version of the generation-mismatch test was VACUOUS — it faked
a createdAt matching no live job, so the round-15 enqueue fence rejected it
for the wrong reason and the test passed with the guard removed. Rewritten to
replace the run for real; it now fails with 'Expected 404, Received 202'.
2026-07-30 12:36:13 -04:00
Danny Avila
e8a943c7e8
📦 chore: bump @librechat/agents to v3.3.8 (#14525)
Activates activity-label continuity end to end. The host side landed with
the activity-groups feature (#14391) — the per-run accumulator that reads
committed headers at request-build time, the `previousLabels` payload
field, resume seeding, and the bridge passthrough — but the SDK had no
field to receive them, so the traced generation path ignored the context
and only the direct fallback rendered it. v3.3.8 carries
danny-avila/agents#356, which adds `previousLabels` to
`RunActivityLabelOptions` and renders it as the label prompt's first
section (capped at 3, each entry whitespace-collapsed and clipped at 200
chars so one malformed header cannot forge prompt sections or inflate
every later request in the run).

Effect: consecutive same-activity batches now extend the run's story
instead of restating a line already on screen, and setup batches stop
being labeled with conclusions their tools had not yet established.

Also included between v3.3.7 and v3.3.8: danny-avila/agents#354, which
anchors summary coverage to a source message id.

Lockfile carries no transitive churn — 3.3.8's dependency tree is
identical to 3.3.7's.

Verified against the installed package: 73 activityLabels tests and 218
api agents-controller tests pass, `tsc --noEmit` clean on packages/api,
and the published build renders the capped, sanitized header section
(oversized labels clipped, embedded newlines flattened to inert text).
2026-07-30 08:23:09 -04:00
Dustin Healy
af795be0c2
🪢 feat: Langfuse Fanout Connection Setting (#14108)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: encrypt tenant Langfuse secret in admin config

Add generic per-field secret encryption to the admin config layer: registered
secret paths (langfuse.secretKey) are encrypted with encryptV3 on write and a
non-secret fingerprint companion is stored. Admin config reads (base + per
principal) redact registered secrets so they are never returned; the fingerprint
is kept so the UI can show which key is configured.

The Langfuse fanout read path decrypts the tenant secret before export. Adds
secretKeyFingerprint to langfuseConfigSchema and tests for the encrypt/redact
policy.

* fix(api): secure admin config secret handling

* fix(api): preserve encrypted langfuse config secrets

* fix(api): couple config secret fingerprint deletion

* fix(api): read langfuse fanout collector url from env

* fix(api): display langfuse secret key hint

* fix(api): remove langfuse secret fingerprint breadcrumbs

* fix(api): use langfuse destination keys for tenant config

* fix(api): remove langfuse config compatibility fallbacks

* refactor(api): simplify langfuse secret helpers

* refactor(api): simplify langfuse config secret handling

* feat: in-app Langfuse connection settings panel

Add a discoverable, admin-gated Langfuse connection panel inside LibreChat
Settings (Dify-style): enable toggle, host, public key, masked write-only secret,
configured-key fingerprint, and a test-connection action. Backed by a dedicated
/api/admin/langfuse/connection endpoint that encrypts the secret at rest, returns
metadata plus fingerprint on read, and validates credentials. Builds on the
per-field encryption and fanout decrypt from the langfuse-config-encryption branch.

* refactor: align Langfuse secret field to CustomUserVars pattern

Use the established SecretInput plus Set/Unset state pill (com_ui_set/com_ui_unset)
from the MCP CustomUserVars UI for the saved-secret state, instead of a bespoke
masked input.

* fix: drop em dash from saved-secret placeholder

* feat: show loading state on Langfuse test connection button

* feat: gate in-app Langfuse settings on fanout config and admin role

* test: align Langfuse connection spec with SecretInput refactor

* feat(langfuse): refine tenant connection controls

* fix(admin): refine Langfuse connection verification

* fix(langfuse): refine tenant connection settings

* fix(langfuse): simplify export enablement controls

* fix(langfuse): validate tenant export configuration

* fix(langfuse): align startup fanout gate

* fix(admin): time out Langfuse verification

* fix(ui): rename Langfuse connection setting

* fix(admin): enforce Langfuse config capability

* feat(langfuse): require explicit tenant export activation

* feat(langfuse): support single-tenant connection settings

* fix(i18n): remove obsolete integrations label

* fix(langfuse): authenticate ingestion verification

* fix(langfuse): validate public key independently

* fix(langfuse): localize connection errors

* perf(config): skip Langfuse checks for non-admins

* fix(langfuse): preserve trace sampling for feedback

* test(langfuse): fix feedback sampling fixture

* fix(langfuse): align secret preview field

* fix(langfuse): harden connection settings state

* fix(langfuse): preserve trace destination state

* fix(langfuse): enforce tenant-wide routing invariants

* fix(langfuse): preserve verified connection invariants

* fix(langfuse): preserve stable project identity

* fix(langfuse): warm project identity asynchronously

---------

Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 18:33:10 -04:00