Lazy code-env provisioning was gated on a loaded LIBRECHAT_CODE_API_KEY, so
JWT-auth deployments (which mint bearer tokens via getCodeApiAuthHeaders and
need no legacy key) silently never provisioned attachments. The gate now
accepts either auth mode and checkSessionsAlive composes X-API-Key with the
minted bearer headers.
Separately, pre-categorization added every codeEnvRef file to
processedResourceFiles before the provisioning loop ran, so the staleness
branch was unreachable and expired sandbox refs were never cleared or
re-provisioned. Staleness is now repaired ahead of the processed guard,
clearing both the legacy ref and its route entry so getCodeEnvRefs cannot
resolve the dead session.
Round-3 Codex follow-ups completing the round-2 fixes:
- The file schema/type dropped codeEnvRef.provisionedAt, so the liveness fast-path
was dead after reload. Add provisionedAt to CodeEnvRef + the Mongoose subschema.
- Lazily provisioned agent-scoped file_search files were only added to
tool_resources.file_search.files, so primeFiles treated them as user attachments
and queried without entity_id, missing the agent-scoped vectors. Add agent-scoped
files to file_ids instead so they are queried with entity_id.
Finding B stamps codeEnvRef with a provisionedAt timestamp at code-env upload
time; update the four exact-shape assertions to expect it (expect.any(Number)).
Mock getAgent in the process.spec ~/models mock (my C fix calls db.getAgent) and
add a test: an agent upload (endpoint=agents) whose provider has a none fallback
resolves llmDeliveryPath from the provider config, not the generic agents config.
Addresses two Codex round-2 P1 findings:
- Liveness: checkSessionsAlive trusted file.updatedAt to skip the code-env live
check, but updateFilesUsage bumps updatedAt on resend, so a usage-touched file
with an expired sandbox session was wrongly treated as alive. Stamp codeEnvRef
with provisionedAt at upload time and gate the fast-path on that instead; refs
without the marker fall through to a live check.
- Routing: an agent upload carries endpoint=agents, so provider-specific
defaultLLMDeliveryPath overrides (endpoints.<Provider>) were ignored and a
none/tool-only file could be persisted as text/provider. Resolve the file config
from the agent's own provider when agent_id is present.
Addresses Codex P1 findings on the lazy provisioning path:
- Scope per file like the direct upload path: current-message chat attachments
(context=message_attachment) provision to the user's code sandbox / unscoped
vector index (entity_id undefined); only agent setup files use entity_id=agentId.
Previously every lazily provisioned file was agent-scoped, so a user's chat
attachment landed in the agent sandbox and file_search queries (unscoped for
fromAgent=false) missed the agent-scoped embedding.
- Surface provisioned files to the tool loaded immediately after by adding them to
ctx.tool_resources.<resource>.files, which primeCodeFiles/primeFiles read. Before,
a freshly provisioned unified upload was invisible to the first code/file_search
call even though provisioning had completed.
provisionFiles only treated execute_code / run_tools_with_code as code execution,
but that capability now expands into bash_tool / read_file / run_tools_with_bash.
So a unified upload in provisionState.codeEnvFiles was never provisioned before the
first bash_tool/read_file call — the file was missing from the sandbox (Codex P1).
Broaden the needsCode guard to the current tool names. The lazy-provisioning e2e
now emits the actually-advertised code-exec tool (bash_tool) so it exercises the
real path. Removes the temporary diagnostics.
EnvVar.CODE_API_KEY was removed from @librechat/agents, so loadCodeApiKey
resolved authFields to [undefined], threw on undefined.split in loadAuthValues,
and lazy code-env provisioning silently bailed with a warning. Read the
canonical LIBRECHAT_CODE_API_KEY (symmetric with LIBRECHAT_CODE_BASEURL) and
degrade gracefully when unset.
Rebase onto current dev brought in the metadata.fileIdentifier →
metadata.codeEnvRef migration (HEAD uploadCodeEnvFile now returns
{ storage_session_id, file_id } and requires kind/id). Update the
unified-upload code paths to match:
- provision.js: provisionToCodeEnv now derives kind/id from entity_id,
calls uploadCodeEnvFile with the new signature, and returns codeEnvRef
- checkSessionsAlive/checkCodeEnvFileAlive: read storage_session_id and
remote file_id from metadata.codeEnvRef instead of parsing the legacy
fileIdentifier string
- resources.ts: primeResources gates on metadata.codeEnvRef and clears
it on staleness; TProvisionToCodeEnv reflects the new return shape
- initialize.js: provisionFiles closure destructures codeEnvRef
- process.spec.js: align two legacyFileUploadUX tests with the
endpoint-level check landed in 7384947 and update the execute_code
expectation to the codeEnvRef metadata shape
- resources.test.ts: import FileSources for the typed source field and
guard the optional attachments map
Move file provisioning from eager (at chat-request start) to lazy
(at tool invocation time via ON_TOOL_EXECUTE). Files are now only
uploaded to code env / vector DB when the LLM actually calls the
respective tool.
- resources.ts: primeResources no longer provisions; computes
provisionState (which files need code env / vector DB uploads)
with staleness check and single credential load
- handlers.ts: add provisionFiles callback to ToolExecuteOptions,
called once per tool-call batch before execution
- initialize.ts: pass provisionState through InitializedAgent
- initialize.js: implement provisionFiles closure that provisions
files in parallel, batches DB updates, clears state after use;
store provisionState in agentToolContexts for all agent types
- loadCodeApiKey: load CODE_API_KEY once per request, pass to both
checkSessionsAlive and provisionToCodeEnv (was N+1 lookups)
- provisionToCodeEnv/provisionToVectorDB now return fileUpdate objects
instead of writing to DB immediately
- primeResources batches all DB updates via Promise.allSettled after
provisioning completes
- Remove updateFile import from provision.js (no longer writes directly)
Phase 2 fixes for the unified file experience:
- Add code env file staleness detection via batch session checks
(checkSessionsAlive) — groups files by session_id, one API call per
session, skips files updated within 6h safe window
- Parallelize file provisioning across files using Promise.allSettled
- Surface provisioning failures as warnings on InitializedAgent
- Fix temp file path safety (use file_id + extension, not raw filename)
- Fix inconsistent return types (normalize to [] instead of undefined)
- Wire checkSessionsAlive through initialize.js → initialize.ts →
primeResources
Introduces the foundation for a unified file upload experience where users
upload files once without choosing a tool_resource upfront. Files are stored
in the configured storage strategy and lazily provisioned to tool environments
(execute_code, file_search) at chat-request time based on agent capabilities.
Phase 1 - Schema + Server-Side Unified Upload:
- Add FileInteractionMode enum (text/provider/deferred/legacy) to fileConfigSchema
- Add defaultFileInteraction field to EndpointFileConfig and FileConfig types
- Update mergeFileConfig/mergeWithDefault to propagate the new field
- Modify processAgentFileUpload to support uploads without tool_resource
using effectiveToolResource resolved from config (default: deferred)
Phase 2 - Lazy Provisioning + Multi-Resource Support:
- Create provision.js with provisionToCodeEnv and provisionToVectorDB
- Extend primeResources with lazy provisioning step that provisions
deferred files to enabled tool environments at chat-request start
- Remove early returns in categorizeFileForToolResources so files can
exist in multiple tool_resources simultaneously
- Wire provisioning callbacks through initializeAgent dependency injection
* fix: recover missing MCP marketplace catalogs
* fix: make MCP catalog recovery passive
* test: type MCP catalog recovery fixtures
* fix: bound and back off passive MCP catalog recovery
Passive recovery runs inline on `GET /api/mcp/tools` and its results are
request-local by design, so every list request re-dialed the same cold
servers with the default connection timeout. Three limits keep that cost
proportional to what recovery can actually recover:
- Cap the discovery timeout at 5s instead of inheriting the connection
default (`initTimeout ?? 30s`); a server configured to connect faster
keeps its own shorter limit.
- Skip a server the config tier already marked `inspectionFailed`, leaving
it to that tier's retry window rather than re-dialing it per request.
- Skip a server whose declared `customUserVars` are unset, matching the
gate `reinitMCPServer` applies for issue #10969 — connecting without them
fails auth, so the attempt is spent for nothing.
Servers that still fail discovery enter a one-minute per-process cooldown,
which is what stops an unreachable server from being re-dialed by every
subsequent list request. A server that recovers clears its own entry, and
expired entries are swept at most once per window so the map stays bounded.
Skipped servers render exactly as they did before recovery existed: present
in the catalog with an empty tool list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B
* fix: bound passive MCP recovery by deadline, key cooldowns by config
Both follow-ups address the same mistake: recovery expressed its own
request-level constraints in terms borrowed from other layers.
`connectionTimeout` bounds one connection attempt, and
`MCPConnectionFactory.discoverToolsInternal` spends it twice — once on the
authenticated connection, then again in `attemptUnauthenticatedToolListing`
— so capping it bounded no total this layer could reason about. Recovery now
enforces its own wall-clock deadline per server with `withTimeout`, which
holds however many attempts the factory makes; `connectionTimeout` is left to
do only its own job, still honouring a shorter operator `initTimeout`. An
attempt abandoned by the deadline disposes its own connection when it
settles, and `Promise.race` keeps a handler on it, so a late rejection is
not unhandled.
A per-request budget now caps total recovery regardless of server count.
A server is dialed only if the remaining budget can fund a full deadline;
never dialing one is not evidence against it, so a skipped server records no
cooldown and a later request reaches it once those ahead are cached or
cooling down.
Cooldown identity now includes the publication generation — the same
effective-config identity the tool caches fence on — instead of just user and
server name. Correcting a server's URL or transport keys a new entry, so the
refetch the client issues on update is no longer skipped for up to a minute
by the previous configuration's failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B
* refactor: keep passive MCP recovery stateless and bounded by its own work
Reverts the cooldown, request budget and deadline race added in 257d5cf and
fc3e3c9, and keeps only the three stateless limits.
The tool cache refuses unfenced writes (`tools.ts`), and a discovery
connection owns no publication generation and is disposed, so a recovered
catalog cannot be retained by design. Those commits responded by building a
cache-shaped memory in front of it — per-process failure state, a scheduling
budget, an identity, an eviction sweep — and each round of review found
another way that hand-rolled cache differed from a real one: wrong identity
for configuration, wrong identity for credentials, no fairness across
requests, and a limiter slot released while its network operation was still
running. None of that machinery was asked for; all of it was compensation for
a result the architecture does not allow keeping.
Recovery is now stateless. It skips only what configuration alone proves
pointless — a server the config tier already marked `inspectionFailed`, and
one whose declared `customUserVars` are unset — and bounds the work itself
rather than racing it, so a limiter slot is held for exactly as long as its
network operation runs and the concurrency limit of three is real.
The attempt timeout is not a compromise: recovery exists for a server that is
reachable and authorized but whose catalog cache expired, and such a server
answers tools/list well inside 1.5s. Anything slower cannot be rescued here,
so failing fast costs nothing. The factory spends that value per attempt, so
a server's ceiling is it times the attempts made; the constant documents that
rather than hiding it behind a number tuned to today's attempt count.
Consequences that were bugs are now gone by construction: every cold server
is attempted on every request, so none is starved by those ahead of it, and
correcting a server's configuration or credentials takes effect on the next
refetch instead of waiting out a stale cooldown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B
* fix: correct the inspection-failure skip and bound catalog fan-out
Three fixes that belong to this layer; a fourth issue does not, and is
described below.
The `inspectionFailed` skip was too broad. `MCPServersInitializer` stores a
YAML server that was unreachable at startup via `addServerStub`, which stamps
`source: 'yaml'`, and only config-tier entries get the timed retry in
`ensureSingleConfigServer`. Skipping every failed stub therefore hid a
recoverable server from the marketplace permanently — the exact state this
recovery exists to escape. It now defers only `source === 'config'`, matching
what `reinitMCPServer` already does.
Plugin auth is read only when some cold server actually declares
`customUserVars`, and only for those servers. The common unauthenticated case
no longer pays a MongoDB round trip whose result nothing can consume.
Snapshot refreshes are now bounded by the same limiter as discovery. They are
not local reads: both connection paths reach `fetchOrderedToolsSnapshot` and
issue a real `tools/list`, so a cache reset across many servers previously
burst unbounded outbound requests while discovery was capped at three.
Not fixed here, because it cannot be: `connectionTimeout` does not bound
discovery. It covers `connection.connect()` only, and `fetchToolsSnapshot`
then applies its own `TOOLS_LIST_TIMEOUT_MS` (30s) to `tools/list`, so a
server that connects fast and stalls while listing still holds its slot for
that window. The factory also does not cancel a timed-out connect before
starting the unauthenticated fallback. Bounding this end to end needs a
deadline threaded through `MCPConnectionFactory` into both `connect()` and
`fetchToolsSnapshot()`, which is a change to shared connection machinery
rather than to this caller. The constant's comment now states what it does
and does not bound instead of implying an end-to-end guarantee.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B
* fix: thread live-session OBO context into passive catalog discovery
The merge of #15334 sources OBO tokens from the live OpenID session via
request-boundary closures. Passive catalog recovery is a discovery call
site too; without these options an OBO server whose stored token went
stale fails recovery — the exact cold-catalog class this PR fixes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B
---------
Co-authored-by: Claude <noreply@anthropic.com>
* 🧊 fix: Inline-refresh OpenID session tokens at MCP OBO call time
Resolves the walk-away failure mode where MCP tool calls using OBO auth
fail with "No valid OpenID access token is available for OBO exchange"
after a user idles past their access-token lifetime. The strategy-time
snapshot on `user.federatedTokens` could expire mid-stream before
`resolveOboToken` ran, while `req.session.openidTokens` carried a still-
valid (or refreshable) token that nothing read.
- New OpenIDSessionRefresh service: per-user single-flighted closure that
reads `req.session.openidTokens` at OBO time and inline-refreshes via
`openid-client.refreshTokenGrant` when expired (30s skew), persisting
via `req.session.save()`. No cookie writes (headers already flushed).
- `resolveOboToken` gains a required UpstreamTokenProvider parameter
(typed as `() => Promise<OIDCTokens | null>`, reusing the shared shape
from @librechat/data-schemas). Compile-time guarantee that every call
site is updated.
- New `session_refresh_failed` OboTokenResolutionReason distinguishes
"session expired and IdP rejected refresh" from "no upstream token
ever existed."
- `req` threaded through createMCPTool/createMCPTools/createToolInstance
to construct the closure with captured request, plus fail-closed
guards in MCPConnectionFactory.getOboTokens and MCPManager.callTool
when the closure isn't plumbed.
- Startup warning in MCPServersInitializer when OBO is configured but
OPENID_REUSE_TOKENS is unset (the strategy populating
user.federatedTokens is only registered under reuse, so OBO would
fail every call without it).
Tests: 16 new in OpenIDSessionRefresh.spec.js; obo.spec.ts extended
for the new param + error reason; wiring smoke tests in MCPManager,
MCPConnectionFactory, MCPServersInitializer, and MCP.spec.js.
* 🛡️ fix: Harden OBO inline-refresh against token type and session edge cases
- Token-preference asymmetry: live-token reuse and expires_at derivation
now strictly gate on the access_token, not the id_token. Added a
required `tokenPreference` parameter on isLiveSessionTokenStillValid,
buildOIDCTokensFromSession, and createOpenIDSessionTokenProvider
so every call site is explicit. Dropped the bogus id_token-exp
fallback in performIdpRefresh — id_token TTL is governed by IdP
session policy and would mark a short-lived access_token reusable
past its real lifetime.
- Missing req in /reinitialize route: the manual reconnect
endpoint now forwards req into reinitMCPServer, so OBO servers can
build a session-aware upstream-token closure instead of failing with
missing_upstream_token.
- Single-flight key collisions: composed key as
tenantId:openidIssuer:openidId:sessionId via getSingleFlightKey.
Concurrent calls in the same session still coalesce; separate sessions
never share an in-flight refresh, preventing refresh-token rotation
from breaking sibling sessions and preventing cross-tenant token
crossover when distinct users share an IdP sub.
- Opaque access token reuse): persist accessTokenExpiresAt
(unix seconds, from tokenset.expires_in) on each refresh AND on initial
login / SPA refresh in setOpenIDAuthTokens. New getAccessTokenExp
helper falls back to it when the access token isn't a JWT, avoiding
redundant inline refreshes for Microsoft Graph and Auth0 default
audiences.
- Log hygiene: the single-flight key (containing sessionId,
openidId, openidIssuer, tenantId) is now SHA-256-hashed in the
"Joining in-flight refresh" debug log. Preserves cross-line correlation
via a 12-char prefix without leaking credential or PII material.
Documented req.session.openidTokens shape contract via JSDoc typedef so
the new accessTokenExpiresAt field has a discoverable home alongside the
existing accessToken/idToken/refreshToken/expiresAt/lastRefreshedAt.
Tests: OpenIDSessionRefresh.spec.js up to 30 passing (added coverage for
opaque-token reuse, JWT-access-token-exp fallback, no-id_token-fallback
regression, cross-session no-coalesce, persistence on refresh, and a
guard against stale accessTokenExpiresAt carryover). AuthService.spec.js
adds two cases covering accessTokenExpiresAt persistence on login.
mcp.spec.js (route) gains a regression test asserting req flows into
reinitMCPServer.
* 🔍 fix: Detect OBO-only MCP admin config overrides
Admin Config overlays for YAML-defined MCP servers compare only
ADMIN_CONFIGURABLE_FIELDS to decide whether to lazy-init a config-tier override.
The OBO config field was added after that fingerprint list, so an override that
only added or changed `obo` was treated as unchanged YAML and skipped.
Include `obo` in the admin-configurable field list and add a regression test for
an OBO-only override.
* 🔊 fix: Mock MCP OAuth timeout in SDK integration test
MCPConnectionFactory.attemptToConnect reads mcpConfig.OAUTH_HANDLING_TIMEOUT
when building the OAuth connection timeout. The SDK OAuth integration test
mocked mcpConfig without that field, which made the timeout calculation produce
NaN and caused the test to fail before the OAuth refresh/start path completed.
Add OAUTH_HANDLING_TIMEOUT to the test mock.
* ♻️ refactor: Pass OBO upstream-token closure into MCP instead of req
Build the OpenID upstream-token provider at the request boundary and thread
only the closure through MCP handling, so the MCP service layer no longer
receives the raw Express request. The closure still reads/refreshes the live
session at tool-call time, preserving the walk-away recovery.
- Drop `req`/`capturedReq` from createMCPTools, createMCPTool, reconnectServer,
createToolInstance, and reinitMCPServer; forward `upstreamTokenProvider`
instead. Closure is constructed in loadTools, loadToolDefinitionsWrapper, and
the reinitialize route, where req/res are in scope.
- OBO: fall back to user.federatedTokens when the provider yields no live
session, so OIDC remote-agent calls (verified bearer, no session) still work.
- Inline refresh: mirror a rotated refresh token to the refreshToken cookie via
a shared setRefreshTokenCookie helper, guarded by !res.headersSent (no-op on
the streaming path; session copy stays authoritative).
- Single-flight: hydrate a joining request's own session from the resolved
tokens so a later OBO call doesn't replay a rotated-away refresh token.
Addresses owner feedback and three review findings.
* 🔒 fix: Recover OIDC refresh-token rotation after SSE OBO refresh
When an inline OBO refresh rotates the OpenID refresh token after SSE headers
have already been sent, the browser refreshToken cookie cannot be updated. Store
a short-lived encrypted bridge from the stale cookie token to the rotated token
so /api/auth/refresh can recover after express-session loss.
Use the signed openid_user_id cookie to load user context for bridge validation,
retry only on invalid_grant, and delete the bridge only after the bridged refresh
succeeds.
* 🔨 fix: hydrate joined OIDC refresh sessions with stable refresh tokens
Update single-flight OIDC refresh joiners whenever refreshed access token
state changes, even if the IdP keeps the refresh token unchanged.
This prevents joined requests from retaining stale accessToken or
accessTokenExpiresAt values and redundantly refreshing later in the same run.
* 🌉 Persist OIDC refresh-token recovery bridges in MongoDB
Store SSE OBO refresh-token recovery bridges in MongoDB instead of
process-local memory so /api/auth/refresh can recover after worker
restarts or cross-worker routing.
Derive bridge expiry from REFRESH_TOKEN_EXPIRY so the recovery window
matches the stale refreshToken cookie it repairs, and delete bridges
after successful recovery.
* 🤝 Coordinate OIDC inline refreshes across workers
Add a short-lived Mongo-backed refresh-flight record so concurrent
OBO refreshes for the same OpenID session do not redeem the same
rotating refresh token on different workers.
The winning worker performs the IdP refresh and stores an encrypted
result; joiners wait for that result, hydrate their request session,
and return without calling the IdP.
* ⚓ Keep OpenID marker cookies aligned on inline refresh
Refresh token_provider and openid_user_id with the same expiry as the
rotated refreshToken cookie when an inline OBO refresh can still write
headers.
Share the marker-cookie writer with the normal OpenID auth refresh path
so the fallback /api/auth/refresh branch continues to recognize valid
OpenID refresh tokens after session expiry.
* 🔑 fix: include refresh token in OIDC local refresh flight key
Key the process-local OIDC refresh coalescing by the current session
refresh token, matching the Mongo-backed flight key. This prevents a
request with a newly rotated token from joining an older pending refresh
and inheriting its failure/result.
* 🌉 fix: store OIDC refresh bridge without cookie response
Treat missing or non-cookie responses like headers-sent streaming
responses during inline OIDC refresh. When the IdP rotates the refresh
token and cookies cannot be written, persist a recovery bridge so a later
/auth/refresh can recover after session expiry.
* 🫙 fix: preserve stale OIDC cookie bridge key
Track the refresh token last written to the browser cookie separately
from the current session refresh token. When inline OIDC refreshes rotate
tokens without a writable response, keep bridging from the browser-stale
token directly to the latest session token.
* 🙌 fix: keep OIDC bridge recovery success on cleanup failure
Make refresh-token bridge cleanup best-effort after a bridged OIDC
refresh succeeds. A transient delete failure now logs a warning but does
not convert the already-refreshed session and cookies into a 403 response.
* 📦 test: Exclude RefreshTokenBridge from tenant-isolation coverage
Add RefreshTokenBridge to the tenant-isolation coverage allowlist because
refresh bridge lookups run during unauthenticated OpenID refresh recovery.
The controller first recovers user context from the signed OpenID marker
cookie, then the bridge methods apply explicit user and tenant filters.
Ambient tenant isolation would bind this recovery path to request-local
tenant context that is not available at the point the stale cookie is being
resolved
* ⚡ Fix OpenID refresh flight retry and marker hydration
Allow failed OpenID refresh flights to be reclaimed immediately instead of pinning transient errors.
Preserve the browser refresh-token marker when joined refreshes hydrate session tokens from a shared flight result.
Stabilize AuthService tests by isolating mocked module imports from prior suites.
* 🛠️ fix: centralize OBO identity scoping
Add shared auth identity helpers for app user ids, OpenID subjects,
tenant ids, and normalized OpenID issuers.
Thread a non-placeholder-visible OBO identity context from the real
request user through MCP connection, tool-call, reinit, and refresh
paths. Keep tenantId and openidIssuer out of createSafeUser so MCP
user placeholders do not expose those fields.
Scope OBO token cache and in-flight exchange keys by tenant, issuer,
OpenID subject, scopes, and a SHA-256 hash of the upstream assertion.
This prevents cross-tenant/cross-issuer collisions and avoids reusing
tokens minted from stale rotated assertions.
Use the shared identity helpers for OpenID refresh-flight keys and
refresh-token bridge recovery records so related OBO refresh paths share
the same identity normalization rules.
The helper is intended for auth-boundary and credential-cache code, not
as a blanket replacement for ordinary app user id ownership checks.
* 🛠️ fix: preserve OIDC refresh-token sync on save failures
Sync OpenID refresh-token cookie/bridge state before persisting the
session so a transient session-store failure cannot lose an IdP-rotated
refresh token.
Also trigger sync when the session refresh token differs from the
browser refresh-token marker, not only when the current grant rotates
the token. This lets later writable refreshes repair stale browser
cookies left behind by SSE refreshes.
Route refresh bridge identity through the shared identity helper with
the threaded OBO identity context, falling back to request/user context
when needed.
Add regression coverage for session-save failures, stale browser cookie
repair, non-writable bridge storage, and shared-helper identity fallback.
* 🛠️ fix: keep OIDC refresh bridge during recovery grace
After successful bridged refresh recovery, re-store the stale-cookie bridge
with a short grace TTL instead of deleting it immediately. This lets parallel
/api/auth/refresh requests that already sent the stale browser cookie recover
before they can observe the first response's Set-Cookie.
Retarget the bridge to the refresh token returned by the bridged retry so
B-to-C refresh-token rotation remains recoverable. The grace TTL is parsed with
math() and defaults to 60s, which shrinks the replay window from the original
REFRESH_TOKEN_EXPIRY bridge lifetime to the short recovery grace period.
Remove the now-unused explicit bridge delete path from the service and
data-schemas method surface. Add regression coverage for grace re-store,
identity symmetry, retry failure behavior, and same-key upsert replacement.
* 🛠️ fix: fail closed on OBO MCP user identity mismatch
Add an OBO-specific guard before MCP tool execution that requires the
effective invocation user and captured request user to both have ids and
to match. This prevents OBO tool calls from falling back to a separate
configurable.user_id identity after request-bound OBO context has already
been captured.
Keep the existing user id fallback behavior for non-OBO MCP calls.
Tests cover mismatched OBO users, missing user ids, and the matching-user
path ignoring a conflicting configurable.user_id.
* 🛠️ fix: Guard OpenID bridge retry user identity
Extract the shared OpenID refresh/user-resolution flow in AuthController
so the normal refresh path and bridge-recovery retry use the same grant,
claims, issuer, user lookup, and diagnostic logging code.
Preserve the existing path-specific behavior: the normal path still owns
migration updates and 401 login redirects, while the bridge retry still
falls through to the existing 403 invalid-token response.
Add a bridge-recovery guard that rejects retry results whose resolved
user id differs from the signed openid_user_id cookie before issuing
tokens or re-storing the grace bridge. Cover both the successful
matching-user recovery and the mismatched-user rejection.
* 🛠️ fix: type-safety polish on OBO data layer
Replace refresh token bridge query/update Record<string, unknown> usage
with typed Mongoose FilterQuery and UpdateQuery definitions.
Harden OpenID marker cookie JWT expiry handling by converting refresh
expiry milliseconds to integer seconds and rejecting invalid or
non-positive durations.
Add focused CSRF tests for fractional refresh expiry values and invalid
expiry configuration.
* 🛠️ fix: Bind OpenID session tokens to authenticated identity
Stamp OpenID session token state with the LibreChat user id, OpenID subject,
tenant id, and normalized issuer when tokens are stored.
Fail closed before OBO inline token reuse/refresh when the session token
identity does not match the current authenticated identity, preventing a stale
or mixed Express session from supplying another user's upstream assertion.
Also validate the normal /api/auth/refresh session-token reuse shortcut against
the signed marker-cookie user before returning cached session tokens.
Note: sessions created before this change carry no identity stamp and are
treated as a mismatch. This is self-healing — the reuse path forces a full IdP
refresh (which re-stamps the session) and the OBO path throws, surfacing as a
one-time re-authentication for active OBO users at deploy time. The session
re-stamps within one session lifetime (SESSION_EXPIRY, default 15 min).
* 🛠️ fix: Recover OpenID refresh token drift
Prefer the browser refresh-token cookie when it differs from the
server-side OpenID session state, and force a real IdP refresh in that
case instead of reusing stale session tokens.
Store a short-lived refresh-token bridge when inline OBO refresh writes
a rotated browser cookie but session persistence fails, so follow-up
refreshes can still recover from the old token.
Keep the bridge grace TTL centralized in RefreshTokenBridge so both
recovery paths use the same env-backed value.
Note: drift is measured against the last-synced browserRefreshToken
marker, so the SSE path (intentionally stale cookie, authoritative
session) does not false-positive. Sessions predating the marker have no
browserRefreshToken; for those, drift falls back to comparing the cookie
against the session refresh token and prefers the cookie on difference.
This is the same self-healing pre-change-session window as the identity
binding fix and re-syncs within one session lifetime.
Tests cover cookie/session drift selection, reusable-session bypass on
drift, bridge storage after session-save failure, and the shared bridge
constant wiring.
* 🛠️ fix: Harden OBO token caching and expiry handling
Reject malformed OBO grant responses before writing them to the exchanged-token cache so a missing access_token cannot poison the cache.
Store absolute expires_at values with cached OBO tokens and ignore legacy cache entries without usable expiry metadata. This keeps cached-token freshness based on the token’s real
remaining lifetime instead of reusing the original relative expires_in on cache hits.
Move OBO expiry normalization and skew helpers into packages/api and use them from both the JS exchange service and the TS MCP resolver. Apply a 30-second safety margin with a one-
second floor for short-lived tokens, covered by direct helper tests and caller-level regression tests.
Tests:
- packages/api: npm run build
- packages/api: npx jest src/mcp/oauth/expiry.spec.ts src/mcp/oauth/obo.spec.ts
- api: npx jest server/services/OboTokenService.spec.js
* 🛠️ fix: Harden OBO refresh-token bridge lookup and indexing
Reuse getValidOpenIDReuseUserId for the bridge-recovery user lookup in
refreshController instead of re-verifying openid_user_id inline. The shared
helper enforces the JWT_REFRESH_SECRET presence check and a strict
typeof payload.id === 'string' guard, rejecting tokens whose id claim is
present but not a string (e.g. a numeric id) that the inline check accepted.
Fail closed on issuer mismatch in getRefreshTokenBridge. Both the stored and
the expected issuer are now normalized and compared for equality, so a bridge
is recovered only when both sides agree (both absent, or both present and
equal after normalization). Previously the check was skipped whenever the
stored issuer was absent, allowing recovery across mismatched issuer context.
Drop the unused {oldRefreshTokenHash, userId, tenantId, openidIssuer} index
and the openidIssuer field on RefreshTokenBridgeQuery. The data-layer filter
only queries the 3-field {oldRefreshTokenHash, userId, tenantId} index; the
issuer is verified in application code, not the query. Hoist the repeated
model accessor into getRefreshTokenBridgeModel.
Note: issuer is now load-bearing for recovery. A bridge stored with an issuer
recovers only when the lookup supplies a matching issuer; the recovery lookup
reads user.openidIssuer via AUTH_REFRESH_USER_PROJECTION (an exclusion
projection that retains the field). If a user's persisted openidIssuer is
empty while the stored bridge has one, recovery fails closed (falls through to
normal re-authentication) until the bridge TTLs out — no security regression.
Tests cover invalid signed-cookie payloads bypassing the bridge, both
asymmetric issuer-presence cases, issuer normalization before comparison, and
an index-alignment assertion guarding against re-adding the dropped index.
* 🛠️ fix: Degrade OBO discovery on token resolution failures
Catch expected OboTokenResolutionError failures during MCP tool discovery and
fall back to unauthenticated tool listing instead of aborting discovery. This
keeps discovery aligned with the existing unauthenticated listing behavior while
preserving unexpected errors as real failures.
Also correct OBO tool-call freshness comment and tighten the OBO trust-check
permissions type to the existing role permission shape.
Tests:
- npx jest src/mcp/__tests__/MCPConnectionFactory.test.ts --runInBand --coverage=false
- npx jest src/mcp/oauth/obo.spec.ts --runInBand --coverage=false
* 🛠️ fix: tighten OBO tool-call errors, bridge logging, and flight typing
Move resolveToolCallUserId inside the tool-call try/catch so an OBO
identity mismatch surfaces with serverName/toolName context and the
standard tool-call-failed message instead of an opaque bare Error.
Raise the refresh-token bridge lookup failure log from debug to warn so
transient infrastructure failures on the unauthenticated /api/auth/refresh
path are observable, and guard the message access against non-Error values.
Replace the unknown+cast in isDuplicateKeyError with a hasErrorCode type
predicate so the duplicate-key check reads error.code without an assertion.
Preserve real math/isEnabled in the MCPConnectionFactory test mock (mock
only processMCPEnv) so mcpConfig timeouts no longer resolve to NaN, fixing
the TimeoutNaNWarning that masked slow OAuth retry behavior.
* 🧪 fix: Restore the Flight Uniqueness Index and Buffer the Graph Cache TTL
Two CI failures on the merge, both in suites this environment cannot run
(their MongoDB binary download is blocked).
`GraphApiService.spec.js` still asserted the unbuffered TTL. Graph tokens
route through the same `getTokenCacheTtlMs` as the OBO and openidStrategy
caches, so the entry now expires 30s before the credential does.
`openidRefreshFlight.spec.ts` dropped the database between tests, which takes
the indexes with it, and Mongoose builds them only once when the model is
compiled. Whether the unique `key` index survived into a test was a race with
that one-time build. Without it a second `create` inserts instead of raising a
duplicate-key error, so every worker believes it won the flight — the
mutual exclusion the file exists to prove. Indexes are now rebuilt after each
drop, which also makes the reclaim and complete cases reach those paths for
the right reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: address OBO review findings
* 🔐 fix: Install Bridge Indexes and Carry OBO Through Assistant Recovery
Two findings from the Codex pass on d08c82f0d.
The refresh-token bridge relied on Mongoose auto-indexing for both of its
indexes, and `MONGO_AUTO_INDEX=false` is a supported deployment setting. A
bridge holds an encrypted refresh token and the TTL index is the only thing
that ever deletes one, so under that setting they would accumulate for the
life of the collection while concurrent upserts lost the compound uniqueness
the filter assumes. Installed before the first write, matching the flight
methods and the session and schedule methods before them.
`recoverServerTools`, the assistant create/update path that reruns
`reinitMCPServer` when a referenced server's catalog and connection snapshot
are both missing, was the last reinit site not carrying the upstream-token
closure. For an OBO server the factory rejects the connection outright, so the
assistant write failed with unavailable MCP definitions. It now builds the
provider at that request boundary like the other entry points; assistant
writes have no `res`, so a rotation there falls back to the recovery bridge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: harden OBO refresh coordination
* ⏳ fix: Keep Elapsed Expiries Elapsed and Revoke the Superseded Session
Two of the five findings from the Codex pass on fcdc15885 — the two that are
defects in code this branch introduced rather than design questions about the
bridge.
`getSkewedTokenExpiresAtMs` floored every result at a second in the future,
including an expiry the provider had already declared elapsed. An exchange
answering `expires_in: 0` or a past `expires_at` was handed to the MCP
connection stamped valid for another second, which only moves the failure
downstream. The floor now applies to a lifetime that is still live, which is
what it was for; an elapsed one stays elapsed so the caller rejects it. Same
for the cache TTL, which falls back to the elapsed-credential floor.
Bridge recovery left the stale token's durable Session behind. Only the token
it recovered through was passed as `existingRefreshToken`, so that one's
session was replaced while the token the browser actually presented kept its
record until its original expiry. That record, with the marker cookie still
bound to it, is what authorizes local image access for OpenID users — so a
copy of the stale cookie outlived the rotation it had lost. Revoked
explicitly on successful recovery.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: close OBO refresh review findings
* 🎟️ fix: Carry the Bridged Token Through a Non-Rotating Recovery
Bridge recovery passes the browser's stale token as `existingRefreshToken` so
the durable Session naming it is the record replaced. That also makes the
stale token the fallback the installed session and the refresh cookie use when
a tokenset carries no `refresh_token` of its own, which holds only while the
recovery grant rotates.
An IdP that answers that grant without rotating sends the browser back to the
very token the bridge exists to retire: `storeOpenIDSession` installs and
deletes the same stale record in one call, and the cookie is rewritten to a
token the IdP already rejected — a sign-out on the next refresh. The grace
bridge one line above already guards this with `|| bridgedRefreshToken`; the
resolved tokenset now does the same, so leader and followers alike publish the
recovered token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* 🚫 fix: Reject an OBO Exchange That Returns an Expired Credential
Preserving an elapsed expiry through the skew helper only helps if something
acts on it, and nothing did: `MCPManager.callTool` checks the access token and
nothing else before setting the Authorization header, so a credential the IdP
declared spent still went downstream to fail there. It is rejected at the
exchange now, where the reason is known, and retryably — the exchange itself
worked, so a fresh grant can succeed.
Completes the elapsed-expiry change in cb26a6f7d, which made the stamp honest
without giving anyone a reason to look at it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: close OpenID refresh review findings
* fix: coordinate OpenID refresh entry points
* chore: sort OpenID flight imports
* fix: fence OpenID refreshes during logout
* fix: close OpenID logout publication races
* fix: narrow completed refresh flight
* test: cover bridge cleanup failure after ownership loss
The compensating delete in storeRefreshTokenBridgeWithLease swallows its own
failure so the lease error stays the one the caller sees. Nothing asserted
that, so removing the inner catch left every suite green while callers began
receiving the cleanup error instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: compensate OpenID bridges only on proven ownership loss
The post-write lease assertion deletes the bridge it just published when it
throws, but it threw for two different reasons: a coordination record that is
no longer ours, and a coordination read that simply failed. Treating the second
as the first destroys the only mapping from the token the browser still holds
to the one the IdP already rotated to, so a transient Mongo error on the
headers-already-sent path signed the user out.
Tag the ownership error where the lease raises it and compensate only for that,
preserving the bridge whenever ownership is merely undetermined. A preserved
bridge stays behind the logout revocation fence, so the safe default costs
nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: reject spent OpenID refresh results
Two ways a refresh could report success while handing back a credential
nothing can use.
An inline refresh carries the previous id_token forward when the IdP omits
one on rotation, so tokenset.id_token is not necessarily freshly issued.
setOpenIDAuthTokens applied its freshness guard only to the session copy and
took tokenset.id_token unconditionally, so /refresh returned an expired
bearer even though the grant produced a usable access token. Skip it only
when it is provably expired: an id_token whose expiry cannot be read stays
preferred, since access_token may be opaque or scoped to another audience.
normalizeExpiresIn preserves a zero or negative lifetime rather than
discarding it, so a grant declaring an already-spent access token still
published, rotating the refresh token and returning a token every freshness
check rejects. Each OBO call then repeated the grant. Reject an elapsed
lifetime before publishing; an unknown lifetime still publishes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* fix: harden OpenID refresh publication
* fix: keep identity and results intact through OpenID refresh cleanup
Two follow-ons from the last round's fixes.
Stripping an expired carried-forward id_token from the refresh result removed
the only identity material a rotation without id_token leaves behind. The
result is rebuilt by buildOIDCTokensFromSession, so it carries no provider
claims() either, and getTokenClaims accepts only those two — bridge recovery
failed with "no usable identity claims" before setOpenIDAuthTokens could hand
back the fresh access token. The stripped token now travels in a
non-enumerable marker, alongside the existing browser and predecessor markers,
which identity resolution reads and the authentication response never sees.
The lease drained a pending renewal by awaiting it in finally, so a transient
coordination failure there threw from finally and replaced the operation's
result. The refresh had already settled and published, so the caller saw a
failure on credentials that had rotated. Proven ownership loss is recorded on
ownershipLost and checked before the return, so the drain has nothing to add
but noise; it now absorbs and logs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F
* refactor: fence OpenID recovery publication
* fix: close OpenID publication transaction
* refactor: make OpenID publication transactional
* fix: satisfy OpenID publication type checks
* fix: fence OpenID session publication
* fix: authorize OpenID refresh publication
* fix: bind OpenID replay generations
* fix: fence OpenID response generations
* fix: authorize OpenID token delivery
* fix: linearize OpenID publication delivery
* chore: sort OpenID refresh flight imports
---------
Co-authored-by: J.C. Bartle <jcbartle@users.noreply.github.com>
Co-authored-by: jbartle <jbartle@rand.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: jcbartle <7274202+jcbartle@users.noreply.github.com>
* 🕰️ fix: Guard `expires_in` So a Token Response Cannot Outlive Its Credential
RFC 6749 §5.1 makes `expires_in` only RECOMMENDED, so a token response may legally omit it.
Four sites derived a lifetime from the raw field, where `undefined * 1000` is `NaN`.
`NaN` is not a short TTL, it is no TTL. `@keyv/redis` writes the key without `PX` because
`NaN` is falsy, so the entry is stored in Redis with no expiration at all; the in-memory
backend embeds `expires: NaN` and every check compares with `>`, always false against `NaN`.
The namespace default does not stand in either, since Keyv applies it with `??=` and `NaN` is
neither `null` nor `undefined`. The exchanged access token was therefore cached permanently at
`openidStrategy.js` and `GraphApiService.js`, and once it genuinely expired the poisoned entry
kept being served with no path to eviction.
The same omission is sharper in `ActionService.js`, where `new Date(NaN).toISOString()` throws
`RangeError: Invalid time value`. Both call sites are inside a `try`, so the failure surfaces as
a generic "Failed to authenticate OAuth tool" that names nothing, and the refresh site falls
through to `requestLogin()` on every attempt, looping with no exit.
The rule had six hand-written homes and three of them were wrong, so it now has one. A new
`packages/api/src/oauth/expiry.ts` normalizes `expires_in` to a positive finite number of
seconds or nothing, and exposes the two shapes callers actually need: a cache TTL that falls
back rather than returning `NaN`, and an absolute expiry that is absent rather than Invalid.
The four unguarded sites adopt it, and the two ad-hoc guards in `openidStrategy.js` and
`OboTokenService.js` are consolidated onto it.
`createHandleOAuthToken` is folded in as well. Its guard already handled `null` and unparseable
strings but admitted `NaN`, since `typeof NaN === 'number'` satisfied its first branch.
The `mcp/oauth` sites are deliberately left alone: `tokens.ts` guards on truthiness and carries
richer logic that reads a JWT access token's own expiry when the response omits one, and the
file is being reworked in #13901.
Closes#15318Closes#15319
* 🕰️ fix: Address `expires_in` Guard Review Round 1
Preserve an explicitly elapsed lifetime instead of collapsing it into "unknown". `expires_in: 0`
is the provider stating the credential is already dead, which is information; treating it as
absent handed it the one-hour fallback in `createHandleOAuthToken` and dropped the expiry
entirely in `ActionService`, so a credential declared expired could be used and retained for up
to an hour. Both sites preserved that value before this branch, so the collapse was a regression
introduced here.
`normalizeExpiresIn` now returns any finite number, positive or not, and reports `undefined` only
for a lifetime that is genuinely unusable. `getTokenExpiresAt` therefore yields a past timestamp
for an elapsed lifetime, so callers refresh rather than guess.
Cache TTLs cannot pass such a value through raw: Keyv reads a TTL of exactly `0` as "no expiry",
turning a dead credential into the immortal entry this module exists to prevent. `getTokenCacheTtlMs`
floors an elapsed lifetime at one millisecond, which expires immediately without ever writing an
entry that outlives its credential.
Parse numeric strings with `Number` rather than `parseInt`, which truncates a complete value such
as `"3.6e3"` to `3` and would expire an hour-long credential after three seconds, re-exchanging
against the identity provider on every request. An empty or blank string is rejected rather than
read as zero, since `Number('')` is `0`.
* 🕰️ fix: Bound `expires_in` to Lifetimes a Date Can Represent
Parsing the complete numeric string last round made an overflow reachable that `parseInt` had
been masking. `parseInt('1e13', 10)` was `1`; `Number('1e13')` is `1e13`, and `1e13` seconds is
1e16 ms, past the ECMAScript time value range of ±8.64e15. Every derived timestamp was therefore
an Invalid Date whose `toISOString()` throws `RangeError: Invalid time value` — the exact failure
this branch exists to remove, reintroduced by its own fix. The token model derives the same way
at `packages/data-schemas/src/methods/token.ts:19`, so storage and authentication would fail with it.
A lifetime is now reported as unusable unless it can still produce a valid `Date`. The bound is
the time value range halved, leaving room for the `Date.now()` every derived timestamp adds. At
roughly 137,000 years it rejects nothing a provider could mean: a one-year refresh token and even
a hundred-year lifetime still pass through untouched, while `1e13`, `Number.MAX_SAFE_INTEGER` and
`1e300` take the caller's fallback instead of poisoning a timestamp.
The invariant tests now carry the overflow shapes rather than a fixed list of small ones, since a
guard that only sees the inputs its author imagined is how the previous round's regression got in.
* 🤖 feat: make event actor HITL durable
* 🤖 fix: break event actor outcome cycle
* 🤖 fix: close durable actor terminal races
* fix: harden durable event actor recovery proofs
* fix: close event actor resume publication races
* feat: wire Keenable web-search provider into config, schema, and UI
Keenable landed as a search provider in @librechat/agents (#285, shipped in
3.2.58+), but LibreChat did not yet expose it. This adds the config/schema/UI
glue so it can be selected, mirroring the existing Tavily provider.
- data-provider: add `keenable` to SearchProvider type + SearchProviders enum,
keenableApiKey/keenableApiUrl schema fields, and a keenableSearchOptions block
(maxResults, site, attributionTitle, timeout).
- data-schemas: register keenable in webSearchAuth.providers and default the
key/URL placeholders in loadWebSearchConfig.
- api/web: pass keenableSearchOptions through to the provider and handle
Keenable's keyless model. Unlike other providers it authenticates with no key
(the public endpoint), picking up an optional key/URL when set; the URL
override is SSRF-preflighted like other user-provided URLs.
- client: add Keenable to the provider dropdown with an optional API-key input.
- docs: document KEENABLE_API_KEY/KEENABLE_API_URL in .env.example and a
webSearch example in librechat.example.yaml.
- tests: keyless + keyed auth resolution, config defaults, and schema parsing.
* fix: ESLint no-unused-vars and clarify Keenable yaml example
- Remove the now-unused RerankerTypes import in data-schemas web.ts (the lint
job runs with --max-warnings 0 on changed files, so this latent warning failed
CI once the file was touched).
- Note in the librechat.example.yaml Keenable stanza that a scraper (and
reranker) is still required for web search to load, and include a Firecrawl
scraper in the example.
* chore: fix import order drift (sort-imports)
* feat: add Keenable as a keyless scraper and select it without a pinned provider
The Keenable scraper landed in @librechat/agents#337, so wire the scraper
category the same way the search provider already is: `scraperProvider:
keenable` reads pages through Keenable's public fetch endpoint with no key
(a key only lifts rate limits, and the endpoint is overridden with
KEENABLE_FETCH_URL). Paired with `rerankerType: none` this makes a fully
keyless web-search stack possible for the first time.
Also closes the Codex finding on this PR: because none of Keenable's auth
fields are required, the generic auth loop skips it whenever it isn't pinned,
so a key submitted through the API-key dialog (which cannot pin a provider)
left the providers category unauthenticated. Keenable is now selected in that
case, gated on one of its values actually being present so installs that
configured nothing keep their current behavior. The scraper gets the same
fallback, additionally gated on Keenable being the resolved search provider,
so it never silently scrapes for another provider.
* fix: select the Keenable scraper from a supplied key, not only for Keenable search
The API-key dialog submits credentials and cannot pin a provider, so choosing
Keenable as the scraper while search stays on Serper/SearXNG/Tavily had no
effect: the unpinned-scraper fallback required Keenable to also be the resolved
search provider.
A supplied Keenable value now triggers it as well, which is the only signal the
dialog can send. The fallback still runs only when no keyed scraper
authenticated, and with neither trigger the category stays unauthenticated, so
a deployment that never configured Keenable is unaffected.
Note the fully keyless choice still cannot be expressed through the dialog:
Keenable's key is optional, so picking it with no key submits nothing at all.
librechat.example.yaml now documents pinning scraperProvider: keenable for that
case.
* style: Sort Keenable imports
* fix: Harden Keenable auth resolution
* fix: Preserve Keenable selection intent
* fix: Fail closed on invalid web search auth
* fix: close keenable auth gaps
* style: sort web auth imports
* fix: preserve web search selection integrity
* fix: isolate web search auth ordering
* fix: silence expected credential misses
* chore: bump agents sdk
* fix: preserve web search preference ownership
* fix: forward cleared Keenable endpoint
* style: sort web search hook imports
* fix: require intent for credential clears
---------
Co-authored-by: Ilya Bogin <ilya.bogin@keenable.ai>
* feat: resume event actors from checkpoint forks
* fix: fence event actor checkpoint uncertainty
* fix: satisfy event actor type contracts
* fix: make actor reconciliation recoverable
* fix: preserve event actor lifecycle transitions
* fix: fence event actor lifecycle outcomes
* fix: enforce event actor lifecycle ownership
* fix: retain event actor settlement proof
* 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal
Repair and compensation deleted the reconciliation row they resolved, which
was the only durable proof that the invocation had already applied an external
action. A delayed duplicate owner could then reacquire the same invocation id
and repeat that action. Both resolutions now retire their receipt to `settled`
and record how it settled, so the same-id tombstone survives; `history_repaired`
and `action_compensated` still force a cold rebuild. Compensation undoes the
effect without re-authorizing the delivery, so a legitimate retry must arrive
under a new invocation id. A retried repair converges on its own receipt.
Bound the journal so a long-lived actor cannot grow its conversation document
without limit: a new fence is admitted only when no active lifecycle row
exists, so a capped push can evict nothing but the oldest settled receipts.
Stop shipping the unbounded source payload on every bound-child continuation.
It rode the delivery body regardless of the feature flag while the sibling
`fire` body deliberately sends event identity alone, so a large webhook payload
could push a previously working delivery past the chat route's body limit. The
actor binds an invocation from identity and never builds the prompt from it.
Blank the positional token map on warm continuations. It is derived from the
full DB history, while a warm run executes on checkpoint-restored state, so its
indices address different messages and the pruner never recounts them —
misattributing cached counts to the wrong messages in both directions.
Keep the replaced-claim exit on its cleanup path when preserving reconciliation
fails: the committing CAS already left a blocking row, so the failed status
upgrade costs provenance, not safety.
* 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry
Give the warm-continuation client test a populated token map and a real
cross-run summary so its assertions bite: the positional map must arrive
blank (checkpoint-restored state no longer matches DB-derived indices, and
the pruner never recounts a populated entry) while `initialSummary` must
pass through unchanged — it rides the system tail and summarizes
pre-boundary turns that were excluded from the very history the committed
checkpoint was built from, so blanking it would silently drop context no
warm run can recover.
* ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal
A compensated receipt still tombstones its invocation id, but its external
effect was explicitly undone — the terminal handler nonetheless replayed
every settled lifecycle's stored action as authoritative and settled the
public outcome as applied, telling an action-aware source the operation
stands and suppressing the new-invocation retry compensation requires. The
handler now settles a compensated invocation as failed with an explicit
compensation error, overriding even fresh applied run evidence from a
replayed generation; verified and repaired receipts continue to replay
applied.
Receipt eviction is now primarily age-based: a stale same-id owner is
bounded by time, not by how many newer invocations settle, so the previous
count-only slice let a high-rate actor evict a tombstone while its delayed
duplicate owner could still wake and repeat the action. Admission prunes
only settled receipts older than a retention window that dwarfs every
generation, job, and delivery-retry lifetime, and the count cap is demoted
to a raised document-size backstop.
* 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts
The terminal handler read its lifecycle snapshot, verified history, and then
settled the public outcome — so a compensation resolving the same receipt
during that window lost: the handler settled applied from its stale snapshot
and no retry could ever change the replay-identity-locked outcome. The
receipt's status CAS is now the serialization point: verification resolves
the receipt BEFORE settling, whichever transition wins determines the public
outcome, and a crash between resolve and settle converges through the
retained receipt's replay. The verified-replay probe requires the receipt's
own resolution, so a compensated receipt can never satisfy a verification
retry. This inverts the settle-before-receipt ordering deliberately: that
ordering guarded proof that resolution used to delete, and the receipt now
retains its full action proof through resolution.
The document-size cap is no longer an eviction quota. A receipt inside its
retention window is never discarded: when the journal holds a full cap of
unexpired receipts, new invocations are refused fail-closed until receipts
age out, making duplicate protection and document integrity simultaneous
invariants instead of a rate-dependent trade.
* 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path
Skill primes are spliced into the message list directly ahead of the newest
message, and a warm continuation forwards only that newest message — so a
checkpoint-restored actor would keep serving the prime bodies baked in at
its last cold start and never observe an edited or newly attached skill.
Until the actor head carries a context fingerprint that forces a cold
rebuild when the agent's skill context changes, agents with always-apply or
manual skill primes stay on the legacy path, which re-primes fresh bodies
every turn: correct on every event, just never warm.
* 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes
History-derived re-priming was a third path into the same staleness class:
an actor that previously invoked a skill carries no request-time prime
arrays, yet primeInvokedSkills re-resolves that skill's current body from
history each turn and the warm slice drops the reconstruction — leaving the
checkpoint's old body active after edits. The fork gate now keys on the
priming hook itself (present exactly when the skills capability is enabled)
alongside the request-time arrays, so every skill-body path routes to the
legacy rebuild until #15235's context fingerprint restores warm
continuation for skill-bearing actors.
* 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage
The executor read applied-action evidence from the run-step collection the
instant sendMessage resolved, but that collection is populated
asynchronously — an applied invocation could classify as actionless
(runSteps still empty while the tool result already streamed), discarding
its fork and stranding the actor cold while the terminal handler later
settled the same delivery as applied from the persisted evidence.
Authoritative proof is now recorded in graph context the moment the
expected tool executes: the request-owned recorder observes the tool-end
chain (which ToolNode dispatches synchronously with both input and output)
and applies the same fences as run-step evidence — exact tool name with the
MCP-suffixed form, the declared argument subset against the execution
input, an error-free result, and the background non-execution receipt
exclusion. readAppliedAction consults the receipt first; run-step
inspection remains the fallback for paths that bypass the tool-end chain.
Regression coverage reproduces the observed ordering: the real executor
commits the head from the receipt while run steps are empty, warm-continues
the next event, and never re-executes the action; recorder fences and the
receipt-first controller wiring are covered separately.
* 🎯 fix: Supply Execution Arguments to the Tool End Callback
The live Vertex + MCP canary exposed a contract mismatch the synthetic
fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end
callback with output only, while the action recorder must verify the
declared argument subset against the execution input. The receipt never
qualified, every turn fell back to cold history rebuilds, and the
tournament advanced with zero actor heads and zero retained checkpoints
while looking successful.
The execution handler owns both halves at the same moment, so the fix is
at the source rather than a correlation store: ToolEndCallbackData gains
the executed call's input and every callback site passes tc.args. A
handler-level regression drives the real createToolExecuteHandler and
asserts the callback receives both fields; recorder regressions pin the
production shapes — an output-only tool end must starve an
argument-fenced receipt rather than trust an unfenced match, and still
qualifies a name-only expected action.
* 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions
The background-claim callback reports the ORIGINAL tool's name for artifact
attribution on the poll turn that harvests a completed task. A name-only
expected action could therefore be impersonated by work some earlier turn
dispatched: the recorder would attribute that delivery to the current
invocation and commit a head whose state never contained the invocation's
own action. The run-step evidence path never had this hole — it sees the
poll tool's name — so the recorder must match its provenance discipline.
Delivery callbacks now carry an explicit backgroundDelivery marker set at
the one site that rewrites the name, and the recorder ignores marked
deliveries outright. Regressions pin both halves of the contract: the
delivery callback must carry the marker with the poll call's arguments,
and a marked delivery can never qualify even a name-only expected action.
* 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions
Three closeout-round findings, each converted into an invariant.
Every legacy-path invalidation now advances a durable epoch — including for
headless and already cold-marked actors, where the marker alone leaves no
CAS-visible trace — and the actor-head CAS requires the epoch observed at
preparation. A concurrently prepared fork whose history predates an
intervening legacy turn can no longer commit past it; the commit reports an
ordinary conflict and journals.
Execution identity is now emitted before post-execution output policy: when
a side-effecting tool succeeds but its returned content is withheld by the
output filter, the callback delivers an outputFiltered receipt with blank
content — the recorder accepts it as proof (rejecting model-detached calls
it cannot distinguish through the blank shape), the artifact path never
sees it, and an applied action is no longer reclassified actionless and
re-executed on retry.
Background-capable expected actions stay off the fork path: dispatch
returns a launch handle every evidence fence correctly rejects, and the
completion is provenance-marked as another turn's work, so a fork would
settle actionless before the external effect lands with nothing to stop a
retry from dispatching it again. The gate mirrors the MCP-suffix name
matching of the evidence path.
* 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance
The epoch fenced only the legacy turn's start: a fork preparing after the
begin invalidation but before the turn's message persistence observed the
new epoch and cold marker, rebuilt from history that did not yet contain
the turn, and committed cleanly because nothing advanced the epoch again —
making the incomplete rebuild authoritative and clearing the marker.
Every legacy event turn now seals its invalidation at terminal persistence
with a second epoch advance, on the success, replaced-claim, and error
exits alike. Sealing deliberately carries no quiescence requirement — it
must succeed while a fork fence is active, because that is exactly the
mid-turn race it defeats — and a fork that already committed against the
begin epoch is healed the same way: the seal re-marks the head cold, so
the next event rebuilds with complete history. Seal failure never diverts
the turn's own exit; the begin bump still fences everything prepared
before the turn.
* 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence
The second epoch advance could not make a legacy turn atomic, and three
findings shared that root cause: two conditional updates left a headless
gap a fork could create the head inside; the error exit sealed before
saveErrorTurn made the error history durable; and any crash or failure
between persistence and sealing left an incomplete fork authoritative,
because the seal was best-effort and its failure was swallowed.
A legacy turn now carries one durable fence. A token is written before
execution by a single update-pipeline write — no two-write gap, and the
cold marker is applied only where a head exists via $cond/$$REMOVE. While
the token is present no fork may prepare (the adapter refuses) or commit
(the CAS requires its absence), because the turn's messages are not yet
durable. One atomic write clears the exact token and advances the epoch
once history is persisted — after saveErrorTurn on the error exit — and
success, replacement, and error exits all route through it.
Failure is now fail-closed rather than silent: a failed seal leaves the
token set, which keeps blocking forks and is logged as such, and a fence
abandoned by a crashed turn is reclaimed only once stale, advancing the
epoch and marking any head cold so the next event rebuilds from whatever
history actually survived.
* fix: serialize legacy event actor turns
* fix: close legacy actor fence ownership gaps
* fix: preserve legacy actor persistence fences
