14 commits

Author	SHA1	Message	Date
Danny Avila	520af663bc	🧵 feat: Background Tool Calls for Agents & Model Specs (#14197) ... * 🧵 feat: Background Tool Calls for Agents & Model Specs Opt-in, poll-based background tool execution. The model marks an eligible tool call with `run_in_background: true`; the host executor registers a task, returns a handle immediately (so the graph turn resolves), runs the tool as a detached promise, and the model retrieves the result via a new `check_background_task` poll tool. Host-side only — no `@librechat/agents` change. - Opt-in mirrors `deferred_tools`: admin capability `run_in_background` (off by default) + per-tool `tool_options.run_in_background`. - Model specs / ephemeral agents: `TModelSpec.runInBackground` / `TEphemeralAgent.run_in_background` synthesize per-tool options; both paths converge at `initializeAgent`. - In-process task registry: scoped per user+conversation, idempotent by toolCallId (safe across resume/replay), capped, TTL-swept. - Excludes direct-path / host-special / code-session tools. Subagents and push notifications are deferred follow-ups. * 🩹 fix: Harden background tool calls (Codex review) - Reliable per-agent execution gate: thread the injected `run_in_background` tool names from `initializeAgent` through `configurable.backgroundToolNames` (`toolRegistry` only reaches the executor for PTC/tool_search), fixing the silent no-op + unstripped-arg leak for ordinary event-driven tools. - Enforce the per-tool opt-in at execution (`backgroundToolSet.has(name)`) so a non-opted-in tool can't be backgrounded via an extra arg. - Gate the `check_background_task` interception on the run actually enabling background, so a user tool sharing that name still executes. - Forward `backgroundToolsAvailable` to added-convo (multi-convo) agents. - Exclude `web_search`/`file_search` from eligibility — their results are turned into user-visible attachments/citations only by the foreground toolEndCallback. * 🩹 fix: Address Codex round 2 on background tool calls - Idempotency scoped to run+turn: provider tool-call ids repeat across turns (e.g. `call_0`), so key the dedupe map by `runId::toolCallId` and sweep orphaned mappings — a later turn no longer collides with a retained task. - Artifacts preserved: a backgrounded tool's artifact is processed through the same `toolEndCallback` as the foreground path (images/files/citations no longer silently dropped), best-effort/guarded. - Forward the `run_in_background` capability to connected-agent discovery and subagent `processAgent` init, so a child agent's own event-driven tools work the same as when it runs as primary. - Strip the injected flag on foreground calls of background-capable tools (the model may emit it as `false`) so strict MCP/action schemas don't reject. - `check_background_task` list path returns metadata only (result_available / result_chars), never full results — prevents context overflow; the full result is returned only when a specific id is requested. * 🩹 fix: Address Codex round 3 on background tool calls - Exclude background-capable tools from eager execution (run.ts): a speculative eager dispatch of a `run_in_background` call could launch the detached task with partial/stale args, and that side effect can't be canceled. - Reserve the `check_background_task` name: overwrite a colliding user/MCP tool with the host poll schema (with a warning) so the advertised schema matches the executor's interception instead of hijacking a mismatched tool. - Don't inject background schemas into pure subagents (spawn-tool child graphs) whose tools don't reach the host interceptor; keep it for primary/added/ connected agents. Subagent background is the durable follow-up. - Thread `backgroundToolsAvailable` + `backgroundToolNames` through the OpenAI-compatible and Responses agent routes (was chat-only), so the same agent/model spec behaves consistently across surfaces. - Exclude image-generation built-ins (dalle/flux/gemini_image_gen/image_gen_oai/ image_edit_oai) — artifact-first tools whose files can't reliably attach to an already-saved turn when backgrounded. * 🩹 fix: Address Codex round 4 on background tool calls - Sanitize self-spawn subagent inputs: strip `run_in_background` + the `check_background_task` def from the parent AgentInputs reused for self-spawn, so the isolated child (direct/child-graph path) doesn't advertise a background schema it can't honor. The SDK resolver keeps a provided `agentInputs` even with `self: true`. - Exclude `check_background_task` from PTC (`run_tools_with_code`) tool definitions — it's host-only and not callable from generated code. - Parse stringified JSON args before deciding background dispatch and before stripping the flag, so string-delivered `run_in_background` is honored and never leaks to strict object-schema tools. - Skip injection for tools that already declare their own `run_in_background` param (would otherwise hijack/strip it), and for non-object (string-input) schemas (would otherwise rewrite the input contract). * 🩹 fix: Address Codex round 5 on background tool calls - check_background_task now parses stringified JSON args, so providers that deliver args as a string can retrieve a specific task by id (not just list). - Include agentId in the background dedupe key (`agentId::runId::toolCallId`): two agents in the same run emitting the same provider id (e.g. `call_0`) now launch independent tasks instead of colliding. - Self-spawn sanitization also strips the background entries from the reused toolRegistry (not just toolDefinitions), so a child using tool_search/deferred loading can't rediscover the host-only run_in_background / check_background_task. * 🩹 fix: Strip run_in_background from PTC target tool schemas (Codex round 6) The PTC path already filtered out the host-only check_background_task poll tool but still exposed target tool schemas with the injected `run_in_background` param (the shared toolRegistry entries were mutated by applyBackgroundToolCalls). PTC codegen doesn't go through the host background interceptor, so it could pass the flag to an MCP/action tool (strict-schema rejection or silent foreground with no poll). Sanitize the PTC toolDefs like the self-spawn path does. * 🩹 fix: Sanitize background from explicit subagent inputs (Codex round 7) A child agent reachable as a top-level/handoff agent is initialized WITH the background capability, then reused as an explicit subagent via buildSubagentConfigs. Round 4 only sanitized the self-spawn case; this now applies the same stripBackgroundFromToolDefinitions/Registry to explicit child agentInputs when `child.backgroundToolNames` is non-empty, so an isolated child graph doesn't advertise a run_in_background / check_background_task contract it can't honor. * 🩹 fix: Reap stuck/expired background tasks (Codex round 8) - get() now sweeps before returning, so repeatedly polling a known background_task_id can't keep an expired completed task (and its retained result, up to 100k chars) alive past the one-hour completed TTL. - sweep() now reaps `running` tasks older than a 30-min running TTL, marking them errored. Previously a detached call that never settled (hung network / lost MCP connection) held a running slot forever, exhausting the per-conversation cap and rejecting every later dispatch. * 🩹 fix: Evict oldest settled tasks instead of blocking at the cap (Codex round 9) Only the running-task cap gates dispatch now. The total-tasks cap (MAX_TASKS_PER_BUCKET) bounds memory but no longer rejects new background calls: when full, it evicts the oldest settled (completed/error) tasks to make room. Previously 200 quick background calls in one conversation would block all new dispatches for up to the completed-task TTL, since polling doesn't remove settled tasks. Running is already capped, so room always frees. * 📝 docs: Frame background tool calls as within-turn (Codex P1 contract) Codex escalated the request-lifecycle findings to P1 on the grounds that the advertised "poll later" contract can't be honored for genuinely long-running calls (request-scoped MCP connections + the run abort signal are torn down at turn end). Align the model-facing contract with what the same-run implementation actually delivers: the run_in_background param, check_background_task, and the dispatch handle now instruct the model to collect the result WITHIN THE SAME TURN (backgrounded work isn't guaranteed to survive past the turn). This is within-turn parallelism; cross-turn survival of long-running calls remains the deliberate durable subagent follow-up. Copy/comment-only; no behavior change. * ♻️ refactor: Cross-turn background tool calls, leak-free Extend background tool calls from within-turn to cross-turn on a single process, since the mechanism already supports it: the run's abort signal never reaches the detached invoke (the graph forwards only configurable/ metadata to the tool-execute handler), so the floating promise keeps running past turn completion and its result stays in the in-process registry for a later turn to poll (get/list key only on user::conversation + id, never the dispatch run/turn). Guarantee no connection leak: ephemeral request-scoped MCP tools (runtime {{LIBRECHAT_BODY_*}} placeholders) capture their request-scoped store at creation and fall back to it, so config manipulation can't redirect them; their connection is torn down at request end. Tag such tools in createToolInstance and run them in the foreground instead of backgrounding them. Pooled/app-level MCP and structured tools are unaffected and survive cross-turn via their managed pools. Reword the model-facing contract (run_in_background, check_background_task, handle message, fileoverview) from within-turn to cross-turn on this server (not across restart/replica, which stays the durable follow-up). Tests: cross-turn poll retrieval; ephemeral MCP tool runs foreground. * 🐛 fix: Guard ephemeral MCP tag against a null server config createToolInstance can be reached with a null/stale capturedServerConfig (cached availableTools + getServerConfig returns null, as several MCP unit tests construct tools). The new unconditional requiresEphemeralUserConnection call then dereferenced config.source and threw during tool construction (CI: Tests api shard 2/3). Guard with the same serverConfig ? ... : false pattern the other callers use; a missing config is not request-scoped. * 🎨 fix: Deliver backgrounded tool artifacts on the poll turn A slow backgrounded MCP/action tool resolves after its dispatch turn is finalized: createToolEndCallback only appends to that turn's artifactPromises (already awaited) and writes to a closed stream, so the artifact (file/citation/ UI resource) was silently dropped — check_background_task recorded only the hasArtifact boolean. The cross-turn contract made this the common case. Hold the artifact on the task and deliver it through the LIVE poll turn's toolEndCallback the first time check_background_task collects that id (once, then cleared to free memory), attributed to the original tool. Same-turn and cross-turn now share this path since the model must poll to collect any result. Tests: registry claim-once; artifact delivered on poll not dispatch, idempotent. * ✨ feat: Agent-builder toggle for background tool calls + cap tool descriptions Add a per-MCP-tool "run in background" toggle in the agent builder, mirroring the programmatic/deferred pattern: gated on the admin `run_in_background` capability via useAgentCapabilities, read/written on tool_options[id] .run_in_background through useMCPToolOptions (per-tool + bulk mark-all), and rendered as a Zap toggle in MCPToolItem and McpSection with new locale keys. Also cap the section tool/server descriptions (McpSection, ToolSection, SkillSection) with max-h-40 overflow-y-auto so a long description scrolls instead of overflowing the dialog, matching MCPToolItem's existing cap. Tests: MCPToolItem renders/toggles the background button only when enabled. * 🧪 fix: Mock new background hook functions in McpSection spec * 🎨 fix: Restore background artifact when poll-turn delivery fails * 🛡️ fix: Harden background tool call edges from review findings - Error immediately (matching foreground) when a background-requested tool failed to load, instead of returning a success handle for a dead task - Exclude ephemeral request-scoped MCP tools at injection time so the model never sees a run_in_background param the executor would silently downgrade; flip the execute-time tag to fail closed on a missing server config - Source image-tool background exclusions from the shared imageGenTools set (adds missing stable-diffusion, an artifact-first live tool) instead of a hand-copied list - Add check_background_task to the eager-execution exclusion list: artifact collection is a one-shot claim that must not fire from a speculative snapshot the SDK may discard - Strip an imitated run_in_background arg on tools the executing agent never opted in (multi-agent history bleed), unless the tool's own schema declares the parameter - Truncate oversized stored results with an explicit marker via the shared truncateMiddle (moved to utils/text) instead of a silent slice - Document the at-most-once artifact delivery semantics honestly (the callback's downstream persistence is fire-and-forget, as in foreground) * ♻️ refactor: Deduplicate background tool-call plumbing and tighten types - Use the SDK's JsonSchemaType instead of a local duplicate; drop all as-unknown casts and type the poll-tool serializer explicitly - Drop derivable BackgroundTask state (progress, hasArtifact) and the dead `enabled` param/return on applyBackgroundToolCalls (guarded at the call site), which also skips the defs pass when nothing opted in - Fold the enable expression into synthesizeBackgroundToolOptions so the three load/added call sites can't drift - Throttle the registry's all-buckets sweep and always sweep the accessed bucket, so a hot poll loop is no longer O(total tasks server-wide); bound retained artifact memory with a size cap - Single-pass stripBackgroundFromToolDefinitions; pass metadata through to the poll-turn callback instead of a no-op reconstruction - Collapse the client's copy-pasted boolean option families into a keyed factory (also removes the shared-object mutation in the bulk toggles) and the six toggle-button copies into one OptionToggle component * 🧪 test: e2e coverage for cross-turn background tool calls Proves the full contract through the real pipeline (mock harness): an agent opts an MCP tool in via tool_options.run_in_background, the model dispatches it detached and receives the synthetic handle while the tool is still running (status=running in the rendered ack — the non-blocking guarantee without timing assertions), the tool completes after its turn finalized, and a later user turn recovers the task id from replayed history, polls check_background_task, and renders the collected result. - fake-mcp-server: slow_echo fixture tool (delayed echo) - fake-model: E2E_BACKGROUND_DISPATCH / E2E_BACKGROUND_COLLECT markers - e2e yaml: agents capabilities = defaults + run_in_background * 🔧 fix: Close two background capability gaps from review - Thread backgroundToolsAvailable through the OpenAI-compatible service (derived from app capabilities like codeEnvAvailable/statefulSessions), so agents with tool_options.run_in_background keep the feature on that route; fold the three capability derivations into one helper - Index ephemeral MCP servers by normalizeServerName when excluding tools from background injection: tool names embed the normalized server name while mcpConfig keys the original, so exotic server names previously escaped the injection-time exclusion * 🛂 fix: Fall back to configurable user identity for background task scoping The in-repo routes merge req into the tool-execute configurable, but external hosts of the exported OpenAI-compatible service inject their own loadTools and may not — tasks would then register under an empty user id, collapsing registry isolation to conversationId alone. Resolve the scoping id from req.user.id, then configurable.user_id / user, and cover the isolation with a foreign-user not_found test. * 🧹 chore: Apply repo import sorter to PR-touched files	2026-07-13 12:51:36 -04:00
Danny Avila	397ddc5366	🧠 feat: Add Memory as an Agent Capability with Inline Tools and Ephemeral Badge (#13869) ... * 🧠 feat: Memory Agent Capability with Inline Tools and Ephemeral Badge Add `AgentCapabilities.memory`, which expands into the inline set_memory/delete_memory tool pair (mirroring the execute_code expansion via registerMemoryTools) when a run-level memoryAvailable gate holds: capability enabled, memory configured, MEMORIES.USE permission, and personalization not opted out. Surfaces the memory artifact as an attachment in the agents tool-end callback. Adds the ephemeral path (TEphemeralAgent.memory, load/added agent tool injection), a fully-gated memory badge plus tools-dropdown entry, the agent-builder Memory toggle with form round-trip, and a mock e2e test asserting the badge reaches the request payload. Additive to and independent of the existing post-turn memory extraction agent. * 🩹 fix: Address Codex review on memory capability (gating, validKeys, usage guard) - Strip the memory capability from the served agents capabilities when memory is not configured/enabled, so the badge, tools dropdown, agent-builder toggle, and backend capability gate stay consistent instead of exposing an inert toggle on default installs (where MEMORIES.USE defaults true). - Surface configured memory.validKeys in the inline tool definitions so the model is told the allowed keys up front, matching the runtime createMemoryTool schema. - Append a strict explicit-request usage guard to the agent instructions when inline memory tools are registered, preserving the memory-agent's privacy behavior. - Add AppService tests covering memory-capability stripping. * ✅ test: Update AppService capability snapshots for memory strip AppService now strips the memory capability from the served agents defaults when no memory block is configured; update the spec's expected capability lists to defaultAgentCapabilitiesWithoutMemory for the no-memory-config cases. * 🛡️ fix: Address Codex re-review on memory capability (round 2) - Strip the memory capability from the FINAL served agents config, not just defaults; loadEndpoints reparses any endpoints.agents block, so memory was still exposed in that common shape (packages/data-schemas/src/app/service.ts) + regression test. - Re-check the full memory gate (config, opt-out, MEMORIES.USE) inside handleTools before constructing set_memory/delete_memory, so an unsolicited tool call from a model/custom endpoint can't bypass the runtime gates (api/app/clients/tools/util/handleTools.js). - Restore the persisted memory toggle for model-spec conversations via applyModelSpecEphemeralAgent (client/src/utils/endpoints.ts). - Clear LAST_MEMORY_TOGGLE_ on logout and clear-all-chats so a stale memory preference can't leak across users on a shared browser (client/src/utils/localStorage.ts). * 🧠 fix: Address Codex re-review on memory capability (round 3) - Serialize set_memory writes and advance a running token total inside createMemoryTool, so parallel batched calls in one event-driven turn can't each pass the limit check against a stale total and collectively exceed memory.tokenLimit (packages/api/src/agents/memory.ts) + tests. - Inject the keyed memory context (withKeys) instead of withoutKeys when the running agent has the inline memory capability, so delete_memory has a visible key to target (api/server/controllers/agents/client.js). * 🔐 fix: Address Codex re-review on memory capability (round 4) - Detect inline memory by tool NAME (set_memory/delete_memory) across an initialized agent's tools + toolDefinitions, since the 'memory' marker is expanded at init and the prior string check never matched; inject the keyed memory context for any primary OR sub-agent that carries the inline memory tools (api/server/controllers/agents/client.js). - Enforce memory WRITE permissions in the inline tool gate: set_memory requires CREATE+UPDATE and delete_memory requires UPDATE (matching the REST memory routes), so a USE-only role can't mutate/delete memories via agent tool calls (api/app/clients/tools/util/handleTools.js). * 🔒 fix: Address Codex re-review on memory capability (round 5) - Gate inline memory registration (memoryAvailable) on the memory WRITE permissions (USE+CREATE+UPDATE), so a read-only-memory role no longer has set_memory/delete_memory shown to the model only for the runtime loader to refuse them (api/server/services/Endpoints/agents/initialize.js). - Enforce the per-agent memory opt-in at execution: handleTools now refuses to construct set_memory/delete_memory unless the agent actually declared them (toolDefinitions/tools), blocking hallucinated/undeclared memory tool calls from mutating memory. - Fail closed when getFormattedMemories errors with a configured tokenLimit, instead of writing as if storage were empty and bypassing the cap (api/app/clients/tools/util/handleTools.js). * 🩹 fix: Address Codex re-review on memory capability (round 6) - Fix a P1 regression from the prior round: the execution-context agent keeps the raw 'memory' capability marker (not the expanded set_memory/delete_memory names), so the opt-in check now matches the marker. This restores memory writes/deletes AND avoids hijacking an MCP tool that merely shares the set_memory/delete_memory name (api/app/clients/tools/util/handleTools.js). - Count repeated set_memory writes to the same key as replacements, not additions, against tokenLimit — set_memory upserts, so a same-key rewrite swaps its prior token contribution instead of double-counting (packages/api/src/agents/memory.ts) + test. - Gate the memory badge, tools dropdown, and agent-builder toggle on the full memory write permissions (USE+CREATE+UPDATE) via a shared useHasMemoryAccess hook, so a read-only-memory role no longer sees an enabled Memory control the backend would refuse to wire up. * 🧷 fix: Address Codex re-review on memory capability (round 7) - Recognize inline memory across both execution-context agent shapes: initializeAgent now sets a LibreChat-only memoryToolsRegistered flag on the InitializedAgent, and the opt-in/detection checks accept that flag OR the raw 'memory' marker. Fixes memory failing for processAddedConvo agents (which store the initialized config, marker already expanded) while staying MCP-name-collision-safe (api/app/clients/tools/util/handleTools.js, packages/api/src/agents/initialize.ts, api/server/controllers/agents/client.js). - Scope keyed memory context to memory-enabled agents only: useMemory now returns both keyed and unkeyed contexts, and buildMessages injects the keyed one (memory keys + token metadata) only to agents that can call delete_memory, while the primary/post-turn path keeps the unkeyed values — so a primary without memory tools no longer sees memory keys it doesn't need. * 🔏 fix: Address Codex re-review on memory capability (round 8) - Enforce memory size limits on inline writes: createMemoryTool now rejects keys over 1000 chars and values over memory.charLimit, matching the REST memory routes, so an inline-memory agent can't persist blobs the memory UI/API would reject (packages/api/src/agents/memory.ts, api/app/clients/tools/util/handleTools.js) + test. - Recheck the agents 'memory' endpoint capability at execution time, so a stale/hallucinated set_memory/delete_memory call can't mutate memory after an admin removes the capability while the agent document still carries the marker (api/app/clients/tools/util/handleTools.js). * ♻️ refactor: Move inline-memory backend logic into packages/api + share memory load Workspace boundary: the inline-memory gating/detection logic that had crept into /api now lives in packages/api/src/agents/memory.ts (TS), with /api kept as thin wrappers. - Add agentHasInlineMemoryTools, isMemoryToolAllowed, and buildInlineMemoryTool to packages/api; handleTools.js now calls buildInlineMemoryTool instead of constructing/gating the tools inline, and client.js imports agentHasInlineMemoryTools instead of redefining it. - Optimize repeated memory loads: getRequestMemories memoizes getFormattedMemories per request (WeakMap keyed by req), so the run's memory-context load and every memory-enabled agent's set_memory token-usage load share a single DB fetch instead of one per agent. * 🧠 fix: Invalidate request memory cache after inline writes Inline set_memory/delete_memory now invalidate the request-scoped getFormattedMemories cache on a successful write, so a later tool round in the same response is seeded with the post-write usage total instead of the stale pre-write one (multi-round writes no longer collectively exceed tokenLimit, and a set after a delete is not over-counted). The within-round sharing across multiple memory-enabled agents is preserved. * 🧠 fix: Persist memory capability on saved agents; honor registration flag - Add Tools.memory to the v1 systemTools allowlist so filterAuthorizedTools no longer silently drops the memory marker when an agent with the Memory capability is created/updated/duplicated through the builder (previously the capability only worked for ephemeral chats, not persisted agents). - agentHasInlineMemoryTools now honors an explicit memoryToolsRegistered boolean before falling back to the raw `memory` marker, so an initialized config whose registration was denied (memoryAvailable false) is not given keyed memory context just because the marker survives in tools. * 🧩 fix: Bring memory tool to parity with other ephemeral tools - Add `memory` to the model-spec schema/type and honor `modelSpec.memory` in both ephemeral paths (load.ts, added.ts) and the frontend spec application, so admins can pre-enable Memory from a model spec exactly like webSearch/fileSearch/executeCode. - Add LAST_MEMORY_TOGGLE_ to the timestamped-storage cleanup list so stale per-conversation memory toggles are purged on startup like the others. - Hide the agent-builder Memory toggle for users who disabled memory in personalization (memories === false), mirroring the chat badge's opt-out gate, so the setting isn't shown as inert/misleading. * ✅ test: Cover memory in applyModelSpecEphemeralAgent spec defaults Update the exact-object assertions to include the new `memory` field and add positive coverage that `modelSpec.memory` maps to the ephemeral agent's `memory` flag. Fixes the shard 2/4 failure from 672a03b05.	2026-06-24 17:14:13 -04:00
Danny Avila	49f4b659f6	🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart (#13814) ... * 🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart MCPServersRegistry was built once at boot from getAppConfig({ baseOnly: true }), freezing allowedDomains/allowedAddresses to YAML. Admin-panel mcpSettings overrides were ignored by both inspection (addServer/ reinspectServer/updateServer/lazyInitConfigServer) and runtime connection enforcement (assertResolvedRuntimeConfigAllowed), so a domain allowed only via the panel failed inspection and never connected. Make the registry's effective allowlists mutable and refresh them from the merged admin-panel config: seed at boot, and re-apply on every config mutation via invalidateConfigCaches -> clearMcpConfigCache. Both inspection and connection paths read the same getters, so both honor overrides without a restart. Fail-safe: current allowlists are preserved when the merged read fails. * 🛡️ fix: Scope MCP allowlist refresh to global config, fail-safe on DB error Address Codex P1 review findings on the allowlist-refresh path: - Tenant-scoped config mutations no longer push one tenant's merged mcpSettings into the process-wide registry singleton (read by all MCP connection paths), which would leak allowlists across tenants. Only global (non-tenant) mutations refresh the registry; tenant mutations still evict the config-server cache. - The refresh read now uses strictOverrides:true so a transient DB error throws instead of silently returning YAML base config — preserving the last-known allowlists rather than overwriting them with fallback values. Adds the strictOverrides option to getAppConfig (default off, no behavior change for existing callers). * ♻️ refactor: Resolve MCP allowlists per-request (tenant-scoped) instead of a global singleton Supersedes the prior global-mutation approach. MCP allowlists live in mcpSettings, which is tenant/principal-scoped admin config, so a process-wide singleton value is the wrong model — it caused cross-tenant bleed and stale reads. Instead, inject a resolver (from the app layer, where the merged config lives) that the registry calls per inspection and per connection. It reads the ALS tenant context via getAppConfig and accepts the acting user so user/role-scoped overrides resolve; config-source inspection (no user) resolves at tenant scope. Falls back to the YAML base allowlists when no resolver is set or the lookup fails, so a transient error fails to the operator baseline rather than disabling the allowlist. Removes the now-unnecessary setAllowlists / boot-seed / invalidateConfigCaches refresh / getAppConfig.strictOverrides machinery. * 🔒 fix: Scope config-source cache by allowlist; resolve OAuth allowlists per-request Address Codex review of the per-request resolver: - Config-source cache key now folds in the resolved allowlists, not just the raw-config hash. Inspection results became allowlist-dependent, so without this a tenant whose allowlist rejects a URL could poison the shared key with an inspectionFailed stub for a tenant that allows it (and vice versa). The tenant-scoped allowlist is resolved once per ensureConfigServers pass and threaded through the cache key + inspection. - The two remaining request-time OAuth allowlist reads now use the merged config instead of the YAML base getters: the fallback OAuth-initiate path (routes/mcp.js) via resolveAllowlists, and OAuth revocation (UserController.maybeUninstallOAuthMCP) via the request's already-merged appConfig.mcpSettings. Without this, an OAuth endpoint allowed only by an admin-panel override was rejected while inspection/connection allowed it. * ✅ test: Update MCP OAuth registry/config mocks for per-request allowlists CI fix for the Finding-12 change. The OAuth-initiate route now calls registry.resolveAllowlists() and the revocation path reads the merged appConfig.mcpSettings, so the affected specs' mocks were asserting the old base-getter values: - routes/__tests__/mcp.spec.js: add resolveAllowlists to the registry mock. - UserController.mcpOAuth.spec.js: provide mcpSettings on the getAppConfig mock so revokeOAuthToken still receives the expected allowlists. * 🧪 test: e2e proof that admin-panel MCP allowlist override takes effect Adds a Playwright mock-harness spec for #13809. A URL-based MCP fixture (e2e-http, streamable-http SDK server) boots inspectionFailed because its origin is omitted from the YAML mcpSettings.allowedDomains; the spec adds that origin via an admin config override (PUT /api/admin/config/user/:id) and asserts the server reinitializes — exercising the real resolver path through the backend + DB. Before the fix, reinspection used the frozen YAML allowlist and the server stayed unreachable. - e2e/setup/fake-mcp-http-server.js: streamable-HTTP MCP fixture (health GET /). - e2e/playwright.config.mock.ts: boot the fixture as a second webServer. - e2e/config/librechat.e2e.yaml: mcpSettings.allowedDomains (excludes 127.0.0.1) + the e2e-http server. - e2e/specs/mock/mcp-allowlist-override.spec.ts: login → baseline reinit fails → apply override → reinit succeeds.	2026-06-17 20:14:53 -04:00
Danny Avila	db7011d567	📊 feat: Real-Time Context Window & Token Usage Tracking (#13670) ... * 📊 feat: Real-Time Context Window & Token Usage Tracking * 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps * 🩹 fix: Resolve Codex Findings for Context Usage Tracking * 📊 feat: Granular Tool Token Breakdown with Deferred Splits * 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors * 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated) * 🧪 test: Local Real-Provider Multi-Turn E2E Harness * 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate * 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events * 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output - carry the post-snapshot output estimate into the context snapshot at finalize so the gauge keeps the last response after live resets - accumulate per-rate billable units and price the session cost at render, so usage events arriving before the token-config load still count once it resolves - pass user-scoped token-config cache keys through loadConfigModels fetches and drop the controller's unscoped fallback to prevent serving another user's resolved config - tag emitted usage events with a per-run seq so resume dedupe never drops a distinct call with an identical payload - admit the static tokenConfig override in the custom endpoint schema so it survives zod parsing into req.config * 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics - classify cache tokens by provider (shared inputTokensIncludesCache from data-provider, consumed by both the backend billing path and the client) instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache is smaller than uncached input no longer under-bill input - mirror resolveCompletionTokens on the client so Vertex-style hidden thinking tokens are reflected in the Output row and session cost - prefer endpoint pricing over adapter-provider pricing so a custom endpoint can price a known model name without built-in rates shadowing it - carry static cacheRead/cacheWrite overrides through the tokenConfig schema and buildTokenConfigMap * 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness - initializeCustom now uses a static endpoint tokenConfig as the agent's endpointTokenConfig (billing + balance checks), not just the advertised UI config — previously the gauge showed admin rates while the agent billed against built-in tables - invalidate the token-config query alongside models on user-key add/ revoke so context windows and pricing refresh without a reload - include maxContextTokens in ChatForm's stabilized conversation memo so the gauge reflects a changed context-window setting immediately - feed the live output estimate from the legacy content path (direct and assistants streams), setting from cumulative part text rather than accumulating deltas * 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing - fold usage events idempotently by (runId, seq) so resume backfill no longer resets the conversation totals — a mid-stream reconnect keeps the usage of prompts already completed earlier in the session - tap replayed pending message/reasoning/content events so output streamed past the resume snapshot reaches the live estimate, not just the message - resolve cost against the agent's backing endpoint (Agents conversations report endpoint `agents` / provider `openAI`, neither of which keys a custom endpoint's tokenConfig) - getMultiplier/getCacheMultiplier fall back to the standard tables for models absent from a partial endpointTokenConfig, so a partial static override no longer bills non-listed models at defaultRate while the UI shows the correct pattern rate * 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup - live/completed gauge counts the repaired completion (normalized output), so under-reporting providers don't drop the response from used context - translate static tokenConfig cacheWrite/cacheRead onto the write/read keys getCacheMultiplier reads, so cache tokens bill at the configured rate instead of the prompt-rate fallback - clear the token index and usage atoms when leaving a conversation, so visited histories don't accumulate in memory for the tab's lifetime - wait for startupConfig before mounting the gauge, so a deployment with contextUsage disabled never briefly mounts it or fires the token-config query on first load * 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo - extract the token-config resolution (override gathering + cache lookup + buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule) - getConvoKey prefers the user message's real conversationId once the `created` event stamps it, so a new chat's first-response live gauge and totals land under the id TokenUsage subscribes to instead of NEW_CONVO * 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config - DEL the Redis job hash before re-creating it so a reused streamId can't inherit a prior run's contextUsage/tokenUsage and backfill stale usage - tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic streams) into the live estimate, not just the content path - copy a deduped fetch's token config to every sibling endpoint sharing the baseURL/key/headers, so /token-config resolves each by its own name * ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume) createJob is an idempotent join — a second replica calls it for the same streamId to share an in-flight stream's state. DELeting the hash wiped the prior replica's persisted created/usage state, so a joining replica missed the created event (GenerationJobManager cross-replica integration test). Reverts the F1 change from 2bfce0c34b; the stale-usage concern doesn't arise in practice (streamId is unique per generation). * 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage - wrap the ModelEndHandler usage emit in try/catch so a failed telemetry delivery (closed SSE / Redis publish error) can't abort the handler before thought-signature capture, which would break resumed tool calls - tag hidden sequential-agent usage as 'sequential' (non-primary) so the client folds it into session cost/totals but not the live context gauge, instead of letting an undefined usage_type inflate the visible gauge * 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup - useTokenConfigQuery refetches on mount when stale, so a user-key change that invalidates tokenConfig while the gauge is unmounted takes effect on return instead of serving the prior key's resolved config - normalize a Vertex-backed agent's provider (vertexai) to the google token-config key, so Gemini context windows and rates resolve instead of showing unknown context / $0 cost * ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge) Move usage-cost pricing to the single source of truth. The backend prices each model call with the same billing functions (premium tiers via getMultiplier(inputTokenCount), cache rates) and emits the USD cost on on_token_usage when interface.contextCost is enabled; the client sums emitted costs instead of re-deriving from base token-config rates. - computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend so the emitted cost matches what is billed (incl. premium thresholds) - getDefaultHandlers gains a usageCost pricing context; initialize.js wires db.getMultiplier/getCacheMultiplier gated on contextCost (agents path) - client UsageTotals carries a summed costUSD; retire the client-side rate lookups (costFromUnits/calcUsageCost) that drifted from backend pricing and produced the provider-keying / cache-key / Vertex / premium findings - keep normalizeUsageUnits for the displayed token counts; token-config is still used for the context-window meter Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1 above their input thresholds). * 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast - re-anchor the context snapshot from the user message to the response message at finalize. Regenerating a response branches off a shared user message, so anchoring on it made the snapshot read as "active" on both branches — switching to the sibling branch showed the wrong (other branch's) context. The response message is branch-unique, so sibling branches now correctly fall back to their own per-branch totals. - raise the gauge ring's track/fill contrast (muted track, prominent fill) so the used portion reads clearly as a fill-level indicator * 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors - tag hidden sequential-agent usage `usage_type: 'sequential'` on the COLLECTED usage (not just the emit), and treat it as non-primary in recordCollectedUsage (billed, excluded from the reported output total) so hidden intermediate output stops inflating the parent's tokenCount/pruning - emit on_token_usage from the subagent usage sink (tagged `subagent`, with authoritative cost when contextCost is on) so the gauge's session cost/totals include billed subagent usage; it stays out of the live meter - call resetLive on the resumable 404 and max-retry terminal branches so the gauge doesn't keep counting stale in-flight tokens after the stream ends * 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle - raise the popup breakdown's context progressbar contrast (muted surface-tertiary track, prominent text-primary fill) — that's the bar the contrast feedback was about - revert the gauge ring restyle (kept its original border-heavy track / text-secondary fill); the ring wasn't the element in question * 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo - a null-anchor context snapshot was treated as active on every branch, leaking one generation's granular breakdown onto sibling branches. Require a non-null (response-message) anchor on the viewed branch instead, so siblings without a matching snapshot fall back to their own totals. - revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds from shallow copies) so the memo was behaviorally identical, but it was the feature's only change to core branch-navigation selectors — removing it matches upstream and rules it out of branch-navigation debugging. * 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits Custom-endpoint agents resolve an endpointTokenConfig during agent init but it never reached the AgentClient, so spending, emitted cost, and runtime max-token resolution all fell back to default rates for those agents. - Surface options.endpointTokenConfig on the returned InitializedAgent. - Pass it to the AgentClient (this.options.endpointTokenConfig) so the spending path bills at configured rates. - Thread it through usageCost to computeUsageCostUSD so emitted per-event cost matches billing. - getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map for models absent from a partial override (matches buildTokenConfigMap); consolidates the duplicated fallback in pricing.ts. * 🪙 fix: Preserve Granular Breakdown Across Branch Switches The granular context breakdown lives only in the live on_context_usage snapshot — a single per-conversation slot, anchored to the latest response and overwritten by each generation. Switching to a branch generated earlier this session lost its tool/skill/system rows and fell back to coarse totals. Retain each generation's finalized snapshot in a per-conversation map keyed by its branch-unique response id (snapshotsByAnchorFamily). When the live snapshot is off the viewed branch, walk the branch tail for its deepest stored anchor and render that breakdown. Bounded by generation count and cleared on conversation switch; the live/just-generated path is unchanged. * 🪙 fix: Harden Resume Seeding and Subagent Usage Emission - useResumableSSE: skip the trailing-output live seed when the resume carries a context snapshot; the snapshot's messageTokens already counts produced output, so seeding it again inflated usage until the next reset. - AgentClient subagent emitter: await GenerationJobManager.emitChunk like every other caller (it persists before publishing), so a floating promise can't race job cleanup and a Redis/publish failure is caught by the emitter's try/catch instead of surfacing as an unhandled rejection. * 🧪 test: Playwright Coverage for Context Breakdown Granularity Add a test-only data-testid distinguishing the granular snapshot breakdown (context-breakdown) from the coarse message-history estimate (context-estimate), then assert granularity in the mock e2e harness: - renders the granular breakdown from the live on_context_usage snapshot (guards that the snapshot event actually reaches the popover, not just the usage totals). - preserves the granular breakdown after switching branches — regenerate to overwrite the single live snapshot, switch back, and confirm the rows survive via the per-anchor snapshot history map. Branch regenerate/sibling selectors mirror the existing chat.spec branch test. All three usage specs pass against the mock pipeline. * 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush Codex round on the prior commit: - countTrailingOutputChars now counts only output at the very END of the aggregated content (0 when the model paused at a tool call), and the resume path always seeds it. The earlier skip-trailing-tool-parts behavior plus the skip-seed-when-snapshot gate together over- or under-counted in-flight output on resume; one rule fixes both — pre-invoke snapshot budget is never double-counted, and genuine in-flight output is no longer dropped. - useTokenUsage re-indexes from the messages cache on tail change while submitting. The cache subscriber is muted during streaming, so without a context snapshot (non-agent streams) sumBranch missed the created tail and dropped history + prompt until finalize. Bounded — tailId only shifts on created/finalize/branch-switch. - AgentClient tracks subagent usage emit promises and flushes them in chatCompletion's finally. The sink fires the emitter without awaiting, and resume reads the usage emitChunk persists (HSET), so cleanup must not race it or resumed clients miss billed subagent usage.	2026-06-13 19:38:28 -04:00
Michael Harvey	05eb986097	💬 feat: Conversation Starters for Model Specs (#13710) ... * 💬 feat: Conversation Starters for Model Specs Adds an optional conversation_starters field to model specs in librechat.yaml. When the active conversation uses a spec that defines starters (and no agent/assistant starters apply), the chat landing renders clickable starter prompts between the landing content and the chat input; clicking one submits it as the first message. - data-provider: add conversation_starters to TModelSpec and tModelSpecSchema so the field survives strict config parsing - client: ConversationStarters falls back to the active spec's starters via getModelSpec; entity (agent/assistant) starters take precedence; starter cards are centered, size to content, wrap at word boundaries, stagger their fade-in, and gain a focus-visible ring - sanitizeModelSpecs passes the field through (denylist); covered by a new unit test - e2e: mock spec + tests for rendering, absence, click-to-submit, and the MAX_CONVO_STARTERS cap Closes #3619 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: Sort ChatView imports --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Danny Avila <danny@librechat.ai>	2026-06-13 11:38:49 -04:00
Danny Avila	b39ec16ff0	🔌 fix: Preserve Ephemeral MCP Selections Across Model Switches (#13697) ... The no-spec branch of `useApplyModelSpecEffects` (added in #11796) reset `ephemeralAgentByConvoId` to null on every `newConversation` call when model specs are configured. On in-place model/endpoint switches (modular chat, same conversation or new-chat draft), BadgeRowContext never refills from localStorage — its init effect only re-runs when the storage suffix or spec changes — so the MCP selection (and tool toggles) were silently dropped from subsequent request payloads while the MCP badge kept displaying them. Reset now only happens on context transitions (leaving a spec, or moving to a different conversation key), where a BadgeRowContext refill is guaranteed; in-place non-spec switches preserve the ephemeral agent. - Gate the no-spec reset on `prevSpecName` / `prevConvoId`, passed from `newConversation` via a snapshot read of the pre-switch conversation - Add jest coverage for all five branches of the no-spec path - Add e2e spec asserting `ephemeralAgent.mcp` stays in the chat payload after a new-chat model switch and after regenerate on a switched conversation (verified failing before the fix, passing after) - Add non-spec "Mock Provider D" endpoint to the e2e config so tests can switch between two real ephemeral endpoints; widen `MockEndpoint` type	2026-06-11 18:13:41 -04:00
Danny Avila	470be2395f	✨ feat: Surface Model Spec Branding on Landing and Selector (#13662) ... Adds an opt-in showOnLanding flag to model specs. When set, the chat landing shows the spec's label and description in place of the time-of-day greeting; specs without the flag are unaffected, so existing deployments see no behavior change. HTML-valued descriptions (inline icons + markup) render sanitized via the shared config-HTML sanitizer with a new media tag/attribute allowlist, both on the landing and in model selector items. Excludes e2e specs from the typed client lint block so staged e2e files no longer fail pre-commit with 'file not found in project'.	2026-06-10 21:02:22 -04:00
Teresa Blanco	9628930958	✅ ci: Add mock e2e coverage for agents, prompts, MCP, and chat flows (#13589) ... * ✅ Add mock e2e coverage for agents, prompts, MCP, and chat flows * 🎯 fix: Change enforce modelSpecs to false --------- Co-authored-by: Danny Avila <danny@librechat.ai>	2026-06-10 09:06:52 -04:00
Danny Avila	da6b74e8eb	🪶 fix: Prevent Soft Default Model Spec from Overriding User Selections (#13642) ... * 🎯 fix: Soft Default Model Spec Overriding User Selections * 🎯 fix: Detect Agents-Only Allow-List Before Endpoints Config Loads * 🎯 fix: Preserve Explicit Soft Default Selections over Older History * 🎯 fix: Limit Soft Default Residue to Spec-Named State, Disable E2E Enforcement	2026-06-10 08:52:28 -04:00
Danny Avila	2a956f143d	🪞 fix: Preserve Model Spec Icons Across Stream Resume and Abort (#13603)	2026-06-08 17:14:21 -04:00
Danny Avila	4b699fb60f	📌 fix: Preserve Project Scope Through Enforced Model Specs (#13586)	2026-06-08 08:41:27 -04:00
Danny Avila	6357ea10c1	🧭 feat: Scope Model Spec Skills (#13522) ... * feat: scope model spec skills * style: format skill catalog limit * fix: serialize model spec skill resolution * test: satisfy model spec load config typing * fix: apply model spec skills to added conversations * fix: support alwaysApply frontmatter alias * fix: address model spec skills review	2026-06-05 10:22:02 -04:00
Danny Avila	a1bfa3b298	🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model (#13508) ... * 🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model Replace the standalone HTTP mock LLM server with an in-process fake model injected into the real createRun -> Run.create pipeline via run.Graph.overrideTestModel, so the mock suite exercises the agents integration end-to-end without a live provider or a separate server. - Bump @librechat/agents to 3.2.2 for the FakeChatModel/createFakeStreamingLLM exports - Add an env-gated applyTestRunHook seam in packages/api createRun (no /api changes) - Add e2e/setup/fake-model.js to drive default replies + the skill-authoring tool-call flow - Drop the mock-llm webServer from playwright.config.mock.ts and set LIBRECHAT_TEST_RUN_HOOK * 🧹 test: Retire Standalone Mock LLM Server From E2E Recorder Migrate the `--profile=mock` recorder onto the same in-process fake model as the Playwright mock suite, then delete the now-unused HTTP mock server so the fake-LLM logic lives in a single place. - Point record.js mock profile at the fake model via LIBRECHAT_TEST_RUN_HOOK - Remove the mock-llm-server spawn/wait and MOCK_LLM_PORT plumbing from record.js - Delete e2e/setup/mock-llm-server.js (e2e/setup/fake-model.js is now the only source) - Update e2e/README.md to describe the in-process fake LLM * 🏷️ ci: Rename Playwright Mock E2E Check to Playwright E2E Tests	2026-06-04 08:33:28 -04:00
Teresa Blanco	b45e4aeae5	🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) ... * 🧪 feat: add e2e playwright tests * 🧪 feat: Add Playwright Recording Harness * test: fix mock playwright config * test: harden mock e2e environment * test: preserve mock dotenv secrets * test: harden mock isolation setup * ci: cache mock e2e builds * test: harden e2e cache and recorder checks * test: preserve data-provider exports in oauth route test * test: isolate mock auth logout state * test: allow isolated logout smoke setup * test: prepare logout smoke auth via api * test: isolate oauth route module mock --------- Co-authored-by: Danny Avila <danny@librechat.ai>	2026-06-02 16:36:39 -04:00