Commit graph

166 commits

Author SHA1 Message Date
Marco Beretta
14d8bb8d3c
⚡ perf: Throttle Code Highlighting While a Message Streams (#16073)
* ⚡ perf: Throttle Code Highlighting While a Message Streams

* fix: harden streamed code highlight throttling

* test: cover wall clock changes during highlighting

* style: tidy highlight regression test

* fix: restart highlight work across lifecycle changes

* style: avoid nested highlight timing ternary

* fix: propagate highlight throttle to startup config

* fix: share and validate highlight throttle

* fix: keep streamed code visible during highlighting

* test: stream highlight tool arguments

* fix: preserve highlights when cadence config resolves

* fix: reschedule highlights when cadence changes

* test: preserve default fake tool chunking

* fix: prevent stale streamed highlights

* test: open the tool pane before asserting streamed highlights

Every scenario in this file waited for highlight tokens without opening the
card. Dev's #16143 stopped passing a closed pane any code at all, so the four
runs polled a card that renders raw text and had nothing to find: the failed
traces show the disclosure at aria-expanded="false" for the whole run, with no
click against it and no tokens in the code element.

Highlighting is now asserted through the card's disclosure, which ProgressText
owns and which carries an aria-expanded state only once the card has input to
show, so waiting on it also waits for the first streamed argument chunk. The
open happens inside expectHighlightedCode, so a scenario cannot assert tokens
without it, and it lands while arguments are still streaming, which is the only
window where the throttle is under test at all.

* test: reopen the rebuilt card for the settled highlight

Opening the pane got the streaming assertions passing and the traces show 240
highlight spans mid-stream, so the throttle itself lands. The settled check
still failed because persisting the streamed message rebuilds the card, and a
rebuilt card starts closed while autoExpandTools is off: the disclosure goes
back to aria-expanded="false" and a closed pane renders raw text.

The settled and cancelled states now assert through the same open-then-wait
helper, so each one measures the card the reader would actually be looking at
instead of the tokens the streaming card happened to leave behind.

* test: leave a window to cancel a streaming highlight

With the settled assertions fixed, the cancellation scenario timed out on the
Stop button instead: it waited for a highlight before stopping, and the ~40
argument chunks it streams at 35 ms are gone in under two seconds, so the run
had already finished and there was nothing left to cancel. It failed on all
three attempts.

The card is now opened and the run stopped before any token is awaited, since
tokens after cancellation are what the scenario is about, and the cancel
variant streams its arguments at a wider cadence so the window does not depend
on how loaded the runner is. Only that label changes cadence; every other
scenario keeps the timing it had.

* fix: preserve initial highlights and honor shared-page cadence

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-21 07:15:32 -04:00
Danny Avila
29ed0993a8
✂️ refactor: Name Tool Rounds From the Trace, and Match a Cut Response From Its End (#16113)
* ✂️ fix: Name Tool Rounds From the Trace, and Match a Cut Response From Its End

* 🧵 fix: Join a Cut Model Call's Round Only When One Is Waiting

* 🏷️ fix: Read Labels From the End Only for the Response the Limit Cut

* 🪪 fix: Say a Tool Round Came From the Conversation Only When It Did

* 🧾 fix: Carry Where a Tool Round's Calls Came From
2026-09-20 08:24:33 -04:00
lia-by-librechat[bot]
9acf9f0ac8
🫗 fix: Roll Back Incomplete Skill Archive Imports (#16052)
* 🫗 fix: Roll Back Incomplete Skill Archive Imports

A .zip or .skill import created the skill first and persisted its bundled
files one by one, keeping the skill and answering 201 when a file failed
validation, a size limit, storage, or its database row. The client never
read the internal `_importSummary`, so it showed "Skill created" and
navigated to a skill that silently lacked the resources SKILL.md names.

Import is now atomic: any failed bundled file rolls back the skill, its
file rows, the ACL grant, and the blobs already written to storage, and
the request answers 422 `skill_import_incomplete` with the failed paths
and reasons. The upload dialog stays open and lists them.

* fix: Localize Import Failure Reasons and Report Every Lost File

Addresses the first review round on the archive import rollback.

Failure reasons are now stable codes (`invalid_path`, `file_too_large`,
`archive_too_large`, `archive_entry_changed`, `persistence_failed`) that
the client localizes, so a non-English locale no longer sees English text
and storage or driver messages stay in the server log.

The decompression budget stops the scan mid-archive, and the entries it
never reached were missing from the response, telling the user they had
seen the whole failure set. Those paths are now reported too.

Rollback deletes stored blobs only once `deleteSkill` confirms the skill
is gone; otherwise the blobs stay and the request answers 500
`skill_import_rollback_failed`, because a surviving skill pointing at
deleted files is worse than unreferenced storage.

The failure list scrolls instead of overflowing the dialog, and a
response that resolves after the dialog was dismissed no longer
repopulates it for the next session.

* fix: Ignore Late Skill Import Successes

* fix: Make Skill Import Rollback Retryable

* fix: Propagate Incomplete Skill Cleanup

* fix: Settle Rollback Blob Cleanup

* test: Type Partial Cleanup Fixture

* fix: Preserve Settled Skill Cleanup Steps

* fix: Export Skill Cleanup Step Type

* fix: Reconcile Partially Deleted Skills

* fix: Reconcile Skill Rollback Caches

* style: format cleanup reconciliation changes

* fix: complete skill rollback cleanup

* chore: leave generated locale catalogs unchanged

* fix: preserve synced skill blob cleanup

* test: type synced blob cleanup mock

* fix: Validate Archive Paths and Bound Rollback Cleanup

---------

Co-authored-by: Lia <lia@librechat.ai>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-19 12:41:30 -04:00
lia-by-librechat[bot]
2452f499a8
🗼 fix: Serialize MCP OAuth Token Refresh Across Replicas (#15993)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Has been cancelled
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Serialize MCP OAuth token refresh across replicas

`MCPTokenStorage.inflightRefreshes` coalesces refresh-token redemptions
inside one Node process. Behind a load balancer without session affinity,
one user's concurrent requests land on different replicas, each reads the
same not-yet-rotated refresh token and redeems it. RFC 9700 servers treat
the second redemption as replay and revoke the whole grant family, so the
user is asked to authorize the MCP server again.

`forceRefreshTokens` now takes a cross-replica refresh flight before
redeeming, using the same `FlowStateManager.acquireLease` primitive the
OAuth teardown fence uses under a distinct key. A replica that waited for
the flight adopts the tokens the holder rotated instead of redeeming
again; when the flight is still held after the wait window it falls back
to the unfenced redemption every earlier release performed.

* fix: Redeem unfenced when the refresh flight lease store fails

* fix: Fence MCP refresh waiters instead of redeeming unfenced

A waiter that could not take the cross-replica refresh flight fell through to
an unfenced redemption after 10s, while the holder's own stale abort does not
fire until 60s. A refresh taking 11 to 60 seconds therefore still let two
replicas redeem one refresh token, the replay this fence exists to prevent.

The waiter now polls until it either adopts the tokens the holder rotated or
acquires the flight, and fails as MCPTokenRefreshUnavailableError rather than
redeeming beside a live holder. `getTokens` callers defer connection recovery
on that error, and the stored credential is left intact for a later attempt.

The flight is keyed by the stored credential (tenant, user, server name)
rather than the caller's OAuth binding digest, so a rolling config change
cannot hand two replicas different locks over one stored token. The wait is
an operator lever, `oauthRefreshWaitTimeout`, clamped to half the stale
window. An adoption read that fails no longer gives up the held flight.

* fix: Annotate derived refresh-flight constants for isolatedDeclarations

* fix: Recapture the credential generation a peer published on adoption

* fix: Hold the MCP refresh flight until redemption settles

Four corrections to the cross-replica refresh flight.

The flight lease equalled the window that aborts a stalled redemption, on
the claim that an expired flight could never belong to a redemption still
able to reach the token endpoint. Aborting proves no such thing: the
request may have been processed with its response lost, and a stalled
event loop can delay the abort past its own deadline. The lease now
outlives the abort, so a peer cannot redeem a credential the provider has
already rotated. A live replica still releases on settle, so the margin is
paid only by one that died holding the flight.

The credential snapshot was taken after the first failed acquisition, so a
holder that stored and released in that gap was snapshotted post-rotation:
the next attempt saw an unchanged record and redeemed the credential it
should have adopted. It is taken before the first attempt now.

`oauthRefreshWaitTimeout` accepted zero while the runtime mapped every
non-positive value to the default, so the config validated and then
behaved contrary to its value. Zero is rejected at load.

Adoption announced a second credential change through
`handleOAuthRefreshSuccess`, whose `onOAuthCredentialsChanged` advances
authorization state after persistence this replica did not perform. That
retired the generation recaptured beside it and fenced the build against
its own tool publication. Adoption now updates the local token-flow cache
and recaptures, without announcing.

* fix: Check every held MCP refresh flight for a peer's rotation

Four corrections, two of them consequences of moving the credential
observation ahead of the first lease attempt.

That move created a storage read inside `beginRefreshFlight`, and its
failure reached a handler written for a lease-store outage, so a transient
read error became an unfenced redemption beside a live peer. The flight
now handles its own reads: losing the observation costs adoption and
nothing else, and only the lease store failing may redeem unfenced.

The observation was also never compared when the first acquisition
succeeded. A peer that rotated and released before this replica contended
left an acquisition that looked uncontended, and its fresh credential was
redeemed a second time. Every acquisition now runs one rotation check,
which also collapses two code paths into one.

`getTokens` already loads the refresh record to decide a refresh is
needed, so it is passed on as the observation baseline instead of read
again, and the read taken under the flight is reused as the credential
redeemed. Two reads on a latency-counted path where there were three.

`runSilentRefresh` collapsed contention into null, sending the 401 path to
interactive OAuth, so a slow peer prompted the user to authorize a server
whose credential was about to be valid. The retryable outcome now travels
through the silent-refresh layers and the connection defers, matched by
name as well as identity because these errors cross the package boundary.

* chore: Regenerate the Agents OpenAPI spec for repositoryInstructions

The drift check added by #15928 fails on dev: #16008 added
`repositoryInstructions` to the agent schema without regenerating the
committed spec, so every branch merging dev inherits the failure. This is
the remedy the check itself prescribes and carries no hand-written change.
Drop this commit if dev regenerates first.

* fix: preserve MCP refresh outcomes across replica boundaries

* fix: keep MCP adoption fenced through publication

* fix: anchor MCP refresh adoption to rejected credentials

* fix: Complete OAuth Adoption Adapters and Isolate Legacy Flow Readers

* fix: Preserve OAuth Request Identity Through Recovery and Discovery

* fix: Fence OAuth Adoption Against All Credential Writers

* fix: Gate Coordinated OAuth Rollout and Preserve Unauthenticated Identity

* fix: Preserve OAuth Coalescing Across the Staged Rollout

* style: Sort OAuth Timeout Constant Import

* fix: Invalidate Both OAuth Token Flow Protocols on Rotation

* test: Complete the Rollback Token Flow Fixture

---------

Co-authored-by: Lia <lia@librechat.ai>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-16 21:58:23 -04:00
Marco Beretta
9633edcb88
🗣️ fix: Keep Provider Error Text on Unclassified Agent Failures (#16034)
* 🗣️ fix: Keep Provider Error Text on Unclassified Agent Failures

An upstream failure LangChain does not classify discarded the provider's own
message and answered with the generic upstream sentence plus a status, so a
gateway or privacy-proxy rejection lost the only account of what happened.

The provider text now rides along in the persisted payload as `message`,
withheld only where a content policy inspects the traffic — the same condition
`getUserFacingProviderError` and `getUserFacingRequestError` already decide by.

* fix: Harden Provider Error Detail Retention

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-16 21:32:41 -04:00
lia-by-librechat[bot]
a26a6709ff
📪 feat: Send Per-Request MCP Headers Without Hiding the Tool Catalog (#15988)
* 📪 feat: Send Per-Request MCP Headers Without Hiding the Tool Catalog

A `{{LIBRECHAT_BODY_*}}` placeholder in an MCP server's `headers` marks the
server request-scoped, and catalog discovery has no conversation to resolve it
against, so LibreChat skipped tool discovery and the server appeared with no
individual tools.

Operators can now declare those headers separately in `requestHeaders`, which
discovery omits and chat turns resolve and merge over `headers`.

* fix: normalize requestHeaders at each resolution entry point

Registering the new field in every consumer that enumerates header-bearing
config fields left five gaps: startup inspection, case-insensitive override,
the admin-configurable comparison surface, Graph token preprocessing, and
direct-bearer recovery.

Fold `requestHeaders` into `headers` at the entry of each resolution pipeline
instead, so Graph preprocessing, direct-bearer detection, `processMCPEnv` and
the transports keep reading one header map and never learn a second exists.
Catalog paths strip it at the same boundary, startup inspection included.

* fix: keep servers with chat-only headers off the shared app connection

A static `requestHeaders` map left a server eligible for app-level sharing, and
`ConnectionsRepository.loadConnection` handed the raw config to the factory, so
the shared session's own `initialize` and `tools/list` carried headers
documented as chat-only — and every later catalog read reused that session.

One session cannot serve both sides of the field. Exclude such configs from
`canUseAppConnection`, the single predicate both `connectAppServers` and the
repository's own app-level gate already consult, so the chat turn keeps its
headers on a user-scoped connection while catalog work never sees them.

* fix: normalize requestHeaders before the manager's bearer decisions

`createUserConnectionInternal` reads `usesDirectOpenIDBearerRecovery` and
`resolveDirectOpenIDBearerConfig` on the config it receives, while the merge sat
in `resolveRuntimeConfig`, a leaf helper below those calls. An `Authorization`
template declared in `requestHeaders` therefore never entered direct-bearer
mode: a `tools/list` 401 skipped the force-refresh and the connection was
returned with a stale bearer.

Normalize at the birth of that pipeline's config instead, ahead of all four
decisions, and make `getMCPAppToolsPublicationGeneration` normalize before
hashing. Connection paths hold a merged config while cache paths hold the
declared one, so without a canonical hash the two would address different
catalog slices.

* fix: Preserve MCP request header ownership across lifecycle boundaries

* fix: Narrow remote header identity before rebuilding config

* fix: Preserve request header precedence over catalog API keys

* docs: Synchronize generated agent API schemas

* fix: Ignore shadowed generated MCP user key requirements

---------

Co-authored-by: Lia <lia@librechat.ai>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-16 15:16:40 -04:00
Danny Avila
dea7346d72
🧷 feat: Carry Ask User Answers Past Compaction (#15944)
* 🧷 feat: Carry Ask User Answers Past Compaction

The user's answers to `ask_user_question` lived only in the tool result that
received them. A checkpoint summary, the tool-output pruner or a full context
window could remove that message, and the agent then asked again or acted on
a summary's paraphrase.

Every answered ask on the conversation branch is now rendered verbatim into
the run's dynamic instructions, on the first request and on every resume,
where the answer just given is read from the seeded content. Older answers
drop first past a token budget; the newest set is always kept.

New lever: `endpoints.agents.askUserQuestion.retainedAnswers` (`enabled`,
`maxTokens`), on by default with a 4096-token budget.

* 🧷 feat: Carry Ask User Answers Past Compaction

The user's answers to `ask_user_question` lived only in the tool result that
received them. A checkpoint summary, the tool-output pruner or a full context
window could remove that message, and the agent then asked again or acted on
a summary's paraphrase.

Every answered ask on the conversation branch is now quoted verbatim at the
top of the current user turn's prompt copy, in the user's own role: rebuilt
from the stored rows every turn, never persisted, excluded from the memory
copy, and counted in the prompt total with the active encoding. A warm
event-actor turn reads the branch on demand. Older answers drop first past a
token budget that includes separators; the newest set is always kept.

New lever: `endpoints.agents.askUserQuestion.retainedAnswers` (`enabled`,
`maxTokens`), on by default with a 4096-token budget.

* 🧷 refactor: Read The Warm Branch Inside The Module

The warm event-actor read lived as a conditional and a `getMessages` call in
the CJS client. The typed module now takes `historyLoaded`, the identity and
the query as dependencies, decides when to read, and owns the failure policy
for both the read and the block itself; the client passes plumbing only.

* 🧷 fix: Complete The Branch Behind A Summary Cut

`loadHistory` already stops its walk at a checkpoint summary, so the rows
`buildMessages` receives never held the answers this feature carries. The
module now detects a branch that stops short of its root, whatever cut it,
and completes it through the injected stored-row query, stored rows winning.

The block goes on the latest user-authored row so a continued generation
keeps it, and the render measures the whole block before reserving the
omission note, so a set is only dropped when something truly has to go.

* 🧷 fix: Reuse Loaded History And Read Every Ask Stamp

A post-compaction turn read the conversation twice: once in `loadHistory`,
which cuts at the checkpoint summary, and again in the module to complete
the branch. The loader now keeps the rows it read and the module prefers
them, querying only for a turn that made no history read.

Two stored shapes were missed: a pause record whose `args` is the bare
question string, now read as the legacy question, and a call whose input
failed validation, whose persisted error output is no longer quoted as an
answer.

* 🧷 fix: Keep The Block Off The Stored Row And Out Of Memory Copies

Array-form content is replaced instead of edited through the array the
stored row shares, the block's cost is measured as two fresh counts of the
same copy rather than against a possibly calibrated stored count, the
memory copy is only built when memory extraction will read it, the loaded
rows reach the agents client through a hook and are released once the block
exists, and the tool-call-id merge is gone: rows are unique by message id
and providers may reuse ids across turns.

* 🧷 refactor: Apply The Answers Block Inside The Module

The placement lookup, the prepend, the count delta and the memory-copy
decision lived in the client after the context kickoff. They now live in
`applyRetainedAnswers`, which receives the rows, the prompt copies, the
count map, the counter and the memory dependency; the client keeps one
call and passes plumbing.

* fix: Apply Retained Answers After Summary Replay 🧷

* perf: Count Retained Context Exactly Within Its Budget 🧮

* fix: Preserve Retained Counts Across Checkpoint Recounts 🧾

* fix: Isolate Retained Context Counting and Provenance 🧵

* fix: Replace Warm Retained Context and Verify Answer Provenance 🪪

* fix: Use Workspace-Compatible Array Traversal 🧭

* fix: Refresh Retained Context at the Warm Turn Boundary 🧲
2026-09-14 21:35:07 -04:00
Danny Avila
2ffee7e143
🪂 feat: Opt-In Text Fallback for Files No Tool Can Read (#15940)
* 🪂 feat: Opt-In Text Fallback for Files No Tool Can Read

An upload routed to tools (`llmDeliveryPath: none`) reaches nothing when the
agent handling the turn runs neither Run Code nor File Search, or runs one that
cannot read the type. With `fileConfig.endpoints.<endpoint>.textFallbackWithoutTools`
(also accepted at the top level and inherited from `default`), such a turn
delivers the text extracted at upload instead. The setting is off by default,
which keeps today's behavior.

- Upload: when the setting is on and an inferred `none` route finds no reading
  tool, the built-in document parser or native text reader stores the text on
  the file, best effort, under the extracted-text size cap and content policy.
- Turn: `resolveTurnLLMDeliveryPath` re-resolves each inferred route against
  the tools the agent runs, and `applyTurnTextFallback` marks copies of the
  records a turn loads, so endpoint filtering, limits, inspection, history
  replay and steer media all see the text the turn delivers. The stored route
  is never rewritten.
- Explicit destinations, records predating routing, and turns whose tools are
  unknown are left as they are.

* 🧪 test: Resolve Custom Endpoint Fallback Under the Endpoint Name

Uploads route a custom endpoint agent by its saved provider, the endpoint name. initializeAgent marks turn copies before getProviderConfig swaps the provider for the backing client, and the endpoint keeps the name afterward, so the client and child encoders resolve the opt-in under the same endpoint. Cover both the initialization path and the post-initialization agent shape.

* ♻️ refactor: Share One Extracted-Text Storage Cap Across Uploads

The fallback defined its own copy of the 15 MiB cap context uploads already apply to extracted text. Export MAX_STORED_EXTRACTED_TEXT_BYTES from files/extract and use it in both places, documenting that it bounds what a MongoDB file document can hold rather than operator policy.

* 🧭 fix: Keep Fallback Text Wherever a Later Turn Can Deliver It

Extraction skipped uploads filed under a reading tool, so a handoff agent
without that tool, or the same agent after its tools or grants changed, had
no text to fall back to. Store fallback text for every inferred route on a
message attachment; files kept on an agent's tool resources never reach a
prompt and still skip it.

Turn marking only ran while the fallback was enabled, so a stored tool-routed
record whose type the endpoint now routes to text stayed marked `none` and
`extractFileContext` skipped it. applyTurnTextDelivery (renamed from
applyTurnTextFallback) now marks any stored `none` record with text that the
turn resolves to text, and returns early when no such record is loaded.

* 🚦 fix: Admit Every Attachment by the Route Its Turn Delivers

A record upload routed to tools that the current endpoint sends to the
provider stayed marked `none` on the turn copy, so admission skipped it while
BaseClient encoded its bytes, bypassing attachment count and size limits.
applyTurnDelivery (renamed from applyTurnTextDelivery) now gives every record
whose route upload inferred the route the turn resolves, before endpoint
filtering, limits and inspection, matching the run-file encoder.
hasInferredLLMDeliveryPath shares the inferred-route rule with the resolver.

Fallback extraction selected the document parser inside the CJS upload path.
resolveUploadFallbackText now takes the upload and picks parseDocument or
parseTextNative itself, with the extractors injectable, so process.js passes
request data only. Its tests run the real parser against the xlsx fixture and
the real native reader against temporary files.

* 🎚️ fix: Read Custom Endpoint Dialect From Config on Both Sides of Init

initializeAgent materializes turn routes before getProviderConfig swaps the
saved custom endpoint name in agent.provider for its backing client, so
routing read that name as a non-OpenAI dialect. Media a custom endpoint opted
into could then be materialized off the provider path, skipping admission,
while the finalized client still sent the bytes. resolveAgentDeliveryRouting
now reads the dialect with getCustomEndpointProvider, as upload does, which
holds before and after the swap.

The upload fallback plan now keys on the chosen-destination marker the turn
resolver reads, so an upload recorded as chosen, including every legacy
chooser upload, never pays for text a turn cannot deliver.

* 🛡️ fix: Apply Turn Routes Only While the Record Stays Admitted

Resolution while initializing and at delivery can disagree: delivery reads a
Responses API choice only the finished client config holds. Applying a route
that takes a record out of admission, off the model path or to text it never
stored, would then skip the limits for a file the client still sends.
applyTurnDelivery now applies a route only while the record stays model-bound;
admitting a record the turn leaves out cannot slip past a limit.

* 🕹️ feat: Run Declared Project Actions on Attached Workers (#15943)

* feat: use declared attached project actions

* Complete named action definitions and approval previews

* Sort named environment imports

* Sort native environment fixture imports

* 🔓 fix: Keep Code Approval Mode Selectable Mid-Run (#15938)

* 🔓 fix: Keep Code Approval Mode Selectable Mid-Run

The composer's code approval mode selector was disabled while a run streamed, and a mode picked during a run was reverted when the final or abort event merged the server conversation back. The selector now stays usable in flight and both merges retain a mid-run selection, which the next send carries.

* 🔓 fix: Retain Mid-Run Approval Mode Across Cache and Recovery

Stamp the live approval mode onto the submission conversation at send so the retention baseline matches what was read, keep a retained mode in the conversation query cache on final and cancel, and rebuild failed or aborted conversations from a preset that carries the mid-run pick.

* 🔓 fix: Scope Retained Approval Mode to the Submitted Conversation

Retention now requires the live conversation to be the one the run submitted, or the id the server assigned to a new chat, so navigating elsewhere mid-run cannot write that conversation's mode back. A single recovery helper rebuilds failed or aborted conversations from the retained preset and patches the detail cache for every recovery site.

* 🔓 refactor: Keep the Local Approval Mode on Every Server Merge

Replace per-path retention with one rule: a locally picked code approval mode is newer than any server copy of the same conversation, so the final and abort merges and the error recovery preset keep it. The selector writes the pick into the conversation's detail cache, which navigation rebuilds from, so the pick is conversation-scoped without reading the index-global atom from the SSE handlers.

* 🔓 fix: Prefer the Live Conversation When Caching the Final Merge

The final handler's detail-cache merge now takes the local approval mode from the open conversation when it is the one that finished, falling back to the cached record only for a conversation that is no longer on screen.

* 🔓 fix: Keep the Local Approval Mode Through Settled-Start Reconciliation

The resumable transport's settled-start reconciliation and replacement handoff wrote a fetched conversation straight into conversation state and the detail cache. Both now apply the same local-mode rule as the event handlers, treating a new chat's pending id as the settled conversation's own.

* 🔓 fix: Seed the Detail Record When a Mode Is Picked

A conversation that has its id but no detail record yet now gets one from the live conversation when a mode is picked, the same key the resumable transport seeds optimistically, so recovery and navigation find the pick. A chat with no id yet still keeps the pick in conversation state alone.

* ⬆️ chore: bump agents to 3.8.7 (#15947)

* 🧭 refactor: Settle Turn Delivery Routing Once in Agent Initialization

`initializeAgent` resolved the provider and its client options only after the
turn's files were loaded and admitted, and the two delivery readers each rebuilt
the attachment routing from the agent object at their own moment: `BaseClient`
took the custom-endpoint dialect from the already-swapped `agent.provider`, the
run-file encoder from whatever child config it was handed.

Move `getProviderConfig`/`getOptions` ahead of file discovery, where nothing in
between fed them, and settle one `deliveryRouting` value with every input final:
the file policy under the endpoint's own name, the dialect its config declares,
the Responses API decision the model call uses, and the transcription setting.
`InitializedAgent`, the child encoder and `BaseClient` consume that value;
`resolveTurnLLMDeliveryPath` is the one place a stored route is resolved again.

* fix: Preserve attachment reachability across handoffs and late steers

* test: Type the untrusted attachment reference fixture

---------

Co-authored-by: Ravi Kumar L <upman@users.noreply.github.com>
2026-09-14 21:02:54 -04:00
Danny Avila
854645ffc0
🔁 feat: Retry OpenID Discovery After Startup Failures (#15899)
* fix: retry OpenID discovery after startup failures

* fix: keep OpenID background retries resilient

* refactor: move OpenID retry policy into API package

* 🔁 feat: Expose OpenID Discovery Retries Through Registration Config

* 🧭 refactor: Own OpenID Discovery Registration and Defaults in TypeScript

---------

Co-authored-by: insoln <insoln@ya.ru>
2026-09-13 23:16:26 -04:00
Danny Avila
696bcb6c1e
🏦 fix: Reserve Credits for In-Flight Requests (#15879)
* 💳 fix: Reserve Balance for In-Flight Requests

Admit each balance-checked request against the credits that other
in-flight requests have not already reserved, and hold its prompt cost
on the balance record until the turn settles. Apply a due auto-refill in
the same fenced write so one refill window credits exactly once.

- data-schemas: reserveBalance / releaseBalanceReservation replace
  createAutoRefillTransaction; the write is a compare-and-swap on
  tokenCredits, reservations and lastRefill (no pipeline updates)
- balance.reservations is select:false so existing reads never see it
- checkBalance returns a reservation handle; BaseClient and the
  assistants controllers release it once the turn settles
- balance.reservationTtlMs bounds how long an unreleased reservation
  counts (default 30 minutes)

* 🧮 refactor: Admit Reservations Without Contention and Record Refills Durably

- admission is a conditional $push/$inc against a reservedCredits total,
  so concurrent admissions of a funded balance commit on their first write
- expired reservations are pruned per element with their credits
- auto-refill is fenced on every setting its decision reads and leaves a
  pendingRefill ledger marker in the same write; any later read records the
  transaction under the fixed id and clears the marker
- a missing record is created with $setOnInsert inside reserveBalance
- reservation lifecycle moves to packages/api (createBalanceReservations,
  withBalanceReservations); BaseClient releases once usage is recorded

* 🪪 fix: Resolve Each User to One Balance Record

- balance records a creator inserts are keyed by the user id, so creators
  racing on a missing record converge on one document; every balance read
  and write resolves a user to their oldest record, so existing duplicates
  stay inert
- upsertBalanceFields takes insert-only fields; the balance-config
  middleware, schedules and lazy initialization pass the starting credit
  through them so it never overwrites a record another writer created
- the auto-refill write is also fenced on reservedCredits
- the scheduled-chat balance pre-skip counts reserved credits as unavailable

* ⏳ fix: Count Only Unexpired Reservations in Scheduled Balance Checks

findBalanceByUser with includeReservedCredits now totals the reservations
that have not expired, so a reservation left by a crashed request stops
pre-skipping scheduled chats at its expiry instead of when a later
reservation write prunes it.

* 🔁 fix: Renew Balance Reservations While Their Request Runs

A reservation is renewed every half TTL until it is released, so only a
reservation whose process stopped expires. Pruning removes a reservation
only while it is still expired at the write, so one renewed after the
pruning read survives.

* 🛟 fix: Keep Reservations Alive Through Renewal Failures and Lost Acks

- renewal reschedules itself: every half TTL, capped to the timer range,
  and a failed renewal retries within a tenth of that window
- an admission write that errors removes its possibly committed
  reservation before rethrowing

* 🛑 fix: Hold a Stopped Turn's Reservation Until Its Charge Lands

- an aborted turn lets its reservations lapse after a minute (or the TTL)
  instead of releasing them, so the Stop request's charge lands before the
  credits are admitted again
- balance.reservationTtlMs has a 10 second minimum, enforced in the schema
  and at runtime, so renewal stays bounded

* ↩️ revert: Release a Stopped Turn's Reservation When the Turn Settles

An aborted turn releases its reservations when it settles, the same as
every other exit, instead of holding them for a fixed lapse window that
refused funded requests after every Stop. The minimum reservation TTL
stays.

* ⏸️ fix: Hold Assistants Reservations Through Background Run Retries

An in-progress Assistants run continues in the background after the
handler responds; the turn's reservations now stay held until that run
settles (createBalanceReservations().holdUntil).

* 🔁 fix: Apply an Auto-Refill Once per Admission

A refill interval that is due again the moment it is applied, such as a zero
interval, let one admission refill repeatedly until the retry bound. Each
admission now applies at most one refill, as the previous check did.

* 🧹 fix: Prune Expired Reservations in One Write

An admission that found expired reservations issued one write per hold, so a
backlog left by a stopped process fanned out into as many concurrent writes.
The prune is now a single write fenced on every hold it removes; a hold renewed
after the read makes the write miss and the admission re-read.

* ⏱️ fix: Renew Reservations From Their Stored Expiry

The next renewal was scheduled half a TTL after the previous write settled,
and the first one half a TTL after the admission returned, so a slow write
could let a live hold expire before its renewal. Renewals are now scheduled
from the expiry the last write stored.

Expired holds are pruned as one bulk write of per-hold fenced updates, so a
hold that changed after the read is skipped while the rest are removed, and
the admission no longer spends a retry on the whole prune.
2026-09-13 12:53:19 -04:00
Danny Avila
9cf33cc8be
🧿 fix: Bind Login Flows to the Browser That Started Them (#15880)
* 🧿 fix: Bind Login Flows to the Browser That Started Them

Social OAuth logins (Google, Facebook, GitHub, Discord, Apple) now store a random state in an HttpOnly cookie scoped to the callback path and only exchange the returned code when the callback carries that state. Apple's form_post callback uses a SameSite=None cookie, and passport-apple's own fixed state no longer bypasses the store.

Local login, temp-token 2FA verification and admin local login reject requests a browser sent from another origin (Sec-Fetch-Site, falling back to Origin vs Host), trusting DOMAIN_CLIENT and DOMAIN_SERVER. Requests without browser fetch metadata, such as server-side callers, pass unchanged.

The sessionless OAuth callbacks drop failureMessage, which wrote to req.session and turned any rejected callback into a 500.

* 🧿 fix: Configure Login State Cookies and Localize Origin Rejections

- OAuth state cookies take secureCookie and maxAgeMs from the caller; registration.oauthStateTtlMs in librechat.yaml sets the lifetime (default 10 minutes).
- Secure deployments use a __Host- cookie at Path=/ so a sibling subdomain cannot plant a state.
- The cookie keeps up to three pending states, so logins started in separate tabs all complete, and an unmatched callback no longer discards them.
- passport-apple's preset state is dropped by deferStateToStore in packages/api.
- The Origin fallback compares scheme and host against the request itself.
- The same-origin rejection carries ErrorTypes.AUTH_CROSS_ORIGIN, which the login and 2FA screens show as a localized message.

* 🧿 refactor: Sign Login State Against a Per-Browser Binding

- The OAuth state store keeps a random per-browser binding in a __Host- cookie (reused across flows) and signs each state as issuedAt.nonce.HMAC(JWT_SECRET, provider, binding, issuedAt, nonce). Nothing is stored per flow, so any number of tabs complete, each state expires on its own clock, and concurrent starts no longer overwrite each other.
- The experimental server entrypoint passes its app config to configureSocialLogins.
- The same-origin guard also trusts an explicitly configured ADMIN_PANEL_URL.

* 🧿 fix: Keep Separate Bindings for Racing First Logins

A browser without a login-state binding now gets one under a cookie name with a random suffix, so two tabs that start their first login at once each keep their own binding. Later starts reuse an existing binding, and a callback is accepted when its state was signed for any binding the browser presents.

* 🧿 fix: Sanitize Request-Derived Values in the Cross-Site Rejection Log

The same-origin guard logged the request Origin and path verbatim. A non-browser client fully controls the Origin header, and the console transport uses a printf format that does not escape control characters, so a crafted value could forge log lines. Strip ASCII control characters (CR, LF, NUL and the rest) before logging, matching the pattern apiNotFound already uses.
2026-09-13 11:47:29 -04:00
Danny Avila
e18606e5ce
🚀 v0.8.8-rc3 (#15859)
Some checks failed
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* 🚀 chore: prepare v0.8.8

* 🚀 chore: prepare v0.8.8-rc3

* docs: mention Agent workspace defaults
2026-09-13 02:17:48 -04:00
Danny Avila
f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00
Joseph Licata
f25242df98
🦘 feat: Share Run-Scoped Files With Subagents (#15848)
* feat: add run-scoped file sharing for subagents

Delegate current-message files through authorized execution catalogs and publish selected immutable code-output versions through the existing file pipeline.

Requires the companion agents SDK context adapter before the dependency can be released and pinned.

* fix: preserve child tool artifacts and verify file-sharing lifecycle

Keep citations and other non-sandbox artifacts on their existing delivery path while retaining private code outputs and inspected images until publication.

Cover native and text delivery, nested and sibling authorization, concurrent runs, cancellation cleanup, and checkpoint recovery through the browser harness.

* test: align file-sharing fixtures and integration cleanup

* test: repair LibreOffice conversion fixtures

* test: verify deployment skill bytes across checkout formats

* fix: align subagent file budgets and refresh boundaries

* chore: update agents SDK to 3.8.6

* refactor: move code output processing into API workspace

* fix: enforce shared file budgets for parents and teams

* fix: declare the shared file encoder interface
2026-09-12 20:14:22 -04:00
Danny Avila
442d477acf
🛎️ fix: Preflight MCP Readiness for Scheduled Chats (#15782)
* fix: Validate Scheduled MCP Readiness 🕰️

* fix: Keep Schedule Permission Store Failures Retryable

* perf: Parallelize Scheduled MCP Preflight

* fix: Preserve Scheduled MCP Invariants

* fix: Close scheduled MCP recovery gaps

* fix: harden scheduled MCP recovery

* fix: bound scheduled MCP validation

* fix: harden scheduled MCP admission bounds

* fix: align scheduled MCP graph admission

* fix: bound scheduled MCP admission

* fix: align scheduled MCP preflight with runtime

* fix: persist scheduled MCP failure outcomes

* fix: match scheduled MCP runtime semantics

* fix: parallelize scheduled MCP admissions

* fix: mirror scheduled subagent traversal

* fix: enforce scheduled subagent config limits

* fix: bound scheduled MCP initialization

* fix: mirror lazy subagent metadata budgets

* fix: complete scheduled MCP recovery

* fix: attribute missing MCP tools

* fix: harden scheduled MCP admission

* fix: preserve scheduled admission invariants

* fix: retain inaccessible MCP requirements

* fix: make MCP admission recovery durable
2026-09-09 22:18:45 -04:00
Danny Avila
65bb64bc68
🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784)
* fix: recover CodeAPI uploads after throttling

* fix: cover eager CodeAPI uploads

* fix: scope CodeAPI upload recovery

* test: clarify upload recovery deadlines

* fix: isolate CodeAPI recovery state

* fix: cancel CodeAPI recovery waits

* test: provide CodeAPI registry in handler fixture

* fix: propagate CodeAPI recovery controls

* fix: bound CodeAPI recovery by run

* fix: preserve skill upload cancellation

* test: update deferred loader signal contract

* fix: preserve recovery cancellation

* fix: cancel recovery source reads

* fix: classify Axios upload cancellation

* fix: preserve lazy provisioning cancellation

* fix: close resource recovery boundaries

* fix: preserve provisioning failures compatibly
2026-09-09 15:48:11 -04:00
Danny Avila
9f311b810d
🪔 feat: Expose Bounded Attached Bash Timeouts (#15776)
* 🕰️ feat: Add BYOM Bash Execution Timeouts

* 🔧 fix: Type Bash Timeout Bounds

* 🔒 fix: Bound BYOM Bash timeouts by policy

* 🧪 test: Propagate BYOM timeout policy fixture
2026-09-09 09:22:33 -04:00
Danny Avila
559fd7d607
🪓 feat: Cancel Background Tool Invocations (#15773)
* ✨ feat: Cancel ordinary background tools

* 🛡️ fix: Keep cancellation receipts rollout-safe

* 🛡️ fix: Harden Background Tool Cancellation

* 🧪 test: Cover cancellation config wiring

* 🎨 style: Sort cancellation imports

* 🧪 test: Align Cancellation Claim Result

* 🛑 fix: Complete Background Cancellation Lifecycle

* 🎨 style: Align Cancellation Test Formatting
2026-09-09 09:06:26 -04:00
Danny Avila
af6e97f0ae
🏠 fix: Keep MCP Caches Fresh on Demand (#15765)
* fix: stop refreshing MCP caches on every chat message

* fix: reconcile MCP caches independently of chat turns

* fix: scope MCP polling to visible controls

* fix: refresh status in the unpinned MCP submenu
2026-09-09 00:36:04 -04:00
Danny Avila
ab6cde0c79
⏳ fix: Separate Temporary and General Chat Retention (#15734)
* fix: separate temporary and general chat retention

* fix: inherit retention when branching messages

* fix: harden retention lifecycle boundaries

* fix: preserve subagent retention context

* fix: trust stored chat retention for uploads

* perf: overlap generated file retention lookup

* fix: derive retention from trusted chat state

* fix: preserve retention across the full turn lifecycle

* test: accept the conversation retention read adapter

* fix: carry retention deadlines through durable job saves

* fix: resolve retention at cached and new chat admission

* fix: resolve direct tool retention before execution

* fix: preserve retention across assistant writes and direct tools
2026-09-08 11:22:54 -04:00
Danny Avila
5ca1093ed9
🛫 feat: Enable Full Access for BYOM Coding (#15718)
* feat: Enable Full Access for BYOM Coding 🛫

* test: Preserve endpoint policy under Full Access

* fix: Gate Full Access on expected root agent metadata
2026-09-07 22:46:01 -04:00
Danny Avila
f1bd85498e
🪫 fix: Recharge Direct OpenID MCP Bearer Sessions (#15704)
* fix: recover direct OpenID MCP bearer sessions

* fix: harden direct bearer recovery lifecycle

* fix: close direct bearer lifecycle races

* test: align direct bearer checks with current dev

* fix: unify direct bearer recovery budget

* fix: infer direct bearer recovery from placeholder

* fix: preserve bearer precedence and recovery budget

* fix: propagate bearer recovery across joiners

* fix: preserve direct bearer fallback without session provider

* fix: close direct bearer recovery race windows

* chore: sort bearer recovery test imports

* fix: align bearer recovery with OAuth cleanup lifecycle

* fix: fence request-scoped MCP creation during config mutations

* fix: retain suspended MCP catalog authentication outcomes

* fix: cancel MCP bearer recovery and reuse refreshed credentials

* fix: close MCP bearer recovery lifecycle boundaries

* fix: preserve ordered catalog authentication rejection

* fix: isolate bearer recovery across request and transport boundaries

* fix: preserve shared recovery ownership and credential precedence

* fix: wait for recovery borrowers and validated OpenID publication

* fix: observe the validated browser publication flight

* fix: cancel abandoned OpenID publication followers

* fix: fence refresh grant admission after lease acquisition
2026-09-07 21:05:26 -04:00
Danny Avila
e370ce4b21
🎒 fix: Bound Agent Attachment Context (#15694)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
2026-09-07 08:16:34 -04:00
Danny Avila
b181ef0232
🪬 feat: Configure Langfuse Trace User Identity and Metadata (#15667)
* 🪪 feat: Configurable Langfuse Trace User Identity and Metadata

* 🪪 fix: Prefer createRun's conversationId and warn once on an unset userIdField

* 🪪 fix: Validate Langfuse trace field names at the point of use

* 🧹 chore: Sort identity imports

* 🪪 fix: Source the trace model label from the initializer, not a prompt-affecting option

* 🧪 fix: Build the trace context without a prototype so partial client contexts resume

* 📦 chore: Bump `@librechat/agents` to v3.8.0 for the Langfuse trace userId override
2026-09-06 17:30:23 -04:00
Danny Avila
8f6efaf6ed
🛂 feat: Add Configurable BYOM Enrollment Policy (#15673) 2026-09-06 15:04:43 -04:00
Marco Beretta
81154adcea
🍋 feat: Add AMD Lemonade Provider Recognition and Branding (#15664)
* feat: add AMD Lemonade provider recognition and branding

* test: enforce packaged provider asset paths
2026-09-06 20:46:08 +02:00
Dustin Healy
aec2121923
🔐 feat: Add M2M Authentication for Agent Management (#15555)
* feat: add M2M authentication for Agent Management

* fix: Harden Agent Management authentication

* fix: Preserve OIDC authentication safeguards

* fix: Normalize management principal IDs

* fix: Bound OIDC signing key lookups

* fix: Allow non-browser Agent Management clients
2026-09-03 20:07:00 -04:00
Danny Avila
f9f1b2fb95
✨ v0.8.8-rc2 (#14845)
Some checks failed
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* chore: prepare v0.8.8-rc2 release

* test: satisfy mongo meili static checks

* test: avoid unsafe logger spy assertion

* chore: refresh v0.8.8-rc2 release

* test: restore mongo meili test isolation

* chore: normalize v0.8.8-rc2 versions

* docs: refresh v0.8.8-rc2 release surfaces
2026-09-02 22:18:48 -04:00
Danny Avila
d5b2a8580e
📖 feat: Add Claude Fable 5.1 Support (#15512)
Some checks are pending
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Claude Fable 5.1 (`claude-fable-5-1`, GA 2026-09-01) succeeds Claude Fable 5
at the same input/output prices, with a 1M context window and 128K max output.
The Mythos-class helpers added for Fable 5 already match it — `isMythosClassModel`
tests `claude-(?:fable|mythos)[-.]?\d`, so adaptive thinking, summarized-display
opt-in, sampling omission, 1M context, the 128K output clamp, prompt caching,
the Vertex multi-region gate, and the Bedrock PDF exemption all cover 5.1
without a code change. What was missing was registration and pricing.

Cache reads are the one rate that differs, and the substring matcher would have
silently gotten it wrong: `claude-fable-5-1` falls through to the `claude-fable-5`
key, which prices cache hits at the usual 0.1x base input ($1/MTok). Fable 5.1
and Mythos 5.1 price them at 0.025x ($0.25/MTok), so every cache read would have
billed 4x — on exactly the long agentic sessions this model is built for.

- Register `claude-fable-5-1` in the shared Anthropic list and
  `global.anthropic.claude-fable-5-1` in the Bedrock list
- Add 1M context / 128K output entries for `claude-fable-5-1` and
  `claude-mythos-5-1`
- Add $10/$50 rates and `{ write: 12.5, read: 0.25 }` cache rates for both
- Update the `.env.example` and Vertex `librechat.example.yaml` examples
- Test that 5.1 resolves to its own key rather than collapsing onto 5.0, across
  token maps, pricing, cache rates, and Mythos-class detection

The three documented breaking changes from Fable 5 need no handling here:
LibreChat never sends a forced `tool_choice` on the Anthropic path (the agents
SDK omits the param when unset, so the API default `auto` applies); thinking
blocks an earlier model can't read are dropped server-side, unbilled; and
history is rebuilt from stored content parts rather than replayed signed
thinking blocks, so the prefix-binding check has nothing to reject.

Refs #15506
2026-09-02 10:45:54 -04:00
Danny Avila
057039570b
🔭 feat: Audit PII Filter Findings (#15508)
* feat: add audit-only content filter action

* refactor: align filter action typing and tests

* fix: preserve audit-only filter behavior

* fix: enforce audit mode across content paths

* fix: honor audit mode for canonical files
2026-09-02 10:36:06 -04:00
Danny Avila
51edb2a660
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* style: format code execution headers

* fix: close self-service pairing gaps

* fix: satisfy code environment type checks
2026-09-01 18:46:33 -04:00
Danny Avila
a9ccac8656
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing

* fix: satisfy code environment type checks

* fix: secure code environment administration

* fix: isolate code pairing control plane

* fix: validate code pairing control responses

* fix: secure code pairing transport

* fix: validate code pairing wire format

* fix: harden pairing secret lookup
2026-08-30 17:12:20 -04:00
Danny Avila
c1cb591d49
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments

* fix: include code environment in lazy agent type

* fix: harden stateful environment routing

* fix: complete code environment route isolation

* fix: declare code environment map type

* fix: preserve configured code execution routes

* fix: harden stateful environment updates

* test: assert route-scoped sandbox readiness

* fix: harden stateful environment lifecycle

* fix: isolate migrated code sessions

* test: assert route-qualified code sessions
2026-08-30 15:38:45 -04:00
Danny Avila
16c2bb4149
📬 feat: Establish Background Continuation Admission (#15348)
* feat: establish background continuation admission

* fix: preserve routed subagent wakeup controls

* fix: probe configured subagent task store
2026-08-30 06:51:36 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Danny Avila
03fa98eb37
🕸️ feat: Complete Keenable Web Search Provider (#15288)
* feat: wire Keenable web-search provider into config, schema, and UI

Keenable landed as a search provider in @librechat/agents (#285, shipped in
3.2.58+), but LibreChat did not yet expose it. This adds the config/schema/UI
glue so it can be selected, mirroring the existing Tavily provider.

- data-provider: add `keenable` to SearchProvider type + SearchProviders enum,
  keenableApiKey/keenableApiUrl schema fields, and a keenableSearchOptions block
  (maxResults, site, attributionTitle, timeout).
- data-schemas: register keenable in webSearchAuth.providers and default the
  key/URL placeholders in loadWebSearchConfig.
- api/web: pass keenableSearchOptions through to the provider and handle
  Keenable's keyless model. Unlike other providers it authenticates with no key
  (the public endpoint), picking up an optional key/URL when set; the URL
  override is SSRF-preflighted like other user-provided URLs.
- client: add Keenable to the provider dropdown with an optional API-key input.
- docs: document KEENABLE_API_KEY/KEENABLE_API_URL in .env.example and a
  webSearch example in librechat.example.yaml.
- tests: keyless + keyed auth resolution, config defaults, and schema parsing.

* fix: ESLint no-unused-vars and clarify Keenable yaml example

- Remove the now-unused RerankerTypes import in data-schemas web.ts (the lint
  job runs with --max-warnings 0 on changed files, so this latent warning failed
  CI once the file was touched).
- Note in the librechat.example.yaml Keenable stanza that a scraper (and
  reranker) is still required for web search to load, and include a Firecrawl
  scraper in the example.

* chore: fix import order drift (sort-imports)

* feat: add Keenable as a keyless scraper and select it without a pinned provider

The Keenable scraper landed in @librechat/agents#337, so wire the scraper
category the same way the search provider already is: `scraperProvider:
keenable` reads pages through Keenable's public fetch endpoint with no key
(a key only lifts rate limits, and the endpoint is overridden with
KEENABLE_FETCH_URL). Paired with `rerankerType: none` this makes a fully
keyless web-search stack possible for the first time.

Also closes the Codex finding on this PR: because none of Keenable's auth
fields are required, the generic auth loop skips it whenever it isn't pinned,
so a key submitted through the API-key dialog (which cannot pin a provider)
left the providers category unauthenticated. Keenable is now selected in that
case, gated on one of its values actually being present so installs that
configured nothing keep their current behavior. The scraper gets the same
fallback, additionally gated on Keenable being the resolved search provider,
so it never silently scrapes for another provider.

* fix: select the Keenable scraper from a supplied key, not only for Keenable search

The API-key dialog submits credentials and cannot pin a provider, so choosing
Keenable as the scraper while search stays on Serper/SearXNG/Tavily had no
effect: the unpinned-scraper fallback required Keenable to also be the resolved
search provider.

A supplied Keenable value now triggers it as well, which is the only signal the
dialog can send. The fallback still runs only when no keyed scraper
authenticated, and with neither trigger the category stays unauthenticated, so
a deployment that never configured Keenable is unaffected.

Note the fully keyless choice still cannot be expressed through the dialog:
Keenable's key is optional, so picking it with no key submits nothing at all.
librechat.example.yaml now documents pinning scraperProvider: keenable for that
case.

* style: Sort Keenable imports

* fix: Harden Keenable auth resolution

* fix: Preserve Keenable selection intent

* fix: Fail closed on invalid web search auth

* fix: close keenable auth gaps

* style: sort web auth imports

* fix: preserve web search selection integrity

* fix: isolate web search auth ordering

* fix: silence expected credential misses

* chore: bump agents sdk

* fix: preserve web search preference ownership

* fix: forward cleared Keenable endpoint

* style: sort web search hook imports

* fix: require intent for credential clears

---------

Co-authored-by: Ilya Bogin <ilya.bogin@keenable.ai>
2026-08-27 14:29:20 -04:00
Danny Avila
8ba76507ae
🧷 fix: Verify Scheduled HITL State Before Pausing (#15284)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
* fix: require durable storage for scheduled HITL

* fix: verify scheduled HITL checkpoints before pause

* fix: narrow scheduled HITL capability checks

* fix: honor ask tool filters in schedule preflight

* fix: enforce durability across scheduled HITL resumes

* fix: scope scheduled HITL admission to active tools

* refactor: move scheduled HITL admission to TypeScript

* fix: close scheduled HITL admission gaps

* fix: bind scheduled pauses to current checkpoint

* fix: cap pending actions at binding deadline

* fix: reject expired pending actions

* fix: guard approval deadlines atomically
2026-08-27 11:16:24 -04:00
Danny Avila
52fa88fc54
📬 feat: Serialize Bound Actor Event Mailboxes (#15260)
* feat: serialize bound actor event mailboxes

* test: harden actor mailbox batch ordering

* fix: treat batch roots as mailbox authority

* test: pin mailbox replay semantics

* fix: preserve mailbox authority through settlement
2026-08-26 10:02:47 -04:00
Danny Avila
68fc46a055
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks

* fix: fence event actor checkpoint uncertainty

* fix: satisfy event actor type contracts

* fix: make actor reconciliation recoverable

* fix: preserve event actor lifecycle transitions

* fix: fence event actor lifecycle outcomes

* fix: enforce event actor lifecycle ownership

* fix: retain event actor settlement proof

* 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal

Repair and compensation deleted the reconciliation row they resolved, which
was the only durable proof that the invocation had already applied an external
action. A delayed duplicate owner could then reacquire the same invocation id
and repeat that action. Both resolutions now retire their receipt to `settled`
and record how it settled, so the same-id tombstone survives; `history_repaired`
and `action_compensated` still force a cold rebuild. Compensation undoes the
effect without re-authorizing the delivery, so a legitimate retry must arrive
under a new invocation id. A retried repair converges on its own receipt.

Bound the journal so a long-lived actor cannot grow its conversation document
without limit: a new fence is admitted only when no active lifecycle row
exists, so a capped push can evict nothing but the oldest settled receipts.

Stop shipping the unbounded source payload on every bound-child continuation.
It rode the delivery body regardless of the feature flag while the sibling
`fire` body deliberately sends event identity alone, so a large webhook payload
could push a previously working delivery past the chat route's body limit. The
actor binds an invocation from identity and never builds the prompt from it.

Blank the positional token map on warm continuations. It is derived from the
full DB history, while a warm run executes on checkpoint-restored state, so its
indices address different messages and the pruner never recounts them —
misattributing cached counts to the wrong messages in both directions.

Keep the replaced-claim exit on its cleanup path when preserving reconciliation
fails: the committing CAS already left a blocking row, so the failed status
upgrade costs provenance, not safety.

* 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry

Give the warm-continuation client test a populated token map and a real
cross-run summary so its assertions bite: the positional map must arrive
blank (checkpoint-restored state no longer matches DB-derived indices, and
the pruner never recounts a populated entry) while `initialSummary` must
pass through unchanged — it rides the system tail and summarizes
pre-boundary turns that were excluded from the very history the committed
checkpoint was built from, so blanking it would silently drop context no
warm run can recover.

* ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal

A compensated receipt still tombstones its invocation id, but its external
effect was explicitly undone — the terminal handler nonetheless replayed
every settled lifecycle's stored action as authoritative and settled the
public outcome as applied, telling an action-aware source the operation
stands and suppressing the new-invocation retry compensation requires. The
handler now settles a compensated invocation as failed with an explicit
compensation error, overriding even fresh applied run evidence from a
replayed generation; verified and repaired receipts continue to replay
applied.

Receipt eviction is now primarily age-based: a stale same-id owner is
bounded by time, not by how many newer invocations settle, so the previous
count-only slice let a high-rate actor evict a tombstone while its delayed
duplicate owner could still wake and repeat the action. Admission prunes
only settled receipts older than a retention window that dwarfs every
generation, job, and delivery-retry lifetime, and the count cap is demoted
to a raised document-size backstop.

* 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts

The terminal handler read its lifecycle snapshot, verified history, and then
settled the public outcome — so a compensation resolving the same receipt
during that window lost: the handler settled applied from its stale snapshot
and no retry could ever change the replay-identity-locked outcome. The
receipt's status CAS is now the serialization point: verification resolves
the receipt BEFORE settling, whichever transition wins determines the public
outcome, and a crash between resolve and settle converges through the
retained receipt's replay. The verified-replay probe requires the receipt's
own resolution, so a compensated receipt can never satisfy a verification
retry. This inverts the settle-before-receipt ordering deliberately: that
ordering guarded proof that resolution used to delete, and the receipt now
retains its full action proof through resolution.

The document-size cap is no longer an eviction quota. A receipt inside its
retention window is never discarded: when the journal holds a full cap of
unexpired receipts, new invocations are refused fail-closed until receipts
age out, making duplicate protection and document integrity simultaneous
invariants instead of a rate-dependent trade.

* 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path

Skill primes are spliced into the message list directly ahead of the newest
message, and a warm continuation forwards only that newest message — so a
checkpoint-restored actor would keep serving the prime bodies baked in at
its last cold start and never observe an edited or newly attached skill.
Until the actor head carries a context fingerprint that forces a cold
rebuild when the agent's skill context changes, agents with always-apply or
manual skill primes stay on the legacy path, which re-primes fresh bodies
every turn: correct on every event, just never warm.

* 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes

History-derived re-priming was a third path into the same staleness class:
an actor that previously invoked a skill carries no request-time prime
arrays, yet primeInvokedSkills re-resolves that skill's current body from
history each turn and the warm slice drops the reconstruction — leaving the
checkpoint's old body active after edits. The fork gate now keys on the
priming hook itself (present exactly when the skills capability is enabled)
alongside the request-time arrays, so every skill-body path routes to the
legacy rebuild until #15235's context fingerprint restores warm
continuation for skill-bearing actors.

* 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage

The executor read applied-action evidence from the run-step collection the
instant sendMessage resolved, but that collection is populated
asynchronously — an applied invocation could classify as actionless
(runSteps still empty while the tool result already streamed), discarding
its fork and stranding the actor cold while the terminal handler later
settled the same delivery as applied from the persisted evidence.

Authoritative proof is now recorded in graph context the moment the
expected tool executes: the request-owned recorder observes the tool-end
chain (which ToolNode dispatches synchronously with both input and output)
and applies the same fences as run-step evidence — exact tool name with the
MCP-suffixed form, the declared argument subset against the execution
input, an error-free result, and the background non-execution receipt
exclusion. readAppliedAction consults the receipt first; run-step
inspection remains the fallback for paths that bypass the tool-end chain.

Regression coverage reproduces the observed ordering: the real executor
commits the head from the receipt while run steps are empty, warm-continues
the next event, and never re-executes the action; recorder fences and the
receipt-first controller wiring are covered separately.

* 🎯 fix: Supply Execution Arguments to the Tool End Callback

The live Vertex + MCP canary exposed a contract mismatch the synthetic
fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end
callback with output only, while the action recorder must verify the
declared argument subset against the execution input. The receipt never
qualified, every turn fell back to cold history rebuilds, and the
tournament advanced with zero actor heads and zero retained checkpoints
while looking successful.

The execution handler owns both halves at the same moment, so the fix is
at the source rather than a correlation store: ToolEndCallbackData gains
the executed call's input and every callback site passes tc.args. A
handler-level regression drives the real createToolExecuteHandler and
asserts the callback receives both fields; recorder regressions pin the
production shapes — an output-only tool end must starve an
argument-fenced receipt rather than trust an unfenced match, and still
qualifies a name-only expected action.

* 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions

The background-claim callback reports the ORIGINAL tool's name for artifact
attribution on the poll turn that harvests a completed task. A name-only
expected action could therefore be impersonated by work some earlier turn
dispatched: the recorder would attribute that delivery to the current
invocation and commit a head whose state never contained the invocation's
own action. The run-step evidence path never had this hole — it sees the
poll tool's name — so the recorder must match its provenance discipline.

Delivery callbacks now carry an explicit backgroundDelivery marker set at
the one site that rewrites the name, and the recorder ignores marked
deliveries outright. Regressions pin both halves of the contract: the
delivery callback must carry the marker with the poll call's arguments,
and a marked delivery can never qualify even a name-only expected action.

* 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions

Three closeout-round findings, each converted into an invariant.

Every legacy-path invalidation now advances a durable epoch — including for
headless and already cold-marked actors, where the marker alone leaves no
CAS-visible trace — and the actor-head CAS requires the epoch observed at
preparation. A concurrently prepared fork whose history predates an
intervening legacy turn can no longer commit past it; the commit reports an
ordinary conflict and journals.

Execution identity is now emitted before post-execution output policy: when
a side-effecting tool succeeds but its returned content is withheld by the
output filter, the callback delivers an outputFiltered receipt with blank
content — the recorder accepts it as proof (rejecting model-detached calls
it cannot distinguish through the blank shape), the artifact path never
sees it, and an applied action is no longer reclassified actionless and
re-executed on retry.

Background-capable expected actions stay off the fork path: dispatch
returns a launch handle every evidence fence correctly rejects, and the
completion is provenance-marked as another turn's work, so a fork would
settle actionless before the external effect lands with nothing to stop a
retry from dispatching it again. The gate mirrors the MCP-suffix name
matching of the evidence path.

* 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance

The epoch fenced only the legacy turn's start: a fork preparing after the
begin invalidation but before the turn's message persistence observed the
new epoch and cold marker, rebuilt from history that did not yet contain
the turn, and committed cleanly because nothing advanced the epoch again —
making the incomplete rebuild authoritative and clearing the marker.

Every legacy event turn now seals its invalidation at terminal persistence
with a second epoch advance, on the success, replaced-claim, and error
exits alike. Sealing deliberately carries no quiescence requirement — it
must succeed while a fork fence is active, because that is exactly the
mid-turn race it defeats — and a fork that already committed against the
begin epoch is healed the same way: the seal re-marks the head cold, so
the next event rebuilds with complete history. Seal failure never diverts
the turn's own exit; the begin bump still fences everything prepared
before the turn.

* 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence

The second epoch advance could not make a legacy turn atomic, and three
findings shared that root cause: two conditional updates left a headless
gap a fork could create the head inside; the error exit sealed before
saveErrorTurn made the error history durable; and any crash or failure
between persistence and sealing left an incomplete fork authoritative,
because the seal was best-effort and its failure was swallowed.

A legacy turn now carries one durable fence. A token is written before
execution by a single update-pipeline write — no two-write gap, and the
cold marker is applied only where a head exists via $cond/$$REMOVE. While
the token is present no fork may prepare (the adapter refuses) or commit
(the CAS requires its absence), because the turn's messages are not yet
durable. One atomic write clears the exact token and advances the epoch
once history is persisted — after saveErrorTurn on the error exit — and
success, replacement, and error exits all route through it.

Failure is now fail-closed rather than silent: a failed seal leaves the
token set, which keeps blocking forks and is logged as such, and a fence
abandoned by a crashed turn is reclaimed only once stale, advancing the
epoch and marking any head cold so the next event rebuilds from whatever
history actually survived.

* fix: serialize legacy event actor turns

* fix: close legacy actor fence ownership gaps

* fix: preserve legacy actor persistence fences
2026-08-26 08:39:00 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Marco Beretta
33e42e6d5d
🎛️ feat: Configurable SearXNG Search Options (#14987)
* feat: configurable SearXNG search options

SearXNG queries were hardcoded to google,bing,duckduckgo with no way to
change the engine list, the result language, or the request timeout. Most
self-hosted instances get served CAPTCHAs by DuckDuckGo, so a third of
every query silently returns nothing and operators have no lever to pull.

Add a searxngSearchOptions block to the webSearch config that accepts
engines (as a comma-separated string or a list), language, timeRange, and
timeout, and thread it through to the search tool. Engines are normalized
to the comma-separated form SearXNG expects, with blank entries dropped so
a stray comma cannot produce an empty engines parameter.

Refs #14117

* fix: normalize SearXNG engines on the runtime config path

The engines transform lived only on the zod schema, but loadCustomConfig
returns the raw YAML object rather than result.data, so nothing downstream
ever saw the transformed value. A YAML list reached the SDK as an array and
threw "options?.engines?.trim is not a function" when the search tool was
built, taking web search down entirely for the exact block the example yaml
documents. An untrimmed string reached SearXNG with spaces still in it.

Extract the normalization into normalizeSearxngEngines and apply it in
loadWebSearchConfig as well as the schema, so both the parsed and the raw
path produce the same comma-separated value. Widen the loader's parameter to
TWebSearchConfigInput, which models engines as the list or string an operator
actually writes, and cover the raw path with tests that call the loader rather
than the schema.

* chore: drop unused RerankerTypes import in web config loader
2026-08-21 11:36:23 -04:00
Marco Beretta
29f6ec6eae
🙈 feat: Config Option to Hide Response Feedback Buttons (#15085)
* feat: add interface option to hide response feedback buttons

Adds `interface.feedback` to librechat.yaml. When set to false, the
thumbs up/thumbs down buttons are removed from the message action row
and the feedback endpoint rejects writes with 403, so deployments that
do not consume the data can stop collecting it. Defaults to true.

* refactor: hoist the feedback gate out of the message row and into typed middleware

Reading startup config inside HoverButtons put a query observer and two Recoil
subscriptions on every message row, and rows never unmount, so the cost grew
with the conversation. Resolve the flag once per chat in useChatHelpers and
carry it on TMessageChatContext; useMessageActions withholds handleFeedback
when it is off, which the action row already treats as "no feedback controls".
The flag now stays false until the config resolves, so a disabled deployment
never flashes controls whose writes are rejected.

Move the server-side policy into requireFeedbackEnabled under packages/api so
the route keeps no policy of its own.

* test: stub the feedback gate in specs that replace the api package

The messages router now imports requireFeedbackEnabled, and express rejects an
undefined handler at require time, so every spec that mocks @librechat/api
wholesale has to carry the export.
2026-08-21 11:16:44 -04:00
Danny Avila
8c14f03432
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects

Adds an optional chat-project destination to a schedule, plus the operator
config to require one — or to pin every scheduled run to a specific project.

Feature:
- `chatProjectId` on the schedule row, accepted on create/update, projected on
  the wire, and carried into the run's conversation through the durable trigger
  envelope's `run` context.
- `interface.schedules.requireProject` refuses schedules that are not filed
  under a project; `interface.schedules.projectId` pins every run to one
  project and implies the requirement.
- Dialog gains a project picker (required when configured, a read-only row when
  pinned); the card shows the destination and the new disabled reasons.

Invariants:
- ONE resolver (`resolveScheduleProjectId`) decides the destination for the
  write handler, the fire path, and the wire projection alike, and an operator
  pin OUTRANKS the stored id in all three. Tightening the config therefore
  redirects — or stops — existing schedules instead of grandfathering where
  their runs land. A pin implies `requireProject` for the same reason: without
  it, a row created before the pin would keep firing with no project at all.
- Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write
  this handler accepts is one the next fire also accepts. Any edit leaving a
  schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like
  the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips
  the requirement, or a schedule auto-disabled for `project_required` could
  never be turned off.
- Fire-time enforcement auto-disables rather than filing runs loose, matching
  agent_deleted: new `project_required` (requirement raised after creation) and
  `project_deleted` (gone, or pinned to a project this owner does not have)
  reasons, both refused BEFORE a billed generation is dispatched, and both
  advancing so a schedule can never wedge on the occurrence.
- `computeCreateDigest` appends the field only when present, so a payload
  without a project digests byte-identically to one from before this change —
  a create retried across the upgrade still matches its own row instead of
  reading as key reuse.
- Project reads are scoped to the owner, so ownership and existence are the
  same lookup; a read ERROR propagates instead of failing closed, so a Mongo
  blip retries the fire rather than auto-disabling the schedule.

The trigger idempotency key hashes principal/event/target and never
`envelope.run`, so the added run field cannot destabilize delivery identity.

* 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget

The project picker landed as a new ROW in the schedule dialog, which broke the
e2e edit spec: `md:overflow-visible` turns off the template's scrolling from
`md` up, so the dialog's content must fit the viewport. The extra row pushed the
footer's Save button below a 720x1280 window, where Playwright reported a
visible, enabled button it could never click — 226 scroll-into-view retries and
a 2-minute timeout, on all three attempts.

Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720):
  dev              688   (32px slack)
  project row      ~790  (off-screen, CI failure)
  3-column row     704   (16px slack — half the budget spent)
  this commit      686   (34px slack, 2px better than dev)

The identity row is now three columns — name, agent, project — and its caption
moved out of the agent cell to sit full width beneath the row: at a third of the
dialog that sentence wraps an extra line, and the row is the tallest thing
competing for the budget. The caption is grouped with the row rather than left
to the form's own 4-unit rhythm, which spent more height on the gap than the
caption occupies.

The e2e spec now asserts the button is in the viewport before clicking it, so
the next field that overflows this dialog says so in one line instead of a
two-minute timeout on a visible element. FOLLOWUPS.md records what the planned
dialog controls (multi-day weekly, timezone, attachments) need first: give
`ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the
popovers into the dialog content, and let the form scroll again.

* 🧹 fix: Address Codex Review on Scheduled Chat Project Scope

Four P2 findings, all real.

Unreachable clearing path (ScheduleDialog). The picker only held live projects, so
`com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a
schedule had a project the owner could never take it away, leaving the server's
`chatProjectId: null` path reachable only by API. The picker now carries a real
"No project" option whenever a project is optional, and omits it when one is
required, where there is nothing valid to select.

Placeholder shown for a real project (ScheduleDialog). A stored or pinned project
outside the first loaded page had no name in the paged map, and the combobox
renders its placeholder for an empty display value — telling the owner a scoped
schedule had no project. That one project is now read by id, with the raw id as a
last resort: a poor label, but an honest one.

Project policy skipped at the resume boundary (service.ts). `claimScheduleResume`
re-applied the schedules gate, the revision fence, the kill switch and
SCHEDULES:USE, but not the project policy this PR added. Approving a paused run
whose project was deleted — or whose owner now sits under a requirement or a pin
it no longer satisfies — billed a continuation the very next scheduled fire would
refuse and auto-disable the schedule for. The effective-project resolution now
runs there too, refused before the lease and the capacity slot so a policy refusal
costs nothing and leaves no state to unwind. NOTE: agent access and balance are
still not rechecked on resume; that gap predates this PR and is left alone.

Per-card project derivation (ScheduleCard). Every card ran the projects hook and
rebuilt the full option array, name map, and one icon element per project, to use
a single name — O(schedules x projects) per render and per project-list refresh.
The hook is split: `useChatProjectNames` (map only, for the panel, which resolves
every card's name once and passes it down) and `useChatProjectPicker` (options and
pagination, for the dialog's one combobox). The panel skips the query entirely
until some schedule actually has a scope.

Tests: four at the resume boundary (verified failing without the gate) and four in
the dialog spec. The picker selection in one existing test now goes through the
search field — the popover's VIRTUALIZED renderer sizes its window from a scroll
height jsdom always reports as 0, so with three options it materialized only two.
Full schedules e2e re-run green against the rebuilt client.

* 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent

Second Codex round, four P2s. Three were consequences of the resume gate added in
the previous commit, which was half-built: it admitted where it should not and
stranded the run where it refused.

Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy`
branch. Both entry points consult that branch FIRST, and both already route its
refusal through abort-and-settle — so a policy stop now settles the occurrence
instead of answering a bare 409 while the job stays `requires_action`, the card
keeps reading "Needs approval", and every retry repeats the same 409 until expiry.
No change to resume.js: the existing branch does the work.

The rule is deliberately NARROW. It refuses only where no valid destination is
left — the requirement is on with nothing satisfying it, or the schedule's own
project is gone (which also unset it on the conversation). It does NOT refuse
because an operator's pin moved: the paused conversation cannot be rebound
(`chatProjectId` is excluded from the resume context and the continuation reuses
the same conversationId), so refusing would strand a pending approval over a
destination it can never reach, for a pin that governs only where the NEXT run
lands — which the fire path already redirects.

Create retries are idempotent again. Project policy had been applied BEFORE the
`clientRequestId` replay lookup, so a raised requirement, a deleted project, or a
moved pin could answer 400 for a create that already committed — pushing the
client to rotate its key and create a DUPLICATE schedule, the exact failure the
key exists to prevent. Policy now applies only to a genuinely new insert, and the
digest is computed from the CLIENT's payload rather than the resolved destination,
so today's policy can no longer re-digest a genuine retry into a mismatch.

An explicit `chatProjectId: null` under a pin is refused rather than silently
resolved to the pin. The payload contract defines `null` as clearing the scope;
answering 201 while filing under the pin reported success for the opposite of what
was asked. Only an OMITTED field takes the pin silently.

Tests: five on the policy branch (both refusals verified failing without it, plus
guards that a moved pin and a live project still admit) and two on the handlers
(the pinned explicit clear, and a committed create recovered by retry after the
policy tightened — verified answering 400 without the reordering).

* 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved

Third Codex round, four P2s.

The root confusion behind the resume findings: an operator pin outranks the stored
id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row
keeps its old value. The row therefore LIED about where that occurrence's
conversation went, and every later re-validation — the resume boundary above all —
checked a project the conversation was never filed under. A schedule storing A,
pinned to B, with B later deleted and A still live, was admitted for resume into a
conversation that had just been unscoped.

Fixed at the source rather than at each reader: a fire that resolves a destination
different from the stored one writes it back, claim-token fenced like every other
worker-side write and deliberately WITHOUT a configRevision bump — this is the
server reconciling itself to policy, not an owner edit, and a bump would fence an
in-flight occurrence off its own run. Written only AFTER the destination validates,
so an unusable pin never lands in the row, and best-effort: the envelope already
carries the right destination, so a failed write costs accuracy on a later recheck,
never the run itself. The wire projection already reported the pin, so this also
stops the row and the UI disagreeing.

An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit
path too. The pin check and the requirement are independent rules, and folding them
together let `{enabled: false, chatProjectId: null}` skip the pin check entirely,
unset the row, and answer with a wire projection still naming the pin. Only the
requirement is waived for a disabling edit.

The dialog no longer requires a project for an edit that leaves a schedule DISABLED.
The server waives the requirement there precisely so a row auto-disabled for
`project_required` can still be renamed or tidied up; requiring it in the form made
that unreachable, and an owner with no projects could not edit the stopped schedule
at all.

FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second
deletion race inside the resume claim window (which needs the effective project
persisted per OCCURRENCE plus a distinct policy conflict routed through
abort-and-settle), and the fact that convergence happens only when a schedule fires.

Tests: four on convergence (pin written, no write when unchanged, no write for a
destination that failed validation, fire survives a failed write), two on the
disabling-edit pin rules, two in the dialog. Schedules e2e re-run green.

* 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest

`computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and
an explicit `null` produced the same digest. Because the replay lookup and
`matchesCreateIntent` deliberately run before project policy, a request could reuse a
pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and
receive 201 describing the pinned row — success reported for the opposite of what it
asked, and the pinned-clear refusal the normal create path applies never reached.

Now `!== undefined`: an omitted field still digests byte-identically to a payload from
before project scope existed, so a create in flight across the upgrade still matches
its own row, while an explicit clear is a distinct intent and digests differently. A
pre-scope client never sent the field at all, so nothing legacy can carry an explicit
null.

* 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used

The schedule-wide convergence from the previous commit was not enough, and the
reason is the single-active run index: it covers `status: 'started'` only, so a
PAUSED run does not block the next occurrence. While run 1 sat paused in project A,
a pin move plus one later fire rewrote the schedule row to B — and the resume
policy then validated B while run 1's conversation was still filed under A. Delete
A and the continuation was admitted into a conversation that had just been unscoped.
That window lasts as long as the pause, not the sub-second race the previous commit
documented.

The reservation now records the destination THIS occurrence used, and
`isScheduleLive` validates that record when given the occurrence's `scheduledFor` —
which `resume.js` already reads two lines above the call. No new conflict type and
no settlement-path surgery: the refusal rides the abort-and-settle branch that check
already has.

An ABSENT record falls back to the schedule-level resolution rather than reading as
unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as
evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as
they do today.

Schedule-wide convergence stays: it keeps the row honest for the UI and for every
check that has no occurrence in hand.

FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the
claim window, which needs a distinct policy conflict routed through abort-and-settle
rather than the bare 409 an `inactive` conflict produces.

Tests: three on occurrence-vs-row precedence (the decisive one verified failing
without the lookup), two on what the reservation records, and the resume controller
spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green.

* 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One

The occurrence fallback added in the previous commit conflated two different
absences. A post-upgrade run that deliberately went unscoped omitted the field
exactly like a row written before the field existed, so both took the fallback —
and a paused unscoped run was then validated against the schedule's CURRENT
project. Under a requirement or a pin added while it sat paused, that admitted a
billed continuation into a conversation satisfying no present policy.

The reservation now ALWAYS records its decision, `null` for unscoped, and the read
reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record
— a pre-scope row, or no row at all — falls back to the schedule-level resolution;
a recorded null is the genuinely unscoped occurrence it says it is, and is refused
once a project becomes required.

The distinction rests entirely on a stored `null` surviving as a present key while a
never-written field stays absent, so that is asserted against real Mongo rather than
assumed: if it ever stopped holding, unscoped runs would silently start being
validated against the schedule's current project again.

Tests: two against mongodb-memory-server (recorded null vs never-written vs missing
row), plus refusal of a recorded-unscoped occurrence under a new requirement and the
preserved fallback for a pre-scope one. Schedules e2e green.

* 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence

The initial-start policy check in `request.js` called `isScheduleLive(..., { policy:
true })` without `scheduledFor`, so it fell back to the schedule-level resolution
even though the run row already carries the occurrence's recorded scope. An
occurrence reserved unscoped, with a pin introduced while its loopback request sat
queued, was therefore admitted against the new pin — producing a billed unscoped
conversation under a requirement it does not satisfy, from an envelope already built
without a project.

`scheduledFor` was already in scope there. Passing it makes the initial start and the
resume validate the same way: against the destination the occurrence itself recorded.
2026-08-21 11:09:04 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Marco Beretta
7569404a7c
🎛️ feat: Make Max Subagents Configurable via librechat.yaml (#15023)
* feat: make max subagents configurable via endpoints.agents.maxSubagents

The per-agent subagent cap was hardcoded at 10 in MAX_SUBAGENTS, leaving
orchestration-heavy deployments no option but patching limits.ts and
rebuilding. Add an optional endpoints.agents.maxSubagents key to
librechat.yaml (default 10, hard ceiling 50) that drives request
validation, model spec presets, and the agents panel UI cap.

* style: fix import order in OrchestrationHub
2026-08-20 11:16:08 -04:00
Danny Avila
a47ba7168f
🪢 feat: Custom Request Headers For Langfuse (#14945)
* ✨ feat: Custom Request Headers For Langfuse

Self-hosted Langfuse behind an authenticating proxy or gateway could not
be reached: every outbound Langfuse request hardcoded `Authorization` and
nothing else. Adds `langfuse.headers`, mirroring `endpoints.custom`
headers, and applies it to all four request surfaces — trace/media export
(via the agents run config), feedback scores, central project-identity
lookup, and admin credential verification.

Values resolve through the same pipeline as endpoint headers, so
`${ENV_VAR}` interpolation and header-safe encoding come along.
`extractEnvVariable` continues to refuse infrastructure secrets, so a
config cannot exfiltrate `MONGO_URI` through a header. A header whose
variable is unset is dropped with a one-time warning rather than sent as
a literal `${...}`, which a gateway would read as a wrong credential
instead of a missing one.

Headers merge beneath LibreChat's own `Authorization` on the REST
surfaces, matching `mergeHeaders`, so a custom header can never displace
the Langfuse credential.

These are deployment-level and documented as such: trace export batches
spans from every user through a single exporter, so unlike endpoint
headers they cannot carry per-user placeholders.

The central project-id cache key now includes the headers, so the
header-less module warm-up cannot record a proxy rejection against the
entry the request path later reads.

* 🔒 fix: Keep Langfuse Headers Out Of Stored Overrides

The generic admin config API accepts any field path inside an allowed
section, so `langfuse.headers` could be written through it. Unlike
`langfuse.secretKey`, headers are a map rather than one scalar path, so
the config secret registry cannot encrypt them at rest or mask them on
read — an admin-written map would sit in Mongo in plaintext and come
back in plaintext, widening exposure of what are gateway credentials.

Rejects them on both the dotted-patch and object-upsert routes, the same
way process-backed MCP servers are held to librechat.yaml. This is what
makes "deployment-level" true rather than merely documented.

* 🐛 fix: Wire Config Middleware And Header Collisions For Langfuse

Two codex review findings.

P1 — `api/server/routes/admin/langfuse.js` never mounted
`configMiddleware`, so `req.config` was undefined in production and
credential verification silently ran without the deployment's proxy
headers: exactly the deployments this feature targets could not save a
connection. The handler unit tests injected `config` into their mock
requests, so they stayed green. Mounts the middleware after the access
checks (unauthorized callers still short-circuit first) and adds
route-level tests that assert the handler actually receives a resolved
config — the composition root, not the component.

P2 — spreading custom headers under `Authorization` only replaced an
exact-case collision. A configured `authorization` survived alongside
the managed `Authorization` and fetch appends rather than replaces,
sending both credentials in one combined value. All four request sites
now use `mergeHeaders`, which already merges case-insensitively with the
override winning; tests cover the lower- and upper-case variants.

* 🔒 fix: Mask Langfuse Headers On Read And Harden Value Handling

Three codex round-2 findings.

P1 — the write guard blocked storing `langfuse.headers` in Mongo but did
nothing for the read path: `GET /api/admin/config/base` serves the
resolved AppConfig through `redactConfigSecrets`, which only knows
registered scalar secrets, so a yaml-configured gateway credential was
returned in full to any admin with Langfuse read access. Adds a
secret-map registry that masks values while keeping key names, so an
admin can still see which headers a deployment sets. Masking is safe
precisely because these are yaml-only — a masked read cannot be
round-tripped back over the real values. A malformed non-object value at
that path is dropped rather than serialized.

P2 — `mergeHeaders` indexes one spelling per lowercase name, so a config
holding both `authorization` and `AUTHORIZATION` had only one displaced;
the survivor was then appended by `Headers` into a combined value. Case
variants are now collapsed at resolution, before any consumer sees them.

P2 — `resolveHeaders` encodes only values it substitutes a user field
into, and no user is supplied here, so a literal or interpolated
character above U+00FF reached `Headers` unencoded and threw. Final
values now go through `encodeHeaderValue`; Latin-1 still passes verbatim.

* 🔒 fix: Keep Langfuse Header Credentials Out Of Logs And Validate Names

Three codex round-3 findings, plus a documented boundary for the fourth.

P1 — `loadCustomConfig` logs the parsed config at startup (`printConfig`
defaults true), so a literal gateway credential in `langfuse.headers` was
copied into application logs on every boot, undoing the masking the admin
read path had just gained. The printed copy now goes through
`redactConfigSecretMaps`, reusing the same registry. Scoped to map-valued
secrets so scalar-secret log behavior is unchanged; the live config keeps
its real values.

P2 — a nonempty but invalid field name (` X-Token`, `X Proxy Token`)
passed the emptiness check and then threw in the `Headers` constructor,
which would break export, verification, lookup, and feedback for the whole
deployment rather than that one header. Names are trimmed and validated
against the RFC 7230 token grammar, and dropped with a warning otherwise.

P2 — unresolved `${VAR}` detection tested the *resolved* value, so a
credential legitimately containing `${...}` was mistaken for a failed
substitution and dropped. Detection now inspects the configured text and
checks the referenced variables directly, which also drops references to
denylisted infrastructure secrets instead of forwarding them verbatim.

The fourth (fanout gateway forwards only `Authorization`, so a tenant
Langfuse behind its own proxy is not covered) is a real limitation in a
separate component. Documented on the schema field and in the example
config rather than left implied.

* 🔒 fix: Scope Langfuse Headers To Configured Origins

Three codex round-4 findings.

P1 — one header map was attached to every destination a run resolves to.
Under fanout that means a credential meant for an internal gateway was
also sent to the central destination, typically Langfuse Cloud: an
unrelated third-party origin. Headers are now attached only when the
destination's origin is one the deployment explicitly configured (a
self-hosted base URL, the fanout collector, or a tenant destination set
by env). The built-in `*.cloud.langfuse.com` defaults are excluded
precisely because nobody pointed at them. For trace export this also
means attaching after the export branch settles on a `baseUrl` rather
than before, since which destination wins depends on the branch.

P2 — `encodeHeaderValue` only encodes above U+00FF, so a newline, CR, or
NUL passed through and threw in `Headers`, breaking every request rather
than the one header. Values are trimmed (the common trailing-newline
case) then validated against the legal field-value bytes; an embedded
CRLF is a request-splitting attempt and is dropped, not stripped.

P2 — the write guard matched only the exact `headers` property, so
`{ langfuse: { "headers.X-Token": "..." } }` and root-level dotted
variants slipped through into the Mixed overrides document, where the
nested-map redactor never walks them and a later read returns them in
plaintext. All dotted spellings are now rejected.

* 🔒 fix: Bind Langfuse Headers To One Configured Origin

Four codex round-5 findings.

P1 — the round-4 allowlist still authorized every configured origin, so a
deployment with both a collector and an explicit central host sent the
same credential to both. `langfuse.headers` is one map with no way to say
which endpoint it authenticates to, so it is only unambiguous when the
deployment configures exactly one Langfuse origin. Iterating on which
origins to guess was the wrong axis; headers are now sent only when there
is a single configured origin and the destination is it, with a warning
when several make the intent unresolvable. That covers the self-hosted
case this feature exists for; multi-destination deployments need
per-destination headers the schema cannot yet express.

P1 — `fetch` defaults to following redirects, and Node strips
`Authorization` across origins but keeps arbitrary headers, so a redirect
off an allowed origin would hand the gateway credential to a host that
passed no check. Requests carrying custom headers now refuse redirects;
requests without them keep the default, so nothing changes for existing
deployments.

P2 — `extractEnvVariable`'s whole-string branch is anchored and greedy, so
`${CLIENT_ID}:${CLIENT_SECRET}` parsed as one variable name and the raw
template was sent as the credential. References are expanded here now, so
only literal values reach that path.

P2 — a valid token name is not necessarily usable: `Transfer-Encoding`
makes `fetch` throw and a fixed `Content-Length` misdescribes the body of
every other request sharing the map. Request-framing names are dropped.

* 🐛 fix: Expand Langfuse Header References Exactly Once

Codex round 6 (P2). After expanding `${VAR}` references myself I still
handed the result to `resolveHeaders`, which runs `extractEnvVariable`
over it again — so a credential containing `${PATH}`, or any other name
that happens to be set, was silently rewritten on export, verification,
lookup, and feedback. The round-3 test only used an *unset* embedded
name, which the second pass leaves alone, so it could not catch this.

Resolution no longer round-trips through `resolveHeaders`. The only part
still wanted from it was stripping `{{...}}` user placeholders, which is
now applied directly; expansion, encoding, and validation were already
local. Adds a test whose embedded variable is set, which fails against
the previous pipeline.

* 🐛 fix: Process Langfuse Header Templates Before Substitution

Codex round 7 (P2), the mirror of round 6. Having stopped re-expanding
the resolved credential, the placeholder strip was still running over it:
a token containing `{{LIBRECHAT_USER_ID}}` had that span deleted and
`abc{{...}}ghi` went out as `abcghi`.

Establishes the invariant the last two rounds were circling. Every
template operation — placeholder strip, unresolved-reference check,
expansion — now runs on the operator's configured text, and the
credential is substituted last and never touched again. Gateway
credentials are arbitrary strings, so none of their bytes are syntax.

Also moves the unresolved-reference check after the strip, so it no
longer reports a variable inside a `{{...}}` span that the strip removes.
2026-08-17 20:39:23 -04:00
Danny Avila
57ea1137f6
🛡️ feat: Let Admins Restrict Stateful Workspace Scopes (#14910)
* feat: let admins restrict stateful workspace scopes

* fix: enforce stateful scope policy across agent paths

* fix: close stateful scope policy activation gaps
2026-08-17 01:29:19 -04:00
Danny Avila
7d850c308a
🧠 feat: Add Live Reasoning Labels (#14893)
* feat: add live reasoning labels

* fix: Stabilize reasoning label checks

* fix: Address reasoning label review findings

* chore: Bump Agents SDK for reasoning labels

* fix: Reset reused reasoning step evidence

* fix: Reconcile cleared reasoning labels

* fix: Fence reasoning label resets

* fix: Reset reasoning ownership before gap labels

* fix: Preserve THINK type through label reset

* test: Expect run-global reasoning revision
2026-08-16 18:11:55 -04:00
Marco Beretta
3bd2358805
🗜️ feat: Let Users Toggle Client-Side Image Resizing (#14883)
* feat: allow users to toggle client-side image resizing

Client-side image resizing could only be configured in librechat.yaml and
defaulted to off, so users had no way to enable it for themselves.

Add a "Resize images before upload" toggle in Settings > Chat > Sending,
stored per device in localStorage. When librechat.yaml sets
clientImageResize.enabled, mergeFileConfig marks the value as enforced and
the toggle renders the admin value read-only. Admin resize parameters still
apply without locking the toggle when enabled is omitted.

Also fix shouldResizeImage, which compared file size against 10% of a 512MB
fallback limit and so only triggered above roughly 51MB. It now uses a 512KB
floor, which lets the setting affect everyday photos.

* fix: harden client image resizing

* fix(client): restrict image resizing and localize toast

* fix: harden client image resizing

* fix(client): recognize static WebP image chunks

* fix(client): clamp resized image dimensions

* fix(client): enforce safe image resize uploads

* fix(client): recheck duplicates after transforming uploads

* fix(client): keep selected files after input reset

* fix(client): disable image resizing when file config fails

* fix(client): decode resize candidates without a base64 copy

* fix(client): coordinate upload batches across hook instances

* test(client): drop the untyped conversation from shared upload state

* fix(client): start the config wait before queueing uploads

* fix(client): disable the resize switch while file config is pending

The switch stayed clickable during the initial file-config load even
though the checked state cannot update until that query settles.

* fix(client): only track upload reservations for observable state
2026-08-16 18:04:03 -04:00