Commit graph

78 commits

Author SHA1 Message Date
Ravi Kumar L
44638c06d2
🔑 fix: Drop Legacy Langfuse Ingestion Check from Connection Test (#17581) 2026-10-29 15:56:09 +01:00
Danny Avila
aa8e5849e0
⬆️ feat: Migrate to Zod v4 (#17442)
* ⬆️ feat: Migrate to Zod v4 (GHSA-mrhx-c7xc-m3w5), openai 6.49, zod-openapi 6

* 🔒 chore: Regenerate bun.lock

* 🐛 fix: Keep omitted fields absent under Zod 4 partial/optional defaults

* fix: Preserve Config Record-Key Diagnostics After Zod 4 Migration

* fix: Require Agents SDK 4.1.1 for Zod 4 Accounting

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-26 14:16:22 +01:00
Danny Avila
080e5f500c
🏘️ fix: Resolve Tenant Profiles in Their Own Partition and Keep Role Operations in the Caller's (#17417)
Some checks failed
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Vote Gate / Gate (push) Waiting to run
Codegraph E2E Votes / vote (shard 1/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
- GET /resolved resolves a group or user profile in the partition its document belongs to, as
  writes judge it: for an admin without a tenant reading a tenant's profile, that tenant's
  librechat.yaml and __base__. GET /catalog resolves such a scope there too, instead of
  answering 404, and GET /effective finds a user by id the way profiles are addressed and
  resolves them in their own tenant: their role, groups and documents there.
- findPartitionUser becomes findPrincipalUser, which also returns the user's tenant.
- Admin role operations address the role of the caller's tenant partition: the role lookup
  (getPartitionRole), listing, creation's name check, the members a rename migrates or a deletion
  reassigns, the members listed and counted, and the ACL entries (deletePrincipalAclEntries) and
  grants a deletion removes. A member added or removed by id must be a user of that partition.
- getRoleByName, which every request's permission checks read, reads the role of the caller's
  partition too: outside a tenant and under the system context the platform-level one, never an
  older tenant role of the same name. updateRoleByName without an id and updateAccessPermissions
  for a role read without its _id write that role, so the legacy permission route changes the
  platform role. The grants route checks a role exists with getPartitionRole.
- runInPartition moves a read into another partition only from outside a tenant, and rejects
  leaving an active one.
- deleteGrantsForPrincipal and findGrantsForPrincipal without a tenantId reach only
  platform-level grants, as moveGrantsToPrincipal does.
- updateRoleByName takes { id } to change one role document; renames, their rollback,
  description updates and updateAccessPermissions write the document they read.
- The exported keyed rename helpers (applyConfigChange rename, renameConfigEntryTombstones)
  refuse a new name no keyed path can address, as the admin routes do.
2026-10-24 18:33:12 +02:00
Atef Bellaaj
d481b6b695
🏬 feat: Sync Langfuse Prompts into the Catalog (#17095)
* feat(prompts): sync Langfuse prompts into the catalog

* refactor(prompts): compose the Langfuse prompt sync runner outside the admin handler

Add createLangfusePromptSyncRunner, which binds the lock, the prompt-group
methods, the ACL grant and the Mongo error predicates once and returns a
runner that takes the per-request source, connection-bound listPrompts,
lock lease and page cap. The admin handler receives that runner as one
dependency instead of forwarding its parts, and its config method
signatures return a LangfuseAdminConfig interface in place of Mongoose
document types.

In api, api/cache/distributedLock.js creates the one DistributedLock for
the process and api/server/services/LangfusePromptSync.js composes the
runner once with it; the admin route imports the composed runner.

* fix(prompts): report incomplete Langfuse syncs and gate the sync action

- Refresh the prompt-group and effective-permission queries after a failed
  sync too, because earlier imports can already be committed.
- Return `complete: false` when the page limit stops a sync before the last
  upstream page, and show a warning instead of the success message.
- Show the Sync prompts action only to users with `manage:prompts`, the
  capability the sync route requires. Resolve the Langfuse startup-config
  capabilities in one batched lookup.
- Document that an interrupted import can leave a group without its initial
  PUBLIC grant, and that a page in progress can outlast an expired lease.

---------

Co-authored-by: Rebase Probe <probe@example.com>
2026-10-24 13:48:52 +02:00
Atef Bellaaj
02613ce212
⛲ feat: Resolve Native and Langfuse Prompts by Source (#16909)
The prompt service dispatches resolvePrompt by group.source. A read-only
Langfuse adapter fetches the text prompt on each use, with no cache, after
a resolver checks the deployment gate, the tenant, prompt sync, the
connection and the stored source identity. GET
/api/prompts/groups/:groupId/resolve returns resolved content for native
and Langfuse groups, with a per-user limit set by rateLimits.promptResolve
(default 60 per minute). Agent-instruction links stay native only.
2026-10-24 13:47:52 +02:00
Atef Bellaaj
202756b37f
🧐 feat: Preview Langfuse Prompts in Settings (#16854)
* feat(langfuse): test prompt listing and resolution from settings

Add a dry-run dialog in Settings > Langfuse that lists Langfuse prompts
and resolves one text prompt (production label or an exact version)
through the tenant's Langfuse connection. Nothing is written. A tenant
user's prompt reads never use the central env keys, even while
TENANT_ISOLATION_STRICT is off.

DevOps makes the feature available with LANGFUSE_PROMPT_SYNC_AVAILABLE
(timeout LANGFUSE_PROMPT_SYNC_TIMEOUT_MS). The tenant admin turns it on
with langfuse.promptSync.enabled.

Related: Linear AI-2271, ClickHouse/ai#4032

* fix(langfuse): isolate prompt credentials and expose prompt settings

Use stored credentials with deployment headers for prompt reads.
Show the settings tab when connection editing or prompt sync is available.
Keep connection controls and trace links under their existing gates.
Audit prompt-sync updates and reuse shared prompt response types.
2026-10-24 13:46:40 +02:00
Danny Avila
03eb7be12e
🛡️ fix: Keep Azure Keys Off Unserved Groups, Let Hides Take Their Specs, and Report Inert Azure Groups (#17411)
Some checks are pending
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Vote Gate / Gate (push) Waiting to run
Codegraph E2E Votes / vote (shard 1/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
- A group a layer names that the config below does not serve (one a lower layer's tombstone of
  a field inside it, or of its groupMap entry, left unable to resolve) gets no key from the
  YAML-shaped group: a higher layer that re-supplies the field and a new instance or base URL
  no longer sends librechat.yaml's key there.
- /changes and PATCH report unknown_spec_endpoint only where the write touches the spec, so
  hiding, removing or unsetting the endpoint a profile's stored spec selects is no longer
  refused; the merge leaves the spec out, as the single-field routes already allowed.
- Without CONFIG_REDERIVE_ENDPOINTS, a layer's Azure groups and its tombstones of them (whole,
  keyed or inside a group) are listed as endpoint_rederive_off in each resolution layer, in GET
  ignored and in the /changes report. With it, a keyed tombstone that leaves its group unable to
  resolve is listed as azure_groups_unresolved. Keyed tombstones are located by entry key.
- The startup report lists documents whose only Azure override is a keyed group tombstone.
- Docs: hides never refused for the specs they take, the new ignored reasons.
2026-10-24 13:08:47 +02:00
Danny Avila
9f617fc4ad
🧬 feat: Re-Derive Azure Groups, Vertex AI, Custom Parameters and Spec Defaults for Config Profiles (#17399)
* 🧬 feat: Re-Derive Azure Groups, Vertex AI, Custom Parameters and Spec Defaults for Config Profiles

AppService derives part of the served config from librechat.yaml at startup, and config
profiles merged into that output without deriving it again. Azure `groups` now merge by `group`
like `endpoints.custom` by `name`: a profile's partial group merges over the group of that
name, groups it does not name stay, indexed writes are refused, change batches take entry ops
on them, and groups that do not resolve over librechat.yaml and Base (or whose written `${VAR}`
is not set) answer `invalid_azure_groups`. Each merged group keeps the key below it unless the
profile supplies one, so the rebinding guard drops a key a partial item moves and a key a lower
layer dropped stays dropped; Azure re-entry compares groups merged the same way.

With `CONFIG_REDERIVE_ENDPOINTS=true` (opt-in for this release), a profile's Azure `groups`
resolve into what the Azure clients read (unconditional before, now behind the switch so stored
group overrides do not go live on upgrade), a profile's `endpoints.anthropic.vertex` derives
`vertexConfig` and the Anthropic models it serves where librechat.yaml configures Vertex AI, and
the Azure Assistants endpoint follows a profile's `assistants`. Startup logs the documents that
hold such overrides, the groups that would not resolve over librechat.yaml and Base, and the
Vertex AI overrides that never apply. Always on: Vertex AI's key file and region are base-only,
a profile may turn Vertex AI on only where librechat.yaml configures it, its settings may
reference only librechat.yaml's own Vertex AI variables (reads return them resolved), a
profile's `vertexConfig` is never applied and writes skip it (`derived_path`,
`rules.derivedPaths`), a model's own `deploymentName` resolves its `${VAR}` like the default one,
the region, project ID and the deployment name each model's requests send (the model name where
the client sends it as written) must stay in their URL host label or path segment, a `${VAR}`
there is a whole `${NAME}` reference only, the Anthropic client refuses any other deployment name
before building a request, and an invalid setup answers `invalid_vertex_config`. The rebinding
guard no longer binds the derived Vertex AI region, which is librechat.yaml's.

Custom endpoint `customParams` are expanded by `expandCustomEndpoint` into the served copy, so
`appConfig.config` stays librechat.yaml as written, and each endpoint a profile writes is
expanded again from its merged form (a changed `defaultParamsEndpoint`, OpenRouter defaults
for a profile's own endpoint); invalid definitions answer `invalid_param_definitions`. After
tenant scoping, `getAppConfig` derives `modelSelect`, `parameters` and `presets` again for the
specs a principal is served where nothing sets them, the agents `memory` capability follows
the merged memory config, and a model spec naming an unconfigured endpoint answers
`unknown_spec_endpoint`. Meta flags: `azureGroupMergeKey`, `customParamsExpansion`,
`specInterfaceRederive` and `endpointRederive`.

* 🪜 fix: Judge Re-Derived Config on What Every Profile Layer Leaves

Two re-derivations decided whether a profile had set a field from whether any layer ever set
it, so a lower profile's value tombstoned by a higher one still counted as set.

- Vertex AI (CONFIG_REDERIVE_ENDPOINTS=true): the marker that keeps a profile's
  `endpoints.anthropic.models` over Vertex AI's is cleared by a later tombstone of those
  models (or of `endpoints.anthropic` or `endpoints`), and such a tombstone derives Vertex AI
  again, so the Anthropic endpoint serves Vertex AI's models instead of none. A tombstone of
  librechat.yaml's own models does the same.
- Spec-dependent controls: `modelSelect`, `parameters` and `presets` are derived again where
  their served value differs from the base config's, not only where the librechat.yaml-shaped
  value does, so a control one profile sets and a higher one tombstones, or a tombstone of one
  librechat.yaml leaves unset, follows the specs served instead of being dropped.

The memory capability, Azure Assistants, Azure groups and custom parameters already derive
from the merged result; tests now cover each set in one layer and tombstoned in a higher one,
and the reverse.

* 🧮 fix: Derive Each Config Part From What the Layers Leave

A tombstone of an object above a derived part's inputs, alone or followed by a layer that
recreates part of that object, left the part without what startup derives it to:

- An Azure endpoint recreated after a tombstone of `endpoints.azureOpenAI` or `endpoints` had
  no groupMap, modelGroupMap or model lists, so its model lists fell back to the environment's
  default Azure models, which requests could not map to a group. It now serves no group, with or
  without CONFIG_REDERIVE_ENDPOINTS (completeAzureResolution).
- A tombstone of the whole `interface` left modelSelect, parameters and presets out, which the
  client reads as off. They are derived from the specs served into a new interface.
- A tombstone of `endpoints.agents` or `endpoints` left no agents endpoint, while a tombstone of
  its capabilities served the defaults. The agents endpoint startup derives is served there.
- A tombstone of a Vertex AI setting that leaves the setup unable to load kept the vertexConfig
  derived below it. Vertex AI is off there, with a warning.

A table in service.spec.ts runs every derived part against every way a layer changes its inputs
(set, unset, a tombstone of the path and of each object above it, a layer recreating after such
a tombstone, Base on either side, tenant scoping, the switch off) through getAppConfig, and
compares each cell with what startup derives from the librechat.yaml the layers leave.

* 🧭 fix: Serve Only Model Specs Whose Endpoint the Layers Leave

A tombstone of `endpoints.custom` or `endpoints` in a higher profile (or the Base profile)
left the model specs of the custom endpoints it removes served, librechat.yaml's or a lower
profile's: the spec list pointed at endpoints /api/endpoints no longer offers, with the model
selector, parameters and presets still off for them. The write check only ran when the write
held `modelSpecs.list`, and the merge served stored specs as they were.

After the last layer, the merge now serves only the specs whose endpoint is a built-in endpoint
or a custom endpoint the YAML-shaped config the layers leave defines, the rule startup skips
librechat.yaml's specs by (deriveServedSpecs). A removed endpoint hides its specs until a layer
defines it again; where no spec is left, none is served (as with a tenant-hidden endpoint) and
the spec-dependent controls come back on. Tombstones of a built-in endpoint's settings keep its
specs, as startup does, and `addedEndpoints` stays as merged. A stored spec for an endpoint that
is not configured is left out on its own instead of being served; writes of one still answer
400 unknown_spec_endpoint.

The derived-part table in service.spec.ts gains the model specs served (with addedEndpoints and
the controls they decide) against tombstones of `endpoints.custom`, `endpoints`, and each of
`endpoints.openAI`, `azureOpenAI` and `anthropic`: by a profile, by Base under a profile, a lower
profile under a higher one, recreated above or by the same layer, with no spec left, and with
the switch off.

* 🧩 fix: Reconcile Re-Derived Config With Dev's Keyed Tombstones, Write Report and Read APIs

- Keyed tombstones of Azure groups (endpoints.azureOpenAI.groups["g"], now merged by key)
  resolve the groups left again where the merge resolves them, and stay inert without
  CONFIG_REDERIVE_ENDPOINTS.
- Keyed tombstones count as touching endpoints.custom, so served specs and custom parameters
  are derived again after one.
- The served-spec derivation keeps modelSpecs while added endpoints remain, as the hidden
  endpoint scoping does; hidden endpoint names leave addedEndpoints.
- Derived paths join the held paths every write skips, built on first use.
- Tests follow the merged behaviour: Azure groups merge by group, the agents endpoint is
  derived again below a tombstone.
2026-10-24 11:46:01 +02:00
Danny Avila
0716094734
🏷️ feat: Hide and Rename Custom Endpoints by Name in Config Profiles (#17396)
* 🏷️ feat: Hide and Rename Custom Endpoints by Name in Config Profiles

Config profiles merge `endpoints.custom` over librechat.yaml entry by entry on `name`, but
could not take one entry away. A tombstone that named one, such as `endpoints.custom["Name"]`,
passed every check, was stored with a 200 and did nothing, because the merge removes object
paths only and stops at arrays; a tombstone at a position (`endpoints.custom.0.headers`) was
stored and did nothing either. Renaming a profile's own endpoint through PUT or PATCH lost its
stored key, since the `renames` hint covered MCP servers only.

POST `/fields/tombstone`, DELETE `/fields` and the `tombstone` and `unset` ops of a change
batch now take a keyed path, `endpoints.custom["<name>"]` or
`endpoints.custom["<name>"].<path>`, whose name is a JSON string compared as the merge folds
it. A tombstone hides the inherited endpoint, or one of its fields or record keys, for the
profile's principals and removes the profile's own entry or path; hiding an endpoint also hides
the model specs that select it, whether librechat.yaml or a lower profile defines it. An unset
resets the profile's own entry or path with the keyed tombstones at or below it, and writing an
entry clears the tombstone that hides it whole. Malformed keyed paths, keyed paths after arrays
not merged by key, the entry's name, its key, preview or destination fingerprint, keyed writes
and positional tombstones answer 400, and the store's keyed mutations refuse a location whose
stored tombstone would name another path. A `segments` target names keys as written on every
route, so a segment such as `custom["Name"]` never reaches an entry past these guards: a write
by it stores a key of that spelling and clears none of the entry's tombstones. A stored
tombstone reads as keyed only after an array merged by key, so an older one such as
`mcpServers.a["b"]` keeps hiding that server.

PUT and PATCH `renames` also take `{ path: 'endpoints.custom', previousName, name }`: the
renamed entry gets the stored ciphertext and preview, and the keyed tombstones of the old name
move to the new one, as they do for a batch `rename`. The carried key stays bound to its
destination (400 `CONFIG_SECRET_REENTRY_REQUIRED` when the renamed entry sends it elsewhere)
and keeps the destination fingerprint it was entered with; a hint that cannot apply answers 400
`CONFIG_RENAME_INVALID` with a reason. Removing the destination of a profile's own keyed entry
beside its stored key needs the key again, and removing an inherited one drops the inherited
key at merge and lists it in `ignored`.

The merge-key registry moves to its own module so mutations can match entries by identity
without an import cycle. Meta reports `keyedTombstones` and `renameHints`,
`rules.renameHintPaths` and `limits.maxEntryKeyLength`, and `docs/endpoint-overrides.md`
documents the contract.

* 🙈 fix: Scope Hidden Endpoint Allow-Lists and Refuse Duplicate Rename Sources

- A hidden custom endpoint leaves endpoints.agents.allowedProviders as it
  leaves modelSpecs; scopeHiddenEndpointSpecs becomes
  scopeHiddenEndpointReferences.
- modelSpecs collapses only when no spec and no added endpoint remain, so
  unrelated addedEndpoints survive when the hidden endpoint owned every spec
  (tenant scoping shares the helper).
- An addedEndpoints or allowedProviders list that names only hidden endpoints
  is kept whole: an empty list allows every endpoint.
- Two endpoints.custom rename hints from one previousName answer
  CONFIG_RENAME_INVALID duplicate_source; MCP hints may still duplicate a
  server.

* 🔀 fix: Keep Keyed Tombstones Whole Across Dev's Patch Steps, Write Report and Resolved API

- PATCH resets (dev's request-ordered steps) still move a renamed entry's
  keyed tombstones, in validation and in the stored state; lifted
  tombstones come from one data-schemas helper for both paths.
- The write report of POST /fields/tombstone and DELETE /fields names a
  keyed location by its keyed path, collection, key and entryPath, as a
  change ref does, instead of the bare collection.
- resolveConfigLayers scopes the model specs and allow-lists of hidden
  endpoints as getAppConfig does, so the resolved API keeps its parity.
- GET /meta reports maxEntryKeyLength beside dev's catalogModelsTtlMs.
2026-10-24 10:44:57 +02:00
Danny Avila
babf6f908b
🧾 feat: Admin Config Write Report, Structured Issues and Deployment-Only Paths (#17395)
* 🧾 feat: Admin Config Write Report, Structured Issues and Deployment-Only Paths

Admin config writes dropped what the API never stores (base-only sections, interface
permissions, dedicated sections) with a 200 and no word of it, stored values nothing reads
(boot-time and operator-only settings, overrides of MCP servers librechat.yaml runs as local
processes, tombstones inside arrays), answered most 4xx without a stable code, returned
validation issues without the bound or option that failed, and refused every admin write
once the terms-acceptance modal was on for an admin who had not accepted it, including the
write that turns it off.

PUT, PATCH /fields, POST /fields/tombstone and DELETE /fields now answer with `applied` and
`skipped` (with a reason per location); a whole `interface` PATCH keeps its UI fields;
base-only paths are skipped on every route. A new DEPLOYMENT_ONLY_CONFIG_PATHS list (rate
limits, CloudFront, permission leases, OpenAPI, skill deletion, the tool filters, the agents
and MCP boot settings, `cache`) is skipped on write, kept at librechat.yaml's value by the merge
and published in GET /meta. Writes and tombstones inside a librechat.yaml stdio MCP server
answer 400 PROCESS_MCP_SERVER_OVERRIDE, and a whole-section value leaves those servers out.
GET of a document lists what the merge leaves out of it (`ignored`, `ignoredBasis`).
Validation issues carry allowlisted `params`; each refused allowedAddresses entry is one issue
with a `reason`; agent recursion limits must be whole numbers of at least 1, and the runtime
ignores any other value. The config handlers' 4xx carry a `code`, the capability middleware adds
AUTH_REQUIRED/FORBIDDEN, and the admin config router opts out of the terms gate. PATCH checks a
section manager on every section it names, skipped entries included, as /changes does. New meta
flags: writeReport, issueParams, adminTermsExempt.

* 🧪 test: Expect Issue Params in the Whole-Document Override Validation Scenario

Validation issues now carry `params` (here the expected and received types), so the scenario that
asserts the exact issue list for a rejected PUT expects them too.

* 🧪 test: Cover Merged Read Codes, Tenant-Scoped Read Reports and Deployment-Only Resolution
2026-10-24 10:01:13 +02:00
Danny Avila
f05f50460a
🔭 feat: Admin Config Read APIs (Resolved and Effective Config, Scope Catalog, Env Status) (#17383)
* 🔭 feat: Admin Config Read APIs (Resolved and Effective Config, Scope Catalog, Env Status)

Admin clients could read one config document at a time, or the deployment's merged config,
but nothing showed what a profile or a user resolves to: a client had to re-implement the
merge (Base-first precedence, merge keys, tombstones, the interface permission filter,
base-only fields, runtime stops, secret rebinding, the env allowlist) to preview it, and had
no way to see why part of a profile is ignored. Endpoint, model and tool pickers came from the
end-user routes, called as the signed-in admin, so they showed the admin's own scope and keys
and needed a browser User-Agent.

The admin config API now has four read routes. GET /resolved/:principalType/:principalId and
GET /effective?userId= return librechat.yaml, the documents that merge over it in merge order,
each with the parts the merge does not apply and why, and the result, in librechat.yaml shape.
They come from resolveConfigLayers, which tracks the same fold the served merge runs and is
pinned to it by parity tests, and from the stored documents rather than the override cache.
GET /catalog lists the endpoints (with the reason one is not served), models (served,
configured and defaults), MCP servers and tools a scope serves, resolved for the scope rather
than the caller and never with a user's stored key; GET /env-status reports whether each env
name the config uses is set, never its value. Every read is filtered to the sections the caller
can read, redacted as reads redact, and scoped to the caller's tenant. The admin Langfuse routes
now say why a deployment offers no connection settings. Meta reports resolvedApi, catalogApi,
envStatus and limits.catalogModelsTtlMs.

* 🪞 fix: Report Duplicate Custom Endpoints from the Served Definition

When librechat.yaml defines a custom endpoint name twice, or both `Ollama` and `ollama`, the
catalog kept the first definition for a name's env names, reason and models, while
`loadCustomEndpointsConfig` serves the last servable one. An entry could pair the served config
of one definition with the env names of another.

`resolveServedCustomEndpoints` now holds the runtime rule (the last definition with a name, key,
base URL and models wins under its served name), and `loadCustomEndpointsConfig`,
`loadConfigModels` and the catalog all use it. A name with no servable definition is reported
from its last one.

* ⏱️ feat: Make the Admin Catalog Model-List Lifetime Configurable

`GET /api/admin/config/catalog` reused a scope's model lists for a fixed two minutes. The new
librechat.yaml field `endpoints.catalogModelsTtlMs` sets that lifetime (milliseconds, `0` to one
hour, two minutes by default); `0` resolves the lists on every read. It is a base-only path, so a
profile cannot change it and `GET /meta` reports one value, now read from the deployment's config
on each request as `limits.catalogModelsTtlMs`.
2026-10-24 09:23:22 +02:00
Danny Avila
4d061fdd46
🔄 fix: Judge Tenant Profiles on Their Own Base, Clear Config Caches on Principal Deletion, and Add a Config Reload Route (#17397)
Profile writes are judged in the tenant partition the profile merges in. For a role, or a
caller inside a tenant, that is the caller's partition. For a group or user profile written by
an admin without a tenant, it is the stored document's tenant: PUT, PATCH and POST …/changes
validate on librechat.yaml as that tenant resolves it with that tenant's __base__ over it, use
the same Base for the recheck after commit and for key binding, tombstones and field deletes
judge re-entry on that tenant's librechat.yaml, and every route reports `ignored` over that
tenant's Base. Each attempt reads the document in that partition only, and a document that
moved partition while the write ran answers 409 CONFIG_VERSION_CONFLICT. The config store's
partition rules are unchanged; another tenant's Base is read in that tenant's context.

Deleting a group, role or user clears the config caches when it removes or restores the
principal's profile (`invalidatingConfigStore`), awaited before the response and broadcast with
Redis as config writes do; a call that rejects clears too, since it may have committed.

POST …/changes lets `unset` remove any part of a top-level section configSchema does not
define; the section's own `unknown_section` issue is no longer blamed on the removal, and every
other op there still answers 400 UNKNOWN_SECTION.

POST /api/admin/config/reload, for an admin without a tenant holding broad manage:configs (a
tenant admin gets 403, since the reload reaches every tenant), runs the config cache clear without
a write, loads librechat.yaml again on the answering instance, and once it loaded publishes the
clear over Redis so every other instance loads the file on its next read. It uses
ConfigCacheInvalidation.reload, which reports the clear, the load and the publish instead of only
logging a failure (writes keep `invalidate`, which leaves the load to the next read). It answers
200 `{ success: true, broadcast }`, `broadcast` false without Redis; 422 CONFIG_RELOAD_INVALID
with each schema issue's location when the loader refuses the file, keeping the last good config
and publishing nothing; 500 CONFIG_RELOAD_FAILED when a clear fails or the config cannot be built
from the file; and 503 CONFIG_RELOAD_BROADCAST_FAILED when this instance reloaded but the publish
failed, since an instance that misses the broadcast keeps its cached librechat.yaml, which never
expires. Every reload that runs is audited as `config.reloaded`, a failed one with outcome
`failure` and its code. The meta reports `features.reloadRoute`. Writes still reload the file.
The e2e content-filter helper uses the route instead of a sentinel profile.

docs/mcp-overrides.md describes the current timing: the awaited clear, the Redis broadcast, what
still waits for a TTL, that a process the clear misses keeps its old librechat.yaml, and hiding
as an access boundary at checkout.
2026-10-24 08:45:54 +02:00
lia-by-librechat[bot]
8d921e6f7e
🥸 test: Cover Overlapping Web Search Secret Patches (#17390)
Co-authored-by: Lia <lia@librechat.ai>
2026-10-23 15:57:05 +02:00
Danny Avila
db225c0919
🔐 feat: Admin Config Secrets v2 (MCP and Azure Credentials, Re-entry, Env Allowlist, Rebinding) (#17353)
* 🔐 feat: Admin Config Secrets v2 (MCP and Azure Credentials, Re-entry, Env Allowlist, Rebinding)

The admin config API stored MCP server credentials and Azure group keys in plaintext, returned
them to config readers, let an admin who cannot read a stored credential send it to a host of
their choosing, and let override values reference any environment variable. MCP credentials
and Azure keys, in `groups` and in the `groupMap` the Azure clients read, are now encrypted
with masked previews, hidden from every read and decrypted once after the merge. A write that
keeps a stored MCP credential or web search key while changing where it is sent must enter it
again, judged by the value the written document holds. `${VAR}` references in overrides are
limited to an allowlist, named where a change batch leaves them, and the merge drops a
credential a layer moves to a new destination without entering it, listing it in write
responses only for sections the caller can read. A custom endpoint's key is bound to its base
URL, its provider and the destination options in addParams and dropParams (such as the
anthropicApiUrl the native Anthropic client sends to), matched across layers under the name the
merge folds (ollama is Ollama). A restated `${VAR}` counts as entered, so allowlisted names
are not bound to a destination. Azure's `${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` stay
templates in a group's `baseURL`, and a scheduled job's MCP consent no longer depends on the
masked previews that change when an admin rotates the key in place.

* ♻️ refactor: Move the MCP Key Re-entry Guard Out of the Binding Helpers to Break an Import Cycle

With #17351's OpenID provisioning on dev, the app config service reached itself through the
rebinding guard: app/service -> app/overrides -> app/rebinding -> mcp/registry/binding ->
mcp/errors (and mcp/types) -> mcp/oauth/obo -> utils/identity -> auth/openid -> auth/provision
-> app/resolve -> app/service. The binding helpers are pure diffs; only
requireApiKeyReentryForRebinding needs the MCP error class, so it moves to
mcp/registry/reentry.ts, and the helpers take MCPOptions from librechat-data-provider as
mcp/headers.ts does. No behaviour changes.

* 🧪 test: Drop a Base-Held Endpoint Key That a Lower-Priority Profile Moves

Base merges first since #17352, so a profile below Base's priority still
lands over it. The rebinding guard drops the key Base stores for a custom
endpoint whose baseURL the profile changes, and the PATCH response lists
it in `ignored`.

* 🔐 fix: Serve Admin Azure Group Keys and Keep Deployment Keys at Their Own Destinations

Azure groups a profile writes now take effect: the merge resolves a layer's
`endpoints.azureOpenAI.groups` as AppService resolves librechat.yaml's
(`resolveAzureGroups`, shared with `azureConfigSetup`) into the `groupMap`,
`modelGroupMap`, model list and assistant models the Azure clients read. A
layer's groups replace the groups below whole, as its array does; groups that
do not resolve stay unread with a warning. The rebinding guard judges the
resolved groups like any `groupMap` entry, and an Azure group's destination now
also covers a deployment name its base URL takes its host from and the
destination options of its `addParams`/`dropParams`.

A write that keeps a stored Azure group key (in `groups` or `groupMap`) while
changing where the group sends it answers 400 `CONFIG_SECRET_REENTRY_REQUIRED`
on PUT, PATCH, field deletes, tombstones and change batches, judged as the
document resolves over librechat.yaml and on each `groups` item as written.

OCR fills an empty key from `OCR_API_KEY` only for its default destination (no
base URL, or `${OCR_BASEURL}`), so a key the guard dropped for an override's
`${VAR}` base URL is not replaced by the deployment's. The env reference policy
includes the `${OCR_API_KEY}` loading fills in for an `ocr` section without a
key, so a client can write back what GET `/base` returned.

The custom endpoint guard no longer needs the layer's overrides to name an
entry: an entry changed some other way is judged as inherited (positional
tombstones inside the list are not applied by the merge, so none reached one).

A web search URL a layer points at a `${VAR}` other than librechat.yaml's
without entering its key is set back: web search sends the deployment's key,
or its client's environment default, to any `${VAR}` URL.

* 🔐 fix: Keep Azure Assistants Served, Opaque Keys as Entered and Batch Env Checks on the End State

Azure assistants: a profile's `groups` replace the groups below whole, so
groups with no model that supports assistants under librechat.yaml's
`assistants: true` left `assistants` on with an empty `assistantModels`, and
the Azure Assistants initializer picked `assistantModels[0]` (undefined).
`azureConfigSetup` refuses that combination at startup; the merge now turns
`assistants` off when none of the models it leaves supports them, with a
warning, judged on the merged result so a later layer whose groups serve
assistants keeps them on. The check is `lacksAzureAssistantModels`, shared
with `azureConfigSetup`.

Opaque credentials: every runtime reader of the OCR, speech, web search and
Langfuse keys uses the decrypted value as is (`resolveConfigSecret`,
`resolveStoredConfigSecret`, `decryptConfigSecret`), so a `${...}` inside a
key the write encrypts is part of the key. Those fields are marked `opaque`:
the write-time check skips a value the write encrypts there, and the runtime
pass keeps their ciphertext instead of blanking it. A whole `${VAR}`
reference there, which the field passes through and its reader resolves, is
still checked, and so is stored plaintext. Custom endpoint (`initializeCustom`),
Azure group (`mapModelToAzureConfig`) and MCP (`processMCPEnv`) credentials
are expanded by their clients once decrypted and stay checked; the Assistants
keys have no reader yet and stay checked too.

Change batches: env references are judged on the document the batch leaves,
as its other issues are. An issue counts only when the end state still holds
a disallowed reference (ciphertext by its plaintext) where the batch sent one,
at a location the batch changed, so a value a later op removes or replaces
(`set ocr.baseURL`, then `unset ocr`) is no longer refused, and a stored
reference the batch puts back where it was is left to merge time.

* 🔐 fix: Proxy-Bound OAuth Secrets, Consistent Azure Group State, Batch Issues by Entry Key and Loaded Web Search References

- The MCP proxy is part of the OAuth client secret's binding (user tier, admin re-entry and the
  merge's rebinding guard share `getChangedOAuthSecretBindingFields`): the access token the
  secret obtains travels through it.
- A tombstone of `endpoints.azureOpenAI.groups` resolves the groups to none in the merge (empty
  `groupMap`, `modelGroupMap` and model lists, so the model list does not fall back to the
  environment's defaults), and removes what they resolve to in `applyConfigTombstones`.
- The merged Azure model lists name only models a merged group serves
  (`narrowAzureServedModels`), so an override's `assistantModels` no group serves no longer keeps
  Azure assistants on.
- `/changes` env issues are held by entry identity (`toConfigIdentityPath`), followed through
  renames and resolved on the batch's end state, so a removal or rename no longer shifts them.
- References loading librechat.yaml introduces (OCR's `${OCR_API_KEY}`, the web search defaults)
  are allowed at the location they are loaded at, so GET /base's sections are accepted written
  back, and nowhere else.

* 🔐 fix: Re-entry for Endpoint, OCR and Speech Keys, Own Keys Bound Over Base and User Key Destinations Kept

- The document that stores a custom endpoint, OCR or speech key can no longer
  move its destination without the key: PUT, PATCH, field deletes, tombstones
  and change batches answer CONFIG_SECRET_REENTRY_REQUIRED at the changed field
  (getBoundKeyReentryIssues), judged by value as the document alone sends it.
- A profile's own key for an endpoint, MCP server, OCR or speech service, or
  Azure groupMap group that Base defines is bound to that destination when it
  is entered: the rebinding guard reports a fingerprint of where it goes, the
  write stores it beside the key (<secret>Binding), and the guard keeps the key
  there until a lower layer moves it. Reads and the merged config leave the
  fingerprint out; kept keys carry it; client values are replaced.
- An inherited custom endpoint or MCP server that sends a credential each user
  holds (user_provided, customUserVars, apiKey.source user, OpenID or Graph
  token placeholders) keeps its destination: a layer's move is set back and
  listed in ignored.
2026-10-23 06:37:54 +02:00
Danny Avila
76e3c6b134
🪜 fix: Base-First Config Precedence, Profile Lifecycle and Role Renames That Keep Their Profile (#17352)
* 🪜 fix: Base-First Config Precedence, Profile Lifecycle and Role Renames That Keep Their Profile

Config documents merge in one total order (`compareConfigPrecedence`): Base (`role` /
`__base__`) first whatever its priority, then ascending priority, then role < group < user,
then document `_id`. The merge, `getApplicableConfigs` and `listAllConfigs` all use it, so a
tie no longer goes to the newest document whatever its type, and Base no longer outranks a
profile through its priority. A profile that writes fields of a librechat.yaml MCP server
therefore undoes a Base hide of that server for its principals. A write that creates a document
without a priority gives it its principal type's default (role 10, group 20, user 100, Base 0),
and a write that omits the priority keeps the stored one, so a broad PUT no longer resets a
profile to 10. Langfuse writes no longer pass a priority.

Profile writes (PUT, PATCH `/fields`, POST `/changes`) validate on librechat.yaml with the
active Base document resolved over it (`resolveRawConfig`, the fold the merge itself runs), so
a role write that is valid where it lands is accepted and one the merge would strip is refused.
Base still validates on librechat.yaml alone, an inactive Base is ignored, and `/changes` still
judges what librechat.yaml itself defines on the raw YAML.

`PATCH /api/admin/config/:principalType/:principalId/priority { priority, expectedRevision? }`
sets only the priority, for broad `manage:configs`, keeping overrides, tombstones and
`isActive`; it is audited as `operation: 'priority'`. A write that would create a principal's
first document (PUT, PATCH `/fields`, POST `/fields/tombstone`, POST `/changes` including a dry
run or a priority-only batch) answers 404 `PRINCIPAL_NOT_FOUND` unless the role (exact name),
group or user (lowercase ObjectId) exists in the caller's tenant partition, or at platform
level outside a tenant (`principalExists`). Each route checks inside its compare-and-swap
write, against the document the write replaces; the tombstone route now plans its write like
the others, so `AdminConfigDeps` no longer takes `tombstoneConfigField`. A PUT with
`expectedRevision: 'absent'` creates the profile even when it carries no overrides.

Renaming a role moves its config profile, system grants and ACL entries to the new name in the
caller's tenant partition. The profile moves through the store contract (`movePrincipal`: a
create-only copy, then removal at the copied revision). A profile already at the new name
answers 409 `ROLE_CONFIG_CONFLICT`, and any failed step rolls back the users, the role document
and every move (500 `ROLE_RENAME_FAILED`). Profiles, grants and ACL entries that renames made
before this change left at an old name are neither moved nor removed, and a role created under
that name still picks them up. The Base principal id is a reserved role name (400
`ROLE_NAME_RESERVED`), and a legacy role by that name never moves or deletes Base. The admin
config meta reports `deterministicPrecedence`, `priorityRoute`, `principalCheck` and
`roleRenameMovesProfile`.

* 🧪 test: Give the E2E Reload Sentinel a Role of Its Own

Admin config writes now answer 404 PRINCIPAL_NOT_FOUND for a principal that does not exist, so
the content-filter helpers' reload sentinel (a made-up user id) could no longer write and every
spec that reloads librechat.yaml failed. The sentinel is now a role the helper creates once (409
when it already exists) and deletes after its profile in cleanup; no user is a member, so its
empty profile changes no one's config.

* 🛟 fix: Keep a Moved Profile Whose Removal Cannot Be Confirmed, Validate on the Caller's Base and Restore Rolled-Back Descriptions

- `movePrincipal` no longer treats a failed read-back of the source, after its removal
  failed, as the source still being there. It keeps the copy, which may be the only one
  left, and rethrows the removal's failure instead of deleting the copy.
- Profile writes read the Base validation floor from the caller's tenant partition, so a
  caller without a tenant validates on the platform-level Base and never on a tenant's.
- A role rename that rolls back gives a role stored without a description the schema
  default back, instead of a `$set` of `undefined` that kept the new description.

* 🧭 fix: Scope Profile Priority Changes to the Caller's Tenant Partition

`setConfigPriority`, behind PATCH `/api/admin/config/:principalType/:principalId/priority`,
read and wrote through the unpartitioned config store. A platform-level admin without a
tenant runs in the default non-strict context, where the tenant-isolation plugin adds no
predicate, so the route could reorder a tenant's profile for the same principal, and answer
with it, when no platform-level profile existed.

It now runs on the partitioned operations that the role rename and the partitioned
`findConfigByPrincipal` use. The read is limited to the caller's partition: that tenant's
document, or without a tenant only a platform-level one (no tenant, `null` or `''`). The
write is pinned to the `_id` of the document that read returned. A tenantless caller
therefore gets 404 `CONFIG_NOT_FOUND` for a profile only a tenant holds. An
`expectedRevision` naming such a profile gets a revision conflict and changes nothing.

Tests: `config.spec.ts` (data-schemas) adds "setConfigPriority in the caller tenant
partition". It covers the platform profile changing alone, a profile that only a tenant holds
being neither changed nor returned, and only the active tenant's profile changing inside a
tenant. The tenant helpers are now shared with the rename suite.
`config.precedence.integration.spec.ts` adds the same check through the handler. With the old
wiring, a tenantless PATCH of a profile only tenant-a holds answers 200 and moves tenant-a's
profile to 40. With the fix it answers 404 and tenant-a keeps 10.

* 🧷 fix: Roll Back Only What a Rename Changed and Validate Profile Writes on the Base They Land Over

- Grant moves report what they changed (re-keyed grants, and superseded grants whose removal
  was confirmed); the undo re-keys only grants still at the new name and re-creates a
  superseded grant only while the new name still holds the grant that superseded it, so a
  grant revoked during a rename is never resurrected.
- The Mongoose binding settles a create whose acknowledgement was lost by the `_id` it chose
  (found at version 1: committed, returned), as the store contract now requires; a profile
  move whose copy committed finishes instead of leaving an orphan copy. The memory engine and
  the conformance suite cover it, plus a copy that cannot be withdrawn being logged.
- A rename rollback moves back only the profile it moved (by identity, edits included), puts
  back only members still on the new name, and reverts the description only while it is still
  the one the rename set.
- PUT, PATCH /fields and POST /changes read Base beside the profile on every attempt and read
  it again once the write commits; a write that Base changed under is judged on that Base and
  withdrawn (only at the revision it wrote) when it fails there.
- The priority route rereads a lost race's current revision in the caller's tenant partition.
2026-10-22 23:32:27 +02:00
Danny Avila
0abbf3313d
🔐 fix: Apply Base Config Overrides Before Sign-In and Refuse Script Policy Links (#17351)
* 🔐 fix: Apply Base Config Overrides Before Sign-In and Refuse Script Policy Links

Sign-in decisions made before the user is known now read librechat.yaml plus the `__base__`
override of the request's tenant, or of the deployment outside a tenant, and fail closed
(`resolvePreAuthAppConfig`). Registration, the first phase of password reset and the first
email-domain check of the LDAP, OpenID, SAML and social strategies use it. The second check
applies the account's role overrides with or without a tenant, and a new account is checked
against the role it will be created with before anything is inserted. OpenID checks again when
the admin claim or role sync changes the role. Password reset issues no link when the
account's config cannot be read.

The signed-out `GET /api/config` serves Base `registration.socialLogins` and the pre-login
interface fields (`resolvePreLoginAppConfig`), falling back to librechat.yaml when overrides
cannot be read. `turnstile` becomes a base-only section, so Turnstile comes only from
librechat.yaml.

Config reads that merge documents stay in one tenant partition (`tenantPartition`, Mongoose
binding `tenantPartitionFilter`): the active tenant's documents, or the tenantless ones outside
a tenant. The config store's Mongoose binding reads, writes, lists and restores role documents,
`__base__` included, in that partition, and its `listActive` draws every document from it;
group and user documents stay addressed by id. Only a `list` with `acrossTenants` reads every
partition, which `listConfigTombstones` uses so the startup MCP tombstone report still names
each tenant's documents. The tenant-isolation conformance suite gains single-partition cases.

Privacy policy and terms of service `externalUrl` must be http(s), `mailto:`, `tel:`, relative
or blank (`isSafeLinkUrl`). Admin writes of another scheme are refused, a librechat.yaml link is
dropped with a warning at load, a stored one is dropped at merge, and every payload and the
client remove one that reaches them. The admin config meta reports
`features.preAuthBaseOverrides: true`.

* 🧪 test: Layer a Required-Field Override Through interface.currency Instead of Turnstile

Turnstile is set in librechat.yaml only now, so profile overrides of it are skipped and the
layering scenario could no longer see a role's Turnstile options completed by a user's site key.
The scenario now layers `interface.currency`, whose `code` and `rate` are both required: the role
stores only the rate, the user only the code, and /api/config serves both.

* 🛂 fix: Admit a First OpenID Login Under Its Claimed Role Before Creating the Account

The admin-role claim and role sync are now decided before a first-login OpenID account is created, and createUserOnce admits the email under that role's config, read in parallel with the USER role's, before inserting anything. A login the claimed role refuses creates no account and no start balance. Admitted logins are unchanged: the account is still created as USER (now explicitly), moved into the claimed role, re-checked and audited as before, with the claims decided once. An account a concurrent first login created gets the claims decided again for its own role and tenant.
2026-10-22 22:49:54 +02:00
Danny Avila
6189220697
🏷️ fix: Resolve endpoints.all Title Settings per Field and Fold Ollama Endpoint Names (#17316)
Some checks failed
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Publish `@librechat/client` to NPM / pack (push) Waiting to run
Publish `@librechat/client` to NPM / publish-npm (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* 🏷️ fix: Resolve endpoints.all Title Settings per Field and Fold Ollama Endpoint Names

Conversation titles picked `endpoints.all` as a whole block, so any `all`
block (even one holding only streamRate, headers or activity label keys)
hid every per-endpoint titleConvo, titleModel, titleMethod, titleEndpoint
and title prompt. `resolveTitleConfig` resolves each field from `all`,
then the named endpoint, then the custom endpoint entry, keeping
titleEndpoint and titleModel as one choice from the first layer that sets
either.

Config overrides merged `endpoints.custom` by exact name while the runtime
serves names through `normalizeEndpointName`, so an override `ollama`
survived beside the YAML `Ollama` and split the endpoint between two
baseURLs and keys. The merge key now folds the same way (the YAML spelling
is kept), merge-key issues, issue attribution and secret preservation use
the same identity, and the effective config serves
`modelSpecs.addedEndpoints`, `modelSpecs.list[].preset.endpoint` and
`endpoints.agents.allowedProviders` in served names without repeats.

A stored document that already names one endpoint in two spellings is
combined at merge in document order, so requests keep the host and key
of the first item instead of moving to the YAML entry; omit-to-keep
secret preservation reads the same combined entry. A new write naming
both spellings is rejected as a duplicate.

An agent saved as `Ollama` now runs on the served `ollama`: the model
catalog lookup (`resolveModelCatalogKey`) falls back to the served name,
and `initializeAgent` folds the provider before the allowedProviders
check and provider setup.

* 🏷️ fix: Report Per-Field endpoints.all Title Resolution in the Admin Config Meta

resolveTitleConfig resolves endpoints.all title settings one field at a time, so GET /api/admin/config/meta now reports rules.titleResolution as 'field'. meta.spec ties the reported value to resolveTitleConfig: with endpoints.all = { streamRate: 1 } and openAI.titleModel = 'x', the rule is 'field' exactly when the endpoint's titleModel survives.

* 🧪 test: Expect Segments on the Folded duplicate_merge_key Write Issue

Admin config write issues carry segments since #17322; the folded Ollama/ollama duplicate test now asserts them.

* 🏷️ fix: Match Folded Custom Endpoint Names in Admin Config Change Ops

The changes endpoint from dev matched entry keys exactly, while the config
merge, merge-key issues and secret preservation fold `Ollama`/`ollama` into
one custom endpoint. An upsertEntry `ollama` over a stored `Ollama` copied the
stored key, appended a second entry and failed as duplicate_merge_key;
removeEntry `ollama` left a stored `Ollama` in place; a rename onto another
spelling passed entry_exists; and the librechat.yaml entry check missed
`ollama` against the YAML `Ollama`.

- data-schemas: `getConfigEntryIdentity` folds a key by its override path;
  `getConfigEntryMatcher` backs upsertEntry, removeEntry, rename, written-path
  attribution and the stored-counterpart lookup; issue locations use the same
  identity. A rename of an entry stored in two spellings moves the one entry
  the runtime merge combines (`combineFoldedConfigEntries`).
- api: the YAML-entry rename guard and the per-batch duplicate check compare
  entries by the same identity.
- Tests: unit coverage for each op across spellings, and handler integration
  tests against MongoDB for upsert with secret preservation (single and
  two-spelling documents), removal, entry_exists, inherited_entry and
  duplicate changes in one batch.
2026-10-22 16:57:42 +02:00
Danny Avila
a3e8426588
🛡️ fix: Reject Unknown Admin Config Sections and Scope Write Responses to Read Access (#17315)
* 🛡️ fix: Reject Unknown Admin Config Sections and Scope Write Responses to Read Access

- Admin config writes (PUT, PATCH, tombstone) now accept only top-level sections that configSchema defines. PUT reports `unknown_section` issues under CONFIG_OVERRIDE_INVALID; PATCH and tombstone return 400 UNKNOWN_SECTION without echoing the path. This closes override writes to AppService runtime state (`paths`, the raw-YAML `config`, `availableTools`, and the `interfaceConfig`/`turnstileConfig`/`mcpConfig` aliases).
- mergeConfigOverrides ignores stored override keys and tombstones outside configSchema, with one warning per document, so data an older server stored is neutralized without a migration. DELETE /fields still removes such stored entries, which is the only per-field way to clear a legacy tombstone.
- Write responses (PUT, PATCH, tombstone, field delete, toggle) are redacted and then filtered to the sections the caller can read plus the sections the write managed; broad managers see the whole document as before.
- Admin reads of the resolved config redact literal Azure group keys (merged `groups`, `groupMap`, raw YAML copy) and the Vertex `serviceKeyFile` (`vertex` and `vertexConfig`), keeping `${ENV}` and `user_provided` references readable. Write behaviour for these paths is unchanged.
- The experimental cluster entry boots from librechat.yaml only (`baseOnly`), like server/index.js, so `__base__` overrides no longer reach file storage, skill sync, skill deletion or the expired-file sweep at boot.

* 🛡️ fix: Resolve Admin Config Write Read Access Before Committing

- Section-scoped writes (PATCH, tombstone, field delete) resolve the caller's readable sections before the mutation, alongside the pre-write reads they already make, instead of after the write and audit have committed. The batched lookup covers every `configSchema` section, so it no longer waits on the stored document.
- A failed grant lookup now fails the request before anything is stored, so a 500 never follows a committed write and invites a duplicate retry. Response filtering after the write is pure.
- Assign-only scope creation and toggles match only empty documents (`expectEmpty`), so their responses skip the lookup entirely.

* 🚩 feat: Report the Section Allowlist and Read-Filtered Write Responses in Config Meta

- `GET /api/admin/config/meta` now reports `sectionAllowlist: true` and `readFilteredWriteResponses: true`, which this branch implements (BE-17, BE-18).
- The meta.spec probes check each flag against the handlers rather than the validator alone: PUT answers `unknown_section` with `segments`, PATCH and tombstone answer `UNKNOWN_SECTION`, a stored `paths` override and tombstone are ignored at merge, and PATCH, tombstone and field delete responses omit a section the caller cannot read.
- A dotted top-level key is reported once, as `dotted_key`, instead of also as `unknown_section`.
- The unknown-section handler and integration tests expect the `segments` that issues carry since #17322.

* 🧪 test: Type the Tombstoned Meta Probe Document Through storedConfig

* 🧪 test: Align Allowlist and Filtered-Response Specs with Version-Checked Web Search Writes

dev's version-checked webSearch writes pass a session and precondition to
patchConfigFields/unsetConfigField, read the stored document before removing a
binding path, and have getBaseConfig load the librechat.yaml config beside a
merged read. The field delete spec now stores the document it removes from and
asserts the precondition alongside the filtered response.

* 🧪 test: Pin Section-Manager Write Responses to What a Read Shows

The grant layer resolves read:configs:<section> from the same section's
manage:configs:<section> (getParentCapabilities, #14472), so a section
manager's GET already returns the whole managed section. The comment on
resolveWriteReadAccess said the opposite; it now states the model.

A new integration spec runs the handlers over real SystemGrant documents
through generateCapabilityCheck: a caller holding only
manage:configs:endpoints patches one endpoints field, and its write
response equals what GET returns it, other endpoint overrides and
tombstones included, with memory and webSearch left out of both.

* 🧪 test: Mark Config-Service Generation Tests With a Schema Section

The base-config generation tests that came with the portable config store marked each base
config and override layer with an `x` key, which no section in `configSchema` defines. This
branch dropped `x` from the spec's `TestConfig` and ignores such keys when overrides are merged,
so those tests no longer typechecked, and the override-layer ones would stop seeing the layer
they read. They now mark layers with the `version` section, as this branch's other service tests
do.

* 🧪 test: Expect the yamlMcpHide Flag Beside the Section Allowlist Flags

#17327 publishes features.yamlMcpHide on dev, so the exact feature list this PR asserts in
meta.spec now includes it.
2026-10-22 16:52:56 +02:00
Danny Avila
8c51a3f9f8
🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields (#17327)
* 🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields

Config profiles are the `__base__`, role, group and user documents written through `/api/admin/config`. A profile can tombstone `mcpServers.<name>`, and the merge then removes that server from the principal's resolved `mcpConfig`.

* 🛡️ fix: Apply MCP Hiding to Server Instructions, Failed Removals and the Startup Audit

- Server instructions bound for the model now leave out librechat.yaml servers hidden from the acting user, including one a model spec pins. `applyContextToAgent` passes the request user through `formatInstructionsForContext`, which drops the user's hidden set from the user-less catalog it reads.
- A config-tier candidate that removes a librechat.yaml value (a field, or a key of a record such as `headers`) and fails inspection now stands as a failed config-source entry instead of falling back to the librechat.yaml base that still carries the removed value. A failed change that removes nothing still falls back, and an entry with no transport (fields written over a hidden server) keeps the documented librechat.yaml fallback.
- The startup tombstone report reads every tenant's profiles in the system tenant scope, so it also runs from entry points without an outer system scope under strict tenant isolation, and names each profile's tenant.

* 🗂️ fix: Project the Startup MCP Tombstone Report Query and Run It Alongside Connections

* 🔑 fix: Keep Removed OAuth Headers Out of the MCP OAuth Callback

* 📝 docs: Note That Removed OAuth Headers Stay Out of MCP Sign-In

* 🔐 fix: Recheck the MCP OAuth Flow's Server Before Initiation Redirects

OAuth initiation redirected to a flow's stored authorization URL, or started
a new authorization from the flow's stored server URL, without resolving the
server again, so a principal whose profile hid the server after the flow was
created could still be sent to its authorization endpoint. The callback's
active-server check and its OAuth header fallback lived in the CJS route.

resolveMCPOAuthTarget (packages/api/src/mcp/oauth/target.ts) now owns both:
it resolves the server in the principal's current config with a strict read,
refuses a flow admitted for another server, a server the principal can no
longer use, or a redefined one, and selects the headers the flow sends (the
initiation snapshot, else the principal's config). Initiation calls it before
either redirect (404 or 400); the callback calls it before the exchange and
from the token-store fence.

* 📝 docs: Note That a Hidden MCP Server Ends OAuth Sign-Ins Already Started

* 🚩 feat: Publish the yamlMcpHide Admin Config Meta Flag

Profiles can hide librechat.yaml MCP servers and a hidden server is an
access boundary, so GET /api/admin/config/meta now reports
features.yamlMcpHide for the admin panel to gate its hide UI on. The
meta.spec probe checks the behaviour itself: the admin API stores an
mcpServers.<name> tombstone, the merged config drops the server, the
registry withholds it from principal lookups and checkout but not from
app-level lookups, process-backed servers stay, and an unreadable
principal config withholds every hideable server.

* 🧪 test: Align MCP Tombstone Specs With the Portable Config Store

The admin config handlers now write whole documents through
writeConfigDocument, so the end-to-end tombstone spec wires it in place
of the removed upsertConfig/patchConfigFields deps. A reset of a whole
MCP record now also prunes the parents it leaves empty.
2026-10-22 14:10:31 +02:00
Danny Avila
4e0d9145e0
⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename) (#17328)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* ⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename)

Admin clients can change a principal's config overrides only one kind of change at a time:
- PATCH `/fields` for sets, at most 100 entries;
- POST `/fields/tombstone` for one tombstone;
- DELETE `/fields` for one reset.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🧷 fix: Tighten Config Change Batches Around Array Entries, Cancelled Changes and MCP Tombstones

- A tombstone with an index segment below a field configSchema does not define (an unknown key, or what an intersection or any holds) is refused as inside_array: the merge removes inherited values from objects only, so it was stored and reported applied while the element stayed visible.
- unset joins set and rename in the merged-by-key position check: unset endpoints.custom.0.headers answers 400 indexed_merge_key_write, since the position can name another endpoint by the time the batch applies. upsertEntry and removeEntry address entries by key.
- A batch whose changes cancel out (set interface.customWelcome, then unset interface) writes nothing: no empty document, revision bump, audit entry or invalidation. Its changes are reported as unchanged.
- A whole-section mcpServers tombstone reports each librechat.yaml stdio server it leaves in place as ignored (process_mcp_server).
- removeEntry no longer attributes the whole collection to itself, and existing issues of merged-by-key entries are matched by key rather than index, so removing an entry beside a legacy-invalid one (or shifting it) is no longer refused for that entry's issue.

* 🪪 fix: Keep a Requested Priority on New Principals and Report Ignored MCP Renames

POST …/changes discarded a broad manager's priority for a principal without a document when no change altered it: the batch answered written: false instead of creating the priority-only document PUT creates. An explicit priority on an absent principal now counts as a change.

A rename under an MCP server librechat.yaml runs as a process persisted an override the runtime merge drops, but was not reported in ignored. It now is, like set and tombstone. An unset there stays unreported: it asks for the YAML value, which is what runs.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧪 test: Reload librechat.yaml After a Committed Change Batch and Not After an Unwritten One

* 🧮 fix: Validate Change Batches on Their End State and Report Base-Only Fields a Tombstone Keeps

A batch's validation scope was built from every intermediate write, located in the state each
change produced. A change a later one undid (set `balance.startBalance`, then restore `balance`)
still widened the scope, so an issue the stored document already had there rejected an
unrelated change, and an entry a later change moved (upsert one entry, then remove one before
it) pointed its written index at a neighbour with a legacy issue. The written paths are now
located in the batch's end state and kept only where that state differs from the stored
document, with entries of merged-by-key arrays compared by key.

A tombstone above a base-only field (`endpoints.agents`, `endpoints.agents.pullRequests`) is
stored and hides the rest of the subtree, while the runtime merge puts back the provider, host,
OAuth app and shared cache slots librechat.yaml sets. The response now lists those fields in
`ignored` with reason `base_only_path`, as it already does for the stdio MCP servers a
whole-section tombstone leaves in place.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.

* 🔐 fix: Require the Stored Web Search Key Again When a Change Batch Moves Its URL

#17323 refuses a PUT, PATCH, tombstone or field delete that changes or removes a web search
URL beside a stored API key unless the key is entered again, so a stored key cannot be sent
to a destination it was not entered for. POST /changes skipped that check: a batch could set,
reset or tombstone `webSearch.searxngInstanceUrl` (or replace `webSearch` without the key,
which keeps the stored key) and point the stored key at another host.

A batch is now judged on the document it would store, like PATCH: a URL that ends up beside
a stored key it did not travel with answers 400 CONFIG_SECRET_REENTRY_REQUIRED and writes
nothing, dry runs included. Only keys the request sent count as entered again; a secret a set
kept from the stored document does not.

* 🚩 feat: Publish the changesBatch Flag and the maxChangeOps Limit in Config Meta

GET /api/admin/config/meta now reports `features.changesBatch: true` and
`limits.maxChangeOps` (1000, from MAX_CONFIG_CHANGE_OPS), so the admin panel can gate its
one-request save on the endpoint and size its batches from the server. The meta.spec probe
checks the behaviour itself: an oversized batch answers TOO_MANY_OPS, a batch with one invalid
op writes nothing, and a valid set plus tombstone lands as one write.
2026-10-22 13:38:06 +02:00
Danny Avila
0b4dc238b3
🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes (#17326)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.
2026-10-22 13:04:35 +02:00
Danny Avila
9ea3c5647d
🔑 fix: Use Admin-Stored Web Search Credentials at Runtime (#17323)
* 🔑 fix: Use Admin-Stored Web Search Credentials at Runtime

Use admin-stored (encrypted or literal) web search credentials as system
credentials at runtime, add the Keenable key to the secret registry, bind a
stored key to the layer that set its URL, and require re-entering the key when
a write changes or removes that URL (checked atomically with a version
precondition).

* 🧹 fix: Keep Dotted-Key Write Guarantees on Version-Checked Web Search Writes

A version-checked write the caller saw absent first creates an empty
document. When the database then refuses the write (a key name it cannot
store, answered 422 CONFIG_KEY_NOT_STORABLE), remove that document again
so a refused write stores nothing, unless another write has changed it.

Secret re-entry issues now carry `segments` like every other config
issue.

* 🚩 feat: Publish the webSearchStoredKeys Feature Flag in the Config Contract

GET /api/admin/config/meta now reports `webSearchStoredKeys: true`, so an
admin client can stop warning that only ${VAR} web search references work at
runtime. meta.spec probes the behaviour behind it: an encrypted stored key
reaches web search auth, the Keenable key is encrypted on write, and changing
a URL beside a stored key requires entering the key again.

* 🧪 test: Type the Refused-Write Driver Stub in the Config Methods Spec

* 🧪 test: Expect the Write Precondition Arguments in the Admin Config Route Test

The handler passes a session slot and an optional version precondition to
patchConfigFields since the web search re-entry check; the route test added
in #17322 asserted the earlier five-argument call.

* 🧹 fix: Surface a Failed Cleanup of the Empty Config Document

When a version-checked write the database refused cannot remove the empty
document created for it, let that failure reach the handler (500) instead
of logging it and answering 422 CONFIG_KEY_NOT_STORABLE, which promises that
nothing was stored.

* 📝 docs: State the Web Search Stored-Key Binding Rule Exactly in the webSearchStoredKeys Flag

The flag's doc said a stored key goes only to a URL the same layer set. The merge rule keeps a stored key over an inherited ${VAR} or default URL, so the doc now names the exact rule: a stored key reaches a URL stored in config only when one layer set both, a later layer that changes, blanks or removes the URL drops the key, and a write that changes or removes the URL must enter the key again.

* 🧹 refactor: Read the Secret Registry Through getConfigSecretRules Only

The dev merge kept this branch's getConfigSecretFields and getConfigArraySecretFields beside dev's getConfigSecretRules, which returns the same registry data. Only the secret specs used the two extra accessors. They now read getConfigSecretRules().secretFields and .arraySecretFields, so the accessors and their exported ConfigSecretField and ArraySecretField interfaces (duplicates of AdminConfigSecretFieldRule and AdminConfigArraySecretFieldRule) are gone.
2026-10-22 11:29:20 +02:00
Danny Avila
2591a15c10
🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract (#17322)
* 🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract

Keep dotted record keys in admin config write bodies behind a path-scoped
sanitizer, reject $-prefixed and prototype keys after auth, report dotted_key
issues with segments, and publish GET /api/admin/config/meta (features, rules,
limits, versions) so clients can gate on backend capabilities.

* 🧪 test: Expect Issue Segments in the Config Override e2e Scenario

Override validation issues now carry `segments`; assert the full issue shape.
2026-10-22 09:59:08 +02:00
Marco Beretta
de9d46fcd4
🔎 perf: Index-Bound Word-Prefix Search for Users and Groups (#16719)
* perf: Debounce People Picker Search

* perf: Index-Bound Word-Prefix Search for Users and Groups

Principal search and admin user search matched name, email and username
with case-insensitive regexes, which scan the whole users collection.
Users and groups now store lowercase, accent-folded token arrays derived
from each searchable field, indexed with tenantId, and searches match
anchored token prefixes. Schema middleware keeps the tokens in sync on
save, insertMany and update queries. Documents written before the tokens
existed stay findable through a fallback branch until
npm run migrate:search-tokens backfills them; startup warns while any
remain.

* perf: Build Search Token Indexes in the Backfill Migration

* fix: Reconcile Stale Search Tokens and Close Write-Path Gaps

The backfill now compares every document's tokens with its source fields and
repairs stale ones, so writes from an older server during a rolling deploy are
recoverable by rerunning it. Undefined assignments no longer clear tokens,
upsert inserts initialize every token field, every query word is required,
relevance scoring folds accents like the filter does, and the startup check
counts both collections in parallel and owns its error logging.

* perf: Drop the Second People Picker Debounce

SearchPicker already debounces query changes by 500ms; the extra 300ms delay
in UnifiedPeopleSearch stacked on top of it.

* fix: Match Nothing for Queries Over the Word Cap

Dropping the query word cap let one request build an unbounded filter. Restore
a 16 word cap, but a longer query now matches nothing instead of ignoring its
extra words, so a cap never produces a false positive.

* test: Cover Word-Prefix People Search and the Token Backfill End to End

* fix: Tokenize Fields an Upsert Copies From Its Filter

MongoDB copies equality conditions from an upsert's filter into the inserted
document, so their token fields now derive from that value instead of being
initialized empty.

* fix: Rank Accent-Folded User Matches Like the Search Filter

* test: Run the Token Backfill Without Blocking the Playwright Worker

* perf: Index Search Tokens Tenant First for Scoped Searches and Ignore Undefined Unsets

* test: Skip the Module-Scope Index Rebuild When the Scenario Runs the Backfill

* fix: Exit the Search Token Migration When Redis Clients Stay Open

* perf: Probe for Missing Search Tokens at Startup Instead of Counting Them

* test: Explain the Startup Probe the Warning Actually Issues

* fix: Keep Search Token Fields Out of Serialized Users and Groups

* fix: Keep the Admin Search Legacy Fallback Anchored at the Field Start

* fix: Score Groups Only on the Fields Search Matches

* perf: Start the Search Token Probe Alongside the Permission Checks

* fix: Warn at Startup When the Search Token Indexes Were Never Built

* fix: Score People Search Matches With the Filter's Word Separators

* perf: Check the Token Indexes Before Probing for Unmigrated Documents

* test: Drive the Failed Startup Probe Past the Index Check

* fix: Score People Search Matches Independently of Word Order
2026-10-16 22:41:37 +02:00
Marco Beretta
9adabde72e
🗂️ feat: Audit MCP, Code Worker, Langfuse, Skill Credential, Passkey and Audit Log Access (#17054)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: Remove Group Dependents Before Deleting the Group

* feat: Audit Code Worker Pairing and Langfuse Connection Changes

* feat: Audit Skill Sync Credential Changes

* feat: Audit Passkey Enrolment and Removal

* feat: Audit Audit Log Reads and CSV Exports

* feat: Audit Audit Log Retention Purges

* feat: Audit MCP Server Create, Update and Delete

* fix: Keep Purge Result When the Purge Audit Write Fails

* test: Cover Admin Integration Audit Scenarios

* fix: Tighten Admin Integration Audit Scope and Group Cleanup Order

* fix: Run Security Fences Before Awaiting Audit Writes

* fix: Move MCP Post-Commit Audit Ordering Into the Auditor and Audit Verify Reads

* fix: Fail Closed When Recording Audit Log Access

* fix: Audit Group Deletion Before the Cascade and Restore Dependents on Failure

* fix: Restore Only Config Overrides When Group Deletion Fails

* fix: Audit Worker Pairing and Export Before the Work and Append Failures

* fix: Record Group Deletion as Pending Without Delaying Revocation

* fix: Skip Config Restore When a Concurrent Delete Removed the Group

* fix: Restore Group Config When the Existence Recheck Fails

* fix: Keep Group Config Restricted When Delete State Is Unknown

* fix: Record Worker Pairing and Revocation as Pending Until Confirmed

* fix: Deliver the Pairing Code Before the Terminal Audit Write

* test: Assert Pairing Code Delivery Precedes the Terminal Audit Write

* fix: Send Revocation and Group Deletion Responses Before the Terminal Audit Write

* fix: Snapshot Group Config Before the Destructive Cascade Step

* refactor: Type the Group Config Snapshot Dependency Without Mongoose Shapes

* fix: Fence Shared MCP Users Before the Deletion Audit and Correlate Audit Attempts

* fix: Diff MCP Server Audits Against the Committed Config

* fix: Read the Committed MCP Config at Commit Time for the Audit Diff

* refactor: Start the Committed MCP Config Read Inside the Auditor

* fix: Audit Langfuse Patches When Activation Fails and Surface Failed Group Config Restores

* fix: Record Member Removals When Group Deletion Completes After a Failed Restore

* test: Serve the Committed Config on the MCP Update Audit Reread

* fix: Audit the Group Returned by the Compensating Deletion

* fix: Keep Member Removal Audits Fail-Open During Group Deletion

* fix: Retry ACL Cleanup After a Compensating Group Deletion

* fix: Mark Unavailable Audit Detail and Order Member Removals After the Pending Deletion

* fix: Guard Synchronous Committed-Config Read Failures in the MCP Auditor

* fix: Return the Persisted MCP Config and Audit Commits That Outlive Cache Invalidation

* style: Sort Imports in MCP Audit Files

* style: Sort Imports After Restack

* fix: Audit MCP Commits That Outlive Any Post-Write Step and Pend Audit Exports Until Complete

* test: Provide the Fail-Closed Setting in the Code Route Mock

* fix: Refresh MCP Read Caches When a Repository Step Fails After the Write

* fix: Finish Audit Exports on Response Completion and Keep Error Details Out of Messages and Logs

* fix: Watch Response Completion Before the Truncation Footer Write

* fix: Skip Audit for No-op MCP Deletes and Close Remaining Post-Failure Audit Gaps

* fix: Keep Causes Out of Structured Logs and Treat a Lost Compensating Delete as Not Found

* fix: Mark Failed MCP Post-Commit Steps and Null Langfuse Activation in Audits
2026-10-15 00:30:26 +02:00
Marco Beretta
76192bdfd0
🛂 feat: Audit Resource Sharing Changes and Fail Closed on Reversible Audit Writes (#17053)
* feat: Audit Resource ACL Changes and Fail Closed by Default

* fix: Keep Non-Reversible Admin Audit Handlers Fail-Open

* docs: Narrow AUDIT_LOG_FAIL_CLOSED Description

* fix: Audit and Restore ACL and Insights Changes in One Pass

* fix: Break Audit and Insights Permission Type Cycle

* test: Cover Resource ACL Audit Scenarios

* refactor: Audit Resource Permission Changes Before Writing

* test: Align Concurrent Revoke Audit Expectation

* fix: Skip Deleting ACL Entries That Were Not Audited

* fix: Read ACL Snapshots Through the Write Session

* fix: Never Block Revocations on Audit Outages and Normalize Principal Keys

* fix: Let an Explicit Public Disable Win Over a Public Update

* fix: Audit Duplicate ACL Rows as Combined Permissions

* fix: Audit Permission Changes Under One Target and Dedupe Revokes

* fix: Audit Capability Grants Before Writing Them

* feat: Serialize Resource Permission Updates With a Per-Resource Lease

* fix: Fence the Resource Lease Before Writes and Always Audit Grant Assignments

* fix: Renew the Resource Lease Before Each ACL Write Phase

* fix: Serialize System Grant Assignment and Revocation With a Per-Grant Lease

* fix: Keep Leases Alive During Audits, Bound Lock Index Wait, Canonicalize Audit Target IDs

* fix: Retry Deferred Revoke Audits Under the Lease and Audit Canonical Principal and Role IDs

* test: Expect Role IDs in Permission Audit Metadata

* fix: Read Mutation Snapshots From the Primary Outside Transactions

* fix: Flush Deferred Audits Before Cache Invalidation, Keep Renewals Monotonic, Keep Role Case in Grant Lease Keys

* refactor: Map the Busy Permission Error to Its Code in the API Package

* style: Sort Lease Imports and Type the Renewal Spec Read

* fix: Keep Success Audits After Commit, Report Downgrades as Revocations, Code Audit Outages

* refactor: Move the Audit Outage Code to Its Own Module

* fix: Translate Lock Failures, Lease Non-Transactional Sessions, and Code Grant Audit Outages

* fix: Sanitize Lease Renewal Failures at the Permission Boundary

* fix: Annotate the Unavailable Error Code for Isolated Declarations

* fix: Keep the Admin Route Fail-Closed Opt-In and Document Lease Clock Skew Safety

* fix: Translate Grant Lease Store Failures to a Retryable 503

* fix: Log Failed Grant Audit Writes With Safe Metadata Only

* test: Detect Raw Error Arguments in the Grant Audit Log Spec

* feat: Make Permission Lease Limits Configurable and Localize Retryable Sharing Failures

* fix: Bound Lock Index Readiness by the Lease Wait and Carry the Resource ID in Every Audit Entry

* test: Cover Lease Config Reaching the Leases and the Sharing Dialog Failure Messages

* fix: Load App Config on the Sharing and Grant Routes and Log Failed Lease Releases

* perf: Load App Config Only on the Grant Mutation Routes
2026-10-15 00:30:16 +02:00
Marco Beretta
ccae5b988d
📜 feat: Audit Admin Role, Group, Config and IdP Role Mutations (#17052)
* feat: Extend the Audit Action Vocabulary to Admin, Sync and Credential Mutations

* feat: Audit Admin Role, Group and Config Mutations

Role deletion now removes the role's grants, ACL entries and config
overrides before the role itself and returns ROLE_CLEANUP_FAILED instead
of success when any of them fails, so a recreated role cannot inherit
leftover privileges.

* feat: Audit IdP and Startup Role Changes

* fix: Keep the Startup Role Sync Running When Its Audit Verification Read Fails

* test: Cover Admin Role Audit Trail End to End

* fix: Audit a Remote Agent Role Sync Only After It Persists

* fix: Restore Role Dependents When Role Deletion Fails Midway

* fix: Restore Role Dependents Only When the Role Still Exists

* style: Sort Role Admin Imports

* fix: Keep the ACL Snapshot When Role Cleanup Fails Ambiguously

* test: Read Role Audit Entries From the Audit Log Route

* fix: Audit Only Real Membership Changes and Restore Deleted Role Config in Full

* fix: Restore Deleted Role Grants in Full and Sanitize Audit Write Logging

* fix: Audit and Report a Role Deletion Confirmed After a Lost Response

* fix: Never Restore Dependents Onto a Recreated Role

* fix: Snapshot Role Dependents Before Deleting and Detect Membership Changes Atomically

* fix: Restore From the Data a Cascade Step Actually Removed

* fix: Restore Only Config Overrides When a Role Delete Fails

* fix: Log Only Safe Metadata When the Startup Audit Read Fails

* fix: Audit Grants Removed by a Failed Role Delete

* fix: Audit Removed Grants on Unverifiable Role Delete Exits

* fix: Honor Fail-Closed Auditing in the Admin Config, Group and Role Routes

* fix: Bind Role Deletion to the Captured Id and Audit Only Atomic Role and Config Changes

* fix: Skip the Config Restore When the Targeted Role Is Gone

* fix: Audit Initial Group Members and Skip No-Op Role Description Updates

* fix: Record Role Deletion as Pending Before Cleanup and Append Its Outcome

* fix: Audit Reassigned Role Members and Return the Persisted Config After a Tracked Toggle

* fix: Keep a Committed Role Deletion Successful When Cascade Audit Writes Fail

* fix: Decide Group Update Audits From the Atomic Write and Sanitize Permission Update Logs

* fix: Audit Grants From a Snapshot, Fail on Member Lookup Errors and Chunk Member Audit Entries

* fix: Audit Group Deletion Members and Previous Role Removal, Abort on Grant Snapshot Failure

* fix: Invalidate Config Caches After Partial Secret Writes

* fix: Retry Failed Cascade Restores, Report Changed Group Fields and Audit Every Reassigned Member

* fix: Log Safe Metadata When Cache Invalidation Fails After a Partial Write

* fix: Keep Error Messages Out of the Audit Write Failure Log

* style: Sort Audit Log Spec Imports

* fix: Sanitize Admin Handler Error Logs and Dispatch System Audit Writes Without Waiting
2026-10-14 23:57:07 +02:00
Marco Beretta
20d2c0d43d
🧹 fix: Run the Full Erasure Cascade When an Admin Deletes a User (#17050)
* fix: Run the Erasure Cascade in Admin User Deletion

* refactor: Share Generation Stop Logic Between Self-Service and Admin Deletion

* refactor: Move Admin Erasure Dependency Assembly Into packages/api

* fix: Bind getCleanupJob When Waiting for Generation Persistence

* test: Pin the Deletion Fence Before Admin Generation Stop

* fix: Resolve the Erasure Target Config Fail Closed

* fix: Carry the Target Profile Into Admin Erasure Storage Cleanup

* fix: Keep the Erasure Target Contract Storage Neutral and Reuse the Loaded Identity

* fix: Refuse New Generations Once Account Deletion Has Begun

* fix: Enforce Account Deletion Admission Once in createJob and Narrow the Erasure File Type

* fix: Install the Admission Guard in Every Entrypoint and Purge Refused Jobs

* fix: Guarantee Refused Job Purge and Sanitize Admission Failures

* fix: Route Remote Admission Through createJob and Keep Refusal Cleanup Sanitized

* fix: Log Admission Cleanup Failures With Safe Error Metadata

* refactor: Check Account Deletion Before Creating a Generation and Drain Twice

* refactor: Order Admission Against the Deletion Marker Without a Grace Period

* fix: Sanitize Post-Commit Admission Retirement Failures

* fix: Wire Shared-Server MCP OAuth Cleanup Into Admin Erasure

* fix: Scope Admin Erasure Lookups to the Target Tenant and Await Shared-Link Cleanup

* fix: Resolve the Erasure Target Config Under the Target Tenant

* fix: Run Admin Erasure OAuth Fencing in the Target Tenant

* fix: Match the Split OAuth Invalidate and Disconnect Hooks in Admin Erasure

* refactor: Inject the Tenant Executor Into the Erasure Resolver
2026-10-14 23:54:32 +02:00
Marco Beretta
1994a4949d
🔏 fix: Encrypt Provider Keys With v3, Reject Plaintext in decryptV2 and Default to Strong Backup Codes (#17051)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
* fix: Write User Provider Keys With Random-IV v3 Format

* fix: Reject Undelimited Input in decryptV2 Instead of Returning It

* fix: Default 2FA Backup Codes to the Strong Format

* fix: Limit Admin Secret Preview to the Last Four Characters

* fix: Re-Derive Stored Secret Previews Under the Last-Four Policy

* test: Build Legacy Credential Fixtures With the Server Key

* fix: Load Request Config for User Key Writes and Mask Legacy Previews
2026-10-13 22:19:37 +02:00
Marco Beretta
0bc344fcb9
🔑 feat: Let Users Connect Their Own Account for Pull Request Lookups (#16989)
* feat: Let Users Connect Their Own Account for Pull Request Lookups

* fix: Refuse Cross-Tenant Pull Request Callbacks and Surface Key Failures

* fix: Keep Pull Request Grants Through Outages and Remove Them on Disconnect

A token endpoint server error is an outage even with an OAuth error body, so
it no longer deletes the grant, and a refresh outage reaches the lookup as
unavailable instead of falling back to no token. Disconnect removes every
pr_oauth row of the user, whatever app or host is configured now. A grant that
never expires extends its retention when used, and the flow, request and
retention limits are configurable. The connection routes are built in
packages/api, mounted behind the pre-auth tenant middleware in both servers,
and answer failures with stable codes.

* fix: Keep the Pull Request OAuth App Out of Overrides

A role, group, user or tenant override can no longer change or remove pullRequests.oauth, so the connect capability and the callback see the operator's app.

* fix: Read a User's Pull Request Token Only With the Feature On

* fix: Report a Malformed Stored Pull Request Grant as a Failure

* fix: Remove a User's Stored Tokens When an Administrator Deletes the User

* fix: Answer No Pull Request for a Conversation Without a Lane When the User Token Fails

* fix: Share One Grant Read Between Concurrent Callers and Report an Expired Grant Disconnected

* fix: Remove a Pull Request Grant Only on a Definite Refusal and Past Its Retention

* fix: Version Pull Request Grants and Coordinate Refresh, Refusal and Retention Through Them

* fix: Invalidate a Refused User Token and Read It Only for a Lane That Needs It

* fix: Finish a Pull Request Connection Under the User's Effective Config Behind the Login Limiter

* fix: Type the Pull Request User Token for Isolated Declarations and Settle Retention Before Answering

* fix: Bind a Pull Request Connection Flow to the Generation a Disconnect Replaces

* fix: Start a Batch's User Token Read Only Within Its Deadline

* fix: Fence Pull Request Token Reads Against a Disconnect and Report a Failed Retention Cleanup

* fix: Fence Connection Status Like Token Reads and Keep a Newer Read Joinable

* fix: Report a Pull Request Grant Without Its App and Name the Sign-In That Made It

* fix: Read a User's Unexpired Pull Request Grants Newest First

* fix: Mark the Connection Callback as an OAuth Navigation and Report a Failed Extension

* fix: Name the Pull Request Sign-In Flow on Its Landing Page

* fix: Remove a Pull Request Grant of a Replaced App When It Is Read

* fix: Keep a Replaced App's Grant Untouched and Extend Retention From Each Day's Use

* fix: Accept a Pull Request Grant Only Under the User's Current Connection Generation

* fix: Renew the Connection Generation With a Grant Still in Use

* fix: Restore the Previous Connection When a Sign-In Cannot Store Its Grant

* fix: Decide a Sign-In by What Was Stored When a Write Does Not Answer

* fix: Roll Back a Sign-In's Generation Along With a Grant It Must Remove

* fix: Refresh an Access Token a Lookup Was Refused With Before Ending the Connection

* fix: Undo a Sign-In Only for a Confirmed Fence, Not a Failed Check

* fix: Read a User Token Again When the Connection Changes Under the Read

* fix: Apply the Current Retention Again When Another Replica Changed the Grant First

* test: Answer Each Concurrent Sign-In by Its Own Code

* fix: Skip Retention Rewrites for Grants That Expire Without a Refresh Token

* fix: Retry a Token Rejection That Loses to a Concurrent Grant Write

* fix: Skip User Token Reads for Lanes Without a Commit and Reread Status After a Reconnect
2026-10-12 15:58:23 +02:00
Marco Beretta
4617302116 🔑 feat: Passkey Authentication with WebAuthn Sign-In and Management (#14715)
* feat: passkey authentication with WebAuthn sign-in and management

* refactor: tighten passkey account protections

Require password confirmation before adding or removing a passkey, and rate
limit those confirmations per user in their own namespace. Restrict passkey
enrollment and sign-in to local accounts, and re-evaluate account status once
the credential owner is resolved. Require user verification on the sign-in
ceremony. Invalidate outstanding access tokens on password reset. Tighten
redirect target validation and sanitize ceremony failure logs.

Adds coverage across the passkey controller, the WebAuthn helpers, the step-up
limiter, both JWT strategies, the redirect utility and the passkey settings UI.

* fix: localize passkey default name and respect router basename

Send a localized default credential label at enrollment so non-English
locales no longer persist the server's English fallback, and route the
passkey post-login and 2FA navigations through the deployment base path
so subdirectory installs stay inside the router.

Also mock syncStaticTools in the server boot specs; without it the
post-listen MCP initialization calls undefined and exits the worker.

* fix: harden passkey counter checks and reach stranded credentials

Make the signature counter write a compare-and-swap so two assertions
that verified against the same stored counter cannot both commit, and
reject the losing one; a cloned authenticator is otherwise accepted
inside that window. Counterless authenticators keep restamping at 0.

Keep the passkey panel reachable for identity-provider accounts, which
already get passwordless removal server-side but had no way in, while
hiding the add control they cannot use. Cascade passkey deletion from
the admin user-deletion path so credential IDs are not stranded.

* fix: reject stale 2FA temp tokens and surface passkey load failures

A temporary 2FA token proves only the first factor, so honouring one
minted before a password reset let anyone holding the second factor
survive account recovery. Apply the same credentials-changed cutoff the
JWT strategy uses.

Also distinguish a failed passkey query from an empty list; the error
state previously rendered as "no passkeys yet", which misreports what
is protecting the account.

* fix: fail closed when the passkey counter write cannot commit

Returning success on a storage error bypassed the compare-and-swap it
exists to enforce, so a cloned authenticator could be accepted while the
stored counter stayed stale. A database that cannot take this write
cannot mint the session either, so little is gained by failing open.
Counterless credentials carry no clone signal and keep their old path.

* fix: keep the passkey 2FA handoff inside the router

A full-page navigation to the 2FA screen remounted AuthContextProvider,
which silent-refreshes on mount; no refresh session exists yet at that
point, so it bounced back to login before the code could be entered.
Route through the router like the password flow does.

Also reveal the removal step-up field when the server refuses a
passwordless attempt, so an account migrated to an identity provider
without its old hash cleared can still clear its credentials; drop the
staged password when the dialog closes; and revoke passkeys in the CLI
password reset, which bypassed the HTTP controller's cleanup.

* fix: apply one email-verification policy across login methods

Accounts predating mandatory verification are marked verified as they
sign in when no email is configured, but only the password strategy did
it, so the same legacy account was admitted by password and refused by
passkey. Extract the rule into a shared helper and call it from both.

* fix: refuse a passkey sign-in when the ban check cannot complete

checkBan defaults next to a no-op, so calling it bare discarded any
internal failure and left req.banned unset, letting the request fall
through and be issued tokens. Capture the error the middleware chain
would have propagated and reject instead.

* fix: refine passkey management dialogs

* test: stabilize passkey tooltip assertions

* test: reconcile passkey specs with the canary reset and deletion flows

The credentialsChangedAt stamp now lives in commitPasswordReset, so its
assertion moves to the passwordReset unit spec. Restores the describe
blocks the rebase merged together.

* test: mock passkey routes in the cross-site login suite

* refactor: move passkey handlers and credential checks into packages/api

The passkey controller, the credentialsChangedAt predicate and the legacy
email-verification grandfathering now live in TypeScript with injected
dependencies; /api keeps only the wiring that builds the handlers.

* style: align passkey UI with Tailwind v4 order and design-rule lint

* fix: revoke earlier access tokens on a CLI password reset

passwordVersion is not a user schema field, so the write was dropped and
tokens issued before the reset kept verifying.

* test: cover passkey sign-in and credential revocation end to end

* fix: hide passkey sign-in when email login is disabled

The passkey endpoints already refuse under ALLOW_EMAIL_LOGIN=false, so
advertising the button only led to a 403.

* fix: let only one concurrent verification consume a passkey challenge

Without a native getDel the challenge was read then deleted, so two
concurrent verifications could both use it. The delete now arbitrates:
only the caller whose delete removed the key gets the challenge.

* test: run passkey ceremonies on the localhost relying party

WebAuthn refuses an IP address as the RP ID, and the harness serves the
app on 127.0.0.1. The reset scenario runs once per project, so the mock
harness lifts the per-IP reset request limit as it does for forks.

* fix: keep passkey management when email login is disabled

Gate only the sign-in button on emailLoginEnabled, since only the sign-in
routes sit behind that gate; list, rename and remove stay reachable.

* fix: build the passkey credential index before the first registration

With MONGO_AUTO_INDEX=false the unique credentialId index was never
created, so concurrent registrations could store the same credential.

* test: enable password reset in the mock harness and cover email-login-off passkeys

* fix: stop re-exporting the passkey cap from data-schemas

The bundler left every import of methods/passkey.ts unlinked while that
module re-exported MAX_PASSKEYS_PER_USER from librechat-data-provider, so
the index build and the counter-failure log threw ReferenceError at
runtime. Consumers now import the cap from librechat-data-provider.

* test: close nested settings dialogs before logging out in passkey scenarios

* fix: reject cross-site passkey assertions

The verify route accepted a form posted from another site, which let an
attacker sign a victim into the attacker's account. It now runs the same
requireSameOrigin guard as password login and the 2FA step.

* fix: end refresh sessions on a CLI password reset

A surviving refresh session could mint an access token issued after the
credentialsChangedAt stamp, so the CLI now deletes sessions like the web
reset does.

* fix: keep the passkey button locked when it supersedes autofill

Starting the modal ceremony aborts the pending autofill one, and the
autofill error path cleared the shared lock and busy state while the
modal ceremony was still running.

(cherry picked from commit 471abaa64d)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14715
2026-10-01 03:46:16 +00:00
Marco Beretta
d263da5271
🧾 fix: Validate Principal Config Overrides Against configSchema (#16433)
* fix: validate principal config overrides against configSchema

Reject admin config writes whose fields fail configSchema, and drop
invalid fields from stored overrides at merge time so the base value
from librechat.yaml survives.

* fix: require the merge key on partial custom endpoint overrides

* test: register a dedicated user for config override scenarios

* test: log in once per worker in config override scenarios

* fix: strip invalid overrides under dotted record keys

* fix: ignore a stored overrides document that is not an object

* fix: reject indexed patches into merged-by-name override arrays

* fix: match union options by shape, apply supplied-value refinements, reject paths past scalars

* refactor: validate config overrides on the merged result instead of the patch

* test: cover record refinements on written config fields

* fix: validate config field writes on top of the stored override

* fix: attribute id-addressed issues to their item and keep valid sections after repairs

* fix: judge field writes by what they change, over the principal's tombstoned base

* fix: validate whole-document config writes on the principal's tombstoned base

* fix: report config override issues as stable codes instead of schema messages

* fix: reject indexed writes into keyed arrays and keep filtered MCP servers out of validation state

* fix: judge each override layer only on the values it supplies

* fix: reject merged-by-key array items that are not objects

* fix: report repeated merge keys on the items the keyed merge discards

* fix: reject keys a union option drops when another option defines them
2026-09-28 16:22:22 -04:00
Danny Avila
f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00
Danny Avila
fb1322401d
🧪 test: Restore Two Rotted Integration Suites and Run Every One in CI (#15866)
* 🧪 test: Restore Two Rotted packages/api Integration Suites

MCPReinitRecovery.integration.test.ts mocked @librechat/data-schemas down to
its logger, so all eight tests threw once MCPServersRegistry started calling
scopedCacheKey (#15028). The mock now spreads the real module under the logger
override; ServerConfigsDB stays mocked, so the suite still never touches Mongo.

secrets.integration.spec.ts drove langfuse.secretKey through the generic admin
config API, which has stripped the langfuse section since #14108 moved it to
the dedicated Langfuse connection API. The generic-write matrix drops that row,
and a langfuse block covers the current contract against a real Config
collection: the stored secret stays encrypted at rest, generic reads redact it,
generic patch, tombstone and field-delete writes leave it untouched, and a full
base-config replacement carries it forward.

* 👷 ci: Run Every packages/api Integration Suite in CI

test:ci excludes every path containing "integration.", but the jobs meant to
run those suites selected by folder: the cache workflow only
src/{cache,cluster,mcp,stream}, the agents workflow only
src/agents/*.integration.spec.ts. Thirteen suites ran in no job, including
every *.integration.test.ts.

test:integration selects *.integration.(spec|test).ts anywhere under src and
replaces test:agents-integration in the (renamed) Integration Tests workflow.
The cache scripts also accept .test.ts, stream selects by suffix alone, and
core picks up src/middleware. Every suite test:ci excludes is now selected by
exactly one CI script.

* 🧪 test: Read the Tool Snapshot in the Concurrent Reinit Simulation

The concurrent reinitialize test copied reinitMCPServer but still called
fetchTools(), which returns [] when tools/list fails. Concurrent reinspections
of a recovered server can each succeed, and a later write replaces the app
connection an earlier request was already handed, so that request's tools/list
fails. The test failed in 9 of 25 iterations on Linux, and in the first CI run
that included the suite. Since #14686 production reads the snapshot and keeps
cached tools when it is incomplete; the simulation now does the same, and still
requires every complete snapshot to list both tools and at least one to complete.

A deterministic test pins the property that tolerance relies on, without
depending on the race: a newer config write replaces the held app connection,
whose snapshot is then incomplete, while the replacement lists the real tools.
2026-09-12 16:20:22 -04:00
Ravi Kumar L
000349681e
🩻 feat(insights): Scope Insights to Authorized Agents (#15549)
Some checks failed
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
* feat(insights): add agent-scoped access

* fix(insights): exclude unattributed assistant messages

* fix(sharing): key role insight toggles by principal

* fix(sharing): preserve insight grant snapshot

* fix(permissions): preserve insight bits atomically

* fix(permissions): guard insight audit rollback

* fix(insights): keep admin grant controls accessible

* fix(permissions): reconcile duplicate principal updates

* fix(insights): show automatic admin access

* fix(permissions): harden insights access updates

* fix(insights): simplify authorization and preserve client state
2026-09-04 20:16:58 -04:00
Danny Avila
84323d8c48
🛰️ fix: Repair BYOM Live Integration (#15613)
* fix: Repair BYOM live integration

* fix: Isolate mutable tool schemas
2026-09-04 15:38:04 -04:00
Danny Avila
164f146bdf
🛞 feat: Manage Personal Code Environment Lifecycles (#15369)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* feat: manage personal code environments

* fix: harden personal code environment lifecycle

* fix: satisfy backend lifecycle types

* fix: harden principal environment lifecycle and quotas

* fix: fence environment lifecycle races

* fix: reconcile code environment stack

* style: format code execution headers

* fix: reconcile code environment lifecycle

* fix: close self-service pairing gaps

* fix: close code environment reconciliation gaps

* fix: close code environment UI review findings

* fix: satisfy code environment type checks

* fix: route code priming to selected worker

* fix: harden code environment lifecycle fences

* fix: type lease renewal interval

* fix: harden environment lifecycle recovery

* fix: close lifecycle recovery races

* fix: type compensated agent timestamps

* fix: fence lifecycle cleanup by tenant and revision

* fix: scope environment revision fencing
2026-09-01 21:10:05 -04:00
Danny Avila
51edb2a660
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* style: format code execution headers

* fix: close self-service pairing gaps

* fix: satisfy code environment type checks
2026-09-01 18:46:33 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
a9ccac8656
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing

* fix: satisfy code environment type checks

* fix: secure code environment administration

* fix: isolate code pairing control plane

* fix: validate code pairing control responses

* fix: secure code pairing transport

* fix: validate code pairing wire format

* fix: harden pairing secret lookup
2026-08-30 17:12:20 -04:00
Ravi Kumar L
a70bcf66a4
🪢 feat: show Langfuse session link in shared chats (#15273)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: show Langfuse sessions in shared chats

* fix: wait for optional auth before loading shares

* fix: require Langfuse access for shared session links

* refactor: move shared Langfuse policy into API package

* fix: wrap shared chat actions on mobile

* fix: coordinate shared Langfuse link loading

* perf: batch shared Langfuse capability checks
2026-08-27 14:29:57 -04:00
Ravi Kumar L
56f0cde9a5
🩻 feat(langfuse): add tenant export telemetry (#15247) 2026-08-26 07:38:30 -04:00
Marco Beretta
0383030817
⚡ perf: Avoid Parallel Full and Paged Prompt Loading on Startup (#15031)
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-25 21:21:17 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Danny Avila
757fbebc37
🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored (#15067)
* 🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored

`discoverSkills` dropped any file whose path failed `isSafeRelativePath`
with no warning, no count, and no record. The skill published, reported
`succeeded`, and was missing files — invisible from the mirrored copy.

Real case: NVIDIA/skills has two such files (spaces in the filename), so
that repository syncs "cleanly" while silently losing them.

Matches what zip import already does — record the file, keep the skill —
and mirrors the existing `skippedSkills` shape into `skippedFiles` /
`skippedFileCount` on the sync status. Dropped files now make a run
`partial`, since a run reporting `succeeded` while dropping content is
the bug.

Only dropped *skills* can still make a run `failed`: a source that
published everything it found is a real mirror even if a file inside one
skill could not come along.

* 🔧 fix: Charge Dropped Files to the Skill That Published Them

Codex review: the up-front accounting counted a skill's unsupported files
whether or not that skill went on to publish. Two consequences — the status
described a skipped skill as published-but-incomplete, and enough failed
skills could consume the 20-entry sample and crowd out drops from skills
that actually published, which is the case the record exists for.

Now recorded at the two points a skill is counted as synced, matching what
`ISkillSyncSkippedFile` already documented ("the skill itself is live").

Also replaces `Array.prototype.at` in the new tests: it is outside this
package's lib target, so `tsc` rejected it even though jest ran it fine.
2026-08-21 03:36:27 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
a47ba7168f
🪢 feat: Custom Request Headers For Langfuse (#14945)
* ✨ feat: Custom Request Headers For Langfuse

Self-hosted Langfuse behind an authenticating proxy or gateway could not
be reached: every outbound Langfuse request hardcoded `Authorization` and
nothing else. Adds `langfuse.headers`, mirroring `endpoints.custom`
headers, and applies it to all four request surfaces — trace/media export
(via the agents run config), feedback scores, central project-identity
lookup, and admin credential verification.

Values resolve through the same pipeline as endpoint headers, so
`${ENV_VAR}` interpolation and header-safe encoding come along.
`extractEnvVariable` continues to refuse infrastructure secrets, so a
config cannot exfiltrate `MONGO_URI` through a header. A header whose
variable is unset is dropped with a one-time warning rather than sent as
a literal `${...}`, which a gateway would read as a wrong credential
instead of a missing one.

Headers merge beneath LibreChat's own `Authorization` on the REST
surfaces, matching `mergeHeaders`, so a custom header can never displace
the Langfuse credential.

These are deployment-level and documented as such: trace export batches
spans from every user through a single exporter, so unlike endpoint
headers they cannot carry per-user placeholders.

The central project-id cache key now includes the headers, so the
header-less module warm-up cannot record a proxy rejection against the
entry the request path later reads.

* 🔒 fix: Keep Langfuse Headers Out Of Stored Overrides

The generic admin config API accepts any field path inside an allowed
section, so `langfuse.headers` could be written through it. Unlike
`langfuse.secretKey`, headers are a map rather than one scalar path, so
the config secret registry cannot encrypt them at rest or mask them on
read — an admin-written map would sit in Mongo in plaintext and come
back in plaintext, widening exposure of what are gateway credentials.

Rejects them on both the dotted-patch and object-upsert routes, the same
way process-backed MCP servers are held to librechat.yaml. This is what
makes "deployment-level" true rather than merely documented.

* 🐛 fix: Wire Config Middleware And Header Collisions For Langfuse

Two codex review findings.

P1 — `api/server/routes/admin/langfuse.js` never mounted
`configMiddleware`, so `req.config` was undefined in production and
credential verification silently ran without the deployment's proxy
headers: exactly the deployments this feature targets could not save a
connection. The handler unit tests injected `config` into their mock
requests, so they stayed green. Mounts the middleware after the access
checks (unauthorized callers still short-circuit first) and adds
route-level tests that assert the handler actually receives a resolved
config — the composition root, not the component.

P2 — spreading custom headers under `Authorization` only replaced an
exact-case collision. A configured `authorization` survived alongside
the managed `Authorization` and fetch appends rather than replaces,
sending both credentials in one combined value. All four request sites
now use `mergeHeaders`, which already merges case-insensitively with the
override winning; tests cover the lower- and upper-case variants.

* 🔒 fix: Mask Langfuse Headers On Read And Harden Value Handling

Three codex round-2 findings.

P1 — the write guard blocked storing `langfuse.headers` in Mongo but did
nothing for the read path: `GET /api/admin/config/base` serves the
resolved AppConfig through `redactConfigSecrets`, which only knows
registered scalar secrets, so a yaml-configured gateway credential was
returned in full to any admin with Langfuse read access. Adds a
secret-map registry that masks values while keeping key names, so an
admin can still see which headers a deployment sets. Masking is safe
precisely because these are yaml-only — a masked read cannot be
round-tripped back over the real values. A malformed non-object value at
that path is dropped rather than serialized.

P2 — `mergeHeaders` indexes one spelling per lowercase name, so a config
holding both `authorization` and `AUTHORIZATION` had only one displaced;
the survivor was then appended by `Headers` into a combined value. Case
variants are now collapsed at resolution, before any consumer sees them.

P2 — `resolveHeaders` encodes only values it substitutes a user field
into, and no user is supplied here, so a literal or interpolated
character above U+00FF reached `Headers` unencoded and threw. Final
values now go through `encodeHeaderValue`; Latin-1 still passes verbatim.

* 🔒 fix: Keep Langfuse Header Credentials Out Of Logs And Validate Names

Three codex round-3 findings, plus a documented boundary for the fourth.

P1 — `loadCustomConfig` logs the parsed config at startup (`printConfig`
defaults true), so a literal gateway credential in `langfuse.headers` was
copied into application logs on every boot, undoing the masking the admin
read path had just gained. The printed copy now goes through
`redactConfigSecretMaps`, reusing the same registry. Scoped to map-valued
secrets so scalar-secret log behavior is unchanged; the live config keeps
its real values.

P2 — a nonempty but invalid field name (` X-Token`, `X Proxy Token`)
passed the emptiness check and then threw in the `Headers` constructor,
which would break export, verification, lookup, and feedback for the whole
deployment rather than that one header. Names are trimmed and validated
against the RFC 7230 token grammar, and dropped with a warning otherwise.

P2 — unresolved `${VAR}` detection tested the *resolved* value, so a
credential legitimately containing `${...}` was mistaken for a failed
substitution and dropped. Detection now inspects the configured text and
checks the referenced variables directly, which also drops references to
denylisted infrastructure secrets instead of forwarding them verbatim.

The fourth (fanout gateway forwards only `Authorization`, so a tenant
Langfuse behind its own proxy is not covered) is a real limitation in a
separate component. Documented on the schema field and in the example
config rather than left implied.

* 🔒 fix: Scope Langfuse Headers To Configured Origins

Three codex round-4 findings.

P1 — one header map was attached to every destination a run resolves to.
Under fanout that means a credential meant for an internal gateway was
also sent to the central destination, typically Langfuse Cloud: an
unrelated third-party origin. Headers are now attached only when the
destination's origin is one the deployment explicitly configured (a
self-hosted base URL, the fanout collector, or a tenant destination set
by env). The built-in `*.cloud.langfuse.com` defaults are excluded
precisely because nobody pointed at them. For trace export this also
means attaching after the export branch settles on a `baseUrl` rather
than before, since which destination wins depends on the branch.

P2 — `encodeHeaderValue` only encodes above U+00FF, so a newline, CR, or
NUL passed through and threw in `Headers`, breaking every request rather
than the one header. Values are trimmed (the common trailing-newline
case) then validated against the legal field-value bytes; an embedded
CRLF is a request-splitting attempt and is dropped, not stripped.

P2 — the write guard matched only the exact `headers` property, so
`{ langfuse: { "headers.X-Token": "..." } }` and root-level dotted
variants slipped through into the Mixed overrides document, where the
nested-map redactor never walks them and a later read returns them in
plaintext. All dotted spellings are now rejected.

* 🔒 fix: Bind Langfuse Headers To One Configured Origin

Four codex round-5 findings.

P1 — the round-4 allowlist still authorized every configured origin, so a
deployment with both a collector and an explicit central host sent the
same credential to both. `langfuse.headers` is one map with no way to say
which endpoint it authenticates to, so it is only unambiguous when the
deployment configures exactly one Langfuse origin. Iterating on which
origins to guess was the wrong axis; headers are now sent only when there
is a single configured origin and the destination is it, with a warning
when several make the intent unresolvable. That covers the self-hosted
case this feature exists for; multi-destination deployments need
per-destination headers the schema cannot yet express.

P1 — `fetch` defaults to following redirects, and Node strips
`Authorization` across origins but keeps arbitrary headers, so a redirect
off an allowed origin would hand the gateway credential to a host that
passed no check. Requests carrying custom headers now refuse redirects;
requests without them keep the default, so nothing changes for existing
deployments.

P2 — `extractEnvVariable`'s whole-string branch is anchored and greedy, so
`${CLIENT_ID}:${CLIENT_SECRET}` parsed as one variable name and the raw
template was sent as the credential. References are expanded here now, so
only literal values reach that path.

P2 — a valid token name is not necessarily usable: `Transfer-Encoding`
makes `fetch` throw and a fixed `Content-Length` misdescribes the body of
every other request sharing the map. Request-framing names are dropped.

* 🐛 fix: Expand Langfuse Header References Exactly Once

Codex round 6 (P2). After expanding `${VAR}` references myself I still
handed the result to `resolveHeaders`, which runs `extractEnvVariable`
over it again — so a credential containing `${PATH}`, or any other name
that happens to be set, was silently rewritten on export, verification,
lookup, and feedback. The round-3 test only used an *unset* embedded
name, which the second pass leaves alone, so it could not catch this.

Resolution no longer round-trips through `resolveHeaders`. The only part
still wanted from it was stripping `{{...}}` user placeholders, which is
now applied directly; expansion, encoding, and validation were already
local. Adds a test whose embedded variable is set, which fails against
the previous pipeline.

* 🐛 fix: Process Langfuse Header Templates Before Substitution

Codex round 7 (P2), the mirror of round 6. Having stopped re-expanding
the resolved credential, the placeholder strip was still running over it:
a token containing `{{LIBRECHAT_USER_ID}}` had that span deleted and
`abc{{...}}ghi` went out as `abcghi`.

Establishes the invariant the last two rounds were circling. Every
template operation — placeholder strip, unresolved-reference check,
expansion — now runs on the operator's configured text, and the
credential is substituted last and never touched again. Gateway
credentials are arbitrary strings, so none of their bytes are syntax.

Also moves the unresolved-reference check after the strip, so it no
longer reports a variable inside a `{{...}}` span that the strip removes.
2026-08-17 20:39:23 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Danny Avila
8f1f961212
🧱 refactor: Require Broad Config Management for Base Field Mutations (#14775) 2026-08-12 22:32:38 -04:00