Commit graph

69 commits

Author SHA1 Message Date
Danny Avila
8f13845da5
📿 fix: Inspect Stored File Locators Per Message (#15841)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: inspect stored file locators per message

* fix: declare traversal diagnostics for isolated declaration builds

* refactor: inject locator traversal reporting from application callers

* fix: propagate traversal diagnostics across protected entry points

* fix: complete middleware reporters and hydration message counts
2026-09-11 16:47:48 -04:00
Danny Avila
0b9dc209b0
🧫 fix: Verify Early Stream Recovery (#15676)
* feat: harden early buffer recovery observability

* fix: fence recovery to emitted frontier

* fix: close recovery observability races

* fix: require atomic recovery capabilities

* fix: fail incomplete recovery attachments closed

* fix: preserve legacy store contract

* fix: correlate failed overflow persistence

* fix: break recovery type dependency cycle

* fix: reconcile distributed recovery lifecycle

* fix: preserve content snapshot contract

* fix: harden cross-replica recovery lifecycle

* fix: lease recovery subscriber state

* fix: close recovery lifecycle edge cases

* fix: close recovery validation races

* test: align subscriber lease cleanup

* fix: harden subscriber lease lifecycle

* fix: finalize subscriber lease handoffs

* fix: reconcile recovery error replacements

* fix: preserve durable recovery frontier

* test: expect durable frontier gaps

* fix: scope durable snapshot protocol

* fix: trust durable terminal recovery payloads

* fix: close overflow recovery rollout races

* test: bound overflow race fixture memory

* fix: bound overflow marker refresh

* test: reduce recovery fixture memory

* fix: fence overflow recovery admission

* fix: close overflow recovery admission races

* fix: scope overflow admission fencing

* fix: preserve owner replay after remote admission

* test: await pre-admission append fencing

* test: reflect pre-admission durability
2026-09-06 22:28:17 -04:00
Danny Avila
c302ae6f35
🪁 fix: Honor Endpoint Drop Params in Parameter Panels (#15610)
* Remove parameters from UI when parameter is dropped through dropParams

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Addressed copilot and codex review suggestions

* 1. Normalize drop parameter names in client/src/components/SidePanel/Parameters/Panel.tsx
  Admin-configured dropParams for Azure/OpenAI-compatible custom endpoints use the effective backend field names (maxTokens, topP, frequencyPenalty, presencePenalty), but the panel's filter compared them against the UI's
  snake_case keys (max_tokens, top_p, etc.), so the controls stayed visible and silently discarded whatever the user set. Added a dropParamsBackendToUIKey map (in packages/data-provider/src/parameterSettings.ts) and normalize
  each dropParams entry through it before filtering.

  2. Apply the same drop-parameter filtering in client/src/components/SidePanel/Agents/ModelPanel.tsx
  The Agent builder's model panel built its parameter list independently and never consulted endpointsDropParamsMap at all, so it kept offering controls for Azure/custom providers whose values the backend drops. Reused the
  same resolution logic (including the new backend→UI key normalization) there, sourced from useGetStartupConfig.

  3. Remove any escapes from Azure fixtures in packages/api/src/app/config.test.ts
  Three tests built azure groupMap/modelGroupMap fixtures via ... as any as AppConfig['endpoints'], bypassing type checking for that shape. Replaced them with two typed helpers, createAzureGroupMap and createAzureConfig, built
  from the existing TAzureConfig/TAzureGroupMap/TAzureModelGroupMap types, so the fixtures are now fully type-checked with no any.

* HAL-1081 Deploy librechat open

  1. Gate the backend-name alias to OpenAI-compatible parameter sets (packages/data-provider/src/parameterSettings.ts)
  dropParamsBackendToUIKey had been applied unconditionally, rewriting a dropped topP to top_p even for a custom endpoint whose defaultParamsEndpoint is anthropic/google (or
  a native bedrock-* endpoint) — where topP is the UI key, so the rewrite broke hiding the control. Replaced the plain map with resolveDropParamsUIKeys(dropParams,
  endpointKey), which only aliases backend names for OpenAI-compatible endpoint keys (openAI, azureOpenAI, custom, openRouter) and passes native-provider keys through
  unchanged otherwise. Updated both Panel.tsx and ModelPanel.tsx to call it with overriddenEndpointKey, and added a resolveDropParamsUIKeys test suite in
  parameterSettings.spec.ts.

  2. Prune stale model_parameters when a control becomes hidden (client/src/components/SidePanel/Agents/ModelPanel.tsx)
  When a parameter an agent already had set gets added to its endpoint's dropParams, the control disappeared but the value stayed in model_parameters and was saved unchanged
  by composeAgentUpdatePayload — invisible to the user and silently reactivated if the endpoint later stopped dropping it. Added a useEffect, mirroring the conversation
  panel's existing pruning effect, that strips any model_parameters key no longer present in the currently-visible parameters list. Also fixed a pre-existing test regression
  (missing useGetStartupConfig mock) and added two tests covering pruning-on-drop and value retention when the control stays visible.

* fix: Preserve parameters for unknown agent providers

* fix: Sanitize agent parameters on submission

* fix: Preserve agent model overrides during pruning

---------

Co-authored-by: Marc Amick <MarcAmick@jhu.edu>
Co-authored-by: MarcAmick <5194465+MarcAmick@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-04 13:52:23 -04:00
Danny Avila
d346e5ab5e
🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short (#15532)
* 🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short

* fix: Await generation settlement on shutdown instead of publishing the drain

* fix: Spend the real shutdown budget and cover the initialization window

* fix: Keep the shutdown registration literal the index spec asserts on

* fix: Keep draining settlements registered while shutdown is waiting

* fix: Track provider executions at the begin/drain chokepoints instead of per caller

* fix: Open the provider tracker before the begin CAS and cap cluster workers to the primary deadline

* fix: Propagate the cluster deadline and release trackers on reconfigure and abandonment

* fix: Hold SIGTERM until the worker acknowledges the cluster deadline

* fix: Refuse provider starts after shutdown begins and harden the cluster deadline handoff

* test: Assert the multi-execution shutdown wait with both executions begun before shutdown
2026-09-03 07:11:38 -04:00
Danny Avila
1c7a172321
🕹️ feat: Configure BYOM Tool Permissions (#15504)
* feat: add configurable BYOM permissions

* test: cover BYOM permission settings live

* fix: harden BYOM permission settings

* perf: reuse principals for environment discovery

* chore: normalize BYOM settings style

* fix: Align BYOM Discovery and Admission

* fix: Harden BYOM Permission Boundaries

* fix: Align BYOM Skill Approval Routing

* fix: Preserve Lazy BYOM Approval Capabilities

* perf: Parallelize Cached Environment Reads
2026-09-02 10:38:00 -04:00
Danny Avila
51edb2a660
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* style: format code execution headers

* fix: close self-service pairing gaps

* fix: satisfy code environment type checks
2026-09-01 18:46:33 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
04a7577821
🧵 fix: Retry Shared Links After Message Persistence Gaps (#15306)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Retry shared links after message persistence gaps

* chore: Sort share test imports

* fix: Address share review feedback

* fix: Normalize hydrated share target
2026-08-28 08:05:52 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Danny Avila
de59da9636
🎟️ refactor: Require Credentials for Local Image Access by Default (#15252)
* 🔐 fix: Protect Local Image Access by Default

* 🔐 fix: Scope Image Authorization to Active Sessions

* 🧹 style: Format Image Authorization Checks

* 🛡️ fix: Harden Image Avatar Authorization

* 🧭 style: Sort Image Authorization Imports

* 🔐 fix: Close Image Authorization Review Gaps

* 🧭 fix: Normalize Stored Avatar Base Paths

* 🏢 fix: Resolve Tenant Assistant Image Policy

* 🛂 fix: Enforce Effective Image Access Policy

* 🧹 style: Flatten Assistant Config Selection

* 🧷 fix: Preserve Image Access Compatibility

* 🪪 fix: Make Image Sessions Revocable

* 🏗️ fix: Move Image Session Policy Into API
2026-08-27 09:55:27 -04:00
Danny Avila
44edcbe014
🧾 feat: Store Durable Event Actor Receipts (#15265)
* 🧾 feat: Store durable event actor receipts

* 📊 fix: Scope actor delivery metrics

* 🧱 fix: Close durable receipt recovery gaps

* fix: serialize actor action admission

* fix: retire terminal batch members

* fix: close actor terminal recovery races

* fix: reclaim lanes after terminal receipts

* fix: close actor receipt recovery gaps

* fix: preserve ambiguous legacy handling

* fix: recover legacy actor outcomes

* fix: admit leased actor actions

* fix: serialize actor admission recovery

* fix: close actor recovery races

* fix: token-fence actor admissions

* fix: preserve mixed-version actor fences

* fix: harden actor rollout and metrics

* fix: gate durable actor receipt rollout

* fix: enforce base actor receipt rollout

* fix: align experimental actor rollout
2026-08-27 06:00:43 -04:00
Danny Avila
52fa88fc54
📬 feat: Serialize Bound Actor Event Mailboxes (#15260)
* feat: serialize bound actor event mailboxes

* test: harden actor mailbox batch ordering

* fix: treat batch roots as mailbox authority

* test: pin mailbox replay semantics

* fix: preserve mailbox authority through settlement
2026-08-26 10:02:47 -04:00
Danny Avila
6360821470
🪢 feat: Coalesce Bound Agent Event Bursts (#15208)
* 🪢 feat: Coalesce Bound Agent Event Bursts

* fix: Restore API Package Build

* fix: Keep Trigger Batches Within One Window

* fix: intersect trigger batch windows

* fix: fence trigger batch lifecycle

* style: sort trigger service imports

* fix: make batch requeue crash-recoverable

* fix: fence concurrent batch requeues
2026-08-25 10:48:56 -04:00
Danny Avila
d641c398d5
🧳 fix: Port Subagent Control Receipt Writes to DocumentDB-Safe Operators (#15171)
* fix: harden subagent control receipt persistence

* fix: harden durable subagent control replay

* fix: await terminal control receipts on shutdown

* fix: close subagent control replay races

* test: type stale-owner transport fixture

* fix: quiesce durable subagent controls

* test: await subagent shutdown durability boundary

* fix: serialize durable subagent controls

* fix: fail shutdown on cleanup errors

* fix: report cancellable result availability accurately

* fix: fence subagent control receipt ownership

* fix: close distributed control receipt races

* test: type control receipt race fixture

* chore: require authoritative control receipts

* fix: close subagent control lifecycle races

* style: separate control reservation member

* test: harden subagent settlement wait

* fix: preserve authoritative control replay state
2026-08-24 20:05:39 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
6daafda86f
🧯 ci: Disarm the Graceful-Shutdown Force-Exit Timer on Test Reset (#14972)
* fix(shutdown): disarm the force-exit timer when test state is reset

`shutdown()` arms a 60s timer that calls `process.exit(1)` as a safety net for
drains that never finish. It is cleared only when the drain runs to completion,
so a drain that never settles — an HTTP server whose close callback never
fires, a task that hangs — leaves it armed.

`__resetShutdownStateForTests()` clears the task list, the shutting-down flag
and the server reference, but not that timer. A suite that triggers a signal
therefore leaves a live self-destruct behind: `unref` keeps it from holding the
process open, but it still fires if anything else keeps the process alive to
the timeout, and `process.exit(1)` then takes down whatever is running a minute
later. Jest reports that as a bare `process.exit called with "1"` with no
failing test, because the run dies before it can print a summary.

Track the timer at module scope, clear it from the reset helper, and clear it
from a `finally` so a throwing drain step cannot leak it either.

The new test fails without the reset change: it starts a drain that never
settles, resets state, advances 120s, and asserts the process was not exited.

* Scope the force-exit timer to the shutdown that armed it

Hoisting the timer to module scope introduced an aliasing hazard: a drain that
settles late runs its `finally` against whatever `forceExitTimer` points at by
then. If state was reset and a second shutdown armed its own timer in the
meantime, the late `finally` cleared the second shutdown's safety net instead
of its own.

Keep a local handle per shutdown, always clear that, and null the module
reference only while it still identifies the same timer.

The added test fails without this: it starts a drain whose close callback is
withheld, resets state, starts a second shutdown, then releases the first
callback and asserts the second net still force-exits.
2026-08-18 08:57:37 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
bce93f9c55
🎯 refactor: Infer Agents Endpoint for Model Specs Naming an Agent (#14889)
* 🎯 fix: Infer Agents Endpoint for Model Specs Naming an Agent

A model spec whose preset names an `agent_id` but omits `endpoint` was
unusable. `isModelSpecEndpointMatch` compares the request's endpoint to
`preset.endpoint` by strict equality, so an undefined endpoint matched
nothing and every request selecting the spec was rejected with a bare
`Model spec mismatch` — an error naming neither the spec nor the missing
field.

The selector had the matching half of the same gap: `handleSelectSpec`
read `preset.endpoint` directly, so it sent no endpoint and skipped
assigning `agent_id` to `model`. Fixing only the server would leave the
request malformed, so the resolution is shared between both.

- Add `resolveModelSpecEndpoint` to `librechat-data-provider`, inferring
  the agents endpoint when a preset names an agent and none is set. An
  explicit `endpoint` always wins, so configured specs are unaffected.
- Use it for endpoint matching and in the selector, so the menu and the
  request pipeline resolve a spec identically.

* 🔁 refactor: Materialize Inferred Spec Endpoints at Config Load

The review showed the lazy-resolver approach was unsound end to end:
config validation rejected an endpoint-less spec before the resolver
could ever run (`tPresetSchema` requires the `endpoint` key), and the
resolver was applied at 2 of ~8 read sites, leaving selection handlers,
startup presets, access filters, and provider-key reachability reading
the raw preset.

Materialize once at the boundary instead:

- `tModelSpecPresetSchema` now makes `endpoint` optional (`nullish`).
  This is barely a widening — `endpoint: null` already validated — and
  only for model-spec presets; `tPresetSchema` is untouched.
- `materializeModelSpecEndpoints` writes each spec's resolved endpoint
  back onto its preset. `createAppConfigService` applies it at both
  effective-config assembly points — YAML base load and DB-override
  merge — so admin-panel specs stored in override documents are covered.
  Identity-preserving, so cached configs see no new references when
  nothing needs filling in.
- Every consumer now reads complete specs; the client's lazy resolve in
  `handleSelectSpec` is reverted to a raw read. `getModelSpecPreset` and
  the two hand-rolled preset constructions resolve the endpoint
  explicitly, which the narrowed preset type now enforces at compile
  time for any `TPreset`-shaped destination.
- `isModelSpecEndpointMatch` keeps the resolver as request-time defense.

* 🩹 fix: Materialize Spec Endpoints Before the YAML Missing-Endpoint Guard

`processModelSpecs` warns and skips any spec whose preset lacks an
endpoint, and it runs inside `loadBaseConfig` — so the previous commit's
materialization received a YAML list from which the inferable spec had
already been dropped. Only DB-override specs (merged after the guard)
actually benefited.

- Materialize at the entry of `processModelSpecs`, so inference happens
  before the guard and YAML agent specs survive it. The guard keeps
  skipping genuinely endpoint-less specs. The `createAppConfigService`
  calls stay: the base-path one guards alternate `loadBaseConfig`
  implementations, the merged-path one covers override documents, and
  both are identity-preserving no-ops when specs are already complete.
- Constrain the widened schema: omitting `endpoint` is only legal when
  the preset names an `agent_id`. A preset with neither validated as a
  hard error before the key became optional, and silently accepting it
  would trade that startup-time error for a dead spec. An explicit
  `endpoint: null` (valid before this PR) keeps validating.

* 🩹 fix: Infer Only From Non-Empty Agent IDs, Never Over Explicit Null

Two edge cases in the inference contract:

- `agent_id: ''` (what a form-backed writer persists for an untouched
  field) passed the nullish checks, validating and materializing a spec
  that names no agent. Both the refinement and the resolver now require
  a non-empty id, so such config fails validation loudly instead of
  producing a selectable spec that cannot work.
- `endpoint: null` alongside an `agent_id` was treated as inferable,
  silently activating a spec that validated — and was skipped — before
  this PR. An explicit null is a statement, not an omission: the
  resolver now infers only when the key is absent, preserving prior
  behavior for previously valid configs.
2026-08-16 11:04:52 -04:00
Danny Avila
1596df724a
🫆 chore: Remove Published Credential Defaults (#14680) 2026-08-07 07:25:05 -04:00
Danny Avila
b807292997
📉 perf: Bound Early Event Buffering for Detached Generations (#14612)
Some checks failed
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Has been cancelled
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
* 📉 perf: Bound Early Event Buffering for Detached Generations

A generation streaming with no attached subscriber re-entered buffering
mode on every disconnect and retained each emitted event in
earlyEventBuffer for its remaining duration. A single 26-minute detached
run (~58,800 tool-argument deltas) grew the heap past 2 GiB with GC cost
climbing alongside it, while client reconnects always resume from
durable state and discard that local buffer anyway.

- Close the early buffer after the first attachment drains it in Redis
  mode; the durable chunk log and pub/sub own recovery from then on,
  matching how cross-replica subscribers already attach.
- Enforce hard bounds (5,000 events / 8 MB estimated) in both modes; on
  overflow the buffer is discarded and closed, with recovery falling back
  to the durable chunk log (Redis) or resume snapshot (in-memory).
- Add a generation_stream_early_buffer_overflows_total counter and
  earlyBufferedEvents/Bytes gauges on getRuntimeStats() for visibility.
- Add incident-shaped regression tests and update specs that pinned the
  old post-disconnect re-buffering contract.

* fix: redirect post-overflow first attachments to resume recovery

A buffer discarded by the overflow guard left the initial non-resume
SSE attachment with nothing to replay, silently omitting pre-attach
output until the final event. Track the overflow on the runtime and
close such attachments with the existing reconnect signal instead: the
client already re-attaches with resume=true on transport failure and
its sync frame reconstructs the discarded output from durable/snapshot
state. Adds no per-event work; the check is one boolean per attachment.

* fix: enforce buffer bounds when restoring canceled resume captures

Captured emissions restored by a resume canceled before activation
bypassed the early-buffer hard cap, so one oversized restoration could
persist past the limits with no later emission to trip the guard.
Restoration now applies the same overflow-and-close behavior through a
shared helper, and the restore-cap spec fails before this change
(5 events / ~10MB retained) and passes after.

* chore: add Redis management scripts and update package.json for Redis commands
2026-08-03 19:03:37 -04:00
Danny Avila
becfc5a373
🗝️ feat: Custom Endpoint API Key Encryption via Array Secret Registry (#14510)
Extend the admin-config secret registry (#14509) with array-item secrets,
registering endpoints.custom[*].apiKey: encryptV3 at rest with apiKeyPreview
companions, redaction on admin reads (plaintext-legacy included, with
verbatim-name omit-to-keep preservation, duplicate-identity skip, and
legacy-plaintext self-healing), passthrough for user_provided/${ENV} refs,
strict-payload runtime decryption at getCustomEndpointConfig, the custom
model fetch, and the provider fallback, and rejection of named/indexed/
positional writes beneath the protected array path.
2026-07-29 15:37:46 -04:00
Danny Avila
324584552c
⏱️ feat: Configurable HTTP Server Timeouts (#14481)
* http server config added

* Fix TypeScript compatibility by accepting NodeJS.ProcessEnv directly when applying optional HTTP server timeout configuration.

* fix(api): configure HTTP server timeouts for clustered workers

* 🕰️ fix: Warn When HTTP Timeouts Are Not Enforced

Codex review of the rebased contributor work surfaced two ways these settings
silently do nothing. Both reproduce, and neither was reported to the operator.

Bun accepts the four property assignments and reflects them back, but does not
enforce them: with keepAliveTimeout=100 and buffer=1000, Bun 1.3.13 held a
keep-alive connection past 3s where Node 24 closed it at 1101ms. Since `b:api`
runs the server under Bun, the existing info log confirmed a configuration that
was not in effect. Warn instead.

Node sweeps header/request timeouts on `connectionsCheckingInterval`, a
createServer option that `app.listen()` leaves at 30s, so sub-30s values round
up to it: headersTimeout=2000 returned 408 at 30004ms by default versus 2010ms
with a 250ms interval. Warn on values below the sweep interval rather than
restructure server construction, since every documented value and both Node
defaults already sit well above it. keepAliveTimeout is socket-driven and stays
exact, so it is excluded.

Both caveats documented in .env.example.

* 🩹 fix: Inject Runtime Versions Instead of Mutating `process.versions`

The spec deleted `process.versions.bun` to reset between cases, which failed
typecheck with TS2790: `@types/bun` is a packages/api dependency and augments
NodeJS.ProcessVersions with a required `bun: string`, so the property is not
optional and cannot be deleted. Assigning undefined would fail for the same
reason.

That augmentation also made the production check dishonest: TypeScript saw
`process.versions.bun` as always a string, so `!= null` read as a no-op branch
even though it is correct at runtime under Node.

Both resolved by taking runtime versions as a third injectable parameter,
matching the existing `environment` parameter. Callers in api/server are
unchanged, the narrow `{ bun?: string }` type restores honest narrowing, and
the tests no longer mutate global state, so they assert the same behavior
whether the suite runs under Node or `bun jest`.

* 📏 fix: Stop Claiming a Ceiling on Sweep-Delayed Timeouts

The warning added in 9adc3eb1c said the effective timeout is "up to 30000ms",
which promises a bound that does not hold. Node detects header/request expiry
only on the next connection sweep, so the delay is relative to the deadline
rather than capped by the interval: measured against the default 30s sweep, a
2000ms headersTimeout closed at 30004ms, 15x the configured value, and cases
where the timeout is near the interval did not fire within a 9s window at all.

Reworded to state the mechanism without asserting a ceiling, and to point
operators at values of 30000ms or above for predictable enforcement. Same
correction applied to the .env.example note, which claimed short values "round
up" to the interval.

* ⏳ fix: Clamp Headers Timeout to the Request Timeout

Setting only HTTP_REQUEST_TIMEOUT_MS below the 60s headers default left
headersTimeout > requestTimeout, a pairing createServer rejects outright with
ERR_OUT_OF_RANGE. Assigning the properties after construction skips that
validation, and the mismatch silently defeats the request timeout for a stalled
body: with requestTimeout=4000 and headersTimeout at its 60000 default, a client
that completed its headers and then stopped mid-body was still connected after
12s. Clamping headersTimeout to 4000 closes the same connection at 4017ms.

An earlier round dismissed this after testing partial *headers*, where
requestTimeout does evict on time. The gap only appears once headers are
complete and the body stalls, which is the case these timeouts exist to bound.

Mirrors Node's own rule rather than its constructor default: zero on either side
means disabled and is left alone, and an explicitly configured
HTTP_HEADERS_TIMEOUT_MS that conflicts is warned about before being clamped
instead of failing startup over a config typo.

---------

Co-authored-by: Peter Rothlaender <peter.rothlaender@ginkgo.com>
2026-07-28 09:10:17 -04:00
Danny Avila
73699b5c25
⚡ perf: Reduce Agent Chat Startup Latency (#14423)
* perf: reduce agent chat startup latency

* test: align Redis stream readiness assertions

* perf: overlap remaining agent startup work

* perf: persist initial agent job metadata atomically

* test: add agent startup latency benchmark

* fix: harden resumable agent stream lifecycle

* fix: isolate replacement stream lifecycles

* fix: preserve terminal stream epochs
2026-07-25 07:58:20 -04:00
Ravi Kumar L
6f21be73a9
feat: add low-noise Redis observability (#14309)
* feat(api): add low-noise Redis observability

* fix(api): preserve Redis proxy constructor
2026-07-16 10:22:03 -04:00
Ravi Kumar L
44d1275f36
⚙️ perf: reduce first-load MongoDB round trips (#14101)
* perf(api): reduce first-load database round trips

* docs: move agent guidance to claude docs

* refactor(api): move message validation into api package

* fix(api): narrow active generation job lookup

* fix(api): preserve omitted source identity
2026-07-06 09:36:34 -04:00
Danny Avila
397ddc5366
🧠 feat: Add Memory as an Agent Capability with Inline Tools and Ephemeral Badge (#13869)
* 🧠 feat: Memory Agent Capability with Inline Tools and Ephemeral Badge

Add `AgentCapabilities.memory`, which expands into the inline set_memory/delete_memory tool pair (mirroring the execute_code expansion via registerMemoryTools) when a run-level memoryAvailable gate holds: capability enabled, memory configured, MEMORIES.USE permission, and personalization not opted out. Surfaces the memory artifact as an attachment in the agents tool-end callback.

Adds the ephemeral path (TEphemeralAgent.memory, load/added agent tool injection), a fully-gated memory badge plus tools-dropdown entry, the agent-builder Memory toggle with form round-trip, and a mock e2e test asserting the badge reaches the request payload. Additive to and independent of the existing post-turn memory extraction agent.

* 🩹 fix: Address Codex review on memory capability (gating, validKeys, usage guard)

- Strip the memory capability from the served agents capabilities when memory is not configured/enabled, so the badge, tools dropdown, agent-builder toggle, and backend capability gate stay consistent instead of exposing an inert toggle on default installs (where MEMORIES.USE defaults true).
- Surface configured memory.validKeys in the inline tool definitions so the model is told the allowed keys up front, matching the runtime createMemoryTool schema.
- Append a strict explicit-request usage guard to the agent instructions when inline memory tools are registered, preserving the memory-agent's privacy behavior.
- Add AppService tests covering memory-capability stripping.

* ✅ test: Update AppService capability snapshots for memory strip

AppService now strips the memory capability from the served agents defaults when no memory block is configured; update the spec's expected capability lists to defaultAgentCapabilitiesWithoutMemory for the no-memory-config cases.

* 🛡️ fix: Address Codex re-review on memory capability (round 2)

- Strip the memory capability from the FINAL served agents config, not just defaults; loadEndpoints reparses any endpoints.agents block, so memory was still exposed in that common shape (packages/data-schemas/src/app/service.ts) + regression test.
- Re-check the full memory gate (config, opt-out, MEMORIES.USE) inside handleTools before constructing set_memory/delete_memory, so an unsolicited tool call from a model/custom endpoint can't bypass the runtime gates (api/app/clients/tools/util/handleTools.js).
- Restore the persisted memory toggle for model-spec conversations via applyModelSpecEphemeralAgent (client/src/utils/endpoints.ts).
- Clear LAST_MEMORY_TOGGLE_ on logout and clear-all-chats so a stale memory preference can't leak across users on a shared browser (client/src/utils/localStorage.ts).

* 🧠 fix: Address Codex re-review on memory capability (round 3)

- Serialize set_memory writes and advance a running token total inside createMemoryTool, so parallel batched calls in one event-driven turn can't each pass the limit check against a stale total and collectively exceed memory.tokenLimit (packages/api/src/agents/memory.ts) + tests.
- Inject the keyed memory context (withKeys) instead of withoutKeys when the running agent has the inline memory capability, so delete_memory has a visible key to target (api/server/controllers/agents/client.js).

* 🔐 fix: Address Codex re-review on memory capability (round 4)

- Detect inline memory by tool NAME (set_memory/delete_memory) across an initialized agent's tools + toolDefinitions, since the 'memory' marker is expanded at init and the prior string check never matched; inject the keyed memory context for any primary OR sub-agent that carries the inline memory tools (api/server/controllers/agents/client.js).
- Enforce memory WRITE permissions in the inline tool gate: set_memory requires CREATE+UPDATE and delete_memory requires UPDATE (matching the REST memory routes), so a USE-only role can't mutate/delete memories via agent tool calls (api/app/clients/tools/util/handleTools.js).

* 🔒 fix: Address Codex re-review on memory capability (round 5)

- Gate inline memory registration (memoryAvailable) on the memory WRITE permissions (USE+CREATE+UPDATE), so a read-only-memory role no longer has set_memory/delete_memory shown to the model only for the runtime loader to refuse them (api/server/services/Endpoints/agents/initialize.js).
- Enforce the per-agent memory opt-in at execution: handleTools now refuses to construct set_memory/delete_memory unless the agent actually declared them (toolDefinitions/tools), blocking hallucinated/undeclared memory tool calls from mutating memory.
- Fail closed when getFormattedMemories errors with a configured tokenLimit, instead of writing as if storage were empty and bypassing the cap (api/app/clients/tools/util/handleTools.js).

* 🩹 fix: Address Codex re-review on memory capability (round 6)

- Fix a P1 regression from the prior round: the execution-context agent keeps the raw 'memory' capability marker (not the expanded set_memory/delete_memory names), so the opt-in check now matches the marker. This restores memory writes/deletes AND avoids hijacking an MCP tool that merely shares the set_memory/delete_memory name (api/app/clients/tools/util/handleTools.js).
- Count repeated set_memory writes to the same key as replacements, not additions, against tokenLimit — set_memory upserts, so a same-key rewrite swaps its prior token contribution instead of double-counting (packages/api/src/agents/memory.ts) + test.
- Gate the memory badge, tools dropdown, and agent-builder toggle on the full memory write permissions (USE+CREATE+UPDATE) via a shared useHasMemoryAccess hook, so a read-only-memory role no longer sees an enabled Memory control the backend would refuse to wire up.

* 🧷 fix: Address Codex re-review on memory capability (round 7)

- Recognize inline memory across both execution-context agent shapes: initializeAgent now sets a LibreChat-only memoryToolsRegistered flag on the InitializedAgent, and the opt-in/detection checks accept that flag OR the raw 'memory' marker. Fixes memory failing for processAddedConvo agents (which store the initialized config, marker already expanded) while staying MCP-name-collision-safe (api/app/clients/tools/util/handleTools.js, packages/api/src/agents/initialize.ts, api/server/controllers/agents/client.js).
- Scope keyed memory context to memory-enabled agents only: useMemory now returns both keyed and unkeyed contexts, and buildMessages injects the keyed one (memory keys + token metadata) only to agents that can call delete_memory, while the primary/post-turn path keeps the unkeyed values — so a primary without memory tools no longer sees memory keys it doesn't need.

* 🔏 fix: Address Codex re-review on memory capability (round 8)

- Enforce memory size limits on inline writes: createMemoryTool now rejects keys over 1000 chars and values over memory.charLimit, matching the REST memory routes, so an inline-memory agent can't persist blobs the memory UI/API would reject (packages/api/src/agents/memory.ts, api/app/clients/tools/util/handleTools.js) + test.
- Recheck the agents 'memory' endpoint capability at execution time, so a stale/hallucinated set_memory/delete_memory call can't mutate memory after an admin removes the capability while the agent document still carries the marker (api/app/clients/tools/util/handleTools.js).

* ♻️ refactor: Move inline-memory backend logic into packages/api + share memory load

Workspace boundary: the inline-memory gating/detection logic that had crept into /api now lives in packages/api/src/agents/memory.ts (TS), with /api kept as thin wrappers.

- Add agentHasInlineMemoryTools, isMemoryToolAllowed, and buildInlineMemoryTool to packages/api; handleTools.js now calls buildInlineMemoryTool instead of constructing/gating the tools inline, and client.js imports agentHasInlineMemoryTools instead of redefining it.
- Optimize repeated memory loads: getRequestMemories memoizes getFormattedMemories per request (WeakMap keyed by req), so the run's memory-context load and every memory-enabled agent's set_memory token-usage load share a single DB fetch instead of one per agent.

* 🧠 fix: Invalidate request memory cache after inline writes

Inline set_memory/delete_memory now invalidate the request-scoped
getFormattedMemories cache on a successful write, so a later tool round
in the same response is seeded with the post-write usage total instead
of the stale pre-write one (multi-round writes no longer collectively
exceed tokenLimit, and a set after a delete is not over-counted). The
within-round sharing across multiple memory-enabled agents is preserved.

* 🧠 fix: Persist memory capability on saved agents; honor registration flag

- Add Tools.memory to the v1 systemTools allowlist so filterAuthorizedTools
  no longer silently drops the memory marker when an agent with the Memory
  capability is created/updated/duplicated through the builder (previously
  the capability only worked for ephemeral chats, not persisted agents).
- agentHasInlineMemoryTools now honors an explicit memoryToolsRegistered
  boolean before falling back to the raw `memory` marker, so an initialized
  config whose registration was denied (memoryAvailable false) is not given
  keyed memory context just because the marker survives in tools.

* 🧩 fix: Bring memory tool to parity with other ephemeral tools

- Add `memory` to the model-spec schema/type and honor `modelSpec.memory`
  in both ephemeral paths (load.ts, added.ts) and the frontend spec
  application, so admins can pre-enable Memory from a model spec exactly
  like webSearch/fileSearch/executeCode.
- Add LAST_MEMORY_TOGGLE_ to the timestamped-storage cleanup list so stale
  per-conversation memory toggles are purged on startup like the others.
- Hide the agent-builder Memory toggle for users who disabled memory in
  personalization (memories === false), mirroring the chat badge's opt-out
  gate, so the setting isn't shown as inert/misleading.

* ✅ test: Cover memory in applyModelSpecEphemeralAgent spec defaults

Update the exact-object assertions to include the new `memory` field and
add positive coverage that `modelSpec.memory` maps to the ephemeral
agent's `memory` flag. Fixes the shard 2/4 failure from 672a03b05.
2026-06-24 17:14:13 -04:00
Ravi Kumar L
fbc990f684
📈 fix: Isolate RUM Telemetry Proxy Auth from App Auth (#13765)
* fix(rum): isolate telemetry proxy auth

* feat(rum): track proxy error metrics

* refactor(rum): simplify proxy auth strategy flow

* test(rum): clarify proxy success metric assertion

* test(metrics): use typed supertest import

* test(metrics): add local supertest types

* test(metrics): keep supertest types local

* test(metrics): use official supertest types

* fix(rum): log proxy auth strategy errors

* fix(rum): classify proxy auth errors in metrics

* style(rum): sort telemetry metric imports

* ci: mention import sort check command

* ci: show targeted import sort example
2026-06-15 12:49:44 -04:00
Danny Avila
6bc75d24c8
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown

Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the
@librechat/api package build, mirroring the merged data-schemas migration.

- Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps,
  bundle first-party `~/` + relative imports)
- Annotate exports for isolatedDeclarations (codefix-driven). Collapse the
  tokens.ts model->token maps to Record<string, Record<string, number>> and
  switch validation.ts's runtime `files` field from z.any() to z.unknown()
  so no explicit `any` is introduced
- Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output
- Add src/telemetry.ts entry shim so the two index.ts entries don't collide
  in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts})
- Delete rollup.config.js

Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass,
both entries load via require(), legacy /api consumes them unchanged.

* 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys

The build-api cache keys hashed `packages/api/server-rollup.config.js`,
which never existed (api used `rollup.config.js`, now removed) — a copy-paste
artifact from the data-provider key that matched no file. Replace it with the
new `packages/api/tsdown.config.mjs` so edits to the build config (entry,
format, externals) bust the api build cache, matching the data-schemas key.
2026-06-08 10:54:48 -04:00
Danny Avila
192703e041
⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations (#13578)
* ⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations

Replace Rollup with tsdown (rolldown + oxc) for @librechat/data-schemas. With the source made isolatedDeclarations-clean, oxc emits .d.ts without tsc, dropping the package build from ~5.8s to ~0.8s (~7x).

- Annotate exported model/method factories for isolatedDeclarations (TypeScript's fixMissingTypeAnnotationOnExports codefix plus hand-authored interfaces); type the ~44 mongoose `any`s and add an explicit PromptMethods interface (previously its declaration was silently dropped by the Rollup build).
- Repoint package.json exports/main/module/types to tsdown output; drop rollup config.
- Config lives in tsdown.config.mjs (native ESM) so CI without a TS-config loader can build it; bundle `dotenv` so the package stays self-contained for its env-loading side effect.
- Fix a latent token `metadata` mismatch the accurate types surfaced: widen TokenCreate/UpdateData inputs to accept plain objects, flatten OAuthMetadata at the api boundary.
- Update mongoMeili/aclEntry specs to the precise model types; drop redundant terser minification from data-provider's library build.

All data-schemas tests pass; api builds clean against the new output.

* 🔧 chore: Hash tsdown.config.mjs in data-schemas CI build-cache keys

The data-schemas build switched from rollup to tsdown, but the build-data-schemas / build-api cache keys in backend-review, config-review, and playwright-mock still hashed the (now-deleted) rollup.config.js. Hash tsdown.config.mjs instead so a config-only change invalidates the cached dist/api builds. (Found by Codex review.)

* 🔧 chore: Replace deprecated tsdown `external` with `deps.neverBundle`

tsdown 0.22 deprecated the top-level `external` option in favor of `deps.neverBundle`. Migrate the data-schemas config and set `deps.onlyBundle: false` to silence the (intentional) dotenv bundling hint. Build output and externalization are unchanged — dotenv bundled, all peers external.
2026-06-07 21:40:48 -04:00
Atef Bellaaj
86fe79c37d
🔗 feat: Add Granular Access Control to Shared Links via ACL System (#13051)
* feat: Add granular access control to shared links via ACL system

* fix(shared-links): preserve isPublic on failed migration grants

Transient ACL failures during auto-migration permanently stranded
links — $unset ran unconditionally, removing the legacy flag that
triggers retry. Now only $unset isPublic after all grants succeed.

* fix(config): skip isPublic unset for failed ACL grants

Bulk migration unconditionally removed isPublic from all links,
even those whose ACL writes failed. Failed links then lost the
legacy marker needed for auto-migration retry. Now tracks failed
link IDs per-batch and excludes them from the $unset step.

Also adds sharedLink to AccessRole resourceType schema enum —
was missing, only worked because seedDefaultRoles uses
findOneAndUpdate which bypasses validation.

* ci(config): add jest config and PR workflow for migration tests

config/__tests__/ specs depend on api/jest.config.js module
mappings but had no dedicated runner. Adds config/jest.config.js
extending api config with absolutized paths, npm test:config
script, and a GitHub Actions workflow triggered by changes to
config/, api/models/, api/db/, or packages/ ACL code.

* fix(permissions): honor boolean sharedLinks config

SHARED_LINKS has no USE permission, so boolean config produced
an empty update payload — gate conditions only matched object
form, making `sharedLinks: false` a no-op on existing perms.

* fix(share): resolve role before creating shared link

Role lookup between create and grant left an orphaned link
without ACL entries if getRoleByName threw — retry then hit "Share already exists" with no recovery path.

* fix: Restore Public ACL Access Checks

* fix: Type Public ACL Lookup

* fix: Preserve Private Legacy Shared Links

* chore: Promote Shared Link Permission Migration

* fix: Address Shared Link Review Findings

* fix: Repair Shared Link CI Follow-Up

* fix: Narrow Shared Link Mongoose Test Mock

* fix: Address Shared Link Review Follow-Ups

* fix: Close Shared Link Review Gaps

* fix: Guard Missing Shared Link Permission Backfill

* test: Add Shared Link Mock E2E

* test: Stabilize Shared Link Mock E2E

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-03 14:17:17 -04:00
jcbartle
268f095c1a
🔒 feat: Add On-Behalf-Of (OBO) token exchange support for MCP Servers (#13429)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* Add OBO (On-Behalf-Of) token exchange support for MCP server connections

Enables transparent authentication to Entra ID-backed MCP servers using the logged-in user's federated token via the OAuth 2.0 jwt-bearer grant. Configured via obo.scopes in librechat.yaml server config.

- Extract generic OboTokenService from GraphTokenService (jwt-bearer grant + cache)
- Refactor GraphTokenService to thin wrapper delegating to OboTokenService
- Add obo schema field to BaseOptionsSchema in data-provider
- Add resolveOboToken in packages/api/src/mcp/oauth/obo.ts (validates federated token, calls resolver, returns MCPOAuthTokens)
- Wire oboTokenResolver through MCPConnectionFactory, MCPManager, UserConnectionManager
- OBO tokens injected via request headers (not OAuth transport), refreshed on each tool call
- Explicit error on OBO failure (no fallthrough to standard OAuth redirect)
- Add unit tests for both resolveOboToken (9 tests) and exchangeOboToken (14 tests)

* Add OBO authentication option to MCP server UI configuration

  Enable users to configure On-Behalf-Of (OBO) token exchange for MCP servers created via the UI (MongoDB-stored), in addition to the existing YAML-based configuration.

  - Add "On-Behalf-Of (OBO)" radio option to MCP server auth section with scopes input field
  - Remove obo from omitServerManagedFields so the field passes UI schema validation
  - Add OBO to AuthTypeEnum, obo_scopes to AuthConfig, and OBO handling in form defaults and submission
  - Add .min(1) validation on obo.scopes to reject empty strings
  - Add English localization keys: com_ui_obo, com_ui_obo_scopes, com_ui_obo_scopes_description
  - Add 5 schema validation tests for OBO field acceptance, transport compatibility, and edge cases

* 🧊 fix: Add obo to safe properties in redactServerSecrets. Fixes the OBO configuration not showing up in the MCP UI after app restart

* Address linter errors

* 🧊 fix: fail closed on OBO refresh errors and retry transient token exchange failures

- stop tool calls from falling back to stale Authorization headers when per-call OBO refresh fails
- add one-time retry for transient Entra OBO exchange failures (network/429/5xx)
- preserve structured OBO failure reasons and retryability in resolveOboToken
- improve OBO auth error messaging for connection setup and tool execution
- add tests for transient vs permanent OBO failure paths

* Addressing linting errors / warnings

* 🧊 fix: isolate OBO MCP auth to user-scoped connections

- block OBO-enabled servers from app-level shared MCP connections
- bypass shared connection lookup for OBO servers in MCPManager.getConnection
- add regressions covering OBO connection scoping and preserve non-OBO app connection reuse

* 🛠️ refactor: centralize MCP user-scoped connection policy

- add shared requiresUserScopedConnection helper for OAuth, OBO, and customUserVars
- use the shared predicate in MCPManager and ConnectionsRepository
- add utils coverage for user-scoped connection policy

* 🧊 fix: restrict MCP OBO config to header-capable transports

- Move OBO configuration out of the shared MCP base options schema and allow it
only on SSE and streamable-http transports, where request headers are applied.
- Explicitly reject OBO on stdio and websocket configs to avoid accepted-but-
nonfunctional server definitions. Add schema coverage for admin/config parsing
and user-input websocket validation.

* 🧊 fix: single-flight concurrent OBO token exchanges

Concurrent tool calls that arrive on a cache miss were each issuing
their own jwt-bearer request to the IdP. Under that fan-out, Entra
intermittently returned errors that the retry classifier saw as
non-retryable, surfacing as:

  "The identity provider rejected the OBO token exchange.
   Cannot execute tool <name>. Re-authenticate the user or
   verify the configured OBO scopes and retry."

A user retry then hit the populated cache and succeeded, which matches
the observed flakiness — the cache was empty at the moment of fan-out
but populated by the time the user clicked retry.

- Coalesce concurrent exchanges in `OboTokenService.exchangeOboToken`
keyed by `${openidId}:${scopes}`. Callers that arrive while an exchange
is in flight share the same upstream request and receive the same
result. `fromCache=false` continues to force a fresh, independent
exchange (and is not joined by `fromCache=true` callers). The IdP
call, single-retry path, and cache write are unchanged — they were
moved into a `performOboExchange` helper so the coalescing wrapper
stays small.
- Tests cover: coalescing on the same key, isolation between different
keys, cleanup on success, cleanup on failure, and the
`fromCache=false` bypass.

* 🔒 feat: gate MCP OBO config behind MCP_SERVERS.CONFIGURE_OBO permission

OBO silently mints per-user delegated tokens from the caller's federated
access token and forwards them to whatever URL the server config points at.
Previously, anyone with MCP_SERVERS.CREATE could configure obo.scopes — so
if server creation is ever delegated beyond admins, a user could stand up
an attacker-controlled server, attach it to a shared agent, and exfiltrate
other users' downstream tokens on tool invocation.

Add a dedicated MCP_SERVERS.CONFIGURE_OBO permission (ADMIN: true, USER:
false by default) and enforce it at three layers so the safety property
no longer depends on CREATE staying admin-only:

- Create/update: POST/PATCH /api/mcp/servers returns 403 when the body
  carries `obo` and the caller's role lacks the permission.
- Runtime fail-closed: for DB-sourced configs, MCPConnectionFactory and
  MCPManager.callTool re-check the original author's role before each
  OBO exchange. If the author has been downgraded, the exchange is
  skipped (factory) or refused (callTool) — retained configs lose their
  privileges automatically.
- UI: the OBO option is hidden in the MCP server dialog for users
  without the permission; a CONFIGURE_OBO toggle is exposed in the MCP
  admin role editor.

Existing role docs receive the new sub-key via the permission backfill
in updateInterfacePermissions on next startup, preserving any
operator-set values. YAML/Config-sourced server configs are unaffected
since they're admin-controlled at the deployment level.

* 🧊 fix: wire OBO machinery for servers with requiresOAuth: false

The discovery and user-connection paths gated OAuth wiring (flow
manager, token methods, oboTokenResolver, oboTrustChecker) behind
isOAuthServer(), which only considers requiresOAuth/oauth fields.
A DB-stored OBO server with requiresOAuth: false therefore landed in
the non-OAuth branch, never received an oboTokenResolver, and the
factory's usesObo getter evaluated to false — sending a bare request
that the upstream rejected with invalid_token.

Add requiresOAuthMachinery() (OAuth OR OBO) and use it at those two
gates. isOAuthServer remains for the OAuth-handshake-only check
(shouldInitiateOAuthBeforeConnect), where OBO must not initiate a
handshake. Plumb the OBO resolver/trust-checker through
ToolDiscoveryOptions so reinitMCPServer can pass them on the
discovery path.

* 🧊 fix: lock all OBO-target fields (URL, proxy, headers, auth) without CONFIGURE_OBO

The CONFIGURE_OBO permission was meant to gate control of the endpoint
that receives OBO-minted per-user delegated tokens and the scopes that
are requested. The previous frontend lock + backend gate only covered
obo.scopes and the auth section, leaving url/proxy/headers/etc. editable
by anyone with UPDATE — meaning a non-permission user could still
redirect an existing OBO server's token flow to an attacker endpoint.

Switch to an allowlist policy: when editing an OBO server without
CONFIGURE_OBO, only title/description/iconPath are mutable. Backend
rejects any other field change with 403; frontend disables the
non-allowlist sections (URL, transport, auth, trust) via fieldset.
The comparison surface (MCP_USER_INPUT_FIELDS) is derived from
MCPServerUserInputSchema's union members so it stays in sync with the
schema. New schema fields land in the locked set by default — adding to
the allowlist is the only way to unlock them, which preserves the
security-review boundary.

* 🧊 fix: skip unauthenticated MCP inspection for OBO-only servers

MCPServerInspector.inspectServer() ran an unauthenticated temp connection
unless the config had requiresOAuth or customUserVars set. For OBO-only
servers without standard MCP OAuth advertisement, this caused
MCPConnectionFactory.create to attempt the connection without a user or
oboTokenResolver — failing on servers that reject the MCP initialize
handshake without a valid bearer token, which surfaced as
MCP_INSPECTION_FAILED on create/update.

Add `obo` to the skip list alongside requiresOAuth and customUserVars,
matching the existing pattern for user-scoped auth modes.

* Addressed linting error: watchedTitle is declared but never referenced (the auto-fill logic at line 156 uses getValues('title') instead). Deleted constant.
2026-06-01 22:36:18 -04:00
Danny Avila
983a33fbad
🎛️ refactor: Scope App-Config Override Cache by Isolation Context (#13455)
getAppConfig caches per-principal merged config overrides under a key built by
overrideCacheKey(role, userId, tenantId). The key used the tenantId *argument*
only — but callers that go through the tenant middleware (the common path)
pass no explicit tenantId and rely on the AsyncLocalStorage tenant context.
Those calls were keyed under the shared '__default__' bucket, so the DB query
(correctly scoped to the ALS tenant by the Mongoose plugin) produced a merged
config that was then cached and served to the next tenant resolving the same
role/user — leaking model specs, endpoints, and interface flags across tenants.

Fall back to getTenantId() before '__default__' so the cache key reflects the
actual tenant scope (param or ALS). Tighten the strict-mode warning to fire
only when there is genuinely no tenant anywhere (param nor ALS), since the ALS
case is now scoped rather than defaulted. No-op for single-tenant deployments,
where getTenantId() is undefined and the key stays '__default__'.

Adds tests (real Map-backed cache) proving the ALS tenant scopes the key and
that two tenants resolving the same role each get their own config with no
cache collision.
2026-06-01 18:00:53 -04:00
Danny Avila
83c7d637c3
🎨 chore: prettier --write all workspaces (#13281)
Run `prettier --write` over the source trees of every workspace to align
with the repo's own `.prettierrc` (`printWidth: 100`, `singleQuote: true`,
`trailingComma: 'all'`, etc.). **19 files reformatted total** — purely
whitespace and line-wrap changes, no functional edits and no API changes.

Scope:
- `packages/api/src/**/*.{ts,tsx}` — 14 files
- `packages/client/src/**/*.{ts,tsx}` — 1 file
- `packages/data-schemas/src/**/*.{ts,tsx}` — 4 files
- `api/**`, `client/**`, `packages/data-provider/**` — already prettier-clean

Most of the drift is in argument-list / type-annotation wrapping where
the formatted form fits within `printWidth` but the current source keeps
a hand-wrapped multi-line shape. Example:

  // before
  function countWebSearchDefinitions(
    toolDefinitions: Array<{ name: string }> | undefined,
  ): number { … }

  // after (still well under 100 cols)
  function countWebSearchDefinitions(toolDefinitions: Array<{ name: string }> | undefined): number { … }

`npx prettier --check` across all workspaces is now clean. The local
pre-commit hook (`lint-staged` → `prettier --write`) would have produced
the same result on any future edit to these files.

There are no prettier-checking workflows in CI today, so drift like this
can re-appear if PRs are merged with the hook bypassed. Companion PR
#13282 adds a `prettier --check` step to `eslint-ci.yml` so future
drift gets caught.
2026-05-23 17:52:58 -04:00
Danny Avila
6d6ea08da4
🆔 feat: Built-in Build Metadata for Support Triage (#12756)
* 🏗️ refactor: Derive App Version from Root package.json + Add buildInfo Schema

The hardcoded `Constants.VERSION` in `data-provider` is now replaced at
rollup build time via `@rollup/plugin-replace`, sourcing from the root
`package.json` so version bumps are a single-file change.

Adds the shape needed by the rest of the series:
- `interface.buildInfo` boolean flag (default `true`) — lets self-hosters
  opt out of exposing commit/branch/date.
- `buildInfo` on `TStartupConfig` — commit/commitShort/branch/buildDate.
- `SettingsTabValues.ABOUT` — new settings tab enum value.

Ref: https://github.com/danny-avila/LibreChat/issues/12406

* 🛠️ feat: Add Build Metadata Resolver and Expose via /api/config

Adds `resolveBuildInfo()` in `@librechat/api` that surfaces commit SHA,
branch, and build date from (in order) `BUILD_*` env vars, then local git
metadata. Result is cached per-process.

`/api/config` includes a `buildInfo` field on both authenticated and
anonymous responses when `interface.buildInfo !== false` and at least one
resolver field is populated. Omitted entirely otherwise.

Designed so pre-built Docker images carry metadata via build-arg while
source installs pick it up from `.git` — no manual version tracking.

Ref: https://github.com/danny-avila/LibreChat/issues/12406

* ℹ️ feat: Add Settings → About Panel with Diagnostics Copy

New Settings tab that renders the running build's version, commit (short
SHA), branch, and build date in a monospaced block alongside a "Copy
diagnostics" button that emits a preformatted text blob for pasting into
support issues.

Tab is hidden when `interface.buildInfo` is set to `false`. Reads from
`startupConfig.buildInfo` provided by `/api/config`.

Ref: https://github.com/danny-avila/LibreChat/issues/12406

* 🐳 ci: Inject BUILD_COMMIT/BRANCH/DATE into Docker Images

Adds optional `BUILD_COMMIT`, `BUILD_BRANCH`, `BUILD_DATE` ARGs to both
`Dockerfile` and `Dockerfile.multi`, wired as `ENV` vars in the runtime
stage so the backend's `resolveBuildInfo` picks them up.

All image-publishing workflows (`tag`, `main`, `dev`, `dev-branch`,
`dev-staging`) now compute `${github.sha}`, `${github.ref_name}`, and a
UTC timestamp, then pass them to `docker/build-push-action` as
`build-args`.

Defaults are empty — non-CI builds (local `docker build`) still work,
and the backend falls back to local `.git` metadata if ARGs aren't set.

Ref: https://github.com/danny-avila/LibreChat/issues/12406

* 📝 docs: Direct Bug Reporters to Settings → About for Version Info

The previous instructions (`docker images | grep librechat`,
`git rev-parse HEAD`) only worked for a subset of deployments and
rarely produced a commit SHA for users pulling pre-built images.

Point users to the new in-app Settings → About panel's
"Copy diagnostics" button, which captures version, commit, branch,
build date, and user agent in a single preformatted block. Fallback
instructions preserved for older installs.

Ref: https://github.com/danny-avila/LibreChat/issues/12406

* 🐳 fix: Move BUILD_* ENV to End of Docker Stages to Preserve Layer Cache

Per-commit BUILD_COMMIT/BUILD_DATE changes were being promoted to ENV
before `npm ci` / `npm run frontend` (single-stage) and before
`npm ci --omit=dev` (multi-stage api-build), which invalidated the cache
for every subsequent layer on every CI run.

Move the ARG/ENV block below the heavy install and build steps in both
Dockerfiles. Metadata is still available in the runtime image but no
longer busts layer reuse.

Addresses codex review on #12756.

* 🔧 fix: Propagate interface.buildInfo=false to Unauthenticated /api/config

The unauthenticated branch of `/api/config` was emitting an `interface`
object only when `privacyPolicy` or `termsOfService` was set, which
meant an admin's explicit `interface.buildInfo: false` opt-out was never
visible to anonymous/guest clients. `Settings.tsx` gates the About tab
on `startupConfig?.interface?.buildInfo !== false`, so a missing field
fell through as "enabled" for those clients.

Include `interface.buildInfo: false` in the unauth payload whenever it's
explicitly disabled. Keep the implicit default (true) absent to preserve
the minimal-unauth-payload convention.

Addresses codex review on #12756.

* 🔀 ci: Trigger Dev Image Workflows on Root package.json + Dockerfile Changes

The baked `Constants.VERSION` now reads from the root `package.json` via
rollup-plugin-replace, but the `dev-images.yml` and `dev-branch-images.yml`
path filters only matched `api/**`, `client/**`, `packages/**`. A release
commit that only bumps root `package.json` would not trigger a rebuild,
leaving `latest` dev images with stale Footer/About version metadata.

Include `package.json`, `package-lock.json`, and both Dockerfiles in the
path filters so dependency changes (lockfile rebuilds) and image build
tweaks also rebuild dev images.

Addresses codex review on #12756.

* 🧽 fix: Harden About Panel Lifecycle, A11y, and Loading Gate

Review follow-ups on #12756:

- #1 timer leak: stash the copy-state `setTimeout` in a ref and clear it
  from a `useEffect` cleanup so unmounting the Settings dialog mid-toast
  doesn't fire `setCopied(false)` on an unmounted component.
- #3 flash of About tab: gate `aboutEnabled` on `startupConfig != null`
  so the tab stays hidden until `/api/config` returns. For admins who
  disabled `interface.buildInfo`, the tab no longer briefly appears and
  vanishes on page load.
- #6 aria-live placement: move the live region off the interactive
  button onto a dedicated `<span role="status" aria-live="polite">` so
  screen readers announce the copied state, not the full button content
  on every re-render.
- #2 missing coverage: add `About.spec.tsx` exercising populated/empty
  buildInfo rendering, invalid-date handling, diagnostics clipboard
  payload, copy-state toggling, unmount cleanup, and the live region.

* ⚡ perf: Eagerly Resolve Build Info at Module Load

Review follow-up #4 on #12756: `resolveBuildInfo()` calls `execFileSync`
with a 2s timeout on source installs without `BUILD_*` env vars. Paying
this cost on the first HTTP request blocks the event loop mid-flight.

Call `resolveBuildInfo()` once at config route module load so the
resolver's cache is warm before any request arrives. Docker images with
the BUILD_* env vars set sidestep the git path entirely, so this only
affects the edge case of source installs.

* 📝 docs: Document rollup Version Placeholder Contract

Review follow-ups #5 / #8 on #12756. The `__LIBRECHAT_VERSION__`
placeholder relies on a substring replacement rule that only works
because the token appears inside a string literal, and the substitution
only runs during `npm run build:data-provider`.

- Expand the `Constants.VERSION` JSDoc to spell out that consumers read
  the placeholder through the built dist bundle; source-level test
  imports would see the raw placeholder.
- Add a NOTE above the rollup `replace` config warning future
  contributors not to repurpose the token as a bare identifier without
  switching to a quoted replacement value.

Non-functional; prevents future contributors from stepping on a subtle
constraint.

* 🪪 fix: Only Toast "Copied" When Clipboard Copy Actually Succeeds

Codex R5 on #12756. `copy-to-clipboard` returns a boolean indicating
whether the underlying `execCommand('copy')` / fallback prompt actually
wrote to the clipboard. The previous handler flipped to the "Copied"
state unconditionally, which in hardened browsers or when the
permission prompt is dismissed would mislead users into filing bug
reports without the diagnostics blob attached.

Gate the state/timer/live-region on the boolean return; silently no-op
on failure rather than showing a false positive. Adds a test asserting
the button label stays at "Copy diagnostics" when the clipboard call
fails.

* 🐳 fix: Derive main image metadata from checkout

* 🪪 fix: Keep About enabled until disabled

* ✅ test: Avoid literal Settings mock text

* 🧱 refactor: Rename Build Info Module
2026-05-23 09:41:13 -04:00
Danny Avila
2ce3921f78
⚖️ feat: Add Operational Prometheus Metrics (#13265)
* add operational prometheus metrics

* fix metrics review findings

* fix metrics edge cases

* gate mongoose metrics instrumentation

* gate metrics setup when unconfigured
2026-05-22 20:47:41 -04:00
Pete Hampton
679672ad15
🪂 feat: Graceful HTTP shutdown on SIGTERM/SIGINT (#13211)
* 🪂 feat: Graceful HTTP shutdown on SIGTERM/SIGINT

* Address feedback

* don't treat ERR_SERVER_NOT_RUNNING as fatal; route telemetry shutdown through coordinator
2026-05-20 13:33:53 -04:00
Danny Avila
34dd8d5f2a
📈 feat: Add Prometheus Metrics Endpoint + AWS Credential Providers (#13111)
* feat: add prometheus metrics endpoint

* fix: format metrics route spec

* chore: update dependencies in package.json and package-lock.json correctly

- Bump `@smithy/core` to version 3.24.1
- Update `@aws-sdk/credential-providers` to version 3.1045.0
- Reintroduce `prom-client` dependency in package.json
- Remove unnecessary dependencies from package.json

* chore: import order

* fix: declare s3 presigner peer dependency

* fix: normalize shared link metrics path

* fix: bound metrics path labels

* fix: tighten metrics auth and peers

* fix: collapse partial metrics paths
2026-05-13 16:49:25 -04:00
Ravi Kumar L
05d4e90f91
🌩️ feat: Strict CloudFront signed cookie enforcement via requireSignedAccess (#13078)
* feat(cloudfront): add requireSignedAccess to enforce strict signed access

Introduces cloudfront.requireSignedAccess (default false). When enabled,
initializeCloudFront requires both CLOUDFRONT_KEY_PAIR_ID and
CLOUDFRONT_PRIVATE_KEY, rejects the unimplemented imageSigning="url"
mode, and initializeFileStorage throws to block startup on any
CloudFront init failure. OSS path is unchanged: missing keys still
log-and-continue when requireSignedAccess is false.

Adds low-noise startup and cookie-issuance logs without leaking signed
URLs, policies, signatures, private keys, or cookie values.

* fix(cloudfront): reject requireSignedAccess unless imageSigning is "cookies"

Previously requireSignedAccess=true was accepted with imageSigning="none"
or "url", but setCloudFrontCookies() only runs for "cookies" — leaving
strict mode toothless: CloudFront stayed publicly accessible, or image
delivery broke on a distribution that actually requires signed access.

Adds a Zod refinement plus a runtime guard in initializeCloudFront so
the only currently-functional strict configuration is imageSigning
"cookies". Signed URL mode can lift this restriction once implemented.

* fix(cloudfront): resolve strict access type checks

* chore(cloudfront): reduce strict startup log noise

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-05-11 23:30:01 -04:00
Atef Bellaaj
187ab787da
🌩️ feat: CloudFront CDN File Strategy (#12193)
* 🌩️ feat: CloudFront CDN File Strategy + signed cookies

Squashed from PR #12193:
- feat(storage): add CloudFront CDN file strategy
- feat(auth): add CloudFront signed cookie support

Note: package.json/package-lock.json dependency additions are intentionally
omitted from this commit and will be re-added via `npm install` after rebase
to avoid lock-file merge conflicts. The two new peer deps that need to be
re-installed are:
  - @aws-sdk/client-cloudfront@^3.1032.0
  - @aws-sdk/cloudfront-signer@^3.1012.0

Also fixes 4 missing destructured names in AuthService.spec.js
(getUserById, generateToken, generateRefreshToken, createSession) that
were referenced in tests but not imported from the mocked '~/models'.

* 📦 chore: install CloudFront SDK deps for PR #12193

Adds the two AWS CloudFront packages required by the rebased
CloudFront CDN strategy:
  - @aws-sdk/client-cloudfront
  - @aws-sdk/cloudfront-signer

Following the @aws-sdk/client-s3 pattern:
  - api/package.json: regular dependency (runtime resolution)
  - packages/api/package.json: peerDependency

Generated by `npm install` against the freshly rebased lock file
to avoid the merge conflicts that came from the original PR's
lock-file edits being made against an older base of dev.

* 🐛 fix: CI failures + review findings on CloudFront PR #12193

CI fixes
- Rename packages/data-provider/src/__tests__/cloudfront-config.test.ts
  → src/cloudfront-config.spec.ts. Jest's default testMatch picks up
  __tests__/ directories even inside dist/, so the compiled .d.ts shell
  was being executed as an empty test suite. Moving to .spec.ts (matching
  the rest of the package) avoids the dist/ pickup.
- Add cookieExpiry: 1800 to CloudFront crud.test makeConfig: the schema
  applies a default so CloudFrontFullConfig requires it.

Review findings addressed
- #1 (Codex + comprehensive): Normalize CloudFront domain with /\/+$/
  regex (and key with /^\/+/ regex) in buildCloudFrontUrl, matching the
  cookie code so resource policy and file URLs stay aligned even when
  the configured domain has multiple trailing slashes. Added tests.
- #2: Move DEFAULT_BASE_PATH out of s3Config into shared
  packages/api/src/storage/constants.ts. ImageService no longer imports
  S3-specific config.
- #3: getCloudFrontConfig() returns Readonly<CloudFrontFullConfig> | null
  to discourage mutation of the cached signing config.
- #4: Add cross-field refinement tests for cloudfrontConfigSchema
  (invalidateOnDelete-without-distributionId,
  imageSigning="cookies"-without-cookieDomain).
- #6: Revert unrelated MCP comment re-indentation in
  librechat.example.yaml.
- #7: Add azure_blob to the strategy list comment.

Skipped
- #5 (extractKeyFromS3Url with CloudFront URLs): existing
  deleteFileFromCloudFront tests already cover the path-equivalence
  assumption; renaming the helper is real refactor work beyond this
  PR's scope.
- #8, #9 (NIT, low confidence): leaving for author judgement.

* 🧹 chore: drop dead DEFAULT_BASE_PATH from s3Config test mock

After moving DEFAULT_BASE_PATH to ~/storage/constants, crud.ts no longer
reads it from s3Config — so the entry in the s3Config jest mock was
misleading dead config. The tests still pass because the unmocked real
constants module provides the value.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-05-05 13:21:05 -04:00
Artyom Bogachenko
5683706af5
🔐 feat: OIDC Bearer Token Authentication for Remote Agent API (#12450)
* Remote Agent Auth middleware

* consider migration and update user

* fix eslint errors

* add scope validation

* fix codex review errors

* add filter for use: sig

* add jwks-rsa deps

* Fix remote agent OIDC auth review findings

* Polish remote agent OIDC timeout coverage

* Reject remote OIDC tokens without subject

* Use tenant context for remote agent auth config

* Harden remote agent OIDC scope handling

* Polish remote agent OIDC cache and scope tests

* Resolve remote agent auth review comments

* Reuse OpenID email claim resolver for remote auth

* Skip empty OpenID email fallback claims

* Use pre-auth tenant context for remote auth config

* Downgrade expected OIDC fallback logging

* Require secure remote OIDC endpoints

* Polish remote agent auth edge cases

* Enforce unique balance records

* Bind remote OpenID users to issuer

* Fix issuer-scoped OpenID indexes

* Avoid unique balance index requirement

* Fix remote OpenID issuer normalization boundaries

* Require issuer-bound OpenID lookups

* Enforce tenant API key policy after auth

* Fix remote auth tenant policy types

* Normalize remote OIDC discovery issuer

* Allow normalized remote OIDC issuer validation

* Enforce resolved tenant OIDC policy

* Polish OpenID issuer and scope validation

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-05-04 17:06:35 -04:00
Danny Avila
74307e6dcc
💭 feat: Require Explicit Auto-agent Enablement for Memories (#12886) 2026-05-01 23:56:08 +09:00
Marco Beretta
f6ee2ea0ee 📜 feat: Skills UI + Initial E2E CRUD / Sharing (#12580)
* 🎨 feat: Skills UI — Create/Edit/Share/List with Conditional File Tree

First-pass UI on top of the CRUD API scaffolding (#12613). Ships the full
user-facing flow for inline, single-SKILL.md skills and leaves a clean
drop-in for phase-2 multi-file support.

- Create a skill from /skills/new with name (kebab-case, validated),
  description, and SKILL.md body — wired to the real `useCreateSkillMutation`
  and `TCreateSkill` payload.
- List skills in a sidebar (SkillsSidePanel) via `useListSkillsQuery` with
  live search filtering.
- Edit any skill the caller has EDIT permission on — `useUpdateSkillMutation`
  passes `expectedVersion` for optimistic concurrency and surfaces 409
  conflicts as a warning toast + cache refetch.
- Non-blocking `TSkillWarning[]` (e.g. "description too short") are shown
  inline above the form after a successful create/patch.
- Read-only mode when the current user lacks EDIT — the form still renders
  but inputs are marked `readOnly` and the save/reset buttons are hidden.
- Share via ACL using the existing `GenericGrantAccessDialog` — the
  `ShareSkill` button is gated on the SHARE permission.
- Delete with confirmation, driven by `useDeleteSkillMutation({ id })`.
- Conditional file tree: only rendered when `useListSkillFilesQuery`
  returns > 0 files. The tree groups flat `relativePath` strings into a
  nested view (no `react-arborist` dependency) and supports per-file
  deletion via `useDeleteSkillFileMutation`. Upload is intentionally
  deferred — the backend stubs it at 501 in phase 1.

- New routes: `/skills`, `/skills/new`, `/skills/:skillId`.
- Sidebar accordion (`SkillsAccordion` wrapping `SkillsSidePanel`) added
  to `useSideNavLinks` gated on `PermissionTypes.SKILLS` USE.

The initial UI branch (#12580) shipped a lot of exploration code on top of
a now-superseded placeholder backend. Kept as complementary: the `Skills/`
component tree, translation keys, role descriptions, `PublicSharingToggle`
SKILL mapping, `resources.ts` SKILL config, `useCanSharePublic` SKILL
mapping, and `data-provider/roles.ts` `useUpdateSkillPermissionsMutation`.

Deferred out of this first pass:
- Skill favorites (`useSkillFavorites`, `getSkillFavorites` endpoint) —
  the backend route doesn't exist yet; saving for a follow-up.
- AgentConfig `SkillSelectDialog` integration — the UI branch had this
  gated behind `false &&`; rolled back with the config.
- `InvocationMode` / `CategorySelector` / `parseSkillMd` / tree-node
  mutations — not in the Anthropic skill spec and not in the CRUD API.
- `react-arborist` dependency — replaced with a hand-rolled recursive
  tree built from flat `TSkillFile[]`.

- 38 data-schemas skill model tests: pass
- 25 api skill route tests: pass
- 16 user-controller cleanup tests: pass

* 🔐 feat: Default-On Skills in Interface Config and Role Seeder

The skills accordion was registered in the side nav gated on
`PermissionTypes.SKILLS` USE, but no one was actually seeding that
permission on startup, so a fresh install had the USER role with
zero skill permissions and the accordion never rendered.

Fixes three gaps:

1. `interfaceSchema` in data-provider's `config.ts` had no `skills`
   field at all. Added it alongside the existing agents/prompts shape
   (boolean | { use, create, share, public }) and a default of
   `{ use: true, create: true, share: false, public: false }`.

2. `loadDefaultInterface` in data-schemas passed every interface key
   through to the loaded config EXCEPT `skills`. Added the one-line
   passthrough so `appConfig.interfaceConfig.skills` is actually
   populated on boot.

3. `updateInterfacePermissions` in packages/api/src/app/permissions.ts
   seeds role permissions from the interface config on every restart.
   Added:
   - `SKILLS` case to `hasExplicitConfig`
   - `skillsDefaultUse/Create/Share/Public` extraction (mirrors
     prompts/agents)
   - `PermissionTypes.SKILLS` block in `allPermissions` that falls
     through config → roleDefaults → schema default, same pattern as
     AGENTS and PROMPTS
   - `SKILLS` entry in the share-backfill array so that pre-existing
     SKILL role docs missing SHARE/SHARE_PUBLIC get them filled on the
     next restart

Test expectations updated: seven `expectedPermissionsFor(User|Admin)`
blocks in `permissions.spec.ts` now include SKILLS, matching the
role-default values (USER: use+create true, share/public false;
ADMIN: all true).

Result: on a fresh install, a regular USER gets skill USE/CREATE
and the "Skills" accordion shows up in the chat side panel without
any yaml config. Admins can lock it down per role or per tenant via
`interface.skills` in librechat.yaml.

Tests:
- 34 packages/api permissions.spec.ts: pass
- 151 packages/api app tests: pass
- 38 data-schemas skill.spec.ts: pass
- 928 data-provider tests: pass
- 25 api skills.test.js: pass

* ♻️ fix: Resolve Skills UI Review Findings

Addresses the 13 findings from the PR review against the prior commit.

1. **canEdit consistency** — extracted `useSkillPermissions(skill)` as the
   single source of truth for owner/admin/ACL gating. `SkillsView`,
   `SkillForm`, `ShareSkill` all consume it; `SkillFileTree`'s per-file
   delete button now honors admin + EDIT-bit permissions instead of just
   ownership. Unit tests cover owner, admin, editor-ACL, viewer-ACL,
   owner-ACL, loading, and undefined-skill cases.

2. **Disabled submit buttons** — create/edit form submit buttons now set
   native `disabled` (not just `aria-disabled`) during `isLoading`.
   `onSubmit` also guards with an early return when the mutation is still
   in-flight so a duplicate enter-key submit can't create two skills.

3. **Wrong maxLength error message** — description/name `maxLength` rules
   no longer re-use `com_ui_skill_*_required`. Added dedicated
   `com_ui_skill_name_too_long` and `com_ui_skill_description_too_long`
   keys with the literal limit interpolated (`{{0}}`).

4. **Search debouncing** — `SkillsSidePanel` now threads the filter input
   through the existing `useDebounce` hook (250ms) so typing "skills" no
   longer fires six separate list queries.

5. **Frontend test coverage** — added:
   - `tree.test.ts` (9 tests) covering `buildTree` / `nodeKey` edge cases:
     empty input, single root file, multiple roots, nested folders,
     deeply-nested trees, lexicographic sort, empty paths, stable keys
   - `useSkillPermissions.test.ts` (7 tests) covering every precedence
     branch (owner / admin / EDIT / VIEW / owner-ACL / loading / undef)

   Form integration tests proved flaky against react-hook-form's async
   `isValid` with our jest-dom mock setup; deferred to a follow-up PR
   with a proper `@librechat/client` test harness.

6. **Shared `SKILL_NAME_PATTERN`** — promoted the regex plus the four
   length constants (`SKILL_NAME_MAX_LENGTH`,
   `SKILL_DESCRIPTION_MAX_LENGTH`, `SKILL_DESCRIPTION_SHORT_THRESHOLD`,
   `SKILL_DISPLAY_TITLE_MAX_LENGTH`, `SKILL_BODY_MAX_LENGTH`) out of
   `packages/data-schemas/src/methods/skill.ts` and into
   `packages/data-provider/src/types/skills.ts`. The data-schemas
   module now aliases the shared exports so the backend validator and
   the frontend form share one source of truth. Also fixed a latent bug:
   the client regex was stricter than the backend
   (`^[a-z0-9]+(?:-[a-z0-9]+)*$` vs. the real `^[a-z0-9][a-z0-9-]*$`),
   which would have rejected valid names like `foo--bar` client-side.

7. **Removed hardcoded "Claude"** — replaced `com_ui_skill_description_help`
   ("Claude uses this to...") with a new `com_ui_skill_create_subtitle`
   for the form header and `com_ui_skill_description_field_hint`
   ("This is what the model reads to decide...") for the inline hint.
   LibreChat is LLM-agnostic; the old copy misled GPT/Gemini users.

8. **Lifted tree mutation hook** — `useDeleteSkillFileMutation` is now
   instantiated once in `SkillFileTree` (not per `TreeRow`). A
   `TreeContext` provides `onDeleteFile` + `isDeleting` + `canEdit` to
   rows. A 60-node tree used to instantiate 60 mutation hooks; it now
   instantiates one.

9. **List O(n) re-render** — `SkillListItem` no longer reads
   `useParams()` directly. `SkillList` reads the active id once and
   passes `isActive` as a prop, so navigation only re-renders the two
   items whose `isActive` flipped (memo'd), not all N items.

10. **Deduped help text** — the field-level hint and form-level subtitle
    now use different translation keys with distinct copy instead of
    showing the same sentence twice on the same page.

11. **Removed ineffective `useCallback`** — `DeleteSkill.handleDelete`,
    `CreateSkillForm.onSubmit` / `.handleCancel`, `SkillForm.onSubmit`,
    and `SkillFileTree.handleDeleteFile` all wrapped closures around
    React Query `mutation` refs, whose identities change every render.
    Their dep arrays invalidated every render, making the memo a no-op
    with extra overhead. `SkillFileTree` now destructures the stable
    `mutate` function and inlines the arrow inside the memoized
    `contextValue` — one stable reference per deps change.

12. **Import order** — fixed shortest→longest package ordering and
    longest→shortest local ordering across all touched skill files per
    AGENTS.md. `react` always first where imported.

13. **Memoization principle** — documented the rule with inline comments:
    `memo` on components that appear in repeated contexts (`TreeRow`,
    `SkillListItem`) or as children of frequently-re-rendering parents
    (`ShareSkill` / `DeleteSkill` under `SkillForm`'s per-keystroke
    form-state updates). Removed `memo` from `SkillFileTree` since its
    parent `SkillDetailPanel` only re-renders on query-data changes.

- 38 data-schemas skill.spec.ts
- 34 packages/api permissions.spec.ts
- 25 api skills.test.js
- 16 client unit tests (9 buildTree + 7 useSkillPermissions)
- All type-checks + eslint clean on touched files

* 🧹 fix: Skills Duplication, Input Styling, Remove LLM-specific Copy

Three UI fixes from an in-chat review pass:

1. **Sidebar duplication** — `SkillsView` was rendering its own
   `SkillsSidePanel` aside alongside the chat side panel's
   `SkillsAccordion`, so on `/skills` the user saw the skill list
   twice. Fixed by mirroring the `InlinePromptsView` pattern: the
   route content is now just the detail / create panel and the
   chat side panel is the sole list. Added `/skills → /skills/new`
   redirect and a `/skills/new` literal route so `useParams().skillId`
   is `undefined` for "new" (matches prompts).

2. **Name Input styling** — the big floating-label pattern used by
   prompts/agents for the primary name field was replaced with a
   conventional `<Label>` + `<Input>` above it, diverging from the
   rest of the app. Restored the prompts-style `text-2xl` input with
   the peer-focus animated label on both `CreateSkillForm` and
   `SkillForm`. Kept the conventional pattern for description and
   body since they're textareas.

3. **Remove LLM-specific copy from skill translations** — dropped
   `com_ui_skill_description_help` ("Claude uses this to...") and
   the transitional "This is what the model reads..." phrasing.
   Field hint is now a neutral "Be specific about when this skill
   should apply." and the create-page subtitle is a neutral "Author
   a new skill your agents can invoke." LibreChat is LLM-agnostic;
   baking product names into user-facing copy is wrong outside the
   `com_endpoint_anthropic_*` keys where the setting actually only
   applies to Claude models.

Side-effect: the `SkillDetailView` wrapper in `SkillsView` now only
renders the file-tree aside when the skill has > 0 files — same
conditional-tree behavior as before, just scoped to this route
instead of also trying to also render a list sidebar.

- 16 client skill tests still pass
- Type-check + eslint clean on touched files

* 🎁 feat: Restore Skills UI from PR #12580

Brings back everything the original UI PR (#12580, commit da039917c)
shipped that my earlier rebase dropped. Verbatim restores where possible;
adapts the new hooks/types where the backend contract has shifted.

**Scoped-out / gated-off (now restored as inert UI scaffolding):**
- `hooks/useSkillFavorites.ts` + `utils/favoritesError.ts` + the
  `useGetSkillFavoritesQuery` / `useUpdateSkillFavoritesMutation` additions
  in `data-provider/Favorites.ts`. The backend route doesn't exist yet —
  the data-service functions resolve with empty arrays so the Star UI is a
  visual-only no-op until phase 2.
- `dialogs/SkillSelectDialog.tsx` + the "Add Skills" section in
  `SidePanel/Agents/AgentConfig.tsx` (still gated behind the original
  `false &&`) + `skills?: string[]` on `AgentForm` / `Agent` /
  `AgentCreateParams` / `AgentUpdateParams` + the `skills: []` entry in
  `defaultAgentFormValues`.
- `TUserFavorite.skillId` reserved on the shared favorites type.

**Concept-is-gone / deleted-types (restored as UI-only types + stubs):**
- `InvocationMode` enum and `TSkillNode`, `TSkillTreeResponse`,
  `TCreateSkillNodeRequest`, `TUpdateSkillNodeRequest` types in
  `packages/data-provider/src/types.ts`. UI-facing only; the backend flat
  `TSkillFile[]` contract is unchanged.
- `TSkill.invocationMode?: InvocationMode` as an optional field. Forms
  read/write it in local state and deliberately drop it from the PATCH
  payload until the backend column lands.
- `tree/SkillFileTree.tsx` (`react-arborist`-based), `SkillTreeNode.tsx`,
  `TreeToolbar.tsx`, `SkillFileEditor.tsx`, `SkillFilePreview.tsx` — full
  filesystem-style browser UI restored verbatim.
- `data-provider/Skills/tree-queries.ts` + `tree-mutations.ts` hooks
  (`useGetSkillTreeQuery`, `useCreateSkillNodeMutation`, etc.). The
  `data-service` stubs them: `getSkillTree` returns `{ nodes: [] }`,
  `createSkillNode` / `updateSkillNode` / `updateSkillNodeContent` return
  synthetic node shapes, `deleteSkillNode` resolves void. Hooks compile
  and run; tree is empty until phase 2 wires a real backend.
- `MutationKeys.createSkillNode` / `updateSkillNode` / `deleteSkillNode` /
  `updateSkillNodeContent` + `CreateSkillNodeBody` /
  `UpdateSkillNodeVariables` / `DeleteSkillNodeBody` /
  `UpdateSkillNodeContentVariables` types.
- `QueryKeys.skillTree` / `skillNodeContent` / `skillFavorites` /
  `favorites` and the `skillTree()` endpoint helper.

**Scope-simplified (restored with minimal adaptation):**
- `display/SkillDetailHeader.tsx` + `display/SkillDetail.tsx`. Header now
  falls back to `InvocationMode.auto` when `skill.invocationMode` is
  undefined.
- `forms/SkillContentEditor.tsx` — click-to-edit markdown preview toggle
  for the SKILL.md body field. Wired into both `CreateSkillForm` and
  `SkillForm` replacing the plain `<TextareaAutosize>`.
  (Needed `@ts-ignore` on `remarkPlugins` / `rehypePlugins` for the same
  `PluggableList` vs `Pluggable[]` shape drift `MarkdownLite.tsx` already
  works around.)
- `forms/InvocationModePicker.tsx` + `forms/CategorySelector.tsx` — the
  auto/manual/both dropdown and the skill category selector. Wired into
  both forms inside a `FormProvider` so the Controller-based widgets can
  read `useFormContext`. `category` flows to the PATCH / POST payload as
  before; `invocationMode` is UI-only per the type note above.
- `buttons/CreateSkillMenu.tsx` + `utils/parseSkillMd.ts` — dropdown with
  AI / Manual / Upload SKILL.md entries + the YAML frontmatter parser for
  the upload path. `CreateSkillForm.defaultValues` now accepts the parsed
  shape, so the upload → redirect → pre-populated form flow works again.
- `buttons/AdminSettings.tsx` — admin permissions dialog. Uses the
  existing `useUpdateSkillPermissionsMutation` which was already wired.
- `sidebar/FilterSkills.tsx` — restored filter + AdminSettings +
  CreateSkillMenu wrapper. `SkillsSidePanel.tsx` is back to the original
  `FilterSkills`-based layout.
- `lists/SkillList.tsx` + `lists/SkillListItem.tsx` — restored verbatim.
- `layouts/SkillsView.tsx` — restored the full tree + file editor + file
  preview layout. The chat side panel keeps its own accordion list; this
  view is the inline detail experience.
- `hooks/Generic/useUnsavedChangesPrompt.ts` — route-leave guard hook.

- `useGetSkillByIdQuery` is aliased to `useGetSkillQuery` so restored
  components (`SkillsView`, `SkillForm`) that import the old name resolve
  to the new hook.
- `SkillSelectDialog` + `AgentConfig` coerce `skillsData?.skills` instead
  of `.data` (list response shape drift from the CRUD PR).
- `CreateSkillForm` / `SkillForm` wrap their JSX in `FormProvider` so the
  restored `CategorySelector` and `SkillContentEditor` components —
  which read `useFormContext` — work inside the existing forms without
  another refactor.
- `CreateSkillForm.defaultValues` prop accepts `Partial<Values> &
  { invocationMode?: unknown }` so the upload flow's
  `{ name, description, invocationMode }` shape passes through cleanly.
- `SkillsView` route map gains `/skills/:skillId/edit` and
  `/skills/:skillId/file/:nodeId` so the tree-navigation URLs the original
  view produces actually resolve.
- `client/package.json` gains `react-arborist@^3.4.3`.
- ~60 translation keys the restored files reference — invocation labels,
  edit/create page titles, file editor chrome, tree toolbar tooltips,
  favorites, admin allow-settings, unknown-file-type, sr_public_skill,
  delete/rename _var variants — all added to `en/translation.json`.

- Prompts-style floating-label name input — kept from my earlier commit
  so it matches the rest of the app (user reviewed and approved that
  styling). Hidden skill-body textarea is replaced by `SkillContentEditor`
  in both forms.

- 38 data-schemas skill.spec.ts
- 34 packages/api permissions.spec.ts
- 25 api skills.test.js
- 7 client useSkillPermissions.test.ts
- Type-check: pre-existing error count (188) dropped to 120 because my
  restorations fixed some previously-broken field types.

* chore: Update package-lock.json to include react-arborist and memoize-one

* feat: Add support for react-arborist in Vite configuration

This update introduces a new condition in the Vite configuration to handle the 'react-arborist' package, ensuring it is properly recognized during the build process. This change enhances compatibility with the recently added 'react-arborist' dependency in the project.

* 🩹 fix: Hide InvocationMode, Fix SkillContentEditor Click-to-Edit

1. Hide InvocationModePicker from both CreateSkillForm and SkillForm.
   Component stays on disk for when the backend lands the column.

2. Fix "Click to edit" doing nothing on SkillContentEditor. The
   `onBlur={() => setIsEditing(false)}` on the TextareaAutosize was
   racing with `autoFocus` — React renders the textarea, autoFocus
   fires, then a layout/reconciliation blur fires immediately,
   bouncing back to preview mode before the user can interact.
   Removed onBlur; users toggle via the header button or Escape key.

* 🎨 feat: Reader-First Skills UI — Match Claude.ai Layout

Reworks the Skills UI from form-first to reader-first, matching
Claude.ai's skill detail pattern.

**Default view is now read-only.** Clicking a skill in the sidebar
navigates to `/skills/:id` which renders `SkillDetail` — a clean
content view with:
- Skill name as the primary heading
- Metadata row: "Added by" + "Last updated" (formatted date)
- Description block
- Rendered SKILL.md body in a bordered card with a source/rendered
  toggle (eye + code icons, matching Claude.ai's segmented control)

No form fields, no save/cancel buttons. The user reads the skill
first and takes action deliberately.

**Create is now a dialog.** The `/skills/new` route is gone.
`CreateSkillMenu` (the + dropdown in the sidebar) now opens
`CreateSkillDialog` — a minimal modal with name, description, and
instructions fields. Upload-from-file still works: parse → populate
dialog → create. Matches Claude.ai's "Write skill instructions"
modal.

**Edit is behind an action.** The detail view shows an "Edit" button
(permission-gated) that navigates to `/skills/:id/edit`, rendering
the existing `SkillForm`. The edit route is preserved for direct
linking.

**Navigation goes to detail, not edit.** `SkillListItem` now
navigates to `/skills/:id` (detail) instead of `/skills/:id/edit`.

- `display/SkillMarkdownRenderer.tsx` — shared ReactMarkdown
  component extracted from `SkillContentEditor`. Same remark/rehype
  plugins, no form dependency.
- `display/SkillDetail.tsx` — the reader-first view (replaces the
  old thin wrapper).
- `dialogs/CreateSkillDialog.tsx` — OGDialog modal for skill
  creation.

- `layouts/SkillsView.tsx` — gutted and rebuilt. Three states:
  no-skill (empty state), skillId (SkillDetail), skillId+edit
  (SkillForm). Removed full-page CreateSkillForm, removed TreeView.
- `buttons/CreateSkillMenu.tsx` — opens dialog instead of navigating
  to `/skills/new`. Upload flow: parse → set dialog defaults → open.
- `lists/SkillListItem.tsx` — navigate to detail, not edit.
- `routes/index.tsx` — removed `/skills/new` and file/nodeId routes;
  `/skills` renders SkillsView directly (empty state).
- `display/index.ts`, `dialogs/index.ts` — added new exports.
- `locales/en/translation.json` — added ~10 new keys for metadata,
  toggle labels, dialog title, empty state.

* 🩹 fix: SkillContentEditor click-to-edit z-index — button was z-0 behind rendered content

* 🩹 fix: Align Edit button size with Share/Delete (size-9)

* 🎨 feat: Claude.ai-Style Skill List Panel

Rewrites the skills sidebar to match Claude.ai's panel layout:

- Header: "Skills" title + search icon (toggles input) + add icon
  (opens CreateSkillDialog directly, no dropdown menu)
- Collapsible "Skills" section with chevron toggle
- Skill items: 24px icon badge (rounded square with ScrollText icon)
  + name only. No description text in the list — that lives in the
  detail view. Active item gets highlighted bg + bold font.
- Removed AdminSettings button from sidebar header — admin config
  is accessible via the admin dashboard, not cluttering every user's
  skill list.
- Removed FilterSkills wrapper (was Filter + AdminSettings +
  CreateSkillMenu). The search + create are now inline in the panel
  header.

Files changed:
- sidebar/SkillsSidePanel.tsx — full rewrite
- sidebar/SkillsAccordion.tsx — simplified wrapper
- lists/SkillList.tsx — collapsible section, no description
- lists/SkillListItem.tsx — icon badge + name, memo'd

* 🎨 fix: Align Skills UI Styling with Prompts Patterns

Style alignment pass based on direct comparison with claude.ai and
the existing prompts preview dialog.

SkillsSidePanel search now replaces the title in the header row when
toggled (search icon + input + X close), matching Claude.ai's pattern.
Previously it pushed a separate input below the header, wasting
vertical space. Close button clears the search term.

Replaced `text-text-tertiary` with `text-text-secondary` across
SkillDetail, SkillList, SkillForm, CreateSkillForm, CreateSkillDialog,
SkillContentEditor. Tertiary was too dark / low contrast.

SkillList section chevron label now reads "Personal skills" (matching
Claude.ai) via the existing `com_ui_my_skills` key, instead of the
generic "Skills" which duplicated the header.

Aligned with `PromptDetailHeader` styling:
- 48px round icon (ScrollText in bg-surface-secondary circle)
- Name + public badge in the icon row
- Metadata below the icon: User icon + author, Calendar icon + date
  (text-xs text-text-secondary with gap-3, matching prompts exactly)
- Description uses the same label-above-text pattern as prompts
- Content card uses `bg-transparent` border (not bg-surface-primary-alt)
- Toggle buttons use size-5 icons and text-text-secondary for inactive

Changed from `max-w-lg p-0` to `max-w-5xl` with the same max-height
and padding pattern as the prompts PreviewPrompt dialog:
`max-h-[80vh] p-1 sm:p-2 gap-3 sm:gap-4`. Close button now renders
via default OGDialogContent behavior (removed showCloseButton=false).

* 🩹 fix: SkillDetail fills parent height, tighter spacing (px-6 pb-6 gap-2)

* 🩹 fix: Align Skills panel header padding (px-4) with list content below

* 🩹 fix: Reduce Skills header top padding (pt-2) to align with sidebar icon strip

* 🩹 fix: Tighten Skills header (py-2) and detail top (py-2) to align with sidebar icons and match edit view

* 🩹 fix: Offset SidePanel Nav pt-2 with -mt-2 on SkillsAccordion so Skills header aligns with icon strip

* 🛠️ fix: Increase Node memory limit for production build in package.json

* 🩹 fix: Remove top padding from SkillDetail header row (py-2 → pb-2)

* 🏗️ refactor: Move pt-2 from SidePanel/Nav wrapper to each panel

Removed the global `pt-2` from `SidePanel/Nav.tsx` and pushed it
into each panel's own top-level wrapper. This lets each panel own
its vertical alignment independently — Skills can sit flush at the
top to align with the sidebar icon strip, while other panels keep
their original spacing.

Panels updated with `pt-2`:
- PromptsAccordion (via className on PromptSidePanel)
- BookmarkPanel
- FilesPanel
- MemoryPanel
- MCPBuilderPanel
- AgentPanel (form wrapper)
- AssistantPanel (form wrapper)
- ParametersPanel (already had pt-2)

SkillsAccordion: removed the -mt-2 hack, now naturally flush.

* 🧹 fix: Align CreateSkillDialog field styling + remove 19 unused i18n keys

Dialog fields: all three inputs now use consistent `rounded-xl
border-border-medium px-3 py-2 text-sm` styling. Replaced the
`<Input>` component with a plain `<input>` to avoid the component's
built-in `rounded-lg border-border-light` overriding the dialog's
border style. Labels use `font-medium` for consistency.

Removed 19 unused translation keys from translation.json:
com_ui_skill_body, com_ui_skill_body_placeholder,
com_ui_skill_create_subtitle, com_ui_skill_file_delete_confirm,
com_ui_skill_file_delete_error, com_ui_skill_file_deleted,
com_ui_skill_files_empty, com_ui_skill_files_multi_hint,
com_ui_skill_list, com_ui_skill_load_error,
com_ui_skill_resize_file_tree, com_ui_skill_select_file,
com_ui_skill_select_file_desc, com_ui_skills_load_error,
com_ui_add_first_skill, com_ui_create_skill_page,
com_ui_edit_skill_page, com_ui_save_skill, com_ui_no_skills_title

* 🎁 feat: Upload Skill Dialog + Simplified Create Menu

New `UploadSkillDialog` matching Claude.ai's upload modal:
- Dashed drop zone with drag-and-drop support
- Accepts .md, .zip, .skill files
- Phase 1: processes .md files (parses YAML frontmatter → creates
  skill with body as the full file content)
- Shows file requirements below the drop zone
- On success: navigates to the new skill's detail view

`CreateSkillMenu` now has two flat options (no sub-menu):
- "Write skill instructions" → opens `CreateSkillDialog`
- "Upload a skill" → opens `UploadSkillDialog`

Removed the disabled "Create with AI" option and the old file input
hidden-element approach. The sidebar `+` button now renders
`CreateSkillMenu` directly instead of a standalone create dialog.

- Removed 5 unused i18n keys (com_ui_skill_added_by,
  com_ui_skill_last_updated, com_ui_skills_add_first,
  com_ui_skills_filter_placeholder, com_ui_skills_new)
- Tightened metadata gap in SkillDetail (mt-1 → mt-0.5)
- Added 7 new upload-related i18n keys

* 🔒 feat: Zip/Skill File Upload Support with Safety Limits

Rewrites UploadSkillDialog to properly handle all three accepted
file types:

- `.md` — reads as text, parses YAML frontmatter, creates skill
- `.zip` / `.skill` — reads as ArrayBuffer, extracts with JSZip,
  finds SKILL.md (at root or one level deep), parses its content,
  creates skill. Shows spinner during processing.

Security guards against zip bombs:
- MAX_ZIP_SIZE: 50MB compressed file limit
- MAX_ENTRIES: 500 file limit inside the archive
- Path traversal rejection: skips entries with `..` or leading `/`
- SKILL.md search limited to depth ≤ 2 segments

Added `jszip@^3.10.1` to client dependencies (already in the
monorepo's node_modules from backend usage).

The name is inferred from the zip filename if SKILL.md frontmatter
doesn't have one (e.g. `skills-autofix.zip` → `skills-autofix`).

* 🚀 feat: Backend Skill Import + Live File Upload Endpoints

New endpoint that accepts a single multipart file (.md, .zip, .skill)
and creates a skill with all its files in one request:

- **.md**: parse YAML frontmatter → create skill with body
- **.zip / .skill**: extract with JSZip, find SKILL.md (root or one
  level deep), create skill from its content, then persist every
  additional file via `upsertSkillFile` + local file storage strategy.
  Returns the created skill + an `_importSummary` with per-file
  results.

Security:
- 50MB compressed file size limit (multer)
- 500 max entries in archive
- 10MB per individual file
- Path traversal rejection (no `..`, no absolute, validated charset)
- File type filter: only .md/.zip/.skill accepted
- Rate limited via existing `fileUploadIpLimiter` + `fileUploadUserLimiter`

Handler lives in `packages/api/src/skills/import.ts` with injectable
deps (`createSkill`, `upsertSkillFile`, `saveBuffer`) for testability.

Replaced the 501 stub with a real handler:
- Accepts multipart FormData with `file` + `relativePath`
- Saves file via local storage strategy
- Calls `upsertSkillFile` to persist the SkillFile record
- Returns the upserted document
- Rate limited, ACL-gated (EDIT permission required)
- 10MB per file limit

`UploadSkillDialog` now sends the file to `/api/skills/import` via
`dataService.importSkill(formData)` — no more client-side JSZip.
Removed `jszip` from client dependencies (only backend needs it).

Added `importSkill()` in data-service + `importSkill()` endpoint
builder in api-endpoints.

Updated the file upload test from expecting 501 stub to expecting
400 "no file provided" (live validation). All 25 skill route tests
pass.

* 🔒 fix: Complete Import Handler — Validation, Ownership, Error Surfacing

Fixes several gaps in the skill import flow:

1. **Skill validation now runs and surfaces properly.** The import
   handler calls the real `createSkill(CreateSkillInput)` which runs
   `validateSkillName`, `validateSkillDescription`, `validateSkillBody`.
   Validation errors (SKILL_VALIDATION_FAILED) are caught and returned
   as 400 with the issue messages. Duplicate-key errors return 409.
   Previously all errors were swallowed into a generic 500.

2. **`authorName` is now populated.** The `CreateSkillInput` requires
   `authorName` which was missing — resolved from `req.user.name ??
   req.user.username ?? 'Unknown'`, matching the existing create handler.

3. **SKILL_OWNER permission is granted after import.** Calls
   `grantPermission` with `AccessRoleIds.SKILL_OWNER` so the uploader
   can edit/delete/share the imported skill. This was entirely missing —
   imported skills would have been ownerless.

4. **`tenantId` propagated.** Both the skill and each SkillFile record
   receive `req.user.tenantId` for multi-tenant deployments.

5. **SkillFile records are created in the DB.** Each non-SKILL.md file
   in the zip is saved to file storage via `saveBuffer` and recorded
   via `upsertSkillFile`, which validates the relativePath, infers the
   category from the path prefix, and atomically bumps the skill's
   `fileCount` and `version`.

Import deps now include `grantPermission` from PermissionService,
injected in `api/server/routes/skills.js`.

* 🐛 fix: Import grant uses accessRoleId (not roleId) — fixes skill not appearing in list

* 🎨 fix: Cache invalidation, file tree, frontmatter rendering

Three fixes for the skill detail view:

1. **Cache invalidation after import.** UploadSkillDialog now calls
   `queryClient.invalidateQueries([QueryKeys.skills])` after a
   successful import so the sidebar list picks up the new skill
   without requiring a page refresh.

2. **File tree in detail view.** When a skill has `fileCount > 0`,
   the detail view now queries `useListSkillFilesQuery` and renders
   a file list below the body card — SKILL.md first, then folders
   and root files. Icons: Folder for directories, FileText for files.

3. **Frontmatter stripped and rendered as metadata.** YAML frontmatter
   (`---\nversion: 0.1.0\ntriggers: ...\n---`) is now parsed out of
   the body before markdown rendering. The `name` and `description`
   fields are skipped (already shown in the header). Remaining fields
   (version, triggers, dependencies, etc.) are displayed in a
   Claude.ai–style grid: label on the left, value on the right,
   above the rendered markdown content. Source view still shows the
   full raw body including frontmatter.

* 🩹 fix: Always fetch skill files — fileCount may be stale in cached skill object

* 🌳 feat: Inline File Tree in Sidebar Skill List

Moves the file tree from the bottom of SkillDetail into the sidebar
list, matching Claude.ai's pattern:

- Multi-file skills show a chevron toggle on the right side of the
  skill list item
- Clicking the chevron expands an inline file tree below the skill
  name: SKILL.md first, then folders (with folder icon + right
  chevron) and root files
- File list is fetched lazily (only when expanded) via
  useListSkillFilesQuery
- Clicking a file navigates to the skill detail view
- Files section removed from SkillDetail — the sidebar is now the
  sole file tree location, keeping the detail panel clean

SkillDetail cleaned up: removed groupFiles helper, file-related
state, useListSkillFilesQuery import, FileText/Folder icon imports.

* 🌲 feat: Virtualized inline file tree with react-vtree

Replace hand-rolled recursive FolderRow/FileRow buttons with a proper
virtualized FixedSizeTree from react-vtree for the sidebar skill list.
Dynamic height tracks open folders; capped at 350px with smooth
expand/collapse transitions.

* chore: Remove no longer used SkillFileTree and SkillTreeNode components

* chore: Update Vite config to replace 'react-arborist' with 'react-vtree' for module resolution

* feat: Skill file content viewing with lazy DB caching

- Add `skills` field to `fileStrategiesSchema` so operators can
  configure a dedicated storage backend for skill files. Falls back
  by type (image/document) when unset.
- Fix hardcoded `FileSources.local` in skill save/import — now uses
  the resolved strategy via `getFileStrategy(req.config, { context })`.
- Replace 501 download stub with real handler that streams from any
  storage backend and returns JSON `{ content, mimeType, isBinary }`.
- Binary detection (null-byte + non-printable ratio on first 8 KB)
  flags files on first read so they're never re-fetched.
- Text content ≤ 512 KB is cached in the SkillFile MongoDB document;
  subsequent reads skip storage entirely.
- Clicking a skill row now expands inline files (not just chevron).
- Clicking a file navigates to `?file=<path>` and renders content
  in a new SkillFileViewer (markdown, code, images, binary placeholder).

* chore: Remove react-window and its type definitions from package.json and package-lock.json

- Deleted `react-window` and `@types/react-window` dependencies from both `package.json` and `package-lock.json` to streamline the project and reduce unnecessary bloat.

* fix: Build errors — remove endpoints import, fix Uint8Array cast

- Replace `import { endpoints }` (not public) with inline URL in
  SkillFileViewer
- Remove `as Uint8Array` cast in stream chunk handling
- Extend getSkillFileByPath return type with content/isBinary to
  decouple from data-schemas build artifact resolution

* chore: Remove 8 unused i18next keys

com_ui_create_skill_ai, com_ui_create_skill_manual,
com_ui_delete_folder_confirm_var, com_ui_delete_skill,
com_ui_delete_skill_confirm_var, com_ui_delete_var,
com_ui_rename_var, com_ui_skill_files

* fix: Add configMiddleware to skills router, handle SKILL.md in viewer

- Add configMiddleware to skills router so req.config is populated
  when getLocalFileStream (or any strategy) reads file paths.
- Handle SKILL.md in download handler — serves skill.body directly
  from the Skill document instead of looking for a SkillFile record.
- Clicking SKILL.md in sidebar tree now opens the file viewer
  (matching Claude.ai behavior: file view vs default detail view).

* ci: Run unit tests on PRs to any branch

Remove the branches filter from both test workflows so contributor
PRs targeting feature branches (not just main/dev) get CI coverage.
Path filters are kept so tests only run when relevant files change.

* fix: Update skills route tests for download handler changes

- Mock configMiddleware (sets req.config for file storage access)
- Mock getStrategyFunctions and getFileStrategy (storage strategy deps)
- Replace 501 stub test with SKILL.md content test + 404 test

* fix: Auto-expand files, frontmatter parsing, select-none, prefetch

- Auto-expand file tree when navigating directly to a skill URL
- Prefetch files for the active skill (eliminates first-expand lag)
- Fix frontmatter parser to handle multi-line YAML list values
  (triggers field was missing because it uses list syntax)
- SkillFileViewer now parses frontmatter for .md files — shows
  structured grid + rendered body (matching SkillDetail's display)
  with source/rendered toggle
- Add select-none to all sidebar skill and file tree buttons

* refactor: Derive expanded state from isActive instead of useEffect

Replace useEffect sync with deterministic derivation:
expanded = hasFiles && (isActive || !collapsed)

Active skill is always open. collapsed is a manual toggle that
only takes effect on non-active items.

* fix: Remove empty space above body card — overlay view toggle

Move the rendered/source toggle from a dedicated row (40px of empty
space) to an absolute-positioned overlay in the card's top-right
corner, matching Claude.ai's layout.

* fix: Remove header bars from content editors — overlay action buttons

Collapse the full-width header bars ("Skill Content", "Text") in
SkillContentEditor, PromptTextCard, and PromptEditor. Action buttons
(edit/save toggle, copy, variables) are now absolute-positioned in
the card's top-right corner, reclaiming ~46px of vertical space.

* fix: Spinner visibility in file viewer — use text-text-secondary

* fix: Address review findings — security, correctness, code quality

Codex P1: Use $unset instead of undefined to clear cached content
and isBinary fields on file re-upload (Mongoose strips undefined).

Codex P2: Match skill-file validation errors by error.code instead
of error.message substring.

F1: Zip bomb defense — track cumulative decompressed bytes (500 MB
cap), check declared uncompressed size before buffering each entry.

F2: Remove misleading "atomically" from import handler JSDoc.

F3: Static import for isBinaryBuffer instead of dynamic import().

F4: Replace console.error with logger in upload handler.

F6: Add multer error handler middleware to skills router.

F7: Move React import to top of SkillDetail.tsx.

F9: Fix variable shadowing (trimmed → item) in parseFrontmatter.

F11: Replace JSON.parse(JSON.stringify()) with toJSON() for
Mongoose document serialization.

F12: Remove dead dynamic import('fs') fallback (memoryStorage
always provides file.buffer).

F13: Hoist MIME_MAP to module scope to avoid per-call allocation.

F16: Share single multer.memoryStorage() instance.

* fix: Follow-up review — close zip bomb gap, fix error handler

F1: Add post-decompression cumulative byte check with break (the
pre-decompression check relies on undocumented JSZip internals
that may be absent; this closes the gap unconditionally).

F2+F3: Multer error handler now forwards non-multer errors via
next(err) instead of swallowing them. Also catches file filter
rejections (plain Error, not MulterError) by message prefix.

F4: Move isBinaryBuffer import to local imports section per
CLAUDE.md import order rules.

F5: Simplify dead toJSON branch — createSkill returns a POJO.

* nit: Link filter error message to handler prefix check

* feat: Accordion expansion + active file highlight in sidebar

- Only one skill's file tree can be expanded at a time (accordion).
  Expansion state lifted from SkillListItem to SkillList.
- Selected file gets bg-surface-active highlight in the tree.
  Skill row uses subtle style (no background) when a file is active,
  matching Claude.ai's pattern where the file — not the skill —
  carries the selection state.

* style: Adjust margin for file tree in SkillListItem component

- Reduced left margin from 10 to 5 for improved layout consistency in the file tree display.

* fix: TS error on FileTreeNode, nested ternary, chevron collapse

- Make style prop optional to match react-vtree's NodeComponentProps
- Flatten nested ternary for skill row active styles
- Skill row click expands (but doesn't collapse) files + navigates
- Chevron click explicitly toggles collapse (matching Claude.ai
  where clicking the chevron is how you collapse files)

* fix: Upload basePath, reject SKILL.md uploads, add skills permission route

- Pass basePath: 'uploads' in per-file upload handler (was defaulting
  to 'images' path, inconsistent with the import flow).
- Reject uploads targeting SKILL.md (reserved path — download handler
  special-cases it to return skill.body, making an uploaded file
  unreachable via the API).
- Add skills entry to roles router permissionConfigs so PUT
  /api/roles/:roleName/skills actually reaches a handler instead
  of returning 404.

* feat: Expand content area, move controls to header, reduce padding

Default detail view:
- Remove rounded-xl bordered card wrapper — content flows directly
  into the article, capitalizing on full screen width
- Move eye/code toggle inline with the divider row
- Reduce px-6/pb-6 to px-4/pb-4

File viewer:
- Move eye/code toggle from card overlay to the header bar
- Add copy-to-clipboard button for text files in the header bar
- Remove rounded-xl bordered card wrapper for markdown content
- Remove bordered pre wrapper for non-markdown text
- Reduce px-6/py-4 to px-4/py-3

Both views maximize content space over decorative chrome.

* fix: Stable header height, restore some padding

- Fix layout shift in file viewer header: use fixed h-10 so the
  bar height stays constant whether the eye/code toggle renders
  (markdown) or not (plain text).
- Bump content padding from px-4/py-3 back to px-5/py-4 in both
  views — the previous reduction was too aggressive.

* fix: Grant rollback, path validation, error format, dead code cleanup

F2: grantOwnership now rolls back (compensating delete) on failure,
matching the create handler. Both markdown and zip import paths
check the result and return 500 on grant failure.

F4: Upload handler validates relativePath with regex + traversal
check before calling downstream upsertSkillFile.

F5: Document JSZip _data.uncompressedSize as best-effort; the
post-decompression cumulative check is the real safety net.

F10: Standardize all upload handler error responses to { error }
(was { message }, inconsistent with handlers.ts).

F13: Single-pass fileResults accumulation in import handler.

F1-5: Remove dead uploadFileStubHandler (no route references it).

Codex P2: Fix delete nav from /skills/new to /skills.

F12: Use cn() in UploadSkillDialog instead of template literals.

* perf: Stream-first binary detection + O(1) public skill check

F1: Download handler now reads only the first 8 KB for binary
detection. If binary, the stream is destroyed immediately without
buffering the remaining file. Text files continue reading for
caching. Eliminates buffering up to 10 MB per request for binary
files under concurrent load.

F7: Single-skill GET and PATCH now use hasPublicPermission (O(1)
ACL lookup) instead of getPublicSkillIdSet (queries ALL public
skill IDs). The list handler still uses the Set approach since it
serializes multiple skills. serializeSkill/serializeSkillSummary
now accept boolean | Set for flexibility.

* fix: Update test to match { error } response format

* fix: Critical stream truncation bug, grantedBy, error format

NF-1 (CRITICAL): Rewrite binary detection to single for-await loop.
Breaking out of for-await-of destroys the stream via iterator.return(),
so the previous two-loop approach silently truncated text files > 8KB.
Now: one loop collects chunks, checks binary after 8KB accumulated,
and either destroys+returns (binary) or continues reading (text).

NF-2: Add grantedBy to import handler's grantPermission call and
interface (was missing, inconsistent with create handler).

NF-3: Standardize all import handler error responses from { message }
to { error }, matching handlers.ts convention. Update client's
UploadSkillDialog to read response.data.error accordingly.

* fix: Prefer specific validation message over generic error field

* fix: YAML quote stripping, saveBuffer null guard, dot segment rejection

- Strip surrounding YAML quotes from frontmatter values so
  name: "my-skill" parses as my-skill (not "my-skill" with quotes
  that fails the name validator).
- Guard resolveSkillStorage against backends with saveBuffer: null
  (e.g. OpenAI/vector strategies) — throws a descriptive error
  caught by the handler's try/catch instead of a TypeError.
- Tighten upload path validation to reject . segments (e.g.
  docs/./a.md) matching the model-layer validator, preventing
  storage writes for paths the DB will reject.

* fix: Orphan cleanup, stream errors, malformed zip, cache latency

F1: Upload handler now deletes the stored blob if the subsequent
DB upsert fails, preventing orphaned files on disk/cloud.

F2: Multer error handler returns { error } (was { message }).

F3: Wrap JSZip.loadAsync in try/catch — malformed zip returns 400
instead of falling through to 500.

F4: Raw download stream gets an error handler — logs the error and
destroys the response if headers were already sent.

F8: Strip leading hyphens from inferred skill name so filenames
like _my-skill.zip don't produce -my-skill (invalid name pattern).

F9: Fire-and-forget all updateSkillFileContent cache writes so the
response is sent immediately. Cache failures are logged but don't
block or fail the read.

* fix: Import orphan cleanup + Content-Disposition sanitization

Finding A: Add deleteFile dep to ImportSkillDeps. The per-file loop
in handleZip now cleans up stored blobs when upsertSkillFile fails,
closing the second half of the F1 orphan fix (upload handler was
already fixed).

Finding B: Sanitize filename in Content-Disposition header for raw
downloads — strip quotes, backslashes, and newlines to prevent
header injection from user-uploaded filenames.

* security: Prevent stored XSS via raw file downloads

Non-image files served via ?raw=true now use Content-Disposition:
attachment (force download) instead of inline. An uploaded .html or
.svg file served inline from the LibreChat origin could execute
scripts with access to the user's session — this closes that vector.

Images stay inline (needed for <img> rendering in SkillFileViewer).
X-Content-Type-Options: nosniff added to prevent MIME sniffing.

* security: Block SVG XSS — allowlist safe raster MIME types for inline

SVG (image/svg+xml) passed the startsWith('image/') check and was
served inline, but SVG is a scriptable format — embedded <script>
tags execute in the LibreChat origin. Replace the prefix match with
a Set of safe raster-only MIME types (png, jpeg, gif, webp, avif,
bmp). SVGs and any future scriptable image/* subtypes now get
Content-Disposition: attachment (forced download).

* fix: Cap JSON text response at 1MB, consistent md name inference

F3: Text files > 1MB now return { isBinary: false } with no content
field, forcing the client to use ?raw=true for download. Prevents
buffering 10MB files into heap for JSON serialization. Frontend
shows a download fallback when content is absent.

F4: handleMarkdown now infers skill name from filename (same as
handleZip) when frontmatter has no name, instead of rejecting
with 400. Consistent behavior across import paths.

F1 (reviewer concern): upsertSkillFile is NOT affected — it uses
{ new: false } for insert-vs-replace detection but does a follow-up
findOne (lines 855-859) to return the post-upsert document.

* fix: deleteFile arg shape, raw URL base path, hoist SAFE_INLINE_MIMES

Codex P2: deleteFile expects { filepath } object, not a raw string.
Both upload handler cleanup and import handler cleanup now pass
{ filepath } to match the strategy contract (deleteLocalFile,
deleteFileFromS3 all expect a file object).

Codex P2: Raw download URL in SkillFileViewer now uses apiBaseUrl
prefix so subpath deployments (/chat, etc.) resolve correctly.

NIT: Hoist SAFE_INLINE_MIMES Set to factory scope — was re-allocated
per raw download request inside the if block.

* fix: Remove inert cache write for large text files, localize aria-label

N2: The { isBinary: false } cache write for text files > 1MB had no
effect — subsequent requests still fell through to stream read since
neither isBinary nor content provided a fast-path short-circuit.
Removed the pointless DB updateOne per request.

N4: Replace hardcoded "Back to skill" aria-label with localize().

* refactor: Extract shared parseFrontmatter, widen deleteFile type

N3: Extract parseFrontmatter into Skills/utils/frontmatter.ts —
single implementation shared by SkillDetail and SkillFileViewer.
Accepts optional skipKeys set so callers control which frontmatter
fields are excluded (SkillDetail skips name/description since
they're shown in the header; other .md files show all fields).

N5: Widen ImportSkillDeps.deleteFile file param from { filepath }
to { filepath; [key: string]: unknown } to signal extensibility
if strategies start accessing additional file properties.

* fix: Advance i past list items for skipped keys, DRY parseSkillMd

Finding A: parseFrontmatter now consumes multi-line YAML list items
before checking skipKeys — prevents list lines from leaking into
subsequent key parsing as spurious fields.

Finding B: parseSkillMd now delegates to the shared parseFrontmatter
instead of re-implementing the same frontmatter scanning loop.
Reduces client-side parseFrontmatter implementations from 3 to 1.

* fix: Call apiBaseUrl(), delete storage blob on file removal

- apiBaseUrl is a function, not a string — call it in the template
  literal so raw download URLs resolve correctly.
- deleteFileHandler now looks up the file record before deleting,
  then fire-and-forget deletes the storage blob via the strategy's
  deleteFile. Previously only the DB record was removed, leaving
  orphaned blobs in local/S3/Firebase/Azure storage.

* fix: Clean up storage blobs when deleting an entire skill

deleteHandler now lists all files for the skill before calling
deleteSkill, then fire-and-forget deletes each blob via the
storage strategy. Previously only per-file deletion cleaned up
blobs — deleting a whole skill left all associated files orphaned
in local/S3/Firebase/Azure storage.

* refactor: useImportSkillMutation hook, fix TSkill[] unsafe cast

- Create useImportSkillMutation in mutations.ts + ImportSkillOptions
  type. UploadSkillDialog now uses the mutation hook instead of
  calling dataService.importSkill directly with manual useState
  loading management. Eliminates unmounted-component state update
  risk and aligns with the React Query mutation pattern used by
  every other mutation in the codebase.

- SkillSelectDialog: replace as unknown as TSkill[] with proper
  TSkillSummary typing. SkillCard props updated to TSkillSummary.
  The dialog only uses summary-level fields (name, description,
  category, author) — the cast was hiding a type mismatch.

* fix: Use saved source for import cleanup, delete old blob on replace

Codex P2: Import cleanup now uses file.source (the backend the file
was actually saved to) instead of re-resolving from config. In mixed
strategy setups, the previous approach could target the wrong backend.

Codex P2: When re-uploading a file to an existing relativePath, the
old blob is now deleted after successful upsert. Previously only the
DB record was replaced, leaving the old storage object orphaned.

* fix: Register PUT /:roleName/skills route in roles router

* fix: Re-read skill after zip file processing for fresh metadata

The import response was built from the skill object created before
the file loop, but each upsertSkillFile bumps version and fileCount.
Clients caching the stale response would get 409 conflicts on first
edit and see incorrect file counts.

Now re-reads the skill via getSkillById after the loop so the
response reflects the current version, fileCount, and updatedAt.

* fix: Size-check SKILL.md before decompression, don't gate on fileCount

P1: SKILL.md was decompressed before any size accounting. A crafted
archive could expand SKILL.md past 10MB before validation ran. Now
checks declared size pre-decompression and actual size post, both
against MAX_SINGLE_FILE_BYTES.

P2: File list query was gated on cached fileCount which can be stale
after mutations. Now fetches files for the active skill regardless
of fileCount. hasFiles derived from fetched data with fileCount as
fallback, so newly uploaded files appear without hard refresh.

* fix: Move files declaration before hasFiles to avoid TDZ error

* security: Stream-decompress zip entries with enforced byte cap

Replace zipEntry.async('nodebuffer') (buffers entire entry before
checking limits) with zipEntry.nodeStream('nodebuffer') piped
through a byte counter that destroys the stream when the per-file
or cumulative limit is exceeded.

Previously, when JSZip's _data.uncompressedSize was absent (the
common case), a high-ratio entry could allocate hundreds of MB
before the post-decompression check caught it. Now decompression
is aborted mid-stream at the exact byte threshold — no entry can
exceed its limit regardless of compression ratio.

* refactor: Reorganize access check for prompts in useSideNavLinks hook

Moved the prompts access check to a new position in the code to improve readability and maintainability. This change ensures that the prompts link is added to the navigation only if the user has the appropriate access, without altering the existing functionality.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-04-25 04:02:00 -04:00
Danny Avila
15fc27950d
⚡ refactor: Short-Circuit Config Override Resolution (#12553) 2026-04-07 22:38:08 -04:00
Danny Avila
ea28dbfa89
🧹 chore: Clean Up Config Fields (#12537)
* chore: remove unused `interface.endpointsMenu` config field

* chore: address review — restore JSDoc UI-only example, add Zod strip test

* chore: remove unused `interface.sidePanel` config field

* chore: restrict fileStrategy/fileStrategies schema to valid storage backends

* fix: use valid FileStorage value in AppService test

* chore: address review — version bump, exhaustiveness guard, JSDoc, configSchema test

* chore: remove debug logger.log from MessageIcon render path

* fix: rewrite MessageIcon render tests to use render counting instead of logger spying

* chore: bump librechat-data-provider to 0.8.407

* chore: sync example YAML version to 1.3.7
2026-04-03 12:22:58 -04:00
Danny Avila
fda72ac621
🏗️ refactor: Remove Redundant Caching, Migrate Config Services to TypeScript (#12466)
* ♻️ refactor: Remove redundant scopedCacheKey caching, support user-provided key model fetching

Remove redundant cache layers that used `scopedCacheKey()` (tenant-only scoping)
on top of `getAppConfig()` which already caches per-principal (role+user+tenant).
This caused config overrides for different principals within the same tenant to
be invisible due to stale cached data.

Changes:
- Add `requireJwtAuth` to `/api/endpoints` route for proper user context
- Remove ENDPOINT_CONFIG, STARTUP_CONFIG, PLUGINS, TOOLS, and MODELS_CONFIG
  cache layers — all derive from `getAppConfig()` with cheap computation
- Enhance MODEL_QUERIES cache: hash(baseURL+apiKey) keys, 2-minute TTL,
  caching centralized in `fetchModels()` base function
- Support fetching models with user-provided API keys in `loadConfigModels`
  via `getUserKeyValues` lookup (no caching for user keys)
- Update all affected tests

Closes #1028

* ♻️ refactor: Migrate config services to TypeScript in packages/api

Move core config logic from CJS /api wrappers to typed TypeScript in
packages/api using dependency injection factories:

- `createEndpointsConfigService` — endpoint config merging + checkCapability
- `createLoadConfigModels` — custom endpoint model loading with user key support
- `createMCPToolCacheService` — MCP tool cache operations (update, merge, cache)

/api files become thin wrappers that wire dependencies (getAppConfig,
loadDefaultEndpointsConfig, getUserKeyValues, getCachedTools, etc.)
into the typed factories.

Also moves existing `endpoints/config.ts` → `endpoints/config/providers.ts`
to accommodate the new `config/` directory structure.

* 🔄 fix: Invalidate models query when user API key is set or revoked

Without this, users had to refresh the page after entering their API key
to see the updated model list fetched with their credentials.

- Invalidate QueryKeys.models in useUpdateUserKeysMutation onSuccess
- Invalidate QueryKeys.models in useRevokeUserKeyMutation onSuccess
- Invalidate QueryKeys.models in useRevokeAllUserKeysMutation onSuccess

* 🗺️ fix: Remap YAML-level override keys to AppConfig equivalents in mergeConfigOverrides

Config overrides stored in the DB use YAML-level keys (TCustomConfig),
but they're merged into the already-processed AppConfig where some fields
have been renamed by AppService. This caused mcpServers overrides to land
on a nonexistent key instead of mcpConfig, so config-override MCP servers
never appeared in the UI.

- Add OVERRIDE_KEY_MAP to remap mcpServers→mcpConfig, interface→interfaceConfig
- Apply remapping before deep merge in mergeConfigOverrides
- Add test for YAML-level key remapping behavior
- Update existing tests to use AppConfig field names in assertions

* 🧪 test: Update service.spec to use AppConfig field names after override key remapping

* 🛡️ fix: Address code review findings — reliability, types, tests, and performance

- Pass tenant context (getTenantId) in importers.js getEndpointsConfig call
- Add 5 tests for user-provided API key model fetching (key found, no key,
  DB error, missing userId, apiKey-only with fixed baseURL)
- Distinguish NO_USER_KEY (debug) from infrastructure errors (warn) in catch
- Switch fetchPromisesMap from Promise.all to Promise.allSettled so one
  failing provider doesn't kill the entire model config
- Parallelize getUserKeyValues DB lookups via batched Promise.allSettled
  instead of sequential awaits in the loop
- Hoist standardCache instance in fetchModels to avoid double instantiation
- Replace Record<string, unknown> types with Partial<TConfig>-based types;
  remove as unknown as T double-cast in endpoints config
- Narrow Bedrock availableRegions to typed destructure
- Narrow version field from string|number|undefined to string|undefined
- Fix import ordering in mcp/tools.ts and config/models.ts per AGENTS.md
- Add JSDoc to getModelsConfig alias clarifying caching semantics

* fix: Guard against null getCachedTools in mergeAppTools

* 🔍 fix: Address follow-up review — deduplicate extractEnvVariable, fix error discrimination, add log-level tests

- Deduplicate extractEnvVariable calls: resolve apiKey/baseURL once, reuse
  for both the entry and isUserProvided checks (Finding A)
- Move ResolvedEndpoint interface from function closure to module scope (Finding B)
- Replace fragile msg.includes('NO_USER_KEY') with ErrorTypes.NO_USER_KEY
  enum check against actual error message format (Finding C). Also handle
  ErrorTypes.INVALID_USER_KEY as an expected "no key" case.
- Add test asserting logger.warn is called for infra errors (not debug)
- Add test asserting logger.debug is called for NO_USER_KEY errors (not warn)

* fix: Preserve numeric assistants version via String() coercion

* 🐛 fix: Address secondary review — Ollama cache bypass, cache tests, type safety

- Fix Ollama success path bypassing cache write in fetchModels (CRITICAL):
  store result before returning so Ollama models benefit from 2-minute TTL
- Add 4 fetchModels cache behavior tests: cache write with TTL, cache hit
  short-circuits HTTP, skipCache bypasses read+write, empty results not cached
- Type-safe OVERRIDE_KEY_MAP: Partial<Record<keyof TCustomConfig, keyof AppConfig>>
  so compiler catches future field rename mismatches
- Fix import ordering in config/models.ts (package types longest→shortest)
- Rename ToolCacheDeps → MCPToolCacheDeps for naming consistency
- Expand getModelsConfig JSDoc to explain caching granularity

* fix: Narrow OVERRIDE_KEY_MAP index to satisfy strict tsconfig

* 🧩 fix: Add allowedProviders to TConfig, remove Record<string, unknown> from PartialEndpointEntry

The agents endpoint config includes allowedProviders (used by the frontend
AgentPanel to filter available providers), but it was missing from TConfig.
This forced PartialEndpointEntry to use & Record<string, unknown> as an
escape hatch, violating AGENTS.md type policy.

- Add allowedProviders?: (string | EModelEndpoint)[] to TConfig
- Remove Record<string, unknown> from PartialEndpointEntry — now just Partial<TConfig>

* 🛡️ fix: Isolate Ollama cache write from fetch try-catch, add Ollama cache tests

- Separate Ollama fetch and cache write into distinct scopes so a cache
  failure (e.g., Redis down) doesn't misattribute the error as an Ollama
  API failure and fall through to the OpenAI-compatible path (Issue A)
- Add 2 Ollama-specific cache tests: models written with TTL on fetch,
  cached models returned without hitting server (Issue B)
- Replace hardcoded 120000 with Time.TWO_MINUTES constant in cache TTL
  test assertion (Issue C)
- Fix OVERRIDE_KEY_MAP JSDoc to accurately describe runtime vs compile-time
  type enforcement (Issue D)
- Add global beforeEach for cache mock reset to prevent cross-test leakage

* 🧪 fix: Address third review — DI consistency, cache key width, MCP tests

- Inject loadCustomEndpointsConfig via EndpointsConfigDeps with default
  fallback, matching loadDefaultEndpointsConfig DI pattern (Finding 3)
- Widen modelsCacheKey from 64-bit (.slice(0,16)) to 128-bit (.slice(0,32))
  for collision-sensitive cross-credential cache key (Finding 4)
- Add fetchModels.mockReset() in loadConfigModels.spec beforeEach to
  prevent mock implementation leaks across tests (Finding 5)
- Add 11 unit tests for createMCPToolCacheService covering all three
  functions: null/empty input, successful ops, error propagation,
  cold-cache merge (Finding 2)
- Simplify getModelsConfig JSDoc to @see reference (Finding 10)

* ♻️ refactor: Address remaining follow-ups from reviews

OVERRIDE_KEY_MAP completeness:
- Add missing turnstile→turnstileConfig mapping
- Add exhaustiveness test verifying all three renamed keys are remapped
  and original YAML keys don't leak through

Import role context:
- Pass userRole through importConversations job → importLibreChatConvo
  so role-based endpoint overrides are honored during conversation import
- Update convos.js route to include req.user.role in the job payload

createEndpointsConfigService unit tests:
- Add 8 tests covering: default+custom merge, Azure/AzureAssistants/
  Anthropic Vertex/Bedrock config enrichment, assistants version
  coercion, agents allowedProviders, req.config bypass

Plugins/tools efficiency:
- Use Set for includedTools/filteredTools lookups (O(1) vs O(n) per plugin)
- Combine auth check + filter into single pass (eliminates intermediate array)
- Pre-compute toolDefKeys Set for O(1) tool definition lookups

* fix: Scope model query cache by user when userIdQuery is enabled

* fix: Skip model cache for userIdQuery endpoints, fix endpoints test types

- When userIdQuery is true, skip caching entirely (like user_provided keys)
  to avoid cross-user model list leakage without duplicating cache data
- Fix AgentCapabilities type error in endpoints.spec.ts — use enum values
  and appConfig() helper for partial mock typing

* 🐛 fix: Restore filteredTools+includedTools composition, add checkCapability tests

- Fix filteredTools regression: whitelist and blacklist are now applied
  independently (two flat guards), matching original behavior where
  includedTools=['a','b'] + filteredTools=['b'] produces ['a'] (Finding A)
- Fix Set spread in toolkit loop: pre-compute toolDefKeysList array once
  alongside the Set, reuse for .some() without per-plugin allocation (Finding B)
- Add 2 filteredTools tests: blacklist-only path and combined
  whitelist+blacklist composition (Finding C)
- Add 3 checkCapability tests: capability present, capability absent,
  fallback to defaultAgentCapabilities for non-agents endpoints (Finding D)

* 🔑 fix: Include config-override MCP servers in filterAuthorizedTools

Config-override MCP servers (defined via admin config overrides for
roles/groups) were rejected by filterAuthorizedTools because it called
getAllServerConfigs(userId) without the configServers parameter. Only
YAML and DB-backed user servers were included in the access check.

- Add configServers parameter to filterAuthorizedTools
- Resolve config servers via resolveConfigServers(req) at all 4 callsites
  (create, update, duplicate, revert) using parallel Promise.all
- Pass configServers through to getAllServerConfigs(userId, configServers)
  so the registry merges config-source servers into the access check
- Update filterAuthorizedTools.spec.js mock for resolveConfigServers

* fix: Skip model cache for userIdQuery endpoints, fix endpoints test types

For user-provided key endpoints (userProvide: true), skip the full model
list re-fetch during message validation — the user already selected from
a list we served them, and re-fetching with skipCache:true on every
message send is both slow and fragile (5s provider timeout = rejected model).

Instead, validate the model string format only:
- Must be a string, max 256 chars
- Must match [a-zA-Z0-9][a-zA-Z0-9_.:\-/@+ ]* (covers all known provider
  model ID formats while rejecting injection attempts)

System-configured endpoints still get full model list validation as before.

* 🧪 test: Add regression tests for filterAuthorizedTools configServers and validateModel

filterAuthorizedTools:
- Add test verifying configServers is passed to getAllServerConfigs and
  config-override server tools are allowed through
- Guard resolveConfigServers in createAgentHandler to only run when
  MCP tools are present (skip for tool-free agent creates)

validateModel (12 new tests):
- Format validation: missing model, non-string, length overflow, leading
  special char, script injection, standard model ID acceptance
- userProvide early-return: next() called immediately, getModelsConfig
  not invoked (regression guard for the exact bug this fixes)
- System endpoint list validation: reject unknown model, accept known
  model, handle null/missing models config

Also fix unnecessary backslash escape in MODEL_PATTERN regex.

* 🧹 fix: Remove space from MODEL_PATTERN, trim input, clean up nits

- Remove space character from MODEL_PATTERN regex — no real model ID
  uses spaces; prevents spurious violation logs from whitespace artifacts
- Add model.trim() before validation to handle accidental whitespace
- Remove redundant filterUniquePlugins call on already-deduplicated output
- Add comment documenting intentional whitelist+blacklist composition
- Add getUserKeyValues.mockReset() in loadConfigModels.spec beforeEach
- Remove narrating JSDoc from getModelsConfig one-liner
- Add 2 tests: trim whitespace handling, reject spaces in model ID

* fix: Match startup tool loader semantics — includedTools takes precedence over filteredTools

The startup tool loader (loadAndFormatTools) explicitly ignores
filteredTools when includedTools is set, with a warning log. The
PluginController was applying both independently, creating inconsistent
behavior where the same config produced different results at startup
vs plugin listing time.

Restored mutually exclusive semantics: when includedTools is non-empty,
filteredTools is not evaluated.

* 🧹 chore: Simplify validateModel flow, note auth requirement on endpoints route

- Separate missing-model from invalid-model checks cleanly: type+presence
  guard first, then trim+format guard (reviewer NIT)
- Add route comment noting auth is required for role/tenant scoping

* fix: Write trimmed model back to req.body.model for downstream consumers
2026-03-30 16:49:48 -04:00
Danny Avila
fda1bfc3cc
🔬 ci: Add TypeScript Type Checks to Backend Workflow and Fix All Type Errors (#12451)
* fix(data-schemas): resolve TypeScript strict type check errors in source files

- Constrain ConfigSection to string keys via `string & keyof TCustomConfig`
- Replace broken `z` import from data-provider with TCustomConfig derivation
- Add `_id: Types.ObjectId` to IUser matching other Document interfaces
- Add `federatedTokens` and `openidTokens` optional fields to IUser
- Type mongoose model accessors as `Model<IRole>` and `Model<IUser>`
- Widen `getPremiumRate` param to accept `number | null`
- Widen `bulkWriteAclEntries` ops to untyped `AnyBulkWriteOperation[]`
- Fix `getUserPrincipals` return type to use `PrincipalType` enum
- Add non-null assertions for `connection.db` in migration files
- Import DailyRotateFile constructor directly instead of relying on
  broken module augmentation across mismatched node_modules trees
- Add winston-daily-rotate-file as devDependency for type resolution

* fix(data-schemas): resolve TypeScript type errors in test files

- Replace arbitrary test keys with valid TCustomConfig properties in config.spec
- Use non-null assertions for permission objects in role.methods.spec
- Replace `.SHARED_GLOBAL` access with `.not.toHaveProperty()` for legacy field
- Add non-null assertions for balance, writeRate, readRate in spendTokens.spec
- Update mock user _id to use ObjectId in user.test
- Remove unused Schema import in tenantIndexes.spec

* fix(api): resolve TypeScript strict type check errors across source and test files

- Widen getUserPrincipals dep type in capabilities middleware
- Fix federatedTokens type in createSafeUser return
- Use proper mock req type for read-only properties in preAuthTenant.spec
- Replace `as IUser` casts with ObjectId-typed mocks in openid/oidc specs
- Use TokenExchangeMethodEnum values instead of string literals in MCP specs
- Fix SessionStore type compatibility in sessionCache specs
- Replace `catch (error: any)` with `(error as Error)` in redis specs
- Remove invalid properties from test data in initialize and MCP specs
- Add String.prototype.isWellFormed declaration for sanitizeTitle spec

* fix(client): resolve TypeScript type errors in shared client components

- Add default values for destructured bindings in OGDialogTemplate
- Replace broken ExtendedFile import with inline type in FileIcon

* ci: add TypeScript type-check job to backend review workflow

Add a `typecheck` job that runs `tsc --noEmit` on all four TypeScript
workspaces (data-provider, data-schemas, @librechat/api, @librechat/client)
after the build step. Catches type errors that rollup builds may miss.

* fix(data-schemas): add local type declaration for DailyRotateFile transport

The `winston-daily-rotate-file` package ships a module augmentation for
`winston/lib/winston/transports`, but it fails when winston and
winston-daily-rotate-file resolve from different node_modules trees
(which happens in this monorepo due to npm hoisting).

Add a local `.d.ts` declaration that augments the same module path from
within data-schemas' compilation unit, so `tsc --noEmit` passes while
keeping the original runtime pattern (`new winston.transports.DailyRotateFile`).

* fix: address code review findings from PR #12451

- Restore typed `AnyBulkWriteOperation<AclEntry>[]` on bulkWriteAclEntries,
  cast to untyped only at the tenantSafeBulkWrite call site (Finding 1)
- Type `findUser` model accessor consistently with `findUsers` (Finding 2)
- Replace inline `import('mongoose').ClientSession` with top-level import type
- Use `toHaveLength` for spy assertions in playwright-expect spec file
- Replace numbered Record casts with `.not.toHaveProperty()` in
  role.methods.spec for SHARED_GLOBAL assertions
- Use per-test ObjectIds instead of shared testUserId in openid.spec
- Replace inline `import()` type annotations with top-level SessionData
  import in sessionCache spec
- Remove extraneous blank line in user.ts searchUsers

* refactor: address remaining review findings (4–7)

- Extract OIDCTokens interface in user.ts; deduplicate across IUser fields
  and oidc.ts FederatedTokens (Finding 4)
- Move String.isWellFormed declaration from spec file to project-level
  src/types/es2024-string.d.ts (Finding 5)
- Replace verbose `= undefined` defaults in OGDialogTemplate with null
  coalescing pattern (Finding 6)
- Replace `Record<string, unknown>` TestConfig with named interface
  containing explicit test fields (Finding 7)
2026-03-28 21:06:39 -04:00
Danny Avila
877c2efc85
🏗️ feat: bulkWrite isolation, pre-auth context, strict-mode fixes (#12445)
* fix: wrap seedDatabase() in runAsSystem() for strict tenant mode

seedDatabase() was called without tenant context at startup, causing
every Mongoose operation inside it to throw when
TENANT_ISOLATION_STRICT=true. Wrapping in runAsSystem() gives it the
SYSTEM_TENANT_ID sentinel so the isolation plugin skips filtering,
matching the pattern already used for performStartupChecks and
updateInterfacePermissions.

* fix: chain tenantContextMiddleware in optionalJwtAuth

optionalJwtAuth populated req.user but never established ALS tenant
context, unlike requireJwtAuth which chains tenantContextMiddleware
after successful auth. Authenticated users hitting routes with
optionalJwtAuth (e.g. /api/banner) had no tenant isolation.

* feat: tenant-safe bulkWrite wrapper and call-site migration

Mongoose's bulkWrite() does not trigger schema-level middleware hooks,
so the applyTenantIsolation plugin cannot intercept it. This adds a
tenantSafeBulkWrite() utility that injects the current ALS tenant
context into every operation's filter/document before delegating to
native bulkWrite.

Migrates all 8 runtime bulkWrite call sites:
- agentCategory (seedCategories, ensureDefaultCategories)
- conversation (bulkSaveConvos)
- message (bulkSaveMessages)
- file (batchUpdateFiles)
- conversationTag (updateTagsForConversation, bulkIncrementTagCounts)
- aclEntry (bulkWriteAclEntries)

systemGrant.seedSystemGrants is intentionally not migrated — it uses
explicit tenantId: { $exists: false } filters and is exempt from the
isolation plugin.

* feat: pre-auth tenant middleware and tenant-scoped config cache

Adds preAuthTenantMiddleware that reads X-Tenant-Id from the request
header and wraps downstream in tenantStorage ALS context. Wired onto
/oauth, /api/auth, /api/config, and /api/share — unauthenticated
routes that need tenant scoping before JWT auth runs.

The /api/config cache key is now tenant-scoped
(STARTUP_CONFIG:${tenantId}) so multi-tenant deployments serve the
correct login page config per tenant.

The middleware is intentionally minimal — no subdomain parsing, no
OIDC claim extraction. The private fork's reverse proxy or auth
gateway sets the header.

* feat: accept optional tenantId in updateInterfacePermissions

When tenantId is provided, the function re-enters inside
tenantStorage.run({ tenantId }) so all downstream Mongoose queries
target that tenant's roles instead of the system context. This lets
the private fork's tenant provisioning flow call
updateInterfacePermissions per-tenant after creating tenant-scoped
ADMIN/USER roles.

* fix: tenant-filter $lookup in getPromptGroup aggregation

The $lookup stage in getPromptGroup() queried the prompts collection
without tenant filtering. While the outer PromptGroup aggregate is
protected by the tenantIsolation plugin's pre('aggregate') hook,
$lookup runs as an internal MongoDB operation that bypasses Mongoose
hooks entirely.

Converts from simple field-based $lookup to pipeline-based $lookup
with an explicit tenantId match when tenant context is active.

* fix: replace field-level unique indexes with tenant-scoped compounds

Field-level unique:true creates a globally-unique single-field index in
MongoDB, which would cause insert failures across tenants sharing the
same ID values.

- agent.id: removed field-level unique, added { id, tenantId } compound
- convo.conversationId: removed field-level unique (compound at line 50
  already exists: { conversationId, user, tenantId })
- message.messageId: removed field-level unique (compound at line 165
  already exists: { messageId, user, tenantId })
- preset.presetId: removed field-level unique, added { presetId, tenantId }
  compound

* fix: scope MODELS_CONFIG, ENDPOINT_CONFIG, PLUGINS, TOOLS caches by tenant

These caches store per-tenant configuration (available models, endpoint
settings, plugin availability, tool definitions) but were using global
cache keys. In multi-tenant mode, one tenant's cached config would be
served to all tenants.

Appends :${tenantId} to cache keys when tenant context is active.
Falls back to the unscoped key when no tenant context exists (backward
compatible for single-tenant OSS deployments).

Covers all read, write, and delete sites:
- ModelController.js: get/set MODELS_CONFIG
- PluginController.js: get/set PLUGINS, get/set TOOLS
- getEndpointsConfig.js: get/set/delete ENDPOINT_CONFIG
- app.js: delete ENDPOINT_CONFIG (clearEndpointConfigCache)
- mcp.js: delete TOOLS (updateMCPTools, mergeAppTools)
- importers.js: get ENDPOINT_CONFIG

* fix: add getTenantId to PluginController spec mock

The data-schemas mock was missing getTenantId, causing all
PluginController tests to throw when the controller calls
getTenantId() for tenant-scoped cache keys.

* fix: address review findings — migration, strict-mode, DRY, types

Addresses all CRITICAL, MAJOR, and MINOR review findings:

F1 (CRITICAL): Add agents, conversations, messages, presets to
SUPERSEDED_INDEXES in tenantIndexes.ts so dropSupersededTenantIndexes()
drops the old single-field unique indexes that block multi-tenant inserts.

F2 (CRITICAL): Unknown bulkWrite op types now throw in strict mode
instead of silently passing through without tenant injection.

F3 (MAJOR): Replace wildcard export with named export for
tenantSafeBulkWrite, hiding _resetBulkWriteStrictCache from the
public package API.

F5 (MAJOR): Restore AnyBulkWriteOperation<IAclEntry>[] typing on
bulkWriteAclEntries — the unparameterized wrapper accepts parameterized
ops as a subtype.

F7 (MAJOR): Fix config.js tenant precedence — JWT-derived
req.user.tenantId now takes priority over the X-Tenant-Id header for
authenticated requests.

F8 (MINOR): Extract scopedCacheKey() helper into tenantContext.ts and
replace all 11 inline occurrences across 7 files.

F9 (MINOR): Use simple localField/foreignField $lookup for the
non-tenant getPromptGroup path (more efficient index seeks).

F12 (NIT): Remove redundant BulkOp type alias.
F13 (NIT): Remove debug log that leaked raw tenantId.

* fix: add new superseded indexes to tenantIndexes test fixture

The test creates old indexes to verify the migration drops them.
Missing fixture entries for agents.id_1, conversations.conversationId_1,
messages.messageId_1, and presets.presetId_1 caused the count assertion
to fail (expected 22, got 18).

* fix: restore logger.warn for unknown bulk op types in non-strict mode

* fix: block SYSTEM_TENANT_ID sentinel from external header input

CRITICAL: preAuthTenantMiddleware accepted any string as X-Tenant-Id,
including '__SYSTEM__'. The tenantIsolation plugin treats SYSTEM_TENANT_ID
as an explicit bypass — skipping ALL query filters. A client sending
X-Tenant-Id: __SYSTEM__ to pre-auth routes (/api/share, /api/config,
/api/auth, /oauth) would execute Mongoose operations without tenant
isolation.

Fixes:
- preAuthTenantMiddleware rejects SYSTEM_TENANT_ID in header
- scopedCacheKey returns the base key (not key:__SYSTEM__) in system
  context, preventing stale cache entries during runAsSystem()
- updateInterfacePermissions guards tenantId against SYSTEM_TENANT_ID
- $lookup pipeline separates $expr join from constant tenantId match
  for better index utilization
- Regression test for sentinel rejection in preAuthTenant.spec.ts
- Remove redundant getTenantId() call in config.js

* test: add missing deleteMany/replaceOne coverage, fix vacuous ALS assertions

bulkWrite spec:
- deleteMany: verifies tenant-scoped deletion leaves other tenants untouched
- replaceOne: verifies tenantId injected into both filter and replacement
- replaceOne overwrite: verifies a conflicting tenantId in the replacement
  document is overwritten by the ALS tenant (defense-in-depth)
- empty ops array: verifies graceful handling

preAuthTenant spec:
- All negative-case tests now use the capturedNext pattern to verify
  getTenantId() inside the middleware's execution context, not the
  test runner's outer frame (which was always undefined regardless)

* feat: tenant-isolate MESSAGES cache, FLOWS cache, and GenerationJobManager

MESSAGES cache (streamAudio.js):
- Cache key now uses scopedCacheKey(messageId) to prefix with tenantId,
  preventing cross-tenant message content reads during TTS streaming.

FLOWS cache (FlowStateManager):
- getFlowKey() now generates ${type}:${tenantId}:${flowId} when tenant
  context is active, isolating OAuth flow state per tenant.

GenerationJobManager:
- tenantId added to SerializableJobData and GenerationJobMetadata
- createJob() captures the current ALS tenant context (excluding
  SYSTEM_TENANT_ID) and stores it in job metadata
- SSE subscription endpoint validates job.metadata.tenantId matches
  req.user.tenantId, blocking cross-tenant stream access
- Both InMemoryJobStore and RedisJobStore updated to accept tenantId

* fix: add getTenantId and SYSTEM_TENANT_ID to MCP OAuth test mocks

FlowStateManager.getFlowKey() now calls getTenantId() for tenant-scoped
flow keys. The 4 MCP OAuth test files mock @librechat/data-schemas
without these exports, causing TypeError at runtime.

* fix: correct import ordering per AGENTS.md conventions

Package imports sorted shortest to longest line length, local imports
sorted longest to shortest — fixes ordering violations introduced by
our new imports across 8 files.

* fix: deserialize tenantId in RedisJobStore — cross-tenant SSE guard was no-op in Redis mode

serializeJob() writes tenantId to the Redis hash via Object.entries,
but deserializeJob() manually enumerates fields and omitted tenantId.
Every getJob() from Redis returned tenantId: undefined, causing the
SSE route's cross-tenant guard to short-circuit (undefined && ... → false).

* test: SSE tenant guard, FlowStateManager key consistency, ALS scope docs

SSE stream tenant tests (streamTenant.spec.js):
- Cross-tenant user accessing another tenant's stream → 403
- Same-tenant user accessing own stream → allowed
- OSS mode (no tenantId on job) → tenant check skipped

FlowStateManager tenant tests (manager.tenant.spec.ts):
- completeFlow finds flow created under same tenant context
- completeFlow does NOT find flow under different tenant context
- Unscoped flows are separate from tenant-scoped flows

Documentation:
- JSDoc on getFlowKey documenting ALS context consistency requirement
- Comment on streamAudio.js scopedCacheKey capture site

* fix: SSE stream tests hang on success path, remove internal fork references

The success-path tests entered the SSE streaming code which never
closes, causing timeout. Mock subscribe() to end the response
immediately. Restructured assertions to verify non-403/non-404.

Removed "private fork" and "OSS" references from code and test
descriptions — replaced with "deployment layer", "multi-tenant
deployments", and "single-tenant mode".

* fix: address review findings — test rigor, tenant ID validation, docs

F1: SSE stream tests now mock subscribe() with correct signature
(streamId, writeEvent, onDone, onError) and assert 200 status,
verifying the tenant guard actually allows through same-tenant users.

F2: completeFlow logs the attempted key and ALS tenantId when flow
is not found, so reverse proxy misconfiguration (missing X-Tenant-Id
on OAuth callback) produces an actionable warning.

F3/F10: preAuthTenantMiddleware validates tenant ID format — rejects
colons, special characters, and values exceeding 128 chars. Trims
whitespace. Prevents cache key collisions via crafted headers.

F4: Documented cache invalidation scope limitation in
clearEndpointConfigCache — only the calling tenant's key is cleared;
other tenants expire via TTL.

F7: getFlowKey JSDoc now lists all 8 methods requiring consistent
ALS context.

F8: Added dedicated scopedCacheKey unit tests — base key without
context, base key in system context, scoped key with tenant, no
ALS leakage across scope boundaries.

* fix: revert flow key tenant scoping, fix SSE test timing

FlowStateManager: Reverts tenant-scoped flow keys. OAuth callbacks
arrive without tenant ALS context (provider redirects don't carry
X-Tenant-Id), so completeFlow/failFlow would never find flows
created under tenant context. Flow IDs are random UUIDs with no
collision risk, and flow data is ephemeral (TTL-bounded).

SSE tests: Use process.nextTick for onDone callback so Express
response headers are flushed before res.write/res.end are called.

* fix: restore getTenantId import for completeFlow diagnostic log

* fix: correct completeFlow warning message, add missing flow test

The warning referenced X-Tenant-Id header consistency which was only
relevant when flow keys were tenant-scoped (since reverted). Updated
to list actual causes: TTL expiry, missing flow, or routing to a
different instance without shared Keyv storage.

Removed the getTenantId() call and import — no longer needed since
flow keys are unscoped.

Added test for the !flowState branch in completeFlow — verifies
return false and logger.warn on nonexistent flow ID.

* fix: add explicit return type to recursive updateInterfacePermissions

The recursive call (tenantId branch calls itself without tenantId)
causes TypeScript to infer circular return type 'any'. Adding
explicit Promise<void> satisfies the rollup typescript plugin.

* fix: update MCPOAuthRaceCondition test to match new completeFlow warning

* fix: clearEndpointConfigCache deletes both scoped and unscoped keys

Unauthenticated /api/endpoints requests populate the unscoped
ENDPOINT_CONFIG key. Admin config mutations clear only the
tenant-scoped key, leaving the unscoped entry stale indefinitely.
Now deletes both when in tenant context.

* fix: tenant guard on abort/status endpoints, warn logs, test coverage

F1: Add tenant guard to /chat/status/:conversationId and /chat/abort
matching the existing guard on /chat/stream/:streamId. The status
endpoint exposes aggregatedContent (AI response text) which requires
tenant-level access control.

F2: preAuthTenantMiddleware now logs warn for rejected __SYSTEM__
sentinel and malformed tenant IDs, providing observability for
bypass probing attempts.

F3: Abort fallback path (getActiveJobIdsForUser) now has tenant
check after resolving the job.

F4: Test for strict mode + SYSTEM_TENANT_ID — verifies runAsSystem
bypasses tenantSafeBulkWrite without throwing in strict mode.

F5: Test for job with tenantId + user without tenantId → 403.

F10: Regex uses idiomatic hyphen-at-start form.

F11: Test descriptions changed from "rejects" to "ignores" since
middleware calls next() (not 4xx).

Also fixes MCPOAuthRaceCondition test assertion to match updated
completeFlow warning message.

* fix: test coverage for logger.warn, status/abort guards, consistency

A: preAuthTenant spec now mocks logger and asserts warn calls for
__SYSTEM__ sentinel, malformed characters, and oversized headers.

B: streamTenant spec expanded with status and abort endpoint tests —
cross-tenant status returns 403, same-tenant returns 200 with body,
cross-tenant abort returns 403.

C: Abort endpoint uses req.user.tenantId (not req.user?.tenantId)
matching stream/status pattern — requireJwtAuth guarantees req.user.

D: Malformed header warning now includes ip in log metadata,
matching the sentinel warning for consistent SOC correlation.

* fix: assert ip field in malformed header warn tests

* fix: parallelize cache deletes, document tenant guard, fix import order

- clearEndpointConfigCache uses Promise.all for independent cache
  deletes instead of sequential awaits
- SSE stream tenant guard has inline comment explaining backward-compat
  behavior for untenanted legacy jobs
- conversation.ts local imports reordered longest-to-shortest per
  AGENTS.md

* fix: tenant-qualify userJobs keys, document tenant guard backward-compat

Job store userJobs keys now include tenantId when available:
- Redis: stream:user:{tenantId:userId}:jobs (falls back to
  stream:user:{userId}:jobs when no tenant)
- InMemory: composite key tenantId:userId in userJobMap

getActiveJobIdsByUser/getActiveJobIdsForUser accept optional tenantId
parameter, threaded through from req.user.tenantId at all call sites
(/chat/active and /chat/abort fallback).

Added inline comments on all three SSE tenant guards explaining the
backward-compat design: untenanted legacy jobs remain accessible
when the userId check passes.

* fix: parallelize cache deletes, document tenant guard, fix import order

Fix InMemoryJobStore.getActiveJobIdsByUser empty-set cleanup to use
the tenant-qualified userKey instead of bare userId — prevents
orphaned empty Sets accumulating in userJobMap for multi-tenant users.

Document cross-tenant staleness in clearEndpointConfigCache JSDoc —
other tenants' scoped keys expire via TTL, not active invalidation.

* fix: cleanup userJobMap leak, startup warning, DRY tenant guard, docs

F1: InMemoryJobStore.cleanup() now removes entries from userJobMap
before calling deleteJob, preventing orphaned empty Sets from
accumulating with tenant-qualified composite keys.

F2: Startup warning when TENANT_ISOLATION_STRICT is active — reminds
operators to configure reverse proxy to control X-Tenant-Id header.

F3: mergeAppTools JSDoc documents that tenant-scoped TOOLS keys are
not actively invalidated (matching clearEndpointConfigCache pattern).

F5: Abort handler getActiveJobIdsForUser call uses req.user.tenantId
(not req.user?.tenantId) — consistent with stream/status handlers.

F6: updateInterfacePermissions JSDoc clarifies SYSTEM_TENANT_ID
behavior — falls through to caller's ALS context.

F7: Extracted hasTenantMismatch() helper, replacing three identical
inline tenant guard blocks across stream/status/abort endpoints.

F9: scopedCacheKey JSDoc documents both passthrough cases (no context
and SYSTEM_TENANT_ID context).

* fix: clean userJobMap in evictOldest — same leak as cleanup()
2026-03-28 16:43:50 -04:00
Danny Avila
77712c825f
🏢 feat: Tenant-Scoped App Config in Auth Login Flows (#12434)
* feat: add resolveAppConfigForUser utility for tenant-scoped auth config

TypeScript utility in packages/api that wraps getAppConfig in
tenantStorage.run() when the user has a tenantId, falling back to
baseOnly for new users or non-tenant deployments. Uses DI pattern
(getAppConfig passed as parameter) for testability.

Auth flows apply role-level overrides only (userId not passed)
because user/group principal resolution is deferred to post-auth.

* feat: tenant-scoped app config in auth login flows

All auth strategies (LDAP, SAML, OpenID, social login) now use a
two-phase domain check consistent with requestPasswordReset:

1. Fast-fail with base config (memory-cached, zero DB queries)
2. DB user lookup
3. Tenant-scoped re-check via resolveAppConfigForUser (only when
   user has a tenantId; otherwise reuse base config)

This preserves the original fast-fail protection against globally
blocked domains while enabling tenant-specific config overrides.

OpenID error ordering preserved: AUTH_FAILED checked before domain
re-check so users with wrong providers get the correct error type.

registerUser unchanged (baseOnly, no user identity yet).

* test: add tenant-scoped config tests for auth strategies

Add resolveAppConfig.spec.ts in packages/api with 8 tests:
- baseOnly fallback for null/undefined/no-tenant users
- tenant-scoped config with role and tenantId
- ALS context propagation verified inside getAppConfig callback
- undefined role with tenantId edge case

Update strategy and AuthService tests to mock resolveAppConfigForUser
via @librechat/api. Tests verify two-phase domain check behavior:
fast-fail before DB, tenant re-check after. Non-tenant users reuse
base config without calling resolveAppConfigForUser.

* refactor: skip redundant domain re-check for non-tenant users

Guard the second isEmailDomainAllowed call with appConfig !== baseConfig
in SAML, OpenID, and social strategies. For non-tenant users the tenant
config is the same base config object, so the second check is a no-op.

Narrow eslint-disable in resolveAppConfig.spec.ts to the specific
require line instead of blanket file-level suppression.

* fix: address review findings — consistency, tests, and ordering

- Consolidate duplicate require('@librechat/api') in AuthService.js
- Add two-phase domain check to LDAP (base fast-fail before findUser),
  making all strategies consistent with PR description
- Add appConfig !== baseConfig guard to requestPasswordReset second
  domain check, consistent with SAML/OpenID/social strategies
- Move SAML provider check before tenant config resolution to avoid
  unnecessary resolveAppConfigForUser call for wrong-provider users
- Add tenant domain rejection tests to SAML, OpenID, and social specs
  verifying that tenant config restrictions actually block login
- Add error propagation tests to resolveAppConfig.spec.ts
- Remove redundant mockTenantStorage alias in resolveAppConfig.spec.ts
- Narrow eslint-disable to specific require line

* test: add tenant domain rejection test for LDAP strategy

Covers the appConfig !== baseConfig && !isEmailDomainAllowed path,
consistent with SAML, OpenID, and social strategy specs.

* refactor: rename resolveAppConfig to app/resolve per AGENTS.md

Rename resolveAppConfig.ts → resolve.ts and
resolveAppConfig.spec.ts → resolve.spec.ts to align with
the project's concise naming convention.

* fix: remove fragile reference-equality guard, add logging and docs

Remove appConfig !== baseConfig guard from all strategies and
requestPasswordReset. The guard relied on implicit cache-backend
identity semantics (in-memory Keyv returns same object reference)
that would silently break with Redis or cloned configs. The second
isEmailDomainAllowed call is a cheap synchronous check — always
running it is clearer and eliminates the coupling.

Add audit logging to requestPasswordReset domain blocks (base and
tenant), consistent with all auth strategies.

Extract duplicated error construction into makeDomainDeniedError().

Wrap resolveAppConfigForUser in requestPasswordReset with try/catch
to prevent DB errors from leaking to the client via the controller's
generic catch handler.

Document the dual tenantId propagation (ALS for DB isolation,
explicit param for cache key) in resolveAppConfigForUser JSDoc.

Add comment documenting the LDAP error-type ordering change
(cross-provider users from blocked domains now get 'domain not
allowed' instead of AUTH_FAILED).

Assert resolveAppConfigForUser is not called on LDAP provider
mismatch path.

* fix: return generic response for tenant domain block in password reset

Tenant-scoped domain rejection in requestPasswordReset now returns the
same generic "If an account with that email exists..." response instead
of an Error. This prevents user-enumeration: an attacker cannot
distinguish between "email not found" and "tenant blocks this domain"
by comparing HTTP responses.

The base-config fast-fail (pre-user-lookup) still returns an Error
since it fires before any user existence is revealed.

* docs: document phase 1 vs phase 2 domain check behavior in JSDoc

Phase 1 (base config, pre-findUser) intentionally returns Error/400
to reveal globally blocked domains without confirming user existence.
Phase 2 (tenant config, post-findUser) returns generic 200 to prevent
user-enumeration. This distinction is now explicit in the JSDoc.
2026-03-27 16:08:43 -04:00
Danny Avila
9f6d8c6e93
🧵 feat: ALS Context Middleware, Tenant Threading, and Config Cache Invalidation (#12407)
* feat: add tenant context middleware for ALS-based isolation

Introduces tenantContextMiddleware that propagates req.user.tenantId
into AsyncLocalStorage, activating the Mongoose applyTenantIsolation
plugin for all downstream DB queries within a request.

- Strict mode (TENANT_ISOLATION_STRICT=true) returns 403 if no tenantId
- Non-strict mode passes through for backward compatibility
- No-op for unauthenticated requests
- Includes 6 unit tests covering all paths

* feat: register tenant middleware and wrap startup/auth in runAsSystem()

- Register tenantContextMiddleware in Express app after capability middleware
- Wrap server startup initialization in runAsSystem() for strict mode compat
- Wrap auth strategy getAppConfig() calls in runAsSystem() since they run
  before user context is established (LDAP, SAML, OpenID, social login, AuthService)

* feat: thread tenantId through all getAppConfig callers

Pass tenantId from req.user to getAppConfig() across all callers that
have request context, ensuring correct per-tenant cache key resolution.

Also fixes getBaseConfig admin endpoint to scope to requesting admin's
tenant instead of returning the unscoped base config.

Files updated:
- Controllers: UserController, PluginController
- Middleware: checkDomainAllowed, balance
- Routes: config
- Services: loadConfigModels, loadDefaultModels, getEndpointsConfig, MCP
- Audio services: TTSService, STTService, getVoices, getCustomConfigSpeech
- Admin: getBaseConfig endpoint

* feat: add config cache invalidation on admin mutations

- Add clearOverrideCache(tenantId?) to flush per-principal override caches
  by enumerating Keyv store keys matching _OVERRIDE_: prefix
- Add invalidateConfigCaches() helper that clears base config, override
  caches, tool caches, and endpoint config cache in one call
- Wire invalidation into all 5 admin config mutation handlers
  (upsert, patch, delete field, delete overrides, toggle active)
- Add strict mode warning when __default__ tenant fallback is used
- Add 3 new tests for clearOverrideCache (all/scoped/base-preserving)

* chore: update getUserPrincipals comment to reflect ALS-based tenant filtering

The TODO(#12091) about missing tenantId filtering is resolved by the
tenant context middleware + applyTenantIsolation Mongoose plugin.
Group queries are now automatically scoped by tenantId via ALS.

* fix: replace runAsSystem with baseOnly for pre-tenant code paths

App configs are tenant-owned — runAsSystem() would bypass tenant
isolation and return cross-tenant DB overrides. Instead, add
baseOnly option to getAppConfig() that returns YAML-derived config
only, with zero DB queries.

All startup code, auth strategies, and MCP initialization now use
getAppConfig({ baseOnly: true }) to get the YAML config without
touching the Config collection.

* fix: address PR review findings — middleware ordering, types, cache safety

- Chain tenantContextMiddleware inside requireJwtAuth after passport auth
  instead of global app.use() where req.user is always undefined (Finding 1)
- Remove global tenantContextMiddleware registration from index.js
- Update BalanceMiddlewareOptions to include tenantId, remove redundant cast (Finding 4)
- Add warning log when clearOverrideCache cannot enumerate keys on Redis (Finding 3)
- Use startsWith instead of includes for cache key filtering (Finding 12)
- Use generator loop instead of Array.from for key enumeration (Finding 3)
- Selective barrel export — exclude _resetTenantMiddlewareStrictCache (Finding 5)
- Move isMainThread check to module level, remove per-request check (Finding 9)
- Move mid-file require to top of app.js (Finding 8)
- Parallelize invalidateConfigCaches with Promise.all (Finding 10)
- Remove clearOverrideCache from public app.js exports (internal only)
- Strengthen getUserPrincipals comment re: ALS dependency (Finding 2)

* fix: restore runAsSystem for startup DB ops, consolidate require, clarify baseOnly

- Restore runAsSystem() around performStartupChecks, updateInterfacePermissions,
  initializeMCPs, and initializeOAuthReconnectManager — these make Mongoose
  queries that need system context in strict tenant mode (NEW-3)
- Consolidate duplicate require('@librechat/api') in requireJwtAuth.js (NEW-1)
- Document that baseOnly ignores role/userId/tenantId in JSDoc (NEW-2)

* test: add requireJwtAuth tenant chaining + invalidateConfigCaches tests

- requireJwtAuth: 5 tests verifying ALS tenant context is set after
  passport auth, isolated between concurrent requests, and not set
  when user has no tenantId (Finding 6)
- invalidateConfigCaches: 4 tests verifying all four caches are cleared,
  tenantId is threaded through, partial failure is handled gracefully,
  and operations run in parallel via Promise.all (Finding 11)

* fix: address Copilot review — passport errors, namespaced cache keys, /base scoping

- Forward passport errors in requireJwtAuth before entering tenant
  middleware — prevents silent auth failures from reaching handlers (P1)
- Account for Keyv namespace prefix in clearOverrideCache — stored keys
  are namespaced as "APP_CONFIG:_OVERRIDE_:..." not "_OVERRIDE_:...",
  so override caches were never actually matched/cleared (P2)
- Remove role from getBaseConfig — /base should return tenant-scoped
  base config, not role-merged config that drifts per admin role (P2)
- Return tenantStorage.run() for cleaner async semantics
- Update mock cache in service.spec.ts to simulate Keyv namespacing

* fix: address second review — cache safety, code quality, test reliability

- Decouple cache invalidation from mutation response: fire-and-forget
  with logging so DB mutation success is not masked by cache failures
- Extract clearEndpointConfigCache helper from inline IIFE
- Move isMainThread check to lazy once-per-process guard (no import
  side effect)
- Memoize process.env read in overrideCacheKey to avoid per-request
  env lookups and log flooding in strict mode
- Remove flaky timer-based parallelism assertion, use structural check
- Merge orphaned double JSDoc block on getUserPrincipals
- Fix stale [getAppConfig] log prefix → [ensureBaseConfig]
- Fix import order in tenant.spec.ts (package types before local values)
- Replace "Finding 1" reference with self-contained description
- Use real tenantStorage primitives in requireJwtAuth spec mock

* fix: move JSDoc to correct function after clearEndpointConfigCache extraction

* refactor: remove Redis SCAN from clearOverrideCache, rely on TTL expiry

Redis SCAN causes 60s+ stalls under concurrent load (see #12410).
APP_CONFIG defaults to FORCED_IN_MEMORY_CACHE_NAMESPACES, so the
in-memory store.keys() path handles the standard case. When APP_CONFIG
is Redis-backed, overrides expire naturally via overrideCacheTtl (60s
default) — an acceptable window for admin config mutations.

* fix: remove return from tenantStorage.run to satisfy void middleware signature

* fix: address second review — cache safety, code quality, test reliability

- Switch invalidateConfigCaches from Promise.all to Promise.allSettled
  so partial failures are logged individually instead of producing one
  undifferentiated error (Finding 3)
- Gate overrideCacheKey strict-mode warning behind a once-per-process
  flag to prevent log flooding under load (Finding 4)
- Add test for passport error forwarding in requireJwtAuth — the
  if (err) { return next(err) } branch now has coverage (Finding 5)
- Add test for real partial failure in invalidateConfigCaches where
  clearAppConfigCache rejects (not just the swallowed endpoint error)

* chore: reorder imports in index.js and app.js for consistency

- Moved logger and runAsSystem imports to maintain a consistent import order across files.
- Improved code readability by ensuring related imports are grouped together.
2026-03-26 17:35:00 -04:00