Commit graph

54 commits

Author SHA1 Message Date
Marco Beretta
9adabde72e
🗂️ feat: Audit MCP, Code Worker, Langfuse, Skill Credential, Passkey and Audit Log Access (#17054)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: Remove Group Dependents Before Deleting the Group

* feat: Audit Code Worker Pairing and Langfuse Connection Changes

* feat: Audit Skill Sync Credential Changes

* feat: Audit Passkey Enrolment and Removal

* feat: Audit Audit Log Reads and CSV Exports

* feat: Audit Audit Log Retention Purges

* feat: Audit MCP Server Create, Update and Delete

* fix: Keep Purge Result When the Purge Audit Write Fails

* test: Cover Admin Integration Audit Scenarios

* fix: Tighten Admin Integration Audit Scope and Group Cleanup Order

* fix: Run Security Fences Before Awaiting Audit Writes

* fix: Move MCP Post-Commit Audit Ordering Into the Auditor and Audit Verify Reads

* fix: Fail Closed When Recording Audit Log Access

* fix: Audit Group Deletion Before the Cascade and Restore Dependents on Failure

* fix: Restore Only Config Overrides When Group Deletion Fails

* fix: Audit Worker Pairing and Export Before the Work and Append Failures

* fix: Record Group Deletion as Pending Without Delaying Revocation

* fix: Skip Config Restore When a Concurrent Delete Removed the Group

* fix: Restore Group Config When the Existence Recheck Fails

* fix: Keep Group Config Restricted When Delete State Is Unknown

* fix: Record Worker Pairing and Revocation as Pending Until Confirmed

* fix: Deliver the Pairing Code Before the Terminal Audit Write

* test: Assert Pairing Code Delivery Precedes the Terminal Audit Write

* fix: Send Revocation and Group Deletion Responses Before the Terminal Audit Write

* fix: Snapshot Group Config Before the Destructive Cascade Step

* refactor: Type the Group Config Snapshot Dependency Without Mongoose Shapes

* fix: Fence Shared MCP Users Before the Deletion Audit and Correlate Audit Attempts

* fix: Diff MCP Server Audits Against the Committed Config

* fix: Read the Committed MCP Config at Commit Time for the Audit Diff

* refactor: Start the Committed MCP Config Read Inside the Auditor

* fix: Audit Langfuse Patches When Activation Fails and Surface Failed Group Config Restores

* fix: Record Member Removals When Group Deletion Completes After a Failed Restore

* test: Serve the Committed Config on the MCP Update Audit Reread

* fix: Audit the Group Returned by the Compensating Deletion

* fix: Keep Member Removal Audits Fail-Open During Group Deletion

* fix: Retry ACL Cleanup After a Compensating Group Deletion

* fix: Mark Unavailable Audit Detail and Order Member Removals After the Pending Deletion

* fix: Guard Synchronous Committed-Config Read Failures in the MCP Auditor

* fix: Return the Persisted MCP Config and Audit Commits That Outlive Cache Invalidation

* style: Sort Imports in MCP Audit Files

* style: Sort Imports After Restack

* fix: Audit MCP Commits That Outlive Any Post-Write Step and Pend Audit Exports Until Complete

* test: Provide the Fail-Closed Setting in the Code Route Mock

* fix: Refresh MCP Read Caches When a Repository Step Fails After the Write

* fix: Finish Audit Exports on Response Completion and Keep Error Details Out of Messages and Logs

* fix: Watch Response Completion Before the Truncation Footer Write

* fix: Skip Audit for No-op MCP Deletes and Close Remaining Post-Failure Audit Gaps

* fix: Keep Causes Out of Structured Logs and Treat a Lost Compensating Delete as Not Found

* fix: Mark Failed MCP Post-Commit Steps and Null Langfuse Activation in Audits
2026-10-15 00:30:26 +02:00
Marco Beretta
76192bdfd0
🛂 feat: Audit Resource Sharing Changes and Fail Closed on Reversible Audit Writes (#17053)
* feat: Audit Resource ACL Changes and Fail Closed by Default

* fix: Keep Non-Reversible Admin Audit Handlers Fail-Open

* docs: Narrow AUDIT_LOG_FAIL_CLOSED Description

* fix: Audit and Restore ACL and Insights Changes in One Pass

* fix: Break Audit and Insights Permission Type Cycle

* test: Cover Resource ACL Audit Scenarios

* refactor: Audit Resource Permission Changes Before Writing

* test: Align Concurrent Revoke Audit Expectation

* fix: Skip Deleting ACL Entries That Were Not Audited

* fix: Read ACL Snapshots Through the Write Session

* fix: Never Block Revocations on Audit Outages and Normalize Principal Keys

* fix: Let an Explicit Public Disable Win Over a Public Update

* fix: Audit Duplicate ACL Rows as Combined Permissions

* fix: Audit Permission Changes Under One Target and Dedupe Revokes

* fix: Audit Capability Grants Before Writing Them

* feat: Serialize Resource Permission Updates With a Per-Resource Lease

* fix: Fence the Resource Lease Before Writes and Always Audit Grant Assignments

* fix: Renew the Resource Lease Before Each ACL Write Phase

* fix: Serialize System Grant Assignment and Revocation With a Per-Grant Lease

* fix: Keep Leases Alive During Audits, Bound Lock Index Wait, Canonicalize Audit Target IDs

* fix: Retry Deferred Revoke Audits Under the Lease and Audit Canonical Principal and Role IDs

* test: Expect Role IDs in Permission Audit Metadata

* fix: Read Mutation Snapshots From the Primary Outside Transactions

* fix: Flush Deferred Audits Before Cache Invalidation, Keep Renewals Monotonic, Keep Role Case in Grant Lease Keys

* refactor: Map the Busy Permission Error to Its Code in the API Package

* style: Sort Lease Imports and Type the Renewal Spec Read

* fix: Keep Success Audits After Commit, Report Downgrades as Revocations, Code Audit Outages

* refactor: Move the Audit Outage Code to Its Own Module

* fix: Translate Lock Failures, Lease Non-Transactional Sessions, and Code Grant Audit Outages

* fix: Sanitize Lease Renewal Failures at the Permission Boundary

* fix: Annotate the Unavailable Error Code for Isolated Declarations

* fix: Keep the Admin Route Fail-Closed Opt-In and Document Lease Clock Skew Safety

* fix: Translate Grant Lease Store Failures to a Retryable 503

* fix: Log Failed Grant Audit Writes With Safe Metadata Only

* test: Detect Raw Error Arguments in the Grant Audit Log Spec

* feat: Make Permission Lease Limits Configurable and Localize Retryable Sharing Failures

* fix: Bound Lock Index Readiness by the Lease Wait and Carry the Resource ID in Every Audit Entry

* test: Cover Lease Config Reaching the Leases and the Sharing Dialog Failure Messages

* fix: Load App Config on the Sharing and Grant Routes and Log Failed Lease Releases

* perf: Load App Config Only on the Grant Mutation Routes
2026-10-15 00:30:16 +02:00
Marco Beretta
ccae5b988d
📜 feat: Audit Admin Role, Group, Config and IdP Role Mutations (#17052)
* feat: Extend the Audit Action Vocabulary to Admin, Sync and Credential Mutations

* feat: Audit Admin Role, Group and Config Mutations

Role deletion now removes the role's grants, ACL entries and config
overrides before the role itself and returns ROLE_CLEANUP_FAILED instead
of success when any of them fails, so a recreated role cannot inherit
leftover privileges.

* feat: Audit IdP and Startup Role Changes

* fix: Keep the Startup Role Sync Running When Its Audit Verification Read Fails

* test: Cover Admin Role Audit Trail End to End

* fix: Audit a Remote Agent Role Sync Only After It Persists

* fix: Restore Role Dependents When Role Deletion Fails Midway

* fix: Restore Role Dependents Only When the Role Still Exists

* style: Sort Role Admin Imports

* fix: Keep the ACL Snapshot When Role Cleanup Fails Ambiguously

* test: Read Role Audit Entries From the Audit Log Route

* fix: Audit Only Real Membership Changes and Restore Deleted Role Config in Full

* fix: Restore Deleted Role Grants in Full and Sanitize Audit Write Logging

* fix: Audit and Report a Role Deletion Confirmed After a Lost Response

* fix: Never Restore Dependents Onto a Recreated Role

* fix: Snapshot Role Dependents Before Deleting and Detect Membership Changes Atomically

* fix: Restore From the Data a Cascade Step Actually Removed

* fix: Restore Only Config Overrides When a Role Delete Fails

* fix: Log Only Safe Metadata When the Startup Audit Read Fails

* fix: Audit Grants Removed by a Failed Role Delete

* fix: Audit Removed Grants on Unverifiable Role Delete Exits

* fix: Honor Fail-Closed Auditing in the Admin Config, Group and Role Routes

* fix: Bind Role Deletion to the Captured Id and Audit Only Atomic Role and Config Changes

* fix: Skip the Config Restore When the Targeted Role Is Gone

* fix: Audit Initial Group Members and Skip No-Op Role Description Updates

* fix: Record Role Deletion as Pending Before Cleanup and Append Its Outcome

* fix: Audit Reassigned Role Members and Return the Persisted Config After a Tracked Toggle

* fix: Keep a Committed Role Deletion Successful When Cascade Audit Writes Fail

* fix: Decide Group Update Audits From the Atomic Write and Sanitize Permission Update Logs

* fix: Audit Grants From a Snapshot, Fail on Member Lookup Errors and Chunk Member Audit Entries

* fix: Audit Group Deletion Members and Previous Role Removal, Abort on Grant Snapshot Failure

* fix: Invalidate Config Caches After Partial Secret Writes

* fix: Retry Failed Cascade Restores, Report Changed Group Fields and Audit Every Reassigned Member

* fix: Log Safe Metadata When Cache Invalidation Fails After a Partial Write

* fix: Keep Error Messages Out of the Audit Write Failure Log

* style: Sort Audit Log Spec Imports

* fix: Sanitize Admin Handler Error Logs and Dispatch System Audit Writes Without Waiting
2026-10-14 23:57:07 +02:00
Marco Beretta
20d2c0d43d
🧹 fix: Run the Full Erasure Cascade When an Admin Deletes a User (#17050)
* fix: Run the Erasure Cascade in Admin User Deletion

* refactor: Share Generation Stop Logic Between Self-Service and Admin Deletion

* refactor: Move Admin Erasure Dependency Assembly Into packages/api

* fix: Bind getCleanupJob When Waiting for Generation Persistence

* test: Pin the Deletion Fence Before Admin Generation Stop

* fix: Resolve the Erasure Target Config Fail Closed

* fix: Carry the Target Profile Into Admin Erasure Storage Cleanup

* fix: Keep the Erasure Target Contract Storage Neutral and Reuse the Loaded Identity

* fix: Refuse New Generations Once Account Deletion Has Begun

* fix: Enforce Account Deletion Admission Once in createJob and Narrow the Erasure File Type

* fix: Install the Admission Guard in Every Entrypoint and Purge Refused Jobs

* fix: Guarantee Refused Job Purge and Sanitize Admission Failures

* fix: Route Remote Admission Through createJob and Keep Refusal Cleanup Sanitized

* fix: Log Admission Cleanup Failures With Safe Error Metadata

* refactor: Check Account Deletion Before Creating a Generation and Drain Twice

* refactor: Order Admission Against the Deletion Marker Without a Grace Period

* fix: Sanitize Post-Commit Admission Retirement Failures

* fix: Wire Shared-Server MCP OAuth Cleanup Into Admin Erasure

* fix: Scope Admin Erasure Lookups to the Target Tenant and Await Shared-Link Cleanup

* fix: Resolve the Erasure Target Config Under the Target Tenant

* fix: Run Admin Erasure OAuth Fencing in the Target Tenant

* fix: Match the Split OAuth Invalidate and Disconnect Hooks in Admin Erasure

* refactor: Inject the Tenant Executor Into the Erasure Resolver
2026-10-14 23:54:32 +02:00
Marco Beretta
1994a4949d
🔏 fix: Encrypt Provider Keys With v3, Reject Plaintext in decryptV2 and Default to Strong Backup Codes (#17051)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
* fix: Write User Provider Keys With Random-IV v3 Format

* fix: Reject Undelimited Input in decryptV2 Instead of Returning It

* fix: Default 2FA Backup Codes to the Strong Format

* fix: Limit Admin Secret Preview to the Last Four Characters

* fix: Re-Derive Stored Secret Previews Under the Last-Four Policy

* test: Build Legacy Credential Fixtures With the Server Key

* fix: Load Request Config for User Key Writes and Mask Legacy Previews
2026-10-13 22:19:37 +02:00
Marco Beretta
0bc344fcb9
🔑 feat: Let Users Connect Their Own Account for Pull Request Lookups (#16989)
* feat: Let Users Connect Their Own Account for Pull Request Lookups

* fix: Refuse Cross-Tenant Pull Request Callbacks and Surface Key Failures

* fix: Keep Pull Request Grants Through Outages and Remove Them on Disconnect

A token endpoint server error is an outage even with an OAuth error body, so
it no longer deletes the grant, and a refresh outage reaches the lookup as
unavailable instead of falling back to no token. Disconnect removes every
pr_oauth row of the user, whatever app or host is configured now. A grant that
never expires extends its retention when used, and the flow, request and
retention limits are configurable. The connection routes are built in
packages/api, mounted behind the pre-auth tenant middleware in both servers,
and answer failures with stable codes.

* fix: Keep the Pull Request OAuth App Out of Overrides

A role, group, user or tenant override can no longer change or remove pullRequests.oauth, so the connect capability and the callback see the operator's app.

* fix: Read a User's Pull Request Token Only With the Feature On

* fix: Report a Malformed Stored Pull Request Grant as a Failure

* fix: Remove a User's Stored Tokens When an Administrator Deletes the User

* fix: Answer No Pull Request for a Conversation Without a Lane When the User Token Fails

* fix: Share One Grant Read Between Concurrent Callers and Report an Expired Grant Disconnected

* fix: Remove a Pull Request Grant Only on a Definite Refusal and Past Its Retention

* fix: Version Pull Request Grants and Coordinate Refresh, Refusal and Retention Through Them

* fix: Invalidate a Refused User Token and Read It Only for a Lane That Needs It

* fix: Finish a Pull Request Connection Under the User's Effective Config Behind the Login Limiter

* fix: Type the Pull Request User Token for Isolated Declarations and Settle Retention Before Answering

* fix: Bind a Pull Request Connection Flow to the Generation a Disconnect Replaces

* fix: Start a Batch's User Token Read Only Within Its Deadline

* fix: Fence Pull Request Token Reads Against a Disconnect and Report a Failed Retention Cleanup

* fix: Fence Connection Status Like Token Reads and Keep a Newer Read Joinable

* fix: Report a Pull Request Grant Without Its App and Name the Sign-In That Made It

* fix: Read a User's Unexpired Pull Request Grants Newest First

* fix: Mark the Connection Callback as an OAuth Navigation and Report a Failed Extension

* fix: Name the Pull Request Sign-In Flow on Its Landing Page

* fix: Remove a Pull Request Grant of a Replaced App When It Is Read

* fix: Keep a Replaced App's Grant Untouched and Extend Retention From Each Day's Use

* fix: Accept a Pull Request Grant Only Under the User's Current Connection Generation

* fix: Renew the Connection Generation With a Grant Still in Use

* fix: Restore the Previous Connection When a Sign-In Cannot Store Its Grant

* fix: Decide a Sign-In by What Was Stored When a Write Does Not Answer

* fix: Roll Back a Sign-In's Generation Along With a Grant It Must Remove

* fix: Refresh an Access Token a Lookup Was Refused With Before Ending the Connection

* fix: Undo a Sign-In Only for a Confirmed Fence, Not a Failed Check

* fix: Read a User Token Again When the Connection Changes Under the Read

* fix: Apply the Current Retention Again When Another Replica Changed the Grant First

* test: Answer Each Concurrent Sign-In by Its Own Code

* fix: Skip Retention Rewrites for Grants That Expire Without a Refresh Token

* fix: Retry a Token Rejection That Loses to a Concurrent Grant Write

* fix: Skip User Token Reads for Lanes Without a Commit and Reread Status After a Reconnect
2026-10-12 15:58:23 +02:00
Marco Beretta
4617302116 🔑 feat: Passkey Authentication with WebAuthn Sign-In and Management (#14715)
* feat: passkey authentication with WebAuthn sign-in and management

* refactor: tighten passkey account protections

Require password confirmation before adding or removing a passkey, and rate
limit those confirmations per user in their own namespace. Restrict passkey
enrollment and sign-in to local accounts, and re-evaluate account status once
the credential owner is resolved. Require user verification on the sign-in
ceremony. Invalidate outstanding access tokens on password reset. Tighten
redirect target validation and sanitize ceremony failure logs.

Adds coverage across the passkey controller, the WebAuthn helpers, the step-up
limiter, both JWT strategies, the redirect utility and the passkey settings UI.

* fix: localize passkey default name and respect router basename

Send a localized default credential label at enrollment so non-English
locales no longer persist the server's English fallback, and route the
passkey post-login and 2FA navigations through the deployment base path
so subdirectory installs stay inside the router.

Also mock syncStaticTools in the server boot specs; without it the
post-listen MCP initialization calls undefined and exits the worker.

* fix: harden passkey counter checks and reach stranded credentials

Make the signature counter write a compare-and-swap so two assertions
that verified against the same stored counter cannot both commit, and
reject the losing one; a cloned authenticator is otherwise accepted
inside that window. Counterless authenticators keep restamping at 0.

Keep the passkey panel reachable for identity-provider accounts, which
already get passwordless removal server-side but had no way in, while
hiding the add control they cannot use. Cascade passkey deletion from
the admin user-deletion path so credential IDs are not stranded.

* fix: reject stale 2FA temp tokens and surface passkey load failures

A temporary 2FA token proves only the first factor, so honouring one
minted before a password reset let anyone holding the second factor
survive account recovery. Apply the same credentials-changed cutoff the
JWT strategy uses.

Also distinguish a failed passkey query from an empty list; the error
state previously rendered as "no passkeys yet", which misreports what
is protecting the account.

* fix: fail closed when the passkey counter write cannot commit

Returning success on a storage error bypassed the compare-and-swap it
exists to enforce, so a cloned authenticator could be accepted while the
stored counter stayed stale. A database that cannot take this write
cannot mint the session either, so little is gained by failing open.
Counterless credentials carry no clone signal and keep their old path.

* fix: keep the passkey 2FA handoff inside the router

A full-page navigation to the 2FA screen remounted AuthContextProvider,
which silent-refreshes on mount; no refresh session exists yet at that
point, so it bounced back to login before the code could be entered.
Route through the router like the password flow does.

Also reveal the removal step-up field when the server refuses a
passwordless attempt, so an account migrated to an identity provider
without its old hash cleared can still clear its credentials; drop the
staged password when the dialog closes; and revoke passkeys in the CLI
password reset, which bypassed the HTTP controller's cleanup.

* fix: apply one email-verification policy across login methods

Accounts predating mandatory verification are marked verified as they
sign in when no email is configured, but only the password strategy did
it, so the same legacy account was admitted by password and refused by
passkey. Extract the rule into a shared helper and call it from both.

* fix: refuse a passkey sign-in when the ban check cannot complete

checkBan defaults next to a no-op, so calling it bare discarded any
internal failure and left req.banned unset, letting the request fall
through and be issued tokens. Capture the error the middleware chain
would have propagated and reject instead.

* fix: refine passkey management dialogs

* test: stabilize passkey tooltip assertions

* test: reconcile passkey specs with the canary reset and deletion flows

The credentialsChangedAt stamp now lives in commitPasswordReset, so its
assertion moves to the passwordReset unit spec. Restores the describe
blocks the rebase merged together.

* test: mock passkey routes in the cross-site login suite

* refactor: move passkey handlers and credential checks into packages/api

The passkey controller, the credentialsChangedAt predicate and the legacy
email-verification grandfathering now live in TypeScript with injected
dependencies; /api keeps only the wiring that builds the handlers.

* style: align passkey UI with Tailwind v4 order and design-rule lint

* fix: revoke earlier access tokens on a CLI password reset

passwordVersion is not a user schema field, so the write was dropped and
tokens issued before the reset kept verifying.

* test: cover passkey sign-in and credential revocation end to end

* fix: hide passkey sign-in when email login is disabled

The passkey endpoints already refuse under ALLOW_EMAIL_LOGIN=false, so
advertising the button only led to a 403.

* fix: let only one concurrent verification consume a passkey challenge

Without a native getDel the challenge was read then deleted, so two
concurrent verifications could both use it. The delete now arbitrates:
only the caller whose delete removed the key gets the challenge.

* test: run passkey ceremonies on the localhost relying party

WebAuthn refuses an IP address as the RP ID, and the harness serves the
app on 127.0.0.1. The reset scenario runs once per project, so the mock
harness lifts the per-IP reset request limit as it does for forks.

* fix: keep passkey management when email login is disabled

Gate only the sign-in button on emailLoginEnabled, since only the sign-in
routes sit behind that gate; list, rename and remove stay reachable.

* fix: build the passkey credential index before the first registration

With MONGO_AUTO_INDEX=false the unique credentialId index was never
created, so concurrent registrations could store the same credential.

* test: enable password reset in the mock harness and cover email-login-off passkeys

* fix: stop re-exporting the passkey cap from data-schemas

The bundler left every import of methods/passkey.ts unlinked while that
module re-exported MAX_PASSKEYS_PER_USER from librechat-data-provider, so
the index build and the counter-failure log threw ReferenceError at
runtime. Consumers now import the cap from librechat-data-provider.

* test: close nested settings dialogs before logging out in passkey scenarios

* fix: reject cross-site passkey assertions

The verify route accepted a form posted from another site, which let an
attacker sign a victim into the attacker's account. It now runs the same
requireSameOrigin guard as password login and the 2FA step.

* fix: end refresh sessions on a CLI password reset

A surviving refresh session could mint an access token issued after the
credentialsChangedAt stamp, so the CLI now deletes sessions like the web
reset does.

* fix: keep the passkey button locked when it supersedes autofill

Starting the modal ceremony aborts the pending autofill one, and the
autofill error path cleared the shared lock and busy state while the
modal ceremony was still running.

(cherry picked from commit 471abaa64d)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14715
2026-10-01 03:46:16 +00:00
Marco Beretta
d263da5271
🧾 fix: Validate Principal Config Overrides Against configSchema (#16433)
* fix: validate principal config overrides against configSchema

Reject admin config writes whose fields fail configSchema, and drop
invalid fields from stored overrides at merge time so the base value
from librechat.yaml survives.

* fix: require the merge key on partial custom endpoint overrides

* test: register a dedicated user for config override scenarios

* test: log in once per worker in config override scenarios

* fix: strip invalid overrides under dotted record keys

* fix: ignore a stored overrides document that is not an object

* fix: reject indexed patches into merged-by-name override arrays

* fix: match union options by shape, apply supplied-value refinements, reject paths past scalars

* refactor: validate config overrides on the merged result instead of the patch

* test: cover record refinements on written config fields

* fix: validate config field writes on top of the stored override

* fix: attribute id-addressed issues to their item and keep valid sections after repairs

* fix: judge field writes by what they change, over the principal's tombstoned base

* fix: validate whole-document config writes on the principal's tombstoned base

* fix: report config override issues as stable codes instead of schema messages

* fix: reject indexed writes into keyed arrays and keep filtered MCP servers out of validation state

* fix: judge each override layer only on the values it supplies

* fix: reject merged-by-key array items that are not objects

* fix: report repeated merge keys on the items the keyed merge discards

* fix: reject keys a union option drops when another option defines them
2026-09-28 16:22:22 -04:00
Danny Avila
f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00
Danny Avila
fb1322401d
🧪 test: Restore Two Rotted Integration Suites and Run Every One in CI (#15866)
* 🧪 test: Restore Two Rotted packages/api Integration Suites

MCPReinitRecovery.integration.test.ts mocked @librechat/data-schemas down to
its logger, so all eight tests threw once MCPServersRegistry started calling
scopedCacheKey (#15028). The mock now spreads the real module under the logger
override; ServerConfigsDB stays mocked, so the suite still never touches Mongo.

secrets.integration.spec.ts drove langfuse.secretKey through the generic admin
config API, which has stripped the langfuse section since #14108 moved it to
the dedicated Langfuse connection API. The generic-write matrix drops that row,
and a langfuse block covers the current contract against a real Config
collection: the stored secret stays encrypted at rest, generic reads redact it,
generic patch, tombstone and field-delete writes leave it untouched, and a full
base-config replacement carries it forward.

* 👷 ci: Run Every packages/api Integration Suite in CI

test:ci excludes every path containing "integration.", but the jobs meant to
run those suites selected by folder: the cache workflow only
src/{cache,cluster,mcp,stream}, the agents workflow only
src/agents/*.integration.spec.ts. Thirteen suites ran in no job, including
every *.integration.test.ts.

test:integration selects *.integration.(spec|test).ts anywhere under src and
replaces test:agents-integration in the (renamed) Integration Tests workflow.
The cache scripts also accept .test.ts, stream selects by suffix alone, and
core picks up src/middleware. Every suite test:ci excludes is now selected by
exactly one CI script.

* 🧪 test: Read the Tool Snapshot in the Concurrent Reinit Simulation

The concurrent reinitialize test copied reinitMCPServer but still called
fetchTools(), which returns [] when tools/list fails. Concurrent reinspections
of a recovered server can each succeed, and a later write replaces the app
connection an earlier request was already handed, so that request's tools/list
fails. The test failed in 9 of 25 iterations on Linux, and in the first CI run
that included the suite. Since #14686 production reads the snapshot and keeps
cached tools when it is incomplete; the simulation now does the same, and still
requires every complete snapshot to list both tools and at least one to complete.

A deterministic test pins the property that tolerance relies on, without
depending on the race: a newer config write replaces the held app connection,
whose snapshot is then incomplete, while the replacement lists the real tools.
2026-09-12 16:20:22 -04:00
Ravi Kumar L
000349681e
🩻 feat(insights): Scope Insights to Authorized Agents (#15549)
Some checks failed
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
* feat(insights): add agent-scoped access

* fix(insights): exclude unattributed assistant messages

* fix(sharing): key role insight toggles by principal

* fix(sharing): preserve insight grant snapshot

* fix(permissions): preserve insight bits atomically

* fix(permissions): guard insight audit rollback

* fix(insights): keep admin grant controls accessible

* fix(permissions): reconcile duplicate principal updates

* fix(insights): show automatic admin access

* fix(permissions): harden insights access updates

* fix(insights): simplify authorization and preserve client state
2026-09-04 20:16:58 -04:00
Danny Avila
84323d8c48
🛰️ fix: Repair BYOM Live Integration (#15613)
* fix: Repair BYOM live integration

* fix: Isolate mutable tool schemas
2026-09-04 15:38:04 -04:00
Danny Avila
164f146bdf
🛞 feat: Manage Personal Code Environment Lifecycles (#15369)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* feat: manage personal code environments

* fix: harden personal code environment lifecycle

* fix: satisfy backend lifecycle types

* fix: harden principal environment lifecycle and quotas

* fix: fence environment lifecycle races

* fix: reconcile code environment stack

* style: format code execution headers

* fix: reconcile code environment lifecycle

* fix: close self-service pairing gaps

* fix: close code environment reconciliation gaps

* fix: close code environment UI review findings

* fix: satisfy code environment type checks

* fix: route code priming to selected worker

* fix: harden code environment lifecycle fences

* fix: type lease renewal interval

* fix: harden environment lifecycle recovery

* fix: close lifecycle recovery races

* fix: type compensated agent timestamps

* fix: fence lifecycle cleanup by tenant and revision

* fix: scope environment revision fencing
2026-09-01 21:10:05 -04:00
Danny Avila
51edb2a660
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* style: format code execution headers

* fix: close self-service pairing gaps

* fix: satisfy code environment type checks
2026-09-01 18:46:33 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
a9ccac8656
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing

* fix: satisfy code environment type checks

* fix: secure code environment administration

* fix: isolate code pairing control plane

* fix: validate code pairing control responses

* fix: secure code pairing transport

* fix: validate code pairing wire format

* fix: harden pairing secret lookup
2026-08-30 17:12:20 -04:00
Ravi Kumar L
a70bcf66a4
🪢 feat: show Langfuse session link in shared chats (#15273)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: show Langfuse sessions in shared chats

* fix: wait for optional auth before loading shares

* fix: require Langfuse access for shared session links

* refactor: move shared Langfuse policy into API package

* fix: wrap shared chat actions on mobile

* fix: coordinate shared Langfuse link loading

* perf: batch shared Langfuse capability checks
2026-08-27 14:29:57 -04:00
Ravi Kumar L
56f0cde9a5
🩻 feat(langfuse): add tenant export telemetry (#15247) 2026-08-26 07:38:30 -04:00
Marco Beretta
0383030817
⚡ perf: Avoid Parallel Full and Paged Prompt Loading on Startup (#15031)
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-25 21:21:17 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Danny Avila
757fbebc37
🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored (#15067)
* 🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored

`discoverSkills` dropped any file whose path failed `isSafeRelativePath`
with no warning, no count, and no record. The skill published, reported
`succeeded`, and was missing files — invisible from the mirrored copy.

Real case: NVIDIA/skills has two such files (spaces in the filename), so
that repository syncs "cleanly" while silently losing them.

Matches what zip import already does — record the file, keep the skill —
and mirrors the existing `skippedSkills` shape into `skippedFiles` /
`skippedFileCount` on the sync status. Dropped files now make a run
`partial`, since a run reporting `succeeded` while dropping content is
the bug.

Only dropped *skills* can still make a run `failed`: a source that
published everything it found is a real mirror even if a file inside one
skill could not come along.

* 🔧 fix: Charge Dropped Files to the Skill That Published Them

Codex review: the up-front accounting counted a skill's unsupported files
whether or not that skill went on to publish. Two consequences — the status
described a skipped skill as published-but-incomplete, and enough failed
skills could consume the 20-entry sample and crowd out drops from skills
that actually published, which is the case the record exists for.

Now recorded at the two points a skill is counted as synced, matching what
`ISkillSyncSkippedFile` already documented ("the skill itself is live").

Also replaces `Array.prototype.at` in the new tests: it is outside this
package's lib target, so `tsc` rejected it even though jest ran it fine.
2026-08-21 03:36:27 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
a47ba7168f
🪢 feat: Custom Request Headers For Langfuse (#14945)
* ✨ feat: Custom Request Headers For Langfuse

Self-hosted Langfuse behind an authenticating proxy or gateway could not
be reached: every outbound Langfuse request hardcoded `Authorization` and
nothing else. Adds `langfuse.headers`, mirroring `endpoints.custom`
headers, and applies it to all four request surfaces — trace/media export
(via the agents run config), feedback scores, central project-identity
lookup, and admin credential verification.

Values resolve through the same pipeline as endpoint headers, so
`${ENV_VAR}` interpolation and header-safe encoding come along.
`extractEnvVariable` continues to refuse infrastructure secrets, so a
config cannot exfiltrate `MONGO_URI` through a header. A header whose
variable is unset is dropped with a one-time warning rather than sent as
a literal `${...}`, which a gateway would read as a wrong credential
instead of a missing one.

Headers merge beneath LibreChat's own `Authorization` on the REST
surfaces, matching `mergeHeaders`, so a custom header can never displace
the Langfuse credential.

These are deployment-level and documented as such: trace export batches
spans from every user through a single exporter, so unlike endpoint
headers they cannot carry per-user placeholders.

The central project-id cache key now includes the headers, so the
header-less module warm-up cannot record a proxy rejection against the
entry the request path later reads.

* 🔒 fix: Keep Langfuse Headers Out Of Stored Overrides

The generic admin config API accepts any field path inside an allowed
section, so `langfuse.headers` could be written through it. Unlike
`langfuse.secretKey`, headers are a map rather than one scalar path, so
the config secret registry cannot encrypt them at rest or mask them on
read — an admin-written map would sit in Mongo in plaintext and come
back in plaintext, widening exposure of what are gateway credentials.

Rejects them on both the dotted-patch and object-upsert routes, the same
way process-backed MCP servers are held to librechat.yaml. This is what
makes "deployment-level" true rather than merely documented.

* 🐛 fix: Wire Config Middleware And Header Collisions For Langfuse

Two codex review findings.

P1 — `api/server/routes/admin/langfuse.js` never mounted
`configMiddleware`, so `req.config` was undefined in production and
credential verification silently ran without the deployment's proxy
headers: exactly the deployments this feature targets could not save a
connection. The handler unit tests injected `config` into their mock
requests, so they stayed green. Mounts the middleware after the access
checks (unauthorized callers still short-circuit first) and adds
route-level tests that assert the handler actually receives a resolved
config — the composition root, not the component.

P2 — spreading custom headers under `Authorization` only replaced an
exact-case collision. A configured `authorization` survived alongside
the managed `Authorization` and fetch appends rather than replaces,
sending both credentials in one combined value. All four request sites
now use `mergeHeaders`, which already merges case-insensitively with the
override winning; tests cover the lower- and upper-case variants.

* 🔒 fix: Mask Langfuse Headers On Read And Harden Value Handling

Three codex round-2 findings.

P1 — the write guard blocked storing `langfuse.headers` in Mongo but did
nothing for the read path: `GET /api/admin/config/base` serves the
resolved AppConfig through `redactConfigSecrets`, which only knows
registered scalar secrets, so a yaml-configured gateway credential was
returned in full to any admin with Langfuse read access. Adds a
secret-map registry that masks values while keeping key names, so an
admin can still see which headers a deployment sets. Masking is safe
precisely because these are yaml-only — a masked read cannot be
round-tripped back over the real values. A malformed non-object value at
that path is dropped rather than serialized.

P2 — `mergeHeaders` indexes one spelling per lowercase name, so a config
holding both `authorization` and `AUTHORIZATION` had only one displaced;
the survivor was then appended by `Headers` into a combined value. Case
variants are now collapsed at resolution, before any consumer sees them.

P2 — `resolveHeaders` encodes only values it substitutes a user field
into, and no user is supplied here, so a literal or interpolated
character above U+00FF reached `Headers` unencoded and threw. Final
values now go through `encodeHeaderValue`; Latin-1 still passes verbatim.

* 🔒 fix: Keep Langfuse Header Credentials Out Of Logs And Validate Names

Three codex round-3 findings, plus a documented boundary for the fourth.

P1 — `loadCustomConfig` logs the parsed config at startup (`printConfig`
defaults true), so a literal gateway credential in `langfuse.headers` was
copied into application logs on every boot, undoing the masking the admin
read path had just gained. The printed copy now goes through
`redactConfigSecretMaps`, reusing the same registry. Scoped to map-valued
secrets so scalar-secret log behavior is unchanged; the live config keeps
its real values.

P2 — a nonempty but invalid field name (` X-Token`, `X Proxy Token`)
passed the emptiness check and then threw in the `Headers` constructor,
which would break export, verification, lookup, and feedback for the whole
deployment rather than that one header. Names are trimmed and validated
against the RFC 7230 token grammar, and dropped with a warning otherwise.

P2 — unresolved `${VAR}` detection tested the *resolved* value, so a
credential legitimately containing `${...}` was mistaken for a failed
substitution and dropped. Detection now inspects the configured text and
checks the referenced variables directly, which also drops references to
denylisted infrastructure secrets instead of forwarding them verbatim.

The fourth (fanout gateway forwards only `Authorization`, so a tenant
Langfuse behind its own proxy is not covered) is a real limitation in a
separate component. Documented on the schema field and in the example
config rather than left implied.

* 🔒 fix: Scope Langfuse Headers To Configured Origins

Three codex round-4 findings.

P1 — one header map was attached to every destination a run resolves to.
Under fanout that means a credential meant for an internal gateway was
also sent to the central destination, typically Langfuse Cloud: an
unrelated third-party origin. Headers are now attached only when the
destination's origin is one the deployment explicitly configured (a
self-hosted base URL, the fanout collector, or a tenant destination set
by env). The built-in `*.cloud.langfuse.com` defaults are excluded
precisely because nobody pointed at them. For trace export this also
means attaching after the export branch settles on a `baseUrl` rather
than before, since which destination wins depends on the branch.

P2 — `encodeHeaderValue` only encodes above U+00FF, so a newline, CR, or
NUL passed through and threw in `Headers`, breaking every request rather
than the one header. Values are trimmed (the common trailing-newline
case) then validated against the legal field-value bytes; an embedded
CRLF is a request-splitting attempt and is dropped, not stripped.

P2 — the write guard matched only the exact `headers` property, so
`{ langfuse: { "headers.X-Token": "..." } }` and root-level dotted
variants slipped through into the Mixed overrides document, where the
nested-map redactor never walks them and a later read returns them in
plaintext. All dotted spellings are now rejected.

* 🔒 fix: Bind Langfuse Headers To One Configured Origin

Four codex round-5 findings.

P1 — the round-4 allowlist still authorized every configured origin, so a
deployment with both a collector and an explicit central host sent the
same credential to both. `langfuse.headers` is one map with no way to say
which endpoint it authenticates to, so it is only unambiguous when the
deployment configures exactly one Langfuse origin. Iterating on which
origins to guess was the wrong axis; headers are now sent only when there
is a single configured origin and the destination is it, with a warning
when several make the intent unresolvable. That covers the self-hosted
case this feature exists for; multi-destination deployments need
per-destination headers the schema cannot yet express.

P1 — `fetch` defaults to following redirects, and Node strips
`Authorization` across origins but keeps arbitrary headers, so a redirect
off an allowed origin would hand the gateway credential to a host that
passed no check. Requests carrying custom headers now refuse redirects;
requests without them keep the default, so nothing changes for existing
deployments.

P2 — `extractEnvVariable`'s whole-string branch is anchored and greedy, so
`${CLIENT_ID}:${CLIENT_SECRET}` parsed as one variable name and the raw
template was sent as the credential. References are expanded here now, so
only literal values reach that path.

P2 — a valid token name is not necessarily usable: `Transfer-Encoding`
makes `fetch` throw and a fixed `Content-Length` misdescribes the body of
every other request sharing the map. Request-framing names are dropped.

* 🐛 fix: Expand Langfuse Header References Exactly Once

Codex round 6 (P2). After expanding `${VAR}` references myself I still
handed the result to `resolveHeaders`, which runs `extractEnvVariable`
over it again — so a credential containing `${PATH}`, or any other name
that happens to be set, was silently rewritten on export, verification,
lookup, and feedback. The round-3 test only used an *unset* embedded
name, which the second pass leaves alone, so it could not catch this.

Resolution no longer round-trips through `resolveHeaders`. The only part
still wanted from it was stripping `{{...}}` user placeholders, which is
now applied directly; expansion, encoding, and validation were already
local. Adds a test whose embedded variable is set, which fails against
the previous pipeline.

* 🐛 fix: Process Langfuse Header Templates Before Substitution

Codex round 7 (P2), the mirror of round 6. Having stopped re-expanding
the resolved credential, the placeholder strip was still running over it:
a token containing `{{LIBRECHAT_USER_ID}}` had that span deleted and
`abc{{...}}ghi` went out as `abcghi`.

Establishes the invariant the last two rounds were circling. Every
template operation — placeholder strip, unresolved-reference check,
expansion — now runs on the operator's configured text, and the
credential is substituted last and never touched again. Gateway
credentials are arbitrary strings, so none of their bytes are syntax.

Also moves the unresolved-reference check after the strip, so it no
longer reports a variable inside a `{{...}}` span that the strip removes.
2026-08-17 20:39:23 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Danny Avila
8f1f961212
🧱 refactor: Require Broad Config Management for Base Field Mutations (#14775) 2026-08-12 22:32:38 -04:00
Ravi Kumar L
9980b6221f
🪢 feat: add Langfuse session links (#14776)
* feat: add Langfuse session links

* fix: tighten Langfuse session link resolution

* fix: clear stale Langfuse session links

* test: verify tenant Langfuse session links

* fix: align Langfuse link with client conventions
2026-08-12 22:25:23 -04:00
Marco Beretta
92a8058f02
🛟 fix: Isolate Invalid Skills During GitHub Sync (#14735)
* fix: treat unrecognized SKILL.md frontmatter keys as warnings

An unknown key in one SKILL.md failed that skill outright, and because the
GitHub sync runner marks a source failed on any validation error, a single
stray key took down every other skill in the repository. Syncing
github.com/cloudflare/skills failed entirely because 2 of its 13 skills
carry a `references:` key.

UNKNOWN_KEY is now a warning, so the skill is stored (unknown keys and all)
and the issue is surfaced rather than fatal. `references` joins the allowed
set with a shallow JSON-safety check instead of a strict kind match: real
files use a string, a list of strings, a list of objects, and a map, and
pinning one shape would reintroduce the same failure.

Malformed frontmatter stays fatal: INVALID_TYPE, INVALID_SHAPE and the
non-plain-object check are unchanged.

* fix: skip individual skills instead of failing a whole sync source

Any error inside the discovery or commit loop reached the outer catch and
marked the entire source failed, so one unusable SKILL.md, one oversized
blob, or one duplicate name cost every other skill in the repository.

Each skill now runs inside its own boundary and a failure is recorded
against that skill. Errors that mean nothing else in the run can succeed
(lock loss, GitHub auth failures, rate limiting) still abort the source
rather than being charged to whichever skill hit them first. Skills are
marked seen before the attempt, so the reconcile pass cannot mirror-delete
the previously synced copy of a skill a later run can repair, and duplicate
names now drop the whole colliding group instead of letting tree order pick
an arbitrary winner.

Status gains `partial` (published some, skipped others) plus a capped
sample of the skipped skills with the reason for each. A run that publishes
nothing and skips something is still `failed`, carrying the first skip's
error. The skipped entries name repository paths, so they follow the same
visibility rule as owner/repo/paths; the bare count does not.

Sync warnings are logged too: a background run has no user-facing surface,
so the log is the only place a maintainer sees why an upstream SKILL.md
looks off.

* test: cover skill sync warnings reaching the log

An unrecognized frontmatter key no longer fails the skill, so a background
sync has nowhere to report it except the log. Every mock in this spec
returned an empty warning list, which left that path unexercised.

* fix: describe nested frontmatter values in the shared skill type

`SkillFrontmatterValue` allowed only scalars and string arrays, while the
server has always stored `hooks` and `metadata` as JSON-safe objects, and now
`references` too. A skill carrying any of them could not be represented by
`TSkill`, `TCreateSkill` or `TUpdateSkillPayload` without a cast.

The type stays free of `any` and `unknown`: values remain JSON-safe by
construction, and the server keeps bounding depth, string length and array
size when it validates them.

* fix: protect moved mirrors and rolled-back counts when a skill is skipped

Continuing past a failed skill exposed two problems that aborting the whole
source used to hide.

A moved skill's mirror keeps its old upstream id until the update lands, and
only the new path was marked as seen, so the reconcile pass read the mirror as
stale and deleted the very copy the skip path exists to preserve. The old id is
now marked as seen too.

Deletion counters were incremented when a stale name-conflicting mirror was
removed, but never undone when the following commit failed and the mirror was
restored. The run no longer stops there, so the status persisted a deletion
that did not happen and the reconcile pass counted the restored row again.
Counters are now rolled back when the restore succeeds.

* fix: bound unknown frontmatter values and keep moved mirrors through duplicates

Tolerating an unrecognized key meant its value skipped the shared JSON-safety
check, so a deeply nested or oversized payload was accepted and persisted under
a key nobody validates. The key stays non-blocking; the value is now held to
the same depth, array and string bounds as every structured key.

A skill that moves into a name another discovered skill also claims is dropped
with the rest of its duplicate group before the sync path can reuse its mirror,
which left the still-published copy unmarked and reconciled away. Both paths now
mark the moved mirror through one helper.

* fix: end the source when a skipped skill fails to roll back

A skill that fails and rolls back cleanly is just a skipped skill. One whose
restore or delete also fails leaves a mirror with half-rewritten files or a
half-created row, and the run now continues past it, so the source could report
partial success while that mirror stayed inconsistent and its pre-marked
upstream id kept reconciliation away from it.

Failed rollbacks now raise a source-fatal error carrying the original failure,
which stops the source the way a lost lock or a refused GitHub token does.

* test: cover a skipped skill discovered at the repository root

A repository-level SKILL.md is discovered with an empty path, so this pins
that a skip recorded against it still persists with the rest of the partial
status rather than taking the whole status row down with it.

* docs: describe unknown skill frontmatter warnings

* fix: preserve mirrors after partial skill sync

* fix: preserve skill validation details during sync

* fix: fail sync when mirror identity cannot be restored

* fix: harden skill sync failure boundaries

* fix: preserve skipped skills on fatal sync

* fix: surface skill sync diagnostics and rollback failures

* fix: preserve skill frontmatter extension keys

* fix: reject skill frontmatter keys that collide when normalized

Frontmatter keys are matched case-insensitively against the canonical
key list, so "Name" and "name" both resolve to "name". Every call site
normalized independently, and the last key in iteration order silently
won, meaning the effective value depended on YAML ordering rather than
on anything the author could see.

Centralize the normalization in normalizeSkillFrontmatterKeys and have
it fail when two recognized keys resolve to the same canonical key,
rather than picking one. parse.ts, deployment.ts and the agent handler
now surface that as a parse error; createSkill and updateSkill surface
it as a blocking DUPLICATE_KEY validation issue. Unrecognized keys are
still passed through untouched so extension frontmatter survives.

deriveStructuredFrontmatterFields and both write paths now run on the
normalized map, so a "Disable-Model-Invocation" key derives the same
column a lowercase one does.

* fix: harden github skill sync against dropped requests and failed cleanup

Three failure paths in the GitHub sync could leave a source looking
healthier than it was.

githubJson only handled HTTP-level errors. A fetch that rejected before
producing a response (DNS failure, socket reset, abort) escaped as a
raw TypeError, so the sync reported a generic crash instead of a typed
sync error. Wrap it as GITHUB_REQUEST_FAILED and add that code to the
fatal set, since a source whose requests never complete cannot be
partially synced.

When a synced file failed to persist, the orphaned upload was cleaned
up on a best-effort basis and the cleanup error was only logged. If the
cleanup itself failed, the source still ended with the original error
and left a real orphan behind. Promote that to a rollback failure so
the source reports SYNC_ROLLBACK_FAILED with the triggering error.

Skill warnings were logged inside commitRemoteSkill, before the file
sync and viewer setup that can still roll the skill back. A skill that
never survived publication therefore emitted warnings as though it had.
Return the warnings from the commit and log them once the skill is
fully published.

* fix: report skipped github skills before credential errors

serializeErrorMessage checked isCredentialError first, and that check
matches on the error text. A skipped skill whose path happens to
contain a credential-ish word, for example skills/credential-helper,
was therefore redacted to "GitHub skill sync credentials are not
available" for admins without credential-metadata access, hiding a
parse failure behind a wrong diagnosis.

Check the promoted skipped-skill case first, since it is identified by
error code rather than by text and is the more specific match. The
credential redaction still applies to everything else.
2026-08-12 13:22:06 -04:00
Dustin Healy
af795be0c2
🪢 feat: Langfuse Fanout Connection Setting (#14108)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: encrypt tenant Langfuse secret in admin config

Add generic per-field secret encryption to the admin config layer: registered
secret paths (langfuse.secretKey) are encrypted with encryptV3 on write and a
non-secret fingerprint companion is stored. Admin config reads (base + per
principal) redact registered secrets so they are never returned; the fingerprint
is kept so the UI can show which key is configured.

The Langfuse fanout read path decrypts the tenant secret before export. Adds
secretKeyFingerprint to langfuseConfigSchema and tests for the encrypt/redact
policy.

* fix(api): secure admin config secret handling

* fix(api): preserve encrypted langfuse config secrets

* fix(api): couple config secret fingerprint deletion

* fix(api): read langfuse fanout collector url from env

* fix(api): display langfuse secret key hint

* fix(api): remove langfuse secret fingerprint breadcrumbs

* fix(api): use langfuse destination keys for tenant config

* fix(api): remove langfuse config compatibility fallbacks

* refactor(api): simplify langfuse secret helpers

* refactor(api): simplify langfuse config secret handling

* feat: in-app Langfuse connection settings panel

Add a discoverable, admin-gated Langfuse connection panel inside LibreChat
Settings (Dify-style): enable toggle, host, public key, masked write-only secret,
configured-key fingerprint, and a test-connection action. Backed by a dedicated
/api/admin/langfuse/connection endpoint that encrypts the secret at rest, returns
metadata plus fingerprint on read, and validates credentials. Builds on the
per-field encryption and fanout decrypt from the langfuse-config-encryption branch.

* refactor: align Langfuse secret field to CustomUserVars pattern

Use the established SecretInput plus Set/Unset state pill (com_ui_set/com_ui_unset)
from the MCP CustomUserVars UI for the saved-secret state, instead of a bespoke
masked input.

* fix: drop em dash from saved-secret placeholder

* feat: show loading state on Langfuse test connection button

* feat: gate in-app Langfuse settings on fanout config and admin role

* test: align Langfuse connection spec with SecretInput refactor

* feat(langfuse): refine tenant connection controls

* fix(admin): refine Langfuse connection verification

* fix(langfuse): refine tenant connection settings

* fix(langfuse): simplify export enablement controls

* fix(langfuse): validate tenant export configuration

* fix(langfuse): align startup fanout gate

* fix(admin): time out Langfuse verification

* fix(ui): rename Langfuse connection setting

* fix(admin): enforce Langfuse config capability

* feat(langfuse): require explicit tenant export activation

* feat(langfuse): support single-tenant connection settings

* fix(i18n): remove obsolete integrations label

* fix(langfuse): authenticate ingestion verification

* fix(langfuse): validate public key independently

* fix(langfuse): localize connection errors

* perf(config): skip Langfuse checks for non-admins

* fix(langfuse): preserve trace sampling for feedback

* test(langfuse): fix feedback sampling fixture

* fix(langfuse): align secret preview field

* fix(langfuse): harden connection settings state

* fix(langfuse): preserve trace destination state

* fix(langfuse): enforce tenant-wide routing invariants

* fix(langfuse): preserve verified connection invariants

* fix(langfuse): preserve stable project identity

* fix(langfuse): warm project identity asynchronously

---------

Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 18:33:10 -04:00
Danny Avila
becfc5a373
🗝️ feat: Custom Endpoint API Key Encryption via Array Secret Registry (#14510)
Extend the admin-config secret registry (#14509) with array-item secrets,
registering endpoints.custom[*].apiKey: encryptV3 at rest with apiKeyPreview
companions, redaction on admin reads (plaintext-legacy included, with
verbatim-name omit-to-keep preservation, duplicate-identity skip, and
legacy-plaintext self-healing), passthrough for user_provided/${ENV} refs,
strict-payload runtime decryption at getCustomEndpointConfig, the custom
model fetch, and the provider fallback, and rejection of named/indexed/
positional writes beneath the protected array path.
2026-07-29 15:37:46 -04:00
Dustin Healy
f4e0888f14
🕶 feat: Generalize Admin Config Secret Redaction (#14509)
* feat: generalize admin config secret redaction to a field registry

Replace the single hardcoded langfuse.secretKey handling in the admin
config secrets module with a registry (CONFIG_SECRET_FIELDS) so every
credential-shaped config field is encrypted at rest, redacted on read,
and preserved when omitted on a subsequent write.

Registry covers langfuse.secretKey plus speech tts/stt provider apiKeys,
ocr.apiKey, the webSearch provider apiKeys, and the assistants /
azureAssistants endpoint apiKeys. Fields that conventionally hold
${ENV_VAR} references keep those references as plain, visible values;
literal secrets are always encrypted. langfuse.secretKey behavior
(display companion, always-encrypt, array-section handling) is unchanged.

Wire runtime decryption for the consumers that read these values from
the merged app config: resolveConfigSecret in the speech STT/TTS
services and decryptConfigSecret in the Mistral OCR auth loader. Legacy
plaintext literals and ${ENV_VAR} references continue to resolve.

* feat: add masked display companions for every registered config secret

Every non-langfuse field added to the secret registry was missing the
displayPath that langfuse.secretKey already had, so redacted admin reads
returned nothing for those fields instead of a masked value like
sk-mis...Z789. The registry-driven encrypt/redact/preserve/mutation-path
logic in secrets.ts was already field-agnostic; the only backend fix is
setting displayPath on the other 15 registry entries.

Add the matching optional display<Field> companion to each zod schema
in librechat-data-provider (ocr, speech tts/stt providers, webSearch
providers, the shared assistants/azureAssistants endpoint schema) so the
field is typed for consumers, mirroring langfuse.displaySecretKey. DB
overrides are Mixed-typed, so nothing breaks without this at the storage
layer, but the type is needed for any typed consumer of TCustomConfig.

A display path can never be written as a secret: direct writes to it are
rejected, and an ancestor-object write that includes a spoofed display
value alongside or instead of the real secret is overwritten or dropped,
never encrypted or persisted.

* fix: harden the config secret write path against masking and mixing bugs

getDisplaySecretKey disclosed the entire value for any secret of 10
characters or fewer, since the first-6/last-4 mask overlaps or covers
the whole string at that length (e.g. self-hosted LocalAI tokens).
Short secrets are now fully masked instead.

writeSecretIntoSection/writeDottedSecret encrypted a literal secret's
raw string verbatim, including leading/trailing whitespace, so a
padded paste round-tripped with the whitespace intact and a
whitespace-only value was not treated as empty. Literals are now
trimmed before encrypting and masking.

Both functions also returned early on an env-placeholder value without
clearing the display companion, so replacing a literal secret with
${ENV_VAR} left the previous masked value stale in the stored config,
and a client-supplied display value submitted alongside a placeholder
secret was never overwritten. The placeholder branch now clears the
display companion in both the dotted-patch and object-valued write
paths.

* fix: fail closed in Mistral OCR auth when a stored ciphertext can't decrypt

loadAuthConfig fell back to the raw v3: ciphertext string whenever
decryptConfigSecret returned undefined, so a corrupted or otherwise
undecryptable stored secret was sent to the Mistral API verbatim as
the apiKey instead of triggering the existing env-var fallback.

isEncryptedConfigSecret is now exported so the OCR auth loader can
distinguish "this looks like ciphertext and failed to decrypt" from
"this was never encrypted" and treat only the former as empty,
preserving literal and ${ENV_VAR} values exactly as before.

* fix: omit undecryptable TTS provider headers instead of sending "undefined"

openAIProvider, elevenLabsProvider, and localAIProvider built their
Authorization/xi-api-key headers directly from resolveConfigSecret's
return value, which is undefined on a decrypt failure. That produced a
literal "Bearer undefined" header (or an undefined-valued xi-api-key
header) sent to the provider instead of failing gracefully.

Each provider now resolves the key once and only includes the header
when it's non-empty, matching the pattern already used by
azureOpenAIProvider and STTService's providers.

* fix: strip secret-ancestor arrays at any depth, not just the top level

encryptConfigSecrets/redactConfigSecrets only stripped an array-valued
registered-secret ancestor when it appeared as a literal top-level key
(e.g. a dotted "speech.tts.openai" key). A true nested array at any
depth, e.g. { speech: { tts: { openai: [{ apiKey: "sk-secret" }] } } },
made walkToParent return null and silently skip that field entirely,
so the literal secret was stored unencrypted and returned verbatim to
any reader with section-level read access.

pruneSecretAncestorArrays now walks every registered field's ancestor
chain and deletes any array found at any depth before encryption or
redaction runs, closing the gap for both write and read paths.

* fix: migrate legacy plaintext secrets instead of dropping them on preserve

preserveConfigSecrets only restored an omitted secret when the existing
stored value was already v3-encrypted. Every field this PR newly
registers was previously stored as plaintext with no protection at
all, so any deployment upgrading into this registry has real
credentials sitting in Mongo as plaintext today. The first time an
admin edited an unrelated field in the same section (e.g. mistralModel
next to ocr.apiKey), the omitted plaintext secret failed the
"already encrypted" check and was silently dropped instead of
preserved, breaking the integration.

The existing value is now encrypted in place when it isn't already
ciphertext or an allowed env placeholder, so the credential survives
the edit and gets a computed display companion instead of being lost.

* refactor: derive masked-preview companions as <field>Preview

Replace the display*-prefixed companion names (displaySecretKey,
displayApiKey, displaySerperApiKey, ...) with a uniform <field>Preview
suffix (secretKeyPreview, apiKeyPreview, serperApiKeyPreview, ...) derived
automatically from the registered secret path — registry entries no longer
declare a displayPath, and the name never collides with display-label
config fields like modelDisplayLabel.

Legacy langfuse.displaySecretKey companions (the only shipped instance,
with no released reader) are stripped from writes and reads and migrated
to secretKeyPreview on preserve, so stored documents self-clean.

Also reject MongoDB operator segments ($, $[], $[id]) in admin config
field paths: isValidFieldPath previously accepted them, letting a patch
like webSearch.$[].serperApiKey reach patchConfigFields as a positional
update that bypassed secret-path validation.

* fix: translate legacy displaySecretKey to secretKeyPreview on reads

Redaction previously deleted the legacy companion outright, so the first
admin read of a not-yet-migrated document showed no configured-secret
indication until a later write migrated it. Reads now surface the legacy
value under secretKeyPreview (when no new-name preview exists) while still
stripping the legacy key from the response; stored documents migrate for
real on their next write.

* fix: detect runtime ciphertext by full encryptV3 payload shape

Runtime resolution (resolveConfigSecret, mistral OCR auth) now identifies
decryptable values by the exact v3:<32-hex-iv>:<hex> shape encryptV3
produces instead of the bare v3: prefix, so a legitimate literal credential
that merely starts with v3: (e.g. from YAML, which the admin write path
never encrypts) resolves as a literal instead of failing decryption.
Write-side prefix rejection stays broad as spoof/echo defense.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 13:55:45 -04:00
Dustin Healy
044c134ecf
🤏 fix: Filter Admin Config Reads by Section-Scoped Read Capability (#14472)
listConfigs, getBaseConfig, and getConfig only checked the broad
read:configs capability, so a caller holding nothing but
read:configs:<section> grants got a blanket 403 on all three instead
of a response filtered to the sections they hold. Any deployment
using section-scoped config grants hits this.

Adds hasAnyConfigReadAccess as a cheap pre-flight check covering
broad and section-scoped read and manage grants (manage implies
read), so a zero-access caller still 403s before a DB fetch while a
section-scoped caller gets the response filtered to exactly what
they hold. The same manage-implies-read rule is fixed at its root in
getParentCapabilities so a manage-only caller sees the section they
manage instead of having it stripped after passing the pre-flight.

Resolves every section for a request in one batched
getHeldCapabilities query via getReadableConfigSections instead of
one round trip per section.

Includes AppConfig field-renaming normalization (interfaceConfig,
turnstileConfig, mcpConfig) so the filter checks the canonical
section name rather than the renamed response field, and stops
availableTools from bypassing the filter by gating it on its
filteredTools/includedTools source sections.
2026-07-28 07:38:37 -04:00
Dustin Healy
b0d46b0518
🗝️ feat: Encrypted Langfuse Fanout Config (#14107)
* feat: encrypt tenant Langfuse secret in admin config

Add generic per-field secret encryption to the admin config layer: registered
secret paths (langfuse.secretKey) are encrypted with encryptV3 on write and a
non-secret fingerprint companion is stored. Admin config reads (base + per
principal) redact registered secrets so they are never returned; the fingerprint
is kept so the UI can show which key is configured.

The Langfuse fanout read path decrypts the tenant secret before export. Adds
secretKeyFingerprint to langfuseConfigSchema and tests for the encrypt/redact
policy.

* fix(api): secure admin config secret handling

* fix(api): preserve encrypted langfuse config secrets

* fix(api): couple config secret fingerprint deletion

* fix(api): read langfuse fanout collector url from env

* fix(api): display langfuse secret key hint

* fix(api): remove langfuse secret fingerprint breadcrumbs

* fix(api): use langfuse destination keys for tenant config

* fix(api): remove langfuse config compatibility fallbacks

* refactor(api): simplify langfuse secret helpers

* refactor(api): simplify langfuse config secret handling

---------

Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
2026-07-14 08:19:55 -04:00
Danny Avila
d9251fc3b6
✂️ fix: Cap Audit Chain Verification and Honor Client Cancellation (#13903) 2026-06-23 08:33:16 -04:00
Danny Avila
725a14e409
🪜 fix: Strip Caller-Supplied Priority from Assign-Only Config Upserts (#13911)
* fix: prevent assign-only config priority changes

* fix: preserve assign-only config priority atomically

* style: format config priority guard

* fix: type preserve priority upsert option
2026-06-23 08:31:26 -04:00
Dustin Healy
6a63531eb4
📒 feat: Audit Log Backend for SystemGrant Assign and Revoke Events (#13087)
* 🛡️ feat: Audit log backend for SystemGrants changes

Add an AuditLog Mongoose collection that records every grant assign/revoke as an append-only entry capturing the actor, target principal, capability, timestamp, and tenant scope. Wire the entry-write into the existing admin assignGrant and revokeGrant handlers so the admin panel's audit-log tab populates as grants happen.

The data-schemas package gains the IAuditLog type, a Mongoose schema with tenant + target compound indexes for keyset pagination, a model factory wired through createModels, and an AuditLog methods factory exposing recordAuditEntry, listAuditLogPage (cursor-paginated, faceted, search-aware), findAuditLogEntry, and streamAuditLogEntries.

The packages/api admin layer adds createAdminAuditLogHandlers with three handlers backing the routes the admin panel already consumes: GET /api/admin/audit-log returns paginated entries, GET /api/admin/audit-log/:id returns a single entry for the permalink drawer, and GET /api/admin/audit-log/export.csv streams CSV with formula-injection defang plus UTF-8 BOM.

The Express layer mounts the new router at /api/admin/audit-log behind requireJwtAuth and the ACCESS_ADMIN capability, matching the existing admin route pattern. The audit emission failure is logged via logger.error but never rolls back the grant.

* 🧹 chore: Audit log backend cleanup — offset pagination, name-based filters, type tightening

Switch listAuditLogPage from cursor-based to offset-based pagination with skip().limit() + parallel countDocuments, returning { entries, total } instead of { entries, nextCursor }; the cursor encode and decode helpers are no longer needed and have been removed.

Interpret the actorId and targetPrincipalId filter parameters as case-insensitive partial regex against the denormalized actorName and targetName fields rather than exact-match against the underlying ObjectId. Admin panel users naturally filter by human name, not by Mongo identifier.

Replace the broad Record<string, unknown> casts on req.query with a typed AuditLogQuery shape, drop two unused exported types and the now-unused mongoose Types import, and fix the streamAuditLogEntries Omit literal to match the interface and the offset-based design.

* 🛠️ fix: Address audit log review feedback (CI typecheck, ISO offsets, no-op revoke, deps surface, schema, backpressure, tests)

Resolve the duplicate AuditAction export that broke the data-schemas TypeScript check by importing the canonical declaration from types/admin instead of re-declaring it in types/auditLog.

Accept timezone-offset ISO 8601 timestamps such as 2026-05-01T09:30:00+02:00 in the from and to filter params and reject local-time strings without a zone so every request resolves to an unambiguous instant.

Skip the audit emission on no-op revokes: revokeCapability now returns deletedCount so the admin handler can omit the grant_removed entry when the target grant did not exist, keeping the audit trail factually accurate. Mocks in the existing grants.spec.ts updated to the new return shape.

Drop the required recordAuditEntry from AdminAuditLogDeps since the audit-log handler factory never consumes it; the grants handler factory keeps its optional dep for the write path.

Tighten the tenantId validator on the audit log schema to require a non-empty trimmed string, and rewrite the listing-index comment to describe deterministic offset sort instead of keyset pagination.

Stream the CSV export with explicit backpressure (await drain when res.write returns false) and abort on client disconnect so a cancelled download no longer pins a Mongo cursor or buffers unbounded data in memory.

Add packages/data-schemas/src/methods/auditLog.spec.ts covering tenant and platform scoping, single and multi action filtering, partial-name filtering for actor and capability, the createdAt window, offset pagination with total, ObjectId and date stringification on the wire, regex-metacharacter escape, and streaming completeness.

* 🛠️ fix: Address P1 audit-log review findings (cursor cancel, drain race, filter naming, type dedupe, tenant scope, log enrichment)

The CSV stream handler kept draining Mongo batches after the client
disconnected because the `for await` loop only honored its abort flag
inside `onEntry`. Thread an `isCancelled` callback into
`streamAuditLogEntries` so the methods layer closes the cursor as soon
as the handler sees `close`/`aborted`; a `finally` block guarantees
release on throw. The drain promise in `writeChunk` now races against
the response's `close` event so a destroyed socket cannot strand the
handler on a `drain` that will never fire.

The HTTP filter keys `actorId` and `targetPrincipalId` always did
case-insensitive substring matches on the denormalized `actorName` /
`targetName` columns, never on ObjectIds — a client passing a real id
silently got zero rows. Renamed the wire-level keys to `actorQuery` /
`targetQuery` (matching what the matcher actually does) and kept the
old names as deprecated aliases for one release so the sibling
admin-panel PR can migrate without breaking; each legacy use logs a
deprecation warning. Renamed the corresponding fields in
`AuditLogFilters` too.

`AdminAuditLogEntryWire` duplicated `AdminAuditLogEntry` from
`types/admin.ts` field-for-field, violating the no-duplicate-types
rule. Deleted the duplicate, hoisted `AuditLogPage`,
`RecordAuditEntryInput`, and `AuditLogFilters` from
`methods/auditLog.ts` into `types/auditLog.ts`, and updated the
handler, method factory, and re-exports accordingly.

`tenantFilter` treated `''` as a valid tenant scope, producing a
`{ tenantId: '' }` query that silently returned nothing while the
schema validator rejected `''` on writes. Switched to a strict
`typeof tenantId === 'string' && tenantId.trim().length > 0` check so
reads agree with writes, with new spec coverage for empty and
whitespace-only inputs.

Audit-write failures now log the full forensic payload (action,
capability, tenantId, actorId, target metadata) inside a single meta
object so winston's standard signature surfaces it correctly; a comment
on the catch block explains why the failure mode stays silent (it must
never block a privileged operation).

Stronger filter parsing: invalid `action` values and unknown
`targetPrincipalType` now return 400 instead of silently dropping.
Extracted `MAX_LIMIT` to a constant. Replaced the
`Record<string, Date>` cast in `buildFilter` with a typed local.
Switched the stream cursor to `lean<IAuditLog[]>()` and removed the
`as IAuditLog` cast inside the loop.

* ✅ test: Cover admin audit-log handler with unit tests for auth, validation, tenant isolation, CSV output, and abort

The sibling admin handlers (grants, groups, roles, users) all have
handler specs; this one was missing. The new suite covers 401 on a
missing `req.user`, 400 on malformed ISO `from` / `to`, 400 on
limit > 500, 400 on negative offset, 400 on an unknown action or
`targetPrincipalType`, 400 on a non-ObjectId `:id`, 404 when the
methods layer returns null, that the caller's `tenantId` (not a
forged query-string `tenantId`) is the one passed to the methods
layer, that `actorQuery` / `targetQuery` round-trip, that the
deprecated `actorId` / `targetPrincipalId` aliases still map through,
that the CSV stream emits the BOM as the first chunk with CRLF line
endings and the expected header labels, that quotes, commas, and
newlines are properly escaped, that the formula-injection prefixes
(`=` `+` `-` `@` tab CR) are defanged, that an `isCancelled` callback
reaches the methods layer and flips to true on client `close`, and
that `res.end` is skipped when the client disconnected mid-stream.

* 🛡️ feat: Enforce append-only AuditLog at the schema level

Every field is now marked `immutable: true`, and pre-hooks on the
schema reject `updateOne`, `updateMany`, `findOneAndUpdate`,
`findOneAndReplace`, `replaceOne`, `deleteOne`, `deleteMany`,
`findOneAndDelete`, plus any `save()` against an existing document.
`timestamps` is reduced to `{ createdAt: true, updatedAt: false }`
since a mutable timestamp would imply mutation is allowed, and
`updatedAt` is dropped from `AuditLog` / `IAuditLog`. The methods
spec resets state between tests via the raw driver (`AuditLog.collection.deleteMany`),
which bypasses the pre-hooks; new specs assert that the model-level
update / delete / re-save paths reject with the append-only error and
that `updatedAt` is not stamped on new documents.

* ♻️ refactor: Share MAX_AUDIT_LOG_LIMIT between methods and handler

Renamed the methods-layer constant from the generic `MAX_LIMIT` to
`MAX_AUDIT_LOG_LIMIT`, exported it through `@librechat/data-schemas`,
and consumed it from the handler instead of duplicating `500` there.
Now the limit is single-sourced; bumping it once updates both the
clamp inside `listAuditLogPage` and the 400-error boundary the
handler returns to clients.

* 🛡️ feat: Gate audit-log routes on a dedicated `READ_AUDIT_LOG` capability

The audit-log routes were gated on `ACCESS_ADMIN`, which conflates "can log
into the admin panel" with "can see who granted what to whom." Anyone with
`ACCESS_ADMIN + READ_CONFIGS` (a config reviewer with no people-management
authority) could read the grant history of every user, group, and role —
information they have no need to know.

`READ_AUDIT_LOG` ('read:audit_log') is now an explicit, separately grantable
read capability with no MANAGE counterpart, matching the append-only nature
of the collection. `seedSystemGrants` iterates `Object.values(SystemCapabilities)`
so existing ADMIN-role seeds pick it up automatically on next startup.

This also makes an "auditor" persona possible: hold `ACCESS_ADMIN + READ_AUDIT_LOG`
without any MANAGE_* grants and you can review history without modifying anything.

* ♻️ refactor: Share AUDIT_ACTIONS, tighten audit dep types, document route order

Exports a runtime AUDIT_ACTIONS array from packages/data-schemas alongside the
AuditAction type so the Mongoose schema enum and the HTTP handler's whitelist
consume one source of truth instead of duplicating the literal pair.

Switches the grants handler's recordAuditEntry dep typing from a duplicated
inline object literal returning Promise<unknown> to the published
RecordAuditEntryInput type returning Promise<void>, and tightens the local
emitAudit args to AuditAction. Replaces the local ParsedFilters interface in
the audit-log handler with Omit<AuditLogFilters, 'offset' | 'limit'> to drop
the duplicate definition.

Drops the optional marker on AuditLog.createdAt. Mongoose always sets it at
insert time, so callers treating it as nullable were guarding against a state
the schema does not produce.

Adds a comment on api/server/routes/admin/audit.js noting that /export.csv
must precede /:id so a future contributor does not accidentally reorder them
into a 404 trap.

* 🛡️ feat: Resolve audit names without extra DB round-trips

For the actor name, JWT-authenticated `req.user` already carries `name`,
`username`, and `email`. `resolveUser` now derives the actor display name
from `req.user` directly and threads it through the caller context, so
every grant assign and revoke no longer triggers a separate `getUserById`
lookup.

For the target name, replaces the previous always-store-the-principalId
behavior (which buried opaque ObjectId strings in immutable audit rows
for USER and GROUP targets) with a `resolveTargetName` dep. ROLE
principals continue to use `principalId` directly because the SystemGrant
model stores role names there. USER and GROUP principals route through
the new dep, which in `api/server/routes/admin/grants.js` calls
`db.getUserById` or `db.findGroupById` respectively and falls back to
the principalId on miss or error so the audit row stays intelligible.

Drops the misleading "display name lookup happens in a later iteration"
comment.

* ✅ test: Cover audit emission, scope emitAudit to today's ROLE-only surface

Fixes a misleading test that claimed to verify "idempotent even if the grant
does not exist" while mocking deletedCount: 1 (the grant DID exist). Replaces
it with the actual no-op scenario (deletedCount: 0) and adds an assertion
that recordAuditEntry is NOT called, since the whole point of the
deletedCount > 0 gate is to avoid fictitious revocation rows.

Adds a dedicated audit emission describe block covering: grant_assigned
emission with the actor name resolved from req.user, grant_removed
emission when deletedCount is positive, and the no-emission fallback when
recordAuditEntry is not configured. The actor-name assertions exercise the
name / username / email fallback chain in resolveUser.

The previous commit also added a `resolveTargetName` dep and an
emitAudit branch for USER/GROUP targets. The grants surface is ROLE-only
today (MANAGE_CAPABILITY_BY_TYPE has only PrincipalType.ROLE), so that
code path is unreachable from the handler. Removed the dep and the
branch; the audit row uses principalId as the target name, which is the
human-readable role name for ROLE principals. A comment in emitAudit
flags where to plumb resolveTargetName back in once USER and GROUP
grants are enabled.

* 🛠️ fix: Inclusive `to` date filter and reject inverted ranges

A `?to=2025-01-15` filter previously stopped at midnight UTC of that
day, silently excluding everything that happened on January 15. The
`parseIsoDate` helper now widens a bare `YYYY-MM-DD` to 23:59:59.999Z
when called with the `end` boundary. Full ISO timestamps are honored
exactly, so callers that want minute-precision can still get it.

Also rejects inverted ranges (`from` later than `to`) with a 400 so
operators see a clear error instead of a silent empty result.

* 🛡️ feat: Cap audit-log CSV exports at 100k rows; cover stream error path

Introduces MAX_AUDIT_EXPORT_ROWS (100k) and threads a `maxRows` option
through streamAuditLogEntries. The handler now passes the cap into the
stream so a careless admin script or a hostile auditor cannot pin a
Node worker and a Mongo cursor by exporting unbounded result sets.
Beyond 100k rows, callers should slice exports by from / to date.

Adds a methods-layer spec for the cap behavior, a handler-layer spec
that asserts the option is plumbed through, and a handler-layer spec
that exercises the streamAuditLogEntries-throws-after-headers-sent path
(catch block falls through to res.end instead of attempting JSON).

Documents on buildFilter that case-insensitive substring regex filters
(actorName, targetName, capability, search) cannot use a B-tree index
and degrade to a tenant-scoped partition scan, so deployments with
hundreds of thousands of audit rows per tenant should constrain those
queries with a date window.

* 🧹 chore: Spell CSV_BOM as  and drop a gratuitous optional chain

`revokeCapability` is typed `Promise<{ deletedCount: number }>` so the
`?.` on `revokeResult?.deletedCount` only obscured that the value cannot
be nullish.

`CSV_BOM` was a literal U+FEFF character invisible in most editors. Now
spelled as the Unicode escape so readers can see the constant; the test
that asserts on the first emitted chunk uses the same escape.

* 🔧 chore: Allowlist AuditLog in the tenant-isolation coverage guard

The AuditLog collection carries a tenantId field but scopes tenancy manually
inside listAuditLogPage / streamAuditLogEntries / recordAuditEntry using the
same $exists: false convention as SystemGrant. The tenant-isolation plugin
coverage spec now allows that and asserts it stays accurate.

* 🛠️ fix: Normalize blank tenantId before persisting audit entries

The `recordAuditEntry` write path was treating any non-null tenantId as a
real string, so empty or whitespace-only values reached the schema validator,
failed the non-empty-string check, and silently dropped the audit row. The
read-side `tenantFilter` already treats those values as platform-level scope,
so the write path now mirrors it: blank or whitespace-only tenantId becomes
an omitted field, which matches `{ tenantId: { $exists: false } }` queries
and clears validation. Added a regression test that records two entries with
blank and whitespace tenantId and asserts both persist with the tenantId
field absent.

* 🎨 style: collapse expect.objectContaining onto one line to satisfy prettier

* 🔒 fix: block document-level deleteOne/updateOne on AuditLog

Mongoose registers deleteOne and updateOne pre-hooks as query middleware
by default. The query-level append-only block on AuditLog therefore did
not cover Document.prototype.deleteOne() or Document.prototype.updateOne(),
leaving a path where a caller that had already loaded an audit row via
findOne could call .deleteOne() or .updateOne() on the instance and bypass
the schema contract.

Explicit { document: true, query: false } registrations close the holes,
and the spec now covers both code paths against a real in-memory Mongo.

* 🔒 fix: require ACCESS_ADMIN on audit-log routes

Every other admin router (config, grants, users, roles, groups, auth)
enforces requireJwtAuth followed by requireCapability(ACCESS_ADMIN) before
any feature-specific capability check. The audit-log router only required
READ_AUDIT_LOG, which is independent of ACCESS_ADMIN in CapabilityImplications,
so a role delegated only READ_AUDIT_LOG without ACCESS_ADMIN could read or
CSV-export the audit trail and bypass the admin boundary.

Aligned the middleware chain with the rest of the admin surface so
ACCESS_ADMIN gates entry and READ_AUDIT_LOG gates the feature within it.

* 🎨 chore: re-sort imports after dev rebase

Post-rebase sort-imports against the merge target — six audit-log files
landed with stale import ordering relative to the current scripts/sort-imports.mts
rules on dev. CI's import-order job flagged the drift; running the script
locally rewrites them in place. No semantic changes.

* 🔧 fix: explicit type annotations on audit-log model + schema exports

Dev migrated packages/data-schemas builds from rollup to tsdown with
--isolatedDeclarations enabled, which requires every exported function to
declare its return type and every exported variable to declare its type.
Two of our audit-log exports got swept up:

  TS9007 models/auditLog.ts:12  createAuditLogModel return type
  TS9010 schema/auditLog.ts:12  auditLogSchema variable type

Added Model<t.IAuditLog> on the factory and Schema<IAuditLog> on the
schema variable, matching the sibling SystemGrant convention. No runtime
behavior change.

* 🔧 fix: align revokeCapability type annotation with implementation

The rebase auto-merge of systemGrant.ts kept dev's outer type annotation
(`revokeCapability: ... => Promise<void>`) but our implementation returns
`Promise<{ deletedCount: number }>` (added during the bot-review loop to
let the audit emitter distinguish a real revoke from a no-op against a
nonexistent grant). The mismatch surfaced as TS2719 on the methods record
return at line 520. Updated the type annotation to match the impl.

The caller at packages/api/src/admin/grants.ts:444 reads
`revokeResult.deletedCount` to gate the audit emit, so the wider return
type is what the rest of the code already assumes.

* 🔧 fix: explicit factory return type on createAdminAuditLogHandlers

Same tsdown --isolatedDeclarations migration that hit packages/data-schemas
also applies to packages/api; the audit-log handler factory's inferred
return type tripped TS9013 against the new build pipeline. Annotated the
factory with explicit handler signatures matching the sibling
createAdminGrantsHandlers convention. Used Promise<Response | void> for
the export handler because its final res.end() path returns undefined,
unlike the other two handlers which always return a Response.

* 🛡️ feat: Generalize audit log into a tamper-evident, extensible event substrate

Reworks the SystemGrant-only audit log into a general-purpose, append-only
compliance substrate designed to absorb future event classes (agent runs,
tool/MCP calls, config + permission changes, approvals) without reshaping the
record. Nothing was shipped yet, so this replaces the grant-specific wire
shape rather than layering aliases.

Schema / record shape (packages/data-schemas):
- schemaVersion + two-level taxonomy: category + namespaced action
  (grant.assigned/grant.removed), first-class outcome and severity.
- Structured actor{type,id,name} supporting non-user actors (system, agent,
  service, schedule, webhook, api); generic target{type,id,name}; open
  metadata map; request context{requestId,ip,userAgent,sessionId}.

Tamper-evidence (hash chain):
- Per-tenant chain keyed by chainKey with seq/prevHash/hash. Appends link to
  the previous hash; a unique {chainKey,seq} index serializes concurrent
  writes (dup-key retry) so the chain can never fork. createdAt is explicit so
  it's covered by the hash.
- verifyAuditChain() walks a chain and detects modification, deletion, and
  forged links; exposed via GET /api/admin/audit-log/verify.

Other best-practice gaps from the review:
- Keyset (cursor) pagination over seq alongside offset; stable under
  concurrent appends. nextCursor in the page payload.
- Retention: purgeAuditLogEntries() privileged prefix-purge with a confirm
  latch, returns a checkpoint; verify tolerates a purged prefix.
- Fail-closed option (AUDIT_LOG_FAIL_CLOSED) so a failed audit write can fail
  the grant request instead of being swallowed; default stays fail-open.
- Grant handlers now capture request context and emit the new shape.

CSV export updated for the new columns (incl. seq/hash). data-schemas bumped
to 0.0.54 for the sibling admin-panel consumer. Tests rewritten: 28
methods-layer cases (chain genesis/linking, tamper detection, keyset, purge)
and the handler/grants specs updated for the new shape, fail-closed, and the
verify endpoint.

* 🛠️ fix: Address Codex review on the audit-log substrate

- F1 (fail-closed atomicity): assign/revoke now compensate (rollback grant /
  restore grant) when a fail-closed audit write fails, so a 5xx never leaves an
  unaudited mutation.
- F5: only emit grant.assigned for a real change — skip the audit when the role
  already holds the capability (idempotent re-assert).
- F7: verifyAuditChain no longer silently trusts a non-genesis start; a purged
  prefix must be authorized by a trusted checkpoint (purge now returns
  {throughSeq, prevHash}), else verification fails as tampering.
- F4: block Model.bulkWrite on AuditLog (would bypass the append-only middleware).
- F3: CSV export appends an explicit TRUNCATED marker + logs when the row cap is hit.
- F6: reject out-of-range date-only filters (2025-02-31) instead of normalizing.
- F2: regenerate package-lock.json for the 0.0.54 data-schemas bump.

Tests: +1 methods (bulkWrite) +2 verify (deleted-prefix / checkpoint mismatch),
updated purge test for checkpoint flow; +4 api (re-assert skip, assign/revoke
fail-closed rollback, date reject, CSV truncation marker).

* 🛠️ fix: Address Codex round-2 on the audit-log substrate

- R2-1/R2-5 (P1/P2): base the grant.assigned audit decision on the atomic
  upsert result. grantCapability now returns { grant, created } via
  includeResultMetadata; the handler audits only when created. Removes the racy
  pre-read, which also mis-handled inherited platform grants vs a new
  tenant-scoped insert and concurrent double-assign.
- R2-2 (P2): namespace tenant chain keys (tenant:<id>) so a tenant whose id is
  literally the platform sentinel can't share the platform audit chain.
- R2-4 (P2): validate literal calendar tokens for full ISO timestamps too, so
  2025-02-31T00:00:00Z is rejected instead of normalizing to March 3.

Tests updated for the grantCapability { grant, created } contract (systemGrant +
grants specs) and the namespaced chain key (auditChainKey helper); +1 api date
case. data-schemas 141, api grants/audit 107 green.

R2-3 (deprecated actorId/targetPrincipalId aliases): not reinstating — the
surface is pre-release and its only consumer (admin-panel PR) migrates to the new
shape in lockstep, so there are no legacy clients to support.
R2-6 (role-deletion cascade emits no grant.removed): valid but a separate
workflow in roles.ts; tracked as a follow-up to keep this PR scoped.

* 🛠️ fix: Address Codex round-3 on the audit-log substrate

- R3-3 (P2): make a grant re-assert a true no-op — move grantedAt/grantedBy to
  $setOnInsert so an existing grant is never silently mutated when the audit is
  skipped (created:false now means nothing changed). grantedAt/grantedBy record
  the original grant.
- R3-2 (P2): report CSV export truncation exactly. streamAuditLogEntries returns
  { count, truncated }; truncated is true only when rows existed beyond the cap,
  so an exact-cap export is no longer falsely marked truncated.
- R3-5 (P2): block AuditLog.insertMany (another bulk path that skips the save
  hook and could inject forged seq/prevHash/hash and poison the chain).

Tests: +insertMany rejection, +exact-cap vs truncated stream cases, +exact-cap
export-not-truncated handler case. ds 142, api 108 green.

R3-1 (deprecated query aliases) and R3-4 (role-deletion cascade audit) are
re-flags of R2-3/R2-6 — holding the prior decisions (pre-release surface; separate
roles.ts workflow tracked as a follow-up), pending maintainer direction.

* 🛡️ feat: Audit grant removals from the role-deletion cascade

Closes the forensic gap Codex flagged (R2-6/R3-4): deleting a role removed its
SystemGrants with no audit entries. `deleteGrantsForPrincipal` now returns the
removed grants, and the role-deletion handler emits a `grant.removed` audit entry
per removed grant (actor = caller, target = role, metadata.capability, request
context), matching the explicit revoke endpoint. Fail-open — the role is already
deleted, so a failed audit is logged, not propagated; sequential to keep the
per-tenant hash chain ordered.

Extracted `buildAuditContext` to admin/context.ts (shared by grants + roles).
Tests: role-deletion emits one entry per grant / none when no grants; ds 110,
api admin 202 green.

* 🛠️ fix: Address Codex round-4 on the audit-log substrate

- R4-1 (P2): don't silently drop an audit row under heavy append contention.
  recordAuditEntry now retries duplicate-key seq collisions up to 12× with
  jittered backoff (was 5, no backoff), so realistic bursts of parallel admin
  writes resolve; the failClosed escape still applies on true exhaustion.
- R4-3 (P2): purge a contiguous seq prefix, not a date range. createdAt is
  app-generated, so under multi-instance clock skew a later seq can carry an
  earlier timestamp; a raw date delete could remove an interior row and break
  verification. purgeAuditLogEntries now resolves the date to the first retained
  seq and deletes only strictly-lower seqs, keeping the remaining chain contiguous.

Tests: +clock-skew purge case (no gap created). ds auditLog 33 green.

R4-2 (role-deletion grant audit) is a re-flag of R2-6/R3-4, already implemented
in 15472127d6 (roles.ts emitGrantRemovals + route wiring + tests); the finding's
cited line numbers predate that commit.

* 🛠️ fix: Address Codex round-5 on the audit-log substrate

- R5-1 (P2): scope each cascade grant.removed entry to the removed grant's own
  tenant, not the caller's. A platform admin deleting a role can remove
  tenant-scoped grants; those removals now land in the affected tenant's chain.
- R5-2 (P2): only return a purge checkpoint when rows were actually deleted. A
  no-op confirmed purge no longer mints a trust boundary that could legitimize a
  prefix it didn't authorize.
- R5-3 (P2): ensure the unique { chainKey, seq } index exists before appending
  (memoized createIndexes), so serialization doesn't depend on a background build
  — closes a silent chain-fork window under MONGO_AUTO_INDEX=false or at startup.

Tests: +per-grant-tenant cascade audit, +no-op-purge-no-checkpoint,
+index-built-before-append. ds auditLog 35, api roles 95 green.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-18 15:42:33 -04:00
Dustin Healy
fa20003952
🛂 refactor: Accept Targeted assign:configs for Config Scope-Lifecycle Endpoints (#13773)
* 🔓 fix: Accept Targeted assign:configs for Config Scope-Lifecycle Endpoints

Three admin-config endpoints currently require broad manage:configs: PUT /:principalType/:principalId for empty-overrides scope creation, DELETE /:principalType/:principalId for scope removal, and PATCH /:principalType/:principalId/active for the active toggle. The capability model already defines assign:configs:user|group|role for delegated administrators and validates that shape in isValidCapability, but no handler accepts it, so a delegate granted assign:configs:role via /api/admin/grants cannot manage scope lifecycle for the principal type they were explicitly delegated.

This aligns the server-side auth with the documented capability surface. Every destructive lateral path stays behind broad manage:configs: operations against the base config principal (__base__), non-empty PUT payloads that $set the full overrides field, and DELETE or toggle on a document whose existing overrides are non-empty (which would erase or neutralize sections the caller could not author). The new hasCapability dep on AdminConfigDeps is optional with a false default, so external consumers continue to get pre-PR behavior until they wire the resolver.

* 🛡️ fix: Block Assign-Only Scope-Lifecycle When Existing Doc Has Tombstones

The existing-overrides guard introduced in the prior commit only checked overrides, but configs also carry tombstones (suppressed inherited field paths) which are iterated during cascade resolution. An assign-only caller could delete, toggle, or empty-upsert a doc whose overrides is empty but whose tombstones is non-empty, which would erase or neutralize suppressions on fields they could not author. Extends the guard at all three call sites to treat a non-empty tombstones array as destructive state.

* 🚨 fix: Log TOCTOU Race When Assign-Only Lifecycle Op Hits Non-Empty Doc

The empty-state guard for assign-only callers performs a read-then-write across two DB roundtrips, so a concurrent broad-manage write can land between the guard and the destructive op. Adds post-write detection on the delete and toggle handlers: when the destructive op returns a doc whose state was non-empty at write time, emit logger.warn with the caller id, principal, and observed-state counts so ops can detect the race and restore from audit logs.

A fully atomic fix would require extending deleteConfig, toggleConfigActive, and upsertConfig in packages/data-schemas/src/methods/config.ts to support compare-and-swap filters, which is a wider design change than this PR's auth scope. Empty-payload upsert is not covered because $set replaces overrides, so the post-write doc no longer reflects pre-write state.

* 🔒 fix: Atomic Empty-State Filter for Assign-Only Scope-Lifecycle Writes

Replaces the read-then-check guard with an atomic Mongo filter on the destructive write itself. Adds an options.expectEmpty parameter to deleteConfig, toggleConfigActive, and upsertConfig in the shared data-schemas layer. When set, the filter requires both overrides and tombstones to be empty before the write matches. The TOCTOU race window is eliminated: a concurrent write cannot land between the empty-state check and the destructive op because they are now a single atomic operation.

For upsertConfig, the E11000 retry path returns null instead of falling back to a filterless update when expectEmpty is set, preserving the atomic property. Handlers fall back to findConfigByPrincipal only to disambiguate the null return between 404 (doc absent) and 403 (doc exists with non-empty state). The post-write logger.warn race detection added in the prior commit is removed as unreachable.
2026-06-18 15:40:58 -04:00
Dustin Healy
84886c56fb
🧷 fix: Preserve Document Priority on Section-Scoped Config Writes (#13772)
The patch and tombstone admin-config handlers accept a priority field on the request body, which controls the position of the entire Config document in the precedence cascade. Today, that priority is written unconditionally, even when the caller holds only a section-scoped grant such as manage:configs:memory. On a document containing overrides for other sections, this lets a section-scoped caller silently reorder overrides they have no permission to author.

This is a defense-in-depth fix for deployments using section-scoped grants. In a vanilla setup where admins hold broad manage:configs, the path is unreachable because the broad-capability short-circuit lets every priority change through, so the fix is a no-op for those callers. The change makes the contract safe-by-construction for any deployment that narrows the auth model, at no cost to upstream behavior.
2026-06-18 15:39:44 -04:00
Danny Avila
62ed62ac6e
🏘️ fix: Scope Skill Sync Status (#13771)
* fix: Scope Skill Sync Status by Tenant

* fix: Preserve Unscoped Skill Sync Status

* fix: Filter Inherited Skill Sync Sources
2026-06-15 15:23:49 -04:00
Danny Avila
197a1dc4e2
🧬 feat: Add GitHub Skill Sync (#13293)
Some checks failed
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
* feat: Add GitHub skill sync

* fix: Address GitHub skill sync CI

* fix: Harden GitHub skill sync review paths

* fix: Prevent overlapping skill sync runs

* fix: Address GitHub skill sync review findings

* fix: Satisfy Git ref lint rule

* fix: Address GitHub sync review follow-ups

* fix: Match skill frontmatter closing fence

* fix: Address GitHub sync review cycle

* fix: Address GitHub sync review follow-ups

* fix: Harden GitHub skill sync worker

* fix: Format GitHub sync rollback log

* fix: Address GitHub sync review feedback

* fix: Format skill import parse handling

* fix: Coerce scalar skill frontmatter and correct scheduler timer clear

- parse: coerce numeric/boolean name and description scalars to strings instead of dropping them to empty (restores pre-refactor behavior; preserves absent-vs-empty distinction for the when-to-use fallback)
- scheduler: clear the setTimeout handle with clearTimeout rather than clearInterval
- test: cover non-string scalar frontmatter coercion

* fix: Tolerate trailing whitespace after SKILL.md opening frontmatter fence

extractFrontmatterBlock required the opening fence to be exactly '---\n', so an opener with trailing spaces/tabs (e.g. '---   \n') silently dropped all frontmatter even though the closing-fence regex already tolerates it. Match the opener with /^---[ \t]*\n/ for symmetry. Addresses Codex P3 (parse.ts:24).

* feat: Run GitHub skill sync under a per-source tenant context

Under TENANT_ISOLATION_STRICT, the sync ran with no async tenant context, so the tenant-isolation mongoose hooks threw on every Skill/SkillFile/AclEntry operation; in non-strict mode synced skills were written tenant-less and never matched tenant-scoped reads. Add an optional per-source tenantId to the skillSync config; when set, each source sync runs inside tenantStorage.run({ tenantId }) so skills, files, and public ACL grants are created and listed within that tenant, and the skill row is stamped with the tenantId for correct dedup. Sources without tenantId keep the prior single-tenant behavior. Avoids runAsSystem. Addresses Codex P2 (sync.js:70).

Lock/status/credential bookkeeping stays outside the tenant context (those collections are intentionally global).

* test: Restore dropped tenant-context coverage for GitHub skill sync

The prior commit shipped the getTenantId import in github.spec.ts without the tenant tests that use it (lost in an interrupted edit), which failed the eslint --max-warnings=0 CI job on an unused import. Restore both github.spec.ts tenant tests (tenant-scoped run stamps tenantId and executes inside the tenant ALS context; no-tenant run stays ambient) and the two config-schemas tenant tests (accepts tenantId, rejects __SYSTEM__).

* test: Restore dropped github.spec tenant-context tests

The previous commit's github.spec.ts edit did not apply (anchor mismatch), so the getTenantId import remained unused and failed eslint --max-warnings=0. Add the two tenant tests that use it: a tenant-scoped run stamps tenantId and executes inside the tenant ALS context, and a no-tenant run stays ambient.

* feat: Scope synced skill author to tenant and harden tenant-context sync

Addresses the latest Codex review on the per-source tenant change:
- makeSourceAuthorId now folds tenantId into the synthetic author hash so the
  same source mirrored into different tenants gets distinct author ids (clearer
  audits, no cross-tenant author collisions). Single-tenant author ids stay
  stable (suffix omitted when tenantId is absent).
- syncSourceInTenantContext uses an async callback per the tenant-context
  contract so the ALS store propagates across awaited Mongoose calls.
- Tests: same-source/different-tenant yields distinct authors; mirror cleanup
  is scoped to the source and deletes only its absent-upstream skills.

* fix: Repair tsc error and guard external edits in github skill sync

- Fix TS2352 in github.spec mirror-cleanup test: build the existing-skill mock via makeSkill with authorName instead of an under-typed 'as CreateSkillInput' cast (this was the failing TypeScript CI check on f00ce3c5a).
- 808: commitExistingRemoteSkillAfterFileSync re-reads to clear our own file-sync version bumps, but now compares refreshed content against the pre-sync snapshot (body/name/description/always-apply) and throws SKILL_CONFLICT on a concurrent external edit instead of overwriting it.

* docs: Note skillSync source tenantId is effectively immutable

Changing/adding/removing a source's tenantId orphans previously mirrored skills in the old tenant (a tenant-scoped sync cannot clean another tenant's data without runAsSystem, which is intentionally avoided).

* fix: Key GitHub skill upstream identity on source id and path only

Addresses Codex finding (github.ts:217): makeUpstreamId previously included owner/repo, so repointing a source to a renamed or replacement repository (same source id) changed the upstreamId, made findSkillBySourceIdentity miss the existing mirror, and then collided on the (name, author, tenantId) uniqueness constraint — leaving the source stuck failing. Identity now keys on the stable source id + root path only. The feature is unreleased, so there is no stored-id migration. Updated spec upstreamId fixtures to the new format; the existing ref-independent identity test now also covers repo moves.

* fix: Scope GitHub skill mirror deletion to the source tenant

Addresses Codex P1 (github.ts:1047/1057): an ambient source (no tenantId) runs listSkillsBySource without tenant context, which under non-strict isolation returns github-synced skills across all tenants. The mirror-deletion pass then treated other tenants' skills as absent-upstream and could delete them. Filter existingSyncedSkills to rows whose tenantId matches the source's configured tenantId (absent = its own ambient bucket) before deleting, so a sync never removes another tenant's mirrored skills. Covered by a test where an ambient run leaves a tenant-b-owned skill untouched.

* fix: Apply tenant-scoped mirror deletion implementation

The prior commit (75ccfa3fc) added the test but the source change to github.ts was lost in an interrupted edit, leaving a failing test with no implementation. This adds the actual guard: the mirror-deletion pass skips skills whose tenantId does not match the source's configured tenantId (absent = ambient bucket), so an ambient source whose listSkillsBySource returns cross-tenant rows under non-strict isolation cannot delete another tenant's mirrored skills.

* fix: Resolve global access role outside tenant context for synced skill grants

Addresses Codex P2 (github.ts:1166): default access roles (incl. skill_viewer) are seeded globally with no tenantId under runAsSystem, but a tenant-scoped sync wraps ensurePublicViewer in the source's tenant context. The PermissionService grantPermission resolved the role via a tenant-isolated AccessRole query, so the global role did not match and tenant-scoped syncs failed with 'Role skill_viewer not found'. The sync adapter now resolves the role inside runAsSystem (matching the global seed) and writes the ACL entry in the active tenant context, so the AclEntry is tenant-scoped (visible to tenant users) while the role lookup still succeeds. Covered by service tests for the resolve-vs-write split and the missing-role failure.

* fix: Strip placeholder frontmatter booleans and check skill conflict before file sync

- 1083 (github.ts:759): toCleanFrontmatter now drops a non-boolean always-apply (e.g. the 'always-apply:' / 'always-apply: # TODO' placeholder, which js-yaml yields as null). The boolean is already captured in the dedicated alwaysApply field; persisting null left ambiguous frontmatter on the synced skill.
- 1080 (github.ts:1057): for an existing mirrored skill, check for an external content edit (via getSkillById + hasExternalSkillEdit) BEFORE syncSkillFiles mutates the bundled files, so a concurrently edited skill fails fast with SKILL_CONFLICT without partial file rewrites. The post-file-sync check still guards edits that land during the file sync window.
Tests: placeholder always-apply is dropped from synced frontmatter; concurrent-edit conflict leaves files unmutated (no upsert/delete).

* fix: Harden GitHub skill sync review paths

* fix: Reuse moved GitHub skill mirrors

* fix: Scope GitHub sync identity conflicts

* test: Fix GitHub sync conflict mock typing

* fix: Support nested env-backed skill sync

* fix: Keep skill sync config base-only

* fix: Scope GitHub skill identity lookup by tenant

* fix: Harden GitHub skill sync admin gates

* fix: Guard existing skill sync permission grants

* feat: Trigger skill sync from resolved config

* fix: Scope resolved skill sync by tenant

* test: Allow manual skill sync status tenant scoping

* refactor: Extract skill sync trigger orchestrator

* test: Complete orchestrator status fixture

* chore: Bump data provider version

* fix: Restrict skill sync server credentials

* test: Complete admin skill sync status fixtures

* fix: tighten skill sync trigger safeguards

* fix: preserve alwaysApply skill sync alias

* chore: sort skill sync imports

* fix: preserve skill sync request scope

* fix: harden skill sync review edges

* refactor: move skill sync admin access to api package

* fix: add skill sync declaration return types

* fix: satisfy skill sync type checks

* fix: resolve codex skill sync review findings

* fix: harden skill sync review edges

* fix: resolve codex skill sync edge findings

* fix: satisfy API declaration build after rebase
2026-06-10 21:05:54 -04:00
Danny Avila
6bc75d24c8
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown

Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the
@librechat/api package build, mirroring the merged data-schemas migration.

- Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps,
  bundle first-party `~/` + relative imports)
- Annotate exports for isolatedDeclarations (codefix-driven). Collapse the
  tokens.ts model->token maps to Record<string, Record<string, number>> and
  switch validation.ts's runtime `files` field from z.any() to z.unknown()
  so no explicit `any` is introduced
- Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output
- Add src/telemetry.ts entry shim so the two index.ts entries don't collide
  in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts})
- Delete rollup.config.js

Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass,
both entries load via require(), legacy /api consumes them unchanged.

* 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys

The build-api cache keys hashed `packages/api/server-rollup.config.js`,
which never existed (api used `rollup.config.js`, now removed) — a copy-paste
artifact from the data-provider key that matched no file. Replace it with the
new `packages/api/tsdown.config.mjs` so edits to the build config (entry,
format, externals) bust the api build cache, matching the data-schemas key.
2026-06-08 10:54:48 -04:00
Danny Avila
aeb5adff34
🪦 fix: Add Durable MCP Config Tombstones (#13534)
* fix: add durable MCP config tombstones

* fix: preserve scoped config tombstones

* fix: clean up config tombstone lint

* fix: handle empty model spec skill allowlist

* fix: preserve inactive config tombstones
2026-06-05 15:05:40 -04:00
Dustin Healy
2bcf3e8582
🪟 fix: Apply Admin-Panel Config Overrides To YAML-Defined MCP Servers (#13173)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🪟 fix: Apply Admin-Panel Config Overrides To YAML-Defined MCP Servers

Admin-panel saves of MCP server fields for YAML-defined servers were
silently dropped by the registry. ensureConfigServers filtered out any
merged config entry whose name appeared in YAML, so overrides such as
iconPath, title, and description never reached getAllServerConfigs even
though the override row had been written to the configs collection and
the AppConfig merge layer had produced the correct merged result.

The filter is removed and replaced with a content-equivalence
short-circuit in ensureSingleConfigServer. YAML-defined servers whose
merged config matches the YAML cache entry skip lazy-init, so unmodified
YAML servers still avoid a redundant inspection round trip. The new
private helper matchesYamlConfig reuses the existing content-hash
function on configurable fields only.

getAllServerConfigs now overlays config-tier entries onto the YAML base
while preserving user-DB entries (source: 'user'), giving precedence of
YAML, then Config tier, then User DB. The docstring is updated to
describe the new order.

Multi-tenancy is already enforced upstream of the registry by the
AppConfig layer, so the registry stays tenant-agnostic and overrides
remain isolated per tenant.

Tests cover the new behavior: config-tier override on YAML-defined
server flows through to getAllServerConfigs, YAML servers without
effective overrides skip lazy-init, user-DB entries win over
config-tier overlays, pure config-tier servers still lazy-init, and the
merged config passed to lazy-init preserves all YAML fields when the
override only adds new ones.

* 🛠️ fix: Address Review Feedback For YAML Override Precedence

Both Copilot and Codex flagged matchesYamlConfig as broken in
production: the cached YAML config carries inspector-derived defaults
(requiresOAuth defaulted to false when YAML omits it, serverInstructions
rewritten from the YAML toggle to the fetched server-instructions
string, and so on) that are absent from appConfig.mcpConfig. The
content-hash comparison reports a mismatch for every YAML server with
no admin-panel override, so ensureSingleConfigServer re-inspects all of
them anyway. The optimization never fires in practice and reintroduces
the source tagging it was supposed to avoid.

Remove the short-circuit and the matchesYamlConfig helper. YAML-defined
servers that appear in the merged config go through lazy-init like any
other entry. A smarter optimization that overlays cosmetic-only
override fields onto the YAML cache without re-inspection belongs in a
follow-up once the boundary between configurable fields and
inspector-derived fields is well defined.

Update the existing ensureConfigServers tests that asserted the old
filter behavior (should exclude YAML servers from config-source
detection, should return empty when all servers are YAML) to assert
the new behavior: YAML servers pass through ensureConfigServers and
are lazy-initialized when they appear in the merged config. Make the
inspector mock more realistic by spreading the raw input first and
overlaying only runtime fields, so the test fixtures match production
where the inspector preserves configurable fields. Drop the companion
test in MCPServersRegistry.test.ts that asserted the short-circuit
fires for unchanged YAML servers; the hand-crafted fixture skipped
inspector defaults and was not representative.

Copilot also flagged that getServerConfig short-circuits to
configServers before checking the user DB, so the precedence enforced
in getAllServerConfigs (user-DB beats config-tier) was bypassed in the
single-server lookup. When configServers carries an entry and a userId
is available, check the user DB first and prefer a source: 'user'
entry so per-user servers are never shadowed by an admin-panel
override.

* 🛡️ fix: Harden Admin Override Overlay Against Failure Stubs

Hardens the admin-panel override path against transient inspect failures
and removes a defensive branch that guarded an impossible state.

The getAllServerConfigs overlay now skips failed-inspection stubs so a
healthy YAML or DB entry stays visible during the 5-minute retry window
instead of being clobbered by a stub. When the overlay does land, the
base entry's source tier is preserved, which keeps Tools/mcp.js routing
its failed-inspection recovery to the correct storage location.

The user-DB precedence block in getServerConfig is removed: configServers
is built from appConfig.mcpConfig which only ever carries admin-tier
entries, so the DB lookup defended a state that cannot occur via the
current call graph. The dead yamlServerNames memoization is also gone.

Adds two regression tests covering inspection-failure preservation and
source-tier preservation on successful overlay, and adds a debug log
when an admin override is suppressed by a user-tier entry. The
makeParsedConfig test factory now honors overrides correctly.

* 🧪 test: Strengthen Admin Override Coverage And Docs

Adds an end-to-end regression test that chains MCPServerInspector.inspect
failure through ensureConfigServers and getAllServerConfigs, asserting
the healthy YAML base entry survives a transient inspect failure
intact. The previous regression test hand-built a failure stub and
skipped ensureSingleConfigServer, leaving the production chain itself
untested.

The getAllServerConfigs docstring now spells out both overlay guards
(failed-stub skip and user-tier preservation) and the source-field
preservation contract that downstream recovery logic depends on.
The yamlLangfuseConfig test fixture is frozen so a future test cannot
mutate it and contaminate sibling tests in the describe block.

* 🔧 fix: Skip Lazy-Init For Unchanged YAML MCP Servers

Adds an admin-configurable-field equivalence check so YAML-defined MCP
servers that carry no admin override skip lazy-init in
ensureConfigServers. This avoids the per-request inspect storm and
keeps unmodified YAML servers out of the config-tier cache, so admin
saves that touch unrelated overrides no longer evict and tear down
those YAML connections.

A second guard in getServerConfig prevents failed-inspection stubs in
configServers from shadowing the healthy YAML base entry for the
duration of the retry window. The aggregate path already had this
guard via getAllServerConfigs; this brings the single-server path to
parity, so Tools/mcp.js recovery routes to YAML reinspection rather
than bailing on the config retry timer.

Adds three regression tests covering the unmodified-YAML skip, the
admin-override lazy-init trigger, and the failure-stub fallthrough.
Updates two existing ensureConfigServers tests that previously
documented the now-incorrect "always lazy-init YAML" behavior.

* 🔓 feat: Expose baseOnly Flag On Admin Config Base Endpoint

The admin getBaseConfig handler now reads req.query.baseOnly and
forwards it to getAppConfig so an admin panel client can request the
un-merged YAML and AppService base configuration without DB overrides
applied. The flag is opt-in; existing callers see no behaviour change
because the default remains the merged response.

The query value is coerced through String() so Express array forms
like baseOnly=true&baseOnly=true are treated as false rather than
truthy by accident. A handler test pins the forwarding behaviour and
the default-merged behaviour against future regressions.

* 🧹 fix: Address Codex Review Findings On MCP Registry Precedence Path

Four follow-ups from the Codex review of PR #13173:

P1. getServerConfig now preserves the configServers candidate as a
last-resort fallback when both YAML cache and user DB return nothing,
so admin-defined config-only servers carrying inspectionFailed=true
still surface the failure stub to callers in api/server/services/Tools/mcp.js
that rely on it to return the still-unreachable message. The
not-found memoization is preserved.

P2a. proxy is added to ADMIN_CONFIGURABLE_FIELDS so an admin override
on SSE/streamable-http proxy is no longer treated as an unchanged YAML
server and correctly triggers lazy-init.

P2b. isUnmodifiedYamlServer now treats absent-on-rawConfig fields as
equal, so inspector-derived values on the cached YAML entry
(notably requiresOAuth filled in by detectOAuth at startup) do not
force unmodified YAML servers to re-init on every request.

P3. getBaseConfig parses ?baseOnly strictly against the literal string
true instead of String-coercing, so array shapes like baseOnly[]=true
no longer pass through.

Regression tests cover all four paths.

* 🧹 fix: Drop Misleading Shadow Warning On Config Vs User-DB Collisions

The Config-tier branch of warnOnOperatorManagedNameCollisions logged
that Config MCP servers shadow DB-backed servers, but
getAllServerConfigs actually preserves the user-tier entry on a
Config-vs-user collision and skips the override. The warning was
describing the opposite of what the code does and would mislead
operational debugging.

The YAML-tier call is unchanged because YAML still legitimately
shadows DB-backed servers. The per-entry debug log inside the
collision branch already captures the actual outcome.

Test renamed and rewritten to assert the user-tier entry is
preserved and no shadow warning is emitted.

* 🔒 fix: Keep Tenant-Scoped configServers Candidate Out Of The Global Read-Through Cache

The prior fix for surfacing inspectionFailed stubs from admin-defined
config-only servers wrote the per-call configServers candidate into
readThroughCache when YAML and DB both missed. The cache key is keyed
by serverName plus userId, so a failed stub from one tenant could
satisfy a later no-userId lookup made by another tenant before any
configServers resolution ran.

getServerConfig now caches only the global YAML/DB resolution (still
caching undefined to memoize not-found lookups) and uses the candidate
strictly as an unmemoized function-level fallback that surfaces the
failure stub to the caller without leaking it across tenants.

Regression test exercises a no-userId call after a tenant-scoped
failure and asserts the cache returns undefined rather than the
stub, and that a second tenant sees their own healthy candidate.

* 🔄 fix: Mirror getAllServerConfigs Precedence Exactly In getServerConfig

getServerConfig was short-circuiting with the configServers candidate
on every healthy lookup, which made single-server callers diverge from
the aggregate path for name collisions between config-tier overrides
and user-DB entries. The aggregate path preserves the user-tier entry
on such collisions, so single-server callers saw the admin override
while list views saw the user server for the same name.

getServerConfig now resolves the YAML/DB base first and applies the
same four-step precedence used in getAllServerConfigs:

  1. user-tier base wins absolutely over a config-tier candidate
  2. healthy YAML/DB base wins over a failed (inspectionFailed)
     candidate
  3. healthy candidate overlays its fields onto the base, preserving
     the base entry's source tag so downstream recovery routes to the
     correct storage location
  4. with no base, the candidate is returned as-is for config-only
     servers

readThroughCache still memoizes only the global YAML/DB lookup, so
the per-call configServers candidate never enters the cache and the
tenant-isolation guarantee from the previous fix is preserved.

Regression tests cover the user-wins-over-config case and the
YAML-overlay-with-yaml-source-preserved case.

* ⚡ perf: Batch YAML Cache Read In ensureConfigServers

isUnmodifiedYamlServer was calling cacheConfigsRepo.get(serverName)
per entry. In the Redis aggregate-key backend, get() is implemented
as getAll() then map lookup, so N concurrent per-server lookups
inflate into N full-map reads and deserializations on every
ensureConfigServers pass.

The loop now takes a single getAll() snapshot at the top and hands
it into a synchronous isUnmodifiedYamlServer helper, turning O(n)
remote reads into O(1) regardless of how many MCP entries are
resolved. The snapshot also gives the unchanged-YAML comparison
one consistent view of YAML across all entries.

Regression test spies on cacheConfigsRepo.get and asserts it is
never called from ensureConfigServers, with getAll called exactly
once.
2026-05-23 17:11:13 -04:00
Danny Avila
ea28dbfa89
🧹 chore: Clean Up Config Fields (#12537)
* chore: remove unused `interface.endpointsMenu` config field

* chore: address review — restore JSDoc UI-only example, add Zod strip test

* chore: remove unused `interface.sidePanel` config field

* chore: restrict fileStrategy/fileStrategies schema to valid storage backends

* fix: use valid FileStorage value in AppService test

* chore: address review — version bump, exhaustiveness guard, JSDoc, configSchema test

* chore: remove debug logger.log from MessageIcon render path

* fix: rewrite MessageIcon render tests to use render counting instead of logger spying

* chore: bump librechat-data-provider to 0.8.407

* chore: sync example YAML version to 1.3.7
2026-04-03 12:22:58 -04:00
Danny Avila
f8405e731b
🗂️ fix: Allow Empty-Overrides Scope Creation in Admin Config (#12492)
* fix: Allow empty-overrides scope creation when priority is provided

The upsertConfigOverrides handler short-circuited when overrides was
empty, returning a plain message instead of creating the config document.
This broke the admin panel's "create blank scope" flow which sends
`{ overrides: {}, priority: N }` — the missing `config` property in the
response caused an `_id` error on the client.

The early return now only triggers when both overrides are empty and no
priority is provided. Per-section permission checks are scoped to cases
where override sections are actually present.

* test: Add tests for empty-overrides scope creation with priority

* test: Address review nits for empty-overrides scope tests

- Add res.statusCode/res.body assertions to capability-check test
- Add 403/401 tests for empty overrides + priority path
- Use mockResolvedValue(null) for consistency on bare jest.fn()
- Remove narrating comment; fold intent into test name
2026-03-31 21:46:48 -04:00
Dustin Healy
2451bf54cf
🛡️ fix: Restrict System Grants to Role Principals (#12491)
* 🛡️ fix: restrict system grants to role principals only

Narrows GrantPrincipalType to PrincipalType.ROLE, rejecting GROUP and
USER with 400. Removes grant cascade cleanup from group/user deletion
handlers and their route wiring since only roles can hold grants.

* 🛡️ fix: address review findings for grants roles-only restriction

Add missing GROUP rejection test for revokeGrant (symmetric with
getPrincipalGrants and assignGrant coverage), add extensibility comment
to GrantPrincipalType, and document the checkRoleExists guard.
2026-03-31 19:25:14 -04:00
Danny Avila
c0ce7fee91
🚫 refactor: Remove Interface Config from Override Processing (#12473)
Add INTERFACE_PERMISSION_FIELDS set defining the interface fields that
seed role permissions at startup (prompts, agents, marketplace, etc.).
These fields are now stripped from DB config overrides in the merge
layer because updateInterfacePermissions() only runs at boot — DB
overrides for these fields create a client/server permission mismatch.

Pure UI fields (endpointsMenu, modelSelect, parameters, presets,
sidePanel, customWelcome, etc.) continue to work in overrides as
before.

YAML startup path is completely unaffected.
2026-03-31 11:07:31 -04:00
Dustin Healy
3d1b883e9d
👨‍👨‍👦‍👦 feat: Admin Users API Endpoints (#12446)
* feat: add admin user management endpoints

Add /api/admin/users with list, search, and delete handlers gated by
ACCESS_ADMIN + READ_USERS/MANAGE_USERS system grants. Handler factory
in packages/api uses findUsers, countUsers, and deleteUserById from
data-schemas.

* fix: address convention violations in admin users handlers

* fix: add pagination, self-deletion guard, and DB-level search limit

- listUsers now uses parsePagination + countUsers for proper pagination
  matching the roles/groups pattern
- findUsers extended with optional limit/offset options
- deleteUser returns 403 when caller tries to delete own account
- searchUsers passes limit to DB query instead of fetching all and
  slicing in JS
- Fix import ordering per CLAUDE.md, complete logger mock
- Replace fabricated date fallback with undefined

* fix: deterministic sort, null-safe pagination, consistent search filter

- Add sort option to findUsers; listUsers sorts by createdAt desc for
  deterministic pagination
- Use != null guards for offset/limit to handle zero values correctly
- Remove username from search filter since it is not in the projection
  or AdminUserSearchResult response type

* fix: last-admin deletion guard and search query max-length

- Prevent deleting the last admin user (look up target role, count
  admins, reject with 400 if count <= 1)
- Cap search query at 200 characters to prevent regex DoS
- Add tests for both guards

* fix: include missing capability name in 403 Forbidden response

* fix: cascade user deletion cleanup, search username, parallel capability checks

- Cascade Config, AclEntry, and SystemGrant cleanup on user deletion
  (matching the pattern in roles/groups handlers)
- Add username to admin search $or filter for parity with searchUsers
- Parallelize READ_* capability checks in listAllGrants with Promise.all

* fix: TOCTOU safety net, capability info leak, DRY/style cleanup, data-layer tests

- Add post-delete admin recount with CRITICAL log if race leaves 0 admins
- Revert capability name from 403 response to server-side log only
- Document thin deleteUserById limitation (full cascade is a future task)
- DRY: extract query.trim() to local variable in searchUsersHandler
- Add username to search projection, response type, and AdminUserSearchResult
- Functional filter/map in grants.ts parallel capability check
- Consistent null guards and limit>0 guard in findUsers options
- Fallback for empty result.message on delete response
- Fix mockUser() to generate unique _id per call
- Break long destructuring across multiple lines
- Assert countUsers filter and non-admin skip in delete tests
- Add data-layer tests for findUsers limit, offset, sort, and pagination

* chore: comment out admin delete user endpoint (out of scope)

* fix: cast USER principalId to ObjectId for ACL entry cleanup

ACL entries store USER principalId as ObjectId (via grantPermission casting),
but deleteAclEntries is a raw deleteMany that passes the filter through.
Passing a string won't match stored ObjectIds, leaving orphaned entries.

* chore: comment out unused requireManageUsers alongside disabled delete route

* fix: add missing logger.warn mock in capabilities test

* fix: harden admin users handlers — type safety, response consistency, test coverage

- Unify response shape: AdminUserSearchResult.userId → id, add AdminUserListItem type
- Fix unsafe req.query type assertion in searchUsersHandler (typeof guards)
- Anchor search regex with ^ for prefix matching (enables index usage)
- Add total/capped to search response for truncation signaling
- Add parseInt radix, remove redundant new Date() wrap
- Add tests: countUsers throw, countUsers call args, array query param, capped flag

* fix: scope deleteGrantsForPrincipal to tenant, deterministic search sort, align test mocks

- Add tenantId option to AdminUsersDeps.deleteGrantsForPrincipal and
  pass req.user.tenantId at the call site, matching the pattern already
  used by the roles and groups handlers
- Add sort: { name: 1 } to searchUsersHandler for deterministic results
- Align test mock deleteUserById messages with production output
  ('User was deleted successfully.')
- Make capped-results test explicitly set limit: '20' instead of
  relying on the implicit default

* test: add tenantId propagation test for deleteGrantsForPrincipal

Add tenantId to createReqRes user type and test that a non-undefined
tenantId is threaded through to deleteGrantsForPrincipal.

* test: remove redundant deleteUserById override in tenantId test

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-03-30 23:06:50 -04:00
Dustin Healy
a4a17ac771
⛩️ feat: Admin Grants API Endpoints (#12438)
* feat: add System Grants handler factory with tests

Handler factory with 4 endpoints: getEffectiveCapabilities (expanded
capability set for authenticated user), getPrincipalGrants (list grants
for a specific principal), assignGrant, and revokeGrant. Write ops
dynamically check MANAGE_ROLES/GROUPS/USERS based on target principal
type. 31 unit tests covering happy paths, validation, 403, and errors.

* feat: wire System Grants REST routes

Mount /api/admin/grants with requireJwtAuth + ACCESS_ADMIN gate.
Add barrel export for createAdminGrantsHandlers and AdminGrantsDeps.

* fix: cascade grant cleanup on role deletion

Add deleteGrantsForPrincipal to AdminRolesDeps and call it in
deleteRoleHandler via Promise.allSettled after successful deletion,
matching the groups cleanup pattern. 3 tests added for cleanup call,
skip on 404, and resilience to cleanup failure.

* fix: simplify cascade grant cleanup on role deletion

Replace Promise.allSettled wrapper with a direct try/catch for the
single deleteGrantsForPrincipal call.

* fix: harden grant handlers with auth, validation, types, and RESTful revoke

- Add per-handler auth checks (401) and granular capability gates
  (READ_* for getPrincipalGrants, possession check for assignGrant)
- Extract validatePrincipal helper; rewrite validateGrantBody to use
  direct type checks instead of unsafe `as string` casts
- Align DI types with data layer (ResolvedPrincipal.principalType
  widened to string, getUserPrincipals role made optional)
- Switch revoke route from DELETE body to RESTful URL params
- Return 201 for assignGrant to match roles/groups create convention
- Handle null grantCapability return with 500
- Add comprehensive test coverage for new auth/validation paths

* fix: deduplicate ResolvedPrincipal, typed body, defensive auth checks

- Remove duplicate ResolvedPrincipal from capabilities.ts; import the
  canonical export from grants.ts
- Replace Record<string, unknown> with explicit GrantRequestBody interface
- Add defensive 403 when READ_CAPABILITY_BY_TYPE lookup misses
- Document revoke asymmetry (no possession check) with JSDoc
- Use _id only in resolveUser (avoid Mongoose virtual reliance)
- Improve null-grant error message
- Complete logger mock in tests

* refactor: move ResolvedPrincipal to shared types to fix circular dep

Extract ResolvedPrincipal from admin/grants.ts to types/principal.ts
so middleware/capabilities.ts imports from shared types rather than
depending upward on the admin handler layer.

* chore: remove dead re-export, align logger mocks across admin tests

- Remove unused ResolvedPrincipal re-export from grants.ts (canonical
  source is types/principal.ts)
- Align logger mocks in roles.spec.ts and groups.spec.ts to include
  all log levels (error, warn, info, debug) matching grants.spec.ts

* fix: cascade Config and AclEntry cleanup on role deletion

Add deleteConfig and deleteAclEntries to role deletion cascade,
matching the group deletion pattern. Previously only grants were
cleaned up, leaving orphaned config overrides and ACL entries.

* perf: single-query batch for getEffectiveCapabilities

Add getCapabilitiesForPrincipals (plural) to the data layer — a single
$or query across all principals instead of N+1 parallel queries. Wire
it into the grants handler so getEffectiveCapabilities hits the DB once
regardless of how many principals the user has.

* fix: defer SystemCapabilities access to factory call time

Move all SystemCapabilities usage (VALID_CAPABILITIES,
MANAGE_CAPABILITY_BY_TYPE, READ_CAPABILITY_BY_TYPE) inside the
createAdminGrantsHandlers factory. External test suites that mock
@librechat/data-schemas without providing SystemCapabilities crashed
at import time when grants.ts was loaded transitively.

* test: add data-layer and handler test coverage for review findings

- Add 6 mongodb-memory-server tests for getCapabilitiesForPrincipals:
  multi-principal batch, empty array, filtering, tenant scoping
- Add handler test: all principals filtered (only PUBLIC)
- Add handler test: granting an implied capability succeeds
- Add handler test: all cascade cleanup operations fail simultaneously
- Document platform-scope-only tenantId behavior in JSDoc

* fix: resolveUser fallback to user.id, early-return empty principals

- Match capabilities middleware pattern: _id?.toString() ?? user.id
  to handle JWT-deserialized users without Mongoose _id
- Move empty-array guard before principals.map() to skip unnecessary
  normalizePrincipalId calls
- Add comment explaining VALID_PRINCIPAL_TYPES module-scope asymmetry

* refactor: derive VALID_PRINCIPAL_TYPES from capability maps

Make MANAGE_CAPABILITY_BY_TYPE and READ_CAPABILITY_BY_TYPE
non-Partial Records over a shared GrantPrincipalType union, then
derive VALID_PRINCIPAL_TYPES from the map keys. This makes divergence
between the three data structures structurally impossible.

* feat: add GET /api/admin/grants list-all-grants endpoint

Add listAllGrants data-layer method and handler so the admin panel
can fetch all grants in a single request instead of fanning out
N+M calls per role and group. Response is filtered to only include
grants for principal types the caller has read access to.

* fix: update principalType to use GrantPrincipalType for consistency in grants handling

- Refactor principalType in createAdminGrantsHandlers to use GrantPrincipalType instead of PrincipalType for better type accuracy.
- Ensure type consistency across the grants handling logic in the API.

* fix: address admin grants review findings — tenantId propagation, capability validation, pagination, and test coverage

Propagate tenantId through all grant operations for multi-tenancy support.
Extract isValidCapability to accept full SystemCapability union (base, section,
assign) and reuse it in both Mongoose schema validation and handler input checks.
Replace listAllGrants with paginated listGrants + countGrants. Filter PUBLIC
principals from getCapabilitiesForPrincipals queries. Export getCachedPrincipals
from ALS store for fast-path principal resolution. Move DELETE capability param
to query string to avoid colon-in-URL issues. Remove dead code and add
comprehensive handler and data-layer test coverage.

* refactor: harden admin grants — FilterQuery types, auth-first ordering, DELETE path param, isValidCapability tests

Replace Record<string, unknown> with FilterQuery<ISystemGrant> across all
data-layer query filters. Refactor buildTenantFilter to a pure tenantCondition
function that returns a composable FilterQuery fragment, eliminating the $or
collision between tenant and principal queries. Move auth check before input
validation in getPrincipalGrantsHandler, assignGrantHandler, and
revokeGrantHandler to avoid leaking valid type names to unauthenticated callers.
Switch DELETE route from query param back to path param (/:capability) with
encodeURIComponent per project conventions. Add compound index for listGrants
sort. Type VALID_PRINCIPAL_TYPES as Set<GrantPrincipalType>. Remove unused
GetCachedPrincipalsFn type export. Add dedicated isValidCapability unit tests
and revokeGrant idempotency test.

* refactor: batch capability checks in listGrantsHandler via getHeldCapabilities

Replace 3 parallel hasCapabilityForPrincipals DB calls with a single
getHeldCapabilities query that returns the subset of capabilities any
principal holds. Also: defensive limit(0) clamp, parallelized assignGrant
auth checks, principalId type-vs-required error split, tenantCondition
hoisted to factory top, JSDoc on cascade deps, DELETE route encoding note.

* fix: normalize principalId and filter undefined in getHeldCapabilities

Add normalizePrincipalId + null guard to getHeldCapabilities, matching
the contract of getCapabilitiesForPrincipals. Simplify allCaps build
with flatMap, add no-tenantId cross-check and undefined-principalId
test cases.

* refactor: use concrete types in GrantRequestBody, rename encoding test

Replace unknown fields with explicit string types in GrantRequestBody,
matching the established pattern in roles/groups/config handlers. Rename
misleading 'encoded' test to 'with colons' since Express auto-decodes
req.params.

* fix: support hierarchical parent capabilities in possession checks

hasCapabilityForPrincipals and getHeldCapabilities now resolve parent
base capabilities for section/assignment grants. An admin holding
manage:configs can now grant manage:configs:<section> and transitively
read:configs:<section>. Fixes anti-escalation 403 blocking config
capability delegation.

* perf: use getHeldCapabilities in assignGrant to halve DB round-trips

assignGrantHandler was making two parallel hasCapabilityForPrincipals
calls to check manage + capability possession. getHeldCapabilities was
introduced in this PR specifically for this pattern. Replace with a
single batched call. Update corresponding spec assertions.

* fix: validate role existence before granting capabilities

Grants for non-existent role names were silently persisted, creating
orphaned grants that could surprise-activate if a role with that name
was later created. Add optional checkRoleExists dep to assignGrant and
wire it to getRoleByName in the route file.

* refactor: tighten principalType typing and use grantCapability in tests

Narrow getCapabilitiesForPrincipals parameter from string to
PrincipalType, removing the redundant cast. Replace direct
SystemGrant.create() calls in getCapabilitiesForPrincipals tests with
methods.grantCapability() to honor the schema's normalization invariant.
Add getHeldCapabilities extended capability tests.

* test: rename misleading cascade cleanup test name

The test only injects failure into deleteGrantsForPrincipal, not all
cascade operations. Rename from 'cascade cleanup fails' to 'grant
cleanup fails' to match the actual scope.

* fix: reorder role check after permission guard, add tenantId to index

Move checkRoleExists after the getHeldCapabilities permission check so
that a sub-MANAGE_ROLES admin cannot probe role name existence via
400 vs 403 response codes.

Add tenantId to the { principalType, capability } index so listGrants
queries in multi-tenant deployments can use a covering index instead
of post-scanning for tenant condition.

Add missing test for checkRoleExists throwing.

* fix: scope deleteGrantsForPrincipal to tenant on role deletion

deleteGrantsForPrincipal previously filtered only on principalType +
principalId, deleting grants across all tenants. Since the role schema
supports multi-tenancy (compound unique index on name + tenantId), two
tenants can share a role name like 'editor'. Deleting that role in one
tenant would wipe grants for identically-named roles in other tenants.

Add optional tenantId parameter to deleteGrantsForPrincipal. When
provided, scopes the delete to that tenant plus platform-level grants.
Propagate req.user.tenantId through the role deletion cascade.

* fix: scope grant cleanup to tenant on group deletion

Same cross-tenant gap as the role deletion path: deleteGroupHandler
called deleteGrantsForPrincipal without tenantId, so deleting a group
would wipe its grants across all tenants. Extract req.user.tenantId
and pass it through.

* test: add HTTP integration test for admin grants routes

Supertest-based test with real MongoMemoryServer exercising the full
Express wiring: route registration, injected auth middleware, handler
DI deps, and real DB round-trips.

Covers GET /, GET /effective, POST / + DELETE / lifecycle, role
existence validation, and 401 for unauthenticated callers.

Also documents the expandImplications scope: the /effective endpoint
returns base-level capabilities only; section-level resolution is
handled at authorization check time by getParentCapabilities.

* fix: use exact tenant match in deleteGrantsForPrincipal, normalize principalId, harden API

CRITICAL: deleteGrantsForPrincipal was using tenantCondition (a
read-query helper) for deleteMany, which includes the
{ tenantId: { $exists: false } } arm. This silently destroyed
platform-level grants when a tenant-scoped role/group deletion
occurred. Replace with exact { tenantId } match for deletes so
platform-level grants survive tenant-scoped cascade cleanup.

Refactor deleteGrantsForPrincipal signature from fragile positional
overload (sessionOrTenantId union + maybeSession) to a clean options
object: { tenantId?, session? }. Update all callers and test assertions.

Add normalizePrincipalId to hasCapabilityForPrincipals to match the
pattern already used by getHeldCapabilities — prevents string/ObjectId
type mismatch on USER/GROUP principal queries.

Also: export GrantPrincipalType from barrel, add upper-bound cap to
listGrants, document GROUP/USER existence check trade-off, add
integration tests for tenant-isolation property of deleteGrantsForPrincipal.

* fix: forward tenantId to getUserPrincipals in resolvePrincipals

resolvePrincipals had tenantId available from the caller but only
forwarded it to getCachedPrincipals (cache lookup). The DB fallback
via getUserPrincipals omitted it. While the Group schema's
applyTenantIsolation Mongoose plugin handles scoping via
AsyncLocalStorage in HTTP request context, explicitly passing tenantId
makes the contract visible and prevents silent cross-tenant group
resolution if called outside request context.

* fix: remove unused import and add assertion to 401 integration test

Remove unused SystemCapabilities import flagged by ESLint. Add explicit
body assertion to the 401 test so it has a jest expect() call.

* chore: hoist grant limit constants to scope, remove dead isolateModules

Move GRANTS_DEFAULT_LIMIT / GRANTS_MAX_LIMIT from inside listGrants
function body to createSystemGrantMethods scope so they are evaluated
once at module load. Remove dead jest.isolateModules + jest.doMock
block in integration test — the ~/models mock was never exercised
since handlers are built with explicit DI deps.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-03-30 16:49:23 -04:00
Danny Avila
fda1bfc3cc
🔬 ci: Add TypeScript Type Checks to Backend Workflow and Fix All Type Errors (#12451)
* fix(data-schemas): resolve TypeScript strict type check errors in source files

- Constrain ConfigSection to string keys via `string & keyof TCustomConfig`
- Replace broken `z` import from data-provider with TCustomConfig derivation
- Add `_id: Types.ObjectId` to IUser matching other Document interfaces
- Add `federatedTokens` and `openidTokens` optional fields to IUser
- Type mongoose model accessors as `Model<IRole>` and `Model<IUser>`
- Widen `getPremiumRate` param to accept `number | null`
- Widen `bulkWriteAclEntries` ops to untyped `AnyBulkWriteOperation[]`
- Fix `getUserPrincipals` return type to use `PrincipalType` enum
- Add non-null assertions for `connection.db` in migration files
- Import DailyRotateFile constructor directly instead of relying on
  broken module augmentation across mismatched node_modules trees
- Add winston-daily-rotate-file as devDependency for type resolution

* fix(data-schemas): resolve TypeScript type errors in test files

- Replace arbitrary test keys with valid TCustomConfig properties in config.spec
- Use non-null assertions for permission objects in role.methods.spec
- Replace `.SHARED_GLOBAL` access with `.not.toHaveProperty()` for legacy field
- Add non-null assertions for balance, writeRate, readRate in spendTokens.spec
- Update mock user _id to use ObjectId in user.test
- Remove unused Schema import in tenantIndexes.spec

* fix(api): resolve TypeScript strict type check errors across source and test files

- Widen getUserPrincipals dep type in capabilities middleware
- Fix federatedTokens type in createSafeUser return
- Use proper mock req type for read-only properties in preAuthTenant.spec
- Replace `as IUser` casts with ObjectId-typed mocks in openid/oidc specs
- Use TokenExchangeMethodEnum values instead of string literals in MCP specs
- Fix SessionStore type compatibility in sessionCache specs
- Replace `catch (error: any)` with `(error as Error)` in redis specs
- Remove invalid properties from test data in initialize and MCP specs
- Add String.prototype.isWellFormed declaration for sanitizeTitle spec

* fix(client): resolve TypeScript type errors in shared client components

- Add default values for destructured bindings in OGDialogTemplate
- Replace broken ExtendedFile import with inline type in FileIcon

* ci: add TypeScript type-check job to backend review workflow

Add a `typecheck` job that runs `tsc --noEmit` on all four TypeScript
workspaces (data-provider, data-schemas, @librechat/api, @librechat/client)
after the build step. Catches type errors that rollup builds may miss.

* fix(data-schemas): add local type declaration for DailyRotateFile transport

The `winston-daily-rotate-file` package ships a module augmentation for
`winston/lib/winston/transports`, but it fails when winston and
winston-daily-rotate-file resolve from different node_modules trees
(which happens in this monorepo due to npm hoisting).

Add a local `.d.ts` declaration that augments the same module path from
within data-schemas' compilation unit, so `tsc --noEmit` passes while
keeping the original runtime pattern (`new winston.transports.DailyRotateFile`).

* fix: address code review findings from PR #12451

- Restore typed `AnyBulkWriteOperation<AclEntry>[]` on bulkWriteAclEntries,
  cast to untyped only at the tenantSafeBulkWrite call site (Finding 1)
- Type `findUser` model accessor consistently with `findUsers` (Finding 2)
- Replace inline `import('mongoose').ClientSession` with top-level import type
- Use `toHaveLength` for spy assertions in playwright-expect spec file
- Replace numbered Record casts with `.not.toHaveProperty()` in
  role.methods.spec for SHARED_GLOBAL assertions
- Use per-test ObjectIds instead of shared testUserId in openid.spec
- Replace inline `import()` type annotations with top-level SessionData
  import in sessionCache spec
- Remove extraneous blank line in user.ts searchUsers

* refactor: address remaining review findings (4–7)

- Extract OIDCTokens interface in user.ts; deduplicate across IUser fields
  and oidc.ts FederatedTokens (Finding 4)
- Move String.isWellFormed declaration from spec file to project-level
  src/types/es2024-string.d.ts (Finding 5)
- Replace verbose `= undefined` defaults in OGDialogTemplate with null
  coalescing pattern (Finding 6)
- Replace `Record<string, unknown>` TestConfig with named interface
  containing explicit test fields (Finding 7)
2026-03-28 21:06:39 -04:00