Commit graph

91 commits

Author SHA1 Message Date
lia-by-librechat[bot]
0e3c9088f6
🫰 feat: Configure Tool Approval Modes Per Agent (#16707)
* 🫰 feat: Configure Tool Approval Modes Per Agent

* fix: Fence Remembered Tool Approval Authority

* fix: Unify Tool Approval Execution Invariants

* fix: Revalidate Learned Approval at MCP Invocation

* fix: Preserve One-Time MCP Review at Invocation

* fix: Execute Reviewed Calls on Templated MCP Connections

* fix: Scope Approval Witnesses to Executing Agents

* fix: Retain Approval Bindings for Hook-Rewritten Inputs

* fix: Pin Review Authority and Retire Settled Tool Calls

* fix: Fingerprint Effective MCP Request Headers

* fix: Preserve Routing Around Renewable MCP Token Fragments

* fix: Bind Tool Consent to OAuth Credential Generations

* test: Keep Mobile Approval Save Outside Drawer Escape Race

* fix: Fence OAuth Consent at MCP Transport Retries

* fix: Let Shared Agent Viewers Reset Personal Consent

* fix: Preserve Renewed Consent Across Reset Races

* fix: Resolve Verified Aliases for Personal Tool Reset

* fix: Fence Renewed OAuth Consent and Credential Review

* fix: Require Foreground Execution for Review-Only MCP Calls

* fix: Pin Invocation Ownership and Stable MCP Consent

* fix: Fence Superseded Tool Consent Completion Writes

* fix: Scope Consent Epochs to Effective MCP Authentication

* fix: Scope Approval Admission to Reachable Agent Tools

* fix: Preserve Approval Epochs for Runtime-Detected OAuth

* fix: Address Tool Approval Review Findings

* fix: Preserve Captured Consent When Tool Approval Is Inherited

* fix: Preserve Approval Admission Through Lazy Metadata

* fix: Fence Programmatic Reviews and Unresolved Lazy Approval Keys

* fix: Account for Normalized and Skill-Contributed Lazy Tools

* fix: Scope Lazy Approval to Selected Tool Spellings and Servers

* fix: Honor Disabled MCP Capability During Lazy Approval Admission

* test: Preserve Approval Merge Regressions After Rebase

* fix: Preserve One-Time Review During Non-OAuth Grant Outages

* fix: Retain One-Time Consent Across Resume Recovery

* fix: Preserve Reviewed Consent Through Transport Outages

* fix: Preserve Case-Variant Lazy Approval Hook Admission

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-04 13:14:48 -04:00
lia-by-librechat[bot]
73d704656e
📔 feat: Preserve Owner Text During PII Redaction (#16311)
* 📔 feat: Preserve Owner Text During PII Redaction

* 🧵 fix: Keep Ordinary Chat Views Outside Private Text Provider

* 🧷 fix: Preserve Owner Text Across Copy and Edit Paths

* 🔏 fix: Protect PII on Denial and Abort Paths

* fix: preserve protected text through denial, stop, and owner reload

* test: cover preliminary private metadata in disconnect fixture

* test: carry preliminary metadata in resume controller fixture

* test: preserve Stop prerequisite identity in tenant fixture

* fix: keep protected owner text out of search and exports

* fix: preserve trusted PII placeholders through provider inspection

* fix: coordinate protected turns across Stop and native sharing

* fix: Preserve Private Text Admission and Query Lifecycles

* fix: Track Private Query Effect Dependencies

* fix: Fence Protected Admission and Screenshot Targets

* fix: Share Protected Admission and Preserve Native Token Trust

* test: Wait for Private Message Concurrency Indexes

* fix: Preserve Protected Admission Across Recovery and Titles

* fix: Coordinate Protected Root Admission and Early Stop

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 11:43:10 -04:00
Marco Beretta
e1e1d1b19d 🎒 feat: Carry Project Context Across Conversations (#15684)
* 🎒 feat: Carry Project Context Across Conversations

* fix: fail a project attach whose upload hold cannot be released and keep file deletes independent of project cleanup

- Attaching a project file now fails, with nothing attached, when the file's
  temporary upload hold cannot be released, instead of attaching a file that
  can still expire.
- A failed project cleanup after a file delete is logged rather than failing a
  delete that already happened, and a user-wide delete clears that user's
  project references without reading every file first.
- Both resize handles share the orientation-aware grip, so a primary handle in
  a vertical group renders a rotated grip like the alternate one.

* fix: treat an empty tenant id as no tenant in the project file picker

* fix: attach nothing when a project file is deleted while the attach is in flight

* fix: localize project file association failures instead of showing server text

* test: expect a localized reason for a rejected project file association

* fix: scope project file listing to the caller's tenant and name the file limit when attaching an existing file

- Listing a project's attached files now returns not found when the project
  belongs to another tenant, matching the available-file listing and the
  chat context resolver.
- Choosing an existing file for a full project now names the project file
  limit instead of the generic attach error, like the upload path.

* fix: treat an empty tenant id as no tenant in every project tenant check

A caller whose tenant id is an empty string was refused its own tenantless
project when listing attached or available files, because the handlers
compared '' with null. The project handlers and the chat context resolver now
share one comparison that treats an empty tenant id as no tenant, the rule the
project file queries already follow.

* test: wait for the workspace layout swap and the mobile drawer before acting in the long-metadata spec

The desktop title was measured while the phone layout could still be the one
rendered, and the sidebar row menu was opened before the mobile drawer
committed its open state, which closes a menu opened ahead of it. The spec
now waits for the workspace's phone-only sidebar toggle to leave and for the
drawer to drop inert, the same drawer signal the agents helpers use.

* fix: keep the agent context fingerprint of a turn outside any project unchanged

Every turn now passes a project context key into the agent context
fingerprint, and a turn outside any project passed the no-project sentinel,
which changed the digest of every existing no-project checkpoint. An agent
event conversation written before this change then cold-started and reloaded
its history on its next event. The fingerprint now hashes the no-project key
as absent, so those checkpoints keep matching.

* test: load the long-metadata spec at phone width before driving the mobile drawer

The spec crossed back from desktop to phone width without a reload, which
leaves the drawer on the desktop toggle path whose delayed focus handoff to
the close button dismissed the row menu roughly 200ms after it opened. The
spec now reloads at phone width, as a phone arrives, and waits for the
drawer's own focus handoff before opening the menu.

(cherry picked from commit 4062bc3c00)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/15684
2026-10-01 03:46:19 +00:00
Danny Avila
d33c72e6a8
🛂 fix: Rate Limit Authenticated 2FA Management (#16565)
* fix: Rate Limit Authenticated 2FA Management

* fix: Explain 2FA Rate Limits With Localized Retry Guidance

* fix: Preserve Minimum 2FA Setup Budget
2026-09-30 09:10:10 -04:00
lia-by-librechat[bot]
ea7ac99209
🛶 fix: Keep Trigger Bans Off Shared Loopback IPs (#16535)
Co-authored-by: Lia <lia@librechat.ai>
2026-09-29 20:50:23 -04:00
Dustin Healy
418d62d52d
🔐 feat: Support Cognito Access Tokens for Agent Management (#16224)
* feat: support Cognito M2M for Agent management

* fix: reject compound management scopes

* fix: require scopes for audience-less management tokens
2026-09-22 20:37:40 -04:00
Danny Avila
696bcb6c1e
🏦 fix: Reserve Credits for In-Flight Requests (#15879)
* 💳 fix: Reserve Balance for In-Flight Requests

Admit each balance-checked request against the credits that other
in-flight requests have not already reserved, and hold its prompt cost
on the balance record until the turn settles. Apply a due auto-refill in
the same fenced write so one refill window credits exactly once.

- data-schemas: reserveBalance / releaseBalanceReservation replace
  createAutoRefillTransaction; the write is a compare-and-swap on
  tokenCredits, reservations and lastRefill (no pipeline updates)
- balance.reservations is select:false so existing reads never see it
- checkBalance returns a reservation handle; BaseClient and the
  assistants controllers release it once the turn settles
- balance.reservationTtlMs bounds how long an unreleased reservation
  counts (default 30 minutes)

* 🧮 refactor: Admit Reservations Without Contention and Record Refills Durably

- admission is a conditional $push/$inc against a reservedCredits total,
  so concurrent admissions of a funded balance commit on their first write
- expired reservations are pruned per element with their credits
- auto-refill is fenced on every setting its decision reads and leaves a
  pendingRefill ledger marker in the same write; any later read records the
  transaction under the fixed id and clears the marker
- a missing record is created with $setOnInsert inside reserveBalance
- reservation lifecycle moves to packages/api (createBalanceReservations,
  withBalanceReservations); BaseClient releases once usage is recorded

* 🪪 fix: Resolve Each User to One Balance Record

- balance records a creator inserts are keyed by the user id, so creators
  racing on a missing record converge on one document; every balance read
  and write resolves a user to their oldest record, so existing duplicates
  stay inert
- upsertBalanceFields takes insert-only fields; the balance-config
  middleware, schedules and lazy initialization pass the starting credit
  through them so it never overwrites a record another writer created
- the auto-refill write is also fenced on reservedCredits
- the scheduled-chat balance pre-skip counts reserved credits as unavailable

* ⏳ fix: Count Only Unexpired Reservations in Scheduled Balance Checks

findBalanceByUser with includeReservedCredits now totals the reservations
that have not expired, so a reservation left by a crashed request stops
pre-skipping scheduled chats at its expiry instead of when a later
reservation write prunes it.

* 🔁 fix: Renew Balance Reservations While Their Request Runs

A reservation is renewed every half TTL until it is released, so only a
reservation whose process stopped expires. Pruning removes a reservation
only while it is still expired at the write, so one renewed after the
pruning read survives.

* 🛟 fix: Keep Reservations Alive Through Renewal Failures and Lost Acks

- renewal reschedules itself: every half TTL, capped to the timer range,
  and a failed renewal retries within a tenth of that window
- an admission write that errors removes its possibly committed
  reservation before rethrowing

* 🛑 fix: Hold a Stopped Turn's Reservation Until Its Charge Lands

- an aborted turn lets its reservations lapse after a minute (or the TTL)
  instead of releasing them, so the Stop request's charge lands before the
  credits are admitted again
- balance.reservationTtlMs has a 10 second minimum, enforced in the schema
  and at runtime, so renewal stays bounded

* ↩️ revert: Release a Stopped Turn's Reservation When the Turn Settles

An aborted turn releases its reservations when it settles, the same as
every other exit, instead of holding them for a fixed lapse window that
refused funded requests after every Stop. The minimum reservation TTL
stays.

* ⏸️ fix: Hold Assistants Reservations Through Background Run Retries

An in-progress Assistants run continues in the background after the
handler responds; the turn's reservations now stay held until that run
settles (createBalanceReservations().holdUntil).

* 🔁 fix: Apply an Auto-Refill Once per Admission

A refill interval that is due again the moment it is applied, such as a zero
interval, let one admission refill repeatedly until the retry bound. Each
admission now applies at most one refill, as the previous check did.

* 🧹 fix: Prune Expired Reservations in One Write

An admission that found expired reservations issued one write per hold, so a
backlog left by a stopped process fanned out into as many concurrent writes.
The prune is now a single write fenced on every hold it removes; a hold renewed
after the read makes the write miss and the admission re-read.

* ⏱️ fix: Renew Reservations From Their Stored Expiry

The next renewal was scheduled half a TTL after the previous write settled,
and the first one half a TTL after the admission returned, so a slow write
could let a live hold expire before its renewal. Renewals are now scheduled
from the expiry the last write stored.

Expired holds are pruned as one bulk write of per-hold fenced updates, so a
hold that changed after the read is skipped while the rest are removed, and
the admission no longer spends a retry on the whole prune.
2026-09-13 12:53:19 -04:00
Danny Avila
9cf33cc8be
🧿 fix: Bind Login Flows to the Browser That Started Them (#15880)
* 🧿 fix: Bind Login Flows to the Browser That Started Them

Social OAuth logins (Google, Facebook, GitHub, Discord, Apple) now store a random state in an HttpOnly cookie scoped to the callback path and only exchange the returned code when the callback carries that state. Apple's form_post callback uses a SameSite=None cookie, and passport-apple's own fixed state no longer bypasses the store.

Local login, temp-token 2FA verification and admin local login reject requests a browser sent from another origin (Sec-Fetch-Site, falling back to Origin vs Host), trusting DOMAIN_CLIENT and DOMAIN_SERVER. Requests without browser fetch metadata, such as server-side callers, pass unchanged.

The sessionless OAuth callbacks drop failureMessage, which wrote to req.session and turned any rejected callback into a 500.

* 🧿 fix: Configure Login State Cookies and Localize Origin Rejections

- OAuth state cookies take secureCookie and maxAgeMs from the caller; registration.oauthStateTtlMs in librechat.yaml sets the lifetime (default 10 minutes).
- Secure deployments use a __Host- cookie at Path=/ so a sibling subdomain cannot plant a state.
- The cookie keeps up to three pending states, so logins started in separate tabs all complete, and an unmatched callback no longer discards them.
- passport-apple's preset state is dropped by deferStateToStore in packages/api.
- The Origin fallback compares scheme and host against the request itself.
- The same-origin rejection carries ErrorTypes.AUTH_CROSS_ORIGIN, which the login and 2FA screens show as a localized message.

* 🧿 refactor: Sign Login State Against a Per-Browser Binding

- The OAuth state store keeps a random per-browser binding in a __Host- cookie (reused across flows) and signs each state as issuedAt.nonce.HMAC(JWT_SECRET, provider, binding, issuedAt, nonce). Nothing is stored per flow, so any number of tabs complete, each state expires on its own clock, and concurrent starts no longer overwrite each other.
- The experimental server entrypoint passes its app config to configureSocialLogins.
- The same-origin guard also trusts an explicitly configured ADMIN_PANEL_URL.

* 🧿 fix: Keep Separate Bindings for Racing First Logins

A browser without a login-state binding now gets one under a cookie name with a random suffix, so two tabs that start their first login at once each keep their own binding. Later starts reuse an existing binding, and a callback is accepted when its state was signed for any binding the browser presents.

* 🧿 fix: Sanitize Request-Derived Values in the Cross-Site Rejection Log

The same-origin guard logged the request Origin and path verbatim. A non-browser client fully controls the Origin header, and the console transport uses a printf format that does not escape control characters, so a crafted value could forge log lines. Strip ASCII control characters (CR, LF, NUL and the rest) before logging, matching the pattern apiNotFound already uses.
2026-09-13 11:47:29 -04:00
Joseph Licata
f25242df98
🦘 feat: Share Run-Scoped Files With Subagents (#15848)
* feat: add run-scoped file sharing for subagents

Delegate current-message files through authorized execution catalogs and publish selected immutable code-output versions through the existing file pipeline.

Requires the companion agents SDK context adapter before the dependency can be released and pinned.

* fix: preserve child tool artifacts and verify file-sharing lifecycle

Keep citations and other non-sandbox artifacts on their existing delivery path while retaining private code outputs and inspected images until publication.

Cover native and text delivery, nested and sibling authorization, concurrent runs, cancellation cleanup, and checkpoint recovery through the browser harness.

* test: align file-sharing fixtures and integration cleanup

* test: repair LibreOffice conversion fixtures

* test: verify deployment skill bytes across checkout formats

* fix: align subagent file budgets and refresh boundaries

* chore: update agents SDK to 3.8.6

* refactor: move code output processing into API workspace

* fix: enforce shared file budgets for parents and teams

* fix: declare the shared file encoder interface
2026-09-12 20:14:22 -04:00
Danny Avila
e7dea565d1
🧵 fix: Preserve Per-Message Inspection Budgets Through Provider Preparation (#15860) 2026-09-12 09:24:48 -04:00
Danny Avila
8f13845da5
📿 fix: Inspect Stored File Locators Per Message (#15841)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: inspect stored file locators per message

* fix: declare traversal diagnostics for isolated declaration builds

* refactor: inject locator traversal reporting from application callers

* fix: propagate traversal diagnostics across protected entry points

* fix: complete middleware reporters and hydration message counts
2026-09-11 16:47:48 -04:00
Danny Avila
f1bd85498e
🪫 fix: Recharge Direct OpenID MCP Bearer Sessions (#15704)
* fix: recover direct OpenID MCP bearer sessions

* fix: harden direct bearer recovery lifecycle

* fix: close direct bearer lifecycle races

* test: align direct bearer checks with current dev

* fix: unify direct bearer recovery budget

* fix: infer direct bearer recovery from placeholder

* fix: preserve bearer precedence and recovery budget

* fix: propagate bearer recovery across joiners

* fix: preserve direct bearer fallback without session provider

* fix: close direct bearer recovery race windows

* chore: sort bearer recovery test imports

* fix: align bearer recovery with OAuth cleanup lifecycle

* fix: fence request-scoped MCP creation during config mutations

* fix: retain suspended MCP catalog authentication outcomes

* fix: cancel MCP bearer recovery and reuse refreshed credentials

* fix: close MCP bearer recovery lifecycle boundaries

* fix: preserve ordered catalog authentication rejection

* fix: isolate bearer recovery across request and transport boundaries

* fix: preserve shared recovery ownership and credential precedence

* fix: wait for recovery borrowers and validated OpenID publication

* fix: observe the validated browser publication flight

* fix: cancel abandoned OpenID publication followers

* fix: fence refresh grant admission after lease acquisition
2026-09-07 21:05:26 -04:00
Danny Avila
4b3a6eaf0e
🪧 feat: Surface Conversation Code Approval Modes (#15703)
* feat: Add conversation code approval modes

* refactor: Reuse shared code approval types

* fix: Scope approval selector to stateful agents

* fix: harden code approval state propagation

* fix: fail closed during approval discovery

* test: cover approval mode acceptance

* test: use visible approval mode label

* test: drive approval menu by keyboard

* test: expose accept edits in BYOM acceptance

* fix: read approval mode atomically on send

* fix: align code modes with endpoint policy

* fix: initialize code limiters at startup
2026-09-07 12:24:29 -04:00
Danny Avila
dd5829ed77
🏰 fix: Enforce Platform Grants for Code Workers (#15700)
* fix: Require Platform Grants for Code Workers

* fix: Separate Platform Scope From Tenant Identity
2026-09-07 08:13:47 -04:00
Danny Avila
911b6324d8
🧲 fix: Coalesce and Bound Code Environment Status Polls (#15701)
* fix: Bound Code Environment Status Polling

* refactor: Move Code Limiters Into API Package

* fix: enforce status polling boundaries
2026-09-07 08:13:25 -04:00
Danny Avila
09da878a47
🚰 fix: Throttle Shared Link Reads and Aggregate Audit Findings (#15605)
* 🪵 fix: Aggregate Audit Findings and Rate Limit Shared Link Reads

Audit-mode content filter rules wrote one `logger.info` per matching
fragment, and inspection deliberately continues past every finding. A
shared conversation full of distinct matching strings therefore produced
one log write per fragment, regenerated on every retrieval of
`GET /api/share/:shareId` — a public, unauthenticated route with no
limiter. The 4,096-entry inspection dedupe set is a memory bound, not an
event budget: once full, further values are inspected and logged again.

Audit metadata carries no inspected text — only rule, source, field and
provenance — so thousands of those writes were byte-identical lines.
Findings are now counted inside an aggregation scope and reported once
per distinct key with an `occurrences` count when the scope ends:

- `assertModelBoundContent` opens a scope when audit rules are configured,
  bounding one inspection pass (a single message's fragments).
- `assertConversationImportContentAllowed` opens the outer scope for
  import and shared-link snapshots, so per-message inspections nest into
  one report instead of one per message.
- `createShareContentPreflight` spans the shared-file metadata pass too.

Nested scopes reuse the outermost aggregation, and outside any scope the
previous immediate write is kept.

`GET /api/share/:shareId` re-inspects the whole snapshot on every request,
so it now carries IP and user limiters like the neighbouring fork route
(`SHARE_IP_MAX`/`SHARE_IP_WINDOW`, `SHARE_USER_MAX`/`SHARE_USER_WINDOW`,
`SHARE_VIOLATION_SCORE`); the user limiter skips anonymous viewers.

* 🔒 fix: Address Codex Findings on Share Audit Aggregation

- The aggregation key joined rule fields with a space, but the filter
  schema allows spaces in custom pattern ids and labels, so
  `{id: 'a', label: 'b c'}` and `{id: 'a b', label: 'c'}` collided on the
  same source/field/provenance and merged two rules' counts under one
  identity. Serialize the tuple instead.
- `SHARE_VIOLATION_SCORE` left unset reached `logViolation` as
  `undefined`, which selects that function's default score of 1 rather
  than the documented zero. With default ban settings a viewer refreshing
  a rate-limited shared link would be banned after 20 rejections. Default
  the score to 0 in code, not only in `.env.example`.
- `removePorts` returns a full IPv6 address, so an attacker holding a
  prefix could rotate the host portion for a fresh bucket on every
  request — and for anonymous viewers the IP limiter is the only bound.
  Derive the key through `ipKeyGenerator` so IPv6 clients group by /56.
2026-09-04 12:38:34 -04:00
Dustin Healy
aec2121923
🔐 feat: Add M2M Authentication for Agent Management (#15555)
* feat: add M2M authentication for Agent Management

* fix: Harden Agent Management authentication

* fix: Preserve OIDC authentication safeguards

* fix: Normalize management principal IDs

* fix: Bound OIDC signing key lookups

* fix: Allow non-browser Agent Management clients
2026-09-03 20:07:00 -04:00
Danny Avila
057039570b
🔭 feat: Audit PII Filter Findings (#15508)
* feat: add audit-only content filter action

* refactor: align filter action typing and tests

* fix: preserve audit-only filter behavior

* fix: enforce audit mode across content paths

* fix: honor audit mode for canonical files
2026-09-02 10:36:06 -04:00
Danny Avila
6195273429
🧲 fix: Anchor Memory Partitions to Accessible Agents (#15475)
* fix: validate agent memory partitions

* fix: authorize agent memory mutations

* test: cover agent manager memory access

* fix: centralize agent memory authorization

* fix: harden agent memory authorization

* fix: support synchronous role providers
2026-09-01 18:37:52 -04:00
Danny Avila
74535662fe
🎖️ fix: Apply Fallback Role on Group Overage (#15472) 2026-09-01 16:44:24 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
70f735336d
🛎️ fix: Enroll Remote Agent Runs in the Generation Lifecycle (#15349)
* fix: enroll remote agent runs in generation lifecycle

* fix: close remote lifecycle ownership gaps

* fix: close remote conversation drain races

* fix: fence remote runs during conversation deletion

* fix: reconcile remote deletion and settlement races

* fix: complete owner deletion recovery

* fix: preserve remote cleanup receipts

* fix: consume deletion receipts before cleanup

* fix: expose idempotent deletion option

* chore: sort remote lifecycle imports
2026-08-30 07:14:13 -04:00
Danny Avila
7a0061507e
🪪 fix: Admit Confirmed Generation Retries Before Message Limits (#15341)
* fix: admit idempotent retries before message limits

* fix: bound generation retry admission

* fix: keep retry probe store-compatible

* fix: exclude normalized resume routes

* fix: preserve trusted retry exemptions

* fix: bound retry claim admission

* style: satisfy generation retry static checks
2026-08-29 13:41:26 -04:00
Ravi Kumar L
a70bcf66a4
🪢 feat: show Langfuse session link in shared chats (#15273)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: show Langfuse sessions in shared chats

* fix: wait for optional auth before loading shares

* fix: require Langfuse access for shared session links

* refactor: move shared Langfuse policy into API package

* fix: wrap shared chat actions on mobile

* fix: coordinate shared Langfuse link loading

* perf: batch shared Langfuse capability checks
2026-08-27 14:29:57 -04:00
Danny Avila
0c959deb99
🧢 fix: Cap Batched HITL Answers Before PII and Moderation (#15257) 2026-08-26 10:10:19 -04:00
Danny Avila
8f9fae0a6e
🛂 fix: Preserve Legacy Assistant Attribution (#15118) 2026-08-22 16:21:45 -04:00
Danny Avila
3ebef4c84e
📨 feat: Add Authenticated Agent Event Ingress (#15110)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources
2026-08-22 01:09:04 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Marco Beretta
29f6ec6eae
🙈 feat: Config Option to Hide Response Feedback Buttons (#15085)
* feat: add interface option to hide response feedback buttons

Adds `interface.feedback` to librechat.yaml. When set to false, the
thumbs up/thumbs down buttons are removed from the message action row
and the feedback endpoint rejects writes with 403, so deployments that
do not consume the data can stop collecting it. Defaults to true.

* refactor: hoist the feedback gate out of the message row and into typed middleware

Reading startup config inside HoverButtons put a query observer and two Recoil
subscriptions on every message row, and rows never unmount, so the cost grew
with the conversation. Resolve the flag once per chat in useChatHelpers and
carry it on TMessageChatContext; useMessageActions withholds handleFeedback
when it is off, which the action row already treats as "no feedback controls".
The flag now stays false until the config resolves, so a disabled deployment
never flashes controls whose writes are rejected.

Move the server-side policy into requireFeedbackEnabled under packages/api so
the route keeps no policy of its own.

* test: stub the feedback gate in specs that replace the api package

The messages router now imports requireFeedbackEnabled, and express rejects an
undefined handler at require time, so every spec that mocks @librechat/api
wholesale has to carry the export.
2026-08-21 11:16:44 -04:00
Danny Avila
d0f9d5625e
🧵 fix: Close Child-Thread Read and Search-Cleanup Gaps (#15055)
* fix: close child thread read and cleanup gaps

* fix: preserve child search cleanup invariants

* test: complete mocked update result

* perf: parallelize scoped message reads

* fix: close child thread compatibility gaps

* test: expect preserved cleanup failure

* fix: reconcile legacy Meili cleanup markers
2026-08-21 00:44:37 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Dustin Healy
a33b128c47
🪪 fix: Preserve Stored Access Token Expiry Over ID Token Exp (#14982)
* 🪪 fix: Preserve Stored Access Token Expiry Over ID Token Exp

extractOpenIDTokenInfo let the ID token exp claim overwrite the token set's stored expires_at. The ID token is minted at login and never refreshed, so once a session outlives the ID token TTL, isOpenIDTokenValid reports the access token as expired even when expires_at is hours in the future, and OpenID placeholder substitution silently stops: MCP headers configured with {{LIBRECHAT_OPENID_ACCESS_TOKEN}} ship the literal placeholder string as the bearer credential and the receiving server rejects every connection with an unparseable JWT until the user fully logs out and back in.

The ID token exp now only fills a missing expiresAt instead of overriding a stored one. Identity claim enrichment from the ID token is unchanged, and the exp fallback for token sets without expires_at is preserved.

* 🪪 fix: Validate ID Token Expiry Before ID Token Placeholder Substitution

The precedence fix made isOpenIDTokenValid track only the access token expiry, so an MCP header using {{LIBRECHAT_OPENID_ID_TOKEN}} could substitute an ID token that had already expired. The ID token exp is now preserved separately as idTokenExpiresAt and checked at the ID token substitution site, so an expired ID token substitutes empty rather than a stale credential while access token substitution is unaffected.

* 🪪 fix: Address OpenID Expiry Review Round

Fix expires_at at the source in the OpenID JWT strategy. The stored value described the
incoming bearer's exp even when access_token came from the session or a cookie, so it could
describe a different credential entirely. A new decodeJwtExpiry helper reads the exp of the
token actually stored, and payload.exp is kept only when the raw bearer is the resolved
access token. Opaque session or cookie tokens now store no expiry rather than a wrong one.

Apply a 30 second clock skew buffer in isOpenIDTokenValid and isIdTokenCurrent via a new
exported OPENID_EXPIRY_BUFFER_SECONDS, mirroring OPENID_REUSE_EXPIRY_BUFFER_SECONDS in
AuthController. Tokens that would expire in transit are treated as already expired.

Make isIdTokenCurrent fail closed when idTokenExpiresAt is absent. exp is REQUIRED in an ID
token, so a missing value means the token is malformed or the claims parse threw. The check
uses == null so an exp of 0 counts as present and therefore expired.

Read the ID token exp with a numeric type check so an exp of 0 records idTokenExpiresAt and
fails closed downstream while a non-numeric exp is ignored, and compare the stored expiry
with != null so a gap filled expiry of 0 reads as expired instead of as no expiry at all.

Raise an actionable re authentication error for the ID token placeholder instead of
substituting an empty string. An empty substitution produced a malformed Authorization
header and a 400 downstream rather than a clean signal that the user must re authenticate.

Raise the same re authentication error from processSingleValue when a user has an OpenID
identity, the stored token set is no longer valid, and the value still contains a credential
bearing OpenID placeholder, so the expired access token case that motivated this PR signals
re auth instead of silently shipping or stripping the placeholder. Only the access token, ID
token, and generic token names raise: identity metadata resolves from the user document and
an expiry hint never needed a token, so those keep their existing literal then strip
behaviour. Unknown placeholder names also stay literal and diagnosable, matching the
existing resolvable placeholder policy.

Add the comments the review asked for on the exp fallback heuristic, the EXPIRES_AT
placeholder semantics, why stale ID token claims stay usable for identity fields, and the
advisory nature of the freshness check.

* 🪪 fix: Honour Opaque Access Tokens And Type The OpenID Re-Auth Error

Drop the ID token exp fallback in extractOpenIDTokenInfo. Storing the access token expiry
honestly means an opaque access token now records no expiry, and the fallback then handed the
ID token exp authority over a credential it does not describe. A deployment issuing opaque
access tokens alongside a short lived ID token saw isOpenIDTokenValid go false and the
credential guard reject a perfectly good access token, which worked before this branch. An
unknown access token expiry is now treated as no expiry, and the ID token exp only ever gates
ID token substitution through idTokenExpiresAt.

Give the re-authentication signal a type. OpenIDReauthRequiredError is raised at both the ID
token placeholder and the credential placeholder guard, ErrorController maps it to a 401
carrying the actionable message, and the class exposes statusCode so the agent generation
path answers 401 instead of a bare 500 for the same condition.

Omit rather than blank a header whose credential placeholder is still unresolved on a final
resolution pass, since an empty bearer credential is malformed under RFC 6750 while an absent
header lets the upstream answer its own challenge. Identity placeholders keep stripping to an
empty string.

Move the resolvable placeholder docblock onto the pattern it describes, resolve an EXPIRES_AT
of 0 as the string 0 for consistency with the neighbouring null checks, and let
AuthController consume the exported OPENID_EXPIRY_BUFFER_SECONDS so the 30 second skew
allowance has a single definition.
2026-08-18 22:02:33 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Danny Avila
5e464bc930
📎 fix: Alias Shell Script MIME Variants to application/x-sh (#14817)
* 📎 fix: Alias Shell Script MIME Variants to `application/x-sh`

Chrome on Linux reports `.sh` files as `application/x-shellscript`
(freedesktop shared-mime-info) and libmagic reports `text/x-shellscript`.
Neither string appears anywhere in the source, so uploads were rejected
even though `application/x-sh` is in the default allowlist and
`codeTypeMapping` maps `sh` to it — `inferMimeType` only consults the
extension map when the client sends no type at all, so a non-empty
browser value passed straight through to the allowlist check.

Alias both variants to the canonical `application/x-sh`, matching the
existing treatment of `text/x-markdown` and `application/x-zip-compressed`.

Also attach `statusCode`/`body` to multer file-filter rejections. Without
them the error misses the `isCustomError` branch in `ErrorController` and
falls through to a bare `500 An unknown error occurred.`, so the rejection
reason was logged server-side but never reached the client. The upload
hook already surfaces `error.response.data.message`, so a rejected file
now explains itself instead of showing a generic upload failure.

* 🔁 refactor: Move Upload Error Contract Into `packages/api`

Addresses codex P1 on #14817.

The producer of the `statusCode`/`body` pair now sits beside its consumer:
`isCustomError` and `ErrorController` are already in
`packages/api/src/middleware/error.ts`, and `CustomError` is already in
`packages/api/src/types/error.ts` — only the construction of that pair was
stranded in legacy JS. `createCustomError` is exported from the same module
as the guard that recognizes it, and `multer.js` is back to a thin caller.

Also pins the `.sh` back-compat claim with tests: configs from the
documented workarounds (`application/x-sh` per #4660/#5689/#6297, and the
broad patterns from #14804) still accept a `.sh` upload after the alias
rewrites the type. A negative control confirms the endpoint config is
genuinely in play rather than falling back to the default allowlist.
2026-08-14 01:12:23 -04:00
Danny Avila
7347cfc195
🍡 feat: Batched User Questions With A Single Bounded Answer Form (#14737)
* feat: support batched user questions

* test: align batched question fixtures

* fix: harden batched question lifecycle

* test: submit batched HITL answers in e2e

* fix: address batched question review findings

* fix: preserve invoke return typing
2026-08-11 01:06:16 -04:00
Danny Avila
ed9542ed75
🧬 fix: Rebind Request Context After Remote Agent Auth (#14685) 2026-08-08 14:27:22 -04:00
Danny Avila
dd159c4566
🔐 fix: Preserve Structured JWT Auth Context (#14652)
* 🔐 fix: Preserve structured JWT auth context

* fix: Omit identity from auth correlation logs

* fix: Isolate pre-auth request context

* style: Sort auth context imports

* fix: Preserve structured auth metadata

* fix: Narrow structured log formatter types

* fix: Annotate structured log context keys

* test: Fix request fixture typing

* fix: Namespace request path log context

* fix: Namespace request method log context

* fix: Preserve captured tenant error paths

* style: Sort tenant error imports

* fix: Classify bulk tenant isolation failures

* fix: Enforce safe request correlation invariants
2026-08-06 08:12:00 -04:00
Dustin Healy
3f0a1ec8d9
🛡️ fix: Run message-filter PII patterns on a linear-time regex engine (ReDoS) (#14554)
* 🛡️ fix: Run message-filter PII patterns on a linear-time regex engine

The messageFilter.pii middleware compiled admin-configured customPatterns with the native RegExp engine and ran them synchronously against every message on the shared event loop, so a catastrophic-backtracking pattern such as (a+)+$ could stall the entire process (native RegExp takes tens of seconds at roughly 32 characters) and take the instance down for every user.

Compile these patterns with RE2JS, a linear-time RE2 port with no native addon, so catastrophic backtracking is impossible regardless of the pattern rather than something the code tries to detect. Patterns using features RE2 does not support, such as backreferences, fail to compile and are dropped and logged exactly as an invalid pattern already is. The filter only tests for a match, so this is a drop-in engine swap with no behavior change for valid patterns.

* 🛡️ fix: Reject RE2-incompatible messageFilter patterns at config load

The customPatterns regex was validated with native RegExp at config load, but the runtime now compiles it with a linear-time engine (RE2) that does not support backreferences or lookaround. Such a pattern passed validation, then failed to compile and was silently dropped at request time, quietly removing PII protection after upgrade.

Reject backreferences and lookaround during config validation with an explicit message, and document RE2 syntax in the example config instead of "JavaScript-flavor". The runtime engine remains the authoritative boundary and still drops-and-logs anything this load-time check misses.

* 🧹 test: Use direct MessageFilterPiiConfig annotations in the PII specs

The added ReDoS cases satisfy the exported MessageFilterPiiConfig type directly, so the `as unknown as` assertions were unnecessary. Annotate the config objects directly, matching the repo's type-safety guidance.

* 🧹 fix: Reject named backreferences in messageFilter patterns at config load

Extend the config-load check to also reject named backreferences (\k<name>), which are valid JavaScript regex but unsupported by the linear-time runtime engine, so they surface at load rather than being dropped at request time. Together with the existing numeric-backreference and lookaround checks this covers the RE2-incompatible construct set; the runtime engine remains authoritative.

* 🛡️ fix: Preserve Unicode whitespace matching in messageFilter starter patterns

RE2's \s is ASCII-only, so after the engine swap the built-in api-key and Bearer starters no
longer matched a secret separated by non-ASCII whitespace (e.g. a non-breaking space), which
native RegExp did match. Broaden the whitespace classes to [\s\p{Zs}] so those patterns keep
their original coverage, and add a regression test for a non-breaking-space separator.

* 🛡️ fix: Validate messageFilter patterns with the RE2 engine at config load

Replace the syntax blacklist (numeric/named backreferences, lookaround) with authoritative
validation: config load now compiles each custom pattern with the same linear-time engine the
runtime uses, so any RE2-incompatible construct (including control escapes like \cA) is rejected
at load with a clear error instead of being silently dropped at request time.

The validator is swappable and defaults to native RegExp so browser builds add no engine; the
server wires the RE2-backed check at startup via configureMessageFilterRegexValidator in both
entry points.

* 🛡️ fix: Match the full whitespace set in messageFilter starter patterns

RE2's `\s` omits the vertical tab and `\p{Zs}` omits U+2028, U+2029, and
U+FEFF, so a separator built from one of those characters slipped past the
`api-key` and `Bearer` starter patterns and reached the model. Broaden the
starter whitespace class to the full JavaScript whitespace set so those
separators are covered again.

* fix: fail closed when messageFilter.pii compiles to zero patterns

DB and admin config overrides bypass the RE2 schema validation (it only
runs at YAML load), so an override whose only pattern is RE2-incompatible
was dropped at compile time, left zero patterns, and let the request
through. compile() now returns a failClosed flag when a config declared
patterns but every one failed to compile; the middleware returns 400 and
findPiiMatchInMessages returns a distinct misconfigured match that the
OpenAI and Responses controllers surface with an admin-facing message.

* 🛡️ fix: Fail closed when any messageFilter.pii custom pattern drops

compile() previously set failClosed only when every pattern dropped (patterns.length === 0 && dropped > 0). With the default starters present, a single RE2-incompatible custom override incremented dropped but left patterns.length > 0, so the filter silently enforced only the surviving subset and text matching only the dropped rule passed.

failClosed now keys off dropped > 0, so any dropped custom pattern blocks with the misconfigured 400. YAML patterns are RE2-validated at load, so dropped stays 0 for valid configs and only unvalidated DB or admin overrides can trip it. Reframed the two keeps-others-active specs to assert fail-closed and added a default-starters partial-drop regression.

* 🧹 fix: Correct the misconfigured JSDoc and drop redundant casts in the PII specs

The misconfigured flag now means any configured custom pattern failed to compile, not that every pattern failed, so its JSDoc on PiiMatch is updated to match. The partial-drop regressions now use direct MessageFilterPiiConfig annotations instead of as-unknown-as casts, keeping the specs type-checked, consistent with the rest of the suite.
2026-08-05 13:42:18 -04:00
Dustin Healy
044c134ecf
🤏 fix: Filter Admin Config Reads by Section-Scoped Read Capability (#14472)
listConfigs, getBaseConfig, and getConfig only checked the broad
read:configs capability, so a caller holding nothing but
read:configs:<section> grants got a blanket 403 on all three instead
of a response filtered to the sections they hold. Any deployment
using section-scoped config grants hits this.

Adds hasAnyConfigReadAccess as a cheap pre-flight check covering
broad and section-scoped read and manage grants (manage implies
read), so a zero-access caller still 403s before a DB fetch while a
section-scoped caller gets the response filtered to exactly what
they hold. The same manage-implies-read rule is fixed at its root in
getParentCapabilities so a manage-only caller sees the section they
manage instead of having it stripped after passing the pre-flight.

Resolves every section for a request in one batched
getHeldCapabilities query via getReadableConfigSections instead of
one round trip per section.

Includes AppConfig field-renaming normalization (interfaceConfig,
turnstileConfig, mcpConfig) so the filter checks the canonical
section name rather than the renamed response field, and stops
availableTools from bypassing the filter by gating it on its
filteredTools/includedTools source sections.
2026-07-28 07:38:37 -04:00
Ravi Kumar L
6f21be73a9
feat: add low-noise Redis observability (#14309)
* feat(api): add low-noise Redis observability

* fix(api): preserve Redis proxy constructor
2026-07-16 10:22:03 -04:00
Cha
73c43ded25
💂 fix: Enforce ALLOW_EMAIL_LOGIN on the Backend Login Route (#14180)
* 🔒 fix: Enforce ALLOW_EMAIL_LOGIN on Backend Login Route

ALLOW_EMAIL_LOGIN=false previously only hid the login form; POST
/api/auth/login stayed mounted and accepted valid credentials. Add a
validateEmailLogin middleware (mirroring validateRegistration /
validatePasswordReset) that rejects login with 403 when the flag is
disabled, with an ALLOW_EMAIL_LOGIN_OVERRIDE escape hatch for
intentional direct API login (each use logged with request IP).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: Gate admin local login by email login flag

* fix: Move email login gate into api package

* test: Avoid mutating readonly request ip

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-09 09:55:36 -04:00
Ravi Kumar L
44d1275f36
⚙️ perf: reduce first-load MongoDB round trips (#14101)
* perf(api): reduce first-load database round trips

* docs: move agent guidance to claude docs

* refactor(api): move message validation into api package

* fix(api): narrow active generation job lookup

* fix(api): preserve omitted source identity
2026-07-06 09:36:34 -04:00
Danny Avila
e452a130e9
⚡ perf: Minimize group membership query in principal resolution (#14055)
getUserPrincipals resolves a user's ACL principals on nearly every
authenticated request. It fetched full group documents (including entire
memberIds arrays) only to read each group _id, and always issued a
separate User lookup for idOnTheSource.

- Project { _id: 1 } on the memberIds group query so it returns only ids
  and can be served from the { memberIds: 1 } index instead of fetching
  and decoding whole group docs.
- Accept role and idOnTheSource from the already-loaded request user and
  thread them from the capability middleware, collapsing the hot path to
  a single indexed group query (idOnTheSource: null means known-local).
2026-07-02 08:44:40 -04:00
Danny Avila
6dbf9d5ad3
🪝 feat: Human-in-the-Loop Runtime - Tool Approval + Ask-User-Question (Slice B) (#13942)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* chore: add @langchain/langgraph-checkpoint-mongodb for HITL durable resume

* feat: HITL tool approval runtime — backend (Slice B)

- endpoints.agents.checkpointer config + durable Mongo checkpointer (seam over the app
  connection; SDK MemorySaver fallback) with a TTL index + deleteThread pruning
- HITL run wiring (PreToolUse policy hook + humanInTheLoop) attached in createRun, fully
  inert when toolApproval.enabled is off
- interrupt gate (pause job -> requires_action + emit on_pending_action) and a resume
  route that rebuilds the run from the durable checkpoint and run.resume()s it
- atomic single-winner resolve; agent-consistency guard; expireStaleApprovals terminal
  event; checkpoint pruned on every non-paused completion (thread_id == conversationId)

* feat: HITL tool approval UI — frontend (Slice B)

approve/reject/edit/respond + ask-user controls in the tool card (OAuth-button precedent),
batch-aware single submit, live + reconnect (resumeState.pendingAction) wiring, and resume
mutations posting to /agents/chat/resume.

* fix(hitl): decouple ApprovalProvider from chat context

ApprovalProvider is now pure state (safe to mount in provider-less / shared / test
renders); the context-dependent submit moved to a useResumeSubmit hook the cards call.
Part imports getAskUserQuestionPart from ~/utils/approval directly so suites that
partial-mock ~/utils render Part without throwing.

* fix(hitl): address Codex review — backend

- P1: enforce per-tool allowed_decisions on resume (reject a crafted decision the
  policy disallows) via findDisallowedDecisions
- prune the durable checkpoint on user-abort of a paused run, and before a fresh
  HITL turn, so a new turn cannot rehydrate an expired/aborted interrupt (thread_id
  is the stable conversationId)
- persist + use isTemporary and the original parentMessageId on resume (temporary
  chats stay temporary; initializeAgent scopes thread files off the right parent)
- generate a deferred first-turn title BEFORE completeJob so its event reaches the
  client and the final event carries the real title
- moderateText: skip when there is no text (tool-approval resume) and moderate the
  ask-user answer, instead of denying on an empty input

* fix(hitl): address Codex review — frontend

- render ToolApproval for ANY paused agent tool card (bash/code/file/etc.), not just
  the generic ToolCall, by wrapping the tool-card branch in Part (moved the rendering
  out of ToolCall)
- findPendingActionMessageIndex only matches an assistant message, never the user
  message (the underscore-strip could target the user bubble before the assistant
  placeholder exists)

* fix(hitl): address Codex re-review

- title eligibility checks the user message’s parent (first turn), not the response’s
  parent — the previous check could never be true and skipped title generation
- use client.buildResponseMetadata() for the resumed message so contextUsage /
  thoughtSignatures survive (the abort-only helper dropped them)
- moderate decisions[].responseText (the respond action’s user text)
- give /chat/abort req.config (configMiddleware) so the HITL checkpoint prune on abort
  actually runs
- read resume state BEFORE setContentParts so the in-memory store does not lose the
  pre-pause seed content
- count resumes against LIMIT_CONCURRENT_MESSAGES (increment/decrement) so paused-then-
  resumed turns cannot bypass the limit
- require actionId on resume so a body without it cannot resolve the current action

* fix(hitl): address Codex re-review (round 3) — resume fidelity

Bring the lean resume path to parity with sendMessage for things it bypassed:
- carry userMCPAuthMap into the rebuilt run so approved MCP tools keep the user's creds
- seed initialSessions (buildInitialToolSessions) so approved code/file/skill tools have
  the pre-pause uploaded-file context (esp. cross-replica / after restart)
- await client.artifactPromises and persist them as response attachments (else tool
  artifacts created after the pause vanish on reload / for late subscribers)
- merge metadata: cumulative usage (+ summary marker) from the job, contextUsage /
  thoughtSignatures from the client — fixes the round-2 regression that underreported
  post-resume cost

* fix(hitl): address Codex re-review (round 4) — resume hardening

- resume: require an EXACT paused agent_id match (reject omitted/ephemeral
  agent_id, not just a different one) and reject an endpoint mismatch, so a
  request can't rebuild the claimed checkpoint on a different graph
- moderateText: also moderate a tool-approval decision's reject `reason` and
  stringified `editedArguments`, not just `responseText`
- request: re-mark the paused response `unfinished:true` after BaseClient saves
  it as completed, so an expired / never-resumed approval doesn't leave a
  "finished" response in history; the resume path overwrites it on success

* test(hitl): route-level integration test for the resume controller

Adds api/server/controllers/agents/__tests__/resume.spec.js, a supertest
integration test that drives the real ResumeAgentController over the full
pause -> approve -> resume -> finalize lifecycle with the SDK run, durable
checkpointer, Mongo, and concurrency cache mocked. The pure decision/liveness
helpers run for real via requireActual, so the guard ladder is exercised end to
end rather than stubbed.

25 cases covering:
- the authorization / staleness / agent-and-endpoint / actionId guard ladder
- tool_approval validation (undecided tool call, policy-disallowed decision)
- ask_user_question answer requirement
- the concurrency gate (429) and the atomic single-winner claim (409)
- the happy path: ACK, run reconstruction, decision->SDK mapping, finalize
  (save the now-finished response, emit done, complete job, prune checkpoint)
- first-turn title generation before stream completion
- re-pause (no double finalize), abort-during-resume (no double finalize),
  and the resume-failure terminal path (emitError + completeJob + prune)

* test(hitl): strengthen resume coverage + add approval util tests

Acts on a self-audit of the new resume integration test.

resume.spec.js (25 -> 32 cases):
- replace the tautological emitDone assertion (it only checked the hardcoded
  `final: true`) with a structural check of the finalEvent payload —
  responseMessage content/id/unfinished, requestMessage identity, title
- cover the previously-unwalked finalize branches: tool-artifact attachments
  (null-filtered), the aggregatedContent fallback when live content is empty,
  and client response-metadata attachment
- add guard cases: unsupported pending-action type (400) and the
  pre-multi-tenancy null-tenantId pass-through (must not 403)
- add error-path cases: first-turn title generation throwing must still
  finalize, and a completeJob failure during a resume error must force a
  terminal job state via the last-resort updateJob

client/src/utils/approval.spec.ts (new, 15 cases):
- applyPendingAction tool_approval: join by tool_call_id not position,
  skip completed calls, default allowed_decisions to [], referential
  stability when nothing changes
- applyPendingAction ask_user_question: append, idempotent replace on replay,
  non-array content coercion
- getAskUserQuestionPart type guard; findPendingActionMessageIndex
  assistant-only resolution (never resolves to the user bubble)

* fix(hitl): address Codex re-review (round 5)

Five findings verified against the code before fixing:

- resume: require an EXACT endpoint match (like agent_id) — a resume that OMITS
  endpoint must not fall through, since the shared chat middleware treats a
  missing/non-agents endpoint as the ephemeral agent and could rebuild the
  claimed checkpoint on a different graph
- resume: filter malformed content parts before saving the finished response,
  matching the normal AgentClient path (a resumed turn could otherwise persist
  an empty/invalid tool_call part that breaks reload/rendering)
- resume: accumulate tool artifacts across pause segments — persist them on
  re-pause and MERGE (not overwrite) at finalize, so artifacts produced before
  a second approval pause aren't dropped by the next rebuilt client
- approval (client): findPendingActionMessageIndex returns -1 when a provided
  responseMessageId isn't found, so the caller retries instead of attaching the
  prompt/approval to a prior assistant reply; fall back to the last assistant
  only when no responseMessageId is given
- RedisJobStore: make appendChunk extend-only (XADD + EXPIRE-if-shorter via a
  single eval) so the on_pending_action chunk emitted after a pause can't reset
  the chunk-stream TTL back to the running window and evict pre-pause content
  before the approval is resolved

Tests: +endpoint-omitted/unsupported-type/malformed-filter/attachment-merge/
re-pause-persist cases in resume.spec.js (36); ask-retry -1 semantics in
approval.spec.ts (16); extend-only TTL assertion in the RedisJobStore Redis
integration spec.

* test(hitl): mongodb-memory-server integration test for the checkpointer seam

The checkpointer unit spec covers config/selection with no DB connection; this
exercises the durable Mongo seam against a real (in-memory) MongoDB — the part
correctness actually depends on:

- getAgentCheckpointer builds a real MongoDBSaver when Mongo is connected and
  setup() creates the TTL index (expireAfterSeconds) on the checkpoint collection
- memory type returns undefined (SDK MemorySaver fallback) even when connected
- saver is memoized per resolved config
- deleteAgentCheckpoint prunes a thread's persisted checkpoint (the cross-turn
  isolation guarantee: turn N+1 on the same conversationId can't rehydrate it)
- pruning is thread-scoped — deleting one conversation leaves others intact
- undefined threadId is a no-op

* fix(hitl): address Codex re-review (round 6)

Four findings verified against the code before fixing:

- messageFilterPii: scan the resume payload's user-authored text (ask-user
  `answer`, and a tool-approval decision's `respond` text, `reject` reason, and
  edited tool arguments) — the shared /resume route ran through the PII filter
  but it only inspected req.body.text, so a blocked token rode the resume
  payload back into the model/tool (mirrors the earlier moderateText fix)
- resume: re-prime skill files invoked in the pre-pause segment before rebuilding
  the run, so an approved code/file-backed tool keeps the injected skill-file
  session refs instead of running without them (mirrors the normal path's
  primeInvokedSkills; the pre-pause content stands in for the message payload)
- hitl: pin the graph identity. Persist a fingerprint of the graph-determining
  request fields (endpoint, agent_id, model, spec, ephemeralAgent — normalized)
  on the pending action at pause, and reject a resume whose recomputed
  fingerprint differs. This closes the ephemeral-agent gap, where agent_id is
  undefined so the id guard can't tell two ephemeral configs apart
- resume: reject incomplete edit/respond decisions (findIncompleteDecisions) —
  an `edit` without an object editedArguments or a `respond` without non-empty
  responseText is 400'd before mapping, rather than defaulting to {} / '' and
  resuming with behavior the user never approved

Tests: incomplete-decision + fingerprint match/mismatch cases in resume.spec.js
(41); findIncompleteDecisions + computeAgentRequestFingerprint unit tests; and
resume-field PII cases in messageFilterPii.spec.ts.

* fix(hitl): address Codex re-review (round 7)

Four findings verified against the code before fixing:

- RedisJobStore: clear `agent_id` on createJob (add it to staleHitlFields). The
  job hash is keyed by conversationId and reused across turns; updateMetadata
  only writes agent_id when truthy, so a conversation that switched from a saved
  agent to an ephemeral/no-agent turn kept the old id and the resume guard
  rejected the valid pause as a different agent. (real correctness bug)
- fingerprint: include `promptPrefix` in computeAgentRequestFingerprint, and
  re-send it on resume (ResumeAgentFields + buildResumeFields). Ephemeral agents
  derive their system instructions from promptPrefix, so a resume changing it
  previously passed the pin and rebuilt different instructions. (completes the
  round-6 fingerprint)
- resume: the re-pause branch now persists the segment's accumulated CONTENT
  (filtered), not just artifacts, so an approval that expires/reaps without a
  final resume no longer loses everything streamed during the resumed segment.
- request: carry `manualSkills`/`alwaysAppliedSkills` on the persisted user
  message so a resumed turn's reconstructed requestMessage keeps its skill pills
  instead of dropping them until a full reload.

Deferred (narrow, no safe contained fix yet — see PR thread replies):
- resume rebuild without `addedConvo` for a multi-conversation/added-agent pane
- cross-replica re-prime of manually-selected (not model-invoked) skill files

Tests: stale-agent createJob clearing (Redis integration), promptPrefix
fingerprint match/mismatch (resume.spec.js + policy.spec.ts), re-pause content
persistence (resume.spec.js).

* fix(hitl): address Codex re-review (round 8)

Five findings verified against the code before fixing; the headline is a durable-
resume correctness fix (the fingerprint had surfaced it as a 403):

- resume durability (the important one): persist the graph-determining request
  fields (endpoint, agent_id, model, spec, promptPrefix, ephemeralAgent) on the
  pending action as `resumeContext`, and REPLAY them onto the resume request via
  a router-level middleware that runs before buildEndpointOption. The client
  can't reconstruct the ephemeral-agent config after a reload/cross-session, so
  the round-6/7 fingerprint would 403 a valid durable resume — and even without
  it the rebuilt agent would lose its tools. Replaying server-side rebuilds the
  SAME graph regardless of client state (and a crafted resume can't swap it; the
  fingerprint still matches because the body is restored first).
- RedisJobStore: also clear `isTemporary` on createJob (same class as agent_id):
  a prior temporary turn's flag would otherwise survive a reused conversation
  hash and a later non-temporary resume would save its response as temporary.
- resume: persist `contextMeta` (context-window calibration) onto the saved
  response like BaseClient does, so the next turn can seed its pruner.
- request: carry manualSkills/alwaysAppliedSkills into the onStart metadata
  update (not just the preliminary one it overwrites), so a resumed turn's
  requestMessage keeps its skill pills.

Deferred (narrow — see thread reply):
- saved-agent edited WHILE a run is paused: agent_id matches but the definition
  changed; needs an agent version/config hash, which is a larger change for a
  narrow window.

Tests: resumeContext pick/apply + round-trip (policy.spec.ts), contextMeta +
manualSkills-on-requestMessage (resume.spec.js), isTemporary clearing (Redis
integration).

* style(hitl): prettier line-wrap in policy.spec.ts (R8 lint fix)

* fix(hitl): address Codex re-review (round 9)

Five findings, all fixed (addedConvo — deferred in rounds 7/8 — is now trivial
thanks to the round-8 replay):

- replay addedConvo: add it to RESUME_CONTEXT_KEYS so the resume middleware
  restores the parallel/secondary-agent config from the paused request; the
  client can't reconstruct it, and it determines the rebuilt graph.
- skill pills (the real fix this time): the round-8 onStart metadata write was
  overwritten by trackUserMessage (the authoritative userMessage writer). Carry
  manualSkills/alwaysAppliedSkills in the emitted `created` message and persist
  them in trackUserMessage; widen UserMessageMeta + SerializableJobData.userMessage.
- execute-code files on resume: seed the paused user message's own files onto
  req.body.files before initializeClient — they're excluded from the
  parent-walk code-session rebuild, so an approved code/read-file tool would
  otherwise resume without them.
- in-memory pending-action UI: route ApprovalEvents.ON_PENDING_ACTION in the
  resume replay/pending-event loops to applyPendingActionToMessages (mirror the
  live handler), so a pause that lands in the snapshot window still renders its
  approval controls instead of sitting paused with no UI.
- abort isTemporary: the /chat/abort partial-save now sources isTemporary from
  the job metadata, not req.body (the stop button posts only conversationId), so
  aborting a paused temporary chat no longer persists an orphaned partial.

Tests: addedConvo in pickResumeContext (policy.spec.ts), file-restore on resume
(resume.spec.js), abort-from-job-isTemporary (abort.spec.js).

* fix(hitl): address Codex re-review (round 10) — resume/expiry races

Three concurrency/coherence findings, verified against the code before fixing:

- expiry-sweep CAS scope: both stale-approval sweeps (GenerationJobManager
  expireStaleApprovals and the RedisJobStore requires_action cleanup) called
  expire()/transitionStatus WITHOUT the observed pendingAction.actionId, so the
  CAS only checked status===requires_action. Between the read and the CAS a user
  could resolve the observed action and the run re-pause on a FRESH action; the
  stale sweep would then abort that valid new pause. Now both pass the observed
  actionId as expectActionId, so the CAS only fires for the action read as stale
  (a re-paused action has a different id → no-op).
- resume graph cache: resumeCompletion cached the rebuilt graph (created with
  messages:[]) via setGraph; RedisJobStore.getContentParts prefers a cached
  graph over reconstructing from the chunk log, so a same-replica reload/status
  poll mid-resume returned aggregatedContent missing the pre-pause content. Skip
  setGraph on resume so introspection falls back to the complete chunk
  reconstruction (setContentParts still seeds the in-memory store).
- pending-action UI: applyPendingActionToMessages scheduled a SINGLE
  animation-frame retry then dropped the pending action; Recoil/React updates can
  take several frames under load, leaving a valid requires_action run with no
  approval controls. Retry across frames (bounded at 120) until the target
  message commits.

Test: expire() with a mismatched expectedActionId no-ops while the matching id
expires (pendingAction.spec.ts).

* chore(deps): update @librechat/agents to version 3.2.53 and @langchain/langgraph to version 1.4.7 in package-lock.json and related package.json files

* refactor(hitl): add resolveToolApprovalPolicy seam for layered policy

Extract the single point where tool-approval policy is resolved for a turn
(`resolveToolApprovalPolicy`) and route the run call site through it instead
of reading `endpoints.agents.toolApproval` inline.

Behaviour-preserving: only the `endpoint` layer is wired today, so the result
is identical to reading the app policy directly. The `agent` and `skills`
layers are reserved seams with documented precedence (endpoint owns the
`enabled` kill switch; agent overrides mode/allow/deny/ask/reason; skills may
only tighten), so future per-agent and per-skill policy plumbing lands in one
function rather than at the `createRun` site. Adds focused unit tests.

* fix(hitl): address Codex re-review (round 11) — resume hardening

F1 (P2, security) — applyResumeContext now DELETES any RESUME_CONTEXT_KEY
absent from the persisted context, so the resume body carries exactly the
graph-determining fields the pause had. Previously only defined keys were
overwritten, leaving a client-supplied `addedConvo` (which the request
fingerprint does not cover) in place — a crafted resume could rebuild a
single-agent checkpoint as a different multi-agent graph/tool set.

F3 (P2) — the resume route ACKs (res.json) before initializeClient, so a
post-ACK getMCPRequestContext(req, res) saw the response as finished and
returned undefined, leaving the resumed run without its run-scoped MCP
connection store (approved MCP / OAuth-overlay tools then ran without their
request-scoped connections). Pre-seed the store with a null res +
cleanupOnResponse:false before the ACK and tear it down in the finally,
mirroring the normal stream path (request.js). userMCPAuthMap was already
preserved separately, so credentials were not lost — only the connection store.

Declined: the ApprovalContext NEW_CONVO guard (P2) is a false positive — the
`created` SSE event updates the conversation atom before any pause renders, so
the id is concrete by click time (details in the PR thread).

Tests: policy.spec (absent-key delete) + resume.spec (MCP context pre-seed/cleanup order).

* fix(hitl): address Codex re-review (round 12) — resume fidelity + multi-tool UI

F4 (P2) — temporal prompt vars: resume rebuilt the agent without restoring
req.conversationCreatedAt or req.body.timezone, so {{current_datetime}}-style
vars compiled a different system prompt than the paused graph (resume wall-clock,
unzoned). Add 'timezone' to RESUME_CONTEXT_KEYS (persisted at pause, replayed by
the resume middleware) and restore conversationCreatedAt from the convo before
initializeClient — mirroring the normal path's resolveConversationCreatedAt.

F5 (P2) — multi-tool approval: applyPendingActionToMessages stopped retrying once
ANY tool-call part was tagged, so siblings that rendered on later frames never got
approval controls and the resume route 400'd the partial batch. Add
countTaggedApprovalParts and keep the bounded RAF retry going until every
action_request is tagged (ask_user_question unchanged — one synthetic part).

F6 (P3) — Edit accepted `null`/`[]` (valid JSON, non-object), enabling Submit for
a value the resume route rejects via findIncompleteDecisions. Mirror the server's
plain-object check in the client (store + editIsValid) so Submit only enables for
an accepted value.

Tests: policy.spec (timezone round-trip), resume.spec (conversationCreatedAt
restore), approval.spec (countTaggedApprovalParts).

* fix(hitl): address Codex re-review (round 13) — recurse into subagent approvals

F9 (P2) — a tool paused INSIDE a subagent has its tool_call_id in the parent
subagent tool_call's nested `subagent_content`, not as a top-level message part.
applyToolApproval and countTaggedApprovalParts only scanned top-level content, so
the approval never attached and the round-12 retry loop counted 0 tagged parts and
spun to its frame cap with no controls. Both now recurse into `subagent_content`
(immutably, so React refs update): the nested call gets tagged and is counted, so
the retry terminates. Added approval.spec cases for the nested tag + count.

Note: surfacing the interactive approve/reject controls inside the subagent view is
a deliberate follow-up — ToolApproval -> useResumeSubmit -> useChatContext crashes
when rendered in the portaled subagent dialog (outside the chat/approval providers),
so that needs the controls scoped to the in-provider inline render (or the dialog
wrapped with the providers). This commit fixes the data/traversal layer only.

F7 (discovered-tool history on resume) and F8 (redis chunk TTL pause race) were
verified false positives — see the PR threads.

* fix(hitl): address Codex re-review (round 14) — resume fidelity + expiry relay

F13 (P2) — manualSkills are graph-determining (skill allowed-tools union into the
tool set before tools load) but weren't replayed, so a reload lost the skill tools
and a crafted resume could inject a different skill past the fingerprint. Add
'manualSkills' to RESUME_CONTEXT_KEYS (same replay-only pattern as timezone/
addedConvo; the delete-absent half blocks injection). Not alwaysAppliedSkills —
that's resolved server-side from the DB, not req.body.

F12 (P2) — the resume final SSE built requestMessage from job.metadata.userMessage
(persisted without files), so attachments vanished from the user bubble on resume.
Spread the already-restored req.body.files onto it, matching the normal path.

F11 (P2) — multi-replica approval expiry: RedisJobStore.cleanupRequiresActionIndex
on another replica can win the requires_action->aborted CAS (it sets the hash error
but has no event transport), and the local sweep then skips because the job is no
longer requires_action, so a client subscribed here never gets the terminal error
until the reap path. expireStaleApprovals now relays APPROVAL_EXPIRED_ERROR for a
locally-subscribed job already aborted FOR approval expiry (error-string gated,
idempotent via the errorEvent flag). emitError already publishes cross-replica.

Tests: policy.spec (manualSkills round-trip + inject-drop), resume.spec (final
requestMessage carries restored files).

* fix(hitl): render approval controls for subagent-nested tool pauses (F10)

Round-13 made applyToolApproval/countTaggedApprovalParts recurse into
subagent_content (data), but SubagentDialogPart rendered nested TOOL_CALL parts
with <ToolCall> only and never mounted <ToolApproval>, so a tool paused inside a
subagent showed no controls and the run was unresolvable.

Render <ToolApproval> in SubagentDialogPart's TOOL_CALL branch when the nested
tool_call carries an approval and isn't yet resolved, mirroring the top-level
Part.tsx render. The subagent dialog portals (OGDialog → ReactDOM.createPortal),
but React context flows through the React tree, not the DOM tree, so ToolApproval
resolves ApprovalProvider/ChatContext and the controls work + submit.

Also harden useResumeSubmit: read ChatContext via useContext (non-throwing)
instead of the throwing useChatContext wrapper, so the cards never crash when
rendered outside a ChatContext.Provider (e.g. a search/citation render that passes
chat context as a prop) — they degrade to inert (buildResumeFields returns null).

* style(hitl): re-sort run.ts imports after dev rebase

* fix(hitl): address Codex re-review (round 15) — resume content fidelity

F14 (P2) — hide_sequential_outputs was applied in chatCompletion before
saving/emitting content but not on resume, so a sequential-agent chain that
pauses for HITL and resumes persisted/emitted intermediate outputs the setting
is meant to hide. Extracted the filter into applyHideSequentialOutputsFilter()
and call it from both chatCompletion and resumeCompletion (after handleRunInterrupt,
covering the finalize + re-pause reads of client.contentParts).

F16 (P2) — on a reloaded HITL pause, the DB already holds the paused user row +
partial assistant row; useResumeOnLoad fed those as submission.messages, then
finalHandler/createdHandler appended the same pair via requestMessage/responseMessage,
duplicating the turn (buildTree doesn't dedupe children by messageId). buildSubmission-
FromResumeState now strips the paused user/response rows (by messageId, incl. the
padded/unpadded response id) from submission.messages — they're re-supplied by the
placeholders + final event. Frontend-only; live (non-reload) pause path untouched.

Deferred: F15 (collapsed-card subagent approval registration/visibility) — see thread.

Tests: client.test (filter keeps last + tool_call parts / no-op when off),
useResumeOnLoad.spec (paused pair stripped from submission.messages).

* fix(hitl): address Codex re-review (round 16) — chunk TTL, slot, job replacement

F17 (P2) — chunk-stream TTL on pause-before-chunk. CHUNK_APPEND_LUA derived its
ceiling only from the chunk key's current TTL, so when the chunks key didn't exist
at pause (fire-and-forget append in flight, or an ask-user pause before any chunk),
the on_pending_action append created the stream with only the 20m running TTL while
the approval window is 24h — content evicted before resume. The Lua now also reads
the job key (KEYS[2]); when status == requires_action it takes max(running, TTL(jobKey))
(the approval window transitionStatus set), else the running TTL. Extend-only preserved;
gated on paused status so normal runs never inflate. Both keys share {streamId} (cluster-safe).

F19 (P2) — with LIMIT_CONCURRENT_MESSAGES, the approval prompt was emitted before the
original request released its slot, so a fast Approve got /resume 429'd. handleRunInterrupt
now releases the slot (idempotent via pendingRequestReleased) right after the pause, before
the prompt; the request.js pause branch and resume.js finally only release if it didn't
(no double-release).

F20 (P2) — finalizeResumedTurn never checked the job wasn't replaced before emitDone/
completeJob/saveMessage, so a stale resume could clobber a newer turn that reused the
conversationId. Added the createdAt guard the normal request path uses (skip finalization
when the live job's createdAt != the paused job's).

Deferred: F18 (subagent_content not reconstructed on Redis resume) — joins the subagent
cluster (F15). See thread.

Tests: RedisJobStore integration (pause-before-chunk gets approval TTL; running stays short),
resume.spec (skip finalization on replacement; no double slot release on re-pause).

* 🛡️ fix: Guard HITL terminal side-effects against job replacement

Jobs are keyed by streamId == conversationId, so a new request REPLACES the
running one on the same conversation. The replaced generation's tail must not
clobber the live generation's state. Each path now re-reads the live job and
compares createdAt against the generation's captured identity before acting.

- Thread the generation's createdAt onto the client (request.js + resume.js)
  as client.jobCreatedAt — the identity every guard compares against.
- handleRunInterrupt: skip approvals.pause when this run is no longer the live
  job, so a stale interrupt can't flip the NEWER job to requires_action.
- chatCompletion finally: skip the checkpoint prune when replaced, so an older
  run's late finally can't delete the newer run's resume checkpoint.
- resume catch-path: gate emitError/completeJob/prune behind a stillLive check
  (fail-open if the read throws), mirroring finalizeResumedTurn's success guard.
- Persist the turn's uploaded files on job.metadata.userMessage (authoritative
  trackUserMessage writer) and prefer them on resume over the user DB row, whose
  save can still be racing a fast /resume.

Tests: 13 guard-predicate cases in jobReplacement.spec.js.

* 🔁 fix: Harden HITL resume — ownership re-check, file seeding, deferred-tool replay

Three follow-ups to the round-17 job-replacement guards (Codex review 4594099963):

- G1 (resume.js): the success-path ownership guard runs at the START of
  finalizeResumedTurn, but saveMessage + first-turn title generation await long
  enough for a new request to replace the job on the same conversationId. Re-read
  the live job immediately before emitDone/completeJob/prune so the terminal writes
  can't tear down the REPLACEMENT job — mirrors the catch-path guard.

- G2 (request.js): onStart's metadata/chunk writes that persist the turn's files
  are fire-and-forget, so a fast approval could read job.metadata.userMessage before
  files landed. Seed files into getPreliminaryUserMessage instead — that write is
  AWAITED before the run starts, so files are durable before any interrupt can emit.

- G3 (run.ts + client.js + resume.js + IJobStore.ts): the resumed graph is rebuilt
  with messages: [], so createRun's tool_search-discovery scan finds nothing. A
  deferred tool discovered earlier in the turn (and targeted by the paused call) was
  therefore absent from the rebuilt schema-only toolMap — resume would throw "unknown
  tool" (no loadRuntimeTools fallback is wired). Capture discovered tool names at
  pause via extractDiscoveredToolsFromHistory(run.getRunMessages()), persist them on
  job.metadata.discoveredTools, and replay them into createRun's new discoveredToolNames
  input (merged with message-extracted names, gated on hasAnyDeferredTools — inert
  otherwise). A new createRun test proves the deferred tool is promoted with the replay
  and absent without it (reproducing the bug).

Tests: real createRun deferred-replay suite (run-summarization.test.ts) + G1/G2/G3
guard predicates (jobReplacement.spec.js). Full suite green.

* 🔒 fix: Close HITL resume metadata + file-substitution + pause-race gaps

Four findings on the round-18 commit (Codex review 4594430222):

- H1 (P1, regression in round-18 G3): the discoveredTools captured at pause never
  reached resume — three metadata allowlists dropped it: GenerationJobManager
  .updateMetadata, RedisJobStore.deserializeJob, and buildJobFacade (plus the
  GenerationJobMetadata type). Added discoveredTools to all four, so the deferred-tool
  replay actually works end-to-end (in-memory store already kept it via Object.assign).

- H2 (P2, security): /resume honored a client-supplied `files` array, letting a crafted
  client resume an approved code/read-file tool against a DIFFERENT file set than the one
  approved (files aren't in the resume fingerprint/context). Resume now ALWAYS sources
  files from the paused job (metadata → DB row), clearing any client-supplied set.

- H3 (P2, ephemeral fidelity): non-default model parameters (temperature, max tokens,
  custom endpoint params) were lost on resume — ephemeral agents derive them from the
  request body, which the resume payload omits. Capture the resolved model_parameters in
  resumeContext at pause and replay them onto the body on resume (excluding `model`, which
  is replayed via the fingerprinted RESUME_CONTEXT_KEYS path). Saved agents already source
  these from the DB.

- H4 (P2, Redis race): a pause landing between the resume snapshot and the Pub/Sub
  subscription reached neither resumeState.pendingAction nor (Redis) pendingEvents, and
  approval events aren't persisted to replayEvents — the client attached to a paused job
  with no approval UI. subscribeWithResume now re-reads the live job AFTER subscribing and
  surfaces the pending action if the snapshot missed it (live read, no staleness).

Tests: discoveredTools metadata round-trip + subscribeWithResume re-read (pendingAction
.spec.ts); client-file substitution rejection (resume.spec.js); model-parameter replay
predicate (jobReplacement.spec.js).

* 🧹 fix: Clear stale discovered tools, release slot on claim error, extend run-step TTL

Three follow-ups on the round-19 commit (Codex review 4594783691):

- I1 (P2): the round-19 discoveredTools field wasn't cleared on Redis streamId reuse.
  HSET only overwrites listed fields and handleRunInterrupt only writes discoveredTools
  when THIS turn discovers a deferred tool — so a replacement turn that pauses without its
  own discovery inherited the prior run's tool names and force-loaded undiscovered deferred
  tools on resume. Added discoveredTools to createJob's staleHitlFields HDEL list (the
  in-memory store already builds a fresh object, so it was Redis-only).

- I2 (P2): with LIMIT_CONCURRENT_MESSAGES, approvals.resolve runs after the slot increment
  but before the run's try/finally, so a store/Redis error there leaked the slot until the
  counter TTL expired (spurious 429s on retry of the still-paused approval). Wrapped the
  claim in try/catch that decrements the slot and returns 500.

- I3 (P3): saveRunSteps did SET ... EX running unconditionally, resetting the run-steps key
  to the 20-min running TTL even while the job is paused for the longer approval window —
  a reload after that window lost the tool timeline. Now uses a paused-window TTL script
  mirroring the chunk-stream no-shrink behavior (extends to the approval window when the
  job hash is requires_action).

Also fixes a latent strict-tsc cast error in the round-19 pendingAction test.

Tests: claim-throws-releases-slot (resume.spec.js); discoveredTools cleared on reuse +
saveRunSteps preserves the paused TTL (RedisJobStore integration, USE_REDIS).

* 🛡️ fix: Guard fast-resume save race, gate HITL to resumable routes, expire on stale submit

Three findings on the round-20 commit (Codex review 4595045652):

- J2 (P1): a fast /resume can claim + finalize the COMPLETED response while the original
  request's pause branch is still awaiting `response.databasePromise`; the later
  unfinished-save then overwrites the completed content. Re-check the job is still paused on
  THIS generation's action (a claim leaves requires_action; a replacement bumps createdAt)
  before marking the row unfinished; fail open on a read error.

- J3 (P1): the tool-approval wiring (humanInTheLoop + PreToolUse hook + checkpointer) was
  applied to EVERY createRun caller when toolApproval.enabled, but the OpenAI-compatible and
  Responses controllers never inspect run.getInterrupt() or persist a pending action — an
  approval-gated tool would pause there with no approval surface or resume endpoint and the
  route would emit a normal final response / [DONE] with the tool call dangling. Gate the
  wiring on a new createRun `hitlCapable` flag, set only by AgentClient (chat + resume).

- J4 (P2): a stale-action 409 on submit returned without driving expiry, leaving the job
  requires_action with a dead action until the periodic sweeper ran — any attached SSE client
  got no terminal event and the stream appeared to hang. Extracted GenerationJobManager
  .expireApproval(streamId, actionId) (expire CAS + terminal SSE, shared with the sweeper) and
  call it from the resume route when the observed action is stale.

J1 (nested subagent approval controls not mounting while the details dialog is closed) is a
valid frontend issue in the deferred subagent-HITL path — tracked separately (replied on the
thread) since the fix touches the shared dialog primitive and needs UI verification.

Tests: HITL-gate both directions (run-summarization.test.ts); expire-on-stale-submit
(resume.spec.js); fast-resume unfinished-save guard predicate (jobReplacement.spec.js).

* 💄 style: Wrap captureAgents signature to satisfy prettier (CI lint)
2026-06-29 16:56:41 -04:00
Danny Avila
5eb1c2c107
🖇️ feat: Reference Selected Chat Text with Multi-Quote Popup (#13868)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🖇️ feat: Reference Selected Chat Text with Multi-Quote Popup

Add a ChatGPT/Codex-style quote feature: selecting text in any message shows
an 'Add to chat' popup that accumulates removable quote chips above the
composer. On submit, the excerpts are merged into the user message text as
Markdown blockquotes (counted in the user message token count, not a system
message) and persisted on the message so they render on the user bubble and
survive reload.

- packages/api: add getReferencedQuotes + mergeQuotedText helpers (blockquote merge, length/count caps) with unit tests
- BaseClient.sendMessage: temporarily merge req.body.quotes into userMessage.text before buildMessages, restore clean text, persist quotes array
- data-schemas + data-provider: add optional quotes field to message schema/type
- client: pendingQuotesByConvoId atom, QuoteButton selection popup, PendingQuoteChips composer row, MessageQuotes persistent display
- useChatFunctions: drain pending quotes onto the message, carry forward on regenerate
- add localization keys and component/integration tests

* 🧪 test: Add Playwright e2e for chat quote feature

Add e2e/specs/mock/quotes.spec.ts covering select -> 'Add to chat' popup ->
chip -> send -> persistent reference block -> reload, plus multi-select
accumulation and chip removal. Selection is driven programmatically (real DOM
Range + dispatched mouseup) to summon the popup deterministically.

Add data-testid hooks (add-to-chat-button, pending-quote-chips, message-quotes)
to the quote components for stable selectors.

* 🛡️ fix: Address Codex review on quote feature

- Run PII filter + OpenAI moderation over req.body.quotes (P1): quoted excerpts
  are merged into the model-facing user message, so they must clear the same
  filters; a crafted quotes payload could otherwise bypass them. Adds tests.
- Carry quotes through edit/save-and-submit replays (overrideQuotes in
  EditMessage), mirroring overrideManualSkills, so edited turns keep context.
- Hide the quote UI for Assistants endpoints (which bypass BaseClient merge),
  so users can't queue quotes the assistant never receives.
- Clear pending quote/skill queues by resolved conversationId in useClearStates,
  not the UI index, so queued-but-unsent selections don't linger in Recoil.
- Cap queued quotes client-side at 10 to match the backend QUOTE_MAX_COUNT, so
  the composer never shows more quotes than are actually sent.

* 🧵 fix: Durably re-merge quotes + Codex round 2

Address Codex's re-review of the quote feature:

- Durable history re-merge (per maintainer decision): quotes are no longer
  merged at request time and stripped; instead each user message's persisted
  message.quotes is merged into its formatted content in AgentClient.buildMessages
  (new prependQuotes helper) for current AND historical turns. The model
  receives the referenced context on every prompt and the token count stays
  consistent with what was persisted; stored text stays clean for display.
- Attach normalized quotes to the user message in handleStartMethods (before
  getReqData/onStart) so the optimistic bubble, resumable abort metadata, and
  saved row all carry them (fixes the abort-metadata gap).
- Skip the quote drain entirely for Assistants endpoints in useChatFunctions,
  leaving the pending atom intact (UI is already hidden there).
- Normalize req.body.quotes via getReferencedQuotes before moderation/PII so
  only the trimmed/truncated/capped excerpts the model will receive are checked.
- Tests: prependQuotes unit tests; BaseClient quote tests assert early
  attachment + clean text; e2e now verifies the model receives the merged
  blockquote on the current turn and re-merged from history on a later turn
  (new E2E_ASSERT_QUOTE mock marker).

* 🔗 fix: Quote share/memo/abort/PII gaps (Codex round 3)

- Shared links: include quotes in the anonymized projection + SharedMessage
  type (+test) so the /share view renders the same reference blocks as the
  owner, mirroring manualSkills/alwaysAppliedSkills.
- MessageRender memo: compare quotes length so a server/resume copy whose only
  change is the quote list re-renders (the block no longer goes stale/missing).
- Resumable job metadata: include quotes in the userMessage written to
  GenerationJobManager so a reload/reconnect mid-stream reconstructs the chips.
- PII + moderation: also scan the merged blockquote+text exactly as the model
  receives it, so a secret split across a quote and the typed body (each clean
  alone) is caught (+cross-boundary test).
- e2e: make quote-add robust against the auto-scroll-dismisses-selection race
  via a retried select+click helper.

* 🛑 fix: Keep quotes on aborted turn's request message (Codex round 4)

abortMiddleware reconstructs finalEvent.requestMessage from jobData.userMessage
but only copied ids + text; include quotes so a stopped quoted turn keeps its
MessageQuotes in the UI and a regenerate-before-reload still sends the
referenced context. Completes the resumable-metadata fix from the prior round.

* 🧮 fix: Quote recount + preliminary abort metadata (Codex round 5)

- Force a canonical token recount for messages carrying quotes in
  AgentClient.buildMessages, so a plain text-only Save edit (which recomputes
  tokenCount from text alone) can't leave a stale, quote-excluding count that
  undercounts context on later turns — recount from the quote-merged copy
  self-heals it.
- Seed normalized quotes into the preliminary userMessage metadata
  (getPreliminaryUserMessage), so an abort during init/tool-loading (before
  onStart) still reconstructs the stopped turn's MessageQuotes.

* ✅ fix: Add getReferencedQuotes to controller test mocks (CI)

request.js's getPreliminaryUserMessage now calls getReferencedQuotes; the
agents controller specs mock @librechat/api wholesale, so the mock must export
it or the call throws and cascades. Added a faithful mock (normalize/cap,
null when empty) to request.resumeMetadata.spec.js and jobReplacement.spec.js.

* 📐 fix: Quotes in context projection + resumable metadata (Codex round 6)

- Context-usage projection (resolveContextProjection): select message.quotes,
  prepend them into the projected user text, and recount quoted messages so the
  context gauge counts the same prompt the model receives (a text-only Save edit
  no longer makes the gauge undercount / over-report remaining budget).
- Resumable job metadata: trackUserMessage (created-event rewrite) and abortJob
  (final requestMessage) now carry quotes; SerializableJobData.userMessage and
  CreatedEvent.message gained an optional quotes field. With the cross-replica
  created-event spread, stopping/reconnecting a quoted turn after the created
  event keeps its MessageQuotes.

* 💬 feat: Collapse multi-select quotes into one chip with hover popup

Composer feedback: the quote chip area now shows a single chip — the excerpt
text for one selection, or a collapsed "{n} selections" pill for multiple,
with a hover popup (HoverCard) listing every excerpt and a per-item remove. The
chip is taller (py-1.5/text-sm) to read less skinny. Adds com_ui_quote_selections
and com_ui_remove_all_quotes; updates unit + e2e tests (e2e drives the count via
a data-quote-count hook and exercises the hover popup).

* ♿ fix: Make multi-selection quote popup keyboard accessible

The collapsed "{n} selections" pill used a HoverCard, which Radix only opens on
pointer hover — its interactive content was unreachable by keyboard. Replaced it
with a Popover: the trigger is a real button that opens on click / Enter / Space
(focus moves into the list, each excerpt's × is tab-navigable, Escape closes and
restores focus), with hover-open preserved for mouse via controlled open state +
a close grace period. Hover-initiated opens skip auto-focus so they don't pull
focus off the composer. Adds an e2e asserting keyboard open/close.

* 📐 fix: Clamp the Add-to-chat button within the viewport (Codex round 7)

The floating selection button positioned via translate(-50%,-100%) (bottom-center
anchor) but clamped top/left as if they were its top-left, so a selection near
the viewport top or sides could render the button partly/fully offscreen. Now it
measures the button (ref + useLayoutEffect) and computes an on-screen top-left —
clamping by the full width within side margins and flipping below the selection
when there's no room above — with no transform, and stays hidden until measured
so it never flashes at an unclamped spot.

* ↩️ fix: Restore pending quotes on early-abort draft (Codex round 8)

When a turn is stopped before the created event (e.g. during tool/MCP init), the
final handler restores requestMessage.text to the draft, but the pending-quote
atom was already drained on submit — so a retry sent no quotes. The abort
requestMessage now carries quotes (preliminary metadata + abort fixes), so the
three early-abort/no-response draft-restore paths in useEventHandlers now also
re-queue pendingQuotesByConvoId from requestMessage.quotes.

* ♿ fix: Use Ariakit Popover for quote selections (keyboard focus)

The multi-selection popup used a hand-rolled Radix Popover with Popover.Anchor +
a manual button, so Radix had no trigger to return focus to — Escape dumped
focus to the page top. Refactored to Ariakit (the codebase's popover primitive,
per DropdownPopup/Fork): the `PopoverDisclosure` is the real trigger, so Escape
closes and returns focus to the composer instead of the top of the page. Keyboard
opens (Enter/Space) autofocus into the list and tab through each excerpt's remove;
hover opens for mouse with autofocus suppressed so it never pulls focus off the
composer. e2e asserts the keyboard open/navigate/Escape flow keeps focus on a
real control (never BODY).
2026-06-21 08:33:11 -04:00
Danny Avila
98704f28c1
🌐 fix: Centralize Outbound Proxy Handling (#13726)
* fix: centralize outbound proxy handling

* chore: sort proxy imports

* test: update proxy helper mocks

* fix: honor proxy bypasses consistently

* fix: support http axios proxy targets
2026-06-14 10:47:49 -04:00
Dustin Healy
e0f715bd24
🔒 fix: Scan All Message Roles in messageFilter.pii (#13677)
A Codex security finding flagged that findPiiMatchInMessages was
gating on msg.role === 'user' and silently skipping every other
role. The OpenAI-compatible validator accepts system, assistant, and
tool from the caller; the Responses input conversion accepts and
converts developer and system. All of those roles flow into
formatAgentMessages and then createRun, so an authenticated remote
agent caller could place a credential-shaped value in any non-user
role and reach the model despite the configured filter.

Drops the role gate. The helper now scans every caller-supplied
message regardless of role; the loop count is unchanged (one outer
over messages, one inner over content parts) and the early-exit on
first match still holds. Spec adds explicit cases for system,
assistant, and tool roles in place of the now-incorrect
skips-non-user assertion.
2026-06-11 09:53:25 -04:00
Dustin Healy
5867f1a065
🛡️ feat: Configurable Message PII Filter (#13602)
* 🛡️ feat: Reject chat messages matching configured credential patterns

Adds an opt-in `messagePiiFilter` middleware mounted on the agent
chat route ahead of `moderateText`. When the configured patterns
match the user's input the request is refused with 400, so the
credential never reaches OpenAI moderation, the model, or MongoDB.
Three starter patterns ship by default and operators can subset
them or add their own regex via `customPatterns` in librechat.yaml.

* 🧪 test: Memoize compiled patterns + add middleware spec

Memoize the compiled pattern array via a WeakMap keyed by the
messagePiiFilter config object so repeat requests against the same
config skip the per-request RegExp construction. Cache entries are
released automatically when the config object itself rotates.

Adds packages/api/src/middleware/messagePiiFilter.spec.ts covering
the default-starter rejections, the starterPatterns subset and
empty-array semantics, customPatterns matching layered on top of and
in place of the starters, the no-config and empty-text pass-through
paths, and a memoization regression check.

* 🛡️ fix: Skip invalid customPattern regexes instead of crashing the request

Admin DB overrides for `messagePiiFilter.customPatterns` reach
`req.config` via `mergeConfigOverrides`, which deep-merges raw
override values without re-running `configSchema`. A typo'd regex
like `(` would slip past the YAML-load validation and throw inside
`new RegExp(...)` during `compile()`, returning 500 for every chat
request until the operator rolled the override back.

Wrapped the per-pattern compile in a try/catch that logs the
invalid pattern id + reason and skips it, so other valid patterns
(starters and other custom entries) keep filtering. Added a
regression test alongside the existing spec.

* 🛡️ feat: Extend PII filter to OpenAI-compatible and Responses agent APIs

The chat-route middleware operates on `req.body.text`, but the remote
agent API endpoints (`/api/agents/v1/chat/completions`,
`/api/agents/v1/responses`) accept the same prompt content as a
`messages` array or an `input` field. A caller using their API key
could send a credential-shaped value through either route and bypass
the configured PII filter even though they share the same agent and
model backbone the middleware is meant to guard.

Factored out `findPiiMatchInMessages`, a tolerant walker that handles
both `content: string` and `content: ContentPart[]` user-message
shapes against the same compiled, cached pattern list. Wired it into
the OpenAI-compat controller after agent lookup and into the
Responses controller right after `convertToInternalMessages`. Each
returns the endpoint's native 400 error shape
(`sendErrorResponse` / `sendResponsesErrorResponse`) with the
`message_pii_filter_block` code when a user message matches.

* 🩹 test: Add findPiiMatchInMessages to OpenAI + Responses controller mocks

The OpenAI-compat and Responses controller specs mock `@librechat/api`
with a hand-listed object. The new `findPiiMatchInMessages` export
wired into both controllers in 3ea35af9a was missing from those
mocks, so the production lookup returned undefined and the controllers
threw at request time under jest. Added the missing entries (default
mock: returns null so the handlers fall through to the existing happy
paths). All 278 agents-controller tests pass locally.

* 🧹 refactor: Namespace messagePiiFilter under messageFilter.pii + fix import order

Renames the yaml field `messagePiiFilter` to `messageFilter.pii`, the
module to `messageFilterPii`, the factory to `createMessageFilterPii`,
the type to `MessageFilterPiiConfig`, and the error code to
`message_filter_pii_block`. The wrapper `messageFilter` namespace
gives future safety filters (e.g. `messageFilter.toxicity`) a place
to plug in without restructuring the config later. The
`findPiiMatchInMessages` helper kept its name because it already
describes what it does at the value level.

Also fixes import order Danny flagged on the OpenAI-compatible and
Responses controllers: `findPiiMatchInMessages` was appended at the
bottom of two `require('@librechat/api')` destructures rather than
placed in the length-sorted slot the house style expects.

* 🧹 chore: Length-sort the general require destructure in responses.js

Reorders the general sub-group inside the `require('@librechat/api')`
destructure shortest to longest so the whole block conforms to the
length-sort rule the file's `// Responses API` sub-group already
follows. Pure reorder, no other changes.

* 🧹 chore: Length-sort the defaultConfig block in AppService

Reorders the `defaultConfig` keys in `packages/data-schemas/src/app/service.ts`
shortest-line to longest-line, with the explicit-value entries
(`mcpConfig`, `fileStrategies`, `cloudfront`) trailing the shorthand
ones. Pure reorder, no behavior change.
2026-06-10 09:03:05 -04:00
Danny Avila
6bc75d24c8
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown

Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the
@librechat/api package build, mirroring the merged data-schemas migration.

- Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps,
  bundle first-party `~/` + relative imports)
- Annotate exports for isolatedDeclarations (codefix-driven). Collapse the
  tokens.ts model->token maps to Record<string, Record<string, number>> and
  switch validation.ts's runtime `files` field from z.any() to z.unknown()
  so no explicit `any` is introduced
- Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output
- Add src/telemetry.ts entry shim so the two index.ts entries don't collide
  in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts})
- Delete rollup.config.js

Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass,
both entries load via require(), legacy /api consumes them unchanged.

* 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys

The build-api cache keys hashed `packages/api/server-rollup.config.js`,
which never existed (api used `rollup.config.js`, now removed) — a copy-paste
artifact from the data-provider key that matched no file. Replace it with the
new `packages/api/tsdown.config.mjs` so edits to the build config (entry,
format, externals) bust the api build cache, matching the data-schemas key.
2026-06-08 10:54:48 -04:00
Danny Avila
192703e041
⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations (#13578)
* ⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations

Replace Rollup with tsdown (rolldown + oxc) for @librechat/data-schemas. With the source made isolatedDeclarations-clean, oxc emits .d.ts without tsc, dropping the package build from ~5.8s to ~0.8s (~7x).

- Annotate exported model/method factories for isolatedDeclarations (TypeScript's fixMissingTypeAnnotationOnExports codefix plus hand-authored interfaces); type the ~44 mongoose `any`s and add an explicit PromptMethods interface (previously its declaration was silently dropped by the Rollup build).
- Repoint package.json exports/main/module/types to tsdown output; drop rollup config.
- Config lives in tsdown.config.mjs (native ESM) so CI without a TS-config loader can build it; bundle `dotenv` so the package stays self-contained for its env-loading side effect.
- Fix a latent token `metadata` mismatch the accurate types surfaced: widen TokenCreate/UpdateData inputs to accept plain objects, flatten OAuthMetadata at the api boundary.
- Update mongoMeili/aclEntry specs to the precise model types; drop redundant terser minification from data-provider's library build.

All data-schemas tests pass; api builds clean against the new output.

* 🔧 chore: Hash tsdown.config.mjs in data-schemas CI build-cache keys

The data-schemas build switched from rollup to tsdown, but the build-data-schemas / build-api cache keys in backend-review, config-review, and playwright-mock still hashed the (now-deleted) rollup.config.js. Hash tsdown.config.mjs instead so a config-only change invalidates the cached dist/api builds. (Found by Codex review.)

* 🔧 chore: Replace deprecated tsdown `external` with `deps.neverBundle`

tsdown 0.22 deprecated the top-level `external` option in favor of `deps.neverBundle`. Migrate the data-schemas config and set `deps.onlyBundle: false` to silence the (intentional) dotenv bundling hint. Build output and externalization are unchanged — dotenv bundled, all peers external.
2026-06-07 21:40:48 -04:00