Commit graph

207 commits

Author SHA1 Message Date
lia-by-librechat[bot]
42219ec8e9
🚜 feat: Support Native PostgreSQL Tool Favorites (#18032)
* 🚜 feat: Support Native PostgreSQL Tool Favorites

* fix: Guard PostgreSQL Reconciliation and Connection Leases

* fix: Scope Favorites Erasure Fences to Deletion Attempts

* fix: Preserve Account Erasure Ownership Until Cancellation Completes

* fix: Reject Unsupported PostgreSQL Command Flags Before Connecting

* fix: Keep Command Validation Compatible With the TypeScript Target

* fix: Keep Migration Previews Read-Only and Preserve Erasure Recovery Identity

* fix: Fence Erasure Recovery Across Tenant and Ownership Changes

* fix: Isolate Migration Dependencies and Keep Mongo Scans Outside Transactions

* fix: Inject PostgreSQL Backend Factories at Composition Roots

* ci: Shard Complete Mutation Work Without Relaxing the Aggregate Gate

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-11-11 12:07:07 -05:00
Dustin Healy
8596615d1d
🌐 feat: Wire Responses and Channel Lifecycle APIs (#17611)
* 🌐 feat: Wire Responses and Channel Lifecycle APIs

* fix: Align Channel Application Policy and Source Authority

* fix: Preserve Configured App Budgets in Canonical Response Commits

* Expose Fenced Channel Receipt Reconciliation Routes

* fix: Keep Ordinary Chat Capture In Its Existing Owner

* fix(channels): scope worker authority and defer browser activation

* fix(responses): gate admission on canonical queue readiness

* fix: Recover Durable Response Dispatch Across Queue Shutdown

* fix: Preserve Upstream Endpoint Config Insertions

* test: Align Channel Runtime Configuration Contract
2026-11-10 21:04:03 -08:00
Danny Avila
9f617fc4ad
🧬 feat: Re-Derive Azure Groups, Vertex AI, Custom Parameters and Spec Defaults for Config Profiles (#17399)
* 🧬 feat: Re-Derive Azure Groups, Vertex AI, Custom Parameters and Spec Defaults for Config Profiles

AppService derives part of the served config from librechat.yaml at startup, and config
profiles merged into that output without deriving it again. Azure `groups` now merge by `group`
like `endpoints.custom` by `name`: a profile's partial group merges over the group of that
name, groups it does not name stay, indexed writes are refused, change batches take entry ops
on them, and groups that do not resolve over librechat.yaml and Base (or whose written `${VAR}`
is not set) answer `invalid_azure_groups`. Each merged group keeps the key below it unless the
profile supplies one, so the rebinding guard drops a key a partial item moves and a key a lower
layer dropped stays dropped; Azure re-entry compares groups merged the same way.

With `CONFIG_REDERIVE_ENDPOINTS=true` (opt-in for this release), a profile's Azure `groups`
resolve into what the Azure clients read (unconditional before, now behind the switch so stored
group overrides do not go live on upgrade), a profile's `endpoints.anthropic.vertex` derives
`vertexConfig` and the Anthropic models it serves where librechat.yaml configures Vertex AI, and
the Azure Assistants endpoint follows a profile's `assistants`. Startup logs the documents that
hold such overrides, the groups that would not resolve over librechat.yaml and Base, and the
Vertex AI overrides that never apply. Always on: Vertex AI's key file and region are base-only,
a profile may turn Vertex AI on only where librechat.yaml configures it, its settings may
reference only librechat.yaml's own Vertex AI variables (reads return them resolved), a
profile's `vertexConfig` is never applied and writes skip it (`derived_path`,
`rules.derivedPaths`), a model's own `deploymentName` resolves its `${VAR}` like the default one,
the region, project ID and the deployment name each model's requests send (the model name where
the client sends it as written) must stay in their URL host label or path segment, a `${VAR}`
there is a whole `${NAME}` reference only, the Anthropic client refuses any other deployment name
before building a request, and an invalid setup answers `invalid_vertex_config`. The rebinding
guard no longer binds the derived Vertex AI region, which is librechat.yaml's.

Custom endpoint `customParams` are expanded by `expandCustomEndpoint` into the served copy, so
`appConfig.config` stays librechat.yaml as written, and each endpoint a profile writes is
expanded again from its merged form (a changed `defaultParamsEndpoint`, OpenRouter defaults
for a profile's own endpoint); invalid definitions answer `invalid_param_definitions`. After
tenant scoping, `getAppConfig` derives `modelSelect`, `parameters` and `presets` again for the
specs a principal is served where nothing sets them, the agents `memory` capability follows
the merged memory config, and a model spec naming an unconfigured endpoint answers
`unknown_spec_endpoint`. Meta flags: `azureGroupMergeKey`, `customParamsExpansion`,
`specInterfaceRederive` and `endpointRederive`.

* 🪜 fix: Judge Re-Derived Config on What Every Profile Layer Leaves

Two re-derivations decided whether a profile had set a field from whether any layer ever set
it, so a lower profile's value tombstoned by a higher one still counted as set.

- Vertex AI (CONFIG_REDERIVE_ENDPOINTS=true): the marker that keeps a profile's
  `endpoints.anthropic.models` over Vertex AI's is cleared by a later tombstone of those
  models (or of `endpoints.anthropic` or `endpoints`), and such a tombstone derives Vertex AI
  again, so the Anthropic endpoint serves Vertex AI's models instead of none. A tombstone of
  librechat.yaml's own models does the same.
- Spec-dependent controls: `modelSelect`, `parameters` and `presets` are derived again where
  their served value differs from the base config's, not only where the librechat.yaml-shaped
  value does, so a control one profile sets and a higher one tombstones, or a tombstone of one
  librechat.yaml leaves unset, follows the specs served instead of being dropped.

The memory capability, Azure Assistants, Azure groups and custom parameters already derive
from the merged result; tests now cover each set in one layer and tombstoned in a higher one,
and the reverse.

* 🧮 fix: Derive Each Config Part From What the Layers Leave

A tombstone of an object above a derived part's inputs, alone or followed by a layer that
recreates part of that object, left the part without what startup derives it to:

- An Azure endpoint recreated after a tombstone of `endpoints.azureOpenAI` or `endpoints` had
  no groupMap, modelGroupMap or model lists, so its model lists fell back to the environment's
  default Azure models, which requests could not map to a group. It now serves no group, with or
  without CONFIG_REDERIVE_ENDPOINTS (completeAzureResolution).
- A tombstone of the whole `interface` left modelSelect, parameters and presets out, which the
  client reads as off. They are derived from the specs served into a new interface.
- A tombstone of `endpoints.agents` or `endpoints` left no agents endpoint, while a tombstone of
  its capabilities served the defaults. The agents endpoint startup derives is served there.
- A tombstone of a Vertex AI setting that leaves the setup unable to load kept the vertexConfig
  derived below it. Vertex AI is off there, with a warning.

A table in service.spec.ts runs every derived part against every way a layer changes its inputs
(set, unset, a tombstone of the path and of each object above it, a layer recreating after such
a tombstone, Base on either side, tenant scoping, the switch off) through getAppConfig, and
compares each cell with what startup derives from the librechat.yaml the layers leave.

* 🧭 fix: Serve Only Model Specs Whose Endpoint the Layers Leave

A tombstone of `endpoints.custom` or `endpoints` in a higher profile (or the Base profile)
left the model specs of the custom endpoints it removes served, librechat.yaml's or a lower
profile's: the spec list pointed at endpoints /api/endpoints no longer offers, with the model
selector, parameters and presets still off for them. The write check only ran when the write
held `modelSpecs.list`, and the merge served stored specs as they were.

After the last layer, the merge now serves only the specs whose endpoint is a built-in endpoint
or a custom endpoint the YAML-shaped config the layers leave defines, the rule startup skips
librechat.yaml's specs by (deriveServedSpecs). A removed endpoint hides its specs until a layer
defines it again; where no spec is left, none is served (as with a tenant-hidden endpoint) and
the spec-dependent controls come back on. Tombstones of a built-in endpoint's settings keep its
specs, as startup does, and `addedEndpoints` stays as merged. A stored spec for an endpoint that
is not configured is left out on its own instead of being served; writes of one still answer
400 unknown_spec_endpoint.

The derived-part table in service.spec.ts gains the model specs served (with addedEndpoints and
the controls they decide) against tombstones of `endpoints.custom`, `endpoints`, and each of
`endpoints.openAI`, `azureOpenAI` and `anthropic`: by a profile, by Base under a profile, a lower
profile under a higher one, recreated above or by the same layer, with no spec left, and with
the switch off.

* 🧩 fix: Reconcile Re-Derived Config With Dev's Keyed Tombstones, Write Report and Read APIs

- Keyed tombstones of Azure groups (endpoints.azureOpenAI.groups["g"], now merged by key)
  resolve the groups left again where the merge resolves them, and stay inert without
  CONFIG_REDERIVE_ENDPOINTS.
- Keyed tombstones count as touching endpoints.custom, so served specs and custom parameters
  are derived again after one.
- The served-spec derivation keeps modelSpecs while added endpoints remain, as the hidden
  endpoint scoping does; hidden endpoint names leave addedEndpoints.
- Derived paths join the held paths every write skips, built on first use.
- Tests follow the merged behaviour: Azure groups merge by group, the agents endpoint is
  derived again below a tombstone.
2026-10-24 11:46:01 +02:00
Danny Avila
2591a15c10
🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract (#17322)
* 🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract

Keep dotted record keys in admin config write bodies behind a path-scoped
sanitizer, reject $-prefixed and prototype keys after auth, report dotted_key
issues with segments, and publish GET /api/admin/config/meta (features, rules,
limits, versions) so clients can gate on backend capabilities.

* 🧪 test: Expect Issue Segments in the Config Override e2e Scenario

Override validation issues now carry `segments`; assert the full issue shape.
2026-10-22 09:59:08 +02:00
Marco Beretta
41d53a2989
🩺 fix: Report Not Ready on /readyz While Redis Is Disconnected (#17232)
* fix: Report Not Ready on /readyz While Redis Is Disconnected

/readyz only reflected the one-shot startup flag, so a pod whose Redis
connection dropped after startup kept answering 200 while sign-in failed.
It now also returns 503 when USE_REDIS is on and either shared client
(ioredis or the Keyv node-redis client) is not connected. /livez and
/health stay unconditional so a Redis blip cannot cause a restart loop.

Related to #16814

* test: Cover Readiness Probes Across a Redis Outage

* fix: Inject Redis Clients Into Readiness and Check Every Cluster Master
2026-10-18 21:40:00 +02:00
Dustin Healy
73847f2db5
🔌 feat: Integrate Durable Skill Lifecycle Across Writers and Routes (#17043)
* feat: integrate durable Skill lifecycle across writers and routes

* 🧪 test: Cover Skill Management Deletion End to End

* fix: Queue superseded Skill blobs for durable cleanup

* fix: Reject Skill file storage moves without new identity

* 🧪 test: Cover Skill UI Delete and Cleanup Restart

* 🧪 test: Assert Durable Cleanup for Replaced Skill Files

* fix: release Skill names during account deletion marking

* fix: use reserved Skill blob plans across writers

* fix: hold superseded Skill blobs until sync commits

* test: preserve account deletion retry coverage after restack

* test: verify durable Skill cleanup after account erasure

* fix: complete Skill cleanup leases and avoid historical metadata scans

* fix: bound Skill cleanup manifest processing
2026-10-16 09:05:05 -07:00
Marco Beretta
ccae5b988d
📜 feat: Audit Admin Role, Group, Config and IdP Role Mutations (#17052)
* feat: Extend the Audit Action Vocabulary to Admin, Sync and Credential Mutations

* feat: Audit Admin Role, Group and Config Mutations

Role deletion now removes the role's grants, ACL entries and config
overrides before the role itself and returns ROLE_CLEANUP_FAILED instead
of success when any of them fails, so a recreated role cannot inherit
leftover privileges.

* feat: Audit IdP and Startup Role Changes

* fix: Keep the Startup Role Sync Running When Its Audit Verification Read Fails

* test: Cover Admin Role Audit Trail End to End

* fix: Audit a Remote Agent Role Sync Only After It Persists

* fix: Restore Role Dependents When Role Deletion Fails Midway

* fix: Restore Role Dependents Only When the Role Still Exists

* style: Sort Role Admin Imports

* fix: Keep the ACL Snapshot When Role Cleanup Fails Ambiguously

* test: Read Role Audit Entries From the Audit Log Route

* fix: Audit Only Real Membership Changes and Restore Deleted Role Config in Full

* fix: Restore Deleted Role Grants in Full and Sanitize Audit Write Logging

* fix: Audit and Report a Role Deletion Confirmed After a Lost Response

* fix: Never Restore Dependents Onto a Recreated Role

* fix: Snapshot Role Dependents Before Deleting and Detect Membership Changes Atomically

* fix: Restore From the Data a Cascade Step Actually Removed

* fix: Restore Only Config Overrides When a Role Delete Fails

* fix: Log Only Safe Metadata When the Startup Audit Read Fails

* fix: Audit Grants Removed by a Failed Role Delete

* fix: Audit Removed Grants on Unverifiable Role Delete Exits

* fix: Honor Fail-Closed Auditing in the Admin Config, Group and Role Routes

* fix: Bind Role Deletion to the Captured Id and Audit Only Atomic Role and Config Changes

* fix: Skip the Config Restore When the Targeted Role Is Gone

* fix: Audit Initial Group Members and Skip No-Op Role Description Updates

* fix: Record Role Deletion as Pending Before Cleanup and Append Its Outcome

* fix: Audit Reassigned Role Members and Return the Persisted Config After a Tracked Toggle

* fix: Keep a Committed Role Deletion Successful When Cascade Audit Writes Fail

* fix: Decide Group Update Audits From the Atomic Write and Sanitize Permission Update Logs

* fix: Audit Grants From a Snapshot, Fail on Member Lookup Errors and Chunk Member Audit Entries

* fix: Audit Group Deletion Members and Previous Role Removal, Abort on Grant Snapshot Failure

* fix: Invalidate Config Caches After Partial Secret Writes

* fix: Retry Failed Cascade Restores, Report Changed Group Fields and Audit Every Reassigned Member

* fix: Log Safe Metadata When Cache Invalidation Fails After a Partial Write

* fix: Keep Error Messages Out of the Audit Write Failure Log

* style: Sort Audit Log Spec Imports

* fix: Sanitize Admin Handler Error Logs and Dispatch System Audit Writes Without Waiting
2026-10-14 23:57:07 +02:00
Marco Beretta
20d2c0d43d
🧹 fix: Run the Full Erasure Cascade When an Admin Deletes a User (#17050)
* fix: Run the Erasure Cascade in Admin User Deletion

* refactor: Share Generation Stop Logic Between Self-Service and Admin Deletion

* refactor: Move Admin Erasure Dependency Assembly Into packages/api

* fix: Bind getCleanupJob When Waiting for Generation Persistence

* test: Pin the Deletion Fence Before Admin Generation Stop

* fix: Resolve the Erasure Target Config Fail Closed

* fix: Carry the Target Profile Into Admin Erasure Storage Cleanup

* fix: Keep the Erasure Target Contract Storage Neutral and Reuse the Loaded Identity

* fix: Refuse New Generations Once Account Deletion Has Begun

* fix: Enforce Account Deletion Admission Once in createJob and Narrow the Erasure File Type

* fix: Install the Admission Guard in Every Entrypoint and Purge Refused Jobs

* fix: Guarantee Refused Job Purge and Sanitize Admission Failures

* fix: Route Remote Admission Through createJob and Keep Refusal Cleanup Sanitized

* fix: Log Admission Cleanup Failures With Safe Error Metadata

* refactor: Check Account Deletion Before Creating a Generation and Drain Twice

* refactor: Order Admission Against the Deletion Marker Without a Grace Period

* fix: Sanitize Post-Commit Admission Retirement Failures

* fix: Wire Shared-Server MCP OAuth Cleanup Into Admin Erasure

* fix: Scope Admin Erasure Lookups to the Target Tenant and Await Shared-Link Cleanup

* fix: Resolve the Erasure Target Config Under the Target Tenant

* fix: Run Admin Erasure OAuth Fencing in the Target Tenant

* fix: Match the Split OAuth Invalidate and Disconnect Hooks in Admin Erasure

* refactor: Inject the Tenant Executor Into the Erasure Resolver
2026-10-14 23:54:32 +02:00
Marco Beretta
219a48b55d
⏳ fix: Expire Dead-Lettered Trigger Deliveries and Clean Up Orphaned Image Uploads (#17049)
* fix: Expire Dead-Lettered Trigger Deliveries After a Review Window

* fix: Delete Stored Image Uploads When Record Creation Fails

* fix: Clean Up Stored Objects on Every Storage-Before-Record Upload Path

* refactor: Resolve the Delete Strategy Lazily in Stored Upload Cleanup

* test: Cover Dead Letter Expiry After the Review Window

* fix: Carry Owner to Stored Upload Cleanup and Keep Batch Dead Letter Deadlines

* refactor: Select the Stored Upload Cleanup Target in the API Package

* fix: Check for a Committed Record Before Deleting an Upload and Keep Dead Letters Unless Configured

* fix: Check the Persisted Record Id for Converted Assistants Images

* fix: Arm a Dead Batch Root Deadline After Its Members Settle

* fix: Settle Expired Dead Batches and Skip the Record Guard Before the File Write

* fix: Skip the Record Guard for Pre-Write Failures and Build the Lookup in packages/api

* fix: Build the Record Lookup Lazily in the Upload Service

* fix: Pass the Dead-Letter Retention Window Through Experimental Startup

* fix: Carry the Vector Copy Into Failed-Upload Cleanup

* fix: Cover the Code Environment and Indeterminate Vector Copies in Upload Cleanup

* fix: Give URL-Upload Cleanup a Request When the Caller Omits One

* fix: Route Failed-Upload Cleanup Through Secondary-Storage Deletion

* fix: Attempt Every Copy Independently When Cleaning Up a Failed Upload

* fix: Track the Record Write in Every Upload Path Whose Retention Lookup Can Fail
2026-10-14 17:58:56 +02:00
Marco Beretta
37ec4b56c9
🎨 feat: Paint the Operator Theme on a First-Ever Visit (#16962)
* feat: Paint the Operator Theme on a First-Ever Visit

The server embeds the deployment's base interface.theme in the HTML shell, with both modes the build resolved for a bundled name, so the boot script and the first commit of DeploymentTheme paint it before /api/config answers. A tab's owned cache still outranks it, shared links skip it, and a request scoped by a trusted X-Tenant-Id header gets the shell without it.

* test: Type the Inline Shell Theme Fixture as Raw JSON

* test: Tag the First-Visit Scenarios Literally

* fix: Ignore a Theme Boot File That Is Not an Object

* fix: Keep the Base Theme for a Tenant Header the Config Rejects

* fix: Report a Theme Boot File That Exists but Cannot Be Read

* fix: Embed the Base Config Theme in the Experimental Shell

* test: Stand In the Shell Theme for an Owned Copy Whose Session Ended
2026-10-14 13:16:44 +02:00
Marco Beretta
5cb396e85c
👋 feat: Optional Greeting Heading and Footer-Free Welcome Screen (#17055) 2026-10-14 11:10:08 +02:00
Marco Beretta
e216f08289
🌗 feat: Hand the Colour Mode Over on an Entry Link (#17056) 2026-10-13 12:27:28 +02:00
Marco Beretta
0bc344fcb9
🔑 feat: Let Users Connect Their Own Account for Pull Request Lookups (#16989)
* feat: Let Users Connect Their Own Account for Pull Request Lookups

* fix: Refuse Cross-Tenant Pull Request Callbacks and Surface Key Failures

* fix: Keep Pull Request Grants Through Outages and Remove Them on Disconnect

A token endpoint server error is an outage even with an OAuth error body, so
it no longer deletes the grant, and a refresh outage reaches the lookup as
unavailable instead of falling back to no token. Disconnect removes every
pr_oauth row of the user, whatever app or host is configured now. A grant that
never expires extends its retention when used, and the flow, request and
retention limits are configurable. The connection routes are built in
packages/api, mounted behind the pre-auth tenant middleware in both servers,
and answer failures with stable codes.

* fix: Keep the Pull Request OAuth App Out of Overrides

A role, group, user or tenant override can no longer change or remove pullRequests.oauth, so the connect capability and the callback see the operator's app.

* fix: Read a User's Pull Request Token Only With the Feature On

* fix: Report a Malformed Stored Pull Request Grant as a Failure

* fix: Remove a User's Stored Tokens When an Administrator Deletes the User

* fix: Answer No Pull Request for a Conversation Without a Lane When the User Token Fails

* fix: Share One Grant Read Between Concurrent Callers and Report an Expired Grant Disconnected

* fix: Remove a Pull Request Grant Only on a Definite Refusal and Past Its Retention

* fix: Version Pull Request Grants and Coordinate Refresh, Refusal and Retention Through Them

* fix: Invalidate a Refused User Token and Read It Only for a Lane That Needs It

* fix: Finish a Pull Request Connection Under the User's Effective Config Behind the Login Limiter

* fix: Type the Pull Request User Token for Isolated Declarations and Settle Retention Before Answering

* fix: Bind a Pull Request Connection Flow to the Generation a Disconnect Replaces

* fix: Start a Batch's User Token Read Only Within Its Deadline

* fix: Fence Pull Request Token Reads Against a Disconnect and Report a Failed Retention Cleanup

* fix: Fence Connection Status Like Token Reads and Keep a Newer Read Joinable

* fix: Report a Pull Request Grant Without Its App and Name the Sign-In That Made It

* fix: Read a User's Unexpired Pull Request Grants Newest First

* fix: Mark the Connection Callback as an OAuth Navigation and Report a Failed Extension

* fix: Name the Pull Request Sign-In Flow on Its Landing Page

* fix: Remove a Pull Request Grant of a Replaced App When It Is Read

* fix: Keep a Replaced App's Grant Untouched and Extend Retention From Each Day's Use

* fix: Accept a Pull Request Grant Only Under the User's Current Connection Generation

* fix: Renew the Connection Generation With a Grant Still in Use

* fix: Restore the Previous Connection When a Sign-In Cannot Store Its Grant

* fix: Decide a Sign-In by What Was Stored When a Write Does Not Answer

* fix: Roll Back a Sign-In's Generation Along With a Grant It Must Remove

* fix: Refresh an Access Token a Lookup Was Refused With Before Ending the Connection

* fix: Undo a Sign-In Only for a Confirmed Fence, Not a Failed Check

* fix: Read a User Token Again When the Connection Changes Under the Read

* fix: Apply the Current Retention Again When Another Replica Changed the Grant First

* test: Answer Each Concurrent Sign-In by Its Own Code

* fix: Skip Retention Rewrites for Grants That Expire Without a Refresh Token

* fix: Retry a Token Rejection That Loses to a Concurrent Grant Write

* fix: Skip User Token Reads for Lanes Without a Commit and Reread Status After a Reconnect
2026-10-12 15:58:23 +02:00
Danny Avila
741978c954
🔭 feat: Recover Stale Chunks Without the Error Page and Ship Client Logs to RUM (#16701)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* 🔭 feat: Recover Stale Chunks Without the Error Page and Ship Client Logs to RUM

Stale chunks: a tab left open across a deploy that opens a code-split panel
requests a hashed chunk the new deploy removed. Vite's preload handler hands
the failure to the stale-asset recovery, which reloads, but the import then
resolves to `undefined` and `React.lazy` throws a TypeError into the route
error boundary, so the error page flashes before the reload. Lazy imports now
go through `lazyWithRecovery`/`importWithRecovery`, which turn that result into
a `ChunkLoadError`; the route boundary recognizes chunk failures (and any error
while a recovery reload is underway), asks the page recovery once, and renders
a localized "Updating to the latest version…" status. When the loop guard
declines, the normal error page with its reload action shows instead. The
page recovery is now idempotent within a page, so concurrent callers join the
in-flight reload rather than tripping the guard.

Client logs: with RUM in proxy mode, the client logger's warn/error calls,
error-boundary catches and stale-asset recovery events are exported as OTLP
log records through the authenticated `/api/rum/v1/logs` proxy. Records carry
allowlisted, scrubbed fields only, follow RUM session sampling, dedupe repeats
with a count, are budgeted per minute and per page, batch with backoff, flush
with keepalive on page hide, and stop after persistent failures. The proxy
now refuses non-OTLP content types and applies a per-user rate limit.
`RUM_CLIENT_LOGS=false` turns the export off.

* 🔒 fix: Keep Client Logs Opt-In and Drop Application Error Messages

- RUM_CLIENT_LOGS now defaults to off, like console capture, so enabling
  RUM proxy mode never starts a new log stream on its own.
- Error messages are exported only for engine, browser and library
  generated types (TypeError, SyntaxError, DOMException, AxiosError,
  chunk failures); application errors keep type, status and stack only,
  since their messages can echo prompt or response text. All quoted
  fragments are now redacted.
- Batches are measured in encoded UTF-8 bytes of the exact request body,
  and page hide drains every queued record that fits the keepalive quota.
- Queued stale-asset events are delivered as soon as the log exporter
  starts, independent of the RUM SDK loading, and marked so they are
  never logged twice.

* 🔒 fix: Export Only Known Generated Error Messages and Ack Queued Events

- Error messages are exported only when the whole text matches a known
  engine, browser or library message shape (property access on
  undefined, "x is not a function", fetch/abort/axios status, chunk and
  CSS preload failures). The error class alone no longer qualifies, since
  any code can throw a TypeError carrying arbitrary text.
- Queued stale-asset events are marked logged only after the collector
  accepts the record carrying them, so an undelivered event replays on the
  next page; in-flight events are skipped by the SDK path.
- A fractional RUM_PROXY_USER_MAX below one falls back to the default
  instead of flooring to a limit of zero.

* 🔒 fix: Hold Unacknowledged Asset Events, Re-send In-Flight Logs on Hide

- The RUM SDK flush no longer drops queued stale-asset events whose client
  log record is still unacknowledged: they stay queued and persisted,
  flagged so their SDK action is not repeated, and leave the queue once
  the collector accepts the record.
- Page hide re-sends the records of an unanswered regular request with
  keepalive; each record belongs to the latest request carrying it, so a
  late response to the original request neither marks nor requeues it.
- Client-log export stops when the authenticated layout unmounts.
- The proxy refuses /v1/logs while client logs, console capture and replay
  are all off, so turning RUM_CLIENT_LOGS off also stops tabs that still
  hold an older startup config.

* 🔒 fix: Export Only Complete Stack Frames, Never the Message Header

Stack reduction now strips the V8 `name: message` header (including every
line of a multi-line message) and keeps only lines with a complete V8,
Gecko or WebKit frame shape that ends in a URL or absolute path with a
line number. A message ending in a source-position-like suffix can no
longer pass as a frame.

* fix: centralize RUM delivery and asset recovery lifecycle

* fix: preserve RUM page lifetime and collector acknowledgements

* fix: retain page transport quotas and recover functional chunks

* fix: scope RUM client sessions to authenticated identity

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-03 23:17:56 -04:00
lia-by-librechat[bot]
f88f3fb2fe
🪇 perf: Coalesce Ready Background Completions (#16650)
* 🪇 perf: Coalesce Ready Background Completions

* fix: Recover Receipt Batches and Gate Rolling Deploys

* fix: Fence Background Batch Handoffs and Cleanup Epochs

* fix: Bind Resumed Cleanup to Its Observed Batch

* fix: Preserve Background Results Through Recovery Handoffs

* fix: Recover Unconfirmed Manual Background Claims

* fix: Fence Automatic Recovery of Manual Handoffs

* fix: Preserve Background Reconciliation and Recovery Retention

* fix: Preserve Replayed Claims and Finish Follower Proofs

* fix: Recover Stale Background Projection Epochs

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 15:03:22 -04:00
lia-by-librechat[bot]
bb0543fd15
📼 fix: Replay Running Subagent Activity (#16541)
* 📼 fix: Replay Running Subagent Activity

* 📼 fix: Preserve Activity Across Rollouts and Reconnects

* 📼 fix: Backfill Missing Foreground Activity

* 📼 fix: Fence Replay and Drain Admitted Activity

* 📼 fix: Close Subagent Replay Review Gaps

* 📼 fix: Resume Activity After Replay Recovery Expires

* 📼 fix: Use Singular Subagent Omission Copy

* 📼 fix: Preserve Public Replay and Displayed Activity

* 📼 fix: Write Expanded Live Activity Before Queuing

* 📼 fix: Track Disjoint Subagent Replay Coverage

* 📼 fix: Preserve Tool Details and Omission Totals

* 📼 fix: Recover Tool Timing During Replay Backfill

* 📼 fix: Reconcile Covered Replay Timing Metadata

* 📼 fix: Restore Message Phases During Replay Backfill

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 13:24:47 -04:00
Marco Beretta
4519fe09ae ⚖️ refactor: Move Policy Links to the Signup Consent (#16138)
(cherry picked from commit 1bfaa5f96f)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16138
2026-10-01 03:46:16 +00:00
Marco Beretta
50ce28957f
🎚️ fix: Apply librechat.yaml Rate Limits to Route Limiters (#16432)
Route modules build their rate limiters while they load, and both server
entries required them before startup checks copied rateLimits from
librechat.yaml into the environment. The conversationsImport, tts, stt
and route-scoped fileUploads budgets therefore ignored the yaml unless
the matching environment variable was also set. Require the routes only
after performStartupChecks in index.js and experimental.js.
2026-09-28 16:15:37 +02:00
Danny Avila
c606e5065a
🛬 fix: Settle Background Tool Results Before the Server Shuts Down (#16365)
* 🛬 fix: Settle Background Tool Results Before the Server Shuts Down

* 🛬 fix: Preserve Background Shutdown Results Through Fallback

* 🧪 test: Match Formatted Cluster Shutdown Wiring

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-26 20:42:47 -04:00
lia-by-librechat[bot]
ffd3d1e139
🔕 perf: Back Off Idle Durable Agent Recovery Polls (#16303)
* 🔕 perf: Back Off Idle Durable Agent Recovery Polls

* 🎛️ style: Format Queued-Turn Recovery Test

* fix: Preserve Durable Recovery Wakeups and Discovery Invariants

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-24 09:36:51 -04:00
Danny Avila
36efc7b4e2
🧺 fix: Bound Background Completion Batches (#16102)
* 🧺 fix: Coalesce Durable Background Wakeups

* fix: Bound replayed background result claims

* fix: Harden background result coalescing controls

* fix: Reconcile background result claims

* fix: close background batch ownership races

* fix: Keep completion batching atomic and bound serialized metadata

* refactor: Reuse canonical background claim contracts

* fix: Expose serialized claim budget in the persistence contract

* test: Preserve claimed V1 results across later background batches
2026-09-19 21:17:32 -04:00
Atef Bellaaj
69f0dd2a12
🔌 feat: OpenAPI spec and docs for the public Agents API (#15928)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
* 🔌 feat: OpenAPI spec and docs for the public Agents API

* 🔧 fix: address review on Agents API OpenAPI spec and docs:

- adapter: mark JSON request bodies as required
- openapi.js: move gate, config read and spec serving into packages/api
  as an injected router factory; leave /api as wiring only
- docs: compute URLs in-browser and use a relative servers url so docs
  work under a base path (e.g. /chat)
- build: copy the generated spec into dist so the multi-stage image ships it
- agents: reuse the enforced list envelope for the docs so the AgentList
  schema cannot drift from the runtime cursor and strictness

* 🔧 fix: align Agents API OpenAPI contract with enforced runtime

Second review round on the public Agents API spec. Three cases where the
published contract described something the server does not do, so generated
clients would fail against the live API:

- Upload form: document the `purpose` field (enum: file_search, execute_code,
  context) that POST /agents/{id}/files actually requires, instead of the
  internal-only `tool_resource`; requests following the old form were rejected
  with 400 because `purpose` was absent.
- Auth: drop the `apiKeyBearer` scheme from the management and skill endpoints.
  They gate on OIDC/M2M via requireAgentManagementAuth and deliberately reject
  the API-key fallback; the API key belongs to the out-of-scope
  OpenAI-compatible endpoints and returns when those are documented.
- Skill update: document the 409 conflict PATCH /skills/{id} returns on a stale
  expectedVersion. The agent update path collapses its internal 409 to 400, so
  it correctly keeps no 409.

* 🔧 fix: serve docs on experimental server and document the 401

Third review round on the public Agents API spec.

- Experimental server: mount routes.openapi in api/server/experimental.js before its /api 404 handler. Without it, npm run backend:experimental returns 404 for /api/docs and /api/openapi.json even when openapi.enabled is true, because that server never mounted the router.
- Auth failure: document the 401 that requireAgentManagementAuth returns on a missing, expired, malformed, or unbound token. Its body is { "error": "Unauthorized" }, a flatter shape than the management error schema, so add a dedicated UnauthorizedError schema and a shared 401 response to the agent and skill contracts rather than reusing the standard error schema.

* 🔧 fix: document skill-update, auth-conflict, upload-limit, and watch-copy gaps

* 🔧 fix: added error shapes for permission denied and internal server errors in API responses docs

* fix: cover the auth-layer 500 shape and pin the account-deletion code

* fix: update OpenApiRouterDeps documentation and simplify request handling in isEnabled function

* 🔧 fix: gate the Swagger asset mount behind the OpenAPI toggle

* 🔧 fix: accept the malformed-JSON 400 envelope in the contract

* fix: accept the tenant-isolation 403 envelope in the contract

* fix: add description for content property in SkillFileUpdateRequest to clarify byte limit
2026-09-16 13:50:05 -04:00
Marco Beretta
8e6e6f062d
🪧 fix: Emit the Configured-Footer Answer With the HTML Shell (#15886)
* 🪧 fix: Emit the Configured-Footer Answer With the HTML Shell

The chat surface lays out against whether the deployment configured a footer
(customFooter, privacy policy, terms of service). The client answered that from
localStorage — whatever the last resolved `/api/config` recorded — so the answer
was a guess on a first-ever visit and could be stale for a deployment whose
configuration changed.

The backend already knows when it serves client/dist/index.html, so it says so
there: `injectConfiguredFooterBootstrap` writes `hasConfiguredFooter` into
`window.__LIBRECHAT_CONFIG__` ahead of the app's own scripts, once at boot from
the base app config, and the client reads it on its first render instead of
remembering the last answer. The `configured-footer` storage atom is gone.

Client and server share one predicate (`hasConfiguredFooter` in
librechat-data-provider) so the two cannot disagree, and the shell injection is
generalized (`injectBootstrapConfig`), with the query-devtools flag now going
through it.

* 🧷 fix: Keep Bootstrap Values Literal and State the Shell's Contract

`injectBootstrapConfig` passed its script as a replacement string, so a `$&`,
`$1` or `$'` inside an injected value was expanded by `String.replace` instead
of reaching the client. Both insertion points take a replacement function now,
and the spec reads its values back off the global through an explicit result
type rather than an unchecked dictionary.

The shell answers from the deployment's own configuration, which a per-tenant,
role or user override of `interface.privacyPolicy` or `termsOfService` can
disagree with — `/api/config` resolves the caller's and the client prefers it.
That contract was overstated in three comments ("a deployment-lifetime fact",
"the same one the config will give"); they now say what the answer is and what
resolves the rest. Two scenarios cover the disagreement end to end: a document
served without the answer, and a policy link only this caller's config carries.
2026-09-13 21:53:54 -04:00
Danny Avila
9cf33cc8be
🧿 fix: Bind Login Flows to the Browser That Started Them (#15880)
* 🧿 fix: Bind Login Flows to the Browser That Started Them

Social OAuth logins (Google, Facebook, GitHub, Discord, Apple) now store a random state in an HttpOnly cookie scoped to the callback path and only exchange the returned code when the callback carries that state. Apple's form_post callback uses a SameSite=None cookie, and passport-apple's own fixed state no longer bypasses the store.

Local login, temp-token 2FA verification and admin local login reject requests a browser sent from another origin (Sec-Fetch-Site, falling back to Origin vs Host), trusting DOMAIN_CLIENT and DOMAIN_SERVER. Requests without browser fetch metadata, such as server-side callers, pass unchanged.

The sessionless OAuth callbacks drop failureMessage, which wrote to req.session and turned any rejected callback into a 500.

* 🧿 fix: Configure Login State Cookies and Localize Origin Rejections

- OAuth state cookies take secureCookie and maxAgeMs from the caller; registration.oauthStateTtlMs in librechat.yaml sets the lifetime (default 10 minutes).
- Secure deployments use a __Host- cookie at Path=/ so a sibling subdomain cannot plant a state.
- The cookie keeps up to three pending states, so logins started in separate tabs all complete, and an unmatched callback no longer discards them.
- passport-apple's preset state is dropped by deferStateToStore in packages/api.
- The Origin fallback compares scheme and host against the request itself.
- The same-origin rejection carries ErrorTypes.AUTH_CROSS_ORIGIN, which the login and 2FA screens show as a localized message.

* 🧿 refactor: Sign Login State Against a Per-Browser Binding

- The OAuth state store keeps a random per-browser binding in a __Host- cookie (reused across flows) and signs each state as issuedAt.nonce.HMAC(JWT_SECRET, provider, binding, issuedAt, nonce). Nothing is stored per flow, so any number of tabs complete, each state expires on its own clock, and concurrent starts no longer overwrite each other.
- The experimental server entrypoint passes its app config to configureSocialLogins.
- The same-origin guard also trusts an explicitly configured ADMIN_PANEL_URL.

* 🧿 fix: Keep Separate Bindings for Racing First Logins

A browser without a login-state binding now gets one under a cookie name with a random suffix, so two tabs that start their first login at once each keep their own binding. Later starts reuse an existing binding, and a callback is accepted when its state was signed for any binding the browser presents.

* 🧿 fix: Sanitize Request-Derived Values in the Cross-Site Rejection Log

The same-origin guard logged the request Origin and path verbatim. A non-browser client fully controls the Origin header, and the console transport uses a printf format that does not escape control characters, so a crafted value could forge log lines. Strip ASCII control characters (CR, LF, NUL and the rest) before logging, matching the pattern apiNotFound already uses.
2026-09-13 11:47:29 -04:00
Danny Avila
f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00
Danny Avila
65bb64bc68
🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784)
* fix: recover CodeAPI uploads after throttling

* fix: cover eager CodeAPI uploads

* fix: scope CodeAPI upload recovery

* test: clarify upload recovery deadlines

* fix: isolate CodeAPI recovery state

* fix: cancel CodeAPI recovery waits

* test: provide CodeAPI registry in handler fixture

* fix: propagate CodeAPI recovery controls

* fix: bound CodeAPI recovery by run

* fix: preserve skill upload cancellation

* test: update deferred loader signal contract

* fix: preserve recovery cancellation

* fix: cancel recovery source reads

* fix: classify Axios upload cancellation

* fix: preserve lazy provisioning cancellation

* fix: close resource recovery boundaries

* fix: preserve provisioning failures compatibly
2026-09-09 15:48:11 -04:00
Danny Avila
ec15075dc8
🚚 feat: Route Attachments and Provision Tools Lazily (#15763) 2026-09-08 23:28:52 -04:00
Ravi Kumar L
000349681e
🩻 feat(insights): Scope Insights to Authorized Agents (#15549)
Some checks failed
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
* feat(insights): add agent-scoped access

* fix(insights): exclude unattributed assistant messages

* fix(sharing): key role insight toggles by principal

* fix(sharing): preserve insight grant snapshot

* fix(permissions): preserve insight bits atomically

* fix(permissions): guard insight audit rollback

* fix(insights): keep admin grant controls accessible

* fix(permissions): reconcile duplicate principal updates

* fix(insights): show automatic admin access

* fix(permissions): harden insights access updates

* fix(insights): simplify authorization and preserve client state
2026-09-04 20:16:58 -04:00
Danny Avila
57c52de33c
🛎️ fix: Start Code Environment Reconciliation After Mongo Connects (#15620) 2026-09-04 19:08:44 -04:00
Danny Avila
d346e5ab5e
🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short (#15532)
* 🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short

* fix: Await generation settlement on shutdown instead of publishing the drain

* fix: Spend the real shutdown budget and cover the initialization window

* fix: Keep the shutdown registration literal the index spec asserts on

* fix: Keep draining settlements registered while shutdown is waiting

* fix: Track provider executions at the begin/drain chokepoints instead of per caller

* fix: Open the provider tracker before the begin CAS and cap cluster workers to the primary deadline

* fix: Propagate the cluster deadline and release trackers on reconfigure and abandonment

* fix: Hold SIGTERM until the worker acknowledges the cluster deadline

* fix: Refuse provider starts after shutdown begins and harden the cluster deadline handoff

* test: Assert the multi-execution shutdown wait with both executions begun before shutdown
2026-09-03 07:11:38 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
a9ccac8656
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing

* fix: satisfy code environment type checks

* fix: secure code environment administration

* fix: isolate code pairing control plane

* fix: validate code pairing control responses

* fix: secure code pairing transport

* fix: validate code pairing wire format

* fix: harden pairing secret lookup
2026-08-30 17:12:20 -04:00
Danny Avila
487984193a
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable

* fix: align detached actor checks with merged base

* fix: close detached actor lifecycle gaps

* fix: break detached action outcome cycle

* fix: deepen detached action ownership

* fix: fence detached launch handoffs

* fix: preserve detached rollout fencing

* test: type detached recovery failure

* fix: close detached ownership handoffs

* fix: stage detached action activation

* fix: persist detached terminal retries

* fix: fence detached durability ownership

* fix: fence trigger lane publication

* fix: fence detached transition generations

* fix: retire failed detached predecessors

* fix: close detached rollout consumers

* test: type detached retry turns

* fix: close detached ownership gaps

* fix: fence detached action resumes

* fix: fence prior-head event actor resumes

* fix: preserve legacy event actor resumes

* fix: align detached resume boundaries
2026-08-28 14:21:04 -04:00
Danny Avila
8ba76507ae
🧷 fix: Verify Scheduled HITL State Before Pausing (#15284)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
* fix: require durable storage for scheduled HITL

* fix: verify scheduled HITL checkpoints before pause

* fix: narrow scheduled HITL capability checks

* fix: honor ask tool filters in schedule preflight

* fix: enforce durability across scheduled HITL resumes

* fix: scope scheduled HITL admission to active tools

* refactor: move scheduled HITL admission to TypeScript

* fix: close scheduled HITL admission gaps

* fix: bind scheduled pauses to current checkpoint

* fix: cap pending actions at binding deadline

* fix: reject expired pending actions

* fix: guard approval deadlines atomically
2026-08-27 11:16:24 -04:00
Danny Avila
de59da9636
🎟️ refactor: Require Credentials for Local Image Access by Default (#15252)
* 🔐 fix: Protect Local Image Access by Default

* 🔐 fix: Scope Image Authorization to Active Sessions

* 🧹 style: Format Image Authorization Checks

* 🛡️ fix: Harden Image Avatar Authorization

* 🧭 style: Sort Image Authorization Imports

* 🔐 fix: Close Image Authorization Review Gaps

* 🧭 fix: Normalize Stored Avatar Base Paths

* 🏢 fix: Resolve Tenant Assistant Image Policy

* 🛂 fix: Enforce Effective Image Access Policy

* 🧹 style: Flatten Assistant Config Selection

* 🧷 fix: Preserve Image Access Compatibility

* 🪪 fix: Make Image Sessions Revocable

* 🏗️ fix: Move Image Session Policy Into API
2026-08-27 09:55:27 -04:00
Danny Avila
44edcbe014
🧾 feat: Store Durable Event Actor Receipts (#15265)
* 🧾 feat: Store durable event actor receipts

* 📊 fix: Scope actor delivery metrics

* 🧱 fix: Close durable receipt recovery gaps

* fix: serialize actor action admission

* fix: retire terminal batch members

* fix: close actor terminal recovery races

* fix: reclaim lanes after terminal receipts

* fix: close actor receipt recovery gaps

* fix: preserve ambiguous legacy handling

* fix: recover legacy actor outcomes

* fix: admit leased actor actions

* fix: serialize actor admission recovery

* fix: close actor recovery races

* fix: token-fence actor admissions

* fix: preserve mixed-version actor fences

* fix: harden actor rollout and metrics

* fix: gate durable actor receipt rollout

* fix: enforce base actor receipt rollout

* fix: align experimental actor rollout
2026-08-27 06:00:43 -04:00
Danny Avila
f34a49007d
🔣 fix: Escape SPA Language Attribute (#15248) 2026-08-26 07:37:36 -04:00
Danny Avila
290b8664d9
🧾 feat: Track Authoritative Agent Event Outcomes (#15213)
* feat: track authoritative agent event outcomes

* fix: isolate agent event outcome types

* fix: declare agent event handler result

* fix: simplify agent event status selection

* fix: preserve authoritative event outcomes

* fix: preserve terminal event evidence

* test: use completed run-step envelope

* test: scope deferred HITL question locator

* fix: settle every agent event terminal path

* style: sort terminal host action imports

* fix: fence agent event terminal evidence

* fix: recover agent event terminal settlement

* fix: scope terminal retry hints by generation

* fix: settle terminal host actions exactly
2026-08-25 18:20:57 -04:00
Danny Avila
877b9b2f1a
🐚 feat: Nonce-Based Content Security Policy for the SPA Shell (#14446)
* 🛡️ feat: Configurable Baseline HTTP Security Headers

Adds helmet's CSP-independent headers (HSTS, X-Frame-Options,
X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response,
with contentSecurityPolicy explicitly disabled. Every header that can
break a deployment is configurable, so there is no allow-list to go
stale the way #7377's hardcoded CSP directives did.

HSTS includeSubDomains defaults off rather than matching helmet's
on-by-default: it would otherwise pin every sibling subdomain to HTTPS
for a year in every visitor's browser, and undoing that requires
serving max-age=0 from each affected host.

* 🛡️ feat: Nonce-Based Content Security Policy for the SPA Shell

Adds an opt-in, per-response nonce CSP on the HTML response, resolved once
at startup so each request only mints a nonce and concatenates the header.
Report-only by default, since that is the rollout step #7377 skipped.

Rebase and correctness pass over #13226:

- Styles carry no nonce. A nonce in style-src makes browsers ignore
  'unsafe-inline', which would have blocked the <style> element the theme
  script injects at runtime, plus every style third-party components inject.
- frame-ancestors 'self' is now a default rather than opt-in, so enabling
  CSP actually covers the clickjacking half of #7110.
- CSP_SCRIPT_SRC_EXTRA now drops 'strict-dynamic', which would otherwise
  make browsers ignore the very hosts the operator configured.
- Nonce stamping runs after the query-devtools bootstrap injection so that
  injected script is covered too.

* fix: replace frame-ancestors instead of merging it

Merging the configured value into the default turned a deliberate
CSP_FRAME_ANCESTORS='none' into `frame-ancestors 'self' 'none'`, which
browsers resolve back to 'self'. Also bail out if the serialized policy
somehow lacks the nonce slot rather than emitting a header the shell
cannot match.

* fix: address Codex review findings on the CSP defaults

All five were real against LibreChat's actual runtime:

- CSP_REPORT_ONLY now only enforces on an explicit false/off/0/no. A typo
  or `1` previously fell through isEnabled() to enforcing, turning a
  config slip into a blocked SPA. Shares the parse helper with
  headers.ts via a new security/env.ts.
- Module preloads are stamped. A production client/dist/index.html
  carries 32 parser-inserted `<link rel="modulepreload">` tags, which
  'strict-dynamic' does not cover and 'self' cannot rescue.
- Stale nonce attributes are replaced rather than preserved; only the
  current response's nonce is authorized.
- worker-src allows data:, which Monaco's default CDN loader needs to
  bootstrap its workers (there is no loader.config() in the client).
- script-src allows 'wasm-unsafe-eval' for the HEIC upload path, which
  compiles WebAssembly through heic-to. Narrower than 'unsafe-eval'.

Verified against the real built shell: 4 scripts and all 32 preloads
nonced, stylesheets/icons/manifest and <style> untouched.

* fix: address second Codex round on CSP rollout controls

- SECURITY_HEADERS=false now disables CSP too. It is documented as the
  global kill switch, and an operator reaching for it to recover a shell
  broken by an enforcing policy must not be left with that policy on.
- The SPA shell is forced to `no-store` while CSP is enabled, ignoring
  INDEX_CACHE_CONTROL/INDEX_PRAGMA/INDEX_EXPIRES and warning when they
  are set. A cacheable shell pins one nonce across page loads and users,
  which is the whole thing a nonce policy defends against.
- Added CSP_ALLOW_WASM and CSP_ALLOW_DATA_WORKERS. The previous commit's
  .env.example claimed CSP_ADDITIONAL_DIRECTIVES could drop
  'wasm-unsafe-eval' and data:, but merging only ever appends sources, so
  the documented hardening step was impossible. These toggles make it real.
2026-08-25 09:18:52 -04:00
Danny Avila
2ef12b1e1d
🦺 feat: Configurable Baseline HTTP Security Headers (#14445)
Adds helmet's CSP-independent headers (HSTS, X-Frame-Options,
X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response,
with contentSecurityPolicy explicitly disabled. Every header that can
break a deployment is configurable, so there is no allow-list to go
stale the way #7377's hardcoded CSP directives did.

HSTS includeSubDomains defaults off rather than matching helmet's
on-by-default: it would otherwise pin every sibling subdomain to HTTPS
for a year in every visitor's browser, and undoing that requires
serving max-age=0 from each affected host.
2026-08-25 08:21:39 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
c7e355b219
🛑 fix: Confirm Scheduled Stops and Separate Terminal Scheduler Failure from Startup (#15051)
* 🛑 fix: Confirm Scheduled Stops and Separate Terminal Scheduler Failure from Startup

Fixes #15042, fixes #15043.

`resume.js` inferred a confirmed stop from the ABSENCE of `failureReason`, but
`abortJob` had four `success: false` paths that returned no reason at all. Those
settled the occurrence as `interrupted` and pruned the checkpoint on aborts that
never landed — including one where a REPLACEMENT generation owned the
conversation, which pruned the successor's checkpoint.

Every `success: false` return now names itself (`job_not_found`,
`already_settled` added alongside the existing `generation_replaced` /
`job_still_active`), and a single canonical `isStopConfirmed` predicate decides
whether durable state may be settled. `already_settled` confirms a stop —
`awaitProviderDrain` has proven the provider segment can no longer persist — so a
permanently terminal generation is not answered with a retry loop.

Separately, a schedule engine that failed to arm advertised its permanent outage
as a transient 503 with `Retry-After`, so a client obeying it would poll forever.
Readiness is now tri-state (`starting` / `armed` / `unavailable`): the retry
contract applies only while arming is genuinely pending, and a failed arm returns
a terminal `SCHEDULES_UNAVAILABLE` with no `Retry-After` and an error-level log.

* 🏷️ fix: Declare Schedule Write Gate Return Types

`--isolatedDeclarations` requires an explicit return type on the exported factory
and on the middleware it returns (TS9007). Adds a named `ScheduleWriteGate` type
matching the existing `ShareMiddleware` shape.
2026-08-20 18:33:51 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00
Ravi Kumar L
006e421cd2
💡 feat: add DB-backed admin insights (#14898)
* feat: add Mongo-backed admin insights

* feat: gate insights with environment variable

* fix: tighten insights access and activity metrics

* fix: preserve insights date selections

* perf: parallelize insights search aggregation

* test: wait for MCP conflict recovery

* test: satisfy strict MCP recovery typing

* fix: disable insights pagination while loading

* fix: localize insights range shortcuts

* fix: bound insights search input
2026-08-18 07:51:51 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Danny Avila
ee8c0abe2d
🪝 feat: Execute Agent Plugin Command Hooks (#14755)
* 🪝 feat: Execute Agent Plugin Command Hooks

Implement the missing PluginHookExecutor boundary so deployment plugins'
ai.librechat/hooks/hooks.json documents execute instead of loading inert:

- Command executor runs handlers as child processes outside the API
  process: Claude-shaped JSON payload on stdin, exit 0 + JSON stdout as
  sanitized hook output, exit 2 blocks with stderr as the reason, minimal
  allowlisted environment plus PLUGIN_ROOT/PLUGIN_DATA, abort-signal kill
- Plugin loading carries the parsed hooks document on the contribution and
  threads hookCapabilities from startup, gated on the operator opt-in
  DEPLOYMENT_PLUGIN_HOOKS (off by default: parsed-but-inert with warning)
- Runs register every ready plugin hook onto the per-run HookRegistry after
  internal policy hooks, with once-per-conversation SessionStart dedup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Harden Plugin Hook Execution Boundary

Address CI and Codex/Copilot review findings on #14755:

- Break the agents -> plugins import cycle: the run seam now reads a
  PluginHookSource wired at startup (mirrors the tool-approval registry)
- Tighten plugin ask decisions to deny unless the run has HITL wiring,
  so an un-resumable interrupt can never strand OpenAI-compatible callers
- Scope cross-run dedup keys by authenticated user and handler identity:
  caller-supplied conversation ids cannot collide across principals, and
  sibling SessionStart handlers all fire; once handlers persist across runs
- Replace a literal NUL byte in source with an escape (file diffed binary)
- Kill the whole detached process group on abort, not just the shell
- Map exit 2 on events without a decision channel to preventContinuation
- Reserve PLUGIN_ROOT/PLUGIN_DATA against allowlist overrides, quote
  PowerShell args, cap captured output by bytes with one-pass decoding,
  and serialize payloads inside the executor's error boundary
- Fix import ordering flagged by the static checks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Close Plugin Hook Policy and Namespace Gaps

Address the second Codex review round on #14755:

- Drop updatedInput from plugin command outputs: hooks in one dispatch all
  receive the original arguments, so a plugin rewrite would reach the tool
  without the approval policy re-evaluating it (host-only now)
- Translate Claude tool aliases (Bash/Write/Edit/Read) to LibreChat runtime
  names in matchers, with reverse payload mapping, so Claude-authored guards
  fire instead of planning ready and never matching
- Key once-only state by declaration position as well as handler contents,
  so sibling declarations with identical handlers stay independent
- Thread sessionStartSource through createRun and mark the HITL resume
  rebuild as 'resume', so SessionStart matchers see the real lifecycle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Translate Regex-Form Claude Tool Aliases

Address the third Codex review round on #14755: alias translation now
substitutes word-bounded tokens, covering regex matchers like ^Bash$ and
^(Write|Edit)$ that the exact-token pass left registered against Claude
names and silently never firing. A regex whose alias sits inside a
character class or escape is rejected as unmapped so it fails loudly at
plan time instead of never running.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Alias Translation and Reuse Load-Time Plans

Address the fourth Codex review round on #14755:

- Add the WebSearch -> web_search alias so Claude-authored web-search
  guards fire against the LibreChat built-in
- Apply alias translation only to tool-name events; a StopFailure matcher
  like ^Bash failed$ stays untouched and keeps matching the error text
- Reuse each plugin's load-time hook plan at run registration instead of
  re-planning up to 512 handlers on every chat turn

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Translate Aliased Tool Inputs and Harden Hook Domains

- Present aliased tool inputs under Claude field names (file_path,
  old_string, new_string, including nested edits), so Write/Edit/Read
  guards see the fields they check instead of silently allowing
- Derive the alias table from canonical tool-name definitions
  (BashExecutionToolDefinition, CREATE_FILE_TOOL_NAME, Tools.web_search)
  instead of a parallel hand-authored table
- Reject matchers naming Claude built-ins with no runtime equivalent
  (Task, Glob, Grep, WebFetch, ...) as unmapped at plan time instead of
  registering guards that never fire
- Replace per-event Sets and Stop special-cases with an exhaustive
  EVENT_TRAITS record over HookEvent, so new engine events demand
  explicit semantics at compile time
- Move cross-run once-state behind a PluginHookOnceStore seam with a
  least-recently-marked memory default: active conversations refresh
  their keys each turn, so capacity eviction can no longer re-fire a
  conversation that is still in use; the seam admits a shared-cache
  store for multi-replica deployments
- Gate portable-only command handlers at plan time on Windows via a new
  supportsHandler capability (commandWindows or shell powershell
  required) instead of spawning bash that cannot exist
- Kill Windows hook process trees with taskkill /t on abort
- Require declaration indices on execution requests, stamped from the
  plan instead of defaulted at execution time

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Keep Group SIGKILL Escalation Armed After Wrapper Exit

An aborted hook whose descendant ignores SIGTERM could leak that
descendant: the wrapper shell's exit fired close, which cancelled the
scheduled group SIGKILL. The escalation timer is now never cancelled —
it is unref'd and killTree already tolerates a vanished process group,
so a redundant late sweep is harmless while a surviving descendant is
reliably killed at the grace deadline. killGraceMs is configurable on
CommandExecutorOptions, with a regression test driving a trap-protected
descendant past the wrapper's exit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Once Retention by Conversation and Reject Clear Source

- Restructure the once store around conversation scopes: registration
  touches the scope every run, so rarely-matching once handlers keep
  their keys while the conversation is active; eviction removes whole
  idle conversations (capacity counts conversations, not keys)
- Reject SessionStart matchers naming the clear lifecycle source at
  plan time — no LibreChat run-construction path emits clear, so the
  handler would plan ready and never fire; wildcard warning text now
  reflects the sources that actually occur
- Make the SIGKILL-escalation regression test real: the surviving
  descendant redirects its stdio away from the captured pipes so the
  wrapper's close fires while it is still alive, exercising the
  window a close-time cancellation would leak

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Bound Alias Tokens by Tool-Name Characters and Host Shells

- Translate Claude aliases (and reject unsupported built-ins) only when
  delimited by characters that cannot appear in a runtime tool name:
  action tool names preserve hyphens, so an alias embedded in a longer
  name like deploy-Bash-v2_action_example_com stays the literal tool
  name instead of being rewritten into a matcher that never fires
- Reject PowerShell-only command handlers on POSIX hosts at plan time
  (and skip them at runtime): bash cannot run PowerShell syntax, so the
  guard would fail open; a handler with both variants still runs its
  portable command
- Handle rejected asynchronous once-store calls: a failed touch logs
  instead of raising an unhandled rejection during run construction,
  and a failed markOnce lookup fails open per the store's documented
  over-fire direction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Probe Group Liveness Before Cancelled or Delivered SIGKILL

The never-cancelled escalation timer could signal a recycled
process-group id when an aborted hook's whole tree exits early in the
grace window. Escalation now probes the group with signal 0: close
cancels the timer only when the group is verifiably empty, and the
deadline re-probes before delivering the group SIGKILL, so surviving
descendants are still reaped while a fully-dead group never receives a
blind late signal. The residual probe-to-signal race is documented as
irreducible without pidfd support.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Gate Windows Escalation on Root-Process Liveness

Windows taskkill /t walks the tree from the root process, so once Node
observes the root's exit an escalation pass can reap nothing and a late
forced taskkill could only hit a recycled PID. The liveness gate is now
platform-aware in one helper: POSIX probes the process group with
signal 0, Windows checks the root's observed exit state, and both the
close-time cancellation and the deadline delivery consult it — no
platform retains a blind late signal. Orphaned SIGTERM-ignoring
descendants on Windows are documented as the platform limitation they
are without Job Objects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Payload Namespace to Declarations and Reap Stray Workers

- Reverse name/input translation now applies only to declarations whose
  matcher actually required Claude-alias translation: the plan records
  requiresToolNameTranslation per entry, so a native-authored matcher
  like ^create_file$ receives native tool names and fields instead of
  Claude-shaped payloads its guard never expected
- Coordinate the two dedup layers via a shouldExecute gate on the
  executor: a declaration suppressed by spent once-state declines
  before claiming the per-input dedup slot, so an identical handler
  under an overlapping matcher can still claim it and fire its own
  independent once-key instead of being permanently shadowed
- Reap process groups that outlive a successful hook: a backgrounded
  worker left running after normal wrapper exit gets the same
  term-then-escalate sequence an abort uses, since unsupported async
  handlers mean no lifecycle owns such processes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🧰 chore: Vendor Pocock Codebase-Design and Architecture Skills

Adds mattpocock/skills engineering/codebase-design and
engineering/improve-codebase-architecture (MIT, license included) under
.claude/skills so future sessions share the deep-module vocabulary
(module, interface, depth, seam, adapter, leverage, locality) and the
architecture-review process. Force-added past the /.claude/ gitignore
deliberately; relocate if project skills should live elsewhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 refactor: Extract Process-Tree Reaping Into a Reaper Module

Tree lifecycle — five of the last seven review findings — lived as
event-handler wiring inside runCommand with its invariants in comments.
It now sits behind a two-method seam: createReaper(child, graceMs)
exposes reap() and onClose(), hiding the term-grace-escalate state
machine, the per-platform liveness gates, the recycled-id guards, and
the clean-exit sweep. The executor shrinks to capture-and-parse, and
the reaper is unit-tested directly with real process trees through its
own interface instead of only via whole-executor integration runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Translation Per Alternative and Sweep at Root Exit

- Track which runtime tool names alias translation produced, so a
  mixed-namespace matcher like Bash|create_file presents Claude-shaped
  payloads only for bash_tool invocations while the natively-authored
  create_file alternative keeps native names and fields; a capability
  omitting the produced-names list keeps declaration-wide translation
- Sweep the process tree at root exit as well as close: a backgrounded
  descendant holding the captured pipes delays close until it dies, so
  the exit-time sweep terminates it promptly instead of stalling the
  hook until its timeout aborts
- Pass the primary agent's resolved model and identity into the plugin
  hook context, so SessionStart payloads carry model and agent_type
  instead of always omitting them

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Default Wildcard Declarations to the Document Namespace

- Matcherless (or wildcard) tool-payload declarations now inherit the
  hook document's Claude namespace: with no alternatives to carry
  namespace evidence, the plan marks them for declaration-wide reverse
  translation, so a wildcard guard inspecting standard Claude names and
  fields sees Write/file_path instead of silently failing open on
  native payloads; PostToolBatch entries translate the same way
- Recognize aliases delimited by regex metacharacters: dots leave the
  tool-name boundary class (runtime names never contain them — action
  ids underscore domain dots), so ^Bash.*$ translates to ^bash_tool.*$
  instead of registering a guard that never fires
- Expand Claude's ${CLAUDE_PLUGIN_ROOT} spelling in hook commands and
  export it in the child environment alongside PLUGIN_ROOT
- Scope SessionStart once-keys by lifecycle source, so a startup firing
  no longer suppresses the conversation's resume rebuild

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Normalize Claude Structured Hook Output

Stock Claude hooks return decisions under hookSpecificOutput
(permissionDecision/permissionDecisionReason), surface context there,
and use continue:false plus the legacy approve/block decisions — none
of which the sanitizer's native field names recognized, so a guard that
works in Claude silently allowed in LibreChat. Parsed JSON now passes
through a dialect normalizer first: hookSpecificOutput fields map to
decision/reason/additionalContext, continue:false becomes
preventContinuation, approve becomes allow, and block becomes deny on
events that block by denying. Native fields win when both dialects
appear, and the ask-to-deny gate applies to the Claude dialect too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Validate Native Decisions and Slim Once Keys

- Strip malformed native output fields before the dialect merge, so a
  placeholder like {"decision":null} can no longer suppress a valid
  Claude permissionDecision into a silent allow; only recognized
  decision tokens take precedence
- Preserve the caller's working directory in hook payloads: cwd now
  reports the run's session context instead of the plugin installation
  path, which commands already receive as PLUGIN_ROOT and which the
  executor still uses as each process's working directory
- Store a compact sha256 digest instead of the full serialized handler
  in once keys: declarations may carry 32 KB commands and 256 args, and
  the previous key embedded them in every retained conversation scope

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Validate Decisions Per Event Channel and Control Post-Tool Blocks

- Accept native decision tokens only from the target event's own
  vocabulary: "continue" is valid on Stop but malformed on a tool
  event, where it previously survived validation, blocked the Claude
  dialect merge, and was then dropped by sanitization into a silent
  allow
- Translate a structured "block" on events with no deny channel
  (PostToolUse, PostToolUseFailure, and the other prevent-trait events)
  into preventContinuation with the block reason as stopReason, instead
  of discarding it and returning a reason that controls nothing
- Document why LibreChat runs supply no payload cwd: tool paths address
  a remote code-execution sandbox rather than the API host where hook
  commands run, so no host directory describes the run

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 13:21:15 -04:00
Danny Avila
5c939d129b
🔌 feat: Add Agent Plugins (Experimental) (#14704)
* 🔌 feat: Add Agent Plugins v1.0.0 Support

Implements the Agent Plugins 1.0.0 specification so LibreChat can load
portable plugin packages: a `plugin.json` manifest, `skills/` holding Agent
Skills, `mcp.json` describing MCP servers, and reverse-domain extension
directories.

- Validate the closed `plugin.json` schema, selecting rules from `$schema`
  without retrieving it. Unknown top-level fields and a non-object
  `extensions` field are reported and ignored; every other violation rejects
  the plugin.
- Enforce plugin-root containment through realpath, including for paths whose
  leaf does not exist, and apply the narrowest failure boundary per component.
- Map `mcp.json` onto LibreChat MCP options across stdio, Streamable HTTP, and
  legacy HTTP+SSE, bypassing the config loader's `${VAR}` process-env
  expansion so plugin values never resolve against the server environment.
- Expand only `${PLUGIN_ROOT}` and `${PLUGIN_DATA}`, once and non-recursively,
  in `args`, `env` values, and `cwd`; supply both variables to the subprocess
  after configured `env`, and reject entries that declare them.
- Discover skills from the immediate children of `skills/` only, reusing the
  deployment skill loader so plugin skills are ordinary deployment skills with
  a distinct id namespace.
- Read LibreChat's `ai.librechat` extension directory and hand
  `hooks/hooks.json` to the Claude hook compatibility layer.
- Load operator-installed plugins from `DEPLOYMENT_PLUGINS_DIR` at startup,
  merging their skills into the deployment skill registry and their MCP
  servers into the app config. Plugins never displace a configured server or
  deployment skill.
- Add `cwd` to the stdio MCP transport, which the specification requires and
  LibreChat did not previously support.

Component failures stay isolated: a malformed `mcp.json`, an invalid skill, or
a bad hooks document never prevents the rest of a plugin from loading.

* 🔒 fix: Contain Agent Plugins config at the runtime boundary

Review of #14704 surfaced that every real finding sat where the loader's
output crosses into LibreChat's existing runtime, not in the specification
logic. The loader deliberately left plugin placeholders literal, but
downstream layers re-processed the same fields and undid it.

- Mark plugin MCP configuration with `source: 'plugin'` and return it verbatim
  from `processMCPEnv`. Without this a remote plugin could declare
  `Authorization: Bearer ${OPENAI_API_KEY}` and receive host credentials at its
  own origin. The gate reads the configuration rather than a caller-supplied
  flag, so no future call site can reintroduce the leak by omitting it.
- Skip `preProcessGraphTokens` for plugin configuration as well; it resolves
  placeholders into headers, url, and args on the same path.
- Reject plugin server names that change under `normalizeServerName`. Tool keys
  embed the normalized name while request-time resolution uses the raw name, so
  an unstable name published tools that nothing could resolve.
- Reject `__proto__`, `constructor`, and `prototype` as server names, and merge
  plugin servers with `Object.defineProperty` and an own-property conflict
  check, so a package cannot reach a prototype setter or collide with an
  inherited member.
- Enforce manifest-name uniqueness before components are accepted; two packages
  sharing a name would share one `PLUGIN_DATA` directory.
- Isolate a failed data-directory creation to the single plugin instead of
  rejecting the whole scan.
- Prefix rejected-plugin diagnostics with the directory, which is the only
  identifier a package without a valid manifest has.
- Type extension namespace contents as JSON rather than `unknown`, and correct
  the header field-value comment to name obs-text.

Verified end to end from the built package: a plugin declaring an environment
placeholder in a header reaches the transport with the placeholder intact while
operator-authored configuration still resolves normally.

* 🔇 fix: Report Agent Plugin hooks that will not run

The loader reads `ai.librechat/hooks/hooks.json`, but nothing registers the
resulting plan, and startup supplies no hook capabilities. A package declaring
hooks was therefore accepted in silence, leaving an operator to believe the
hooks ran.

Detect the document when no capabilities are registered and report it as
unsupported, so the limitation is visible in startup diagnostics rather than
inferred from behavior that never happens.

* 🧯 test: Restore MCP startup test mocks

Carries the two mock additions from #14711 so this branch can prove itself
green. `initializeMCPs` now calls `syncStaticTools`, which the server startup
specs do not stub, so they fail on every branch that has not picked this up.
Drops out of the rebase once #14711 lands.

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-09 08:10:22 -04:00
Danny Avila
1bd4455c2d
🧭 fix: Make MCP Catalog Redis Cluster-Safe (#14717)
* fix: make MCP catalog Redis startup cluster-safe

* fix: stabilize Redis readiness gate

* fix: type Redis readiness export

* style: apply canonical import order
2026-08-09 06:59:29 -04:00