Commit graph

310 commits

Author SHA1 Message Date
Danny Avila
b6b24741da
🗝️ fix: Strengthen Two-Factor Recovery Credentials (#16742)
* fix: Strengthen Two-Factor Recovery Credentials

* fix: Declare Recovery Helper Return Types

* fix: Consume Recovery Codes Atomically

* fix: Stage Recovery Code Rollout and Improve Input
2026-10-04 12:16:37 -04:00
Danny Avila
741978c954
🔭 feat: Recover Stale Chunks Without the Error Page and Ship Client Logs to RUM (#16701)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* 🔭 feat: Recover Stale Chunks Without the Error Page and Ship Client Logs to RUM

Stale chunks: a tab left open across a deploy that opens a code-split panel
requests a hashed chunk the new deploy removed. Vite's preload handler hands
the failure to the stale-asset recovery, which reloads, but the import then
resolves to `undefined` and `React.lazy` throws a TypeError into the route
error boundary, so the error page flashes before the reload. Lazy imports now
go through `lazyWithRecovery`/`importWithRecovery`, which turn that result into
a `ChunkLoadError`; the route boundary recognizes chunk failures (and any error
while a recovery reload is underway), asks the page recovery once, and renders
a localized "Updating to the latest version…" status. When the loop guard
declines, the normal error page with its reload action shows instead. The
page recovery is now idempotent within a page, so concurrent callers join the
in-flight reload rather than tripping the guard.

Client logs: with RUM in proxy mode, the client logger's warn/error calls,
error-boundary catches and stale-asset recovery events are exported as OTLP
log records through the authenticated `/api/rum/v1/logs` proxy. Records carry
allowlisted, scrubbed fields only, follow RUM session sampling, dedupe repeats
with a count, are budgeted per minute and per page, batch with backoff, flush
with keepalive on page hide, and stop after persistent failures. The proxy
now refuses non-OTLP content types and applies a per-user rate limit.
`RUM_CLIENT_LOGS=false` turns the export off.

* 🔒 fix: Keep Client Logs Opt-In and Drop Application Error Messages

- RUM_CLIENT_LOGS now defaults to off, like console capture, so enabling
  RUM proxy mode never starts a new log stream on its own.
- Error messages are exported only for engine, browser and library
  generated types (TypeError, SyntaxError, DOMException, AxiosError,
  chunk failures); application errors keep type, status and stack only,
  since their messages can echo prompt or response text. All quoted
  fragments are now redacted.
- Batches are measured in encoded UTF-8 bytes of the exact request body,
  and page hide drains every queued record that fits the keepalive quota.
- Queued stale-asset events are delivered as soon as the log exporter
  starts, independent of the RUM SDK loading, and marked so they are
  never logged twice.

* 🔒 fix: Export Only Known Generated Error Messages and Ack Queued Events

- Error messages are exported only when the whole text matches a known
  engine, browser or library message shape (property access on
  undefined, "x is not a function", fetch/abort/axios status, chunk and
  CSS preload failures). The error class alone no longer qualifies, since
  any code can throw a TypeError carrying arbitrary text.
- Queued stale-asset events are marked logged only after the collector
  accepts the record carrying them, so an undelivered event replays on the
  next page; in-flight events are skipped by the SDK path.
- A fractional RUM_PROXY_USER_MAX below one falls back to the default
  instead of flooring to a limit of zero.

* 🔒 fix: Hold Unacknowledged Asset Events, Re-send In-Flight Logs on Hide

- The RUM SDK flush no longer drops queued stale-asset events whose client
  log record is still unacknowledged: they stay queued and persisted,
  flagged so their SDK action is not repeated, and leave the queue once
  the collector accepts the record.
- Page hide re-sends the records of an unanswered regular request with
  keepalive; each record belongs to the latest request carrying it, so a
  late response to the original request neither marks nor requeues it.
- Client-log export stops when the authenticated layout unmounts.
- The proxy refuses /v1/logs while client logs, console capture and replay
  are all off, so turning RUM_CLIENT_LOGS off also stops tabs that still
  hold an older startup config.

* 🔒 fix: Export Only Complete Stack Frames, Never the Message Header

Stack reduction now strips the V8 `name: message` header (including every
line of a multi-line message) and keeps only lines with a complete V8,
Gecko or WebKit frame shape that ends in a URL or absolute path with a
line number. A message ending in a source-position-like suffix can no
longer pass as a frame.

* fix: centralize RUM delivery and asset recovery lifecycle

* fix: preserve RUM page lifetime and collector acknowledgements

* fix: retain page transport quotas and recover functional chunks

* fix: scope RUM client sessions to authenticated identity

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-03 23:17:56 -04:00
Danny Avila
1a8d23e1ed
✅ feat: Default Machine Selection and Linked-Worktree Lanes On (#16678)
* ✅ feat: Default Machine Selection and Linked-Worktree Lanes On

Per-chat machine choice from an agent's saved allowlist and linked-worktree
lane routing were both opt-in, so neither worked until a deployment found and
set its flag. Both are now on unless set to false: machine choice still only
applies to agents with a saved allowlist, and lanes still require a worker
that advertises git_linked_worktree. One helper per flag in data-provider
decides the default for every server reader and the advertised client config.

* feat: Advertise the Code Environment Decision Protocol by Default

Per-chat machine choice needs the conversation decision protocol, which the
client requires before offering allowlisted machines, but the protocol was
only advertised when CODE_ENVIRONMENT_DECISION_VERSION=1. With selection on
by default, authors could save an allowlist the composer never offered.
An unset or blank value now advertises protocol v1; any other value, such as
0, keeps the legacy-safe path.

* fix: Turn Machine Choice Off With the Decision Protocol Opt-Out

CODE_ENVIRONMENT_DECISION_VERSION=0 stopped advertising the protocol but
routing still accepted a non-default machine, so a cached tab or a direct
client could persist a choice during a mixed-version rollout. Machine choice
is recorded through that protocol, so isCodeEnvironmentSelectionEnabled now
requires both, for routing and for the advertised endpoints config. The
protocol resolver moves to code/protocol.ts (re-exported from code/config)
so execution and endpoints can use it without an import cycle.

* style: Format checkout routing regression cases

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 20:08:35 -04:00
Marco Beretta
92432e95d5
🔐 feat: Enforce Required Two-Factor Authentication (#14731)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: enforce required two-factor authentication

Require local and LDAP accounts to enroll in TOTP before they can use the
app when ENFORCE_TWO_FACTOR_AUTHENTICATION is set. Sign-in, refresh, bearer
verification, optional share auth and image authorization all route an
unenrolled account into a staged setup flow, and every bearer credential
minted before enrollment or a password reset is retired.

Password reset retirement uses the existing credentialsChangedAt stamp, and
commitPasswordReset clears any staged enrollment in the same write. Passkey
sign-in follows the same enrollment gate as password sign-in.

* test: cover passkey sign-in handing off to required enrollment

* refactor: keep Mongoose user types out of two-factor auth contracts

* fix: withdraw a minted session when its retirement recheck fails

* fix: hold Agent API keys and the reset CLI to the required two-factor policy

* fix: adopt a setup token handed off after the setup screen mounted

* fix: move every tab that receives an enrollment hand-off into setup

* fix: read the enrollment gate's retirement cutoff in system scope

* fix: clear minted session cookies before the fallible session deletion

* fix: adopt a setup token persisted before the setup screen subscribed
2026-10-01 23:39:51 +02:00
Marco Beretta
eefeb2a7f9 ⚙️ feat: Make the Per-User Passkey Cap Configurable (#16259)
(cherry picked from commit 84180a87e2)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16259
2026-10-01 03:46:17 +00:00
Marco Beretta
4617302116 🔑 feat: Passkey Authentication with WebAuthn Sign-In and Management (#14715)
* feat: passkey authentication with WebAuthn sign-in and management

* refactor: tighten passkey account protections

Require password confirmation before adding or removing a passkey, and rate
limit those confirmations per user in their own namespace. Restrict passkey
enrollment and sign-in to local accounts, and re-evaluate account status once
the credential owner is resolved. Require user verification on the sign-in
ceremony. Invalidate outstanding access tokens on password reset. Tighten
redirect target validation and sanitize ceremony failure logs.

Adds coverage across the passkey controller, the WebAuthn helpers, the step-up
limiter, both JWT strategies, the redirect utility and the passkey settings UI.

* fix: localize passkey default name and respect router basename

Send a localized default credential label at enrollment so non-English
locales no longer persist the server's English fallback, and route the
passkey post-login and 2FA navigations through the deployment base path
so subdirectory installs stay inside the router.

Also mock syncStaticTools in the server boot specs; without it the
post-listen MCP initialization calls undefined and exits the worker.

* fix: harden passkey counter checks and reach stranded credentials

Make the signature counter write a compare-and-swap so two assertions
that verified against the same stored counter cannot both commit, and
reject the losing one; a cloned authenticator is otherwise accepted
inside that window. Counterless authenticators keep restamping at 0.

Keep the passkey panel reachable for identity-provider accounts, which
already get passwordless removal server-side but had no way in, while
hiding the add control they cannot use. Cascade passkey deletion from
the admin user-deletion path so credential IDs are not stranded.

* fix: reject stale 2FA temp tokens and surface passkey load failures

A temporary 2FA token proves only the first factor, so honouring one
minted before a password reset let anyone holding the second factor
survive account recovery. Apply the same credentials-changed cutoff the
JWT strategy uses.

Also distinguish a failed passkey query from an empty list; the error
state previously rendered as "no passkeys yet", which misreports what
is protecting the account.

* fix: fail closed when the passkey counter write cannot commit

Returning success on a storage error bypassed the compare-and-swap it
exists to enforce, so a cloned authenticator could be accepted while the
stored counter stayed stale. A database that cannot take this write
cannot mint the session either, so little is gained by failing open.
Counterless credentials carry no clone signal and keep their old path.

* fix: keep the passkey 2FA handoff inside the router

A full-page navigation to the 2FA screen remounted AuthContextProvider,
which silent-refreshes on mount; no refresh session exists yet at that
point, so it bounced back to login before the code could be entered.
Route through the router like the password flow does.

Also reveal the removal step-up field when the server refuses a
passwordless attempt, so an account migrated to an identity provider
without its old hash cleared can still clear its credentials; drop the
staged password when the dialog closes; and revoke passkeys in the CLI
password reset, which bypassed the HTTP controller's cleanup.

* fix: apply one email-verification policy across login methods

Accounts predating mandatory verification are marked verified as they
sign in when no email is configured, but only the password strategy did
it, so the same legacy account was admitted by password and refused by
passkey. Extract the rule into a shared helper and call it from both.

* fix: refuse a passkey sign-in when the ban check cannot complete

checkBan defaults next to a no-op, so calling it bare discarded any
internal failure and left req.banned unset, letting the request fall
through and be issued tokens. Capture the error the middleware chain
would have propagated and reject instead.

* fix: refine passkey management dialogs

* test: stabilize passkey tooltip assertions

* test: reconcile passkey specs with the canary reset and deletion flows

The credentialsChangedAt stamp now lives in commitPasswordReset, so its
assertion moves to the passwordReset unit spec. Restores the describe
blocks the rebase merged together.

* test: mock passkey routes in the cross-site login suite

* refactor: move passkey handlers and credential checks into packages/api

The passkey controller, the credentialsChangedAt predicate and the legacy
email-verification grandfathering now live in TypeScript with injected
dependencies; /api keeps only the wiring that builds the handlers.

* style: align passkey UI with Tailwind v4 order and design-rule lint

* fix: revoke earlier access tokens on a CLI password reset

passwordVersion is not a user schema field, so the write was dropped and
tokens issued before the reset kept verifying.

* test: cover passkey sign-in and credential revocation end to end

* fix: hide passkey sign-in when email login is disabled

The passkey endpoints already refuse under ALLOW_EMAIL_LOGIN=false, so
advertising the button only led to a 403.

* fix: let only one concurrent verification consume a passkey challenge

Without a native getDel the challenge was read then deleted, so two
concurrent verifications could both use it. The delete now arbitrates:
only the caller whose delete removed the key gets the challenge.

* test: run passkey ceremonies on the localhost relying party

WebAuthn refuses an IP address as the RP ID, and the harness serves the
app on 127.0.0.1. The reset scenario runs once per project, so the mock
harness lifts the per-IP reset request limit as it does for forks.

* fix: keep passkey management when email login is disabled

Gate only the sign-in button on emailLoginEnabled, since only the sign-in
routes sit behind that gate; list, rename and remove stay reachable.

* fix: build the passkey credential index before the first registration

With MONGO_AUTO_INDEX=false the unique credentialId index was never
created, so concurrent registrations could store the same credential.

* test: enable password reset in the mock harness and cover email-login-off passkeys

* fix: stop re-exporting the passkey cap from data-schemas

The bundler left every import of methods/passkey.ts unlinked while that
module re-exported MAX_PASSKEYS_PER_USER from librechat-data-provider, so
the index build and the counter-failure log threw ReferenceError at
runtime. Consumers now import the cap from librechat-data-provider.

* test: close nested settings dialogs before logging out in passkey scenarios

* fix: reject cross-site passkey assertions

The verify route accepted a form posted from another site, which let an
attacker sign a victim into the attacker's account. It now runs the same
requireSameOrigin guard as password login and the 2FA step.

* fix: end refresh sessions on a CLI password reset

A surviving refresh session could mint an access token issued after the
credentialsChangedAt stamp, so the CLI now deletes sessions like the web
reset does.

* fix: keep the passkey button locked when it supersedes autofill

Starting the modal ceremony aborts the pending autofill one, and the
autofill error path cleared the shared lock and busy state while the
modal ceremony was still running.

(cherry picked from commit 471abaa64d)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14715
2026-10-01 03:46:16 +00:00
Marco Beretta
4ffe3825ec 📧 feat: add email address change (#14597)
(cherry picked from commit 9ff251dd96)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14597
2026-10-01 03:46:16 +00:00
Danny Avila
e9d1d330b1
🧬 feat: Resolve GPT Point Releases to Their Family and Add GPT-6.1 Sol (#16526)
* feat: Resolve GPT Point Releases to Their Family, Add GPT-6.1 Sol

gpt-6.1-sol matched no token, pricing or family key, so agents ran with the
default 32K context and summarized constantly. Point releases now inherit
their family's context, output limit, pricing, reasoning options and native
Responses routing until listed explicitly; GPT-6.1 Sol joins the catalog.

* test: Type the point-release routing helper as ResponsesApiRouting
2026-09-29 19:04:06 -04:00
lia-by-librechat[bot]
feba26360b
🌞 feat: Add GPT-6 Sol and Luna Support (#16221)
* feat: add GPT-6 Sol and Luna support

* fix: normalize Sol and Luna reasoning requests

* fix: align Sol and Luna effective Responses routing

* fix: honor server routing policy for model uploads

* test: await system grant indexes before writes

* fix: verify model upload policies against runtime initialization

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-22 22:37:08 -04:00
lia-by-librechat[bot]
16a7e116ef
💌 feat: Authenticate RUM Proxy Exports (#16114)
Co-authored-by: Lia <lia@librechat.ai>
2026-09-20 05:25:26 -04:00
lia-by-librechat[bot]
2f67471b65
🚐 perf: Enable Redis Stream Coalescing by Default (#16093)
* 🚐 perf: Enable Redis Stream Coalescing by Default

* fix: Preserve Legacy Chunk Frames in Coalesced Publications

* fix: Align Coalesced Publication and Append Size Thresholds

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-19 13:04:43 -04:00
Danny Avila
854645ffc0
🔁 feat: Retry OpenID Discovery After Startup Failures (#15899)
* fix: retry OpenID discovery after startup failures

* fix: keep OpenID background retries resilient

* refactor: move OpenID retry policy into API package

* 🔁 feat: Expose OpenID Discovery Retries Through Registration Config

* 🧭 refactor: Own OpenID Discovery Registration and Defaults in TypeScript

---------

Co-authored-by: insoln <insoln@ya.ru>
2026-09-13 23:16:26 -04:00
Danny Avila
1163045cef
🖋️ feat: Seal Code Environment Choices Per Conversation (#15868)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: lock code environments per conversation

* style: Sort code environment imports

* fix: Harden code environment decisions

* fix: Preserve code environment rollout compatibility

* fix: Harden code environment boundaries

* refactor: Centralize code environment policy

* test: Restore partial disconnect policy mock

* fix: Close code environment edge paths

* fix: Preserve implicit stateful code routes

* fix: Stabilize implicit code routing

* test: Align remote workspace contract

* chore: Sort code route import
2026-09-12 20:45:00 -04:00
Zsanz3
b356c3d87e
🈳 fix: Treat Blank Mongo Index Settings as Unset (#15809)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* 📭 fix: Treat Empty Mongo Auto-Index Env as Unset

Empty or whitespace MONGO_AUTO_INDEX / MONGO_AUTO_CREATE values are now
treated as unset so Mongoose keeps its defaults instead of receiving false.

Co-authored-by: Zsanz3 <Zsanz3@users.noreply.github.com>

* fix: Format optional Mongo index setting helper

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Zsanz3 <Zsanz3@users.noreply.github.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-09 23:24:18 -04:00
Danny Avila
09da878a47
🚰 fix: Throttle Shared Link Reads and Aggregate Audit Findings (#15605)
* 🪵 fix: Aggregate Audit Findings and Rate Limit Shared Link Reads

Audit-mode content filter rules wrote one `logger.info` per matching
fragment, and inspection deliberately continues past every finding. A
shared conversation full of distinct matching strings therefore produced
one log write per fragment, regenerated on every retrieval of
`GET /api/share/:shareId` — a public, unauthenticated route with no
limiter. The 4,096-entry inspection dedupe set is a memory bound, not an
event budget: once full, further values are inspected and logged again.

Audit metadata carries no inspected text — only rule, source, field and
provenance — so thousands of those writes were byte-identical lines.
Findings are now counted inside an aggregation scope and reported once
per distinct key with an `occurrences` count when the scope ends:

- `assertModelBoundContent` opens a scope when audit rules are configured,
  bounding one inspection pass (a single message's fragments).
- `assertConversationImportContentAllowed` opens the outer scope for
  import and shared-link snapshots, so per-message inspections nest into
  one report instead of one per message.
- `createShareContentPreflight` spans the shared-file metadata pass too.

Nested scopes reuse the outermost aggregation, and outside any scope the
previous immediate write is kept.

`GET /api/share/:shareId` re-inspects the whole snapshot on every request,
so it now carries IP and user limiters like the neighbouring fork route
(`SHARE_IP_MAX`/`SHARE_IP_WINDOW`, `SHARE_USER_MAX`/`SHARE_USER_WINDOW`,
`SHARE_VIOLATION_SCORE`); the user limiter skips anonymous viewers.

* 🔒 fix: Address Codex Findings on Share Audit Aggregation

- The aggregation key joined rule fields with a space, but the filter
  schema allows spaces in custom pattern ids and labels, so
  `{id: 'a', label: 'b c'}` and `{id: 'a b', label: 'c'}` collided on the
  same source/field/provenance and merged two rules' counts under one
  identity. Serialize the tuple instead.
- `SHARE_VIOLATION_SCORE` left unset reached `logViolation` as
  `undefined`, which selects that function's default score of 1 rather
  than the documented zero. With default ban settings a viewer refreshing
  a rate-limited shared link would be banned after 20 rejections. Default
  the score to 0 in code, not only in `.env.example`.
- `removePorts` returns a full IPv6 address, so an attacker holding a
  prefix could rotate the host portion for a fresh bucket on every
  request — and for anonymous viewers the IP limiter is the only bound.
  Derive the key through `ipKeyGenerator` so IPv6 clients group by /56.
2026-09-04 12:38:34 -04:00
Danny Avila
c00acb36a0
🌌 feat: Add GPT-6 Astra Support (#15567)
* 🌌 feat: Add GPT-6 Astra Support

Registers OpenAI's GPT-6 Astra (`gpt-6-astra`) as a first-class model on the
canonical OpenAI endpoint.

Every model-specific behavior Astra requires — Responses routing for tool
calls, stripping the sampling and logprob parameters it rejects, and
substituting the `none`/`minimal` reasoning efforts it does not accept — lands
in `@librechat/agents` (danny-avila/agents#506) rather than here. None of it is
expressible at configuration time: tools bind after `getOpenAIConfig` runs, and
LangChain's request builders emit `temperature`/`top_p` from instance fields
regardless of what this layer sets. Keeping it in the SDK also means the
routing gate can be the documented rule (tools present) instead of the
over-approximation the config-time GPT-5.6 path is forced into.

What is left here is registration:

- 1,050,000-token context window and 128K max output
- Model dropdown (config.ts) and the `OPENAI_MODELS` example
- Standard pricing: $10/M input, $1/M cached input, $12.50/M cache writes,
  $50/M output
- The >272K long-context tier, which LibreChat already supports through
  `premiumTokenValues`/`premiumCacheTokenValues`: 2x input and cache rates and
  1.5x output for the full request

Batch/Flex (50%) and Fast mode (2x) have no representation in the pricing
model today and are not introduced here; they apply per-request rather than
per-model, so they need a request-shape signal this layer does not yet carry.

Tests pin the pricing against the documented multipliers rather than restating
the literals — cache writes as 1.25x input, and the premium tier as 2x/1.5x of
standard — plus threshold behavior either side of 272K, and that `gpt-6-astra`
resolves to its own key for snapshots and provider-prefixed ids instead of
collapsing onto a shorter `gpt-6` match.

Resolves #15560

Ref: https://developers.openai.com/api/docs/models/gpt-6-astra
Ref: https://developers.openai.com/api/docs/guides/latest-model

* 🧪 test: Guard the GPT-6 Astra Max Output Entry

[Copilot] The Astra tests asserted the context window but not the 128K max
output added alongside it in `anthropicMaxOutputs`' OpenAI counterpart, so that
entry could regress silently. The gpt-5.6 tier test above covers both, and the
delete-the-fix check showed the gap: removing the source entries failed only
the context assertions.

Assert `getModelMaxOutputTokens` alongside `getModelMaxTokens`, and fold the
snapshot/prefix cases into one loop so both dimensions are covered for every id
shape rather than only the bare one. Verified by deleting just the max-output
entry: both Astra tests now fail where previously neither did.

* 🪧 feat: Declare the First-Party OpenAI Endpoint to the Agents SDK

The agents SDK gates GPT-6 Astra's request constraints — Responses-only tool
calls, rejected sampling parameters, unsupported reasoning efforts — on knowing
it is talking to the first-party OpenAI surface, and defaults them off. It takes
that as a declaration rather than inferring it, because a base URL cannot answer
the question: only this layer knows whether a URL is a faithful first-party
route, a gateway, or a proxy with its own semantics.

That decision already exists here, in `isCanonicalOpenAIBaseURL`, used by the
GPT-5.6 Responses default. Declaring it from the same checks keeps it in one
place instead of duplicating it downstream where the two could drift — and if
the endpoint logic ever moves, it moves once.

`firstPartyEndpoint` is declared on `OAIClientOptions` alongside the other agents
SDK fields this layer already sets (`promptCache`, `_lc_stream_delay`,
`includeReasoningContent`), so it typechecks against the current published SDK
and is consumed once the SDK release lands.

Two backward-compatibility expectations gain the field. That is the intended
signal: they assert the full emitted config precisely so a new field cannot slip
through unnoticed.

* 🗂 fix: Keep GPT-6 Astra Out of the Assistants Catalogs

[Codex P2] `sharedOpenAIModels` feeds `defaultModels[assistants]` and
`[azureAssistants]` as well as the OpenAI and agents catalogs. Astra serves tool
calls only from the Responses API, and the Assistants endpoints do not route
through `getOpenAILLMConfig`, so listing it there offered a configuration the
provider rejects.

Astra now sits in its own `responsesOnlyOpenAIModels` list, spread into the
OpenAI and agents catalogs only. Agents keeps it deliberately: that path does go
through `getOpenAILLMConfig`.

* 🎚 fix: Route GPT-6 Astra to the Responses API at Config Time

[Codex P1] `max_tokens` was shaped into `max_completion_tokens` here, from
`llmConfig.useResponsesApi`, while the agents SDK switched Astra to the
Responses API later at invocation time — so the request reached an endpoint
expecting `max_output_tokens` and was rejected.

Deciding the API here fixes that, and generalizes: any field whose shape depends
on the endpoint has the same problem, so the choice has to precede the shaping.
It also lands before Azure replaces `llmConfig.model` with a deployment name,
which is the only point that still knows the real model id.

Unlike the GPT-5.6 rule above, this does not depend on reasoning parameters.
Astra serves tool calls only from Responses, and OpenAI recommends Responses for
it generally, so a static decision loses nothing — which was the reason the SDK
was deciding it dynamically. The existing guards carry over: OpenRouter, custom
gateways, and an explicit `dropParams` opt-out keep their configured path.

The agents SDK is still required, for the request shaping Astra needs on either
API (danny-avila/agents#506); it no longer decides routing.

* 🏷 fix: Record the Served Model Before Azure Overwrites It

[Codex P1] `llmConfig.model` is unconditionally replaced with the Azure
deployment name, so the agents SDK received an alias and could not tell which
model it was talking to — a deployment not literally named `gpt-6-astra` got
none of the request shaping Astra requires.

Records the real id in `servedModel` immediately before the overwrite, which is
the last point that still knows it. The SDK prefers it over `model` when
deciding which model-specific constraints apply (danny-avila/agents#506).

Set for every first-party Azure request rather than only Astra: "what this
deployment serves" is a fact this layer knows and the SDK decides what to do
with, and keeping model-specific knowledge out of here is the point of the
split. The backward-compatibility expectation gains the field, which is that
test doing its job.

* 🎛 fix: Do Not Let a reasoning_effort Drop Disable Astra Routing

[Codex P1] The Astra routing reused the GPT-5.6 opt-out wholesale, which treats
`dropParams: ['reasoning_effort']` as opting out of the Responses API. That
holds for the 5.6 default, which routes *because* of reasoning — removing the
effort removes its reason to route. Astra's routing is not reasoning-driven, so
a rule clearing an unsupported stored effort silently put it back on Chat
Completions, where its tool calls fail.

Astra now honors only an explicit `useResponsesApi` drop. Reusing the guard
without re-deriving whether each clause applied was the error.

* ✂️ fix: Scope GPT-6 Astra to the First-Party OpenAI Endpoint

[Codex P1 + P2] Two findings, both Azure, resolved by removing the Azure support
rather than extending it.

Azure's first-party hosts do not satisfy `isCanonicalOpenAIBaseURL`, so a
configured `*.openai.azure.com` was neither routed to Responses nor declared
first-party; and moving Astra out of `sharedOpenAIModels` also removed it from
the Azure chat fallback, since `getOpenAIModels({ azure: true })` resolves
through `defaultModels[azureAssistants]`.

Both are moot once the scope is right: OpenAI does not document Astra as
available on Azure OpenAI. Its model page lists the API and OpenAI's own
subscription plans, with no third-party cloud surface. The Azure handling was
support for a deployment the model is not documented to reach, and it produced
most of the review findings on this branch and its agents SDK counterpart.

Routing and the first-party declaration are now scoped to
`EModelEndpoint.openAI`, and the Azure-only `servedModel` recording is gone
along with its SDK counterpart. Astra staying out of the Azure catalogs is now
the intended outcome rather than a regression.

* 🚫 fix: Keep Astra Out of the Initial Azure Catalog

[Codex P1] `initialModelsConfig` feeds the OpenAI catalog to Azure OpenAI as
well, so scoping Astra to the first-party OpenAI endpoint left it advertised on
Azure anyway — and being first in the list, it could become the default
selection where no prior choice exists. The server's Azure fallback already
excludes it, so the client would offer a model the server then rejects.

The Azure initial catalog now uses the OpenAI list minus the Responses-only
models, following the same scoping decision the routing guard makes.

The initial Assistants catalog was already safe: `fitlerAssistantModels`
requires `gpt-4` or `gpt-3.5`, which Astra does not match. Pinned that in a test
rather than leaving it to a regex written for another purpose.

* ⬆️ chore: Bump `@librechat/agents` to 3.7.20 for GPT-6 Astra Constraints
2026-09-04 10:23:29 -04:00
Danny Avila
a6cf61abcf
🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565)
Exports application logs as OTel log records when OTEL_LOGS_ENABLED=true,
correlated with the active trace, and gates every signal explicitly so a
disabled signal never falls back to the SDK's default OTLP exporter.
Verified end to end over OTLP/gRPC against an in-process collector.
2026-09-03 21:16:09 -04:00
Danny Avila
2d794a0439
💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539)
* fix: Detect silently dead Redis sockets with deadline-bounded heartbeats

A Redis peer can vanish without a FIN or RST (a dropped NAT entry, a proxy
failover, a migrated VM). The socket stays `ready` from ioredis's point of
view, no error fires, and every command on it waits for the kernel's
retransmission timeout — about fifteen minutes at Linux defaults.

Observed in production: while the main client's socket was dead,
POST /api/agents hung until Cloudflare returned a 524; while the subscriber's
socket was dead, every attach failed with "Timed out synchronizing Redis
subscription" for sixteen minutes across five conversations.

`REDIS_PING_INTERVAL` could not help — it awaits ping() with no deadline, so
the probe just joins everything else that is waiting.

- Add startRedisHeartbeat: races a PING against a deadline and destroys the
  socket when the probe goes unanswered, so ioredis reconnects, replays queued
  commands, and re-subscribes a subscriber's channels.
- Destroy the stream rather than disconnect(true), whose FIN a vanished peer
  never acknowledges; cluster clients fall back to the regular reconnect.
- Add createIoRedisSubscriber so every dedicated subscriber gets an error
  listener and its own heartbeat, and route the generation-stream and both
  subagent-routing subscribers through it.
- Set ioredis keepAlive from REDIS_KEEP_ALIVE (default 10s).
- Add REDIS_PING_TIMEOUT, REDIS_SUBSCRIBER_PING_INTERVAL and REDIS_KEEP_ALIVE,
  documented in .env.example.
- Give the fake RESP server a silent mode and a SUBSCRIBE reply so the
  dead-peer case can be tested against real clients.

* fix: Probe each cluster node and refuse a non-positive heartbeat deadline

A cluster-level PING is routed to one node, so a healthy reply proved nothing
about the other sockets. The heartbeat now resolves its targets each tick —
every node for a cluster, the client itself otherwise — and races an
independent deadline per target, tearing down only the socket that stops
answering. A non-positive deadline disables the heartbeat with a warning
instead of forcing a reconnect on every tick, and the single-probe test no
longer depends on wall-clock slack.
2026-09-03 09:45:16 -04:00
Danny Avila
f9f1b2fb95
✨ v0.8.8-rc2 (#14845)
Some checks failed
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* chore: prepare v0.8.8-rc2 release

* test: satisfy mongo meili static checks

* test: avoid unsafe logger spy assertion

* chore: refresh v0.8.8-rc2 release

* test: restore mongo meili test isolation

* chore: normalize v0.8.8-rc2 versions

* docs: refresh v0.8.8-rc2 release surfaces
2026-09-02 22:18:48 -04:00
Danny Avila
790f2b3153
⚡ feat: Add Gemini 3.8 Flash Support (#15519)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* ⚡ feat: Add Gemini 3.8 Flash Support

Adds first-class support for Google's Gemini 3.8 Flash (`gemini-3.8-flash`,
GA 2026-09-02) for both the Gemini API (AI Studio) and Google Cloud Gemini
Enterprise Agent Platform, following the Gemini 3.7 Flash integration (#14818).

The Flash-family handler that PR generalized already carries the shape 3.8
needs, so registering the model is a one-line rule rather than new behavior:
it inherits the strip of deprecated sampling params (temperature/topP/topK),
the rejected penalty params, and `thinkingBudget`, defaults to `medium`
thinking, and — like 3.7 — errors on `minimal`, so an explicit `minimal` is
substituted with the nearest supported level, `low`.

- Context window (1,048,576) in googleModels; API + cache pricing in tx.ts
- Model dropdown (config.ts) and GOOGLE_MODELS examples for both integrations
- Register the model in the Flash-family thinking-rule table
- Apply the same introductory pricing as 3.6/3.7 Flash ($0.75 in / $3.75 out /
  $0.075 cached, per 1M), reverting to $1.50 / $7.50 / $0.15 on 2027-01-01;
  the existing comments at both call sites now name 3.8 alongside 3.6/3.7
- Tests mirroring the 3.7 Flash coverage: thinking default, legacy-param strip,
  explicit level pass-through, `minimal` substitution, versioned aliases,
  context window, model-key mapping, and rates

Refs #15516

Ref: https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash
Ref: https://ai.google.dev/gemini-api/docs/pricing

* 🔧 chore: Update @librechat/agents dependency to version 3.7.15 in package-lock.json and related package.json files
2026-09-02 15:06:40 -04:00
Danny Avila
d5b2a8580e
📖 feat: Add Claude Fable 5.1 Support (#15512)
Some checks are pending
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Claude Fable 5.1 (`claude-fable-5-1`, GA 2026-09-01) succeeds Claude Fable 5
at the same input/output prices, with a 1M context window and 128K max output.
The Mythos-class helpers added for Fable 5 already match it — `isMythosClassModel`
tests `claude-(?:fable|mythos)[-.]?\d`, so adaptive thinking, summarized-display
opt-in, sampling omission, 1M context, the 128K output clamp, prompt caching,
the Vertex multi-region gate, and the Bedrock PDF exemption all cover 5.1
without a code change. What was missing was registration and pricing.

Cache reads are the one rate that differs, and the substring matcher would have
silently gotten it wrong: `claude-fable-5-1` falls through to the `claude-fable-5`
key, which prices cache hits at the usual 0.1x base input ($1/MTok). Fable 5.1
and Mythos 5.1 price them at 0.025x ($0.25/MTok), so every cache read would have
billed 4x — on exactly the long agentic sessions this model is built for.

- Register `claude-fable-5-1` in the shared Anthropic list and
  `global.anthropic.claude-fable-5-1` in the Bedrock list
- Add 1M context / 128K output entries for `claude-fable-5-1` and
  `claude-mythos-5-1`
- Add $10/$50 rates and `{ write: 12.5, read: 0.25 }` cache rates for both
- Update the `.env.example` and Vertex `librechat.example.yaml` examples
- Test that 5.1 resolves to its own key rather than collapsing onto 5.0, across
  token maps, pricing, cache rates, and Mythos-class detection

The three documented breaking changes from Fable 5 need no handling here:
LibreChat never sends a forced `tool_choice` on the Anthropic path (the agents
SDK omits the param when unset, so the API default `auto` applies); thinking
blocks an earlier model can't read are dropped server-side, unbilled; and
history is rebuilt from stored content parts rather than replayed signed
thinking blocks, so the prefix-binding check has nothing to reject.

Refs #15506
2026-09-02 10:45:54 -04:00
Danny Avila
8f6228095f
🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format (#15453)
* 🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format

`read_file` on a `/mnt/data` image collapsed every failure — rate limit,
missing path, truncated transfer, transport error — into the binary-file
hint, whose image branch reads as a permanent capability limit ("is an
image file and cannot be read as text"). The model stopped retrying, and
the operator had no way to tell which failure had actually occurred.

The failure that surfaced it: the reader windows the bytes through
`/exec` stdout, one call per window, against the Code API's per-user
execution limiter (20 requests / 30s in the reference deployment). At a
fixed 32KB window a 122KB chart cost 4 calls and a 1MB image cost 33 —
more than a whole limiter window, so anything over ~640KB could never
complete. A turn that generates several charts and reads them back
exhausts the budget partway through and reports the 429s as unreadable
images.

- `handlers.ts`: failed reads name the real cause and the affordance that
  matches it — rate-limited waits and retries, a missing path lists the
  directory, a truncated transfer retries. The "cannot be read as text"
  phrasing is now reserved for bytes that are not a supported image. An
  image past the inline budget says to downscale and read the smaller
  copy rather than pointing at `bash_tool`, which cannot show an image.
- `process.js`: the window is derived from the runner's stdout budget
  (48,960B vs 32,768B, so a 122KB chart is 3 calls and 1MB is 22), halves
  and re-reads the same offset when a runner truncates anyway (remembered
  per Code API, with the env remedy logged), waits out a rate limit it
  can afford instead of discarding the bytes already pulled, and stops at
  a 24-call ceiling so no read stalls a turn across limiter windows.
- `utils/code.ts`: shared `getCodeApiRetryAfterMs` (header, then the
  `retry_after_seconds` body); `skillFiles.ts` drops its private copy.

* ♻️ refactor: Move the Sandbox Image Reader Into TypeScript

Addresses the review round on #15453.

- P1: windowing, window sizing, the in-sandbox reader script, response
  parsing and byte assembly move to `packages/api/src/files/code/image.ts`,
  and the Code API rate-limit policy to `packages/api/src/utils/code.ts`.
  `process.js` keeps only the `/exec` transport it owns. Both new modules
  are tested against their real implementations (19 + 13 cases), including
  the round-trip ceiling, the changed-file guard and the shell-injection
  guard on the reader script.
- P2: a configured stdout budget below the old 8KB window floor produced a
  window the runner could never emit, and narrowing was pinned to the same
  floor, so no retry could succeed. The window now aligns to the base64
  group size instead, and the round-trip ceiling bounds what a small budget
  costs.
- P2: `.env.example` documented a 32,768-byte default that no longer holds
  and never mentioned the stdout budget. It now describes
  `LIBRECHAT_CODE_SANDBOX_OUTPUT_MAX_SIZE` as the setting to tune, with the
  chunk override as the escape hatch.
- P3: `process.spec.js` no longer reimplements the retry parser in its
  module mock. Behaviour is tested where it lives; the mock is a declared
  passthrough and the spec covers what the transport still owns — session
  forwarding, the profile header, parser hand-off and error logging.
- P3: `getCodeApiRetryAfterMs` reads the response body through `in`
  narrowing rather than an `as` assertion.
- P3: the window-sizing JSDoc no longer claims a multiple of 3 in the one
  case that is not — an explicit `LIBRECHAT_CODE_IMAGE_CHUNK_BYTES`.

* 🎯 fix: Size Sandbox Image Reads to What the Limiter Can Deliver

Addresses review round 2 on #15453.

- Narrowing now halves the caller's own failing window instead of the
  shared learned value, and the learned value only moves down to the
  narrowest size any read needed. Reads run concurrently (one per tool
  call), so two reads failing at the same size were compounding into a
  window two steps smaller than the runner would have accepted, and
  persisting it for every later read.
- Narrowing continues until a runner accepts a window or a floor below
  which no runner could serve an image at all. Two halvings stopped at
  12,240 bytes — still ~16KB of base64 — so an 8-10KB stdout cap that was
  not configured up front stayed unreachable.
- The round-trip ceiling is now one limiter window (20), and a read that
  cannot finish within it says so from the FIRST response rather than
  spending the whole budget rediscovering it: the first window reveals
  both the file size and what a call really delivers. A near-limit image
  no longer drains the turn's executions before failing, which is what
  made the wait budget look too small.
- The round-trip message names the size this deployment can actually
  inline, so the model has a downscale target instead of a guess.
- `python3: not found` from the sandbox reader was matching the generic
  "not found" in `isSandboxMissingFileError` and being reported as a
  missing image path, sending the model to `ls /mnt/data` while hiding a
  runner dependency. Image reads now classify only path-specific ENOENT
  forms as a missing file.

* 📐 fix: Size a Truncated Read's Retry From the Cap It Revealed

Addresses review round 3 on #15453.

A ~1KB sandbox stdout cap is a real runner default, and the 1,536-byte
narrowing floor sat above it: discovery stopped one step short and
declared every image unreadable, even though small ones fit comfortably
in the 576-byte window the configured path already derives.

Narrowing now runs until no smaller window exists. It also no longer has
to grope its way down: a truncated response emitted exactly as much as
the runner allows, so its length measures the cap this deployment never
declared, and the retry is derived from it directly — one round-trip
instead of the eight blind halvings it takes to fall from ~49KB to ~1KB.
Blind halving remains the fallback and bounds the result either way, so
every retry is strictly smaller than the window that just failed.
2026-09-01 09:08:01 -04:00
Danny Avila
dc77b78d3e
🔇 refactor: Quiet Framework Logging in Unit Tests (#15363)
* 🔇 refactor: Quiet Framework Logging in Unit Tests

Unit test output was overwhelmingly framework log lines rather than test
results. A `packages/api` run printed 9,076 lines for 434 suites; 1,596 of
them were the identical `at Console.log (winston/transports/console.js:87)`
frame that Jest staples onto every winston write.

Three causes:

- Each winston logger built its Console transport with a hard-coded
  `level: 'info'`. Winston resolves a transport's explicit level ahead of
  its parent's, so the logger's own `NODE_ENV`-derived `warn` never applied
  — in tests or in production. `CONSOLE_LOG_LEVEL` now drives it, defaulting
  to the previous `info` so deployments are unaffected. `meiliLogger` had
  the same hard-coded level and is routed through the same helper.
- `LOG_TO_FILE` defaults on, so every Jest worker opened DailyRotateFile
  transports. A run left `packages/api/logs/` holding a 107KB error log, a
  gzipped rotation, audit JSON, and — from fake-timer suites — files dated
  1969 and 2023.
- Client suites carried leftover debug `console.log` in shipped source
  (`Test mode. Skipping silent refresh.` alone accounted for 109 blocks)
  plus three test-harness defects whose React error dumps buried everything
  else: a markdown suite rendering Mermaid with no Router, a `~/Providers`
  mock missing `useSearchContext` (the component's own catch swallowed the
  throw, so the sources path was dead while the suite passed), and
  `getBoundingClientRect` stubs without `left`, which made a computed
  `right` NaN.

A shared `config/jest.setup.logging.cjs` sets the two env vars for both
backend workspaces. It sets env only and requires nothing: eagerly requiring
the logger froze `CREDS_KEY` into `encryptV3` before specs could set it and
broke five suites. `TEST_VERBOSE_LOGS=true` restores the logs for debugging.

Output per full run:

  packages/api           9,076 -> 770 lines   (winston lines 1,596 -> 0)
  packages/data-schemas  2,521 -> 946 lines   (winston lines   290 -> 0)
  client                10,384 -> 3,153 lines (console blocks  285 -> 74)

Test counts are unchanged: client 476 suites / 5,827 tests green, and the
Ariakit `act(...)` warnings the selector suites emitted are gone rather than
suppressed.

* 🔧 fix: Settle Shutdown Timers Before Counting Them

`destroy()` finishes cancelling its fenced retirement timers on the tick
after it resolves. The count only returned to the baseline because a console
write from the logger happened to yield first — silencing that logging left
four timers still pending at the assertion, deterministically failing the
suite in CI's sharded, coverage-enabled run.

Advance the fake clock by zero before counting, so the assertion proves the
timers were cancelled rather than that something incidental yielded.

* 🛡️ fix: Validate CONSOLE_LOG_LEVEL Instead of Trusting It

Winston resolves a level name against the logger's level map, so a name
outside it resolves to `undefined` and the transport drops every message.
Passing `CONSOLE_LOG_LEVEL` straight through meant a typo (`warning`) or a
stray space silently muted all console output — a misconfiguration that reads
as a dead deployment.

Validate against the level set, normalize case and whitespace, and fall back
to the default with a warning on the way past, since the logger cannot report
its own misconfiguration.

The level map itself was duplicated verbatim in both loggers; it now lives in
`utils` beside the validation that depends on it.

* 🎚️ fix: Let CONSOLE_LOG_LEVEL Outrank DEBUG_CONSOLE

`DEBUG_CONSOLE=true` forced the console transport to `debug`, so on those
deployments every non-`silent` value of `CONSOLE_LOG_LEVEL` was discarded —
`CONSOLE_LOG_LEVEL=error` still emitted debug and info, defeating the control
it advertises.

`DEBUG_CONSOLE` now only moves the *default* level and keeps choosing the debug
format, so an explicit level stays in charge of verbosity. With no explicit
level the resolved default is still `debug`, unchanged for existing setups.

Covers the wiring with a spec that re-imports the logger per environment,
rather than only testing the resolver in isolation.
2026-08-30 15:26:30 -04:00
Danny Avila
3fa33b740b
🛫 refactor: Promote Generation Protocol V2 Automatically (#15324)
* refactor: promote generation protocol v2 automatically

* fix: remove unused protocol import
2026-08-28 17:17:09 -04:00
Danny Avila
77c2a51cf3
🪧 fix: Advertise Detached Event Actor Support from the Generation Store (#15322)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
* fix: activate detached event actions automatically

* fix: shield detached terminal generations

* perf: share capability availability index

* style: flatten capability status selection

* fix: preserve mixed-version lifecycle shells

* fix: complete detached action store adapters

* fix: close mixed-version capability races

* fix: honor legacy capability success
2026-08-28 16:33:23 -04:00
Danny Avila
487984193a
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable

* fix: align detached actor checks with merged base

* fix: close detached actor lifecycle gaps

* fix: break detached action outcome cycle

* fix: deepen detached action ownership

* fix: fence detached launch handoffs

* fix: preserve detached rollout fencing

* test: type detached recovery failure

* fix: close detached ownership handoffs

* fix: stage detached action activation

* fix: persist detached terminal retries

* fix: fence detached durability ownership

* fix: fence trigger lane publication

* fix: fence detached transition generations

* fix: retire failed detached predecessors

* fix: close detached rollout consumers

* test: type detached retry turns

* fix: close detached ownership gaps

* fix: fence detached action resumes

* fix: fence prior-head event actor resumes

* fix: preserve legacy event actor resumes

* fix: align detached resume boundaries
2026-08-28 14:21:04 -04:00
Danny Avila
03fa98eb37
🕸️ feat: Complete Keenable Web Search Provider (#15288)
* feat: wire Keenable web-search provider into config, schema, and UI

Keenable landed as a search provider in @librechat/agents (#285, shipped in
3.2.58+), but LibreChat did not yet expose it. This adds the config/schema/UI
glue so it can be selected, mirroring the existing Tavily provider.

- data-provider: add `keenable` to SearchProvider type + SearchProviders enum,
  keenableApiKey/keenableApiUrl schema fields, and a keenableSearchOptions block
  (maxResults, site, attributionTitle, timeout).
- data-schemas: register keenable in webSearchAuth.providers and default the
  key/URL placeholders in loadWebSearchConfig.
- api/web: pass keenableSearchOptions through to the provider and handle
  Keenable's keyless model. Unlike other providers it authenticates with no key
  (the public endpoint), picking up an optional key/URL when set; the URL
  override is SSRF-preflighted like other user-provided URLs.
- client: add Keenable to the provider dropdown with an optional API-key input.
- docs: document KEENABLE_API_KEY/KEENABLE_API_URL in .env.example and a
  webSearch example in librechat.example.yaml.
- tests: keyless + keyed auth resolution, config defaults, and schema parsing.

* fix: ESLint no-unused-vars and clarify Keenable yaml example

- Remove the now-unused RerankerTypes import in data-schemas web.ts (the lint
  job runs with --max-warnings 0 on changed files, so this latent warning failed
  CI once the file was touched).
- Note in the librechat.example.yaml Keenable stanza that a scraper (and
  reranker) is still required for web search to load, and include a Firecrawl
  scraper in the example.

* chore: fix import order drift (sort-imports)

* feat: add Keenable as a keyless scraper and select it without a pinned provider

The Keenable scraper landed in @librechat/agents#337, so wire the scraper
category the same way the search provider already is: `scraperProvider:
keenable` reads pages through Keenable's public fetch endpoint with no key
(a key only lifts rate limits, and the endpoint is overridden with
KEENABLE_FETCH_URL). Paired with `rerankerType: none` this makes a fully
keyless web-search stack possible for the first time.

Also closes the Codex finding on this PR: because none of Keenable's auth
fields are required, the generic auth loop skips it whenever it isn't pinned,
so a key submitted through the API-key dialog (which cannot pin a provider)
left the providers category unauthenticated. Keenable is now selected in that
case, gated on one of its values actually being present so installs that
configured nothing keep their current behavior. The scraper gets the same
fallback, additionally gated on Keenable being the resolved search provider,
so it never silently scrapes for another provider.

* fix: select the Keenable scraper from a supplied key, not only for Keenable search

The API-key dialog submits credentials and cannot pin a provider, so choosing
Keenable as the scraper while search stays on Serper/SearXNG/Tavily had no
effect: the unpinned-scraper fallback required Keenable to also be the resolved
search provider.

A supplied Keenable value now triggers it as well, which is the only signal the
dialog can send. The fallback still runs only when no keyed scraper
authenticated, and with neither trigger the category stays unauthenticated, so
a deployment that never configured Keenable is unaffected.

Note the fully keyless choice still cannot be expressed through the dialog:
Keenable's key is optional, so picking it with no key submits nothing at all.
librechat.example.yaml now documents pinning scraperProvider: keenable for that
case.

* style: Sort Keenable imports

* fix: Harden Keenable auth resolution

* fix: Preserve Keenable selection intent

* fix: Fail closed on invalid web search auth

* fix: close keenable auth gaps

* style: sort web auth imports

* fix: preserve web search selection integrity

* fix: isolate web search auth ordering

* fix: silence expected credential misses

* chore: bump agents sdk

* fix: preserve web search preference ownership

* fix: forward cleared Keenable endpoint

* style: sort web search hook imports

* fix: require intent for credential clears

---------

Co-authored-by: Ilya Bogin <ilya.bogin@keenable.ai>
2026-08-27 14:29:20 -04:00
Danny Avila
8ba76507ae
🧷 fix: Verify Scheduled HITL State Before Pausing (#15284)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
* fix: require durable storage for scheduled HITL

* fix: verify scheduled HITL checkpoints before pause

* fix: narrow scheduled HITL capability checks

* fix: honor ask tool filters in schedule preflight

* fix: enforce durability across scheduled HITL resumes

* fix: scope scheduled HITL admission to active tools

* refactor: move scheduled HITL admission to TypeScript

* fix: close scheduled HITL admission gaps

* fix: bind scheduled pauses to current checkpoint

* fix: cap pending actions at binding deadline

* fix: reject expired pending actions

* fix: guard approval deadlines atomically
2026-08-27 11:16:24 -04:00
Danny Avila
0d0d7d05bb
🧬 fix: Harden SAML Identity Binding (#15264) 2026-08-27 06:45:41 -04:00
Danny Avila
877b9b2f1a
🐚 feat: Nonce-Based Content Security Policy for the SPA Shell (#14446)
* 🛡️ feat: Configurable Baseline HTTP Security Headers

Adds helmet's CSP-independent headers (HSTS, X-Frame-Options,
X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response,
with contentSecurityPolicy explicitly disabled. Every header that can
break a deployment is configurable, so there is no allow-list to go
stale the way #7377's hardcoded CSP directives did.

HSTS includeSubDomains defaults off rather than matching helmet's
on-by-default: it would otherwise pin every sibling subdomain to HTTPS
for a year in every visitor's browser, and undoing that requires
serving max-age=0 from each affected host.

* 🛡️ feat: Nonce-Based Content Security Policy for the SPA Shell

Adds an opt-in, per-response nonce CSP on the HTML response, resolved once
at startup so each request only mints a nonce and concatenates the header.
Report-only by default, since that is the rollout step #7377 skipped.

Rebase and correctness pass over #13226:

- Styles carry no nonce. A nonce in style-src makes browsers ignore
  'unsafe-inline', which would have blocked the <style> element the theme
  script injects at runtime, plus every style third-party components inject.
- frame-ancestors 'self' is now a default rather than opt-in, so enabling
  CSP actually covers the clickjacking half of #7110.
- CSP_SCRIPT_SRC_EXTRA now drops 'strict-dynamic', which would otherwise
  make browsers ignore the very hosts the operator configured.
- Nonce stamping runs after the query-devtools bootstrap injection so that
  injected script is covered too.

* fix: replace frame-ancestors instead of merging it

Merging the configured value into the default turned a deliberate
CSP_FRAME_ANCESTORS='none' into `frame-ancestors 'self' 'none'`, which
browsers resolve back to 'self'. Also bail out if the serialized policy
somehow lacks the nonce slot rather than emitting a header the shell
cannot match.

* fix: address Codex review findings on the CSP defaults

All five were real against LibreChat's actual runtime:

- CSP_REPORT_ONLY now only enforces on an explicit false/off/0/no. A typo
  or `1` previously fell through isEnabled() to enforcing, turning a
  config slip into a blocked SPA. Shares the parse helper with
  headers.ts via a new security/env.ts.
- Module preloads are stamped. A production client/dist/index.html
  carries 32 parser-inserted `<link rel="modulepreload">` tags, which
  'strict-dynamic' does not cover and 'self' cannot rescue.
- Stale nonce attributes are replaced rather than preserved; only the
  current response's nonce is authorized.
- worker-src allows data:, which Monaco's default CDN loader needs to
  bootstrap its workers (there is no loader.config() in the client).
- script-src allows 'wasm-unsafe-eval' for the HEIC upload path, which
  compiles WebAssembly through heic-to. Narrower than 'unsafe-eval'.

Verified against the real built shell: 4 scripts and all 32 preloads
nonced, stylesheets/icons/manifest and <style> untouched.

* fix: address second Codex round on CSP rollout controls

- SECURITY_HEADERS=false now disables CSP too. It is documented as the
  global kill switch, and an operator reaching for it to recover a shell
  broken by an enforcing policy must not be left with that policy on.
- The SPA shell is forced to `no-store` while CSP is enabled, ignoring
  INDEX_CACHE_CONTROL/INDEX_PRAGMA/INDEX_EXPIRES and warning when they
  are set. A cacheable shell pins one nonce across page loads and users,
  which is the whole thing a nonce policy defends against.
- Added CSP_ALLOW_WASM and CSP_ALLOW_DATA_WORKERS. The previous commit's
  .env.example claimed CSP_ADDITIONAL_DIRECTIVES could drop
  'wasm-unsafe-eval' and data:, but merging only ever appends sources, so
  the documented hardening step was impossible. These toggles make it real.
2026-08-25 09:18:52 -04:00
Danny Avila
2ef12b1e1d
🦺 feat: Configurable Baseline HTTP Security Headers (#14445)
Adds helmet's CSP-independent headers (HSTS, X-Frame-Options,
X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response,
with contentSecurityPolicy explicitly disabled. Every header that can
break a deployment is configurable, so there is no allow-list to go
stale the way #7377's hardcoded CSP directives did.

HSTS includeSubDomains defaults off rather than matching helmet's
on-by-default: it would otherwise pin every sibling subdomain to HTTPS
for a year in every visitor's browser, and undoing that requires
serving max-age=0 from each affected host.
2026-08-25 08:21:39 -04:00
Danny Avila
b6e3cf46d2
🕯️ fix: Decay Violation Scores With a Configurable TTL (#15153) 2026-08-24 08:36:21 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Danny Avila
3ebef4c84e
📨 feat: Add Authenticated Agent Event Ingress (#15110)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources
2026-08-22 01:09:04 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
5e3c680761
🪃 feat: Wake Parent Agents on Child Completion (#14975)
* feat: wake parent agents on child completion

* wip: harden child completion wakeup lifecycle

* fix: close the completion-wakeup static failures

Type the durable-claim store fixture, the continue-envelope test helper,
and the terminal message's task metadata so the wakeup suites compile
against the shapes they actually exercise. Replace `Array.prototype.at`,
which the package target library does not provide.

Capture the prepared child thread in a non-optional local before the
provider callback closes over it, and narrow the trigger envelope itself
on `mode === 'continue'` rather than a separately copied mode, so reading
the continue target is sound. Lift the parent-message fallback out of a
nested ternary into a named resolver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the active-predecessor admission fence

The Redis job-creation call gained a thirteenth scalar argument, so the
spec helper reconstructed the HSET pairs one slot early and rebuilt an
invalid job hash; three creation tests failed on that alone.

Give the fence itself direct coverage in both store adapters, which it
had none of despite deciding whether an automatic continuation may
replace a live parent turn. Each proves a running and a requires_action
predecessor are refused with the state a controller needs for a finite
409, that an absent or settled predecessor is admitted, and that an
ordinary user turn without the policy still replaces its predecessor.
The Redis case also asserts a refused continuation leaves the parent's
durable job and chunks untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: harden completion wakeup rollout and claims

* fix: close completion wakeup race windows

* test: keep the child store fixture exact

* fix: close final subagent wakeup gaps

* fix: preserve ambiguous completion claims

* fix: release pre-admission wakeup claims

* fix: stabilize subagent completion recovery

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 12:18:31 -04:00
Ravi Kumar L
006e421cd2
💡 feat: add DB-backed admin insights (#14898)
* feat: add Mongo-backed admin insights

* feat: gate insights with environment variable

* fix: tighten insights access and activity metrics

* fix: preserve insights date selections

* perf: parallelize insights search aggregation

* test: wait for MCP conflict recovery

* test: satisfy strict MCP recovery typing

* fix: disable insights pagination while loading

* fix: localize insights range shortcuts

* fix: bound insights search input
2026-08-18 07:51:51 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Danny Avila
06bf324cf0
🛤️ feat: Per-Agent Code Execution Routing With Stateful Session Scopes (#14848)
* feat: route code execution per agent profile

* chore: sort execution profile imports

* test: preserve stateful environment literal types

* fix: isolate stateful code environments by user

* fix: preserve per-agent code routing end to end

* fix: route code priming by execution profile

* fix: isolate code profile lifecycle state

* fix: preserve mixed-profile code resources

* fix: complete stateful skill routing
2026-08-16 09:42:15 -04:00
Danny Avila
eaef87fa26
🚀 chore: Prepare v0.8.8-rc1 (#14394)
* 🚀 chore: Prepare v0.8.8-rc1 release

* 📚 docs: Complete v0.8.8-rc1 operator references

* 📚 docs: Mark stateful sessions experimental

* 📚 docs: Clarify background code capability

* 📚 docs: Refresh v0.8.8-rc1 operator guidance

* 📚 docs: Highlight v0.8.8-rc1 features in README

* 📦 chore: Bump publishable packages again

* 📚 docs: Add streaming question progress

* 📦 chore: Bump publishable packages again

* 📚 docs: Refresh v0.8.8-rc1 release highlights

* 📦 chore: Bump publishable packages again

* 📚 docs: Refresh v0.8.8-rc1 release guidance

* 📦 chore: Bump publishable packages again

* 📚 docs: Highlight batched Agent questions

* 📦 chore: Bump publishable packages again

* 📦 chore: Bump publishable packages again

* 📦 chore: Bump publishable packages again

* 📦 chore: Refresh v0.8.8-rc1 package versions

* 📦 chore: Refresh v0.8.8-rc1 package versions

* 📦 chore: Refresh v0.8.8-rc1 package versions

* 📄 docs: Note PowerPoint template support

* 📦 chore: Refresh v0.8.8-rc1 package versions

* 📄 docs: Note latest provider and file support
2026-08-14 03:24:59 -04:00
Danny Avila
24d111fde9
⚡ feat: Add Gemini 3.7 Flash Support (#14818)
* ⚡ feat: Add Gemini 3.7 Flash Support

Adds first-class support for Google's Gemini 3.7 Flash (`gemini-3.7-flash`)
for both the Gemini API (AI Studio) and Google Cloud Gemini Enterprise Agent
Platform, following the Gemini 3.6 Flash integration (#14369).

- Context window (1,048,576) in googleModels; API + cache pricing in tx.ts.
- Model dropdown (config.ts) and GOOGLE_MODELS examples for both integrations.
- Register the model in the Flash-family handler so it inherits the existing
  strip of deprecated sampling params (temperature/topP/topK), rejected
  penalty params, and thinkingBudget, and defaults to `medium` thinking.
- Generalize that handler's enumerated table from a [id, level] tuple to a
  rule object, so a model can also declare thinking levels it rejects. Gemini
  3.7 Flash errors on `minimal` (which the Google endpoint offers in its
  thinkingLevel slider), so an explicit `minimal` is substituted with the
  nearest supported level, `low`. Explicit low/medium/high pass through
  unchanged.
- Apply Google's introductory pricing ($0.75 in / $3.75 out / $0.075 cached,
  per 1M) to Gemini 3.7 Flash and correct Gemini 3.6 Flash to the same rates.
  Both revert to $1.50 / $7.50 / $0.15 on 2027-01-01; noted at both call sites.

Resolves #14802

Ref: https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash
Ref: https://ai.google.dev/gemini-api/docs/pricing

* 📝 docs: Match the House Style for Promotional Rate Comments

Align the Gemini 3.6/3.7 Flash introductory-pricing notes with the existing
Sonnet 5 convention in the same file: one comment per group, naming the models
and the exact values to restore, so the manual follow-up is unambiguous.

No rate changes.

* ⬆️ chore: Bump `@librechat/agents` to 3.4.7 for Gemini 3.7 Flash Prefill

Unblocks this PR. `NO_PREFILL_GEMINI_MODELS` is model-enumerated in the agents
SDK, so 3.4.6 does not know `gemini-3.7-flash` forbids a trailing `model`-role
turn — editing an assistant reply and resubmitting would reach Google as a
prefill and return HTTP 400 on a model this PR adds to the default list.

3.4.7 (danny-avila/agents#412, released via #413) adds it. Verified the
published tarball: `3.4.6...3.4.7` touches only
`dist/{cjs,esm}/llm/google/utils/common.*` — the prefill array and its comment.
`dist/types` is byte-identical, so there is no API surface change.

Raises the declared range in both workspaces alongside the lock. `^3.4.6`
already permitted 3.4.7, but the fix is required rather than merely compatible,
so the floor should say so.
2026-08-14 01:12:42 -04:00
snapydziuba
6c46fd1252
📄 feat: accept PowerPoint template MIME type (#14761) 2026-08-14 00:22:01 -04:00
Danny Avila
ee8c0abe2d
🪝 feat: Execute Agent Plugin Command Hooks (#14755)
* 🪝 feat: Execute Agent Plugin Command Hooks

Implement the missing PluginHookExecutor boundary so deployment plugins'
ai.librechat/hooks/hooks.json documents execute instead of loading inert:

- Command executor runs handlers as child processes outside the API
  process: Claude-shaped JSON payload on stdin, exit 0 + JSON stdout as
  sanitized hook output, exit 2 blocks with stderr as the reason, minimal
  allowlisted environment plus PLUGIN_ROOT/PLUGIN_DATA, abort-signal kill
- Plugin loading carries the parsed hooks document on the contribution and
  threads hookCapabilities from startup, gated on the operator opt-in
  DEPLOYMENT_PLUGIN_HOOKS (off by default: parsed-but-inert with warning)
- Runs register every ready plugin hook onto the per-run HookRegistry after
  internal policy hooks, with once-per-conversation SessionStart dedup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Harden Plugin Hook Execution Boundary

Address CI and Codex/Copilot review findings on #14755:

- Break the agents -> plugins import cycle: the run seam now reads a
  PluginHookSource wired at startup (mirrors the tool-approval registry)
- Tighten plugin ask decisions to deny unless the run has HITL wiring,
  so an un-resumable interrupt can never strand OpenAI-compatible callers
- Scope cross-run dedup keys by authenticated user and handler identity:
  caller-supplied conversation ids cannot collide across principals, and
  sibling SessionStart handlers all fire; once handlers persist across runs
- Replace a literal NUL byte in source with an escape (file diffed binary)
- Kill the whole detached process group on abort, not just the shell
- Map exit 2 on events without a decision channel to preventContinuation
- Reserve PLUGIN_ROOT/PLUGIN_DATA against allowlist overrides, quote
  PowerShell args, cap captured output by bytes with one-pass decoding,
  and serialize payloads inside the executor's error boundary
- Fix import ordering flagged by the static checks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Close Plugin Hook Policy and Namespace Gaps

Address the second Codex review round on #14755:

- Drop updatedInput from plugin command outputs: hooks in one dispatch all
  receive the original arguments, so a plugin rewrite would reach the tool
  without the approval policy re-evaluating it (host-only now)
- Translate Claude tool aliases (Bash/Write/Edit/Read) to LibreChat runtime
  names in matchers, with reverse payload mapping, so Claude-authored guards
  fire instead of planning ready and never matching
- Key once-only state by declaration position as well as handler contents,
  so sibling declarations with identical handlers stay independent
- Thread sessionStartSource through createRun and mark the HITL resume
  rebuild as 'resume', so SessionStart matchers see the real lifecycle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Translate Regex-Form Claude Tool Aliases

Address the third Codex review round on #14755: alias translation now
substitutes word-bounded tokens, covering regex matchers like ^Bash$ and
^(Write|Edit)$ that the exact-token pass left registered against Claude
names and silently never firing. A regex whose alias sits inside a
character class or escape is rejected as unmapped so it fails loudly at
plan time instead of never running.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Alias Translation and Reuse Load-Time Plans

Address the fourth Codex review round on #14755:

- Add the WebSearch -> web_search alias so Claude-authored web-search
  guards fire against the LibreChat built-in
- Apply alias translation only to tool-name events; a StopFailure matcher
  like ^Bash failed$ stays untouched and keeps matching the error text
- Reuse each plugin's load-time hook plan at run registration instead of
  re-planning up to 512 handlers on every chat turn

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Translate Aliased Tool Inputs and Harden Hook Domains

- Present aliased tool inputs under Claude field names (file_path,
  old_string, new_string, including nested edits), so Write/Edit/Read
  guards see the fields they check instead of silently allowing
- Derive the alias table from canonical tool-name definitions
  (BashExecutionToolDefinition, CREATE_FILE_TOOL_NAME, Tools.web_search)
  instead of a parallel hand-authored table
- Reject matchers naming Claude built-ins with no runtime equivalent
  (Task, Glob, Grep, WebFetch, ...) as unmapped at plan time instead of
  registering guards that never fire
- Replace per-event Sets and Stop special-cases with an exhaustive
  EVENT_TRAITS record over HookEvent, so new engine events demand
  explicit semantics at compile time
- Move cross-run once-state behind a PluginHookOnceStore seam with a
  least-recently-marked memory default: active conversations refresh
  their keys each turn, so capacity eviction can no longer re-fire a
  conversation that is still in use; the seam admits a shared-cache
  store for multi-replica deployments
- Gate portable-only command handlers at plan time on Windows via a new
  supportsHandler capability (commandWindows or shell powershell
  required) instead of spawning bash that cannot exist
- Kill Windows hook process trees with taskkill /t on abort
- Require declaration indices on execution requests, stamped from the
  plan instead of defaulted at execution time

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Keep Group SIGKILL Escalation Armed After Wrapper Exit

An aborted hook whose descendant ignores SIGTERM could leak that
descendant: the wrapper shell's exit fired close, which cancelled the
scheduled group SIGKILL. The escalation timer is now never cancelled —
it is unref'd and killTree already tolerates a vanished process group,
so a redundant late sweep is harmless while a surviving descendant is
reliably killed at the grace deadline. killGraceMs is configurable on
CommandExecutorOptions, with a regression test driving a trap-protected
descendant past the wrapper's exit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Once Retention by Conversation and Reject Clear Source

- Restructure the once store around conversation scopes: registration
  touches the scope every run, so rarely-matching once handlers keep
  their keys while the conversation is active; eviction removes whole
  idle conversations (capacity counts conversations, not keys)
- Reject SessionStart matchers naming the clear lifecycle source at
  plan time — no LibreChat run-construction path emits clear, so the
  handler would plan ready and never fire; wildcard warning text now
  reflects the sources that actually occur
- Make the SIGKILL-escalation regression test real: the surviving
  descendant redirects its stdio away from the captured pipes so the
  wrapper's close fires while it is still alive, exercising the
  window a close-time cancellation would leak

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Bound Alias Tokens by Tool-Name Characters and Host Shells

- Translate Claude aliases (and reject unsupported built-ins) only when
  delimited by characters that cannot appear in a runtime tool name:
  action tool names preserve hyphens, so an alias embedded in a longer
  name like deploy-Bash-v2_action_example_com stays the literal tool
  name instead of being rewritten into a matcher that never fires
- Reject PowerShell-only command handlers on POSIX hosts at plan time
  (and skip them at runtime): bash cannot run PowerShell syntax, so the
  guard would fail open; a handler with both variants still runs its
  portable command
- Handle rejected asynchronous once-store calls: a failed touch logs
  instead of raising an unhandled rejection during run construction,
  and a failed markOnce lookup fails open per the store's documented
  over-fire direction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Probe Group Liveness Before Cancelled or Delivered SIGKILL

The never-cancelled escalation timer could signal a recycled
process-group id when an aborted hook's whole tree exits early in the
grace window. Escalation now probes the group with signal 0: close
cancels the timer only when the group is verifiably empty, and the
deadline re-probes before delivering the group SIGKILL, so surviving
descendants are still reaped while a fully-dead group never receives a
blind late signal. The residual probe-to-signal race is documented as
irreducible without pidfd support.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Gate Windows Escalation on Root-Process Liveness

Windows taskkill /t walks the tree from the root process, so once Node
observes the root's exit an escalation pass can reap nothing and a late
forced taskkill could only hit a recycled PID. The liveness gate is now
platform-aware in one helper: POSIX probes the process group with
signal 0, Windows checks the root's observed exit state, and both the
close-time cancellation and the deadline delivery consult it — no
platform retains a blind late signal. Orphaned SIGTERM-ignoring
descendants on Windows are documented as the platform limitation they
are without Job Objects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Payload Namespace to Declarations and Reap Stray Workers

- Reverse name/input translation now applies only to declarations whose
  matcher actually required Claude-alias translation: the plan records
  requiresToolNameTranslation per entry, so a native-authored matcher
  like ^create_file$ receives native tool names and fields instead of
  Claude-shaped payloads its guard never expected
- Coordinate the two dedup layers via a shouldExecute gate on the
  executor: a declaration suppressed by spent once-state declines
  before claiming the per-input dedup slot, so an identical handler
  under an overlapping matcher can still claim it and fire its own
  independent once-key instead of being permanently shadowed
- Reap process groups that outlive a successful hook: a backgrounded
  worker left running after normal wrapper exit gets the same
  term-then-escalate sequence an abort uses, since unsupported async
  handlers mean no lifecycle owns such processes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🧰 chore: Vendor Pocock Codebase-Design and Architecture Skills

Adds mattpocock/skills engineering/codebase-design and
engineering/improve-codebase-architecture (MIT, license included) under
.claude/skills so future sessions share the deep-module vocabulary
(module, interface, depth, seam, adapter, leverage, locality) and the
architecture-review process. Force-added past the /.claude/ gitignore
deliberately; relocate if project skills should live elsewhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 refactor: Extract Process-Tree Reaping Into a Reaper Module

Tree lifecycle — five of the last seven review findings — lived as
event-handler wiring inside runCommand with its invariants in comments.
It now sits behind a two-method seam: createReaper(child, graceMs)
exposes reap() and onClose(), hiding the term-grace-escalate state
machine, the per-platform liveness gates, the recycled-id guards, and
the clean-exit sweep. The executor shrinks to capture-and-parse, and
the reaper is unit-tested directly with real process trees through its
own interface instead of only via whole-executor integration runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Scope Translation Per Alternative and Sweep at Root Exit

- Track which runtime tool names alias translation produced, so a
  mixed-namespace matcher like Bash|create_file presents Claude-shaped
  payloads only for bash_tool invocations while the natively-authored
  create_file alternative keeps native names and fields; a capability
  omitting the produced-names list keeps declaration-wide translation
- Sweep the process tree at root exit as well as close: a backgrounded
  descendant holding the captured pipes delays close until it dies, so
  the exit-time sweep terminates it promptly instead of stalling the
  hook until its timeout aborts
- Pass the primary agent's resolved model and identity into the plugin
  hook context, so SessionStart payloads carry model and agent_type
  instead of always omitting them

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Default Wildcard Declarations to the Document Namespace

- Matcherless (or wildcard) tool-payload declarations now inherit the
  hook document's Claude namespace: with no alternatives to carry
  namespace evidence, the plan marks them for declaration-wide reverse
  translation, so a wildcard guard inspecting standard Claude names and
  fields sees Write/file_path instead of silently failing open on
  native payloads; PostToolBatch entries translate the same way
- Recognize aliases delimited by regex metacharacters: dots leave the
  tool-name boundary class (runtime names never contain them — action
  ids underscore domain dots), so ^Bash.*$ translates to ^bash_tool.*$
  instead of registering a guard that never fires
- Expand Claude's ${CLAUDE_PLUGIN_ROOT} spelling in hook commands and
  export it in the child environment alongside PLUGIN_ROOT
- Scope SessionStart once-keys by lifecycle source, so a startup firing
  no longer suppresses the conversation's resume rebuild

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Normalize Claude Structured Hook Output

Stock Claude hooks return decisions under hookSpecificOutput
(permissionDecision/permissionDecisionReason), surface context there,
and use continue:false plus the legacy approve/block decisions — none
of which the sanitizer's native field names recognized, so a guard that
works in Claude silently allowed in LibreChat. Parsed JSON now passes
through a dialect normalizer first: hookSpecificOutput fields map to
decision/reason/additionalContext, continue:false becomes
preventContinuation, approve becomes allow, and block becomes deny on
events that block by denying. Native fields win when both dialects
appear, and the ask-to-deny gate applies to the Claude dialect too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Validate Native Decisions and Slim Once Keys

- Strip malformed native output fields before the dialect merge, so a
  placeholder like {"decision":null} can no longer suppress a valid
  Claude permissionDecision into a silent allow; only recognized
  decision tokens take precedence
- Preserve the caller's working directory in hook payloads: cwd now
  reports the run's session context instead of the plugin installation
  path, which commands already receive as PLUGIN_ROOT and which the
  executor still uses as each process's working directory
- Store a compact sha256 digest instead of the full serialized handler
  in once keys: declarations may carry 32 KB commands and 256 args, and
  the previous key embedded them in every retained conversation scope

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

* 🪝 fix: Validate Decisions Per Event Channel and Control Post-Tool Blocks

- Accept native decision tokens only from the target event's own
  vocabulary: "continue" is valid on Stop but malformed on a tool
  event, where it previously survived validation, blocked the Claude
  dialect merge, and was then dropped by sanitization into a silent
  allow
- Translate a structured "block" on events with no deny channel
  (PostToolUse, PostToolUseFailure, and the other prevent-trait events)
  into preventContinuation with the block reason as stopReason, instead
  of discarding it and returning a reason that controls nothing
- Document why LibreChat runs supply no payload cwd: tool paths address
  a remote code-execution sandbox rather than the API host where hook
  commands run, so no host directory describes the run

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWXQZD2WzeAsvee4eRdWWc

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 13:21:15 -04:00
Danny Avila
1596df724a
🫆 chore: Remove Published Credential Defaults (#14680) 2026-08-07 07:25:05 -04:00
Dustin Healy
c6bb77325a
🔗 feat: Admin Panel Link in Settings for Admins (#14662)
Expose ADMIN_PANEL_URL through the startup config for users holding the
access:admin capability, and render an Admin section in Settings > General
with an external link to the admin panel. The URL is omitted server-side
for unauthenticated requests and users without admin access.
2026-08-06 12:40:44 -04:00
Danny Avila
33c8801ba6
🌊 feat: Wire Adaptive Stream Smoothing Across Google, Bedrock, and Zero-Disable Semantics (#14660)
* 🌊 feat: Wire Adaptive Stream Smoothing Across Google, Bedrock, and Zero-Disable Semantics

With @librechat/agents 3.4.0 smoothing defaults ON (25ms adaptive) for
every provider; this completes the LibreChat side:

- google: fix the unguarded endpoints.all clobber and wire streamRate
  into llmConfig._lc_stream_delay — previously read from config and
  silently dropped, making google streamRate a no-op end to end
- bedrock: wire streamRate (endpoint + endpoints.all) — previously
  absent entirely, so bedrock smoothing was unreachable from config
- anthropic/openai: nullish guards so streamRate: 0 survives as the
  explicit smoothing disable; drop the azure 30/17 hardcoded fallback
  the SDK default now supersedes
- delete dead createHandleLLMNewToken (no call sites since #6886, and
  LangChain backgrounds callbacks so a sleep there never paced anything)
- schema: streamRate gains .min(0) and docs; example docs updated,
  including pairing STREAM_DELTA_COALESCE_MS with the smoothing tick

* 🩹 fix: Address Review — Keep Published Shim, Typed Delay Access, Scoped Docs

- restore createHandleLLMNewToken as a @deprecated compatibility shim:
  it ships in the public @librechat/api root, so removal is reserved for
  a major release
- assign llmConfig._lc_stream_delay via the SDK's typed property (3.4.0
  StreamSmoothingOptions) instead of Record<string, unknown> casts at
  all five sites
- scope the 25ms-default wording to agents SDK-backed providers (legacy
  Assistants/Ollama still per-chunk sleep at DEFAULT_STREAM_RATE=1) and
  clarify coalescing guidance when streamRate: 0
2026-08-06 12:39:12 -04:00
Ravi Kumar L
7775f25b0d
🪢 fix: disable central fanout media uploads for langfuse (#14642)
* fix(langfuse): disable central fanout media uploads

* test(langfuse): cover fanout media policy in run config

* chore(deps): bump agents for Langfuse media policy

* chore(deps): bump agents to 3.3.13

* fix(langfuse): gate central fanout media uploads
2026-08-05 17:23:54 -04:00
Danny Avila
58cdd9cd8f
⚡ feat: Coalesce Redis Streaming Delta Publications into Windowed Batches (#14614)
* ⚡ feat: Coalesce Redis Streaming Delta Publications into Windowed Batches

Every streamed delta currently costs two Redis EVALs (durable append + sequence-allocating publish), and the publish round trip is awaited inside the provider-stream consumption loop. Behind STREAM_DELTA_COALESCE_MS (default off), message/reasoning/run-step deltas now buffer for a small window and flush as one CHUNK_BATCH frame: a single INCRBY reserves consecutive per-event sequences and one EVAL publishes the batch, while a matching batched XADD keeps the durable chunk log on the same cadence so the resume frontier's log-vs-counter timing assumptions hold. Subscribers unpack batch frames at ingress into individually sequenced chunks, so the reorder buffer, duplicate drop, and force-flush behavior are unchanged. Durable, steer-receipt, created, and terminal emissions stay on the awaited per-event path and act as ordering barriers that flush any pending window first; terminal claims flush both sides before the status CAS so a warm tail cannot fence against its own completion.

Benchmarked on local Redis (per-scenario RESETSTAT, INFO cpu/commandstats): at 100-200 ev/s a 25ms window cuts EVAL calls 67-82% and Redis engine CPU 52-70%; at the incident's 40 ev/s it halves EVALs while a 20ms window batches nothing (avg 1.0/frame). Producer await stall drops from ~0.9ms/delta to ~0.05ms/delta, matching the previously measured 16-18% USE_REDIS_STREAMS wall-time overhead. Delivery p95 stays under one window (27-28ms at 25ms).

* 📝 docs: Document STREAM_DELTA_COALESCE_MS in .env.example

* 🚧 fix: Drain Coalesced Windows Before Abort and Shutdown Terminal CAS

abortJob and the graceful-shutdown finalizer claim terminal state through their own CAS calls rather than claimTerminalJob, so the pre-CAS coalescer flush did not cover them: a window tail buffered at abort time flushed against the already-aborted status, fenced (-1), and the false receipts retired the healthy runtime and error-closed subscribers before the abort FINAL frame. Extract the flush into flushCoalescedStreamBuffers and call it from all three terminal paths that can interrupt a live emitter (claim, abort, shutdown); the abort call sits ahead of the content snapshot so a chunk-log reconstruction also observes the flushed tail. Regression test aborts mid-window and asserts the tail is delivered with no subscriber error (fails without the fix). Paused-state terminals (approval expiry, pause-persistence timeout) need no flush: the pause's durable barrier already drained the window and nothing streams while paused.

* 🛡️ fix: Keep Fence Retire a Lost-Signal Backstop on Aborted Runtimes

A cross-replica abort claims its terminal CAS on the aborting replica, so the owner cannot drain its coalesced window pre-CAS; the window flush then fences against the aborted status. When the flush timer lands in the CAS-to-FINAL gap, the false receipts retired the owner runtime and detached its SSE handlers, so the abort FINAL published moments later was dropped and attached clients hung until client-side reconnect. The stop signal reaching the owner (~1ms pub/sub) is proof the abort/replacement flow owns terminal delivery and cleanup, so retireRuntimeAfterDurableFence now returns early for runtimes whose abort signal already landed. The forced teardown remains exactly for its original purpose: a fence observed by a NOT-yet-aborted owner, which is the lost-signal case. Regression test pins the race deterministically via the abort beforePublish hook (which runs between the CAS and the FINAL), forcing the owner flush there: without the guard the FINAL is dropped and the subscriber never completes; with it the FINAL delivers cleanly.

* 🧰 fix: Gate, Isolate, and Bound the Coalesced Delta Path

Three hardening fixes for the coalescing prototype. The manager now enables the fire-and-forget delta path only when the configured services actually batch — presence of flushPendingChunks/flushPendingAppends is the advertisement — so a custom transport that only implements emitChunk keeps the awaited per-event ordering contract even with STREAM_DELTA_COALESCE_MS set, and a batching transport is never paired with a per-event store (which would let the durable log trail the sequence counter by a full window). Batch unpack isolates each event: a throwing subscriber callback now degrades exactly like a lost individual frame (that sequence stalls until the reorder force-flush) instead of discarding the batch tail whose sequences were already reserved. And the emitter tracks outstanding coalesced receipts per stream, awaiting one once 256 accumulate: healthy settlement is a window plus a round trip so the count sits in single digits and the await never runs, while a stalled Redis now paces the producer exactly like the flag-off awaited path instead of accumulating batches, resolver closures, and queued commands without bound.

Unit tests cover the capability gate (hint shape and await behavior for capable, incapable, and window-off configurations) and the backpressure threshold; an integration test pins the unpack isolation (fails without it: the batch tail vanishes instead of recovering via force-flush). Benchmark re-run confirms the counter and gate cost nothing measurable: identical EVAL counts and the serial drain still enqueue-bound.

* 🎛️ fix: Make STREAM_DELTA_COALESCE_MS the Single Coalescing Switch

The per-instance coalesceWindowMs constructor overrides could disagree with the environment the manager reads: overrides without the env silently did nothing, and an enabled env with an override of 0 selected the un-awaited manager path while both services published and appended per-event. Nothing in the repo passed these options, so remove them — the transport, the job store, and the manager now read STREAM_DELTA_COALESCE_MS through one resolver, making a half-enabled process unrepresentable rather than documented against. The capability-presence gate remains for services that do not implement batching at all.

* 🧪 fix: Observe Abort Tail Delivery Before Terminal Teardown in Test

The same-replica abort test waited for the coalesced tail only after abortJob returned, but abortJob's finally-block cleanup tears down local subscription state and publish receipts acknowledge Redis execution, not subscriber delivery. Single-node pub/sub delivers sub-millisecond so the frames always won locally; under the CI Redis Cluster they cross the cluster bus and lost the race, timing out the assertion. Await delivery concurrently with the abort instead — the pre-CAS flush publishes the tail several round trips before the teardown, so observing during the call is deterministic in both topologies. Test-only change.
2026-08-05 13:40:18 -04:00