Commit graph

38 commits

Author SHA1 Message Date
lia-by-librechat[bot]
0e3c9088f6
🫰 feat: Configure Tool Approval Modes Per Agent (#16707)
* 🫰 feat: Configure Tool Approval Modes Per Agent

* fix: Fence Remembered Tool Approval Authority

* fix: Unify Tool Approval Execution Invariants

* fix: Revalidate Learned Approval at MCP Invocation

* fix: Preserve One-Time MCP Review at Invocation

* fix: Execute Reviewed Calls on Templated MCP Connections

* fix: Scope Approval Witnesses to Executing Agents

* fix: Retain Approval Bindings for Hook-Rewritten Inputs

* fix: Pin Review Authority and Retire Settled Tool Calls

* fix: Fingerprint Effective MCP Request Headers

* fix: Preserve Routing Around Renewable MCP Token Fragments

* fix: Bind Tool Consent to OAuth Credential Generations

* test: Keep Mobile Approval Save Outside Drawer Escape Race

* fix: Fence OAuth Consent at MCP Transport Retries

* fix: Let Shared Agent Viewers Reset Personal Consent

* fix: Preserve Renewed Consent Across Reset Races

* fix: Resolve Verified Aliases for Personal Tool Reset

* fix: Fence Renewed OAuth Consent and Credential Review

* fix: Require Foreground Execution for Review-Only MCP Calls

* fix: Pin Invocation Ownership and Stable MCP Consent

* fix: Fence Superseded Tool Consent Completion Writes

* fix: Scope Consent Epochs to Effective MCP Authentication

* fix: Scope Approval Admission to Reachable Agent Tools

* fix: Preserve Approval Epochs for Runtime-Detected OAuth

* fix: Address Tool Approval Review Findings

* fix: Preserve Captured Consent When Tool Approval Is Inherited

* fix: Preserve Approval Admission Through Lazy Metadata

* fix: Fence Programmatic Reviews and Unresolved Lazy Approval Keys

* fix: Account for Normalized and Skill-Contributed Lazy Tools

* fix: Scope Lazy Approval to Selected Tool Spellings and Servers

* fix: Honor Disabled MCP Capability During Lazy Approval Admission

* test: Preserve Approval Merge Regressions After Rebase

* fix: Preserve One-Time Review During Non-OAuth Grant Outages

* fix: Retain One-Time Consent Across Resume Recovery

* fix: Preserve Reviewed Consent Through Transport Outages

* fix: Preserve Case-Variant Lazy Approval Hook Admission

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-04 13:14:48 -04:00
lia-by-librechat[bot]
19092e55bd
🖲️ fix: Enable Running Chat Menus and Preserve Renamed Titles (#16697)
Reconciles current dev while preserving manual title ownership, existing menus, and configuration regressions.

Co-authored-by: Lia <lia@librechat.ai>
2026-10-03 23:17:15 -04:00
Marco Beretta
fb1dd57a9b
🔓 feat: Remember Tool Approvals Per Conversation (#16557)
* feat: Remember Tool Approvals Per Conversation

Adds an opt-in "Always allow" choice to the tool approval card. When
endpoints.agents.toolApproval.allowAlways is true, the server marks eligible
paused calls with allow_always; approving with scope "session" stores that
exact tool name on the conversation, and later runs in the same conversation
fold the stored names into the policy as exact-name allow entries.

The server enforces every step: the resume route rejects a session scope the
pause did not offer, eligibility is rechecked against the live policy before
storing, the list is written only through an owner-scoped bounded method, and
generic saves and imports cannot set or clear it. Admin deny and ask rules and
programmatic hooks still win, stored names are ignored under dontAsk or when
the flag is off, and native code tools stay under the code approval picker.
The flag defaults to false, so existing deployments keep prompting.

* fix: Share the Effective Approval Policy and Persist Allows After Resume Fences

Offering, resume re-check, scheduled admission and createRun now read one
policy from buildEffectiveToolApprovalPolicy: endpoint rules healed against
MCP alias spellings, then the conversation's remembered tools. A call held by
a healed ask rule is no longer offered "Always allow", and a scheduled fire
whose known tools are all remembered is no longer rejected as pause capable.

The remembered-tools cap and name length cap move to
toolApproval.allowAlwaysMaxTools and allowAlwaysMaxToolNameLength (defaults
64 and 256). The card stops offering the choice for new tools once the cap is
reached instead of silently approving once.

The resume route stores remembered tools only after the schedule and project
context fences pass; a storage failure still approves the call once.

* fix: Offer Always Allow Only When It Will Hold

Offer allow_always only when storing the name would auto-approve the next
identical call on the run's own path: every MCP spelling of the tool passes
the healed static policy and no programmatic or plugin approval hook can
apply to it (the pause payload does not record which hook asked).

Remembered names now expand to every alias spelling of the same tool, and a
deny or ask on any spelling keeps the tool ineligible. A table-driven spec
walks the invariant through the real hook wiring.

* fix: Recheck Approval Hooks Before Remembering a Resumed Approval

The resume-time eligibility recheck now applies the same programmatic and
plugin PreToolUse hook check as marking, with a hook context built from the
resume request and the live plugin hook source. A hook registered after the
pause turns the choice into a one-time approval: nothing is stored and the
resume continues.

Document that toolApproval.allowAlways should be enabled only once every
replica supports it, since older replicas ignore the choice.

* fix: Judge Always Allow Offers Against the Run's Live Aliases and Active Cap

* fix: Charge Only New Tool Names Against the Remembered Approval Cap

* fix: Recheck Remembered Approvals Against the Paused Run's Aliases

* chore: Sort Imports in Agent Run After Rebase
2026-10-03 11:47:32 +02:00
Lia
04461d4cdb ⛴️ fix: Keep MCP App and Speech Limits in One E2E Config
(cherry picked from commit e20558b736)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16457
2026-10-01 03:46:18 +00:00
Marco Beretta
2be35b2f91 💬 refactor: ChatForm UI redesign with unified palette (#14473)
* 💬 refactor: ChatForm UI redesign with unified palette

Replace the scattered composer controls (badge row, tools dropdown, attach
menu, floating steer chips) with one system: a searchable palette with
favourites, a composer bar holding active tools plus effort and speech
controls, a tray staging files and manual context, and a queue rail that
buffers messages while a reply is running.

Collapses the 94 development and review-fix commits of PR #14473 into one
commit so the branch could be rebased onto dev once; the upstream merge of
this pull request is a squash merge either way. The pre-rebase history is
kept locally at tag prerebase-backup (afde3e4b9a).

* fix: finish the dev rebase of the composer redesign

The rebase onto dev left four defects the merge itself produced, plus one
review finding the branch had never answered:

- ChatForm's `min-w-0` JSX comment lost its closing brace, so the file did
  not parse at all.
- `useSteering` carried both dev's local draft consumer and the host-injected
  one under the same name; the host's `consumeDraft` owns the storage key
  autosave actually parked on, so the local copy is gone.
- `useUploadOptions` kept dev's `awaitingAgentProvider` gate after the merge
  replaced the block that derived `isSavedAgent`; the predicate is back, from
  the same `isEphemeralAgentId` the shared allowance helper uses.
- `MessagesView` still reserved a band for the in-flight steer overlay and
  read `steerOverlayHeightFamily`, which nothing writes now that pending
  steers render inside the streaming reply. The reservation is gone and
  `ScrollButton`'s lift is optional; composer panels keep standing the button
  down through `composerOverlayCountFamily`.
- The aggregate MCP chip keyed on `mcp:pinned`, which a server literally
  named `pinned` also produces (`mcp:${serverName}`), so both entries shared
  one React key and one packing slot. It uses a key outside that namespace.

Also drops the locale keys whose only consumers the redesign deleted, ports
dev's extracted-text delivery path into `useAttachExisting` (every
attach-existing surface routes through it now), repoints the mock e2e
built-in helpers at the palette, and repairs test fixtures the merge left
stale (a query mock that blanked agent metadata, a Jotai store mismatch, an
ambiguous tray query, an ephemeral-agent fixture standing in for a saved one).

* test: cover the composer redesign's acceptance scenarios end to end

Twelve mock-harness scenarios, one `@scenario:` tagged test each, over the
behaviours this change is accountable for: the single searchable palette
(open, search, favourite across a reload, keyboard-only operation), the
composer bar and tray (active tool chip, a staged file riding with the
message, a per-turn thinking level that applies to the next turn only), the
queue rail (Enter buffers during a run and drains after it, keyboard reorder
changes the send order, the draft survives run completion), and the composer's
portaled surfaces (Stop reaches the run from inside a portal, a pending tool
approval in the bar resumes the run).

Two contract scenarios are deliberately absent, with reasons: split-pane Stop
isolation is not runtime-reachable because `ChatRoute` mounts one pane, and
code-approval mode mid-run needs an attached stateful code environment the
mock lab does not configure, so dev's `ChatForm.approvalMode` unit spec stays
its coverage.

* fix: make the composer's acceptance run and the whole client suite green

The first verification run of the twelve scenarios passed six and failed six,
and the full client jest suite (which CI runs sharded, unlike the diff-scoped
local precheck) failed fourteen tests. One of each group was a real defect; the
rest were queries and fixtures still aimed at surfaces the redesign replaced.

Defects:

- `useQueueDrain` held its send lock after a revealed queued turn settled, so
  the next server-owned row could never be selected. The reveal path now owns
  its own lock and releases it when the revealed turn clears.
- `ChatForm` recomputed the during-run slot without `consumeComposer` in its
  dependency list, and still carried a landing flag the host prop replaced.

Tests and lab:

- The five `ChatForm.attachments` focus cases now drive the palette dialog and
  the tray's quote chips instead of the deleted attach menu and quote popup,
  with the same strict focus assertions; the harness gains the `main` landmark
  the palette portals into, and the touch fixture a complete `matchMedia`.
- `Palette.spec` no longer closes a hoisted `jest.mock` factory over a later
  `const`; `useSteering.spec` follows dev's toast wording.
- The six failing scenarios were repointed at what this lab actually renders:
  uploads go through the palette's local-computer row, the reasoning control
  needs `interface.parameters`, the queued-row contract is the row and its
  drain rather than a control the submit consumes, the palette's keyboard path
  uses a built-in the lab grants, the slow-reply wait uses the fixture's chunk
  marker, and the approval scenario selects its agent through the builder flow
  the passing approval specs use.

* fix: close the eight findings Codex raised on the rebased head

Codex reviewed the rebased heads and reported eight issues; all eight were
real. Two of them concern code this session added.

Behaviour:

- A custom or OpenRouter endpoint that declares neither `paramDefinitions`
  nor a `reasoningFormat` no longer gets a reasoning control inferred from the
  generic settings list. Capability now has to be declared (a reasoning-key
  parameter definition, a non-disabled `reasoningFormat`, or a known reasoning
  provider as the default params endpoint); a `disabled` format always
  resolves nothing, and every consumer — the composer control, the pre-send
  validation and the server — asks the same resolver, so a gateway backed by a
  non-reasoning model can no longer offer a level the turn would fail on.
- Restoring a queued attachment keeps its `llmDeliveryPath`, so an extracted
  document is routed as its text rather than as raw provider content.
- The placeholder-to-durable reasoning migration now requires this pane's own
  live submission, so opening another conversation from a new chat no longer
  carries the pending level into it. The created/sync case it was written for
  still carries.
- The queue rail treats an acknowledged server-owned row as an ordering
  boundary: local rows still reorder among themselves, a server-owned row
  cannot move, and a move that would cross one is refused out loud — the
  queued-turn API has enqueue, list and cancel only, so a client-side reorder
  across that boundary could not be made authoritative.
- The palette resolves an agent's skill scope with `resolveAgentSkillsScope`
  instead of treating an empty list as an open catalog, and the attach target
  resolves `useResponsesApi` with `resolveUseResponsesApi`, matching what the
  run will actually do.

Policy:

- The reasoning-override behaviour moved out of the CJS middleware into
  `packages/api/src/modelSpecs/reasoningOverride.ts`, which returns a
  discriminated result; the middleware is wiring again, and the behavioural
  cases moved with it.
- `showComposerTips` and `showFilesDialog` are Jotai atoms now, the tips
  preference keeping its storage key and reaching the composer as a prop.

* fix: make the redesigned composer reachable from the mock e2e helpers

CI's Playwright shard failed four `file-provisioning` tests on this branch
because the shared upload helpers still clicked `#attach-file-menu-button` and
`#attach-file-button`, entry points the redesign removed — attaching now goes
through the palette, which also owns the file input. Both helpers drive the
palette now, `uploadViaLegacyOption` mapping its two destination names onto the
palette rows that route to the code environment and the vector store, and the
specs that drove the old surfaces directly (`code-upload`, `thread-renderers`,
`unified-upload`) follow.

Also closes the last client/server disagreement the acceptance run exposed: a
custom endpoint that declares a reasoning key in `paramDefinitions` had that
declaration merged into the provider defaults and then discarded by model-name
inference, so the composer offered a level for `mock-model-a` that the server
rejected with "Invalid reasoning override". An explicitly declared key now
selects the merged setting before inference, while an undeclared generic custom
or OpenRouter target still resolves nothing.

The twelfth acceptance scenario is rescoped: a paused approval keeps the
composer usable and resumes to completion, but the composer-hosted approval
panel and chip are not asserted, because the pending action is keyed by the
stream's conversation id while the composer stays bound to the new-chat
sentinel — in code byte-identical to dev, and mirroring the write onto the
sentinel did not surface the panel. Tracked at berry-13/LibreChat#39.

* test: finish the acceptance run and keep the client typecheck green

- The thinking scenario read `reasoningOverride` from a `userMessage` wrapper
  the chat request does not have; the trace shows the request carrying
  `{ key: 'effort', value: 'max' }` at the root, so the assertion reads it
  there. Source behaviour was already correct.
- The approval scenario timed out on mobile because the Agent Builder drawer
  stayed open after selecting the agent and swallowed the pointer events meant
  for the composer; it now closes the sidebar and waits for the drawer to go
  inert before sending.
- Two specs typed a named custom endpoint's label as an `EModelEndpoint`, which
  CI's client typecheck rejects (the label is exactly what the case is about).

All twelve acceptance scenarios now pass in desktop light, desktop dark and
mobile emulation.

* fix: close the nine findings Codex raised on the verified head

Two were defects in code this session added while closing earlier findings:

- A reasoning key declared in `paramDefinitions` but absent from the generic
  settings list was accepted as capability and then never inserted, so the
  resolver exposed and validated the generic `reasoning_effort` instead. A
  declared key is now appended to the effective settings and resolved before
  any fallback, for `effort`, `thinkingLevel` and `thinkingBudget` alike.
- The queue's reorder guard validated from the destination row's own index, so
  `target === index` rejected every pointer drag and only keyboard reordering
  worked. It validates the dragged row's identity now; the server-owned
  boundary, its announcement and the disabled grips are unchanged.

The rest:

- Stop no longer falls through to the document-wide control once a focused
  pane has been resolved: focus in an idle pane while another generates used
  to abort the other pane's run.
- `PendingSteers` is mounted once in `MessagesView`, above the renderer
  switch, so a sent or failed steer keeps its in-flight, retry and
  send-as-new surface with `FLAT_THREAD` off, where the recursive renderer
  had none and the composer no longer carries a fallback.
- The file-list policy moved out of the CJS route into
  `packages/api/src/files/list.ts`, and its cap is `interface`-level config
  (`fileListLimit`, default 100) instead of a literal in the route.
- The steer-arm confirmation window is `interface.steerArmConfirmationTimeoutMs`
  with its current 10s as the default, travelling the same path as the other
  client-consumed timings.
- The pause predicates judge the active branch's tail rather than the last
  assistant in array order, so an inactive sibling branch can no longer
  enable or disable the inline escalation control.

* fix: move persisted reasoning validation into api package

* fix: restore composer entry points and defaults

* style: normalize reasoning helper formatting

* test: open mobile sidebar before account menu

* test: use settled sidebar layout for mobile account menu

* test: handle empty mobile sidebar in account scenario

* test: target account menu in mobile drawer

* fix: keep composer surfaces and mobile drawer reachable

* fix: let thread approvals receive pointer events

* fix: stabilize composer steering and shortcuts

* fix: commit queue drags and centralize file modal

* fix: return a canceled steer's text to the composer

Canceling a pending steer dropped the words the user had written: the
bubble disappeared and nothing carried the text back, so a reclaim that
the server confirmed still lost the message.

Hand the reclaimed text to `pendingComposerTextByConvoId`, which the
composer already drains on its first render, and append rather than
overwrite when the composer holds a draft the user typed while the cancel
request was in flight.

* fix: close the six findings Codex raised on the pre-rebase head

Pending steers keyed off the Recoil conversation id, which lags the route
during warm-cache navigation, so the surface could sit at the destination
thread's tail while its Cancel and Escalate acted on the source run. It
now keys off the rendered tree, as the mount window already does.

Arming a steer removes the control that was activated, dropping keyboard
focus on the document body. `useSteerEscalate` already exposes `onArmed`
for exactly that; the bubble now passes one that moves focus to the row's
Cancel action.

Opening the file manager from the account menu left its dialog with no
opener: the modal rejects an active element inside a menu, and nothing
supplied a fallback, so closing it returned focus to the body. The opener
is published alongside the open flag and cleared on close, which keeps the
shortcut path's composer-focus behaviour intact.

A steer the server rejected was swept into the queue with no marking, and
the run-end drain then sent those words as a fresh turn without the user
touching Retry or Send as new. Converted failures now carry
`needsExplicitSend`, which the drain skips the same way it skips a row the
rail is mid-edit on.

The queued-turn reconciliation window was a hard-coded 60s, unreachable by
an operator behind a slow proxy or a delayed read replica. It ships as
`interface.queuedTurnReconciliationTimeoutMs` with that value as default,
beside the steer arm timeout it belongs with.

Reasoning-override shape validation moved to
`parseReasoningOverrideRequest` in packages/api, leaving the middleware to
map a rejection onto its own error response.

* fix: keep the composer's reasoning level in step with the parameters panel

`stableConversation` memoized the conversation without any reasoning field
in its key, so editing the level in the Parameters panel produced a new
conversation object that every listed dependency ignored. The composer
went on showing the old level while the next ordinary send already used
the new one.

* test: stop the sidebar helper waiting for a drawer desktop never renders

`ensureSidebarOnScreen` measured `#mobile-drawer` whenever the sidebar had
no Pinned region. Desktop renders no drawer, and `boundingBox()` on a
locator that matches nothing waits for it, so every scenario that reached
that branch at a desktop viewport timed out instead of continuing. Check
for the drawer's absence rather than measuring it.

* test: open the mobile drawer from its inert state, not its geometry

A closed drawer stays mounted and keeps its layout box, so measuring the
Pinned region reported it on screen while every control inside was still
untappable. `inert` is the state the component actually sets when it
closes, so the helper reads that and clicks the header opener before
anything else measures.

* fix: return a canceled steer whole, through one recovery boundary

A steer that stops being a steer was handed back three different ways,
and each one lost something. The immediate cancel wrote `steer.text`
into the composer handoff atom, dropping the attachments, quoted
excerpts, manual skill picks and reasoning override that `settleReclaimed`
then deleted with the pending object. The cancel that won its race
against its own POST cleared the pending marker and restored nothing at
all, so those words disappeared. A definitively rejected steer could
only be resent unchanged, because the row offering to edit it was gone.

Give the recovery one boundary. `useSteerRehome` hands the whole steer
to the composer through the guarded restore the queue rail already uses,
and converts it into an ordinary queued follow-up when the composer
refuses: a draft in the box, another chat on screen, a paused question,
or a composer that unmounted while the reclaim was in flight. The chip
goes either way, since a pending row claiming the server still holds the
words is a lie once they live somewhere else.

The composer is a sibling of the message list rather than its ancestor,
so the thread had no way to reach that restore. The pane's host now
holds the reference and both sides read it there, which also means a
reclaim resolving after the composer is gone refuses instead of writing
into a dead form.

Because the restore refuses rather than overwrites, a draft typed while
a cancel is in flight is no longer glued to the returned steer; the
steer becomes its own queued message and the draft is left as typed.

* test: drive the surfaces these scenarios actually reach

Four scenarios were failing against surfaces the user never touches.

The approval scenario clicked Approve on the thread's copy of the
pending card. The composer's review panel floats over the tail of the
thread and owns the batch submit, so that copy sits behind it and the
click retried until the test timed out. Drive the panel instead; the
thread card stays asserted as the record of the request.

The palette scenario counted rows the moment the dialog painted, then
expected clearing the filter to restore that number. The catalog arrives
in waves: the MCP server list is released by the app-wide warmup timer
and the skills catalog only starts fetching once the dialog's open
effect has run, so the count kept climbing under the test. Wait for the
last wave and for the count to hold still, then snapshot.

The sidebar helper opened the mobile drawer twice through two signals
that disagree. It waited for `inert` to clear, which happens on a state
commit several frames ahead of the slide, then fell through to a
bounding box still reading off-screen and clicked the opener again, into
a machine already opening. One open, through `inert`, then wait for the
drawer to settle in the viewport.

Splitting the cancel scenario follows the recovery change: an empty
composer takes the steer back, and a staged draft keeps the box while
the steer becomes a queued message of its own.

* fix: hold a server-rejected steer for an explicit send

Editing a definitively failed steer falls back to the queue when the
composer will not take it. Queued like any other follow-up, the run-end
drain would send it the moment the reply finished, which turns the
server's refusal into an automatic new turn carrying the payload it
refused.

Mark that fallback the way the rest of the rejected-steer path is
marked, so the row waits for the user to send it.

* test: restore two assertions this branch broke while rewriting specs

Both specs were updated for the redesigned surfaces and both lost a step
they needed, and both fail deterministically in CI as a result.

thread-fold pressed Enter mid-run and waited for a queued row, under a
comment claiming Enter queues by default. It does not: the default
during-run action is steer, so Enter produced a pending steer and the
queued row the next line waits for could never appear. Ctrl/Cmd+Enter is
what takes the other action, which is the one this test needs.

mcp-oauth-readiness asserted on the server card's Cancel button straight
after clicking Connect. Connect opens the OAuth dialog over the panel,
and it is modal, so while it is open the card underneath is marked
aria-hidden and Cancel is not in the accessibility tree at all, which is
why the failure reads "element(s) not found" rather than "not visible".
The rewrite dropped the dismissal the previous flow performed; restore it
as a helper both call sites share.

* test: reach the account menu without the pinned section

The two account-menu tests borrowed `ensureSidebarOnScreen`, which also
settles the Pinned section's placement for the pinned specs. They never
touch that region, and on a run where an earlier test left a pinned
conversation the section exists but has not been laid out yet, so both
tests spent fifteen seconds polling it and failed on the wait rather than
on anything they assert.

Give them a helper that does only what they need: open the mobile drawer
from `inert`, the one signal that distinguishes a closed drawer from an
open one while it stays mounted, then wait for the slide to reach the
viewport. Desktop renders no drawer and needs nothing.

* test: open the drawer by what it shows, not one early read

The helper decided once, from a single read of `inert` taken right after
navigation. On a fresh mobile load the drawer can still be mounting, so
that read reports a state the app has not settled into, the helper
returns believing the drawer is open, and the account menu inside it is
never shown.

Retry against the thing the caller actually needs, the account menu being
visible, re-checking before each attempt. While the drawer is open its
control is labelled "Close sidebar", so the retry cannot toggle it shut.

* fix: keep the composer observing MCP authorization where its tools apply

The old badge row mounted its provider with observeToolAuthorization set
whenever the composer showed ephemeral tools, so on a plain endpoint the
chat kept MCP connection status and discovered tools in the query cache.
The redesigned composer mounts the same provider without the prop, so it
defaults to off and nothing on the chat side loads that state any more.

The Agent Builder's tool library reads that cache. With it empty, every
server reads as Unknown with no tools, and clicking one opens its detail
dialog offering to connect instead of adding it to the agent.

Pass the same gate the badge row used: showTools is the old
showEphemeralBadges expression.

* test: let the palette close before reopening it, and name what is asserted

enableBuiltinTool pressed Escape and returned while the palette was still
animating out. A caller that reopened it straight away resumed the same
popover, scroll offset included, so the Attach rows at the top sat outside
the virtualized window and upload helpers timed out looking for them. Wait
for the dialog to be gone before returning.

The staged-file assertions matched the bare file name, which the palette
now also lists under recent uploads while it closes; scope them to the
tray they are about.

The OAuth dialog dismissal waited for a title built from the server's
display name, but the dialog is titled with the server name.

* feat: make the queued-send lock expiry configurable

The per-pane lock that stops the run-end drain and an explicit Send now
from submitting the same queued message twice gives up a broken claim
after a fixed 60 seconds. Until then every drain and every Send now for
that pane is refused, and a deployment had no way to tune that window.

Expose it as interface.queuedSendLockTimeoutMs beside the queued-turn
reconciliation timeout, defaulting to the same 60 seconds so nothing
changes unless an operator sets it. The interface loader copies each
field explicitly, so the value is added there too; without that line a
configured value would never reach the client.

* fix: decline interrupt-and-steer while reasoning is staged

With a reasoning choice staged, the Interrupt & steer row is disabled,
but its keyboard chord still reached interruptSteer, which fell back to
interruptAndSend: it aborted the run and threw away the partial answer
the action promises to keep.

A staged reasoning choice belongs to a new generation that a steer into
the live one cannot carry, so decline the chord exactly as the row does,
the same way the function already declines while paused on an approval.
The words stay in the composer, where Enter queues them as a turn of
their own.

* fix: wake the queue drain when a late steer retry is queued

Retry marks a failed steer sending, and a sending steer is kept out of
the run-end sweep so its own POST settles it. When the run finishes
while that POST is in flight, the drain spends its one-shot signal on a
queue the steer is not in yet; the late ack or NO_ACTIVE_RUN then moves
it into the queue, where it sat until the user sent it by hand or
another run ended.

Every other late conversion re-posts the last completed run end through
the composer's rewakeDrain. The retry lives in the thread, so the
composer now publishes that function through the same host it already
uses to hand a reclaimed steer back to the draft, and the retry calls it
after converting. It stays a no-op unless the run completed cleanly and
no newer signal is armed, or when no composer is mounted.

* fix: honour thinking-budget ranges wider than 200,000

The request-scoped reasoning override capped thinkingBudget at 200,000
in the shared schema. An operator's paramDefinitions may advertise a
wider range, and the composer renders it, but every value above the cap
was dropped by the control's parse and rejected by the server's before
the range-aware check could run.

That check already exists: resolveReasoningOverride validates the value
against the endpoint's resolved range. Drop the fixed ceiling so the
resolved range is the bound, keeping the integer and lower-bound rules.
The new case covers both directions on a custom endpoint whose range
goes to 500,000.

* refactor: persist the preliminary reasoning override through the shared helper

The agents controller validated the request's reasoning override with
the schema itself before writing the preliminary user message. That is
behavior in the CJS layer, and it duplicated resolvePersistedReasoningOverride,
which @librechat/api already exports and BaseClient already uses for the
same field.

Call the helper instead. A preliminary message is always a fresh turn,
so it is never an edit or a compaction. The two controller specs that
replace @librechat/api with a hand-built mock now take the real helper.

* fix: clear the file manager's state at the session boundary

The file manager's open flag and its opener live in Jotai's default
store, which outlives the authenticated route. A file manager left open
at logout reopened for the next account, still holding a ref to the
previous session's unmounted opener. Reset both in useClearStates, next
to the other composer state this branch already clears there.

* fix: decline interrupt-and-steer on the first turn while reasoning is staged

* test: read the phone layout from the viewport and keep the drawer open

* fix: name both reasons a queued message cannot be edited yet

* test: reuse a published file through the composer palette

* style: move the composer onto the design-system rules

* style: move the file row's new layout styles into classes

* fix: keep the composer's reused thumbnails, focus ring and disabled dimming intact

* fix: bring a steer back when its deferred cancel does not remove it

* fix: treat a run as over only when every pane holding it is idle

* refactor: pass enter-to-send into the composer from its host

* feat: make the palette's recent-file count configurable

* fix: let a file already in the composer be reattached at the limit

* test: hide the chip before replaying a deferred cancel

* refactor: restore the resume request through one packages/api facade

* fix: keep a conversation's staged reasoning across a visit to another model

* fix: keep a connected MCP server's variables configurable from the palette

* fix: resolve every pane-scoped shortcut through the pane holding focus

* refactor: keep every reasoning-override decision out of the /api wiring

* perf: skip paging the skill catalog for an agent that can show no skill

* test: type the reasoning snapshot the override helper records

* test: pin the applied budget and the untouched parameters in the override specs

* style: give the stop button the send button's shared submit recipe

* fix: name only the newline key when no chord sends

* fix: name the stop key only once the stop control can act

* fix: clear every staged reasoning selection at sign-out

* fix: stop advertising send-now while staged reasoning forces a queue

* fix: resolve the reasoning target of an ephemeral agent in the composer and the request

* test: type the staged reasoning the sign-out spec seeds

* fix: check the composer will take a queued row before discarding its parked copy

* style: carry the composer's classes onto Tailwind v4

* style: sort the composer's classes in Tailwind v4 order

* fix: return a refused queued row to the durable queue instead of memory

* i18n: localize the whole attachment-limit message

* refactor: validate the reasoning override inside the packages/api facade

* style: give the composer's submit slot its own IconButton variant

* test: pin the submit slot to IconButton's submit variant

* test: stub the seen-state hook the rebased MessagesView now calls

* fix: turn file search on for an embedded file attached through the shared hook

* chore: record the composer's design-rule counts against the rebased base

* fix: keep the composer's container query inert under Tailwind v4

* test: point the v4 focus scenarios at the redesigned composer's controls

* fix: keep a re-queued row on its own lineage after the run ends

* test: assert a re-queued row carries its lineage

* refactor: pass the dictation preferences into the composer from its host

* test: read the usage atom from the store the ask hook renders under

* fix: read a pending steer's pause from its own pane's branch

* a11y: keep focus on the quote chips until the last one is removed

* fix: keep staged reasoning for the next message across a manual compaction

* test: expect focus on the remaining quote chip after removing one of two

* style: sort the imports the canary rebase merged

* refactor: remove the side-panel files table the account menu replaced

* fix: settle a retried steer's late ack against a racing cancel

A cancel clicked while the retry POST was in flight left the optimistic
marker unread: the ack re-minted a live chip and injected words the user
had asked back, and a failed retry after that cancel lost them entirely,
because the failure path maps chips the optimistic cancel had removed.
The ack path now defers to the marker the way the composer's does, the
failure path re-adds the chip before settling it, and the queued row a
late ack produces binds its recovery identity only under protocol v2,
the same default the composer's degradation path trusts.

* fix: strip a replayed reasoning override that no longer validates on resume

A paused turn's reasoning override is replayed onto its resume request
and then validated as if it were fresh client input, so an operator
changing the endpoint's reasoning config between pause and approve made
the checkpoint permanently unresumable with a 400. The replay now marks
the request, and a replayed override that fails validation is stripped
so the resume proceeds on defaults; a fresh override that fails is still
rejected.

* refactor: derive the enforced spec fields inside the modelSpecs facade

The set of keys an enforced model spec locks is the input contract of
the reasoning-override refusal rule, and deriving it in the route
middleware left the policy untestable from packages/api and free to
diverge from the facade that owns the rest of the preset application.
applyModelSpecPreset now returns the enforced fields beside the private
ones it applies, and the middleware consumes them.

* fix: restore the pre-override parameters when a resume degrades

Stripping a replayed reasoning override that no longer validates removed
only the metadata field, while the resume had already replayed the paused
turn's model parameters with the override applied into the endpoint
option, so the provider still received what validation had refused. The
strip now inverts the application from the replayed base: the recorded
pre-override value returns and the thinking flag the override forced on
comes off.

* fix: resolve the effective responses route for upload targets

The attach target returned the conversation or agent's saved
useResponsesApi flag, so an Azure chat auto-routed to the Responses API
by responsesApiRouting preflighted its uploads against the chat
completions route and lost provider-document uploads. The resolver now
applies resolveEffectiveUseResponsesApi with the endpoint, model,
web-search state and routing policy, the same resolution the during-run
upload target makes.

* fix: name the interrupt chord in the hint only while it still interrupts

The during-run hint advertised Alt+Enter Interrupt & send even when the
chord had been claimed: a submitMessage rebound to Alt+Enter, a chord
yielded to a global shortcut, or shortcuts disabled altogether each make
the keydown resolver return something else. The hint now reads the same
resolver verdict the during-run send button does and omits the chord it
no longer owns.

* test: assert the unseen count the scenario adds rather than a zero baseline

The tab title counts every unseen conversation, and a shared
verification run leaves legitimately-unseen conversations from earlier
scenarios behind, so the three title-count scenarios failed on other
scenarios' leftovers instead of the behavior they seed. Each now reads
its baseline and asserts the count it adds and clears.

* test: hold the unseen count to its direction on the mobile project

The away poll that lights a dot also discovers conversations the client
had not counted yet, so on the mobile project the title count can jump
past any baseline taken mid-test. Opening still has to clear this
conversation's contribution and an away reply still has to raise the
count, which is what the assertions now hold.

* test: enable the MCP Apps server through the redesigned palette

The MCP Apps integration spec still drove the old composer's MCP
Servers menu, which the redesign replaced with toggleable rows in the
attach-and-tools palette; the lane failed on the missing menu on every
run. The helper now opens the palette, waits for the server wave, and
flips the row's pressed state.

* fix: preserve numeric reasoning defaults on blur

* test: reopen persisted code cards before checking highlights

* test: follow the current file manager navigation

* fix: offer only reasoning options a request override can carry

* fix: keep the favorited copy when skills share a name

* fix: restore composer focus by the interaction that closed the palette

* test: expect a keyboard close to refocus the composer on every project

* chore: re-record the design-rule backlog on the rebased composer

* chore: drop the files table and label the rebase brought back

* test: tolerate other projects' unseen conversations when marking one unread

* test: open the file table header scenario from the account menu

* fix: let the stop shortcut reach a hidden stop control from a dialog

* fix: keep staged reasoning out of tool-limit recovery and MCP App turns

* fix: degrade a queued turn's stale reasoning override instead of dead-lettering it

* fix: refresh the attach target when web search changes the route

* fix: re-arm the run-end drain after moving a steer to the queue

* test: hover the MCP row before reaching its revealed Connect action

* test: pick the parts scenario's MCP server from the tools palette

* fix: cancel a recovered row's parked steer only after the composer takes it

A recovered row's only durable copy is the steer parked on the server, and
once its run has ended nothing can re-create it, so a composer that refused
the row after the cancel left the words in memory until the next reload.
The composer now takes the words first; a cancel that then fails holds the
row out of the run-end drain instead of sending it a second time.

* fix: name only the idle composer affordances that act

* fix: keep steer escalation off until the pane owns the rendered conversation

The branch tail that decides whether a run is paused is keyed by the pane's
conversation, which can still name the previous chat while a cached one
renders. Until the pane catches up, a pause on any cached branch now keeps
Escalate disabled instead of reading a missing or foreign tail as unpaused.

* fix: route the host's Enter and auto-send preferences into the composer's nested hooks

ChatForm takes enterToSend and autoSendText from its host, but useTextarea,
DuringRunSendButton and both speech engines still read the store atoms, so a
host whose values differ advertised one behavior while the handlers ran
another. Pass the host values through instead.

* fix: settle a browser dictation take only after its final transcript lands

stopListening resolves on the recognizer's end event, after the final result
reached useSpeechRecognition's reducer but before React committed it, so the
take settled while the composer still held the interim text. Settle from an
effect declared after the transcript effect, off a state update queued behind
the result.

* fix: drop a restored reasoning override the current setting no longer offers

The staged override was only checked for a matching parameter key, so a value
restored after a capability change on the same target stayed staged, blocked
live steering, and was dropped at send time. Validate it against the setting's
options and range.

* test: scan the attach and tools palette for accessibility

The a11y suite still opened the removed tools dropdown. Open the palette
instead, and add a scenario that runs Axe over the open palette.

* fix: make the palette's virtualized list the grid that owns its rows

The rows sat under a wrapper grid behind react-virtualized's scroller, which
always renders aria-label and aria-readonly and so could not be made
presentational. Axe reported both the disallowed attribute and the grid's
invalid child. The scroller is now the labelled grid, with a rowgroup holding
the rows.

* fix: state the palette grid's row total and each row's position

Only the rows in view are mounted, so without aria-rowcount and aria-rowindex
assistive technology reads the rendered window as the whole grid.

* fix: key an empty destination's pending steers to the route

An empty tree has no message naming its conversation, so the pending-steer
surface fell back to the chat context, which still names the source chat
during warm navigation, and its controls acted on the source run. Fall back
to the chat's messages key, which follows the route.

* fix: keep assistant uploads available when the file config fails

Assistants upload directly and never needed the destination policy, which is
why the replaced AttachFileChat rendered their uploader outside the policy
gate. Restore that exception so a failed or paused file config request does
not remove every attachment entry.

(cherry picked from commit f568a4ba30)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14473
2026-10-01 03:46:18 +00:00
Dustin Healy
7b27e7941e 🧩 feat: MCP Apps support (#13831)
* feat: add MCP Apps support

* fix: preserve MCP App results and enforce sandbox policy

* fix: bind persisted MCP Apps to their server configuration

* fix: complete MCP Apps runtime integration

* fix: type the optional MCP App binding codec explicitly

* test: provide MCP Apps policy in bridge mocks

* fix: pass admitted App policy to the tool call owner

* fix: align MCP Apps CI contracts

* test: preserve MCP abort reasons in recovery mocks

* fix: preserve MCP compatibility across policy profiles

* test: use profiled MCP catalog generations

* test: expect the standard MCP session profile

* fix: remove the unused MCP abort import

* test: expect profiled MCP fixture arguments

* test: seed profiled MCP manager fixtures

* Fix request-header bearer resolution after canary merge

* Preserve MCP App views across canary message updates

* Document message-owned MCP App placement

* ci: Recheck MCP Apps Against Canary Base

---------

Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit a98d40f61c)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/13831
2026-10-01 03:46:16 +00:00
Marco Beretta
50ce28957f
🎚️ fix: Apply librechat.yaml Rate Limits to Route Limiters (#16432)
Route modules build their rate limiters while they load, and both server
entries required them before startup checks copied rateLimits from
librechat.yaml into the environment. The conversationsImport, tts, stt
and route-scoped fileUploads budgets therefore ignored the yaml unless
the matching environment variable was also set. Require the routes only
after performStartupChecks in index.js and experimental.js.
2026-09-28 16:15:37 +02:00
Joseph Licata
f25242df98
🦘 feat: Share Run-Scoped Files With Subagents (#15848)
* feat: add run-scoped file sharing for subagents

Delegate current-message files through authorized execution catalogs and publish selected immutable code-output versions through the existing file pipeline.

Requires the companion agents SDK context adapter before the dependency can be released and pinned.

* fix: preserve child tool artifacts and verify file-sharing lifecycle

Keep citations and other non-sandbox artifacts on their existing delivery path while retaining private code outputs and inspected images until publication.

Cover native and text delivery, nested and sibling authorization, concurrent runs, cancellation cleanup, and checkpoint recovery through the browser harness.

* test: align file-sharing fixtures and integration cleanup

* test: repair LibreOffice conversion fixtures

* test: verify deployment skill bytes across checkout formats

* fix: align subagent file budgets and refresh boundaries

* chore: update agents SDK to 3.8.6

* refactor: move code output processing into API workspace

* fix: enforce shared file budgets for parents and teams

* fix: declare the shared file encoder interface
2026-09-12 20:14:22 -04:00
Danny Avila
fed09dacb6
📎 test: Verify Stateful Sandbox Upload Continuity (#14363)
Covers the one leg the credential-free mock suite cannot fake: a file
attached through the UI's Code Environment target must land in the live
Code API, get delivered into the sandbox, and stay readable across turns.

- fake-model: E2E_EXEC_UPLOADED:/E2E_EXEC_PERSIST: markers emit real
  bash_tool calls through the production tool pipeline (the sandbox
  surface stateful agents register; execute_code stays host-wired and
  never appears in the tool registry), plus an env-gated dump of each
  run's advertised tools for future spec authoring
- spec: attach uploads a CSV to the Code API, turn 1 reads it back from
  /mnt/data and drops a proof file no upload contained, turn 2 (sent
  without an attachment) reads both — only possible when the run reuses
  the same stateful runtime session
- e2e config: agents capabilities gain stateful_code_sessions, and the
  mock profile pins CHECK_BALANCE=false so a developer's local .env can
  never fail every send with zero-balance users
- skips unless LIBRECHAT_CODE_BASEURL is provided (kept out of CI)
2026-09-09 00:35:28 -04:00
Danny Avila
ec15075dc8
🚚 feat: Route Attachments and Provision Tools Lazily (#15763) 2026-09-08 23:28:52 -04:00
Danny Avila
dfb264811c
🪢 test: Prove MCP OAuth Resume End to End (#15485) 2026-09-01 18:38:39 -04:00
Danny Avila
c1cb591d49
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments

* fix: include code environment in lazy agent type

* fix: harden stateful environment routing

* fix: complete code environment route isolation

* fix: declare code environment map type

* fix: preserve configured code execution routes

* fix: harden stateful environment updates

* test: assert route-scoped sandbox readiness

* fix: harden stateful environment lifecycle

* fix: isolate migrated code sessions

* test: assert route-qualified code sessions
2026-08-30 15:38:45 -04:00
Danny Avila
1489623fa3
🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210)
* 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E

The mock e2e lane's only credential-free model is a hand-authored script:
`fake-model.js` decides responses from ~60 `E2E_*` prompt markers. That
covers scripted shapes well, but no scenario replays a *real* recorded
provider conversation through the assembled chain, so real streaming
shapes — provider chunk cadence, reasoning deltas, usage metadata — are
only ever approximated.

This adds a record-once/replay-forever tier alongside the marker routing.

Record (`E2E_MODEL_FIXTURES=record`, needs a provider key): the run hook
appends a LangChain callback handler to every agent context's
`clientOptions.callbacks` instead of overriding the model, so the REAL
provider streams while each invocation's `AIMessageChunk`s serialize to
`e2e/fixtures/model-replay/<name>.jsonl` — text deltas, tool_call_chunks,
reasoning kwargs, and genuine usage metadata. Only the latest human text
is recorded for binding; system prompts and tool schemas never enter the
fixture.

Replay (default, keyless): `fake-model.js` consults `tryBindReplay` ahead
of marker routing, binding a conversation whose prompt matches the next
unconsumed invocation. The replaying model is not hand-assigned — it is
registered as SDK provider `librechat-e2e-replay` via `registerProvider`
and constructed through the SDK's own `initializeModel`, so registry
lookup, constructor clientOptions, and real `bindTools` all run the way a
live provider's would. Recorded chunks therefore stream through the same
createRun → graph → SSE → persistence chain.

Consumption is enforced rather than assumed: every invocation re-checks
its prompt against the recording, an invocation past the end of the
script throws, and a per-fixture ledger lets the spec assert at teardown
that every recorded invocation and chunk was drained. Streaming
incrementality is asserted from that ledger, not by sampling transient
DOM, which is a race by construction.

The credential-free profile is unchanged when not recording: the record
provider and its selector entry are template markers that stay comments,
and no existing spec's routing is touched (a fixture only binds on an
exact prompt match; everything else falls through).

Verified: record vs the real DeepSeek API 1 passed (15.2s); keyless
replay 1 passed twice (11.4s, 12.7s) with the ledger fully drained (2/2
invocations, 10/10 chunks, no overruns or mismatches); app-load,
completion, and chat 10 passed unchanged.

* fix(e2e): rebind a replay fixture from the top for a new conversation

The replay cursor is process-global while the web server outlives a
Playwright retry, so a fully consumed fixture left the retry unable to
bind its first prompt: it fell through to marker routing and failed
deterministically, burning every configured CI retry. A partially
consumed attempt failed the same way.

Binding now restarts the fixture when the incoming prompt matches its
first recorded invocation, resetting the ledger with the cursor so the
new attempt is judged on its own consumption instead of accumulating the
previous one's counts. Continuing an in-progress binding still outranks
restarting, so a fixture whose opening prompt repeats later in the script
advances rather than rewinding.

The over-consumption guard is untouched — it fires inside the stream when
the cursor passes the end, not at bind time.

* fix(e2e): close three replay-lane gaps found in review

Restart the recorder on a retry. Its state is process-global like the
replay cursor, so a failed attempt that had already recorded invocations
left the counter advanced: the retry appended 2/3 after 0/1, or kept the
previous attempt's `error` line, and the fixture was unusable for replay.
Recording now truncates and restarts when the opening prompt reappears,
mirroring the replay side's rule and its caveat.

Retain a consumed binding for the conversation that drove it. An extra
user turn past the final recorded invocation found no next invocation and
fell through to ordinary fake-model routing, so it was answered with a
mock reply: the over-consumption guard never ran and the already-drained
ledger still passed. Such a conversation is now recognized by its human
turns opening with the fixture's recorded prompts, and stays bound so the
stream raises the overrun. Continuing an in-progress binding still
outranks restarting, which outranks retaining a consumed one, so a
retry's fresh conversation rewinds rather than being read as an extra
turn.

Validate the fixture the recording actually wrote. Record mode honors
`E2E_MODEL_FIXTURE_NAME`, but the spec always inspected the committed
`deepseek-two-turn`; another name wrote elsewhere while the assertions
read the pre-existing file, and because the prompts are fixed the stale
answers could match and green a run that verified nothing it produced.

* fix(e2e): make replay binding correct for tool and subagent fixtures

Round two's retry and consumed-binding fixes both assumed one model
invocation per user turn. A turn that calls a tool breaks that: the model
is invoked again after the tool result under the same latest human
message.

Identify a retry by the conversation boundary, not the prompt. The
recorder ran per invocation and truncated whenever the opening prompt
reappeared, so a tool round trip looked like a retry and discarded the
recorded tool-call invocation. Restart detection now sits in
`installRecorder`, which runs once per `createRun`: a turn whose history
holds no prior human message begins a conversation.

Compare consumed bindings against user turns, not invocations. Several
recorded invocations can share one prompt, so a one-to-one comparison
could not recognize the originating conversation — invocations `[A, A, B]`
against history `[A, B, C]` failed on both length and elements, and the
extra turn fell through to the fake model with the drained ledger still
passing. Fixtures now carry their collapsed turn sequence.

Override the subagent model too. `graph.overrideModel` is not inherited
by child executors, so a fixture recording a subagent call — record mode
captures child invocations already — would leave the child on its
configured provider: an underrun, and a real provider request in a lane
that must stay keyless.

Reject ambiguous prompt matches. Binding order followed filesystem
enumeration, so a second fixture sharing a prompt could silently redirect
a scenario to the wrong chunks and ledger; the spec's choice never
reaches the server-side loop, so ambiguity fails instead of picking a
winner. Fixture identity is the file name for the same reason — a
recorded `meta.name` is descriptive, and trusting it let a copied fixture
collapse onto another's registry key and ledger.

Prove the recording is fresh. The spec removes the selected fixture
before driving, so a run whose hook never installed the recorder fails
instead of greening against a stale artifact whose answers still match
these deterministic prompts.

* fix(e2e): rewind a replay fixture at the conversation boundary

Consecutive invocations can share a prompt — a tool call produces exactly
that — so an attempt stopping mid-turn left the cursor on an invocation
whose text still equalled the opening prompt. Matching the cursor first
meant a retry's fresh conversation resumed after the tool call instead of
rewinding, consuming the post-tool invocation and silently replaying a
different script than was recorded.

A conversation boundary now outranks an in-progress cursor: a fresh
conversation whose prompt opens the fixture rewinds even when the cursor
would have matched. Continuing still outranks restarting within a
conversation, so a turn that calls a tool advances to its post-tool
invocation rather than rewinding on its own repeated prompt.

* fix(e2e): refuse cross-conversation binding and prove content streaming

A fresh conversation could steal a partly consumed fixture's later turn.
Only a conversation opening with the fixture's first prompt was treated
as a boundary, so after `[A, B]` had consumed `A`, an unrelated new
conversation whose first message was `B` matched the cursor, received the
recorded second-turn response, and advanced the shared cursor without
ever having driven `A`. A conversation start may now only rewind a partly
consumed fixture, never continue it; continuation within a conversation
is unaffected.

The incrementality assertion counted empty frames. Providers emit empty
initialization and usage-metadata chunks around the content deltas, so a
total chunk count above one was satisfied by a single delta: the previous
fixture's closing turn had four chunks and one content-bearing delta
carrying the whole answer, and both modes stayed green without proving
incremental assistant-content streaming at all. Fixtures now track
content-bearing chunks separately, the closing prompt asks for prose
rather than a number, and both modes require several content deltas on
that turn. Re-recorded: the closing turn now carries 28 content deltas.

* fix(e2e): scope record mode to the fixture spec

`E2E_MODEL_FIXTURES=record` replaces the fake-model hook globally, so an
unfiltered entry point such as `npm run e2e:mock` sent every spec under
specs/mock to the paid real-provider endpoint, while each fresh
conversation truncated and rewrote the one selected fixture — leaving an
artifact from whichever scenario happened to run last.

Record mode now matches only the fixture spec: an unfiltered recording
run lists one test instead of 203. Replay mode is untouched and still
collects the full suite.

* 🪪 fix: Bind Replay Fixtures by Conversation, Not Prompt Text

Prompt text was standing in for conversation identity, and three review
rounds found the same class of defect underneath it: a tool call repeats
a prompt across invocations, a retry repeats it across attempts, and a
resumed run has neither prompt nor history because `createRun` is rebuilt
with no messages while state comes from the checkpoint. Each fix in that
space created the next gap.

Thread the identity instead. `createRun` accepts a `conversationId` and
passes it to the run hook, which the agents controller supplies at both
call sites — the same value it already uses as the checkpointer's
`thread_id`. The field is optional and the hook is env-gated, so nothing
changes when the harness is not in use.

Binding then collapses to ownership. A fixture is owned by the
conversation that claimed it, and its cursor is authoritative wherever it
stands: an extra turn reaches the over-consumption guard rather than
falling through to the scripted fake model, and a resumed run keeps
replaying with no prompt to match. A different conversation may claim the
fixture only by opening it, which rewinds — what a Playwright retry looks
like. Everything else is refused, so an unrelated conversation can no
longer continue someone else's partly consumed script by repeating a
later prompt. The prompt is still re-checked on every real turn; only a
resume, which structurally carries no human message, is exempt. The
previous text-and-history rules remain as a fallback when identity is
absent.

The recorder keys the same way: a new attempt is a new conversation, so a
resume no longer truncates the fixture mid-turn and discards its
tool-call invocation.

Record summarization too. The summary provider runs on its own model with
its own callback list, so a scenario crossing the context-pruning
threshold recorded the agent's invocations but not the summariser's,
leaving a fixture that could not reproduce the pruned context.

* 🧾 fix: Harden Record Mode and Make the Rendered-Text Assertion Honest

CI caught what local runs had not: the committed fixture was never
replayed locally, because the record run overwrote it after the replay
check rather than before. Re-recording and replaying in that order is
what surfaced the rest of this.

The DOM assertion compared raw recorded text against rendered markdown.
The previous answer opened with `52.`, which Markdown renders as an
ordered-list marker, so those characters never appear in the DOM and the
match failed on all three CI attempts while replay itself was correct.
The closing prompt now asks for prose beginning with a word, a leading
enumerator is stripped before matching, and only a prose prefix is
compared.

Derived configs discarded the record-mode restriction. `config.redis.ts`
and `config.mermaid.ts` spread this config and then replace `testMatch`,
so `e2e:mock:redis` in record mode would still send its specs to the paid
provider. A restriction expressed as an overridable value cannot hold, so
record mode now refuses any config but the mock one.

Superseded recording callbacks could write across a reset. A failed
attempt with a provider call still in flight keeps its handler on the old
graph; after the retry reset, that call would allocate an invocation from
the new counter or append an `error` entry with a cleared mapping.
Handlers now carry the recording generation they were installed for and
ignore everything from an older one, and attachment dedupes against the
current generation so a graph carried across a restart is not left with
an inert handler.

* 🚧 fix: Make Summarization an Explicit Boundary, Not a Half-Feature

Recording summarization invocations without replaying them is worse than
ignoring them. Replay routes the agent model and subagents only, so a
recorded summarization entry takes a slot in the fixture sequence that
replay never consumes, and the next primary call reads the summariser's
chunks — a prompt mismatch or, worse, silently wrong content.

The attachment was also aimed at the wrong shape: the SDK reads
`summarizationConfig.parameters`, not `.parameters` nested under
`.config`, so the previous attempt would have attached to nothing in a
real run. Its test passed only because the test built the shape the code
expected rather than the shape the SDK provides.

Rather than ship a half-routed feature, recording now fails the moment
summarization runs, naming the reason. Both shapes are guarded so the
guard cannot miss the way the recorder did. Summarization fixtures need
replay routing for that model before they can be supported.

The derived-config guard added alongside it was itself broken: workers do
not carry `--config`, and the argument lookup fell through to
`process.argv[0]`, so every recording run aborted claiming the node
binary was an unexpected config. The flag is now located explicitly and
absence is treated as "not the process that parsed the CLI".

* 🔒 fix: Close the -c Config Alias and Pin the Recorder's Fixture Name

Playwright documents `-c` as an alias for `--config`, so record mode
launched as `playwright test -c e2e/playwright.config.redis.ts` slipped
past a guard that recognised only the long spelling. Both spellings and
both `=` and space forms are now parsed.

Accepting arbitrary fixture names also worked against the ambiguity
check. This spec drives one fixed prompt pair, so recording under another
name left two fixtures sharing those prompts; replay then refused to bind
either and the keyless lane stopped working — a successful documented
recording run could disable the suite it exists to serve. The spec now
records only the fixture it owns and says so when asked for another.

* 🔧 test: Record a Real Tool-Call Turn and Replay It Through the Tool Node

The fixture format carried `tool_call_chunks` and the binding advanced
through a turn's invocations, but nothing had recorded a real
tool-calling conversation end to end — the path was covered only by
hand-written synthetic fixtures, and it is the first one a new scenario
would exercise.

This records one: the provider calls the `remember_fact` MCP tool, the
tool runs, and the model is invoked a second time with its result. That
is the shape a single prompt cannot express — one user turn spanning
several model invocations, all sharing one prompt — so it is what proves
the turn-vs-invocation distinction the binding rules were built around.

Replay drives the real tool node rather than replaying its output, so the
tool executes again and the assertion checks its live result.

Two fixtures now coexist, which the record path had to grow for: the
config keeps an allowlist so an unknown name is still refused, record
mode collects every replay spec, and each spec records only the fixture
it owns and stands down for the others.

MCP tools reach the model under a server-qualified name
(`remember_fact_mcp_e2e-memory`); that qualification has changed before,
so the assertions match the base name as a prefix rather than pinning the
suffix.

Verified: record 1 passed (15.6s, real API) then replay 1 passed (13.6s)
against that fixture, ledger drained 2/2 invocations and 35/35 chunks;
both replay specs together 2 passed; app-load, completion, chat and
mcp-ephemeral 12 passed.
2026-08-25 18:29:55 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
fe71ffdf42
🛂 ci: Grant Multi-Convo Permission in E2E Mock Config (#14839)
`agent-skills-added.spec.ts` drives the composer's `+` command, which opens the
added-model popover. That path is gated on MULTI_CONVO.USE:

    if (!hasMultiConvoAccess || !plusCommandEnabled || isAssistantsEndpoint(endpoint)) return;

The mock config never sets `interface.multiConvo`, so the permission falls
through to the seeded role default and `handlePlusCommand` returns before
opening the popover. The spec then fails on a popover that is absent from the
DOM entirely, which reads as a selector or timing problem rather than a missing
permission.

Set it explicitly, the same way `contextCost` is set just above for the usage
gauge — the mock config's job is to make each exercised feature's gate explicit
rather than inherit a default.
2026-08-15 10:47:35 -04:00
Danny Avila
236ee6c1ab
🧭 fix: Re-Anchor Parent Activity Phase Bounds (#14741)
* test: cover parent activity phase finalization

* test(e2e): stabilize parent phase coverage

* fix(agents): reanchor parent activity phase bounds

* fix(agents): preserve delayed tools in activity phases

* test(agents): keep phase slice bounds typed

* fix(agents): preserve sparse activity phase bounds

* test(e2e): read structured phase replies
2026-08-11 10:16:57 -04:00
Ravi Kumar L
7cf4c3f73f
🧪 test(e2e): add Bombadil property exploration (#14462)
* test(e2e): add Bombadil property exploration

* fix(e2e): address Bombadil review feedback
2026-08-10 02:11:06 +02:00
Danny Avila
1bccc2bc18
📡 fix: Refresh MCP Tools After List-Changed Notifications (#14686)
* fix(mcp): handle dynamic tool list changes

Co-authored-by: Pascal Garber <pascal@artandcode.studio>

* test(mcp): fix CI validation

* fix(mcp): keep dynamic tool catalogs live

* fix(mcp): harden dynamic catalog lifecycle

* test(mcp): use typed startup connection

* test(mcp): isolate dynamic e2e fixtures

* fix(mcp): refresh tools after reconnect

* fix(mcp): close dynamic catalog cache gaps

* test(mcp): update OAuth connection mocks

* fix(mcp): preserve app snapshot ownership

* style(mcp): sort connection imports

* fix(mcp): close review race conditions

* fix(mcp): preserve cache ownership edges

* fix(mcp): harden recovery lifecycle

* fix(mcp): guard tool-less app refresh

* fix(mcp): fence distributed cache races

* fix(mcp): retire stale connection state

* fix(mcp): keep tool snapshots authoritative

* fix(mcp): fence stale app tool publications

* style(mcp): sort repository test imports

* test(mcp): mock empty startup publication

* fix(mcp): preserve app publication generations

* fix(mcp): harden publication recovery races

* fix(mcp): address tool catalogs by runtime config

* fix(mcp): load scoped catalogs for assistant writes

* fix(mcp): harden catalog publication recovery

* fix(mcp): serialize forced connection replacement

* fix(mcp): serialize ordinary creation with replacements

* fix(mcp): harden catalog fallback boundaries

* fix(mcp): close lifecycle fencing gaps

* fix(mcp): preserve catalog authority on failures

* fix(mcp): compensate failed catalog mutations

* fix(mcp): fence catalog refresh ordering

* style(mcp): sort agent loader imports

* fix(mcp): cancel stale connection creation

* fix(mcp): fence catalog coordination

* fix(mcp): close catalog race windows

* fix(mcp): harden cross-pod catalog fencing

* fix(mcp): close catalog lifecycle edges

* style(mcp): sort assistant imports

* fix(mcp): reject stale recovery authority

* fix(mcp): restore static catalog on every startup

* fix(mcp): order app catalog publications

* style(mcp): sort catalog revision imports

* fix(mcp): separate catalog allocation and commit fences

---------

Co-authored-by: Pascal Garber <pascal@artandcode.studio>
2026-08-08 13:50:21 -04:00
Danny Avila
45cc53c40b
🛰️ chore: bump @librechat/agents to v3.4.0, cover streamed subagent results e2e (#14647)
* test: cover streamed subagent results end to end

* test: assert real e2e conversation id

* test: harden streamed subagent e2e

* test: stop incompatible subagent fixtures

* chore: update @librechat/agents to version 3.4.0 in package.json and package-lock.json
2026-08-05 22:25:03 -04:00
Danny Avila
cc813f430e
🎯 feat: Tool Intent Label Capability (tool_intents) (#14499)
* 🎯 feat: Tool Intent Label Capability (tool_intents)

Adds the fourth member of the per-tool capability family (defer_loading,
allowed_callers, run_in_background): an admin capability
AgentCapabilities.tool_intents plus a per-tool
tool_options[name].describe_intent flag. Opted-in tools get an optional
intent string injected as the FIRST property of their schema — one
model-authored sentence per call, streamed to the client as the call's
live status label (args already reach the client verbatim, so no new
event plumbing). Native host tools (web_search, create_file/edit_file,
set_memory/delete_memory, ask_user_question) default on while the
capability is enabled; explicit false opts out. SDK-native intent
schemas (@librechat/agents coding suite) are recognized and left alone.

- packages/api/src/agents/intent.ts: structural sibling of
  background.ts — first-key non-mutating injection with registry
  parity (covers deferred/tool_search discovery), eligibility and
  PTC-only skips, arg read/strip helpers, self-spawn strip for defs and
  registry, ephemeral/model-spec synthesis with a tool_options merge so
  the background and intent toggles compose.
- handlers.ts: intent runs BEFORE background injection so the label
  stays the first streamed key when a tool carries both (pinned by
  test); the arg is stripped before invocation unless the tool's own
  schema declares it, on both the foreground and background-dispatch
  paths; PTC target schemas are sanitized like background's.
- Capability plumbing through all four routes (endpoint initialize,
  openai + responses controllers, the exported OpenAI-compatible
  service) plus handoff discovery and added-convo agents, and the
  intentToolNames execution channel via configurable.
- describe_intent on toolOptionsSchema (all three written-out Zod
  annotations), ToolOptions, TEphemeralAgent, TModelSpec (+ zod), and
  data-schemas doc comments (tool_options is Mixed — no migration).
- intent.spec.ts: 28 tests cloned from background.spec.ts structure,
  including the intent+background key-order composition.

* 🧯 fix: Codex Review — Opt-Out Strips SDK-Native Intent, Skip mcp_all Placeholders

- An explicit describe_intent: false now REMOVES an SDK-native intent
  property from the definition and registry entry, so the per-tool
  opt-out actually disables the arg's token cost for tools like
  web_search that carry the schema natively (SDK bodies tolerate its
  absence). Previously the early return left the property in place.
- synthesizeIntentToolOptions skips lazily-expanded mcp_all
  placeholders instead of recording options under names that
  applyIntentLabels' exact-name matching can never match, and documents
  the limitation (parity with synthesizeBackgroundToolOptions).

The P1 about the client not rendering the label is the documented
slicing: the UI streaming-label PR follows once #14391's ToolCallGroup
changes merge — args already reach the client, so that slice is purely
rendering.

* 🧯 fix: Codex Re-Review — Label Marker Guard, Capability Kill Switch, Late Defs, Service Threading

- removeIntentParam is now marker-guarded (the label contract's opening
  instruction discriminates it), so an MCP/action tool's own business
  `intent` parameter is never stripped by an opt-out or the disabled
  path — previously an explicit false could remove a real, possibly
  required argument.
- New sanitizeIntentLabels pass runs AFTER every registration step
  (the skill catalog appends its SDK definition post-injection): with
  tool_intents disabled it strips SDK-native intent labels from all
  definitions and registry entries, making the capability a real kill
  switch over their token cost; with it enabled it enforces explicit
  per-tool opt-outs on late-registered definitions.
- ask_user_question removed from the native default-on set: its graph
  tool is rebuilt in run.ts from its own Zod schema (also the HITL
  card's wire shape), so definition-level injection never reached the
  model. Its intent support lands with the HITL slice, which threads
  the label into the interrupt payload deliberately.
- The exported OpenAI-compatible service now threads intentToolNames
  into the run configurable, so the executor's PTC path can strip
  host-injected intent schemas on that route like the in-repo
  controllers do.

* 🧯 fix: Codex Round 2 — Post-Skill Injection, PTC Native Strip, Service Boundary, Honest Docs

- Intent injection now runs LAST in initializeAgent, after the skill
  catalog — which both appends its own definition and REPLACES upgraded
  ones (skill-aware read_file), clobbering an earlier injection while
  intentToolNames still listed the tool. Injection PREPENDS while
  background APPENDS, so intent stays the first schema property under
  the new ordering (pinned by a reverse-order composition test).
- The PTC target-schema strip is now marker-guarded strip-ALL: SDK-
  native intent labels (which are deliberately never in intentToolNames)
  are removed from sandbox-advertised schemas alongside host-injected
  ones; business intent params survive.
- toolIntentsAvailable on the exported service documents the loader
  boundary: a custom LoadToolsFn returning only structured instances
  bypasses definition/registry injection and sanitize by construction.
- librechat.example.yaml describes tool_intents as backend groundwork
  with UI rendering in an upcoming release rather than promising a live
  label today.

* 📦 chore: bump `@librechat/agents` to v3.3.6

Brings in the SDK half of tool intent labels (danny-avila/agents#347,
#349): intent-first schemas on the coding suite across all three
engines, plus web_search / subagent / skill / tool_search, and the
outcome / outcome_patch result channel.

Activates three host paths that were inert while no SDK tool shipped an
`intent` property — verified against the real 3.3.6 schemas:
- capability OFF now strips SDK-native labels (a real admin kill switch)
- explicit `describe_intent: false` removes them per tool
- host injection stays idempotent against an SDK schema, keeping
  `intent` first and never double-injecting

* 🔬 test: Real-Provider Verification for Tool Intent Labels

Adds the live check the unit tests structurally cannot perform: whether a
real model actually authors the injected arg, places it FIRST, and gives
sibling calls to one tool distinct labels. Reuses the existing
real-provider harness (in-memory Mongo, seeded user, credential
neutralizer) and the existing stdio MCP fixture as a genuine tool, so no
external service is involved.

- e2e/config/librechat.real.yaml: adds the e2e-memory MCP server and the
  tool_intents capability, giving the real model something to call. The
  sibling spec asserts only relative token growth, so the extra schemas
  do not perturb it.
- e2e/playwright.config.real.ts: optional Langfuse passthrough. The
  LANGFUSE_* keys match the credential-neutralizer pattern and were being
  blanked before the server booted; they are preserved explicitly, read
  from the invoking environment only, and never written to the generated
  config.
- e2e/specs/real/tool-intents.spec.ts: two facts stored in one turn, both
  through the same tool, asserting intent is the first key of each call
  and that the two labels differ. Args are read from persistence rather
  than the DOM deliberately — no UI renders the label yet, and
  persistence is what a reloaded conversation and the trace both read.

First run against claude-haiku-4-5 produced 'Recording the location of
the OAuth callback router' and 'Recording the location of the MCP
connection pool configuration' — distinct, first-position, no tool name.

Also updates tool-intent-spec.md: records the 3.3.7 removal of the tense
verb map with the evidence that motivated it, the trimmed description and
the marker's role as an API, and a new mandatory requirement that
client-side label rendering be gated on a server-sent signal rather than
the presence of an intent key (a tool's own business 'intent' parameter
would otherwise render as a status label).

* 📦 chore: bump `@librechat/agents` to v3.3.7 and dedupe the intent contract

Picks up danny-avila/agents#353: the tense verb map is gone (a bare
intent now displays unchanged, with completion carried by UI state), the
model-facing description is trimmed 502 → 289 chars, and both the marker
and the description are exported.

Stops redeclaring the SDK contract here:
- INTENT_LABEL_MARKER is imported instead of duplicated as a string
  literal. Every removal path in this module keys on it, and a local copy
  that drifted from the SDK's would make them all stop recognizing
  SDK-native labels — failing OPEN, with labels left in schemas and
  per-tool opt-outs silently inert.
- INTENT_DESCRIPTION is imported too, so host-injected tools and
  SDK-native tools present the model with one identical instruction.
  Keeping the old local copy would also have meant host-injected tools
  still paying ~126 tokens per schema while SDK tools paid ~72.

Verified live against real Anthropic after the trim: two sibling calls to
one MCP tool produced 'Storing the OAuth callback router file location'
and 'Storing the MCP connection pool configuration file location' —
first-position and distinct, so the shorter description holds compliance.
2026-07-29 15:40:52 -04:00
Danny Avila
7b6900d556
🏷️ feat: Activity Groups With Fast-Model Headers (#14391)
* ✨ feat: Activity Groups with Fast-Model Labels

Groups each contiguous block of reasoning + tool calls into a collapsible
unit headed by a fast-model label (claude.ai-style hierarchy), off the
critical path: a PostToolBatch hook claims a live content slot at the
batch boundary (steering index-offset pattern), renders a deterministic
counts phrase instantly, and swaps in the generated label ~1s later while
the next model call streams. Labels are UI-only — stripped before the SDK
formatter and skipped in the legacy formatter — and reach live clients
via a dedicated on_activity_label SSE event (live/replay/pending paths).

Grouping preserves legacy rendering byte-for-byte when no label part is
present. Generation bridges to Run.generateActivityLabel() when the SDK
ships it (session-grouped Langfuse tracing); falls back to a direct call
today. Env-gated: ACTIVITY_LABELS_POC=true, ACTIVITY_LABEL_MODEL.

* 🧷 fix: Address Codex and Copilot Review Findings for Activity Labels

- Settle in-flight label fills (bounded 3s) before finalization on both
  the main and resume paths, so a label resolving during the final batch
  still reaches the durable log and saved message.
- Overlay on_activity_label chunks in RedisJobStore content
  reconstruction (splice path last-wins per index; replay path
  chronological overwrite), matching steer handling.
- Wire activity labels into the HITL resume createRun so post-resume
  batches keep claiming slots.
- Guard against out-of-order publishes: fill() awaits the claim emit
  before emitting the resolved label, and the client applier ignores a
  stale pending placeholder once a resolved label is present.
- Stamp the batch's groupId onto label parts so parallel-column runs
  place them inside their group instead of filtering them out.
- Localize the counts fallback phrase (10 keys, singular/plural) through
  useLocalize across chat rendering and exports.
- Type the hook with Providers/ClientOptions instead of stringly types;
  drop the unknown cast in the spec; add a dedicated rAF retry ref for
  label events with effect cleanup.

* 🛡️ fix: Address Independent Review — Abort, Usage, Lane Context, Redis Test

- Propagate the run abort signal into label generation (both wiring call
  sites; runtime combines host + dispatch signals with the timeout) so a
  user abort cancels in-flight label calls instead of paying to timeout.
- Record label-call usage like titles: the SDK bridge aggregates via
  chainOptions callbacks, the fallback path via a per-generation callback
  factory; both feed recordCollectedUsage under context 'activity-label'.
- Scope block-context capture: reasoning collection stops at the previous
  block's label part and filters by executingAgentId, so consecutive or
  parallel batches can no longer bleed another block's thinking into the
  payload; intent text still scans past labels (persists across batches).
- Forward the effective charLimit to the SDK call so host and SDK prompts
  agree (SDK default aligned to 600 in agents#327).
- Add a Redis integration test proving last-write-wins reconstruction of
  on_activity_label chunks per claimed index.
- Rebased onto main: only the two activity commits replay (the nine
  steering commits belonged to the old base branch), zero conflicts,
  steering suites green.

* 📐 refactor: Move Activity-Label Wiring to TypeScript, Address Codex Round 2

- [P1] Slot claiming, lane stamping, emit ordering, context capture, and
  settle tracking now live in packages/api (createActivityLabelWiring +
  captureActivityBlockContext); client.js is a thin closure wrapper.
- Register the activity-label hook BEFORE the steer drain so a steer
  draining at the same batch boundary cannot flush the tool block and
  orphan the label outside its group.
- Resolve request-based header placeholders in resolveActivityLabelLLM
  (titleConvo parity) so metadata-keyed proxies work on label calls.
- Trim labels centrally before filling so whitespace-only output from
  either generation path keeps the deterministic counts fallback.

* 🧭 fix: Codex Round 3 — Capture Order, Shared Strip, Token Estimator, Hide Filter

- Capture block context BEFORE pushing the label part: the scan stops at
  ACTIVITY_LABEL parts, so post-push capture hit the just-inserted label
  and silently collected no reasoning excerpts (regression test added).
- Share stripActivityLabelParts from packages/api and apply it in the
  Responses and OpenAI-compatible controllers, closing the replay leak
  for entry points still running SDKs without the formatter skip.
- Exclude activity_label parts from the fallback response-token estimator
  (UI-only parts must not inflate no-usage provider billing).
- Keep label parts explicitly under hide_sequential_outputs — they
  summarize exactly the outputs that mode hides.

* 🔁 fix: Codex Round 4 — Resume Gap, Delta Flush, Agent-Scoped Intent, Token Counter

- Synthesize on_activity_label events for labels claimed or filled in the
  snapshot→subscribe window (the publish is fire-and-forget, so Redis-mode
  reconnects missed them). Feature-gated so the default path adds no
  content re-read; the client applier already ignores duplicates.
- Flush queued deltas before applying a label part, matching the pending-
  action and steer appliers — without it the handler read a stale message
  cache and syncStepMessage pushed a pre-delta copy back.
- Skip another agent's tail text when resolving intent, so parallel runs
  cannot seed a label prompt with a sibling agent's narration.
- Exclude activity_label parts from countFormattedMessageTokens (the
  agent-path counter), not just the legacy BaseClient one.

* 🏗️ refactor: Codex Round 5 — Extract Label Host Logic, Report Usage, Icon Strip

- Move provider/model resolution, usage-metadata mapping, and the settle
  loop into packages/api (activityLabels/host.ts); client.js keeps only
  thin delegations, per the repo's TypeScript-implementation convention.
- Fold label usage into the response rollup with an 'activity-label' tag
  (subagent precedent) so metadata.usage and the live cost gauge account
  for it; tagged, so it stays out of PRIMARY usage/context pairing.
- Narrow tool metadata once in ToolCallGroup so THINK parts in a labeled
  block no longer render phantom generic icons in the stacked strip.
- Import the activity-label helpers by deep path in GenerationJobManager:
  the package barrel now reaches provider-config/cache modules that
  import back into the stream layer, and the cycle broke suite loading.

Declined: resetting steerOffsetState before HITL resume — resume builds a
FRESH AgentClient via initializeClient (initialize.js:978), so the offset
is already zero; the seed wrapper alone accounts for pre-pause parts.

* 🚦 fix: Codex Round 6 — Stream Label Usage, Close Late Fills

- Emit an on_token_usage chunk for label calls (sink push alone left the
  live session gauge blind); retained in pendingSubagentEmits so job
  cleanup cannot race the persist, tagged 'activity-label' as before.
- Close the label scope when settle times out: the wiring gates fill() on
  isClosed and the client fires a label-scoped AbortController, so a
  straggling generation can neither mutate a saved response nor emit into
  a job whose runtime is gone. The controller also chains to the run
  signal, so a user abort still cancels label work.

* 🩹 fix: Repair CI — Package Typecheck and Module Mocks

Local runs covered the client tsconfig and jest, but never packages/api's
own tsconfig, so nine type errors in the extracted host module shipped.

- Type host.ts against the real contracts: ServerRequest, EndpointDbMethods,
  AppConfig from @librechat/data-schemas, IUser for createSafeUser, and a
  MaybeAzureConfig view for the azure instance-name probe and configuration.
- Widen resolveConfigHeaders' llmConfig to Partial<RunLLMConfig>: it only
  reads the three provider header carriers, so auxiliary generations with a
  bare ClientOptions can resolve headers without assembling a run config.
  Type-only widening; every existing caller still satisfies it.
- Add stripActivityLabelParts to the @librechat/api mock in the OpenAI and
  Responses controller specs — those mocks enumerate exports, so a new
  import read as undefined and threw before the assertions ran.
- Use the real activity-label helpers in the ToolCallGroup spec's ~/utils
  mock; stubbing them out would hide the header logic under test.

* ⚙️ feat: Configure Activity Labels via librechat.yaml, Drop Env Vars

Replaces the ACTIVITY_LABELS_POC / ACTIVITY_LABEL_MODEL env gate with
per-endpoint settings, following the title options convention rather than
a top-level block — each endpoint picks its own cheap label model.

- Add activity, activityModel, activityEndpoint, activityPrompt,
  activityMaxPerRun, and activityCharLimit to the endpoint schema, and to
  the endpoints.all pick list (enumerated, so 'all:' would otherwise drop
  them silently).
- resolveActivityConfig reads them with title-style precedence:
  endpoints.all > named endpoint > custom endpoint config.
- Model precedence is now activityModel > titleModel > the agent's model.
  activityEndpoint runs labels on another endpoint's credentials, with
  titleConvo's fallback-on-unknown-name behavior.
- Thread activityPrompt/MaxPerRun/CharLimit through the wiring into the
  hook and the SDK bridge; they were hardcoded defaults.
- The resume gap-repair gate keyed on the env var; it now keys on the
  snapshot actually containing label parts, so deployments without the
  feature still perform no extra content read.
- Document the fields in librechat.example.yaml; add host.spec.ts
  covering precedence, custom-endpoint fallback, and opt-out.

* 📝 refactor: Rename Enable Flag to activityLabel, Document Schema Inheritance

- Rename the boolean from `activity` to `activityLabel`, matching the
  titleConvo/titleModel shape: a verb-object toggle whose prefix matches
  its modifiers (activityModel, activityPrompt, ...). `activity: true`
  alone read ambiguously — it could mean tracking or logging activity.
- Document the two endpoint-schema inheritance paths, which behave
  oppositely and are ~900 lines apart:
  * `endpoints.all` omits from baseEndpointSchema, so new options are
    inherited automatically — nothing to maintain.
  * `azureEndpointSchema` enumerates via .pick(), so a new option is
    silently unavailable on Azure endpoints until listed there.
  The activity block now carries a pointer to the Azure caveat.

* 🔍 fix: Address Codex Findings on the Config Rework

- Pass the matched custom-endpoint config into the label gate. Custom
  endpoints live in the `endpoints.custom` ARRAY, so without it every
  custom endpoint resolved as disabled — including the example this PR
  added to librechat.example.yaml.
- Give label usage a unique `runId:seq`. Label usage is billed but never
  appended to `collectedUsage`, so its length was static: every label
  event reused the last primary usage's pair and collided with itself,
  and the client dedupes on exactly that.
- Attach `cost` to label usage when `interface.contextCost` is on;
  aggregateEmittedUsage treats coverage as all-or-nothing, so an event
  without it suppressed the whole response's cost.
- Honor `activityPrompt` on the direct fallback path, not just the SDK
  bridge — it previously always used the built-in instruction.
- Seed the per-response label cap from labels already on the response so
  a HITL resume cannot mint a fresh quota after every approval.
- Reconcile label gaps on resume via a durable per-job `activityLabels`
  flag instead of probing the snapshot: the FIRST label of a run can be
  claimed inside the snapshot->subscribe window, which the old signal
  missed. The flag is read from a job record already fetched there, so
  runs without the feature still add no content read.
- Auto-collapse labeled single-tool groups; one-call batches are common
  in agent runs and rendering them expanded defeats the grouping.

* 🎯 fix: Correct Label Usage Seq, Cross-Endpoint Pricing, Close Scopes

- Give label usage a NEGATIVE seq namespace. The previous fix was wrong:
  seq is a position in `collectedUsage` (push, then emit with the new
  length), so sink-length + array-length still lands on a real position —
  primary emits 1, the label computes 2, the next primary also emits 2.
  Labels have no position at all (billed separately, never appended), so
  they now occupy a namespace positional sequences cannot reach. The
  client key is a string used for Set membership, so the sign is inert.
- Price cross-endpoint labels with the LABEL endpoint's token config:
  resolveActivityLabelModel now returns the resolved endpointTokenConfig,
  and both the streamed cost and recordCollectedUsage use it instead of
  the agent endpoint's rates.
- Make close state per-wiring rather than per-client. A HITL resume
  rebuilds the wiring, and resetting a shared flag re-opened closures from
  the pre-pause segment whose provider call ignored the abort; settle now
  closes every retained scope, past generations included.

* 🎯 fix: Make the Activity Header Say Something the Cards Cannot

The header read "ran 1 command" next to a card already labeled "Code" —
it restated the UI beneath it instead of adding to it. Two causes, both
about content rather than timing:

- A deterministic tool-type tally was the primary display and also fed
  the prompt, so the best case was a tally and the worst case was a
  tally dressed as prose. Removed from the metadata, the prompt, the
  part type, and the client.
- The instruction only ever reached the fallback path. The wiring
  passed a prompt only when  was configured, so the
  preferred SDK path silently used the published package default. The
  wiring now always supplies one and the hook forwards it on both
  paths.

The register is rewritten around what the cards cannot show:
past-tense git-commit-subject, leading with the distinctive noun,
outcome over attempt, tool names and counts and arguments explicitly
forbidden. The batch entries are labeled as reference material so the
model stops transcribing them.

Claiming a slot no longer emits. The slot still reserves its index so
streamed parts never collide, but with nothing to say there is nothing
to render: until a description exists the block looks exactly as it
does without the feature.

* 🧹 fix: Drop the Localize Hook Left Unused by the Counts Removal

* ✅ test: Add Activity-Label e2e Coverage with a Recording Label Server

Activity labels are the one model call a mock run does not already fake:
fake-model.js swaps the GRAPH model via overrideTestModel, while
run.generateActivityLabel() calls the endpoint resolved client options
over HTTP. The custom endpoints already point baseURL at 127.0.0.1:8889,
so serving that port exercises the real path with no production seam.

fake-label-server.js answers it in both JSON and SSE form, records each
prompt, and can inject blank/error responses. Recording is what lets the
spec assert the CONTRACT rather than the rendering: that this repo
register and the tool OUTPUTS actually reach the model. That is the bug
class that produced unusable labels before, and rendered text looks
identical whether or not the instruction arrived.

Labels get a dedicated endpoint (Mock Provider E). A labeled block
auto-collapses even at one tool call, which hides the tool cards other
specs assert on -- enabling this on a shared endpoint broke
steering.spec.ts. Provider D is the unlabeled control.

Request-count assertions are scoped to a per-test token: a 5xx label
response is retried by the provider client, and a retry can land after
the next test has reset the server.

* 🩹 fix: Address Review Findings on Activity-Label Indexing and Pricing

Replay index (P1). Reserving the slot only in server memory left no event
for it, so a cross-instance replay rebuilt content as [tool, hole, later],
compacted the hole away, and the fill for the reserved index landed on the
following part and overwrote it. The claim now publishes the empty,
pending part so the index is real for every consumer, and fill publishes
even when generation returned nothing so the client cannot stay pending.

It stays invisible: an empty label still DELIMITS its batch in
groupSequentialToolCalls but is not attached as the header, so grouping
does not re-shuffle when the text lands and the block renders exactly as
it does with the feature off.

Edited-response index (P1). Edit-and-resubmit replays the kept prefix and
the server indexes only new content, so run steps offset by that prefix.
Labels are claimed in the same space and now take the identical shift;
without it a label could land inside the prefix and overwrite it.

Redis flag. deserializeJob never read activityLabels back, so every Redis
reload left it undefined and resume skipped label gap reconciliation.

Executing agent. RunActivityLabelOptions.agentId selects the executing
agent tracing metadata AND its tool-output redaction policy; omitting it
let a handoff be redacted under the default agent configuration.

Label pricing. An undefined endpointTokenConfig is meaningful for a
built-in label endpoint (priced from the shared table), so the nullish
fallback billed those labels at a custom primary rates. Inherit only when
the label runs on the agent own endpoint.

HITL usage sequence. runId is the response message id and the counter was
instance-local, so a resume restarted at -1 and the client runId:seq
deduper discarded the post-approval label usage. Seeded past the labels
already on the response.

Also distinguishes "cannot serve" (undefined) from "no label" (null) in
the SDK bridge, so a missing run falls back to the direct call instead of
filling the slot empty. Version gating already happens at wiring time via
the sdkCapable prototype probe.

* 🩹 fix: Keep Unfilled Activity Labels Invisible and Unmask Endpoint Settings

Follow-up review round. Publishing the reservation on every batch made two
latent rendering paths reachable on every run, and both are fixed here.

Empty labels no longer change grouping. The previous pass still formed a
tool-group for a textless label, which wrapped even a single tool call and
pulled THINK parts inside it — and since a reservation is published the
moment each batch ends, that applied during every generation and
permanently after a blank or failed fill. An empty label now flushes the
legacy way instead: it still delimits its batch, but the block re-splits
exactly as it renders with the feature off.

Parallel lanes no longer show a blank line. Lanes render raw parts, so an
unfilled label had nothing to draw; empty ones are dropped. Making labels
act as collapsible headers inside lanes is still a separate gap.

Edited responses no longer offset on resume. The sync replaces
initialResponse.content with the server's aggregatedContent, which already
contains the kept prefix AND everything generated since — so its length is
not the prefix length, and indices reconciled from that snapshot are
already absolute. Offsetting again pushed the label past its slot onto a
later part. The shift now applies only to a fresh edited submission.

Activity settings resolve field by field. Selecting one config object
whole meant any endpoints.all block — even one carrying nothing but
headers — shadowed the named or custom endpoint and silently disabled
activity labels everywhere. Global still wins per field.

Adds groupToolCalls coverage for the invisible-while-empty contract, which
is the part most likely to regress: it is normal state on every run, not
an edge case.

* 🔒 fix: Scope Detached Label Writes to Their Generation Epoch

Epoch scoping (P1). Label generation is detached and can outlive the
generation that started it. emitChunk only proves that SOME runtime is
current, not that the caller belongs to it, so an aborted generation's
fill(null) -- and its usage event -- could be attributed to whichever
generation replaced it, landing an index from the abandoned response on
top of the new one. Because an empty label renders nothing, that
overwrote content silently. emitChunk now takes an optional jobCreatedAt
and drops the event when the runtime epoch differs, mirroring the
existing setGraph/setContentParts convention, and both label emitters
pass it.

An abort now CLOSES the label scope instead of only cancelling the call:
the rejected generation still runs its catch and calls fill(null), which
would otherwise emit into a stream the next generation may already own.

Edited-response indexing (P1). The previous pass skipped the prefix
offset on resume, which was the wrong half of the problem: a sync
replaces initialResponse.content with the server's aggregatedContent,
which is completion-local, so after a reconnect its length is not the
kept-prefix length and the offset is wrong -- but it is wrong for run
steps in exactly the same way. Tool cards and the label that heads them
must share one index space; a label shifting differently from its tools
lands on another part. The label path now uses the identical expression
as useStepHandler, with no resume special-case. Correcting the
post-resume prefix length belongs in calculateContentIndex, where it
fixes both at once.

titleModel masking. The activity settings were made per-field last pass,
but the titleModel fallback a few lines below still selected an entire
config object, so a partial endpoints.all (for example one carrying only
headers) hid a named endpoint's titleModel and quietly fell the label
back to the main agent model. Both now read through one shared per-field
helper.

Resume reconciliation no longer depends solely on markActivityLabels,
which is best-effort yet had come to gate correctness: a lost flag write
silently dropped a label. The snapshot is consulted as a fallback.

The exported host type for generateLabel now admits undefined, which is
the documented "cannot serve, fall back to the direct call" signal the
hook keys on -- distinct from null, meaning it ran and produced nothing.

* 🧷 fix: Keep Group Identity Stable and Memoize Label Endpoint Resolution

Group remount. Tool-group identity was keyed on the first part in the
block. An activity label absorbs the block's leading THINK part the moment
its text lands, so the key flipped from tool:<id> to fallback:<scope>:<idx>
mid-run, remounting the group and discarding whatever the user had
expanded. The key now scans for the first tool call, which does not move
when the block re-forms.

Label endpoint resolution is memoized per response. It reads provider
config and can hit the database for user keys, yet nothing it depends on
changes between batches of one run — and it ran twice per batch, once for
generation and once for usage accounting. The promise is cached rather
than the value so concurrent batches share a single in-flight resolution,
and a rejection is evicted so one transient credential failure cannot
disable labels for the rest of the response.

* 🎯 fix: Offset Edited Resubmissions by a Prefix Length That Survives Resume

The server indexes only NEW content for an edited resubmission, so the
client offsets incoming indices by the prefix it retained. That prefix was
read as initialResponse.content.length, which is correct only until a
resume: the sync replaces that array with the server's completion-local
snapshot, whose length is unrelated to the prefix. After a reconnect every
offset was therefore wrong -- run steps and activity labels alike -- and
could write over content the edit kept. For a label the symptom is worse
than a bad position: the fill misses its own reservation, so the pending
placeholder is never resolved.

The prefix length is now captured when the submission is built, while
initialResponse.content still IS the retained prefix, and carried on the
submission as editPrefixLength. calculateContentIndex takes that length
instead of deriving it from an array that a resume may have replaced, so
run steps and labels share one index space by construction rather than by
both happening to read the same field.

Note the prefix is the FULL original content with the edited part
substituted in place (useChatFunctions clones latestMessage.content and
mutates one entry) -- it is not a slice, so the length cannot be inferred
from editedContent.index.

Group identity no longer changes when a label fills. Tool-group keys were
derived from the first part in the block; an activity label absorbs the
leading THINK part when its text lands, flipping the key mid-run and
remounting the group, which discarded the user's expansion state. The key
now scans for the first tool call, which does not move.

Label endpoint resolution is memoized per response. It reads provider
config and can hit the database for user keys, yet ran twice per batch --
once to generate, once for usage accounting -- while nothing it depends on
changes within a run. The promise is cached so concurrent batches share one
in-flight resolution, and rejections are evicted so a transient credential
failure cannot disable labels for the rest of the response.

The resume gap passes for steers and activity labels now share a single
lazy content read instead of each issuing its own. The label pass stays
gated on the run flag with a snapshot fallback: reconciling
unconditionally would also close the residual first-label window, but it
would bill a read to every resume of every run, including deployments with
the feature off -- which the steer pass deliberately avoids. That residual
requires a lost flag write, which shares fate with the content writes the
labels live in.

* 💵 fix: Bill Cross-Endpoint Labels at Their Own Rates

recordCollectedUsage never accepted an endpointTokenConfig, so the value
the activity-label caller passed was dropped and the balance transaction
was written at the primary agent's rates. Only the UI cost honored the
label endpoint, so a custom primary pointing activityEndpoint at another
endpoint showed one price and charged another. The parameter is now
accepted, and an explicit config wins outright over per-agent resolution:
that map is keyed by AGENT, so it cannot describe usage that ran on a
different endpoint.

Group identity is stable for id-less tool calls too. The previous pass
anchored the key to the first tool call ID; where a supported tool call
carries no id the fallback still used the block's first part index, which
shifts when a filled label absorbs the leading THINK part. The fallback now
anchors to the first TOOL entry's index, so only a block containing no
tool call at all keys off parts[0].

markActivityLabels is retried rather than fire-and-forget. It gates resume
gap reconciliation and is a SEPARATE write from the durable label append,
so a single lost write silently drops a label the content itself recorded.
The earlier "shared fate with content writes" reasoning was wrong. One
retry at run setup costs nothing and removes the only realistic way the
gate goes stale, without billing a content read to every resume.

* 🧮 fix: Stop Offsetting Once SYNC Drops the Edited Prefix

The edit offset was applied unconditionally, but whether it is correct
depends on which branch SYNC took. SYNC either preserves the content
already loaded for the response -- which still contains the retained
prefix, so the offset is required -- or replaces it with the server's
aggregatedContent, which is completion-local and indexed from zero, after
which any offset writes past the end of a now shorter array.

That is why the two previous attempts each fixed half of it: skipping the
offset on resume was right for the replace branch, applying it
unconditionally was right for the preserve branch, and neither holds on its
own. The offset now tracks the actual state of the rendered content.

For an activity label the replace branch was worse than a bad position:
the fill landed past its own reservation, so the pending placeholder was
never resolved and the block kept its generic header for the rest of the
run.

Applied to run steps as well, not just labels. useStepHandler reads the
prefix from the same submission and had the same unconditional offset, so
after a mid-session resume of an edited response tool cards were misplaced
too. Normalizing at the dispatch boundary keeps both in ONE index space by
construction: a label that shifted differently from the tools it heads
would land on another part.

Note the reload path was already coherent -- useResumeOnLoad rebuilds the
submission without editedContent or editPrefixLength, giving no offset
against server-supplied content -- so only the mid-session SYNC path was
inconsistent.

* 🧾 fix: Keep Label Accounting Out of the Primary Usage Slot

Label usage no longer owns getStreamUsage(). recordCollectedUsage assigned
its result to this.usage unconditionally, so when the primary provider
reported no usage metadata but the label provider did, BaseClient took the
label's output tokens as the assistant response's authoritative count. The
later primary call returns early on an empty collectedUsage and never
replaced it, so the wrong value stood, the text-based fallback was skipped,
and the real generation went unbilled. Secondary usage is still billed but
no longer writes that slot.

Cross-endpoint pricing keys off an explicit discriminator rather than the
presence of a value. A built-in label endpoint prices from the shared
table, so an undefined endpointTokenConfig is its MEANINGFUL value --
reading that as "no override" fell back to the primary's custom rates and
restored the exact mismatch the previous pass set out to fix. The caller
already knows whether the label ran elsewhere and now says so.

markActivityLabels rejects on failure instead of swallowing it. The flag
gates resume gap reconciliation and the caller retries it, but the internal
catch resolved successfully and made that retry unreachable -- so the two
changes cancelled out and a transient write failure still left the flag
absent.

Late label accounting is suppressed with the same gate as the late fill. A
straggler that outlived the settle timeout still ran its finally block, so
it charged the balance and appended to usageEmitSink after the response had
passed its usage flush and metadata snapshot: a cost the user pays but is
never shown.

The cleared-prefix state is scoped to one generation. It was set on a
resume SYNC that replaced the response and then never reset, so a later
edited resubmission in the same mounted hook dispatched run steps and
labels with no offset against content that still held its retained prefix.
Reconnects pass isResume and keep the state; a new generation clears it.

* 🔑 fix: Key Prefix State to the Stream and Honor current_model for Labels

The cleared-prefix reset keyed on isResume, which skips exactly the case it
was added for: a submission whose POST succeeded server-side but lost its
response is retried, comes back resumed: true, and subscribes in resume
mode even though it is a NEW generation. A previous generation's cleared
state then survived into it, and incoming run steps and labels applied no
offset against content that still held its retained prefix. The state is
now keyed to the stream id, which changes with the generation and stays put
across reconnects of one.

activityModel now honors current_model. The options are documented as
title-shaped and the titleModel fallback already excludes the sentinel, but
the higher-precedence activity override passed the literal string through to
getOptions and the provider, so an endpoint following that convention failed
every label instead of using the agent model.

* 🎯 fix: Key Prefix State to the Generation and Resolve the Run Model

The cleared-prefix state was keyed to the stream id, which never changes
within a conversation: request.js sets streamId = conversationId, so once a
reconnect cleared the state every later edited resubmission in that
conversation dispatched run steps and labels with no offset and could
overwrite the prefix it retained. It is now keyed to the response message
id, the only per-generation identity available here -- minted per
submission and carried through a resume unchanged.

That is the third identity tried for this state. isResume missed the
deduplicated-retry path (a lost response returns resumed: true for a new
generation); the stream id is conversation-scoped. The response id is the
boundary that actually matches a generation.

current_model labels now resolve the model the run is really using.
initializeAgent merges the request's endpointOption override into
model_parameters and the run gives it precedence, so preferring the saved
agent.model could send labels to a different, potentially unavailable or
more expensive model than the conversation is on.

* 🆔 fix: Key Prefix State to the Submission and Keep the Origin Title Model

Editing an assistant response reuses that response's messageId as
editedMessageId, and useChatFunctions carries it onto
initialResponse.messageId -- so re-editing the same response produced two
generations with the same key and the cleared-prefix state survived between
them, leaving run steps and labels with no offset against content the edit
retained. Keyed now to clientRequestId, the per-submission uuid, which is
minted fresh per edit attempt and forwarded unchanged on retries.

That is the fourth key this state has had, and each earlier one failed at a
real boundary: isResume missed the deduplicated-retry path, the stream id is
the conversation id, and the response message id is reused across edits of
one response. clientRequestId is the identity that actually means "this
submission".

The titleModel fallback is read from the ORIGINATING endpoint again, matching
how titleConvo captures its config before switching credentials. Reading it
after an activityEndpoint switch meant an OpenAI endpoint configured with
titleModel claude-haiku and activityEndpoint anthropic fell through to the
OpenAI run model and sent that name to Anthropic, failing every label. The
destination endpoint supplies credentials, not the model choice.

* 🧷 fix: Close the Remaining Edit, Epoch, and Scope Gaps for Labels

SYNC clears the edit prefix on the new-row branch too. When a resumed
edited submission cannot match an existing assistant row, that branch
builds the response straight from the server's completion-local
aggregatedContent, so it holds no retained prefix -- but the reset lived
only in the matched branch, leaving later steps and labels adding an
offset to indices that were already absolute.

Label usage is keyed per GENERATION. Editing one assistant response reuses
its responseMessageId while each fresh generation restarts
activityLabelUsageSeq, so a second edit re-emitted the same runId:seq and
the client discarded the newer usage while its balance transaction was
still written. The key now carries jobCreatedAt, the run's own epoch:
stable across reconnects and HITL resumes, distinct between generations.

The scope is revalidated at commit time. Checking once before the await let
a scope that closed mid-flight still charge the balance after finalization,
while the matching fill saw the closed scope and dropped the label --
billed but never surfaced, the exact outcome the guard exists to prevent.

The titleModel fallback no longer reaches the destination endpoint. With
activityEndpoint set and no titleModel on the originating endpoint, it
picked up the destination's, so changing only the credential target
silently changed the model and its cost. Precedence is activityModel, then
the originating endpoint's titleModel, then the run model; the destination
supplies credentials only.

* ✂️ refactor: Confine the Edit-Prefix Offset to Activity Labels

useStepHandler is now byte-identical to dev again. The resume-aware prefix
offset was applied there too, which was more correct in principle -- the
post-resume prefix length is genuinely wrong for run steps as well -- but it
changed index math that EVERY run step flows through, for every user,
including everyone who never enables activityLabel.

That shared correction needed five revisions in two days (isResume, the
stream id, the response message id, clientRequestId, and the SYNC new-row
branch), each passing the full suite and each failing at a boundary only
review found. Carrying it inside an opt-in feature put every user behind
logic with that track record. It belongs in its own change, with tests that
construct the edit-plus-resume states none of the current suites reach.

The offset now applies only where the label handler places its part, so
this PR cannot alter rendering for anyone with the feature off. The known
consequence is recorded in the description: with activity labels ENABLED,
an edited response that reconnects mid-generation can place its label and
its tool cards in different index spaces. That is a bug for opt-in users
rather than a regression for everyone, and it disappears once the shared
fix lands.

submission.editPrefixLength stays: the label path still needs a prefix
length that survives a SYNC replacing initialResponse.content.

* 🧾 fix: Commit Labels Before Billing and Keep Blank Slots Invisible

Round-nine review (all P2, feature-scoped):

- Billing ordering (client.js:409, runtime.ts): usage accounting ran
  BEFORE the slot commit on both generation paths, so the settlement
  deadline could expire during the balance write — charged, then the
  fill dropped as out-of-scope: billed, never shown. `slot.fill` now
  resolves a commit flag, generators register their accounting via
  `deferUsage`, and the hook runs it only after a committed fill.
- Scope gates (client.js:757): the direct-fallback `collect` omitted
  `scopeOpen`; both paths now gate on the OWNING wiring's scope, so a
  pre-pause straggler cannot bill because the resumed generation's
  scope is still open.
- Blank-label grouping (groupToolCalls.ts:81): a blank slot forced a
  flush, splitting adjacent single-call batches into standalone cards
  where the feature-off path merges them. Blank labels now only mark
  the claim boundary — structurally invisible, while a later filled
  label still cannot claim an earlier batch.
- Stale fill indices (wiring.ts:301): the skill-card unshift and the
  hide-sequential filter reshape contentParts before the finalization
  settle, so an in-flight fill emitted its claim-time index against a
  shifted array. Both completion paths now settle label fills before
  any post-run content reshaping (the finally settle stays as the
  error-path net; the second call sees an empty pending list).
- Bounded serialization (runtime.ts:238): `JSON.stringify` fully
  materialized unbounded tool results to keep 200/600 chars per entry.
  A budget-bounded serializer stops at the limit (which also bounds
  cyclic values) and preserves the exact truncate-with-ellipsis output.

Tests: fill/bill ordering + suppression on dropped fills (runtime.spec),
blank-slot merging and claim boundaries (groupToolCalls.test), bounded
serialization equivalence and giant-output truncation (runtime.spec).

* 🧮 fix: Keep Deferred Label Billing Inside the Settle Window

Self-review follow-up to the billing reorder: deferring usage until
after the commit moved it PAST the fill's resolution, so a settle keyed
on fills alone could let finalization flush the usage sink and snapshot
metadata while the label's billing was still in flight — the usage row
would silently miss the message rollup even on the happy path.

The hook now reports its whole detached task (generate → fill →
deferred usage) via a `trackTask` option, wired to the same settle
tracker as the fills, so finalization waits for billing exactly as it
did when accounting preceded the fill. The task never rejects. Pinned
in runtime.spec: the tracked task resolves only after usage collection.

* 🧰 fix: Harden Label Resolution, Output Bounds, and Cache Billing

Round-ten review (all P2, feature-scoped); the sixth finding is the
documented edited+reconnect index-space limitation, answered on-thread
as deliberately out of scope for this PR.

- Rejected-LLM memoization (runtime.ts): the hook cached a rejected
  `resolveLLM()` promise permanently, failing every later batch and
  silently defeating the host resolver's own rejected-cache eviction.
  The memo now evicts on rejection so the next batch retries.
- `current_model` precedence (host.ts): an explicit
  `activityModel: current_model` resolved to `undefined` and then lost
  to a configured `titleModel`. The sentinel now resolves straight to
  the run model; the title fallback applies only when `activityModel`
  is absent.
- Output bounds (runtime.ts): label text was persisted verbatim; a
  model ignoring the 4–9-word instruction (or steered by injection in
  untrusted tool output) could emit thousands of tokens duplicated
  through SSE, the chunk log, persistence, and the UI.
  `normalizeLabelOutput` keeps the first non-empty line, collapses
  whitespace, and hard-caps at 200 chars on both generation paths.
- Cache-token billing (host.ts, client.js): the usage mapper dropped
  cache fields, vanishing Anthropic cache tokens from billing and
  charging OpenAI cache reads at the full input rate. The mapper now
  normalizes Anthropic/OpenAI/LangChain cache shapes into
  `input_token_details`, and the emit + cost path carries them with the
  label endpoint's `provider` (additive-provider adjustment).
- Usage-type union (runs.ts): `TTokenUsageEvent.usage_type` now
  includes the emitted `activity-label` literal; the lone consumer
  keys on `usage_type != null`, so this is type-level completion.

Tests: sentinel/title/explicit model precedence and all three cache
shapes (host.spec), transient-resolution retry and output normalization
with truncation (runtime.spec), the new usage literal (runs.spec).

* 🪗 fix: Let Settled Labels Collapse Void Tools and Keep the Tail Cursor

Round-eleven review (all P2, client-side). Two fixed; the other two
findings restate documented Known limitations (edited+reconnect run-step
index space; parallel-lane collapsible headers), answered on-thread.

- Void-tool auto-collapse (ToolCallGroup.tsx): `allCompleted` keyed
  solely on output truthiness, so a tool that legitimately returns an
  empty string kept its labeled group expanded forever. A settled,
  filled label is itself a completion proof — the PostToolBatch claim
  only happens after every output in the batch returned — so it now
  satisfies `allCompleted`; pending labels keep the group live.
- Trailing-reservation cursor (ContentParts.tsx): a blank label
  reservation at the content tail renders nothing but still counted as
  the last part, stripping the streaming cursor and last-item
  affordances from the last VISIBLE part until the next delta.
  `lastContentIdx` now walks back past empty label slots.

Tests: labeled void-tool group auto-collapses, pending-label group
stays expanded (ToolCallGroup.test).

* 💳 fix: Price Label Cache Correctly, Honor endpoints.agents, Cancel Every Retry

Round-twelve review: four fixed here; the remaining P1 (move the
client.js bridge into packages/api) is an architecture call answered
on-thread for the maintainer.

- Provider on billed entries (client.js, P1): round ten added cache
  details to label usage entries but not `provider`, and `splitUsage`
  treats an unknown provider as additive — re-adding cache_read and
  cache_creation on top of an input count that already contains them,
  double-charging Anthropic/OpenAI cached label calls while the
  streamed cost (which carried the provider) disagreed. Every mapped
  entry now carries the label endpoint's provider.
- endpoints.agents honored (host.ts, client.js): `initializeAgent`
  rewrites `agent.endpoint` to the backing provider, so activity
  settings under the PUBLIC `agents` endpoint — valid config, inherited
  by `agentsEndpointSchema` — were silently ignored. Field resolution
  is now `all` > public endpoint > backing provider/custom, applied to
  both the enable gate and the model/titleModel resolution.
- E2E_LABEL_PORT reaches the YAML (playwright.config.mock.ts): an
  overridden port moved the fake label server and its health check but
  not the generated config's hard-coded 8889 baseURLs, so readiness
  passed while every label request targeted the wrong port. The
  override is now substituted into the generated copy.
- Every retry frame cancelled (useResumableSSE.ts): concurrent label
  retry chains (reservation + fill per slot) overwrote one rAF handle,
  so cleanup cancelled only the newest chain; the rest ran up to 120
  frames past unmount and could apply a stale label to a replacement
  generation reusing the same response id. Outstanding frame ids now
  live in a Set that cleanup drains.

Tests: public-endpoint gate/precedence/all-above-public (host.spec).

* 🖱️ fix: Keep the Last-Part Cursor in Parallel Lanes Too

Round-thirteen review (single P2): `ParallelContentRenderer` computed
`lastContentIdx` from the unfiltered array, so a trailing blank label
reservation — filtered out of every lane — left NO rendered part
carrying the last-part cursor and running-subagent affordances until
the label filled.

The sequential renderer's walk-back is extracted into a shared
`lastVisibleContentIdx` helper (utils/activityLabels) used by both
`ContentParts` and `ParallelContentRenderer`, so the two index spaces
cannot drift again. Behavior pinned in activityLabels.spec: trailing
blank skipped, consecutive blanks skipped, filled label counts,
label-free content unchanged.

* 🧹 chore: Alias the Retry-Frame Set for the Effect Cleanup Lint Rule

* 📏 fix: Let activityCharLimit Reach Tool Inputs

Round-fifteen review: `activityCharLimit` is documented as the
per-entry truncation for tool input AND output, but `buildPrompt`
hard-coded inputs at 200 characters — so raising the setting could
never surface a distinguishing path, query, or operation that appears
past the first 200 characters of a long argument. Inputs now truncate
at the configured limit alongside outputs; the 200-char constant
remains only for the intent line (renamed INTENT_CHAR_LIMIT to match).
Config fidelity pinned in runtime.spec: a 400-char argument survives a
450 limit and truncates under a 50 limit.

The round's other finding is the fifth restatement of the documented
edited+reconnect index-space limitation, answered on-thread with the
prior four cross-references.

* 🤝 fix: No Labels for Pure Handoff Batches

Round-sixteen review: a PostToolBatch containing only `transfer_to_*`
calls claimed a label slot, but transfer parts are never groupable —
the client flushed the handoff card standalone and the label orphaned
into a stray line after it, restating what the card already says.

Two-sided fix:
- Hook (runtime.ts): a batch whose every entry is a transfer call
  claims nothing — no slot, no model call, no `maxPerRun` consumption.
  Mixed batches still label (the header describes the real work).
- Renderer (groupToolCalls.ts): an orphan label whose `tool_call_ids`
  are all transfer calls is dropped instead of rendered standalone,
  covering content persisted before the hook-side skip.

The round's two P1s are repeats answered on-thread: the packages/api
extraction (maintainer-decided follow-up, recorded in the description)
and the sixth restatement of the edited+reconnect index limitation.

Tests: transfer-only batch claims nothing, mixed batch still claims
(runtime.spec); transfer-only orphan label dropped, real-batch orphan
label still renders (groupToolCalls.test).

* 🎛️ fix: Sanitize Label Client Options and Bound the Batch Prompt

Round-seventeen review: two fixed; the other two findings repeat the
maintainer-decided packages/api extraction (follow-up) and the
edited+reconnect index limitation (seventh instance), answered
on-thread.

- Primary-option strip (host.ts): the label client copied the resolved
  `llmConfig` wholesale, so an endpoint whose defaults enable extended
  thinking or carry model-specific output caps forwarded them to the
  (often cheaper) label model — unsupported options failed every label,
  and supported thinking spent real tokens and the settlement window on
  a 4–9 word header. The copy now strips `omitTitleOptions` keys and
  the `modelKwargs` output caps exactly like the title path, restoring
  the Anthropic `clientOptions` carrier by reference so proxy
  `defaultHeaders` still reach label requests.
- Batch prompt budget (runtime.ts): per-entry truncation left the batch
  dimension unbounded — hundreds of parallel calls could build a prompt
  past the fast model's window. The entries section now has a total
  budget (8k chars, scaling with `activityCharLimit` so a raised limit
  still fits several entries); entries past it are skipped without
  paying their serialization cost, and the list notes how many were
  omitted. The first entry always renders in full.

Tests: option strip with header-carrier survival (host.spec); giant
batch bounded with omission marker, small batch untouched
(runtime.spec).

* 🛡️ fix: Keep SSRF Guards on Label Calls, Skip Mixed Handoff Batches

Round-eighteen review: four fixed; the fifth repeats the
maintainer-decided packages/api extraction (eighth instance), answered
on-thread.

- SSRF-safe carrier (host.ts, P1): the sanitize step restored the
  Anthropic `clientOptions` carrier only when `defaultHeaders` existed,
  but for user-provided base URLs `getLLMConfig` stores the guarded
  Undici dispatcher and `redirect: 'error'` there — dropping it
  reopened DNS-rebinding/redirect paths on label calls to
  user-controlled URLs. The carrier (client CONSTRUCTION options, not
  generation params) is now restored whenever present, same reference.
- Primary maxTokens (host.ts): top-level `maxTokens` is not in
  `omitTitleOptions` and survived the strip; the title path deletes it
  explicitly, and a cap sized for the primary model can be rejected by
  the substitute. Deleted on the copy.
- Bounded keys (runtime.ts): the object branch materialized every key
  via `Object.keys` and quoted oversized keys in full before the budget
  check. Enumeration is now lazy (`for..in` + own-property guard) and
  keys slice to the budget before quoting, like string values.
- Mixed handoff batches (runtime.ts, groupToolCalls.ts): the client
  flushes the block at the transfer card, so a mixed batch's label
  orphaned exactly like a pure one. The hook now skips ANY batch
  containing a transfer call, and the renderer drops orphan labels
  covering one (legacy content).

Tests: carrier survival without headers by same reference, maxTokens
strip (host.spec); mixed batch claims nothing (runtime.spec); mixed
orphan dropped, real-batch orphan kept (groupToolCalls.test).

* 🧢 fix: Cap Label Generation, Order the Flag Persist, Detach Settled Listeners

Round-nineteen review: three fixed; the fourth is the ninth instance of
the edited+reconnect index limitation, answered on-thread.

- Generation cap (host.ts): stripping the primary output caps left
  label calls with NO cap at all — `normalizeLabelOutput` bounds what
  persists, not what the provider generates and bills, so a model
  ignoring the 4–9-word instruction (or steered by injected tool
  output) could emit its provider-default output per batch. The
  sanitize step now installs a 256-token label cap (per provider
  family: `maxOutputTokens` for Google-style wrappers, `maxTokens`
  otherwise), after the filter so the omit set cannot remove it.
- Flag-persist ordering (client.js): the `markActivityLabels` write was
  fire-and-forget, so an immediate cross-replica reconnect could read
  the job between the write and the first claim, see neither flag nor
  snapshot label, and skip gap reconciliation. Label emission now
  awaits the (settled-on-failure) persist chain, making "a label event
  exists" imply "the flag is durable" — the race window is gone; only
  the documented double-write-failure residual remains.
- Listener detach (client.js): each HITL approval cycle's wiring adds a
  `once` abort listener to the shared job signal that only an actual
  abort removes; settled segments now detach theirs in
  `settleActivityLabels`, so long multi-approval runs cannot accumulate
  dead closures toward the listener-limit warning.

Tests: the primary cap is REPLACED by the 256-token label cap
(host.spec).

* 🎯 fix: Route the Label Cap Per Model Family

Round-twenty review: the 256-token label cap set maxTokens
unconditionally, but GPT-5+ rejects max_tokens (the OpenAI builder
routes its cap into modelKwargs.max_completion_tokens /
max_output_tokens) and o-series models reject it with no stable kwargs
alternative — every label on those models would have failed. The cap
now mirrors the builder: modelKwargs for GPT-5+ (responses-API aware),
no cap for o-series (title parity; the 200-char persistence bound
still applies), maxOutputTokens for Google, maxTokens otherwise.
Pinned in host.spec for both reasoning families.

The round's other finding is the tenth instance of the documented
edited+reconnect index limitation, answered on-thread.

* ⏱️ fix: Persist the Label Flag at Run Start, Not on the Emit Path

Round-twenty-one review: two fixed; the other three repeat the
maintainer-decided packages/api extraction, the edited+reconnect index
limitation, and the parallel-lane header limitation — all answered
on-thread with their standing decisions.

- Flag ordering, corrected (client.js): sequencing label emission
  behind the flag persist (previous round) delayed the claim-time
  reservation while the shared index offset had ALREADY shifted
  subsequent SDK chunks — reopening the cross-instance
  hole-compaction overwrite the reservation emit exists to prevent.
  The reservation emits immediately again; instead, run start
  (processStream and resume alike) awaits the settled-on-failure
  persist chain, so the flag is durable before any batch can claim a
  label. Same guarantee, zero latency on the emit path.
- Tail-label cursor (ContentParts.tsx): a filled label at the content
  tail is consumed into the group header rather than listed in
  `group.parts`, so the `isLast` check missed it and nothing held the
  streaming cursor until the next delta. The check now includes
  `labelPart.idx`.

* 🔌 fix: Detach Label Abort Listeners Even Without Claims

A segment with labels enabled can end without a single claim (text-only, or handoff batches, which skip labels); the early return in settleActivityLabels skipped the detach added for HITL listener accumulation. The detach now runs on both paths.

* ⚖️ fix: Make the Commit Flag the Sole Billing Authority

Round-twenty-three review: a committed fill racing a late scope close
(user abort or settle timeout during the durable emit) stayed visible
— the part is mutated and persisted before the close — yet the
deferred accounting's scope gates then skipped the charge: a completed
provider call escaping both the label charge and the primary abort
accounting.

The scope gates on the deferred-usage path are removed; the hook's
commit flag is now the single billing authority in BOTH directions. A
dropped fill never reaches the accounting callback (billed-never-shown
stays impossible), and a committed fill bills regardless of when its
scope closed (shown-never-billed now impossible too). The dead
`scopeOpen` payload threading is removed with it; the
`recordActivityLabelUsage` parameter survives, defaulting open, for
callers that own no commit signal.

The round's other finding is the twelfth instance of the documented
edited+reconnect index limitation, answered on-thread.

* 🧮 feat: Bill Labels by Estimate When Providers Omit Usage

Maintainer decision: follow the title convention rather than leaving
label calls unbilled when a provider returns no usage metadata.

The hook now passes a LAZY estimate thunk with the deferred accounting
on the success path — the EXACT prompt the direct path sent (or the
locally built equivalent for the SDK path: same entries, context,
instruction, truncation contract, and continuity headers) plus the
final normalized label. `recordActivityLabelUsage` invokes it only
when no collected entry carries a real token count, counts both texts
with the shared o200k_base tokenizer, and feeds the synthesized entry
through the SAME pipeline (provider-tagged, streamed event, cost,
balance transaction). Real provider usage always wins when present.
The failure path passes NO estimate: a throw before a response bills
only real collected metadata, never a full phantom prompt.

Tests: the estimate thunk carries the exact invoked prompt and final
label; the failure path defers with no estimate (runtime.spec).

* 💵 fix: Estimate From the Raw Completion, Not the Normalized Label

The fallback estimate counted the normalized label (first line, 200-char cap) while the provider generated and would bill the raw output up to the 256-token generation cap — under-recording verbose replies. The estimate thunk now carries the raw pre-normalization text; the persisted label is unchanged. Pinned with a multi-line reply test.

* 🧾 fix: Commit Label Text Only After the Durable Emit, Estimate the Real SDK Prompt

Round review on the billing work: two fixed; the third is the
fourteenth instance of the edited+reconnect index limitation, answered
on-thread.

- Copy-first fill (wiring.ts): the fill mutated the shared content part
  BEFORE its durable emit, so a failed emit left the label text on
  `contentParts` anyway — persistence could save and display a label no
  client ever received and billing (keyed on the commit flag) never
  charged. The new state is staged on a copy; the shared part mutates
  only after the emit succeeds, so content, delivery, and billing move
  together.
- Real SDK prompt for estimates (client.js): the estimate thunk carried
  this module's locally built prompt, but the SDK path frames entries
  differently — the estimated input count was for a prompt never sent.
  Chain-start callbacks (handleLLMStart/handleChatModelStart) now
  capture the prompt the SDK actually rendered, and the deferred
  accounting substitutes it into the estimate when capture succeeded,
  falling back to the local approximation otherwise.
2026-07-29 14:05:47 -04:00
Danny Avila
d8427ffc5e
🛂 test: Cover Tool Approval Workflows End to End (#14427)
* test: cover tool approval workflows end to end

* fix: preserve tool approval state across resume

* fix: preserve agent context in mock stream responses

* fix: preserve nested approvals in collapsed groups
2026-07-26 21:58:25 -04:00
Danny Avila
f3159f9891
🧩 fix: Harden Agent Skill Lifecycles End to End (#14429)
Some checks failed
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Has been cancelled
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
* test: cover agent skill lifecycles end to end

* style: sort agent skill imports
2026-07-25 08:19:12 -04:00
Danny Avila
520af663bc
🧵 feat: Background Tool Calls for Agents & Model Specs (#14197)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🧵 feat: Background Tool Calls for Agents & Model Specs

Opt-in, poll-based background tool execution. The model marks an eligible tool
call with `run_in_background: true`; the host executor registers a task, returns
a handle immediately (so the graph turn resolves), runs the tool as a detached
promise, and the model retrieves the result via a new `check_background_task`
poll tool. Host-side only — no `@librechat/agents` change.

- Opt-in mirrors `deferred_tools`: admin capability `run_in_background`
  (off by default) + per-tool `tool_options.run_in_background`.
- Model specs / ephemeral agents: `TModelSpec.runInBackground` /
  `TEphemeralAgent.run_in_background` synthesize per-tool options; both paths
  converge at `initializeAgent`.
- In-process task registry: scoped per user+conversation, idempotent by
  toolCallId (safe across resume/replay), capped, TTL-swept.
- Excludes direct-path / host-special / code-session tools. Subagents and push
  notifications are deferred follow-ups.

* 🩹 fix: Harden background tool calls (Codex review)

- Reliable per-agent execution gate: thread the injected `run_in_background`
  tool names from `initializeAgent` through `configurable.backgroundToolNames`
  (`toolRegistry` only reaches the executor for PTC/tool_search), fixing the
  silent no-op + unstripped-arg leak for ordinary event-driven tools.
- Enforce the per-tool opt-in at execution (`backgroundToolSet.has(name)`) so a
  non-opted-in tool can't be backgrounded via an extra arg.
- Gate the `check_background_task` interception on the run actually enabling
  background, so a user tool sharing that name still executes.
- Forward `backgroundToolsAvailable` to added-convo (multi-convo) agents.
- Exclude `web_search`/`file_search` from eligibility — their results are turned
  into user-visible attachments/citations only by the foreground toolEndCallback.

* 🩹 fix: Address Codex round 2 on background tool calls

- Idempotency scoped to run+turn: provider tool-call ids repeat across turns
  (e.g. `call_0`), so key the dedupe map by `runId::toolCallId` and sweep
  orphaned mappings — a later turn no longer collides with a retained task.
- Artifacts preserved: a backgrounded tool's artifact is processed through the
  same `toolEndCallback` as the foreground path (images/files/citations no
  longer silently dropped), best-effort/guarded.
- Forward the `run_in_background` capability to connected-agent discovery and
  subagent `processAgent` init, so a child agent's own event-driven tools work
  the same as when it runs as primary.
- Strip the injected flag on foreground calls of background-capable tools
  (the model may emit it as `false`) so strict MCP/action schemas don't reject.
- `check_background_task` list path returns metadata only (result_available /
  result_chars), never full results — prevents context overflow; the full
  result is returned only when a specific id is requested.

* 🩹 fix: Address Codex round 3 on background tool calls

- Exclude background-capable tools from eager execution (run.ts): a speculative
  eager dispatch of a `run_in_background` call could launch the detached task
  with partial/stale args, and that side effect can't be canceled.
- Reserve the `check_background_task` name: overwrite a colliding user/MCP tool
  with the host poll schema (with a warning) so the advertised schema matches
  the executor's interception instead of hijacking a mismatched tool.
- Don't inject background schemas into pure subagents (spawn-tool child graphs)
  whose tools don't reach the host interceptor; keep it for primary/added/
  connected agents. Subagent background is the durable follow-up.
- Thread `backgroundToolsAvailable` + `backgroundToolNames` through the
  OpenAI-compatible and Responses agent routes (was chat-only), so the same
  agent/model spec behaves consistently across surfaces.
- Exclude image-generation built-ins (dalle/flux/gemini_image_gen/image_gen_oai/
  image_edit_oai) — artifact-first tools whose files can't reliably attach to an
  already-saved turn when backgrounded.

* 🩹 fix: Address Codex round 4 on background tool calls

- Sanitize self-spawn subagent inputs: strip `run_in_background` + the
  `check_background_task` def from the parent AgentInputs reused for self-spawn,
  so the isolated child (direct/child-graph path) doesn't advertise a background
  schema it can't honor. The SDK resolver keeps a provided `agentInputs` even
  with `self: true`.
- Exclude `check_background_task` from PTC (`run_tools_with_code`) tool
  definitions — it's host-only and not callable from generated code.
- Parse stringified JSON args before deciding background dispatch and before
  stripping the flag, so string-delivered `run_in_background` is honored and
  never leaks to strict object-schema tools.
- Skip injection for tools that already declare their own `run_in_background`
  param (would otherwise hijack/strip it), and for non-object (string-input)
  schemas (would otherwise rewrite the input contract).

* 🩹 fix: Address Codex round 5 on background tool calls

- check_background_task now parses stringified JSON args, so providers that
  deliver args as a string can retrieve a specific task by id (not just list).
- Include agentId in the background dedupe key (`agentId::runId::toolCallId`):
  two agents in the same run emitting the same provider id (e.g. `call_0`) now
  launch independent tasks instead of colliding.
- Self-spawn sanitization also strips the background entries from the reused
  toolRegistry (not just toolDefinitions), so a child using tool_search/deferred
  loading can't rediscover the host-only run_in_background / check_background_task.

* 🩹 fix: Strip run_in_background from PTC target tool schemas (Codex round 6)

The PTC path already filtered out the host-only check_background_task poll tool
but still exposed target tool schemas with the injected `run_in_background` param
(the shared toolRegistry entries were mutated by applyBackgroundToolCalls). PTC
codegen doesn't go through the host background interceptor, so it could pass the
flag to an MCP/action tool (strict-schema rejection or silent foreground with no
poll). Sanitize the PTC toolDefs like the self-spawn path does.

* 🩹 fix: Sanitize background from explicit subagent inputs (Codex round 7)

A child agent reachable as a top-level/handoff agent is initialized WITH the
background capability, then reused as an explicit subagent via buildSubagentConfigs.
Round 4 only sanitized the self-spawn case; this now applies the same
stripBackgroundFromToolDefinitions/Registry to explicit child agentInputs when
`child.backgroundToolNames` is non-empty, so an isolated child graph doesn't
advertise a run_in_background / check_background_task contract it can't honor.

* 🩹 fix: Reap stuck/expired background tasks (Codex round 8)

- get() now sweeps before returning, so repeatedly polling a known
  background_task_id can't keep an expired completed task (and its retained
  result, up to 100k chars) alive past the one-hour completed TTL.
- sweep() now reaps `running` tasks older than a 30-min running TTL, marking
  them errored. Previously a detached call that never settled (hung network /
  lost MCP connection) held a running slot forever, exhausting the
  per-conversation cap and rejecting every later dispatch.

* 🩹 fix: Evict oldest settled tasks instead of blocking at the cap (Codex round 9)

Only the running-task cap gates dispatch now. The total-tasks cap
(MAX_TASKS_PER_BUCKET) bounds memory but no longer rejects new background calls:
when full, it evicts the oldest settled (completed/error) tasks to make room.
Previously 200 quick background calls in one conversation would block all new
dispatches for up to the completed-task TTL, since polling doesn't remove settled
tasks. Running is already capped, so room always frees.

* 📝 docs: Frame background tool calls as within-turn (Codex P1 contract)

Codex escalated the request-lifecycle findings to P1 on the grounds that the
advertised "poll later" contract can't be honored for genuinely long-running
calls (request-scoped MCP connections + the run abort signal are torn down at
turn end). Align the model-facing contract with what the same-run implementation
actually delivers: the run_in_background param, check_background_task, and the
dispatch handle now instruct the model to collect the result WITHIN THE SAME TURN
(backgrounded work isn't guaranteed to survive past the turn). This is
within-turn parallelism; cross-turn survival of long-running calls remains the
deliberate durable subagent follow-up. Copy/comment-only; no behavior change.

* ♻️ refactor: Cross-turn background tool calls, leak-free

Extend background tool calls from within-turn to cross-turn on a single
process, since the mechanism already supports it: the run's abort signal
never reaches the detached invoke (the graph forwards only configurable/
metadata to the tool-execute handler), so the floating promise keeps
running past turn completion and its result stays in the in-process
registry for a later turn to poll (get/list key only on
user::conversation + id, never the dispatch run/turn).

Guarantee no connection leak: ephemeral request-scoped MCP tools (runtime
{{LIBRECHAT_BODY_*}} placeholders) capture their request-scoped store at
creation and fall back to it, so config manipulation can't redirect them;
their connection is torn down at request end. Tag such tools in
createToolInstance and run them in the foreground instead of backgrounding
them. Pooled/app-level MCP and structured tools are unaffected and survive
cross-turn via their managed pools.

Reword the model-facing contract (run_in_background, check_background_task,
handle message, fileoverview) from within-turn to cross-turn on this server
(not across restart/replica, which stays the durable follow-up).

Tests: cross-turn poll retrieval; ephemeral MCP tool runs foreground.

* 🐛 fix: Guard ephemeral MCP tag against a null server config

createToolInstance can be reached with a null/stale capturedServerConfig
(cached availableTools + getServerConfig returns null, as several MCP unit
tests construct tools). The new unconditional requiresEphemeralUserConnection
call then dereferenced config.source and threw during tool construction
(CI: Tests api shard 2/3). Guard with the same serverConfig ? ... : false
pattern the other callers use; a missing config is not request-scoped.

* 🎨 fix: Deliver backgrounded tool artifacts on the poll turn

A slow backgrounded MCP/action tool resolves after its dispatch turn is
finalized: createToolEndCallback only appends to that turn's artifactPromises
(already awaited) and writes to a closed stream, so the artifact (file/citation/
UI resource) was silently dropped — check_background_task recorded only the
hasArtifact boolean. The cross-turn contract made this the common case.

Hold the artifact on the task and deliver it through the LIVE poll turn's
toolEndCallback the first time check_background_task collects that id (once,
then cleared to free memory), attributed to the original tool. Same-turn and
cross-turn now share this path since the model must poll to collect any result.

Tests: registry claim-once; artifact delivered on poll not dispatch, idempotent.

* ✨ feat: Agent-builder toggle for background tool calls + cap tool descriptions

Add a per-MCP-tool "run in background" toggle in the agent builder, mirroring
the programmatic/deferred pattern: gated on the admin `run_in_background`
capability via useAgentCapabilities, read/written on tool_options[id]
.run_in_background through useMCPToolOptions (per-tool + bulk mark-all), and
rendered as a Zap toggle in MCPToolItem and McpSection with new locale keys.

Also cap the section tool/server descriptions (McpSection, ToolSection,
SkillSection) with max-h-40 overflow-y-auto so a long description scrolls
instead of overflowing the dialog, matching MCPToolItem's existing cap.

Tests: MCPToolItem renders/toggles the background button only when enabled.

* 🧪 fix: Mock new background hook functions in McpSection spec

* 🎨 fix: Restore background artifact when poll-turn delivery fails

* 🛡️ fix: Harden background tool call edges from review findings

- Error immediately (matching foreground) when a background-requested tool
  failed to load, instead of returning a success handle for a dead task
- Exclude ephemeral request-scoped MCP tools at injection time so the model
  never sees a run_in_background param the executor would silently downgrade;
  flip the execute-time tag to fail closed on a missing server config
- Source image-tool background exclusions from the shared imageGenTools set
  (adds missing stable-diffusion, an artifact-first live tool) instead of a
  hand-copied list
- Add check_background_task to the eager-execution exclusion list: artifact
  collection is a one-shot claim that must not fire from a speculative
  snapshot the SDK may discard
- Strip an imitated run_in_background arg on tools the executing agent never
  opted in (multi-agent history bleed), unless the tool's own schema declares
  the parameter
- Truncate oversized stored results with an explicit marker via the shared
  truncateMiddle (moved to utils/text) instead of a silent slice
- Document the at-most-once artifact delivery semantics honestly (the
  callback's downstream persistence is fire-and-forget, as in foreground)

* ♻️ refactor: Deduplicate background tool-call plumbing and tighten types

- Use the SDK's JsonSchemaType instead of a local duplicate; drop all
  as-unknown casts and type the poll-tool serializer explicitly
- Drop derivable BackgroundTask state (progress, hasArtifact) and the dead
  `enabled` param/return on applyBackgroundToolCalls (guarded at the call
  site), which also skips the defs pass when nothing opted in
- Fold the enable expression into synthesizeBackgroundToolOptions so the
  three load/added call sites can't drift
- Throttle the registry's all-buckets sweep and always sweep the accessed
  bucket, so a hot poll loop is no longer O(total tasks server-wide); bound
  retained artifact memory with a size cap
- Single-pass stripBackgroundFromToolDefinitions; pass metadata through to
  the poll-turn callback instead of a no-op reconstruction
- Collapse the client's copy-pasted boolean option families into a keyed
  factory (also removes the shared-object mutation in the bulk toggles) and
  the six toggle-button copies into one OptionToggle component

* 🧪 test: e2e coverage for cross-turn background tool calls

Proves the full contract through the real pipeline (mock harness): an agent
opts an MCP tool in via tool_options.run_in_background, the model dispatches
it detached and receives the synthetic handle while the tool is still running
(status=running in the rendered ack — the non-blocking guarantee without
timing assertions), the tool completes after its turn finalized, and a later
user turn recovers the task id from replayed history, polls
check_background_task, and renders the collected result.

- fake-mcp-server: slow_echo fixture tool (delayed echo)
- fake-model: E2E_BACKGROUND_DISPATCH / E2E_BACKGROUND_COLLECT markers
- e2e yaml: agents capabilities = defaults + run_in_background

* 🔧 fix: Close two background capability gaps from review

- Thread backgroundToolsAvailable through the OpenAI-compatible service
  (derived from app capabilities like codeEnvAvailable/statefulSessions),
  so agents with tool_options.run_in_background keep the feature on that
  route; fold the three capability derivations into one helper
- Index ephemeral MCP servers by normalizeServerName when excluding tools
  from background injection: tool names embed the normalized server name
  while mcpConfig keys the original, so exotic server names previously
  escaped the injection-time exclusion

* 🛂 fix: Fall back to configurable user identity for background task scoping

The in-repo routes merge req into the tool-execute configurable, but external
hosts of the exported OpenAI-compatible service inject their own loadTools and
may not — tasks would then register under an empty user id, collapsing
registry isolation to conversationId alone. Resolve the scoping id from
req.user.id, then configurable.user_id / user, and cover the isolation with a
foreign-user not_found test.

* 🧹 chore: Apply repo import sorter to PR-touched files
2026-07-13 12:51:36 -04:00
Danny Avila
397ddc5366
🧠 feat: Add Memory as an Agent Capability with Inline Tools and Ephemeral Badge (#13869)
* 🧠 feat: Memory Agent Capability with Inline Tools and Ephemeral Badge

Add `AgentCapabilities.memory`, which expands into the inline set_memory/delete_memory tool pair (mirroring the execute_code expansion via registerMemoryTools) when a run-level memoryAvailable gate holds: capability enabled, memory configured, MEMORIES.USE permission, and personalization not opted out. Surfaces the memory artifact as an attachment in the agents tool-end callback.

Adds the ephemeral path (TEphemeralAgent.memory, load/added agent tool injection), a fully-gated memory badge plus tools-dropdown entry, the agent-builder Memory toggle with form round-trip, and a mock e2e test asserting the badge reaches the request payload. Additive to and independent of the existing post-turn memory extraction agent.

* 🩹 fix: Address Codex review on memory capability (gating, validKeys, usage guard)

- Strip the memory capability from the served agents capabilities when memory is not configured/enabled, so the badge, tools dropdown, agent-builder toggle, and backend capability gate stay consistent instead of exposing an inert toggle on default installs (where MEMORIES.USE defaults true).
- Surface configured memory.validKeys in the inline tool definitions so the model is told the allowed keys up front, matching the runtime createMemoryTool schema.
- Append a strict explicit-request usage guard to the agent instructions when inline memory tools are registered, preserving the memory-agent's privacy behavior.
- Add AppService tests covering memory-capability stripping.

* ✅ test: Update AppService capability snapshots for memory strip

AppService now strips the memory capability from the served agents defaults when no memory block is configured; update the spec's expected capability lists to defaultAgentCapabilitiesWithoutMemory for the no-memory-config cases.

* 🛡️ fix: Address Codex re-review on memory capability (round 2)

- Strip the memory capability from the FINAL served agents config, not just defaults; loadEndpoints reparses any endpoints.agents block, so memory was still exposed in that common shape (packages/data-schemas/src/app/service.ts) + regression test.
- Re-check the full memory gate (config, opt-out, MEMORIES.USE) inside handleTools before constructing set_memory/delete_memory, so an unsolicited tool call from a model/custom endpoint can't bypass the runtime gates (api/app/clients/tools/util/handleTools.js).
- Restore the persisted memory toggle for model-spec conversations via applyModelSpecEphemeralAgent (client/src/utils/endpoints.ts).
- Clear LAST_MEMORY_TOGGLE_ on logout and clear-all-chats so a stale memory preference can't leak across users on a shared browser (client/src/utils/localStorage.ts).

* 🧠 fix: Address Codex re-review on memory capability (round 3)

- Serialize set_memory writes and advance a running token total inside createMemoryTool, so parallel batched calls in one event-driven turn can't each pass the limit check against a stale total and collectively exceed memory.tokenLimit (packages/api/src/agents/memory.ts) + tests.
- Inject the keyed memory context (withKeys) instead of withoutKeys when the running agent has the inline memory capability, so delete_memory has a visible key to target (api/server/controllers/agents/client.js).

* 🔐 fix: Address Codex re-review on memory capability (round 4)

- Detect inline memory by tool NAME (set_memory/delete_memory) across an initialized agent's tools + toolDefinitions, since the 'memory' marker is expanded at init and the prior string check never matched; inject the keyed memory context for any primary OR sub-agent that carries the inline memory tools (api/server/controllers/agents/client.js).
- Enforce memory WRITE permissions in the inline tool gate: set_memory requires CREATE+UPDATE and delete_memory requires UPDATE (matching the REST memory routes), so a USE-only role can't mutate/delete memories via agent tool calls (api/app/clients/tools/util/handleTools.js).

* 🔒 fix: Address Codex re-review on memory capability (round 5)

- Gate inline memory registration (memoryAvailable) on the memory WRITE permissions (USE+CREATE+UPDATE), so a read-only-memory role no longer has set_memory/delete_memory shown to the model only for the runtime loader to refuse them (api/server/services/Endpoints/agents/initialize.js).
- Enforce the per-agent memory opt-in at execution: handleTools now refuses to construct set_memory/delete_memory unless the agent actually declared them (toolDefinitions/tools), blocking hallucinated/undeclared memory tool calls from mutating memory.
- Fail closed when getFormattedMemories errors with a configured tokenLimit, instead of writing as if storage were empty and bypassing the cap (api/app/clients/tools/util/handleTools.js).

* 🩹 fix: Address Codex re-review on memory capability (round 6)

- Fix a P1 regression from the prior round: the execution-context agent keeps the raw 'memory' capability marker (not the expanded set_memory/delete_memory names), so the opt-in check now matches the marker. This restores memory writes/deletes AND avoids hijacking an MCP tool that merely shares the set_memory/delete_memory name (api/app/clients/tools/util/handleTools.js).
- Count repeated set_memory writes to the same key as replacements, not additions, against tokenLimit — set_memory upserts, so a same-key rewrite swaps its prior token contribution instead of double-counting (packages/api/src/agents/memory.ts) + test.
- Gate the memory badge, tools dropdown, and agent-builder toggle on the full memory write permissions (USE+CREATE+UPDATE) via a shared useHasMemoryAccess hook, so a read-only-memory role no longer sees an enabled Memory control the backend would refuse to wire up.

* 🧷 fix: Address Codex re-review on memory capability (round 7)

- Recognize inline memory across both execution-context agent shapes: initializeAgent now sets a LibreChat-only memoryToolsRegistered flag on the InitializedAgent, and the opt-in/detection checks accept that flag OR the raw 'memory' marker. Fixes memory failing for processAddedConvo agents (which store the initialized config, marker already expanded) while staying MCP-name-collision-safe (api/app/clients/tools/util/handleTools.js, packages/api/src/agents/initialize.ts, api/server/controllers/agents/client.js).
- Scope keyed memory context to memory-enabled agents only: useMemory now returns both keyed and unkeyed contexts, and buildMessages injects the keyed one (memory keys + token metadata) only to agents that can call delete_memory, while the primary/post-turn path keeps the unkeyed values — so a primary without memory tools no longer sees memory keys it doesn't need.

* 🔏 fix: Address Codex re-review on memory capability (round 8)

- Enforce memory size limits on inline writes: createMemoryTool now rejects keys over 1000 chars and values over memory.charLimit, matching the REST memory routes, so an inline-memory agent can't persist blobs the memory UI/API would reject (packages/api/src/agents/memory.ts, api/app/clients/tools/util/handleTools.js) + test.
- Recheck the agents 'memory' endpoint capability at execution time, so a stale/hallucinated set_memory/delete_memory call can't mutate memory after an admin removes the capability while the agent document still carries the marker (api/app/clients/tools/util/handleTools.js).

* ♻️ refactor: Move inline-memory backend logic into packages/api + share memory load

Workspace boundary: the inline-memory gating/detection logic that had crept into /api now lives in packages/api/src/agents/memory.ts (TS), with /api kept as thin wrappers.

- Add agentHasInlineMemoryTools, isMemoryToolAllowed, and buildInlineMemoryTool to packages/api; handleTools.js now calls buildInlineMemoryTool instead of constructing/gating the tools inline, and client.js imports agentHasInlineMemoryTools instead of redefining it.
- Optimize repeated memory loads: getRequestMemories memoizes getFormattedMemories per request (WeakMap keyed by req), so the run's memory-context load and every memory-enabled agent's set_memory token-usage load share a single DB fetch instead of one per agent.

* 🧠 fix: Invalidate request memory cache after inline writes

Inline set_memory/delete_memory now invalidate the request-scoped
getFormattedMemories cache on a successful write, so a later tool round
in the same response is seeded with the post-write usage total instead
of the stale pre-write one (multi-round writes no longer collectively
exceed tokenLimit, and a set after a delete is not over-counted). The
within-round sharing across multiple memory-enabled agents is preserved.

* 🧠 fix: Persist memory capability on saved agents; honor registration flag

- Add Tools.memory to the v1 systemTools allowlist so filterAuthorizedTools
  no longer silently drops the memory marker when an agent with the Memory
  capability is created/updated/duplicated through the builder (previously
  the capability only worked for ephemeral chats, not persisted agents).
- agentHasInlineMemoryTools now honors an explicit memoryToolsRegistered
  boolean before falling back to the raw `memory` marker, so an initialized
  config whose registration was denied (memoryAvailable false) is not given
  keyed memory context just because the marker survives in tools.

* 🧩 fix: Bring memory tool to parity with other ephemeral tools

- Add `memory` to the model-spec schema/type and honor `modelSpec.memory`
  in both ephemeral paths (load.ts, added.ts) and the frontend spec
  application, so admins can pre-enable Memory from a model spec exactly
  like webSearch/fileSearch/executeCode.
- Add LAST_MEMORY_TOGGLE_ to the timestamped-storage cleanup list so stale
  per-conversation memory toggles are purged on startup like the others.
- Hide the agent-builder Memory toggle for users who disabled memory in
  personalization (memories === false), mirroring the chat badge's opt-out
  gate, so the setting isn't shown as inert/misleading.

* ✅ test: Cover memory in applyModelSpecEphemeralAgent spec defaults

Update the exact-object assertions to include the new `memory` field and
add positive coverage that `modelSpec.memory` maps to the ephemeral
agent's `memory` flag. Fixes the shard 2/4 failure from 672a03b05.
2026-06-24 17:14:13 -04:00
Danny Avila
49f4b659f6
🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart (#13814)
* 🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart

MCPServersRegistry was built once at boot from getAppConfig({ baseOnly:
true }), freezing allowedDomains/allowedAddresses to YAML. Admin-panel
mcpSettings overrides were ignored by both inspection (addServer/
reinspectServer/updateServer/lazyInitConfigServer) and runtime connection
enforcement (assertResolvedRuntimeConfigAllowed), so a domain allowed only
via the panel failed inspection and never connected.

Make the registry's effective allowlists mutable and refresh them from the
merged admin-panel config: seed at boot, and re-apply on every config
mutation via invalidateConfigCaches -> clearMcpConfigCache. Both inspection
and connection paths read the same getters, so both honor overrides without
a restart. Fail-safe: current allowlists are preserved when the merged read
fails.

* 🛡️ fix: Scope MCP allowlist refresh to global config, fail-safe on DB error

Address Codex P1 review findings on the allowlist-refresh path:

- Tenant-scoped config mutations no longer push one tenant's merged
  mcpSettings into the process-wide registry singleton (read by all MCP
  connection paths), which would leak allowlists across tenants. Only
  global (non-tenant) mutations refresh the registry; tenant mutations
  still evict the config-server cache.
- The refresh read now uses strictOverrides:true so a transient DB error
  throws instead of silently returning YAML base config — preserving the
  last-known allowlists rather than overwriting them with fallback values.
  Adds the strictOverrides option to getAppConfig (default off, no behavior
  change for existing callers).

* ♻️ refactor: Resolve MCP allowlists per-request (tenant-scoped) instead of a global singleton

Supersedes the prior global-mutation approach. MCP allowlists live in
mcpSettings, which is tenant/principal-scoped admin config, so a process-wide
singleton value is the wrong model — it caused cross-tenant bleed and stale
reads.

Instead, inject a resolver (from the app layer, where the merged config lives)
that the registry calls per inspection and per connection. It reads the ALS
tenant context via getAppConfig and accepts the acting user so user/role-scoped
overrides resolve; config-source inspection (no user) resolves at tenant scope.
Falls back to the YAML base allowlists when no resolver is set or the lookup
fails, so a transient error fails to the operator baseline rather than
disabling the allowlist.

Removes the now-unnecessary setAllowlists / boot-seed / invalidateConfigCaches
refresh / getAppConfig.strictOverrides machinery.

* 🔒 fix: Scope config-source cache by allowlist; resolve OAuth allowlists per-request

Address Codex review of the per-request resolver:

- Config-source cache key now folds in the resolved allowlists, not just the
  raw-config hash. Inspection results became allowlist-dependent, so without
  this a tenant whose allowlist rejects a URL could poison the shared key with
  an inspectionFailed stub for a tenant that allows it (and vice versa). The
  tenant-scoped allowlist is resolved once per ensureConfigServers pass and
  threaded through the cache key + inspection.
- The two remaining request-time OAuth allowlist reads now use the merged
  config instead of the YAML base getters: the fallback OAuth-initiate path
  (routes/mcp.js) via resolveAllowlists, and OAuth revocation
  (UserController.maybeUninstallOAuthMCP) via the request's already-merged
  appConfig.mcpSettings. Without this, an OAuth endpoint allowed only by an
  admin-panel override was rejected while inspection/connection allowed it.

* ✅ test: Update MCP OAuth registry/config mocks for per-request allowlists

CI fix for the Finding-12 change. The OAuth-initiate route now calls
registry.resolveAllowlists() and the revocation path reads the merged
appConfig.mcpSettings, so the affected specs' mocks were asserting the old
base-getter values:
- routes/__tests__/mcp.spec.js: add resolveAllowlists to the registry mock.
- UserController.mcpOAuth.spec.js: provide mcpSettings on the getAppConfig
  mock so revokeOAuthToken still receives the expected allowlists.

* 🧪 test: e2e proof that admin-panel MCP allowlist override takes effect

Adds a Playwright mock-harness spec for #13809. A URL-based MCP fixture
(e2e-http, streamable-http SDK server) boots inspectionFailed because its
origin is omitted from the YAML mcpSettings.allowedDomains; the spec adds that
origin via an admin config override (PUT /api/admin/config/user/:id) and
asserts the server reinitializes — exercising the real resolver path through
the backend + DB. Before the fix, reinspection used the frozen YAML allowlist
and the server stayed unreachable.

- e2e/setup/fake-mcp-http-server.js: streamable-HTTP MCP fixture (health GET /).
- e2e/playwright.config.mock.ts: boot the fixture as a second webServer.
- e2e/config/librechat.e2e.yaml: mcpSettings.allowedDomains (excludes 127.0.0.1)
  + the e2e-http server.
- e2e/specs/mock/mcp-allowlist-override.spec.ts: login → baseline reinit fails →
  apply override → reinit succeeds.
2026-06-17 20:14:53 -04:00
Danny Avila
db7011d567
📊 feat: Real-Time Context Window & Token Usage Tracking (#13670)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 📊 feat: Real-Time Context Window & Token Usage Tracking

* 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps

* 🩹 fix: Resolve Codex Findings for Context Usage Tracking

* 📊 feat: Granular Tool Token Breakdown with Deferred Splits

* 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors

* 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated)

* 🧪 test: Local Real-Provider Multi-Turn E2E Harness

* 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate

* 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events

* 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output

- carry the post-snapshot output estimate into the context snapshot at
  finalize so the gauge keeps the last response after live resets
- accumulate per-rate billable units and price the session cost at
  render, so usage events arriving before the token-config load still
  count once it resolves
- pass user-scoped token-config cache keys through loadConfigModels
  fetches and drop the controller's unscoped fallback to prevent serving
  another user's resolved config
- tag emitted usage events with a per-run seq so resume dedupe never
  drops a distinct call with an identical payload
- admit the static tokenConfig override in the custom endpoint schema so
  it survives zod parsing into req.config

* 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics

- classify cache tokens by provider (shared inputTokensIncludesCache from
  data-provider, consumed by both the backend billing path and the client)
  instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache
  is smaller than uncached input no longer under-bill input
- mirror resolveCompletionTokens on the client so Vertex-style hidden
  thinking tokens are reflected in the Output row and session cost
- prefer endpoint pricing over adapter-provider pricing so a custom
  endpoint can price a known model name without built-in rates shadowing it
- carry static cacheRead/cacheWrite overrides through the tokenConfig
  schema and buildTokenConfigMap

* 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness

- initializeCustom now uses a static endpoint tokenConfig as the agent's
  endpointTokenConfig (billing + balance checks), not just the advertised
  UI config — previously the gauge showed admin rates while the agent
  billed against built-in tables
- invalidate the token-config query alongside models on user-key add/
  revoke so context windows and pricing refresh without a reload
- include maxContextTokens in ChatForm's stabilized conversation memo so
  the gauge reflects a changed context-window setting immediately
- feed the live output estimate from the legacy content path (direct and
  assistants streams), setting from cumulative part text rather than
  accumulating deltas

* 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing

- fold usage events idempotently by (runId, seq) so resume backfill no
  longer resets the conversation totals — a mid-stream reconnect keeps the
  usage of prompts already completed earlier in the session
- tap replayed pending message/reasoning/content events so output streamed
  past the resume snapshot reaches the live estimate, not just the message
- resolve cost against the agent's backing endpoint (Agents conversations
  report endpoint `agents` / provider `openAI`, neither of which keys a
  custom endpoint's tokenConfig)
- getMultiplier/getCacheMultiplier fall back to the standard tables for
  models absent from a partial endpointTokenConfig, so a partial static
  override no longer bills non-listed models at defaultRate while the UI
  shows the correct pattern rate

* 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup

- live/completed gauge counts the repaired completion (normalized output),
  so under-reporting providers don't drop the response from used context
- translate static tokenConfig cacheWrite/cacheRead onto the write/read
  keys getCacheMultiplier reads, so cache tokens bill at the configured
  rate instead of the prompt-rate fallback
- clear the token index and usage atoms when leaving a conversation, so
  visited histories don't accumulate in memory for the tab's lifetime
- wait for startupConfig before mounting the gauge, so a deployment with
  contextUsage disabled never briefly mounts it or fires the token-config
  query on first load

* 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo

- extract the token-config resolution (override gathering + cache lookup +
  buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving
  the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule)
- getConvoKey prefers the user message's real conversationId once the
  `created` event stamps it, so a new chat's first-response live gauge and
  totals land under the id TokenUsage subscribes to instead of NEW_CONVO

* 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config

- DEL the Redis job hash before re-creating it so a reused streamId can't
  inherit a prior run's contextUsage/tokenUsage and backfill stale usage
- tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic
  streams) into the live estimate, not just the content path
- copy a deduped fetch's token config to every sibling endpoint sharing the
  baseURL/key/headers, so /token-config resolves each by its own name

* ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume)

createJob is an idempotent join — a second replica calls it for the same
streamId to share an in-flight stream's state. DELeting the hash wiped the
prior replica's persisted created/usage state, so a joining replica missed
the created event (GenerationJobManager cross-replica integration test).
Reverts the F1 change from 2bfce0c34b; the stale-usage concern doesn't
arise in practice (streamId is unique per generation).

* 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage

- wrap the ModelEndHandler usage emit in try/catch so a failed telemetry
  delivery (closed SSE / Redis publish error) can't abort the handler
  before thought-signature capture, which would break resumed tool calls
- tag hidden sequential-agent usage as 'sequential' (non-primary) so the
  client folds it into session cost/totals but not the live context gauge,
  instead of letting an undefined usage_type inflate the visible gauge

* 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup

- useTokenConfigQuery refetches on mount when stale, so a user-key change
  that invalidates tokenConfig while the gauge is unmounted takes effect on
  return instead of serving the prior key's resolved config
- normalize a Vertex-backed agent's provider (vertexai) to the google
  token-config key, so Gemini context windows and rates resolve instead of
  showing unknown context / $0 cost

* ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge)

Move usage-cost pricing to the single source of truth. The backend prices
each model call with the same billing functions (premium tiers via
getMultiplier(inputTokenCount), cache rates) and emits the USD cost on
on_token_usage when interface.contextCost is enabled; the client sums
emitted costs instead of re-deriving from base token-config rates.

- computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend
  so the emitted cost matches what is billed (incl. premium thresholds)
- getDefaultHandlers gains a usageCost pricing context; initialize.js wires
  db.getMultiplier/getCacheMultiplier gated on contextCost (agents path)
- client UsageTotals carries a summed costUSD; retire the client-side rate
  lookups (costFromUnits/calcUsageCost) that drifted from backend pricing
  and produced the provider-keying / cache-key / Vertex / premium findings
- keep normalizeUsageUnits for the displayed token counts; token-config is
  still used for the context-window meter

Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1
above their input thresholds).

* 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast

- re-anchor the context snapshot from the user message to the response
  message at finalize. Regenerating a response branches off a shared user
  message, so anchoring on it made the snapshot read as "active" on both
  branches — switching to the sibling branch showed the wrong (other
  branch's) context. The response message is branch-unique, so sibling
  branches now correctly fall back to their own per-branch totals.
- raise the gauge ring's track/fill contrast (muted track, prominent fill)
  so the used portion reads clearly as a fill-level indicator

* 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors

- tag hidden sequential-agent usage `usage_type: 'sequential'` on the
  COLLECTED usage (not just the emit), and treat it as non-primary in
  recordCollectedUsage (billed, excluded from the reported output total) so
  hidden intermediate output stops inflating the parent's tokenCount/pruning
- emit on_token_usage from the subagent usage sink (tagged `subagent`, with
  authoritative cost when contextCost is on) so the gauge's session
  cost/totals include billed subagent usage; it stays out of the live meter
- call resetLive on the resumable 404 and max-retry terminal branches so the
  gauge doesn't keep counting stale in-flight tokens after the stream ends

* 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle

- raise the popup breakdown's context progressbar contrast (muted
  surface-tertiary track, prominent text-primary fill) — that's the bar the
  contrast feedback was about
- revert the gauge ring restyle (kept its original border-heavy track /
  text-secondary fill); the ring wasn't the element in question

* 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo

- a null-anchor context snapshot was treated as active on every branch,
  leaking one generation's granular breakdown onto sibling branches. Require
  a non-null (response-message) anchor on the viewed branch instead, so
  siblings without a matching snapshot fall back to their own totals.
- revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds
  from shallow copies) so the memo was behaviorally identical, but it was the
  feature's only change to core branch-navigation selectors — removing it
  matches upstream and rules it out of branch-navigation debugging.

* 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits

Custom-endpoint agents resolve an endpointTokenConfig during agent init but
it never reached the AgentClient, so spending, emitted cost, and runtime
max-token resolution all fell back to default rates for those agents.

- Surface options.endpointTokenConfig on the returned InitializedAgent.
- Pass it to the AgentClient (this.options.endpointTokenConfig) so the
  spending path bills at configured rates.
- Thread it through usageCost to computeUsageCostUSD so emitted per-event
  cost matches billing.
- getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map
  for models absent from a partial override (matches buildTokenConfigMap);
  consolidates the duplicated fallback in pricing.ts.

* 🪙 fix: Preserve Granular Breakdown Across Branch Switches

The granular context breakdown lives only in the live on_context_usage
snapshot — a single per-conversation slot, anchored to the latest response
and overwritten by each generation. Switching to a branch generated earlier
this session lost its tool/skill/system rows and fell back to coarse totals.

Retain each generation's finalized snapshot in a per-conversation map keyed
by its branch-unique response id (snapshotsByAnchorFamily). When the live
snapshot is off the viewed branch, walk the branch tail for its deepest
stored anchor and render that breakdown. Bounded by generation count and
cleared on conversation switch; the live/just-generated path is unchanged.

* 🪙 fix: Harden Resume Seeding and Subagent Usage Emission

- useResumableSSE: skip the trailing-output live seed when the resume
  carries a context snapshot; the snapshot's messageTokens already counts
  produced output, so seeding it again inflated usage until the next reset.
- AgentClient subagent emitter: await GenerationJobManager.emitChunk like
  every other caller (it persists before publishing), so a floating promise
  can't race job cleanup and a Redis/publish failure is caught by the
  emitter's try/catch instead of surfacing as an unhandled rejection.

* 🧪 test: Playwright Coverage for Context Breakdown Granularity

Add a test-only data-testid distinguishing the granular snapshot breakdown
(context-breakdown) from the coarse message-history estimate
(context-estimate), then assert granularity in the mock e2e harness:

- renders the granular breakdown from the live on_context_usage snapshot
  (guards that the snapshot event actually reaches the popover, not just the
  usage totals).
- preserves the granular breakdown after switching branches — regenerate to
  overwrite the single live snapshot, switch back, and confirm the rows
  survive via the per-anchor snapshot history map.

Branch regenerate/sibling selectors mirror the existing chat.spec branch test.
All three usage specs pass against the mock pipeline.

* 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush

Codex round on the prior commit:

- countTrailingOutputChars now counts only output at the very END of the
  aggregated content (0 when the model paused at a tool call), and the resume
  path always seeds it. The earlier skip-trailing-tool-parts behavior plus the
  skip-seed-when-snapshot gate together over- or under-counted in-flight
  output on resume; one rule fixes both — pre-invoke snapshot budget is never
  double-counted, and genuine in-flight output is no longer dropped.
- useTokenUsage re-indexes from the messages cache on tail change while
  submitting. The cache subscriber is muted during streaming, so without a
  context snapshot (non-agent streams) sumBranch missed the created tail and
  dropped history + prompt until finalize. Bounded — tailId only shifts on
  created/finalize/branch-switch.
- AgentClient tracks subagent usage emit promises and flushes them in
  chatCompletion's finally. The sink fires the emitter without awaiting, and
  resume reads the usage emitChunk persists (HSET), so cleanup must not race
  it or resumed clients miss billed subagent usage.
2026-06-13 19:38:28 -04:00
Michael Harvey
05eb986097
💬 feat: Conversation Starters for Model Specs (#13710)
* 💬 feat: Conversation Starters for Model Specs

Adds an optional conversation_starters field to model specs in
librechat.yaml. When the active conversation uses a spec that defines
starters (and no agent/assistant starters apply), the chat landing
renders clickable starter prompts between the landing content and the
chat input; clicking one submits it as the first message.

- data-provider: add conversation_starters to TModelSpec and
  tModelSpecSchema so the field survives strict config parsing
- client: ConversationStarters falls back to the active spec's
  starters via getModelSpec; entity (agent/assistant) starters
  take precedence; starter cards are centered, size to content,
  wrap at word boundaries, stagger their fade-in, and gain a
  focus-visible ring
- sanitizeModelSpecs passes the field through (denylist); covered
  by a new unit test
- e2e: mock spec + tests for rendering, absence, click-to-submit,
  and the MAX_CONVO_STARTERS cap

Closes #3619

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: Sort ChatView imports

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-13 11:38:49 -04:00
Danny Avila
b39ec16ff0
🔌 fix: Preserve Ephemeral MCP Selections Across Model Switches (#13697)
The no-spec branch of `useApplyModelSpecEffects` (added in #11796) reset
`ephemeralAgentByConvoId` to null on every `newConversation` call when
model specs are configured. On in-place model/endpoint switches (modular
chat, same conversation or new-chat draft), BadgeRowContext never refills
from localStorage — its init effect only re-runs when the storage suffix
or spec changes — so the MCP selection (and tool toggles) were silently
dropped from subsequent request payloads while the MCP badge kept
displaying them.

Reset now only happens on context transitions (leaving a spec, or moving
to a different conversation key), where a BadgeRowContext refill is
guaranteed; in-place non-spec switches preserve the ephemeral agent.

- Gate the no-spec reset on `prevSpecName` / `prevConvoId`, passed from
  `newConversation` via a snapshot read of the pre-switch conversation
- Add jest coverage for all five branches of the no-spec path
- Add e2e spec asserting `ephemeralAgent.mcp` stays in the chat payload
  after a new-chat model switch and after regenerate on a switched
  conversation (verified failing before the fix, passing after)
- Add non-spec "Mock Provider D" endpoint to the e2e config so tests can
  switch between two real ephemeral endpoints; widen `MockEndpoint` type
2026-06-11 18:13:41 -04:00
Danny Avila
470be2395f
✨ feat: Surface Model Spec Branding on Landing and Selector (#13662)
Adds an opt-in showOnLanding flag to model specs. When set, the chat
landing shows the spec's label and description in place of the
time-of-day greeting; specs without the flag are unaffected, so existing
deployments see no behavior change. HTML-valued descriptions (inline
icons + markup) render sanitized via the shared config-HTML sanitizer
with a new media tag/attribute allowlist, both on the landing and in
model selector items. Excludes e2e specs from the typed client lint
block so staged e2e files no longer fail pre-commit with 'file not
found in project'.
2026-06-10 21:02:22 -04:00
Teresa Blanco
9628930958
✅ ci: Add mock e2e coverage for agents, prompts, MCP, and chat flows (#13589)
* ✅ Add mock e2e coverage for agents, prompts, MCP, and chat flows

* 🎯 fix: Change enforce modelSpecs to false

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-10 09:06:52 -04:00
Danny Avila
da6b74e8eb
🪶 fix: Prevent Soft Default Model Spec from Overriding User Selections (#13642)
* 🎯 fix: Soft Default Model Spec Overriding User Selections

* 🎯 fix: Detect Agents-Only Allow-List Before Endpoints Config Loads

* 🎯 fix: Preserve Explicit Soft Default Selections over Older History

* 🎯 fix: Limit Soft Default Residue to Spec-Named State, Disable E2E Enforcement
2026-06-10 08:52:28 -04:00
Danny Avila
2a956f143d
🪞 fix: Preserve Model Spec Icons Across Stream Resume and Abort (#13603)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
2026-06-08 17:14:21 -04:00
Danny Avila
4b699fb60f
📌 fix: Preserve Project Scope Through Enforced Model Specs (#13586) 2026-06-08 08:41:27 -04:00
Danny Avila
6357ea10c1
🧭 feat: Scope Model Spec Skills (#13522)
* feat: scope model spec skills

* style: format skill catalog limit

* fix: serialize model spec skill resolution

* test: satisfy model spec load config typing

* fix: apply model spec skills to added conversations

* fix: support alwaysApply frontmatter alias

* fix: address model spec skills review
2026-06-05 10:22:02 -04:00
Danny Avila
a1bfa3b298
🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model (#13508)
* 🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model

Replace the standalone HTTP mock LLM server with an in-process fake model
injected into the real createRun -> Run.create pipeline via
run.Graph.overrideTestModel, so the mock suite exercises the agents
integration end-to-end without a live provider or a separate server.

- Bump @librechat/agents to 3.2.2 for the FakeChatModel/createFakeStreamingLLM exports
- Add an env-gated applyTestRunHook seam in packages/api createRun (no /api changes)
- Add e2e/setup/fake-model.js to drive default replies + the skill-authoring tool-call flow
- Drop the mock-llm webServer from playwright.config.mock.ts and set LIBRECHAT_TEST_RUN_HOOK

* 🧹 test: Retire Standalone Mock LLM Server From E2E Recorder

Migrate the `--profile=mock` recorder onto the same in-process fake model
as the Playwright mock suite, then delete the now-unused HTTP mock server
so the fake-LLM logic lives in a single place.

- Point record.js mock profile at the fake model via LIBRECHAT_TEST_RUN_HOOK
- Remove the mock-llm-server spawn/wait and MOCK_LLM_PORT plumbing from record.js
- Delete e2e/setup/mock-llm-server.js (e2e/setup/fake-model.js is now the only source)
- Update e2e/README.md to describe the in-process fake LLM

* 🏷️ ci: Rename Playwright Mock E2E Check to Playwright E2E Tests
2026-06-04 08:33:28 -04:00
Teresa Blanco
b45e4aeae5
🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472)
* 🧪 feat: add e2e playwright tests

* 🧪 feat: Add Playwright Recording Harness

* test: fix mock playwright config

* test: harden mock e2e environment

* test: preserve mock dotenv secrets

* test: harden mock isolation setup

* ci: cache mock e2e builds

* test: harden e2e cache and recorder checks

* test: preserve data-provider exports in oauth route test

* test: isolate mock auth logout state

* test: allow isolated logout smoke setup

* test: prepare logout smoke auth via api

* test: isolate oauth route module mock

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-02 16:36:39 -04:00