Commit graph

222 commits

Author SHA1 Message Date
Marco Beretta
e95e696d70
🧾 fix: Count the Tool Results a Tool-Limit Stop Retains (#15893)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 🧾 fix: Count the Tool Results a Tool-Limit Stop Retains

Context snapshots reach the client only through the SDK's pre-invoke
`ON_CONTEXT_USAGE`, so the results of the tools a call requests are never in that
call's snapshot — the next call's snapshot carries them as kept-message context.
A run that stops at the tool-call limit makes no next call, so the tool result it
retains lives in the response and in no snapshot: the gauge reported
`(budget − remaining) + completedOutputTokens` and left the retained result out
of used tokens and out of the tool-call share until the following turn.

The save path now counts those results with the run's own tokenizer and persists
them as `retainedToolTokens`, a second post-snapshot delta alongside
`completedOutputTokens` rather than a number folded into the provider-reconciled
`messageTokens`. `resolveRetainedToolTokens` owns the rule that only a tool-limit
stop retains anything, and the snapshot handler records where its content ended
so the count starts at the right boundary.

Counting had to avoid `Tokenizer.getTokenCount`, whose fallbacks would have put a
guess inside exact accounting: above 4 KiB it returns byte length, several times
the real count on ordinary text, and it estimates from character length while an
encoding loads. `countExactTokens` tokenizes in bounded slices cut on code-point
boundaries and returns nothing at all when the encoding is cold, so an
uncountable result withdraws the figure instead of inflating it.

The client adds the field to used tokens, subtracts it from the runway headroom
and widens the tool-call share, in the live snapshot after finalization and in
the persisted blob after a reload.

* 🧹 style: Wrap the Retained-Counter Assertion as Prettier Requires

* 🧮 fix: Address the Review of the Retained-Tool Count

Three findings from the first round, each a real defect in how the figure was
produced rather than a style point.

The boundary was a content index recorded mid-run, but completion reshapes the
array — skill cards are unshifted onto the front and `hide_sequential_outputs`
replaces it with a filtered one — so a saved index no longer means the same
position. The snapshot now records the tool-call ids it already accounts for, and
the save path counts the results of the calls missing from that set: ids survive
every reshape, and a filtered-away call is correctly left out.

Counting in 4 KiB slices was not exact either: a BPE merge spanning a seam is
charged twice, measured at ~1 token per slice, and the field exists precisely to
be an exact addend. `countExactTokens` now tokenizes the whole input — ~60 ms/MB,
paid once at the end of a stopped turn — and refuses content past 8 MiB rather
than estimating it.

The counter takes its exact-count function instead of reaching for the tokenizer
singleton, so `resolveRetainedToolTokens` owns the default (the run's own
encoding) and a caller or test can supply another. That also removes the mock of
global state from the specs.

`compactionReclaim` now includes the retained result in the total it subtracts the
kept exchange from. `latestExchangeTokens` already counts that result on the
other side, so leaving it out subtracted content the total never carried and
understated the savings — to zero on a large final result.

* 🧯 fix: Bound One Turn's Retained-Result Tokenization

The tokenizer refuses a single result past 8 MiB, but a final call that requested
several tools in parallel would pay that bound once per result. The counter now
holds a budget for the whole turn and withdraws its figure past it, so the save
path cannot be made to tokenize an unbounded pile of output.

* 🎚️ feat: Configure the Retained-Result Tokenization Budget

The exact count the gauge adds costs ~60 ms/MB of retained tool output, and the
ceiling on that work was hard-coded in two places. It is now one lever:
`endpoints.agents.maxRetainedToolCountChars`, defaulting to the 8 MiB that
reproduces today's behavior, shared by the schema and the save path through
`DEFAULT_MAX_RETAINED_TOOL_COUNT_CHARS`. Deployments whose tools legitimately
return more can raise it; slower hardware can lower it, or set `0` to withhold
the figure entirely.

`Tokenizer.countExactTokens` no longer carries a bound of its own — the caller
owns the budget — and `resolveRetainedToolTokens` passes the configured value to
the counter, which spends it across all of a final call's parallel results.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-13 22:20:25 -04:00
Marco Beretta
91e00bbe6e
🧷 fix: Mark a Failed Compaction So Its Rerun Controls Stay Withheld (#15883)
* 🧷 fix: Mark a Failed Compaction So Its Rerun Controls Stay Withheld

A manual compaction that failed with an error part carried no marker: the
server stamped `initiatedBy: 'user'` only on a summary that produced text and
returned early once an error part existed. On a branch ending in a user
message, the resulting turn had no marker and a user parent, so the hover row
kept Edit, Regenerate and Continue — and using one answered that user message
instead of retrying the compaction.

`markCompactionOutcome` now marks whichever part carries the outcome, summary
or error, and `isUserInitiatedCompaction` reads both. The marking moved from
`api/server/controllers/agents/client.js` into `packages/api`, where the
behavior belongs; the controller keeps the call.

* 🏷️ refactor: Type the compaction spec helpers with SummaryContentPart

* 🧯 fix: Record a Marked Failure When a Compaction Produces Nothing

A compaction run that produced neither a summary nor an error part threw
COMPACTION_FAILED out of `sendCompletion`. The turn was then saved by
`saveErrorTurn`, which writes only top-level `text`/`error` and no content, so
the row carried no compaction marker: on a user leaf the hover row kept
Regenerate, and using it answered that user message.

`markCompactionOutcome` now records the typed failure as a marked error content
part instead, so the turn streams and persists like every other failed
compaction. A cancelled run still fails as a typed error, leaving the abort
path to own the turn. `saveErrorTurn` marks the compaction rows it still writes
for failures that are rethrown rather than recorded as content.

Verified end to end by two new mock scenarios that run Compact context against
a summarizer returning nothing, live and after a reload.

* 🧪 test: Run the Empty-Compaction Scenarios Against a Real Compaction

* 🧱 refactor: Resolve Failed-Turn Compaction Content in packages/api

`saveErrorTurn` decided in CJS whether a failed row carries compaction
identity. The decision moves to `resolveFailedTurnContent` in
`packages/api`, which returns the content fields a failed turn is persisted
with — the marked failure for a compaction, nothing for any other turn — and
`request.js` spreads the result.

* 🧹 fix: Keep an Unusable Summary Out of a Failed Compaction's Turn

History loading accepts any nonempty summary as the conversation's checkpoint
(`BaseClient.findSummaryContentBlock` ignores `failed`), so a compaction that
streamed partial text before failing would have persisted that truncated text
beside its recorded failure and replaced the history it failed to summarize.
The fallback now drops the unusable summary, leaving the typed failure as the
turn's whole outcome.

The empty-summarizer scenarios switch the shared fixture summarizer to blank
output, so the reset moves to `afterEach` and runs even when the helper throws
before its caller's `finally`.
2026-09-13 21:53:29 -04:00
Danny Avila
f6335cb9d7
💰 fix: Bill Stopped Agent Turns From Provider Usage (#15889)
* 💰 fix: Record Token Usage for Stopped Agent Turns

The agents Stop route only signals the abort; the run's cleanup skipped recordCollectedUsage on an aborted signal, trusting an abort middleware that is mounted only on the assistants abort routes. Stopped and error-aborted agent turns therefore wrote no transactions and never debited the balance.

The run now records its collected usage on every exit, labelling a stopped turn's transactions with context 'abort'. The owner of the run charges, so a cross-replica Stop is still billed exactly once.

* 💰 fix: Make the Run the Single Charger for a Stopped Turn

The legacy assistants-mounted abort route still billed a non-assistants job — provider usage via spendCollectedUsage, or a text-count fallback — on top of what the run records on exit, and its array clear ran only after its own awaited write, so the two could race even on one replica. The route now only stops and persists; the run records its usage once, labelled through resolveRunUsageContext in packages/api.

* 💰 fix: Keep the Text-Count Fallback From Re-Billing Recorded Usage

BaseClient falls back to text-count billing whenever the recorded usage has no positive output count, so a stopped call that reported input tokens and no output was debited by recordCollectedUsage and then charged its estimated prompt again. AgentClient.recordTokenUsage now returns early once provider usage was recorded (hasRecordedProviderUsage in packages/api); an all-zero report still falls through to the estimate.

* 💰 refactor: Lift Fallback Token Billing Into packages/api

recordFallbackTokenUsage owns the whole fallback operation — the recorded-usage guard, the estimate spend, the reasoning row and the error boundary — taking spendTokens as an injected dependency. AgentClient.recordTokenUsage only builds the transaction metadata from the client and calls it.

* 💰 fix: Label Fallback Billing From the Run's Stop State

BaseClient invokes the estimate fallback without a context, so the sole row for an initial turn stopped before any provider usage was labelled 'message'. recordTokenUsage now defaults the context from the run's own abort signal.

* 💰 refactor: Default the Fallback Billing Label in packages/api

recordFallbackTokenUsage takes the run's raw abort state and derives the transaction label itself; AgentClient.recordTokenUsage passes the signal through and no longer chooses the context.

* 💰 fix: Guard Fallback Billing on the Collected Primary Entries

The stream aggregate recordCollectedUsage returns takes its input from the first primary call only, so a later cancelled call that reported input alone was billed yet invisible to the fallback guard, which then charged the estimated prompt again. recordFallbackTokenUsage now also consults the collected entries themselves (hasRecordedPrimaryUsage), ignoring summarization, subagent and sequential usage; AgentClient passes its collected entries through.
2026-09-13 21:51:05 -04:00
Danny Avila
74fa60cee1
🔎 fix: Enforce WEB_SEARCH Role Permission on Agents (#15875)
* 🔎 fix: Enforce WEB_SEARCH Role Permission on Agents

Squash of #15822 onto dev, preserving authorship:

- 🔎 fix: Enforce WEB_SEARCH Role Permission on Agents
- 🧪 test: Cover the Runtime Tool Loader's Web Search Gate
- 🔒 fix: Gate Web Search on the OpenAI-Compatible Embedder Surface
- 🩹 fix: Authorize Web Search by the Execution Context's User
- 📝 docs: Note the Grant Memo's One-Subject-Per-Request Assumption
- 🧹 fix: Deny Web Search Before an Endpoint Default Can Grant It
- 🙈 fix: Withhold the Web Search Model Parameter From a Denied Role
- 🧱 fix: Strip Provider-Native Search an Endpoint Re-Enables After Denial
- 🔌 fix: Close the OpenRouter Plugin Path and Stop Pruning Hidden Parameters

* 🧭 fix: Gate Provider-Native Web Search on the Built Config

The web search role gate ran before the provider builder and only when the
agent's stored `web_search` was not `false`, so an endpoint's `addParams`
re-enabled native search for a denied role, and every OpenAI-compatible and
Responses request read the role whether or not search was in play.

The gate now reads the builder's output: when it carries a native search tool
or OpenRouter's `web` plugin, the `WEB_SEARCH` grant is resolved and the search
stripped on denial. The routes that reach `initializeAgent` with `runtime` and
no `req` pass a lazy `resolveWebSearchGrant` that joins their request-memoized
grants, forwarded through agent discovery to handoffs and subagents. The
embedder surface wires it whenever `getRoleByName` is supplied, independent of
`appConfig`.

---------

Co-authored-by: TomasPalsson <tomas2212042710@gmail.com>
2026-09-12 23:00:10 -04:00
Danny Avila
1163045cef
🖋️ feat: Seal Code Environment Choices Per Conversation (#15868)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: lock code environments per conversation

* style: Sort code environment imports

* fix: Harden code environment decisions

* fix: Preserve code environment rollout compatibility

* fix: Harden code environment boundaries

* refactor: Centralize code environment policy

* test: Restore partial disconnect policy mock

* fix: Close code environment edge paths

* fix: Preserve implicit stateful code routes

* fix: Stabilize implicit code routing

* test: Align remote workspace contract

* chore: Sort code route import
2026-09-12 20:45:00 -04:00
Dustin Healy
19eb24921e
🔠 feat: Render Safe Upstream Model Errors (#15837)
* ✨ feat: Surface Upstream Model Errors

* 🧪 test: Cover Upstream Error Rendering

* 🛡️ fix: Harden Upstream Error Fallbacks

* 🛡️ fix: Defer Upstream Error Fallbacks
2026-09-12 20:23:54 -04:00
Dustin Healy
7a83b8f5d2
📟 feat: Record Safe Upstream Model Failures (#15800)
* ✨ feat: Add Upstream Model Error Observability

* ♻️ refactor: Move Terminal Error Policy to TypeScript

* 🛡️ fix: Preserve Agent Cancellation Semantics
2026-09-12 20:23:17 -04:00
Joseph Licata
f25242df98
🦘 feat: Share Run-Scoped Files With Subagents (#15848)
* feat: add run-scoped file sharing for subagents

Delegate current-message files through authorized execution catalogs and publish selected immutable code-output versions through the existing file pipeline.

Requires the companion agents SDK context adapter before the dependency can be released and pinned.

* fix: preserve child tool artifacts and verify file-sharing lifecycle

Keep citations and other non-sandbox artifacts on their existing delivery path while retaining private code outputs and inspected images until publication.

Cover native and text delivery, nested and sibling authorization, concurrent runs, cancellation cleanup, and checkpoint recovery through the browser harness.

* test: align file-sharing fixtures and integration cleanup

* test: repair LibreOffice conversion fixtures

* test: verify deployment skill bytes across checkout formats

* fix: align subagent file budgets and refresh boundaries

* chore: update agents SDK to 3.8.6

* refactor: move code output processing into API workspace

* fix: enforce shared file budgets for parents and teams

* fix: declare the shared file encoder interface
2026-09-12 20:14:22 -04:00
Danny Avila
8f13845da5
📿 fix: Inspect Stored File Locators Per Message (#15841)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: inspect stored file locators per message

* fix: declare traversal diagnostics for isolated declaration builds

* refactor: inject locator traversal reporting from application callers

* fix: propagate traversal diagnostics across protected entry points

* fix: complete middleware reporters and hydration message counts
2026-09-11 16:47:48 -04:00
Dustin Healy
03a42d6c7e
🎒 fix: Carry Subagent Identity Past Streaming (#15795)
* fix: resolve saved subagent display identity

* fix: persist explicit subagent execution identity
2026-09-09 23:07:10 -04:00
Danny Avila
a8087ce454
🏕️ feat: Select Attached Code Workspaces (#15768)
* feat: Select Attached Code Workspaces

* fix: Preserve Workspace Boundaries Across Agent Graphs

* fix: Validate Lazy Agent Workspace Bindings Before Persistence

* fix: Include Handoff Environments in Workspace Selection

* fix: Close Workspace Admission and Remote Ingress Gaps

* fix: Preserve Workspace Selection In Chat Runtime Envelopes

* fix: Authorize Deployment Workspace Status And Match Readiness

* fix: Preserve Ephemeral Chats And Pin Runtime Worker Identity

* test: Complete Deployment Pairing Fixtures
2026-09-09 02:14:50 -04:00
Danny Avila
ba12eaa418
⛔ fix: Bind Tool Approvals to Generation Scope (#15724)
* fix: bind tool approvals to generation scope

* fix: gate generation scope on replay lifecycle support

* chore: add npmrc for npm v12 support, fix xlsx tarball dl

* chore: bump agents sdk to v3.8.3

* fix: require scoped approval replay support
2026-09-08 08:02:48 -04:00
Dustin Healy
cdc3339a78
🪬 fix: Fence Agent Checkpoint Ownership and Cleanup Recovery (#15680)
* 🔒 fix: Isolate Checkpoint Cleanup by Generation Ownership

* 🔒 fix: Harden Checkpoint Cleanup Ownership

* 🔒 fix: Retain Checkpoint Cleanup Evidence

* 🔒 fix: Persist checkpoint cleanup receipts

* 🔒 fix: Bound checkpoint cleanup receipt lifecycle

* fix: Bind checkpoint cleanup to durable ownership and deletion intent

* fix: Bind event actor checkpoint storage to authenticated owners

* fix: Resolve exact actor checkpoint ownership before lifecycle cleanup

* fix: Preserve actor storage compatibility with durable ownership and prune work

* fix: Defer checkpoint erasure until conversation deletion commits

* fix: Reclaim orphan actor ownership and batch deletion lookups

* fix: bind actor checkpoint ownership to payload rows

* fix: isolate actor readers and await account persistence

* fix: gate owned suspensions and drain complete account state

* fix: unify checkpoint deletion lifecycle and storage authority

* fix: Replay checkpoint deletion from durable owner state

* fix: retain checkpoint cleanup identity before topology loss

* fix: complete checkpoint catalog capture and retirement lifecycle
2026-09-07 22:04:59 -04:00
Danny Avila
6f05678bfc
🧐 feat: Restore Interactive Tool Approval Review (#15717)
* 🧐 feat: Review Pending Tool Actions in the Composer (#15714)

* feat: add interactive approval review to the composer

* fix: Harden interactive approval boundaries

* fix: Bound every approval display label

* test: exercise composer approval review in BYOM acceptance

* fix: reveal all directional approval controls

* fix: canonicalize code approval targets across replicas

* style: sort resumable approval test imports

* test: Exercise Coexisting Approval Surfaces

* Revert "🧐 feat: Review Pending Tool Actions in the Composer (#15714)"

This reverts commit 5f79ded9553405b697e837e1976a70accbfc3fc3.

* Reapply "🧐 feat: Review Pending Tool Actions in the Composer (#15714)"

This reverts commit 52c7d9c7453da0168338bcf79494927f561a92a0.

* chore: bump agents sdk to v3.8.2
2026-09-07 22:02:56 -04:00
Danny Avila
3e2a6f333b
🪗 feat: Compact Context in a Summarize-Only Turn (#15690)
* 🗜️ feat: Manual Context Compaction as a Summarize-Only Turn

Adds a user-triggered "Compact context" action that summarizes the active
branch on demand. The compaction is an ordinary agent turn: the client
submits it through the normal chat pipeline with `compact: true`, the
controller hangs it off the branch's leaf instead of creating a user
message, and the agent client runs the graph summarize-only (the SDK's new
`summarizeOnly` input), so the summary streams into the response
placeholder under the leaf exactly like the automatic detour's and is
persisted as the boundary every later turn starts from.

Nothing new is needed for job registration, billing, abort, persistence,
or the usage snapshot: the run emits the same summary step and post-summary
context snapshot the automatic path already emits, and the response message
carries the summary part plus the `summaryUsedTokens` baseline the gauge
reads. The controller validates the request (existing conversation, a real
leaf, summarization enabled, no edit/regenerate/continue combination) and
answers 409 when the leaf is already a bare summary.

Client-side the submission borrows the regenerate shape — no new user
bubble, the placeholder parents onto the leaf — while the wire payload
drops `isRegenerate` so the server never treats it as a regenerated user
turn. The action sits in the context-usage popover, gated by the same
`summarization.enabled` switch as the automatic detour.

* 🗜️ fix: Localize a Compaction That Cannot Run

A compaction the graph could not attempt (summarization disabled, the
instructions already exceeding the budget, or an explicit recency window
covering the whole conversation) rejects with the SDK's
`ManualSummarizationSkippedError`, which rendered as the generic admin
error. It now maps to a typed `compaction_skipped` payload with the reason,
and a run that ends with neither a summary nor a recorded error fails as
`compaction_failed`; the Error component localizes both. A run that already
recorded why it stopped keeps that error part instead of failing twice.

* 🗜️ fix: Anchor a Compaction on the Leaf Everywhere It Is Read

Review round 1.

- The client sent the leaf's parent as `parentMessageId`, which the
  controller treats as the anchor, so the summary would have skipped the
  latest reply and landed as its sibling. The leaf's own id is now both the
  placeholder's parent and the server-side anchor.
- A compaction has no user message of its own, and every consumer of the
  job's user-message slot now knows it: the error path parents its row on
  the loaded anchor and never upserts the leaf, and the job records the
  regenerate shape so a reconnecting client rebuilds the turn under the
  leaf instead of recreating it as a user message.
- The controller no longer reads the anchor before the branch is loaded;
  `BaseClient` validates it against the history it loads anyway and
  answers 404 for a missing leaf or a typed `nothing_to_summarize` for a
  finished compaction. That predicate is shared through the data provider
  and ignores a summary still streaming or one that failed, so an
  interrupted compaction can be retried.
- A compaction never consumes files staged in the composer, the action is
  shown only when the server advertises `compactionEnabled`, and the
  in-flight marker is feature-local Jotai state rather than a read of the
  Recoil submission store.

* 🗜️ fix: Keep a Compaction Turn to Its Summary

Review round 2, plus the class it belongs to. A compaction submits no user
turn, so nothing that keys off one may run: the memory pass no longer runs
over the history (it would have spent a model call and could repeat memory
writes), and skill primes are no longer injected into the transcript the
run is about to summarize.

The client's in-flight marker is also reconciled on mount: a compaction
that finished while the view was away left the marker set, and the next
ordinary turn in that conversation would have shown as compacting.

* 🧹 chore: Sort Imports in the Compaction Hook Spec

* 🔧 chore: Update @librechat/agents dependency to version 3.8.1 in package-lock.json and package.json files
2026-09-07 21:51:54 -04:00
Danny Avila
4b3a6eaf0e
🪧 feat: Surface Conversation Code Approval Modes (#15703)
* feat: Add conversation code approval modes

* refactor: Reuse shared code approval types

* fix: Scope approval selector to stateful agents

* fix: harden code approval state propagation

* fix: fail closed during approval discovery

* test: cover approval mode acceptance

* test: use visible approval mode label

* test: drive approval menu by keyboard

* test: expose accept edits in BYOM acceptance

* fix: read approval mode atomically on send

* fix: align code modes with endpoint policy

* fix: initialize code limiters at startup
2026-09-07 12:24:29 -04:00
Danny Avila
e370ce4b21
🎒 fix: Bound Agent Attachment Context (#15694)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
2026-09-07 08:16:34 -04:00
Danny Avila
a3fc3b9dbf
🚧 fix: Gate File Search and Run Code by Role (#15691)
* fix: enforce FILE_SEARCH role permission server-side

The FILE_SEARCH role permission is stored, served by GET /api/roles/:name and
settable through the admin API, but nothing ever checks it.
PermissionTypes.FILE_SEARCH occurs zero times under api/server; the three
occurrences in @librechat/api are all in the interface-to-role sync, which
writes the permission rather than checking it.

Measured on v0.8.8-rc1 and confirmed unchanged in rc2: a user whose role has
FILE_SEARCH.USE = false can still upload a document with
tool_resource=file_search and gets 200 with "embedded": true. The same user
calling execute_code correctly gets 403.

Two gates, both mirroring toolAccessPermType in ~/server/controllers/tools.js:

* Upload (api/server/routes/files/files.js): a tool-resource-to-permission map
  checked before any branching, so it also covers the assistants path. Returns
  403 with the same log line as the RUN_CODE gate. execute_code is included
  because it had the same hole: the tool CALL was gated, the upload was not.

* Tool loading (api/app/clients/tools/util/handleTools.js): checked outside the
  lazy loader so a denied user never gets the tool equipped, rather than one
  that fails when called. Fails closed if the permission check itself throws.
  All four required imports were already present in that file, next to an
  existing FILE_CITATIONS check.

Both changes live in /api (JavaScript) rather than /packages/api (TypeScript)
as CLAUDE.md prefers for new backend code: each mirrors an existing pattern in
exactly these files, and a second permission gate in another language and
package would be harder to keep in step with the first.

Tests: api/app/clients/tools/util/handleTools.fileSearchPermission.test.js
covers permit, deny, the log line, and the fail-closed path.

* fixup! fix: enforce FILE_SEARCH role permission server-side

* 🔐 fix: Gate `file_search` and `execute_code` on Role Permissions

The `FILE_SEARCH` gate added in the previous commit lives in
`handleTools.loadTools`, which `loadAgentTools` only reaches when
`definitionsOnly` is false. That default is false on the chat path
(`Endpoints/agents/initialize.js`) but true on the responses and
OpenAI-compatible controllers, so on those paths a denied user still got
the tool definition advertised to the model and their files primed by
`primeSearchFiles`, and the call failed later in `loadToolsForExecution` —
the "gets one that fails when called" outcome the gate set out to avoid.

Move the decision to the capability filters that both loaders already run,
mirroring how `canUseMCP` resolves the `MCP_SERVERS` permission ahead of the
synchronous filter. `AgentCapabilities` stays the instance-wide deployment
switch; the role grant is a second condition a tool has to clear. Because
`hasFileSearch`/`hasExecuteCode` derive from the filtered list, priming is
skipped for a denied user too.

`RUN_CODE` had the same hole: `toolAccessPermType` in
`~/server/controllers/tools.js` gates only `POST /tools/:toolId/call`, not
the agent run, so a role without `RUN_CODE.USE` could still execute code
through an agent. Both tools go through the same map.

Checks use `checkAccessWithRequestCache` and run only for tools whose
capability is already enabled, so a run costs at most one role read. A check
that throws denies the tool.

Also formats the test file added in the previous commit, which Prettier
rejects as-is.

* 🔐 fix: Close the Remaining `RUN_CODE` and `FILE_SEARCH` Bypasses

Codex review of the previous commit found four ways past the gates. All four
share a cause: the permission map was copied per call site, so every boundary
that was not copied into stayed open.

Moves the maps and the check into `packages/api/src/tools/rolePermissions.ts`,
per CLAUDE.md ("all new backend code must be TypeScript in `/packages/api`",
"keep `/api` changes to the absolute minimum"), and points the four `/api`
boundaries at it.

- `handleTools.loadTools` now gates every tool in `toolRolePermissions`, not
  just `file_search`. It is the shared boundary the Assistants required-action
  flow crosses via `processRequiredActions`, which never passes the agent
  capability filter — so a legacy Assistant with a raw function named
  `execute_code` ran arbitrary code for a role denied `RUN_CODE`.

- The check is now request-cached (`checkAccessWithRequestCache`). The previous
  `checkAccess` call omitted `req`, so the agent path resolved the same grant
  twice and issued a second serial `getRoleByName` during chat startup, against
  the rule in AGENTS.md.

- `codeExecutionEnabled` is gated by the role result at all three loaders, not
  only the `execute_code` entry in the filtered list. It drives tool
  classification and the programmatic bash tool, so an agent pairing
  `execute_code` with an MCP `code_execution` tool still advertised and
  instantiated `run_tools_with_bash` for a denied role.

- The upload map covers `EToolResources.code_interpreter`, which the Assistants
  builder posts instead of `execute_code`, and native `code_interpreter` /
  `file_search` tools are dropped when an assistant is created or updated.
  Those run inside the provider and reach neither loader, so configuration time
  is the only place they can be gated.

Renames `handleTools.fileSearchPermission.test.js` to
`handleTools.rolePermissions.test.js` — it now covers both tools.

* 🔐 fix: Gate the Image Upload Route and the v1 Assistant Writers

Second Codex round found the same shape again: boundaries the map had not been
wired into. Adds two shared helpers next to the maps in
`packages/api/src/tools/rolePermissions.ts` so a new upload handler or assistant
writer gets the check by calling one function.

- `/files/images` accepts `tool_resource` and routes agent uploads to
  `processAgentFileUpload` on its own. The Code Files UI sends images there, so
  every image was a way around the upload boundary for a role denied
  `RUN_CODE`. Both upload routes now call
  `checkToolResourceUploadPermission`.

- That helper passes `req`, so the role read joins the request cache. The
  previous `checkAccess` call omitted it, and an allowed Assistants upload goes
  on to `processFileUpload` → `addResourceFileId` → `updateAssistant`, whose own
  check does pass `req` — two serial lookups per upload, against AGENTS.md.

- `/assistants/v1` is still mounted and its `createAssistant` / `patchAssistant`
  write native tools straight to the provider. Both now filter through
  `resolveAssistantToolPermissions`, which v2 also uses in place of its local
  copy.

Run-time enforcement for native assistant tools is deliberately not here: it
means overriding `body.tools` at `createRun` in both chat controllers, which
changes run semantics, and is better argued on its own.

* 🔐 fix: Gate Legacy `retrieval` and the Code-Environment File Tools

Third Codex round, same shape twice more.

- The v1 assistant builder submits `{ type: 'retrieval' }` where v2 submits
  `file_search` (`AssistantPanel.tsx:203`). Only the v2 spelling was mapped, so
  `resolveAssistantToolPermissions` treated legacy retrieval as ungated and both
  v1 writers preserved it for a role denied `FILE_SEARCH`. Both spellings now
  answer to the same grant.

- `codeEnvAvailable` in the agent initializer was capability-only, and
  `initializeAgent` rebuilds `bash_tool`, `read_file` and the workspace file
  tools from it — after the tool loader has already dropped `execute_code` for a
  denied role. The bash gate stops command execution, but those handlers still
  read, search, create and edit files in an attached code environment. The flag
  now carries the role grant, and short-circuits before the role read when the
  deployment has the capability off.

* 🔐 fix: Resolve Tool Grants Once Per Request and Pair Every Capability Gate

Four review rounds found thirteen issues, all one class: a boundary the
permission check had not been copied into. Rounds 2 and 3 each found gaps
created by the previous round's fix, and a manual sweep of tool constructors
and upload routes still missed the `codeEnvAvailable` family entirely — because
the thing worth enumerating is not "paths that reach a tool" but "reads of the
capability".

There are 28 such reads. This replaces per-site permission checks with one
resolution per request, pairs every read that is a gate, and pins the list so a
new one cannot be added silently.

- `resolveToolRoleGrants` resolves `RUN_CODE` and `FILE_SEARCH` together and
  memoizes on the request. Both perf findings disappear by construction: gates
  share one role read instead of each issuing its own.

- Newly paired gates: `codeEnvAvailable` in the OpenAI-compatible, Responses and
  memory-agent initializers (`initializeAgent` rebuilds `bash_tool`, `read_file`
  and the workspace file tools from it, and hands their handlers the code
  environment); the capability checks in agent upload processing and in
  agent-management upload purposes; and the library chat-completion service,
  behind an optional `getRoleByName` dependency so embedders are not broken.

- The v1 Knowledge upload posts `assistant_id` with no `tool_resource`, so there
  is no resource to authorize. It now reads the assistant's own native tools and
  requires their grants, instead of treating a missing resource as unrestricted.

- The startup role read moved off the critical path into the existing
  `Promise.all`, and the direct tool-call check passes `req` so the shared
  loader gate reuses its result rather than repeating the lookup.

- `toolCapabilityGates.spec.js` pins every file that reads either capability
  with a count and a reason. A new read fails the suite until it is classified,
  so the next boundary announces itself instead of waiting for a reviewer.

* 🔐 fix: Abort Assistant Writes on Role-Lookup Failure, Authorize Legacy Uploads First

Round 5 found no missed boundary — the inventory held. All three findings are
about how the gates behave.

- Failing closed is right where a denial blocks the operation, and wrong where a
  denial instead filters a payload the caller persists. A transient role-store
  outage during an assistant create or patch was silently stripping
  `code_interpreter` / `file_search` / `retrieval` and saving the result, turning
  a momentary failure into permanent config loss. `checkToolRolePermission`
  gains `throwOnError`, and `resolveAssistantToolPermissions` uses it: every one
  of its callers either persists the filtered list or rejects the request, so a
  lookup failure has to propagate.

- The legacy-assistant upload gate ran after `sanitizedUploadFn` had already
  pushed the file to the provider, so a denied role left an untracked remote file
  and got a 500 for what is a 403. Moved to the route, next to the tool-resource
  gate, before any bytes are sent.

- `resendFiles` hydration derived its resource set from unfiltered `agent.tools`,
  so a denied role still paid the thread walk and code-file reads for a tool
  about to be dropped. The `execute_code` half now keys off
  `effectiveCodeEnvAvailable`, which already carries the grant.

The `file_search` half of that last one needs the grant threaded into
`initializeAgent`, which is plumbing in the most delicate file here for a latency
win on an already-correct path — deliberately left for a follow-up.

* 🧪 fix: Declare `resolveToolRoleGrants` in the Upload Spec's Module Stub

`Tests: api (shard 2/3)` failed on `a0b38c3`: `process.spec.js` stubs
`@librechat/api` wholesale, so the `resolveToolRoleGrants` call added to the
upload gate resolved to undefined and threw. Five failures in
`processAgentFileUpload`, plus one cascading `processFileURL` assertion from the
same suite.

The gate was verified through its own spec and the routes above it, but not
through the spec of the file it lives in — `process.spec.js` was never run
locally. Every spec belonging to a file this PR touches now has been, and the
mongodb-backed ones that still cannot run here were checked statically for the
same stub shape: all but `process.integration.spec.js` spread `requireActual`,
and that one never reaches these paths.

* ⚡ fix: Skip the Role Lookup When Code Execution Is Disabled

The browser initializer already guards this; the OpenAI-compatible and Responses
controllers did not, so every request on those endpoints issued a role read whose
result the capability short-circuit then discarded. Both now start the lookup
only when the deployment has `execute_code` enabled, matching the third.

* ⚡ fix: Drop Three Redundant Reads on the Gated Paths

- The memory initializer resolved grants unconditionally, so a deployment
  without `execute_code` paid a role read for a flag that could only be false.
  Guarded like the three chat initializers.

- Agent-management upload purposes resolved grants before `checkCapability`, so
  a purpose the deployment has switched off paid a role read on the way to being
  rejected. Capability first, then the grant.

- The legacy-assistant preflight builds an OpenAI client to read the assistant's
  tools, and `processFileUpload` then built a second one — re-reading the user's
  key expiry and values. The preflight now hands its client on, and processing
  builds one only when it wasn't given one.

---------

Co-authored-by: Paul <200737214+SSIG-IT@users.noreply.github.com>
2026-09-07 07:03:22 -04:00
Danny Avila
b181ef0232
🪬 feat: Configure Langfuse Trace User Identity and Metadata (#15667)
* 🪪 feat: Configurable Langfuse Trace User Identity and Metadata

* 🪪 fix: Prefer createRun's conversationId and warn once on an unset userIdField

* 🪪 fix: Validate Langfuse trace field names at the point of use

* 🧹 chore: Sort identity imports

* 🪪 fix: Source the trace model label from the initializer, not a prompt-affecting option

* 🧪 fix: Build the trace context without a prototype so partial client contexts resume

* 📦 chore: Bump `@librechat/agents` to v3.8.0 for the Langfuse trace userId override
2026-09-06 17:30:23 -04:00
Danny Avila
8a46ca070c
🧊 feat: Persist Context Fading Tier Across Agent Runs (#15505)
* 🧊 feat: Persist Context Fading Tier Across Agent Runs

Carry the pruner's latched context-fading tier from `@librechat/agents`
through `contextMeta` the way `calibrationRatio` already travels, so a
historical tool result keeps the same truncated bytes from one run to the
next and Anthropic's prefix-based prompt cache survives across turns
(danny-avila/agents#497).

- Add `IAgentFadingTier` to the message and conversation `contextMeta`
  types, schemas and the data-provider zod schema
- Add `isAgentFadingTier`, `resolvePersistableFadingTier` and
  `resolveRunContextMeta` in `packages/api`; a tier is persisted once it
  carries information (masking active or a budget below the window), and
  a latched tier is kept even at a neutral calibration ratio
- Seed `createRun({ fadingTier })` from the parent response's contextMeta
  regardless of encoding, since caps are character-based; the field rides
  the existing forward-compatible `runConfig` variable and older SDK
  versions ignore it, as does `Run.getFadingTier` being absent
- Accept a valid `fading` field in event actor context and reject a
  malformed one, matching the calibration checks
- Capture run context meta through one module-level helper for both
  completion paths instead of two inline copies

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Seed Fading Tier Across HITL Resume, Validate at the DB Layer

Address review findings on fading tier persistence:

- Leave the "is this tier worth persisting" decision to the SDK, whose
  `Run.getFadingTier()` now returns only informative tiers; comparing
  against the client's window misclassified untouched conversations
  whenever a reserve ratio was configured
- Carry `contextMeta` through the HITL pause projection (staged approval,
  job metadata, resume state) and seed the rebuilt client from it, so a
  paused turn resumes with the same tier and calibration instead of
  re-deriving a shallower one and rewriting the prefix
- Share run seeding between `chatCompletion` and `resumeCompletion`, and
  make capture tolerate partial runs and bare resume contexts
- Validate `contextMeta.fading` in `commitAgentEventActorState` with a
  shared `isAgentFadingTier` guard in data-schemas, so a malformed tier is
  rejected at the DB layer instead of silently cold-starting the actor
- Reuse `IAgentEventActorContextMeta` for `IMessage.contextMeta` instead of
  extending an inline duplicate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 test: Cover Resume Seeding From Paused Context Meta

Assert the resume controller seeds the rebuilt client from the context meta
captured at the pause before rebuilding the run, and move the capture
helper's JSDoc back above its function.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧹 style: Sort Imports in Fading Tier Plumbing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Persist Fading Tier Through Redis Jobs and Branches

Address Codex and Copilot review findings on fading tier persistence:

- Deserialize `contextMeta` in `RedisJobStore`, whose explicit read mapper
  rebuilt every other pause field but this one, so Redis-backed
  deployments resumed with no tier to seed after every HITL pause
- Copy server-private `contextMeta` onto the assistant message created by
  `POST /api/messages/branch`, so the next turn seeds its pruner the same
  way it would from the parallel source response
- Constrain the `fading` subdocuments in both Mongoose schemas (version
  enum, positive budget, required fields) to match the zod schema
- Throw a distinct "fading tier is invalid" error from both context meta
  validators instead of reporting a calibration failure

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through Stop and Re-Pause

Addresses the second Codex pass on the fading-tier persistence:

- A re-pause whose resumed segment has nothing persistable now clears the
  job's `contextMeta` in the same transition instead of leaving the first
  pause's calibration and tier for the next resume to seed from.
- The client publishes the run's live calibration and fading tier onto the
  job after each pre-invoke context snapshot (deduplicated on value), and
  the Stop path copies `jobData.contextMeta` onto the stopped response, so
  a follow-up from a stopped turn keeps the prompt prefix stable.
- Mid-run captures (HITL pause, Stop) read the graph's live state; `Run`
  refreshes its own getters only after the stream settles, so the pause
  used to persist the seeds it started from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Fence Context Meta Publication Ahead of Stop

A Stop handled by another replica reads the job before the owner is
signaled, so a fire-and-forget publish could lose the race, and a run
stopped before its first context snapshot never published the tier it
inherited. The client now publishes the inherited seed before the run
streams or resumes, and the context-usage handler awaits each
snapshot's publish so the write lands before the model call it
describes begins. Failures still only log; the pre-run publish is
optional-called so bare resume contexts keep working.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Read Back Context Meta After the Abort Claim

The run owner publishes its context meta ahead of each model call and
awaits the write, so a publish landing between the abort's initial job
read and its terminal claim describes the call whose partial output the
abort snapshot carries. abortJob now reads the same-epoch job back after
the claim and content re-read and takes its context meta, so the stopped
response persists the tier that produced its bytes. The refresh is
best-effort and logs on failure like the content refresh beside it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Share In-Flight Context Meta Writes and Clear Neutral Live State

Two snapshot callbacks in a parallel-agent run can capture the same tier
before the first write settles; the publisher now caches the in-flight
promise per serialized value so the second caller awaits the same
durable write instead of treating an uncommitted value as published.
A live snapshot with nothing left to persist after an earlier publish now
writes a neutral record (ratio 1, no tier), since a running job's fields
cannot be deleted through the metadata writer, so a Stop no longer
persists an earlier non-neutral state onto the response. The pre-run
seed publish is unchanged and is told apart from live snapshots by a
flag from the context-usage sink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 chore: Drop Unused Rest Parameter in Context Meta Test

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 feat: Persist Per-Agent Fading Tiers as Compact Context Meta

Reconciles the host with @librechat/agents at 9aacad1a, where graph
history stays canonical and each Run derives a provider-only projection
from the latched tiers. LibreChat therefore persists only compact state:
the calibration ratio, the default agent's tier, and now the per-agent
tiers from Run.getFadingTiers(), stored as validated entries so agent IDs
never become MongoDB field names and restored onto a null-prototype
record for RunConfig.fadingTiers. Truncated messages, canonical tool
content, projection provenance and per-message truncation state are
never persisted; the SDK's latched flag is stripped on capture.

The per-agent map travels everywhere the single tier already did:
response contextMeta, event-actor checkpoint state, HITL pause and
resume job metadata in both stores, the Stop path, and branch creation.
A shared Mongoose definition keeps the message and conversation schemas
identical, the zod schema validates entries, and commitAgentEventActorState
rejects malformed maps. Comments no longer describe graph messages as
truncated; only the provider projection is.

Tests cover: tool results and inputs reach the persisted content parts
at full size; contextMeta carries exactly the compact tier and
calibration fields; a persisted snapshot round-trips into the seeded
RunConfig unchanged and prototype-safe; per-agent tiers survive pause,
Redis job round-trips, and event-actor context validation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Restore Event Actor Types and Publish Seeds Before Abortable Setup

The previous commit's doc rewording in the conversation types dropped the
event-actor interfaces that precede the fading tier, which broke the
declaration bundle in CI; they are restored unchanged.

The inherited context meta is now published onto the job as soon as the
parent's state is loaded and awaited at the top of chatCompletion and
resumeCompletion, ahead of run creation and the other abortable setup
stages, so a Stop during setup still persists the parent's tiers onto the
stopped response. The partial response saved when every subscriber
disconnects now carries the job's context meta like the Stop and pause
paths do. The branch route keeps the source's context meta in the saved
message but strips it from the HTTP response, matching client reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Serialize Distinct Context Meta Publications

Overlapping snapshots from parallel agents could carry different tiers;
each started its own write and both job stores keep whichever write
finished last, so an older snapshot could overwrite a newer one. Each
distinct publication is now issued after the previous one settles, in
call order, so the newest snapshot is what a Stop reads back. Equal
values still share one write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Project Root Message Reads and Keep Inherited Meta on Setup Failure

The root GET /api/messages paths returned raw documents: the
single-message query read called getMessages without a projection and the
cursor page performed an unprojected find. Both now apply
CLIENT_MESSAGE_SELECT, with getMessagesByCursor accepting a select option,
so the persisted fading tiers never reach the client through the
paginated endpoint; the search-hydration read projects the same way.

A turn that loads a parent's context meta but fails before its run exists
used to replace the inherited meta with the capture of a missing run, so
the persisted error response lost the last valid tiers. Both finalizers
now keep the inherited meta when no run was created while a created run's
neutral state may still clear it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 refactor: Move Context Meta Publication Into packages/api

The publication coordinator (ordering, deduplication, in-flight sharing,
failure state) now lives in packages/api as createContextMetaPublisher,
with selectRunContextMetaToPublish deciding what a publication carries;
the legacy client keeps only the wiring to the job metadata writer. A
transiently failing write is retried with a short backoff before the
publication is reported and forgotten, so a snapshot's model call no
longer proceeds on the first rejection while the job holds the previous
record. The SDK swallows handler errors, so retrying is the only way to
strengthen the fence without failing the turn.

A terminal save now supplies contextMeta: null when the finished run has
nothing to carry, and saveMessage unsets a previously stored value in
that case, so a disconnect snapshot's record cannot outlive the run that
completed neutrally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Widen the saveMessage Method Type for the Null Unset

The exported method interface still typed contextMeta as the message
field, so the spec exercising the null unset failed typechecking; it now
mirrors the implementation signature.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Publish Every Agent's Snapshot, Sanitize Search Hits, Unbound Agent IDs

The context-usage handler now awaits the run's context meta publish for
every ON_CONTEXT_USAGE event, hidden sequential agents included, while
the visible-snapshot bookkeeping and the client emission keep their gate;
a Stop before the next visible snapshot finds the tier a hidden agent's
model call latched. Search results are built from the hydrated hit minus
its server-private contextMeta, since Meilisearch hydration projects every
schema field. The per-agent tier entry guard no longer caps the agent ID
length: ephemeral agent IDs encode endpoint, model and sender without a
bound, and dropping such an entry would let that agent alone re-derive
its tier on the next turn. Also formats the publication spec so the
static checks pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Unset Context Meta Atomically, Keep Publication State After a Failed Write

- `saveMessage` folds the `contextMeta` unset into the same update that
  persists the terminal response, on both the plain and the provenance-merge
  paths, so a failure between two writes can no longer leave a completed row
  carrying a disconnect snapshot's state.
- `createContextMetaPublisher` tracks whether any record has committed
  separately from the retryable latest publication, so `hasPublished` stays
  true after a later publication exhausts its retries and a following neutral
  snapshot still overwrites the earlier record on the job.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through the Resumable Stop Route and Neutral Resumed Completions

- The resumable Stop route copies the refreshed `jobData.contextMeta` onto the
  terminal assistant message it persists, as the abort middleware already
  does, so a stopped resumable response keeps the tiers that produced its
  partial output.
- A resumed run that completes with neutral context state now saves
  `contextMeta: null`, which unsets the paused segment's record instead of
  letting an omitted field keep it on the completed row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Trust Only Server-Authored Context Meta, Unset It on a Neutral Stop, Read It From the Job on Disconnect

- `POST /api/messages/:conversationId` and conversation imports drop any
  client-supplied `contextMeta`, and the run seeds its calibration and fading
  tiers only from a server-authored parent response, so a forged tier can never
  shape a provider projection.
- Both Stop paths save `contextMeta: jobData.contextMeta ?? null`, so a job that
  re-paused with neutral state unsets what an earlier pause stored on the row
  instead of leaving it behind.
- The disconnect partial save reads the same-epoch job record for the run's
  published context meta, since the client-facing resume snapshot never carries
  server-private state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧪 test: Mock the Job Store in the Resume Metadata Spec

The disconnect partial save now reads the same-epoch job record through
`GenerationJobManager.getJobStore()`, so the resume metadata spec's manager
mock provides it as the other disconnect spec already does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Keep Context Meta Off the POST Message Response

`POST /api/messages/:conversationId` returned the saved row verbatim, so a
client write against an existing server-authored message echoed the row's
stored context meta. The route now projects the row through the same client
projection the branch route uses, which drops the server-private field.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 📦 chore: Bump @librechat/agents to 3.7.16 for Persisted Fading Tiers

3.7.16 is the first release that carries danny-avila/agents#497, so
`Run.getFadingTier()` / `getFadingTiers()` now report the latched tier
this branch persists and `RunConfig.fadingTier` / `fadingTiers` seed the
next run from it. The dependency set is unchanged from 3.7.15; only the
package entry moves.

Adds an SDK-backed round trip to the fading spec: a first pruner latches
an informative tier, `resolveRunContextMeta` reduces it to the persisted
shape, and a second pruner seeded from that shape reproduces the first
run's projected tool-exchange bytes under instruction and calibration
drift, while an unseeded pruner under the same drift relaxes its tier and
rewrites them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-02 20:29:43 -04:00
Danny Avila
29f738dea8
⏭️ feat: Honor an Interrupt Before the Model Has Answered (#15491)
* ⏭️ feat: Honor an Interrupt Before the Model Has Answered

An interrupt armed while the model is still thinking sat as "Interrupting"
until the entire turn finished, then landed as a terminal continuation. The
cause is upstream: the SDK reads `shouldPreempt` once per streamed chunk, and
its seal gate requires non-empty text — so a silent provider is never polled at
all, and a reasoning-only turn is never sealable.

`@librechat/agents` gains a discard-and-restart path for exactly that window
(`HOOK_PREEMPT_RESTART_CAPABLE`). This wires the host half:

- `GenerationJobManager.subscribePreempt` registers run-scoped wake listeners,
  notified after an ACCEPTED arm — local or cross-replica, since both land in
  `armPreemptIds`. Fenced on `createdAt` like every other preempt entry point,
  and a throwing listener never fails the arm a queued steer depends on.
- `createSteerPreemptPoll` hands that channel to the SDK as
  `StreamPreemption.subscribe`, gated on a new `isSteerPreemptRestartSupported`
  probe so an SDK that cannot act on a wake is never handed one.

The wake is a hint only: `isPreemptRequested` stays the single authority, so
the level-triggered contract the seal path depends on is unchanged.

Inert until `@librechat/agents` publishes the restart contract — the probe
returns false against the pinned 3.7.11 and the run keeps today's behavior.

* 🩹 fix: Replay an Already-Armed Preempt on Subscribe

Preempt requests are level-triggered, and the window between a job becoming
steerable and the SDK installing its model-attempt listener is real. An
interrupt landing there was recorded with no callbacks to notify, and on a
silent or reasoning-only turn no later chunk poll may ever run — leaving the
request to wait out the whole turn, the exact stall this channel removes.

`subscribePreempt` now replays an existing arm to the newly registered listener
only, after registration: waking the whole set would re-notify runs that
already looked, and waking before registration would leave a concurrent arm
with nowhere to land. The `createdAt` fence is unchanged and still gates the
replay.

The SDK also reads the level flag once at each attempt's start, so this is
belt-and-braces rather than the sole cover — but it makes the host correct on
its own terms instead of resting on when the SDK happens to look.

* 🔧 chore: Update @librechat/agents dependency to version 3.7.14 in package-lock.json and related package.json files

* 🔧 chore: Update dependencies in package-lock.json and package.json

- Bump versions for @humanfs/core (0.19.1 to 0.19.2), @humanfs/node (0.16.6 to 0.16.8), @xmldom/xmldom (0.8.13 to 0.8.15), and qs (6.15.2 to 6.16.0).
- Add @humanfs/types (0.15.0) as a new dependency for @humanfs/node.
- Update dependencies for side-channel and side-channel-list to their latest versions.
2026-09-02 13:35:52 -04:00
Dustin Healy
0b96fb9f70
🪪 feat: Propagate Tenant to Agent Model Headers (#15521)
* fix: propagate tenant to agent model headers

* fix: propagate tenant across agent model calls
2026-09-02 13:25:38 -04:00
Danny Avila
1c7a172321
🕹️ feat: Configure BYOM Tool Permissions (#15504)
* feat: add configurable BYOM permissions

* test: cover BYOM permission settings live

* fix: harden BYOM permission settings

* perf: reuse principals for environment discovery

* chore: normalize BYOM settings style

* fix: Align BYOM Discovery and Admission

* fix: Harden BYOM Permission Boundaries

* fix: Align BYOM Skill Approval Routing

* fix: Preserve Lazy BYOM Approval Capabilities

* perf: Parallelize Cached Environment Reads
2026-09-02 10:38:00 -04:00
Danny Avila
b1c56239f9
🧤 feat: Enforce Safe BYOM Tool Approvals (#15503)
* feat: add safe default approvals for BYOM code

* fix: preserve safe BYOM approval baseline
2026-09-02 08:53:16 -04:00
Marco Beretta
2cf1007f3c
🧺 feat: Preserve Partial Work at Tool Call Limits (#15486)
* feat: Handle reaching the agent tool-call limit gracefully

When the graph exhausts its per-turn step budget, persist the turn as
unfinished with finish_reason tool_call_limit instead of a red error.
Warn the model as the budget runs low, and offer Keep going or Answer now.

* docs: Add screenshots of the tool-call-limit notice

Light and dark captures of the unfinished-turn card for the PR description.

* style: Tighten the tool-call-limit notice card

Pin dismiss to the top-right with uniform rounding, use a slightly
smaller hit target, and round the card to xl. Refresh the PR screenshots.

* fix: Address Copilot review findings on the step-budget path

Skip empty and subagent-scoped PostToolBatch events when counting
rounds. Drop the double-cast in the notice test helper and coerce the
tool-call-limit finish reason in the Continue-gate spec.

* fix: Address Codex review findings on budget and dismiss

Pass the resolved recursion limit into the OpenAI-compatible
processStream path so the step-budget hook matches the graph.
Reset the notice dismiss state when the displayed message id changes.

* fix: Unblock CI typecheck and import sorting

Declare recursion_limit on the OpenAI service agent type, sort that
file's local imports, and route the notice test context through unknown
so tsc accepts the partial ChatContext fixture.

* fix: Sort remaining PR imports for static checks

* chore: Remove tool-call-limit notice screenshots

* fix: Keep recovery actions from consuming composer drafts

Pass empty file, skill, and quote overrides on Keep going and Answer
now so a staged compose is not attached or drained.
2026-09-01 21:38:02 -04:00
Danny Avila
a3c2277c38
🚧 fix: Narrow Chat MCP Selections to What the Picker May Offer (#15478)
* 🚧 fix: Narrow Chat MCP Selections to What the Picker May Offer

The chat MCP picker offers a server only when the operator has not hidden it
(`chatMenu: false`) and the user reaches it directly rather than through an
agent (`consumeOnly`). Nothing enforced either condition on the request, so a
stale client, a replayed body, or a hand-written one could name any server and
get its tools — `consumeOnly` in particular is a permission the registry derives
per user, so ignoring it let a user reach in plain chat a server they had only
been granted through an agent.

Narrow the picker's selection in the two ephemeral loaders, against the same
predicate the client's `selectableServers` uses. The resolver is injected, so
`packages/api` stays free of the registry and the loaders keep their existing
`deps` shape.

Provenance is why this lives in the loaders rather than at the tool-resolution
chokepoint: only here is a server name known to have come from the picker.
Servers a model spec pins are the operator's own choice and are added after the
narrowing, so pinning a chat-hidden server to a spec keeps working.

`chatMenu` only ever comes from the config tier, so the overlay settles it with
no lookup at all. Lookups for the rest run concurrently and are cached per
server and user. A name the registry cannot resolve — request-tier, declared on
the body and never registered — is kept, as is one whose lookup fails: this
narrows an already-authenticated selection and is not the authorization
boundary.

* 📝 docs: Say What chatMenu Now Bounds

The schema still described `chatMenu` as dropdown visibility alone, which is
misleading now that it also bars a server from the chat selection a request
carries. Spell out both halves, and what it deliberately does not restrict:
agents, and servers a model spec pins.

* 🛡️ fix: Close Four Gaps in the Chat MCP Narrowing

Dedupe before resolving. The selection is request-supplied and the loaders used
to build a `Set` immediately; mapping the raw array first let one body repeat a
name enough times to launch a resolver — registry, cache, and ACL reads, with no
single-flight — per occurrence.

Resolve names through the tool-loading aliases. A configured `private server` is
addressable as `private_server` downstream, so an exact-key lookup let the
normalized spelling walk past a `chatMenu: false` check and still load tools.
Names are returned as sent; only the decision is canonicalized.

Let the request overlay settle `chatMenu`. The config tier is where the flag
lives and is what the picker itself was built from, so an overlay that re-enables
a server now outranks the durable config the registry returns, which was silently
dropping a valid selection. `consumeOnly` still comes from the registry.

Narrow the request body, not just the loader's copy. `applyContextToAgent` reads
`req.body.ephemeralAgent.mcp` directly and prefers it over the agent's tools, so
a hidden server's `serverInstructions` reached the model context even after its
tools were dropped.

* 🧭 fix: Ask the Registry Which MCP Servers the Picker May Offer

Three rounds of review found the same mistake in three disguises: this narrowing
re-derived the registry's tier precedence instead of asking for it. A config
entry was treated as authoritative over a user-DB or process-backed server of the
same name; the alias map was built from config names only, so a DB server named
`private_server` was misrouted to a config `private server`; and the overlay
short-circuit ran before any of that could be consulted.

Resolve the accessible set through `resolveAllMcpConfigs` instead — the same call
behind `GET /api/mcp/servers`, so what the request may use and what the picker
was offered come from one place and cannot drift. Precedence, cross-tier name
collisions, and the config overlay are the registry's job again, and the
selectability test is a single flag check over what it returns.

The result is smaller: no overlay branch, no per-server lookups, one catalog read
regardless of how long the selection is, and aliases built over the whole
accessible set so an exact name always outranks another server's normalized form.

Publish the servers actually in play back onto the request body, spec pins
included. `applyContextToAgent` prefers that list over the agent's tools, so
narrowing to the picker selection alone had started dropping a spec-pinned
server's `serverInstructions` while suppressing a hidden one's.
2026-09-01 18:47:36 -04:00
Danny Avila
aa7539909b
🪶 feat: Enable Standalone Skill Authoring (#15480)
* feat: enable standalone skill authoring

* test: align skill e2e flow with enable toggle

* fix: preserve legacy skill state on version restore
2026-09-01 18:39:05 -04:00
Danny Avila
dccd7ae496
🧶 fix: Preserve Multi-Agent Tool Call History (#15468)
* 🧭 fix: Preserve Multi-Agent Tool History

* 🧰 fix: Preserve Lazy Subagent Tool History

* 🧪 test: Enable Lazy Code History Fixture

* 🧭 fix: Derive Effective Lazy Tool History

* 🧪 fix: Tolerate Empty MCP Name Audits

* 🧪 test: Mock Lazy MCP Name Audit

* 🧭 fix: Expand Lazy Historical Tool Names

* 🧱 fix: Derive Durable Wildcard History

* 🧬 fix: Traverse Nested Wildcard History

* 🔭 fix: Resolve Wildcard Server Boundaries

* 🔐 fix: Persist MCP Call Identity

* 🪪 fix: Stamp Resolved MCP Execution Identity

* 🕸️ fix: Resolve Graph Member MCP Identity

* 🧭 fix: Fall Back From Empty Graph Member ID

* 🧭 fix: Preserve Aggregated MCP Identity

* 🧭 fix: Harden MCP Identity Resolution

* 🧭 test: Assert Concurrent Graph Limit Invariant

* 🧭 fix: Centralize Lazy History Policy

* 📦 chore: Bump `@librechat/agents` to v3.7.13

* 🧪 test: Align Content Filter Fixture With SDK History
2026-09-01 18:37:45 -04:00
Danny Avila
982a9a5141
🧴 fix: Strip LangChain URLs From Provider Errors (#15421)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* 🧹 fix: Strip LangChain Attribution From User-Facing Provider Errors

LangChain classifies provider failures by mutating `error.message` with a
docs URL, and that message is forwarded verbatim into the error content part
we persist, so users saw `docs.langchain.com/...` credited for a provider's
own 429. The client only intercepted `MODEL_NOT_FOUND`, leaving every other
code to render raw.

Resolve the classification server-side instead: codes with localized copy
become the typed payload the client already knows how to localize, and every
other message keeps the provider's wording with the docs URL removed before
it reaches the database. The untouched error still reaches the logs through
`getSafeErrorMetadata`.

The client keeps a code-driven lookup for messages persisted before this, and
gains `com_error_model_rate_limit` alongside the existing not-found copy. The
OpenAI-compatible and Responses controllers now share one provider-error
helper rather than three copies of it, so the strip applies on those surfaces
too.

* 🛡️ fix: Harden Error Sanitization For Non-String Messages And Legacy Text

Codex review follow-up.

`stripLangChainTroubleshootingUrl` assumed a string, so an error-like object
with a truthy non-string `message` threw `.replace is not a function` inside
the error handler that was meant to report it — the template interpolation it
replaced had coerced such values safely. It now coerces at the boundary, as
does the code lookup.

Legacy messages whose LangChain code has no localized copy still rendered the
docs URL, which is exactly the historical case the client-side path exists to
clean up. The rendered fallback is now stripped too.

Both helpers move to `librechat-data-provider`, which is shared by the client
and the backend, so one implementation serves both instead of a second regex
drifting in the component.
2026-08-31 12:51:34 -04:00
Danny Avila
413906abc3
🖍️ fix: Honor Agent Image Detail Settings (#15407)
* 🖼️ fix: Apply an Agent's Configured Image Detail

The Agent Builder exposes an Image Detail slider and saves it to
`model_parameters.imageDetail`, but nothing ever read it back: `encodeAndFormat`
resolves the level from `req.body.imageDetail`, which the agents route never
populates, so every image was encoded as `detail: "auto"`. Providers that reject
`auto` — MiniMax's OpenAI-compatible API among them — fail every image upload
through an agent, and the setting that would fix it has no effect.

The root cause is a classification error rather than a missing argument.
`extractLibreChatParams` splits LibreChat's own settings out of an agent's model
parameters, and its key set is derived from the `librechat` group in
`parameterSettings`. `imageDetail` sat in `baseDefinitions` instead, so it was
neither extracted for LibreChat's use nor stripped from the options handed to the
provider.

It is now grouped with `resendFiles` and the other LibreChat-level parameters,
which gives it the same path those already take: extracted during agent
initialization, carried on the initialized agent, passed to the client, and read
by `encodeAndFormat`. `AgentClient.getSaveOptions` already expected
`options.imageDetail` — nothing had ever populated it.

The request body still wins where it is set, so the classic endpoints keep
resolving detail from conversation settings exactly as before.

Fixes #15359

* 🔤 fix: Type Image Detail as Its Enum, Not a Bare String

`TConversation.imageDetail` is the `ImageDetail` enum, so the extractor's return
type and the initialized agent's field are now declared as that enum rather than
widened to `string`, and the tests pass enum members instead of raw literals.

Caught by CI's type check, which I had run before adding the tests and not after —
jest compiles the specs through babel, so the suite stayed green over a type error.
2026-08-31 08:34:05 -04:00
Danny Avila
7533d138fa
🧬 perf: Evolve Compaction Guidance on Warm Turns (#15371)
* perf: evolve compaction guidance on warm turns

* style: sort compaction adapter imports
2026-08-30 17:11:20 -04:00
Danny Avila
c1cb591d49
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments

* fix: include code environment in lazy agent type

* fix: harden stateful environment routing

* fix: complete code environment route isolation

* fix: declare code environment map type

* fix: preserve configured code execution routes

* fix: harden stateful environment updates

* test: assert route-scoped sandbox readiness

* fix: harden stateful environment lifecycle

* fix: isolate migrated code sessions

* test: assert route-qualified code sessions
2026-08-30 15:38:45 -04:00
Danny Avila
cd3768ed1f
🍵 feat: Continue Late Steers in Warm Agent Runs (#15357)
* feat: continue late steers in warm agent runs

* chore: bump agents sdk to v3.7.9

* fix: guard terminal steer admission
2026-08-30 11:54:17 -04:00
Danny Avila
e3ccaba5af
🛄 feat: Restore Compaction Guidance Across Continuations (#15356)
* 🧭 feat: preserve compaction guidance across continuations

* 🔧 fix: narrow persisted compaction fields
2026-08-30 08:13:24 -04:00
Danny Avila
10981b8637
🍱 feat: Guide Compaction With a Bounded Semantic Index (#15340)
* 🧭 feat: guide compaction with semantic context

* 🛡️ fix: fail closed on semantic intent collisions
2026-08-29 17:53:04 -04:00
Danny Avila
c06b09c945
🤖 feat: make Event Actor HITL durable (#15305)
* 🤖 feat: make event actor HITL durable

* 🤖 fix: break event actor outcome cycle

* 🤖 fix: close durable actor terminal races

* fix: harden durable event actor recovery proofs

* fix: close event actor resume publication races
2026-08-28 08:05:11 -04:00
Danny Avila
6fba51a35a
🫆 feat: Fingerprint Agent Context for Zero-Read Warm Continuation (#15299)
* feat: add context-fingerprinted warm continuation

* fix: preserve manual skills across warm turns

* perf: keep ordinary agent initialization neutral

* fix: fingerprint complete agent topology

* fix: close warm context replay gaps

* fix: preserve lazy subagent memory scope

* perf: fingerprint only model-bound memory

* fix: project lazy memory capability

* style: sort warm continuation imports

* fix: make warm continuation reconstructable

* fix: preserve lazy skill isolation

* fix: retain scoped skill semantics
2026-08-27 23:01:14 -04:00
Danny Avila
8ba76507ae
🧷 fix: Verify Scheduled HITL State Before Pausing (#15284)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
* fix: require durable storage for scheduled HITL

* fix: verify scheduled HITL checkpoints before pause

* fix: narrow scheduled HITL capability checks

* fix: honor ask tool filters in schedule preflight

* fix: enforce durability across scheduled HITL resumes

* fix: scope scheduled HITL admission to active tools

* refactor: move scheduled HITL admission to TypeScript

* fix: close scheduled HITL admission gaps

* fix: bind scheduled pauses to current checkpoint

* fix: cap pending actions at binding deadline

* fix: reject expired pending actions

* fix: guard approval deadlines atomically
2026-08-27 11:16:24 -04:00
Danny Avila
68fc46a055
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks

* fix: fence event actor checkpoint uncertainty

* fix: satisfy event actor type contracts

* fix: make actor reconciliation recoverable

* fix: preserve event actor lifecycle transitions

* fix: fence event actor lifecycle outcomes

* fix: enforce event actor lifecycle ownership

* fix: retain event actor settlement proof

* 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal

Repair and compensation deleted the reconciliation row they resolved, which
was the only durable proof that the invocation had already applied an external
action. A delayed duplicate owner could then reacquire the same invocation id
and repeat that action. Both resolutions now retire their receipt to `settled`
and record how it settled, so the same-id tombstone survives; `history_repaired`
and `action_compensated` still force a cold rebuild. Compensation undoes the
effect without re-authorizing the delivery, so a legitimate retry must arrive
under a new invocation id. A retried repair converges on its own receipt.

Bound the journal so a long-lived actor cannot grow its conversation document
without limit: a new fence is admitted only when no active lifecycle row
exists, so a capped push can evict nothing but the oldest settled receipts.

Stop shipping the unbounded source payload on every bound-child continuation.
It rode the delivery body regardless of the feature flag while the sibling
`fire` body deliberately sends event identity alone, so a large webhook payload
could push a previously working delivery past the chat route's body limit. The
actor binds an invocation from identity and never builds the prompt from it.

Blank the positional token map on warm continuations. It is derived from the
full DB history, while a warm run executes on checkpoint-restored state, so its
indices address different messages and the pruner never recounts them —
misattributing cached counts to the wrong messages in both directions.

Keep the replaced-claim exit on its cleanup path when preserving reconciliation
fails: the committing CAS already left a blocking row, so the failed status
upgrade costs provenance, not safety.

* 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry

Give the warm-continuation client test a populated token map and a real
cross-run summary so its assertions bite: the positional map must arrive
blank (checkpoint-restored state no longer matches DB-derived indices, and
the pruner never recounts a populated entry) while `initialSummary` must
pass through unchanged — it rides the system tail and summarizes
pre-boundary turns that were excluded from the very history the committed
checkpoint was built from, so blanking it would silently drop context no
warm run can recover.

* ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal

A compensated receipt still tombstones its invocation id, but its external
effect was explicitly undone — the terminal handler nonetheless replayed
every settled lifecycle's stored action as authoritative and settled the
public outcome as applied, telling an action-aware source the operation
stands and suppressing the new-invocation retry compensation requires. The
handler now settles a compensated invocation as failed with an explicit
compensation error, overriding even fresh applied run evidence from a
replayed generation; verified and repaired receipts continue to replay
applied.

Receipt eviction is now primarily age-based: a stale same-id owner is
bounded by time, not by how many newer invocations settle, so the previous
count-only slice let a high-rate actor evict a tombstone while its delayed
duplicate owner could still wake and repeat the action. Admission prunes
only settled receipts older than a retention window that dwarfs every
generation, job, and delivery-retry lifetime, and the count cap is demoted
to a raised document-size backstop.

* 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts

The terminal handler read its lifecycle snapshot, verified history, and then
settled the public outcome — so a compensation resolving the same receipt
during that window lost: the handler settled applied from its stale snapshot
and no retry could ever change the replay-identity-locked outcome. The
receipt's status CAS is now the serialization point: verification resolves
the receipt BEFORE settling, whichever transition wins determines the public
outcome, and a crash between resolve and settle converges through the
retained receipt's replay. The verified-replay probe requires the receipt's
own resolution, so a compensated receipt can never satisfy a verification
retry. This inverts the settle-before-receipt ordering deliberately: that
ordering guarded proof that resolution used to delete, and the receipt now
retains its full action proof through resolution.

The document-size cap is no longer an eviction quota. A receipt inside its
retention window is never discarded: when the journal holds a full cap of
unexpired receipts, new invocations are refused fail-closed until receipts
age out, making duplicate protection and document integrity simultaneous
invariants instead of a rate-dependent trade.

* 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path

Skill primes are spliced into the message list directly ahead of the newest
message, and a warm continuation forwards only that newest message — so a
checkpoint-restored actor would keep serving the prime bodies baked in at
its last cold start and never observe an edited or newly attached skill.
Until the actor head carries a context fingerprint that forces a cold
rebuild when the agent's skill context changes, agents with always-apply or
manual skill primes stay on the legacy path, which re-primes fresh bodies
every turn: correct on every event, just never warm.

* 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes

History-derived re-priming was a third path into the same staleness class:
an actor that previously invoked a skill carries no request-time prime
arrays, yet primeInvokedSkills re-resolves that skill's current body from
history each turn and the warm slice drops the reconstruction — leaving the
checkpoint's old body active after edits. The fork gate now keys on the
priming hook itself (present exactly when the skills capability is enabled)
alongside the request-time arrays, so every skill-body path routes to the
legacy rebuild until #15235's context fingerprint restores warm
continuation for skill-bearing actors.

* 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage

The executor read applied-action evidence from the run-step collection the
instant sendMessage resolved, but that collection is populated
asynchronously — an applied invocation could classify as actionless
(runSteps still empty while the tool result already streamed), discarding
its fork and stranding the actor cold while the terminal handler later
settled the same delivery as applied from the persisted evidence.

Authoritative proof is now recorded in graph context the moment the
expected tool executes: the request-owned recorder observes the tool-end
chain (which ToolNode dispatches synchronously with both input and output)
and applies the same fences as run-step evidence — exact tool name with the
MCP-suffixed form, the declared argument subset against the execution
input, an error-free result, and the background non-execution receipt
exclusion. readAppliedAction consults the receipt first; run-step
inspection remains the fallback for paths that bypass the tool-end chain.

Regression coverage reproduces the observed ordering: the real executor
commits the head from the receipt while run steps are empty, warm-continues
the next event, and never re-executes the action; recorder fences and the
receipt-first controller wiring are covered separately.

* 🎯 fix: Supply Execution Arguments to the Tool End Callback

The live Vertex + MCP canary exposed a contract mismatch the synthetic
fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end
callback with output only, while the action recorder must verify the
declared argument subset against the execution input. The receipt never
qualified, every turn fell back to cold history rebuilds, and the
tournament advanced with zero actor heads and zero retained checkpoints
while looking successful.

The execution handler owns both halves at the same moment, so the fix is
at the source rather than a correlation store: ToolEndCallbackData gains
the executed call's input and every callback site passes tc.args. A
handler-level regression drives the real createToolExecuteHandler and
asserts the callback receives both fields; recorder regressions pin the
production shapes — an output-only tool end must starve an
argument-fenced receipt rather than trust an unfenced match, and still
qualifies a name-only expected action.

* 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions

The background-claim callback reports the ORIGINAL tool's name for artifact
attribution on the poll turn that harvests a completed task. A name-only
expected action could therefore be impersonated by work some earlier turn
dispatched: the recorder would attribute that delivery to the current
invocation and commit a head whose state never contained the invocation's
own action. The run-step evidence path never had this hole — it sees the
poll tool's name — so the recorder must match its provenance discipline.

Delivery callbacks now carry an explicit backgroundDelivery marker set at
the one site that rewrites the name, and the recorder ignores marked
deliveries outright. Regressions pin both halves of the contract: the
delivery callback must carry the marker with the poll call's arguments,
and a marked delivery can never qualify even a name-only expected action.

* 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions

Three closeout-round findings, each converted into an invariant.

Every legacy-path invalidation now advances a durable epoch — including for
headless and already cold-marked actors, where the marker alone leaves no
CAS-visible trace — and the actor-head CAS requires the epoch observed at
preparation. A concurrently prepared fork whose history predates an
intervening legacy turn can no longer commit past it; the commit reports an
ordinary conflict and journals.

Execution identity is now emitted before post-execution output policy: when
a side-effecting tool succeeds but its returned content is withheld by the
output filter, the callback delivers an outputFiltered receipt with blank
content — the recorder accepts it as proof (rejecting model-detached calls
it cannot distinguish through the blank shape), the artifact path never
sees it, and an applied action is no longer reclassified actionless and
re-executed on retry.

Background-capable expected actions stay off the fork path: dispatch
returns a launch handle every evidence fence correctly rejects, and the
completion is provenance-marked as another turn's work, so a fork would
settle actionless before the external effect lands with nothing to stop a
retry from dispatching it again. The gate mirrors the MCP-suffix name
matching of the evidence path.

* 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance

The epoch fenced only the legacy turn's start: a fork preparing after the
begin invalidation but before the turn's message persistence observed the
new epoch and cold marker, rebuilt from history that did not yet contain
the turn, and committed cleanly because nothing advanced the epoch again —
making the incomplete rebuild authoritative and clearing the marker.

Every legacy event turn now seals its invalidation at terminal persistence
with a second epoch advance, on the success, replaced-claim, and error
exits alike. Sealing deliberately carries no quiescence requirement — it
must succeed while a fork fence is active, because that is exactly the
mid-turn race it defeats — and a fork that already committed against the
begin epoch is healed the same way: the seal re-marks the head cold, so
the next event rebuilds with complete history. Seal failure never diverts
the turn's own exit; the begin bump still fences everything prepared
before the turn.

* 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence

The second epoch advance could not make a legacy turn atomic, and three
findings shared that root cause: two conditional updates left a headless
gap a fork could create the head inside; the error exit sealed before
saveErrorTurn made the error history durable; and any crash or failure
between persistence and sealing left an incomplete fork authoritative,
because the seal was best-effort and its failure was swallowed.

A legacy turn now carries one durable fence. A token is written before
execution by a single update-pipeline write — no two-write gap, and the
cold marker is applied only where a head exists via $cond/$$REMOVE. While
the token is present no fork may prepare (the adapter refuses) or commit
(the CAS requires its absence), because the turn's messages are not yet
durable. One atomic write clears the exact token and advances the epoch
once history is persisted — after saveErrorTurn on the error exit — and
success, replacement, and error exits all route through it.

Failure is now fail-closed rather than silent: a failed seal leaves the
token set, which keeps blocking forks and is logged as such, and a fence
abandoned by a crashed turn is reclaimed only once stale, advancing the
epoch and marking any head cold so the next event rebuilds from whatever
history actually survived.

* fix: serialize legacy event actor turns

* fix: close legacy actor fence ownership gaps

* fix: preserve legacy actor persistence fences
2026-08-26 08:39:00 -04:00
Danny Avila
1489623fa3
🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210)
* 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E

The mock e2e lane's only credential-free model is a hand-authored script:
`fake-model.js` decides responses from ~60 `E2E_*` prompt markers. That
covers scripted shapes well, but no scenario replays a *real* recorded
provider conversation through the assembled chain, so real streaming
shapes — provider chunk cadence, reasoning deltas, usage metadata — are
only ever approximated.

This adds a record-once/replay-forever tier alongside the marker routing.

Record (`E2E_MODEL_FIXTURES=record`, needs a provider key): the run hook
appends a LangChain callback handler to every agent context's
`clientOptions.callbacks` instead of overriding the model, so the REAL
provider streams while each invocation's `AIMessageChunk`s serialize to
`e2e/fixtures/model-replay/<name>.jsonl` — text deltas, tool_call_chunks,
reasoning kwargs, and genuine usage metadata. Only the latest human text
is recorded for binding; system prompts and tool schemas never enter the
fixture.

Replay (default, keyless): `fake-model.js` consults `tryBindReplay` ahead
of marker routing, binding a conversation whose prompt matches the next
unconsumed invocation. The replaying model is not hand-assigned — it is
registered as SDK provider `librechat-e2e-replay` via `registerProvider`
and constructed through the SDK's own `initializeModel`, so registry
lookup, constructor clientOptions, and real `bindTools` all run the way a
live provider's would. Recorded chunks therefore stream through the same
createRun → graph → SSE → persistence chain.

Consumption is enforced rather than assumed: every invocation re-checks
its prompt against the recording, an invocation past the end of the
script throws, and a per-fixture ledger lets the spec assert at teardown
that every recorded invocation and chunk was drained. Streaming
incrementality is asserted from that ledger, not by sampling transient
DOM, which is a race by construction.

The credential-free profile is unchanged when not recording: the record
provider and its selector entry are template markers that stay comments,
and no existing spec's routing is touched (a fixture only binds on an
exact prompt match; everything else falls through).

Verified: record vs the real DeepSeek API 1 passed (15.2s); keyless
replay 1 passed twice (11.4s, 12.7s) with the ledger fully drained (2/2
invocations, 10/10 chunks, no overruns or mismatches); app-load,
completion, and chat 10 passed unchanged.

* fix(e2e): rebind a replay fixture from the top for a new conversation

The replay cursor is process-global while the web server outlives a
Playwright retry, so a fully consumed fixture left the retry unable to
bind its first prompt: it fell through to marker routing and failed
deterministically, burning every configured CI retry. A partially
consumed attempt failed the same way.

Binding now restarts the fixture when the incoming prompt matches its
first recorded invocation, resetting the ledger with the cursor so the
new attempt is judged on its own consumption instead of accumulating the
previous one's counts. Continuing an in-progress binding still outranks
restarting, so a fixture whose opening prompt repeats later in the script
advances rather than rewinding.

The over-consumption guard is untouched — it fires inside the stream when
the cursor passes the end, not at bind time.

* fix(e2e): close three replay-lane gaps found in review

Restart the recorder on a retry. Its state is process-global like the
replay cursor, so a failed attempt that had already recorded invocations
left the counter advanced: the retry appended 2/3 after 0/1, or kept the
previous attempt's `error` line, and the fixture was unusable for replay.
Recording now truncates and restarts when the opening prompt reappears,
mirroring the replay side's rule and its caveat.

Retain a consumed binding for the conversation that drove it. An extra
user turn past the final recorded invocation found no next invocation and
fell through to ordinary fake-model routing, so it was answered with a
mock reply: the over-consumption guard never ran and the already-drained
ledger still passed. Such a conversation is now recognized by its human
turns opening with the fixture's recorded prompts, and stays bound so the
stream raises the overrun. Continuing an in-progress binding still
outranks restarting, which outranks retaining a consumed one, so a
retry's fresh conversation rewinds rather than being read as an extra
turn.

Validate the fixture the recording actually wrote. Record mode honors
`E2E_MODEL_FIXTURE_NAME`, but the spec always inspected the committed
`deepseek-two-turn`; another name wrote elsewhere while the assertions
read the pre-existing file, and because the prompts are fixed the stale
answers could match and green a run that verified nothing it produced.

* fix(e2e): make replay binding correct for tool and subagent fixtures

Round two's retry and consumed-binding fixes both assumed one model
invocation per user turn. A turn that calls a tool breaks that: the model
is invoked again after the tool result under the same latest human
message.

Identify a retry by the conversation boundary, not the prompt. The
recorder ran per invocation and truncated whenever the opening prompt
reappeared, so a tool round trip looked like a retry and discarded the
recorded tool-call invocation. Restart detection now sits in
`installRecorder`, which runs once per `createRun`: a turn whose history
holds no prior human message begins a conversation.

Compare consumed bindings against user turns, not invocations. Several
recorded invocations can share one prompt, so a one-to-one comparison
could not recognize the originating conversation — invocations `[A, A, B]`
against history `[A, B, C]` failed on both length and elements, and the
extra turn fell through to the fake model with the drained ledger still
passing. Fixtures now carry their collapsed turn sequence.

Override the subagent model too. `graph.overrideModel` is not inherited
by child executors, so a fixture recording a subagent call — record mode
captures child invocations already — would leave the child on its
configured provider: an underrun, and a real provider request in a lane
that must stay keyless.

Reject ambiguous prompt matches. Binding order followed filesystem
enumeration, so a second fixture sharing a prompt could silently redirect
a scenario to the wrong chunks and ledger; the spec's choice never
reaches the server-side loop, so ambiguity fails instead of picking a
winner. Fixture identity is the file name for the same reason — a
recorded `meta.name` is descriptive, and trusting it let a copied fixture
collapse onto another's registry key and ledger.

Prove the recording is fresh. The spec removes the selected fixture
before driving, so a run whose hook never installed the recorder fails
instead of greening against a stale artifact whose answers still match
these deterministic prompts.

* fix(e2e): rewind a replay fixture at the conversation boundary

Consecutive invocations can share a prompt — a tool call produces exactly
that — so an attempt stopping mid-turn left the cursor on an invocation
whose text still equalled the opening prompt. Matching the cursor first
meant a retry's fresh conversation resumed after the tool call instead of
rewinding, consuming the post-tool invocation and silently replaying a
different script than was recorded.

A conversation boundary now outranks an in-progress cursor: a fresh
conversation whose prompt opens the fixture rewinds even when the cursor
would have matched. Continuing still outranks restarting within a
conversation, so a turn that calls a tool advances to its post-tool
invocation rather than rewinding on its own repeated prompt.

* fix(e2e): refuse cross-conversation binding and prove content streaming

A fresh conversation could steal a partly consumed fixture's later turn.
Only a conversation opening with the fixture's first prompt was treated
as a boundary, so after `[A, B]` had consumed `A`, an unrelated new
conversation whose first message was `B` matched the cursor, received the
recorded second-turn response, and advanced the shared cursor without
ever having driven `A`. A conversation start may now only rewind a partly
consumed fixture, never continue it; continuation within a conversation
is unaffected.

The incrementality assertion counted empty frames. Providers emit empty
initialization and usage-metadata chunks around the content deltas, so a
total chunk count above one was satisfied by a single delta: the previous
fixture's closing turn had four chunks and one content-bearing delta
carrying the whole answer, and both modes stayed green without proving
incremental assistant-content streaming at all. Fixtures now track
content-bearing chunks separately, the closing prompt asks for prose
rather than a number, and both modes require several content deltas on
that turn. Re-recorded: the closing turn now carries 28 content deltas.

* fix(e2e): scope record mode to the fixture spec

`E2E_MODEL_FIXTURES=record` replaces the fake-model hook globally, so an
unfiltered entry point such as `npm run e2e:mock` sent every spec under
specs/mock to the paid real-provider endpoint, while each fresh
conversation truncated and rewrote the one selected fixture — leaving an
artifact from whichever scenario happened to run last.

Record mode now matches only the fixture spec: an unfiltered recording
run lists one test instead of 203. Replay mode is untouched and still
collects the full suite.

* 🪪 fix: Bind Replay Fixtures by Conversation, Not Prompt Text

Prompt text was standing in for conversation identity, and three review
rounds found the same class of defect underneath it: a tool call repeats
a prompt across invocations, a retry repeats it across attempts, and a
resumed run has neither prompt nor history because `createRun` is rebuilt
with no messages while state comes from the checkpoint. Each fix in that
space created the next gap.

Thread the identity instead. `createRun` accepts a `conversationId` and
passes it to the run hook, which the agents controller supplies at both
call sites — the same value it already uses as the checkpointer's
`thread_id`. The field is optional and the hook is env-gated, so nothing
changes when the harness is not in use.

Binding then collapses to ownership. A fixture is owned by the
conversation that claimed it, and its cursor is authoritative wherever it
stands: an extra turn reaches the over-consumption guard rather than
falling through to the scripted fake model, and a resumed run keeps
replaying with no prompt to match. A different conversation may claim the
fixture only by opening it, which rewinds — what a Playwright retry looks
like. Everything else is refused, so an unrelated conversation can no
longer continue someone else's partly consumed script by repeating a
later prompt. The prompt is still re-checked on every real turn; only a
resume, which structurally carries no human message, is exempt. The
previous text-and-history rules remain as a fallback when identity is
absent.

The recorder keys the same way: a new attempt is a new conversation, so a
resume no longer truncates the fixture mid-turn and discards its
tool-call invocation.

Record summarization too. The summary provider runs on its own model with
its own callback list, so a scenario crossing the context-pruning
threshold recorded the agent's invocations but not the summariser's,
leaving a fixture that could not reproduce the pruned context.

* 🧾 fix: Harden Record Mode and Make the Rendered-Text Assertion Honest

CI caught what local runs had not: the committed fixture was never
replayed locally, because the record run overwrote it after the replay
check rather than before. Re-recording and replaying in that order is
what surfaced the rest of this.

The DOM assertion compared raw recorded text against rendered markdown.
The previous answer opened with `52.`, which Markdown renders as an
ordered-list marker, so those characters never appear in the DOM and the
match failed on all three CI attempts while replay itself was correct.
The closing prompt now asks for prose beginning with a word, a leading
enumerator is stripped before matching, and only a prose prefix is
compared.

Derived configs discarded the record-mode restriction. `config.redis.ts`
and `config.mermaid.ts` spread this config and then replace `testMatch`,
so `e2e:mock:redis` in record mode would still send its specs to the paid
provider. A restriction expressed as an overridable value cannot hold, so
record mode now refuses any config but the mock one.

Superseded recording callbacks could write across a reset. A failed
attempt with a provider call still in flight keeps its handler on the old
graph; after the retry reset, that call would allocate an invocation from
the new counter or append an `error` entry with a cleared mapping.
Handlers now carry the recording generation they were installed for and
ignore everything from an older one, and attachment dedupes against the
current generation so a graph carried across a restart is not left with
an inert handler.

* 🚧 fix: Make Summarization an Explicit Boundary, Not a Half-Feature

Recording summarization invocations without replaying them is worse than
ignoring them. Replay routes the agent model and subagents only, so a
recorded summarization entry takes a slot in the fixture sequence that
replay never consumes, and the next primary call reads the summariser's
chunks — a prompt mismatch or, worse, silently wrong content.

The attachment was also aimed at the wrong shape: the SDK reads
`summarizationConfig.parameters`, not `.parameters` nested under
`.config`, so the previous attempt would have attached to nothing in a
real run. Its test passed only because the test built the shape the code
expected rather than the shape the SDK provides.

Rather than ship a half-routed feature, recording now fails the moment
summarization runs, naming the reason. Both shapes are guarded so the
guard cannot miss the way the recorder did. Summarization fixtures need
replay routing for that model before they can be supported.

The derived-config guard added alongside it was itself broken: workers do
not carry `--config`, and the argument lookup fell through to
`process.argv[0]`, so every recording run aborted claiming the node
binary was an unexpected config. The flag is now located explicitly and
absence is treated as "not the process that parsed the CLI".

* 🔒 fix: Close the -c Config Alias and Pin the Recorder's Fixture Name

Playwright documents `-c` as an alias for `--config`, so record mode
launched as `playwright test -c e2e/playwright.config.redis.ts` slipped
past a guard that recognised only the long spelling. Both spellings and
both `=` and space forms are now parsed.

Accepting arbitrary fixture names also worked against the ambiguity
check. This spec drives one fixed prompt pair, so recording under another
name left two fixtures sharing those prompts; replay then refused to bind
either and the keyless lane stopped working — a successful documented
recording run could disable the suite it exists to serve. The spec now
records only the fixture it owns and says so when asked for another.

* 🔧 test: Record a Real Tool-Call Turn and Replay It Through the Tool Node

The fixture format carried `tool_call_chunks` and the binding advanced
through a turn's invocations, but nothing had recorded a real
tool-calling conversation end to end — the path was covered only by
hand-written synthetic fixtures, and it is the first one a new scenario
would exercise.

This records one: the provider calls the `remember_fact` MCP tool, the
tool runs, and the model is invoked a second time with its result. That
is the shape a single prompt cannot express — one user turn spanning
several model invocations, all sharing one prompt — so it is what proves
the turn-vs-invocation distinction the binding rules were built around.

Replay drives the real tool node rather than replaying its output, so the
tool executes again and the assertion checks its live result.

Two fixtures now coexist, which the record path had to grow for: the
config keeps an allowlist so an unknown name is still refused, record
mode collects every replay spec, and each spec records only the fixture
it owns and stands down for the others.

MCP tools reach the model under a server-qualified name
(`remember_fact_mcp_e2e-memory`); that qualification has changed before,
so the assertions match the base name as a prefix rather than pinning the
suffix.

Verified: record 1 passed (15.6s, real API) then replay 1 passed (13.6s)
against that fixture, ledger drained 2/2 invocations and 35/35 chunks;
both replay specs together 2 passed; app-load, completion, chat and
mcp-ephemeral 12 passed.
2026-08-25 18:29:55 -04:00
Danny Avila
ac2aef00f6
🫗 fix: Drain Quoted Excerpts Into Mid-Run Steering (#15175)
* 🧭 fix: Carry Quoted Excerpts Through Mid-Run Steering

"Add to chat" quote chips were dropped by every during-run steer path: the
steer POST had no quotes concept, so a composer-origin steer left the chip
staged (gluing onto the NEXT send) and a queued item steered into the live
run lost its quotes silently.

Quotes now ride the steer protocol end to end:
- POST + admission: `quotes` on the steer body, normalized like the chat
  route's (getReferencedQuotes caps), part of the idempotency fingerprint
  only when present so pre-existing receipts still replay.
- Injection: merged into the model-bound turn as Markdown blockquotes at
  both boundaries (text-only and media paths), mirroring prependQuotes.
- Persistence + replay: the STEER content part stores `quotes` separately
  from the typed text; stampSteerPartMedia re-merges them per turn (even
  with resendFiles off) via the SDK's transient media stamp, with the quote
  block folded into the token budget.
- UI: composer steers/interrupt-steers drain the chips (skill picks stay
  staged — they configure a NEW turn's run); SteerPart and the in-flight
  bubble render the same MessageQuotes reference blocks as user bubbles;
  queued/failed rows show a quote count; reconnect reseeds fall back to the
  server item's quotes when no local chip survives.
- buildMessages keeps its zero-await path to the parallel context kickoff
  via a synchronous stamp-target probe.

* 🧭 fix: Keep Quotes in the Client-Safe Steer Projection

toPendingSteer is the projection behind resume-state pendingSteers, abort
responses, and terminal leftover claims — dropping quotes there would lose
them on exactly the recovery paths the reconnect reseed's server fallback
relies on.

* 🧪 test: In-Flight Steer Bubble Renders Carried Quotes

* 🔁 fix: Re-Stage Quotes When a Pre-Quotes Replica Accepts the Steer

Codex flagged the rolling-deploy window: an old replica 202s a quoted steer
while dropping the excerpts, so the client cleared the chips for context the
model never received.

The 202 (fresh and receipt replay) now echoes quotesAccepted from the
DURABLE item; a missing echo on a quote-bearing composer-origin steer
re-stages the excerpts as composer chips — the pre-steer behavior, so they
ride the next send instead of vanishing — and strips them from the surviving
chip so a later terminal conversion cannot duplicate them. Queued-origin
steers keep quotes on the item, whose restore paths already return it
intact. The residual cross-version lost-ACK retry stays fail-closed as a
409 idempotency conflict (failed chip with retry controls).

* 🔁 fix: Close the Remaining Cross-Version Quote-Loss Windows

Codex round 2:
- Send now of a quoted queued item against a pre-quotes replica now
  re-stages the excerpts too (the row is consumed and the words inject
  bare, so the composer is their only remaining home); the strip clears the
  chip's captured origin copy so reclaims and terminal conversions cannot
  duplicate them.
- A quoted retry whose lost first ACK was accepted by a pre-quotes replica
  now REPLAYS that legacy receipt instead of 409ing: the stored fingerprint
  matching the quote-less hash of the same words proves the cross-version
  case, and the replayed 202's missing echo drives the re-stage. Different
  quotes against a quote-bearing receipt still conflict.
- TSteerAppliedEvent.part gains the quotes field (typed SSE consumers).

* 🧪 test: Drop the Stale Narrow SteerDrainOutput Alias

The spec's local intersection re-declared injectedMessages with
content: string, predating the SDK pin that declares the field natively
(content: string | MessageContentComplex[]). Under CI's clean install the
hook's BaseHookOutput is no longer assignable to that narrower alias; the
plain PostToolBatchHookOutput is the correct type for every drain/boundary
assertion. Verified against the published 3.6.16 dist and the local one.

* 🔁 fix: Honor the Generation Owner's Quote Capability End to End

Codex round 4:
- steerQuotesCapable rides job metadata (createJob + HITL resume rewrite),
  mirroring preemptCapable's owner-recorded pattern: an upgraded admission
  replica no longer stores quotes — or claims them accepted — for a
  generation whose older owning drain would silently drop them at
  injection. The missing echo drives the client re-stage, and a later
  capable handover cannot double-deliver restored context.
- Applied events reconcile dropped quotes: when a quote-less applied part
  settles a quote-bearing chip (the lost-202 ordering the ACK-echo path
  cannot see), resolveSteerChip and both reconnect settle paths re-stage
  the chip's excerpts before removing their only copy. mergeRestagedQuotes
  dedupe keeps every trigger idempotent for the same excerpts.

* 🔁 fix: Re-Read Quote Capability at the Last Moment and Cap Restaged Chips

Codex round 5:
- A HITL resume rewrites steerQuotesCapable without changing the
  generation's createdAt, so the enqueue fence cannot see a
  capable-to-legacy handover landing during admission's awaits. Re-read
  the owner's flag immediately before item construction (paid for only by
  quote-bearing requests); the residual between re-read and enqueue commit
  matches preemptCapable's documented race.
- mergeRestagedQuotes now respects the 10-quote contract with the staged
  chips winning: a restored tail that cannot ride the next send is dropped
  explicitly instead of rendering as a chip the submission would silently
  discard. MAX_QUOTE_COUNT moves to utils/steer as the single client
  source; QuoteButton imports it.

* 🔁 fix: Steer Quote Coverage for Preflights, Memory, and Single-Scan Stamping

Codex round 6:
- Stored-message policy inspection now extracts steer-part quotes as quote
  fragments (path /content/N/quotes/M), so conversation import and shared
  link preflights inspect the newly persisted field exactly like top-level
  message.quotes.
- The memory copy gets its own quote-merge stamp (text only, resendFiles
  false): formatAgentMessages ignores part.quotes, so without it a steer
  whose substance lives in its excerpt reached the chat model but never
  memory extraction.
- collectSteerStampTargets replaces the boolean probe: buildMessages
  collects once and hands the targets to stampSteerPartMedia, keeping the
  zero-await fast path without scanning the history twice.

* 🔁 fix: Redis Quote Plumbing, Conversion-Race Guard, and Quote-Bound Recovery Proof

Codex round 7:
- RedisJobStore.deserializeJob now restores steerQuotesCapable (the explicit
  mapper otherwise dropped it on every read, leaving quote steering inert in
  Redis deployments), with the round-trip spec extended.
- Both Lua parked-steer projections (terminal close + generation
  replacement) forward item.quotes, matching toPendingSteer — a lost final
  no longer strips excerpts from durable recovery in Redis mode.
- The no-echo restage reads the SURVIVING chip (reclaimRejectedChipQuotes):
  a terminal conversion that beat the delayed 202 already moved the quotes
  onto the queued follow-up, and re-staging them again double-delivered.
  Regression-tested with the conversion-before-ACK ordering.
- RecoveredSteerPayload binds normalized, order-significant quotes (builder,
  validator, TS matcher, and the Lua decode+matcher): a stale client
  presenting the same recoverySteerId with altered or missing quotes cannot
  consume the parked source. Quote-less sources keep matching quote-less
  recoveries.

* 🔁 fix: Execution-Bound Quote Capability with an Atomic Enqueue Predicate

Codex round 8:
- steerQuotesCapable becomes a transient assertion translated (at createJob
  and in ApprovalLifecycle.resolve) into steerQuotesExecutionId, valid only
  while it equals the LIVE providerExecutionId. A legacy replica winning a
  HITL resume rewrites the execution id without knowing the marker, so its
  stale assertion self-invalidates — a bare boolean could not be cleared by
  code that predates it.
- The fenced enqueue evaluates that equality atomically (all three Redis
  scripts decode-and-strip like the existing preemptCapable normalization;
  both InMemory sites mirror it) and returns the persisted item, so the
  quotesAccepted echo reflects exactly what was stored even when a handover
  lands between admission's read and the commit. The last-moment re-read is
  gone — the transaction is the authority.
- Tests: capable-resume re-binding, legacy-resume omit-not-clear
  invalidation, the admission-vs-handover race (capability read true, then
  execution rewritten before enqueue), and the Redis round-trip of the
  marker.

* 🔁 fix: Full Redis Parking Coverage and Loss-Moment Quote Restaging

Codex round 9:
- The two remaining Redis parking projections (terminal status CAS and
  stale-running cleanup) forward item.quotes — every field-picked steer
  projection now carries them (audited: 2 Lua 'projected' + 2 Lua
  'clientItem' + toPendingSteer).
- The ordinary no-echo ACK no longer re-stages: the steer has not injected
  yet, so the quotes stay carried on the pending chip. A quote-less applied
  event re-stages them at the actual loss; a terminal leftover conversion
  carries them onto the recovered row, whose normal send delivers quotes on
  any server — re-staging at the ACK let that leftover auto-send bare text
  while the excerpts glued onto an unrelated draft. Only the settled
  receipt replay (already injected, no future event) reclaims immediately.

* 🔁 fix: Legacy-Replayable Receipts with Separate Quote Identity

Codex round 10: an upgraded-first receipt stored a quote-inclusive
fingerprint no pre-quotes replica could recompute, so a lost-ACK retry
routed through one 409'd already-accepted words with duplicate-send
controls.

The durable fingerprint reverts to the quote-independent 3-field hash —
the one shape EVERY deployed version computes, replayable across a rolling
deploy in both directions — and quote identity moves beside it as
requestedQuotesFingerprint (of the REQUESTED quotes, pre any capability
strip, so an incapable-owner acceptance still replays its own retries).
Absent records (legacy-written or quote-less) accept any same-words retry,
preserving the round-5 rule; present records must match exactly, keeping
different-quotes clientSteerId reuse a 409 on quote-aware readers. Under
the keep-on-chip client contract a legacy replay's missing echo is
harmless — the excerpts stay carried on the pending chip.

* 🧪 chore: Re-Trigger CI After Dropped Workflow Events
2026-08-24 22:29:13 -04:00
Danny Avila
5a8700643c
⚡ perf: Build the Memory Message Copy Only When Something Reads It (#15164)
buildMessages formatted every history row twice per turn — a prompt
copy and a memory copy — then discarded the entire memory payload
unless some row carried fileContext, which is the rare case. The
memory copy has exactly two consumers: that payload, and the canonical
recount of a row, where it is content-identical to the prompt copy
unless the row itself has fileContext. So the prompt copy is now the
recount surface for context-free rows, a fileContext row builds its
memory copy at recount time, and the full memory payload is assembled
in a deferred pass — same formatting, same per-row merge order — only
once a row has proven the payload will be kept. The common turn
formats each row once instead of twice and no longer allocates a
payload it throws away.

Also forwards the run's useLegacyContent to formatAgentMessages as the
new legacyContent option, inert on the current SDK release: once the
SDK change ships, text history is emitted pre-flattened so the
per-request legacy projection stops cloning every message and the
context meter's identity-based count reuse holds across the
projection.
2026-08-24 10:33:47 -04:00
Danny Avila
77cb72e50c
🧮 perf: Enable Agent Context Count Reuse (#15130)
* perf: enable agent context count reuse

* fix: declare token counter return type

* fix: initialize cached token counters

* style: sort token counter imports

* fix: Keep cached token counts exact
2026-08-23 03:05:02 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Danny Avila
8ae94afa91
🪡 fix: Thread Parent Message ID Through MCP Request-Scoped Bodies (#15095)
* fix: Unify MCP request-scoped headers

* fix: address request-scoped MCP review findings

* test: preserve request scope on status errors

* fix: treat authorized on-demand MCP servers as ready

* refactor: separate MCP readiness from connection state

* fix: preserve on-demand MCP readiness labels

* test: satisfy OpenAI conversation ownership guard

* fix: keep MCP action predicates boolean

* fix: close deferred MCP request context gaps

* fix: preserve on-demand MCP configuration actions

* fix: fail closed on unavailable MCP parent context

* test: complete MCP connecting-state mocks

* fix: preserve missing MCP parent on continuations

* fix: align native MCP request identities

* fix: preserve edited MCP parent identity

* test: use scoped Agent initializer fixture

* test: expose MCP request body helper

* fix: preserve MCP turn identity across resume

* style: sort stream metadata imports

* fix: carry normalized MCP identity to execution
2026-08-21 16:33:01 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
547bd8c4bf
🧵 feat: Persist View-Only Subagent Threads (#14957) 2026-08-18 07:41:42 -04:00
Danny Avila
7d62be2ad3
🕸️ feat: Run Saved Agent Teams as Subagents (#14944)
* feat: Add graph subagent integration

* style: Sort response usage test imports

* fix: Preserve lazy graph runtime context

* fix: Use isolated graph input helper

* test: Align graph integration fixtures

* fix: Preserve lazy graph runtime capabilities

* fix: Bound lazy graph metadata preload

* fix: Harden lazy graph resolution lifecycle

* fix: Coalesce lazy graph member resolution

* fix: Snapshot initialized graph members only

* fix: Preserve lazy agent runtime context

* fix: Preserve batched lazy context preparation

* fix: Preserve graph member capability bounds

* fix: reconcile graph subagents with execution profiles

* style: align graph subagent types with formatter
2026-08-17 18:02:52 -04:00
Danny Avila
7d850c308a
🧠 feat: Add Live Reasoning Labels (#14893)
* feat: add live reasoning labels

* fix: Stabilize reasoning label checks

* fix: Address reasoning label review findings

* chore: Bump Agents SDK for reasoning labels

* fix: Reset reused reasoning step evidence

* fix: Reconcile cleared reasoning labels

* fix: Fence reasoning label resets

* fix: Reset reasoning ownership before gap labels

* fix: Preserve THINK type through label reset

* test: Expect run-global reasoning revision
2026-08-16 18:11:55 -04:00