Commit graph

82 commits

Author SHA1 Message Date
Marco Beretta
91e00bbe6e
🧷 fix: Mark a Failed Compaction So Its Rerun Controls Stay Withheld (#15883)
* 🧷 fix: Mark a Failed Compaction So Its Rerun Controls Stay Withheld

A manual compaction that failed with an error part carried no marker: the
server stamped `initiatedBy: 'user'` only on a summary that produced text and
returned early once an error part existed. On a branch ending in a user
message, the resulting turn had no marker and a user parent, so the hover row
kept Edit, Regenerate and Continue — and using one answered that user message
instead of retrying the compaction.

`markCompactionOutcome` now marks whichever part carries the outcome, summary
or error, and `isUserInitiatedCompaction` reads both. The marking moved from
`api/server/controllers/agents/client.js` into `packages/api`, where the
behavior belongs; the controller keeps the call.

* 🏷️ refactor: Type the compaction spec helpers with SummaryContentPart

* 🧯 fix: Record a Marked Failure When a Compaction Produces Nothing

A compaction run that produced neither a summary nor an error part threw
COMPACTION_FAILED out of `sendCompletion`. The turn was then saved by
`saveErrorTurn`, which writes only top-level `text`/`error` and no content, so
the row carried no compaction marker: on a user leaf the hover row kept
Regenerate, and using it answered that user message.

`markCompactionOutcome` now records the typed failure as a marked error content
part instead, so the turn streams and persists like every other failed
compaction. A cancelled run still fails as a typed error, leaving the abort
path to own the turn. `saveErrorTurn` marks the compaction rows it still writes
for failures that are rethrown rather than recorded as content.

Verified end to end by two new mock scenarios that run Compact context against
a summarizer returning nothing, live and after a reload.

* 🧪 test: Run the Empty-Compaction Scenarios Against a Real Compaction

* 🧱 refactor: Resolve Failed-Turn Compaction Content in packages/api

`saveErrorTurn` decided in CJS whether a failed row carries compaction
identity. The decision moves to `resolveFailedTurnContent` in
`packages/api`, which returns the content fields a failed turn is persisted
with — the marked failure for a compaction, nothing for any other turn — and
`request.js` spreads the result.

* 🧹 fix: Keep an Unusable Summary Out of a Failed Compaction's Turn

History loading accepts any nonempty summary as the conversation's checkpoint
(`BaseClient.findSummaryContentBlock` ignores `failed`), so a compaction that
streamed partial text before failing would have persisted that truncated text
beside its recorded failure and replaced the history it failed to summarize.
The fallback now drops the unusable summary, leaving the typed failure as the
turn's whole outcome.

The empty-summarizer scenarios switch the shared fixture summarizer to blank
output, so the reset moves to `afterEach` and runs even when the helper throws
before its caller's `finally`.
2026-09-13 21:53:29 -04:00
Danny Avila
f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00
Danny Avila
1163045cef
🖋️ feat: Seal Code Environment Choices Per Conversation (#15868)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: lock code environments per conversation

* style: Sort code environment imports

* fix: Harden code environment decisions

* fix: Preserve code environment rollout compatibility

* fix: Harden code environment boundaries

* refactor: Centralize code environment policy

* test: Restore partial disconnect policy mock

* fix: Close code environment edge paths

* fix: Preserve implicit stateful code routes

* fix: Stabilize implicit code routing

* test: Align remote workspace contract

* chore: Sort code route import
2026-09-12 20:45:00 -04:00
Hao
bd08a363c4
🔣 fix: Encode Images From Storage Keys (#15851)
* fix: Prefer storage keys for image encoding

* fix: Sanitize assistant image errors

* fix: Sanitize agent image storage errors

* fix: Move blob image encoding into API package

* fix: Complete image encoder TypeScript migration

* fix: Keep request-boundary error logs diagnosable

getSafeErrorMetadata reduces an error to { type: 'Error' } and an optional
status. That is the right bound where an error may echo submitted content, and
it stays in place at getS3FileStream, getAzureFileStream and
getFirebaseFileStream, where a signed URL reaches the log.

Applied at ResumableAgentController's initialization catch and the chatV1
generic branch it blinded the whole route: a missing key, an agent that fails to
load and a database error all logged as { type: 'Error' }, with no message,
name, code or stack.

getSafeErrorText keeps the error's own description and stack, reduces every URL
to its origin so an object path and signature cannot ride along, and strips
bearer credentials. It returns text so the two handlers log it inside the
message and pass no winston metadata, which is what stops an SDK error's own
enumerable properties (request.url, config, response.headers) from being
promoted onto the record.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-12 12:39:26 -04:00
Danny Avila
d510b6feb6
🛖 fix: Restore Saved Attached Workspaces (#15811)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
* Restore attached workspaces from the resolved owner conversation

* Carry restored conversation through lazy tool initialization

* fix: Resolve Workspace State From Effective Conversation

* style: Format Conversation Regression Cases

* test: Retain Real Workspace Resolver In Controller Fixtures

* test: Exercise Workspace Envelope Conversation Identity
2026-09-10 11:33:11 -04:00
Danny Avila
6cb0499319
🌳 fix: Guard Ambiguous BYOM Workspace Selection (#15785)
* fix: Preserve BYOM workspace selection state

* 🧭 fix: Centralize Workspace Error Serialization

* 🎨 style: Format Workspace Error Helper
2026-09-09 14:27:54 -04:00
Danny Avila
66faf4c27a
🛑 fix: Preserve BYOM Foreground Cancellation Identity (#15777)
* fix: Preserve BYOM foreground cancellation identity

* fix: Bind foreground cancellation to runtime message
2026-09-09 09:22:08 -04:00
Danny Avila
a8087ce454
🏕️ feat: Select Attached Code Workspaces (#15768)
* feat: Select Attached Code Workspaces

* fix: Preserve Workspace Boundaries Across Agent Graphs

* fix: Validate Lazy Agent Workspace Bindings Before Persistence

* fix: Include Handoff Environments in Workspace Selection

* fix: Close Workspace Admission and Remote Ingress Gaps

* fix: Preserve Workspace Selection In Chat Runtime Envelopes

* fix: Authorize Deployment Workspace Status And Match Readiness

* fix: Preserve Ephemeral Chats And Pin Runtime Worker Identity

* test: Complete Deployment Pairing Fixtures
2026-09-09 02:14:50 -04:00
Danny Avila
ab6cde0c79
⏳ fix: Separate Temporary and General Chat Retention (#15734)
* fix: separate temporary and general chat retention

* fix: inherit retention when branching messages

* fix: harden retention lifecycle boundaries

* fix: preserve subagent retention context

* fix: trust stored chat retention for uploads

* perf: overlap generated file retention lookup

* fix: derive retention from trusted chat state

* fix: preserve retention across the full turn lifecycle

* test: accept the conversation retention read adapter

* fix: carry retention deadlines through durable job saves

* fix: resolve retention at cached and new chat admission

* fix: resolve direct tool retention before execution

* fix: preserve retention across assistant writes and direct tools
2026-09-08 11:22:54 -04:00
Dustin Healy
cdc3339a78
🪬 fix: Fence Agent Checkpoint Ownership and Cleanup Recovery (#15680)
* 🔒 fix: Isolate Checkpoint Cleanup by Generation Ownership

* 🔒 fix: Harden Checkpoint Cleanup Ownership

* 🔒 fix: Retain Checkpoint Cleanup Evidence

* 🔒 fix: Persist checkpoint cleanup receipts

* 🔒 fix: Bound checkpoint cleanup receipt lifecycle

* fix: Bind checkpoint cleanup to durable ownership and deletion intent

* fix: Bind event actor checkpoint storage to authenticated owners

* fix: Resolve exact actor checkpoint ownership before lifecycle cleanup

* fix: Preserve actor storage compatibility with durable ownership and prune work

* fix: Defer checkpoint erasure until conversation deletion commits

* fix: Reclaim orphan actor ownership and batch deletion lookups

* fix: bind actor checkpoint ownership to payload rows

* fix: isolate actor readers and await account persistence

* fix: gate owned suspensions and drain complete account state

* fix: unify checkpoint deletion lifecycle and storage authority

* fix: Replay checkpoint deletion from durable owner state

* fix: retain checkpoint cleanup identity before topology loss

* fix: complete checkpoint catalog capture and retirement lifecycle
2026-09-07 22:04:59 -04:00
Danny Avila
3e2a6f333b
🪗 feat: Compact Context in a Summarize-Only Turn (#15690)
* 🗜️ feat: Manual Context Compaction as a Summarize-Only Turn

Adds a user-triggered "Compact context" action that summarizes the active
branch on demand. The compaction is an ordinary agent turn: the client
submits it through the normal chat pipeline with `compact: true`, the
controller hangs it off the branch's leaf instead of creating a user
message, and the agent client runs the graph summarize-only (the SDK's new
`summarizeOnly` input), so the summary streams into the response
placeholder under the leaf exactly like the automatic detour's and is
persisted as the boundary every later turn starts from.

Nothing new is needed for job registration, billing, abort, persistence,
or the usage snapshot: the run emits the same summary step and post-summary
context snapshot the automatic path already emits, and the response message
carries the summary part plus the `summaryUsedTokens` baseline the gauge
reads. The controller validates the request (existing conversation, a real
leaf, summarization enabled, no edit/regenerate/continue combination) and
answers 409 when the leaf is already a bare summary.

Client-side the submission borrows the regenerate shape — no new user
bubble, the placeholder parents onto the leaf — while the wire payload
drops `isRegenerate` so the server never treats it as a regenerated user
turn. The action sits in the context-usage popover, gated by the same
`summarization.enabled` switch as the automatic detour.

* 🗜️ fix: Localize a Compaction That Cannot Run

A compaction the graph could not attempt (summarization disabled, the
instructions already exceeding the budget, or an explicit recency window
covering the whole conversation) rejects with the SDK's
`ManualSummarizationSkippedError`, which rendered as the generic admin
error. It now maps to a typed `compaction_skipped` payload with the reason,
and a run that ends with neither a summary nor a recorded error fails as
`compaction_failed`; the Error component localizes both. A run that already
recorded why it stopped keeps that error part instead of failing twice.

* 🗜️ fix: Anchor a Compaction on the Leaf Everywhere It Is Read

Review round 1.

- The client sent the leaf's parent as `parentMessageId`, which the
  controller treats as the anchor, so the summary would have skipped the
  latest reply and landed as its sibling. The leaf's own id is now both the
  placeholder's parent and the server-side anchor.
- A compaction has no user message of its own, and every consumer of the
  job's user-message slot now knows it: the error path parents its row on
  the loaded anchor and never upserts the leaf, and the job records the
  regenerate shape so a reconnecting client rebuilds the turn under the
  leaf instead of recreating it as a user message.
- The controller no longer reads the anchor before the branch is loaded;
  `BaseClient` validates it against the history it loads anyway and
  answers 404 for a missing leaf or a typed `nothing_to_summarize` for a
  finished compaction. That predicate is shared through the data provider
  and ignores a summary still streaming or one that failed, so an
  interrupted compaction can be retried.
- A compaction never consumes files staged in the composer, the action is
  shown only when the server advertises `compactionEnabled`, and the
  in-flight marker is feature-local Jotai state rather than a read of the
  Recoil submission store.

* 🗜️ fix: Keep a Compaction Turn to Its Summary

Review round 2, plus the class it belongs to. A compaction submits no user
turn, so nothing that keys off one may run: the memory pass no longer runs
over the history (it would have spent a model call and could repeat memory
writes), and skill primes are no longer injected into the transcript the
run is about to summarize.

The client's in-flight marker is also reconciled on mount: a compaction
that finished while the view was away left the marker set, and the next
ordinary turn in that conversation would have shown as compacting.

* 🧹 chore: Sort Imports in the Compaction Hook Spec

* 🔧 chore: Update @librechat/agents dependency to version 3.8.1 in package-lock.json and package.json files
2026-09-07 21:51:54 -04:00
Danny Avila
e370ce4b21
🎒 fix: Bound Agent Attachment Context (#15694)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
2026-09-07 08:16:34 -04:00
Ravi Kumar L
000349681e
🩻 feat(insights): Scope Insights to Authorized Agents (#15549)
Some checks failed
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
* feat(insights): add agent-scoped access

* fix(insights): exclude unattributed assistant messages

* fix(sharing): key role insight toggles by principal

* fix(sharing): preserve insight grant snapshot

* fix(permissions): preserve insight bits atomically

* fix(permissions): guard insight audit rollback

* fix(insights): keep admin grant controls accessible

* fix(permissions): reconcile duplicate principal updates

* fix(insights): show automatic admin access

* fix(permissions): harden insights access updates

* fix(insights): simplify authorization and preserve client state
2026-09-04 20:16:58 -04:00
Danny Avila
8a46ca070c
🧊 feat: Persist Context Fading Tier Across Agent Runs (#15505)
* 🧊 feat: Persist Context Fading Tier Across Agent Runs

Carry the pruner's latched context-fading tier from `@librechat/agents`
through `contextMeta` the way `calibrationRatio` already travels, so a
historical tool result keeps the same truncated bytes from one run to the
next and Anthropic's prefix-based prompt cache survives across turns
(danny-avila/agents#497).

- Add `IAgentFadingTier` to the message and conversation `contextMeta`
  types, schemas and the data-provider zod schema
- Add `isAgentFadingTier`, `resolvePersistableFadingTier` and
  `resolveRunContextMeta` in `packages/api`; a tier is persisted once it
  carries information (masking active or a budget below the window), and
  a latched tier is kept even at a neutral calibration ratio
- Seed `createRun({ fadingTier })` from the parent response's contextMeta
  regardless of encoding, since caps are character-based; the field rides
  the existing forward-compatible `runConfig` variable and older SDK
  versions ignore it, as does `Run.getFadingTier` being absent
- Accept a valid `fading` field in event actor context and reject a
  malformed one, matching the calibration checks
- Capture run context meta through one module-level helper for both
  completion paths instead of two inline copies

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Seed Fading Tier Across HITL Resume, Validate at the DB Layer

Address review findings on fading tier persistence:

- Leave the "is this tier worth persisting" decision to the SDK, whose
  `Run.getFadingTier()` now returns only informative tiers; comparing
  against the client's window misclassified untouched conversations
  whenever a reserve ratio was configured
- Carry `contextMeta` through the HITL pause projection (staged approval,
  job metadata, resume state) and seed the rebuilt client from it, so a
  paused turn resumes with the same tier and calibration instead of
  re-deriving a shallower one and rewriting the prefix
- Share run seeding between `chatCompletion` and `resumeCompletion`, and
  make capture tolerate partial runs and bare resume contexts
- Validate `contextMeta.fading` in `commitAgentEventActorState` with a
  shared `isAgentFadingTier` guard in data-schemas, so a malformed tier is
  rejected at the DB layer instead of silently cold-starting the actor
- Reuse `IAgentEventActorContextMeta` for `IMessage.contextMeta` instead of
  extending an inline duplicate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 test: Cover Resume Seeding From Paused Context Meta

Assert the resume controller seeds the rebuilt client from the context meta
captured at the pause before rebuilding the run, and move the capture
helper's JSDoc back above its function.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧹 style: Sort Imports in Fading Tier Plumbing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Persist Fading Tier Through Redis Jobs and Branches

Address Codex and Copilot review findings on fading tier persistence:

- Deserialize `contextMeta` in `RedisJobStore`, whose explicit read mapper
  rebuilt every other pause field but this one, so Redis-backed
  deployments resumed with no tier to seed after every HITL pause
- Copy server-private `contextMeta` onto the assistant message created by
  `POST /api/messages/branch`, so the next turn seeds its pruner the same
  way it would from the parallel source response
- Constrain the `fading` subdocuments in both Mongoose schemas (version
  enum, positive budget, required fields) to match the zod schema
- Throw a distinct "fading tier is invalid" error from both context meta
  validators instead of reporting a calibration failure

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through Stop and Re-Pause

Addresses the second Codex pass on the fading-tier persistence:

- A re-pause whose resumed segment has nothing persistable now clears the
  job's `contextMeta` in the same transition instead of leaving the first
  pause's calibration and tier for the next resume to seed from.
- The client publishes the run's live calibration and fading tier onto the
  job after each pre-invoke context snapshot (deduplicated on value), and
  the Stop path copies `jobData.contextMeta` onto the stopped response, so
  a follow-up from a stopped turn keeps the prompt prefix stable.
- Mid-run captures (HITL pause, Stop) read the graph's live state; `Run`
  refreshes its own getters only after the stream settles, so the pause
  used to persist the seeds it started from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Fence Context Meta Publication Ahead of Stop

A Stop handled by another replica reads the job before the owner is
signaled, so a fire-and-forget publish could lose the race, and a run
stopped before its first context snapshot never published the tier it
inherited. The client now publishes the inherited seed before the run
streams or resumes, and the context-usage handler awaits each
snapshot's publish so the write lands before the model call it
describes begins. Failures still only log; the pre-run publish is
optional-called so bare resume contexts keep working.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Read Back Context Meta After the Abort Claim

The run owner publishes its context meta ahead of each model call and
awaits the write, so a publish landing between the abort's initial job
read and its terminal claim describes the call whose partial output the
abort snapshot carries. abortJob now reads the same-epoch job back after
the claim and content re-read and takes its context meta, so the stopped
response persists the tier that produced its bytes. The refresh is
best-effort and logs on failure like the content refresh beside it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Share In-Flight Context Meta Writes and Clear Neutral Live State

Two snapshot callbacks in a parallel-agent run can capture the same tier
before the first write settles; the publisher now caches the in-flight
promise per serialized value so the second caller awaits the same
durable write instead of treating an uncommitted value as published.
A live snapshot with nothing left to persist after an earlier publish now
writes a neutral record (ratio 1, no tier), since a running job's fields
cannot be deleted through the metadata writer, so a Stop no longer
persists an earlier non-neutral state onto the response. The pre-run
seed publish is unchanged and is told apart from live snapshots by a
flag from the context-usage sink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 chore: Drop Unused Rest Parameter in Context Meta Test

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 feat: Persist Per-Agent Fading Tiers as Compact Context Meta

Reconciles the host with @librechat/agents at 9aacad1a, where graph
history stays canonical and each Run derives a provider-only projection
from the latched tiers. LibreChat therefore persists only compact state:
the calibration ratio, the default agent's tier, and now the per-agent
tiers from Run.getFadingTiers(), stored as validated entries so agent IDs
never become MongoDB field names and restored onto a null-prototype
record for RunConfig.fadingTiers. Truncated messages, canonical tool
content, projection provenance and per-message truncation state are
never persisted; the SDK's latched flag is stripped on capture.

The per-agent map travels everywhere the single tier already did:
response contextMeta, event-actor checkpoint state, HITL pause and
resume job metadata in both stores, the Stop path, and branch creation.
A shared Mongoose definition keeps the message and conversation schemas
identical, the zod schema validates entries, and commitAgentEventActorState
rejects malformed maps. Comments no longer describe graph messages as
truncated; only the provider projection is.

Tests cover: tool results and inputs reach the persisted content parts
at full size; contextMeta carries exactly the compact tier and
calibration fields; a persisted snapshot round-trips into the seeded
RunConfig unchanged and prototype-safe; per-agent tiers survive pause,
Redis job round-trips, and event-actor context validation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Restore Event Actor Types and Publish Seeds Before Abortable Setup

The previous commit's doc rewording in the conversation types dropped the
event-actor interfaces that precede the fading tier, which broke the
declaration bundle in CI; they are restored unchanged.

The inherited context meta is now published onto the job as soon as the
parent's state is loaded and awaited at the top of chatCompletion and
resumeCompletion, ahead of run creation and the other abortable setup
stages, so a Stop during setup still persists the parent's tiers onto the
stopped response. The partial response saved when every subscriber
disconnects now carries the job's context meta like the Stop and pause
paths do. The branch route keeps the source's context meta in the saved
message but strips it from the HTTP response, matching client reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Serialize Distinct Context Meta Publications

Overlapping snapshots from parallel agents could carry different tiers;
each started its own write and both job stores keep whichever write
finished last, so an older snapshot could overwrite a newer one. Each
distinct publication is now issued after the previous one settles, in
call order, so the newest snapshot is what a Stop reads back. Equal
values still share one write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Project Root Message Reads and Keep Inherited Meta on Setup Failure

The root GET /api/messages paths returned raw documents: the
single-message query read called getMessages without a projection and the
cursor page performed an unprojected find. Both now apply
CLIENT_MESSAGE_SELECT, with getMessagesByCursor accepting a select option,
so the persisted fading tiers never reach the client through the
paginated endpoint; the search-hydration read projects the same way.

A turn that loads a parent's context meta but fails before its run exists
used to replace the inherited meta with the capture of a missing run, so
the persisted error response lost the last valid tiers. Both finalizers
now keep the inherited meta when no run was created while a created run's
neutral state may still clear it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 refactor: Move Context Meta Publication Into packages/api

The publication coordinator (ordering, deduplication, in-flight sharing,
failure state) now lives in packages/api as createContextMetaPublisher,
with selectRunContextMetaToPublish deciding what a publication carries;
the legacy client keeps only the wiring to the job metadata writer. A
transiently failing write is retried with a short backoff before the
publication is reported and forgotten, so a snapshot's model call no
longer proceeds on the first rejection while the job holds the previous
record. The SDK swallows handler errors, so retrying is the only way to
strengthen the fence without failing the turn.

A terminal save now supplies contextMeta: null when the finished run has
nothing to carry, and saveMessage unsets a previously stored value in
that case, so a disconnect snapshot's record cannot outlive the run that
completed neutrally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Widen the saveMessage Method Type for the Null Unset

The exported method interface still typed contextMeta as the message
field, so the spec exercising the null unset failed typechecking; it now
mirrors the implementation signature.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Publish Every Agent's Snapshot, Sanitize Search Hits, Unbound Agent IDs

The context-usage handler now awaits the run's context meta publish for
every ON_CONTEXT_USAGE event, hidden sequential agents included, while
the visible-snapshot bookkeeping and the client emission keep their gate;
a Stop before the next visible snapshot finds the tier a hidden agent's
model call latched. Search results are built from the hydrated hit minus
its server-private contextMeta, since Meilisearch hydration projects every
schema field. The per-agent tier entry guard no longer caps the agent ID
length: ephemeral agent IDs encode endpoint, model and sender without a
bound, and dropping such an entry would let that agent alone re-derive
its tier on the next turn. Also formats the publication spec so the
static checks pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Unset Context Meta Atomically, Keep Publication State After a Failed Write

- `saveMessage` folds the `contextMeta` unset into the same update that
  persists the terminal response, on both the plain and the provenance-merge
  paths, so a failure between two writes can no longer leave a completed row
  carrying a disconnect snapshot's state.
- `createContextMetaPublisher` tracks whether any record has committed
  separately from the retryable latest publication, so `hasPublished` stays
  true after a later publication exhausts its retries and a following neutral
  snapshot still overwrites the earlier record on the job.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through the Resumable Stop Route and Neutral Resumed Completions

- The resumable Stop route copies the refreshed `jobData.contextMeta` onto the
  terminal assistant message it persists, as the abort middleware already
  does, so a stopped resumable response keeps the tiers that produced its
  partial output.
- A resumed run that completes with neutral context state now saves
  `contextMeta: null`, which unsets the paused segment's record instead of
  letting an omitted field keep it on the completed row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Trust Only Server-Authored Context Meta, Unset It on a Neutral Stop, Read It From the Job on Disconnect

- `POST /api/messages/:conversationId` and conversation imports drop any
  client-supplied `contextMeta`, and the run seeds its calibration and fading
  tiers only from a server-authored parent response, so a forged tier can never
  shape a provider projection.
- Both Stop paths save `contextMeta: jobData.contextMeta ?? null`, so a job that
  re-paused with neutral state unsets what an earlier pause stored on the row
  instead of leaving it behind.
- The disconnect partial save reads the same-epoch job record for the run's
  published context meta, since the client-facing resume snapshot never carries
  server-private state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧪 test: Mock the Job Store in the Resume Metadata Spec

The disconnect partial save now reads the same-epoch job record through
`GenerationJobManager.getJobStore()`, so the resume metadata spec's manager
mock provides it as the other disconnect spec already does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Keep Context Meta Off the POST Message Response

`POST /api/messages/:conversationId` returned the saved row verbatim, so a
client write against an existing server-authored message echoed the row's
stored context meta. The route now projects the row through the same client
projection the branch route uses, which drops the server-private field.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 📦 chore: Bump @librechat/agents to 3.7.16 for Persisted Fading Tiers

3.7.16 is the first release that carries danny-avila/agents#497, so
`Run.getFadingTier()` / `getFadingTiers()` now report the latched tier
this branch persists and `RunConfig.fadingTier` / `fadingTiers` seed the
next run from it. The dependency set is unchanged from 3.7.15; only the
package entry moves.

Adds an SDK-backed round trip to the fading spec: a first pruner latches
an informative tier, `resolveRunContextMeta` reduces it to the persisted
shape, and a second pruner seeded from that shape reproduces the first
run's projected tool-exchange bytes under instruction and calibration
drift, while an unseeded pruner under the same drift relaxes its tier and
rewrites them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-02 20:29:43 -04:00
Marco Beretta
2cf1007f3c
🧺 feat: Preserve Partial Work at Tool Call Limits (#15486)
* feat: Handle reaching the agent tool-call limit gracefully

When the graph exhausts its per-turn step budget, persist the turn as
unfinished with finish_reason tool_call_limit instead of a red error.
Warn the model as the budget runs low, and offer Keep going or Answer now.

* docs: Add screenshots of the tool-call-limit notice

Light and dark captures of the unfinished-turn card for the PR description.

* style: Tighten the tool-call-limit notice card

Pin dismiss to the top-right with uniform rounding, use a slightly
smaller hit target, and round the card to xl. Refresh the PR screenshots.

* fix: Address Copilot review findings on the step-budget path

Skip empty and subagent-scoped PostToolBatch events when counting
rounds. Drop the double-cast in the notice test helper and coerce the
tool-call-limit finish reason in the Continue-gate spec.

* fix: Address Codex review findings on budget and dismiss

Pass the resolved recursion limit into the OpenAI-compatible
processStream path so the step-budget hook matches the graph.
Reset the notice dismiss state when the displayed message id changes.

* fix: Unblock CI typecheck and import sorting

Declare recursion_limit on the OpenAI service agent type, sort that
file's local imports, and route the notice test context through unknown
so tsc accepts the partial ChatContext fixture.

* fix: Sort remaining PR imports for static checks

* chore: Remove tool-call-limit notice screenshots

* fix: Keep recovery actions from consuming composer drafts

Pass empty file, skill, and quote overrides on Keep going and Answer
now so a staged compose is not attached or drained.
2026-09-01 21:38:02 -04:00
Danny Avila
b7ae67307e
🕰️ fix: Anchor Temporal Variables to Turn Start (#15419) 2026-08-31 10:38:10 -04:00
Danny Avila
c7f0a8f247
📬 feat: Move Agent Queued Turns Server-Side (#15370)
* feat: move agent queued turns server-side

* test: update queued turn composition coverage

* fix: harden queued-turn lifecycle ownership

* fix: centralize queued-turn lifecycle ownership

* fix: close queued-turn lifecycle gaps

* fix: unify queued turn lifecycle transitions

* fix: fence queued-turn deletion admission

* fix: preserve queued-turn lifecycle ownership

* fix: close queued-turn reconciliation races

* fix: reconcile quarantined queued-turn admissions

* fix: annotate admission evidence default

* fix: deepen queued turn lifecycle admission

* fix: fence queued turn provider admission
2026-08-30 21:53:58 -04:00
Danny Avila
773127bff2
🎠 refactor: Route Every Event Actor Turn Through One Lifecycle (#15325)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* refactor: unify Event Actor turn lifecycle

* fix: retain Event Actor fence ownership

* fix: preserve mixed-version actor suspension safety
2026-08-28 17:30:23 -04:00
Danny Avila
3fa33b740b
🛫 refactor: Promote Generation Protocol V2 Automatically (#15324)
* refactor: promote generation protocol v2 automatically

* fix: remove unused protocol import
2026-08-28 17:17:09 -04:00
Danny Avila
77c2a51cf3
🪧 fix: Advertise Detached Event Actor Support from the Generation Store (#15322)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
* fix: activate detached event actions automatically

* fix: shield detached terminal generations

* perf: share capability availability index

* style: flatten capability status selection

* fix: preserve mixed-version lifecycle shells

* fix: complete detached action store adapters

* fix: close mixed-version capability races

* fix: honor legacy capability success
2026-08-28 16:33:23 -04:00
Danny Avila
487984193a
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable

* fix: align detached actor checks with merged base

* fix: close detached actor lifecycle gaps

* fix: break detached action outcome cycle

* fix: deepen detached action ownership

* fix: fence detached launch handoffs

* fix: preserve detached rollout fencing

* test: type detached recovery failure

* fix: close detached ownership handoffs

* fix: stage detached action activation

* fix: persist detached terminal retries

* fix: fence detached durability ownership

* fix: fence trigger lane publication

* fix: fence detached transition generations

* fix: retire failed detached predecessors

* fix: close detached rollout consumers

* test: type detached retry turns

* fix: close detached ownership gaps

* fix: fence detached action resumes

* fix: fence prior-head event actor resumes

* fix: preserve legacy event actor resumes

* fix: align detached resume boundaries
2026-08-28 14:21:04 -04:00
Danny Avila
c06b09c945
🤖 feat: make Event Actor HITL durable (#15305)
* 🤖 feat: make event actor HITL durable

* 🤖 fix: break event actor outcome cycle

* 🤖 fix: close durable actor terminal races

* fix: harden durable event actor recovery proofs

* fix: close event actor resume publication races
2026-08-28 08:05:11 -04:00
Danny Avila
6fba51a35a
🫆 feat: Fingerprint Agent Context for Zero-Read Warm Continuation (#15299)
* feat: add context-fingerprinted warm continuation

* fix: preserve manual skills across warm turns

* perf: keep ordinary agent initialization neutral

* fix: fingerprint complete agent topology

* fix: close warm context replay gaps

* fix: preserve lazy subagent memory scope

* perf: fingerprint only model-bound memory

* fix: project lazy memory capability

* style: sort warm continuation imports

* fix: make warm continuation reconstructable

* fix: preserve lazy skill isolation

* fix: retain scoped skill semantics
2026-08-27 23:01:14 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Danny Avila
f4efef4b3a
♟️ fix: Settle First Bound Actor Events (#15300) 2026-08-27 19:06:41 -04:00
Danny Avila
41ba808e6f
🧅 feat: Reveal Subagent Thread History and Event Detail on Demand (#15283)
* ✨ feat: Refine Subagent Thread Activity UX

* test: align subagent completion lifecycle

* chore: satisfy subagent panel static checks

* fix: harden subagent activity pagination

* fix: preserve paged subagent history

* fix: surface unrecoverable activity boundaries

* fix: rebase paged activity after live advances

* fix: bound child activity history state

* fix: preserve rebased child history order

* fix: retain unavailable child history boundary

* fix: fence child history cursor generations

* fix: preserve subagent history continuity

* fix: retain live turns during history rebase

* fix: close subagent history edge cases

* fix: harden subagent history phase transitions
2026-08-27 08:56:06 -04:00
Danny Avila
44edcbe014
🧾 feat: Store Durable Event Actor Receipts (#15265)
* 🧾 feat: Store durable event actor receipts

* 📊 fix: Scope actor delivery metrics

* 🧱 fix: Close durable receipt recovery gaps

* fix: serialize actor action admission

* fix: retire terminal batch members

* fix: close actor terminal recovery races

* fix: reclaim lanes after terminal receipts

* fix: close actor receipt recovery gaps

* fix: preserve ambiguous legacy handling

* fix: recover legacy actor outcomes

* fix: admit leased actor actions

* fix: serialize actor admission recovery

* fix: close actor recovery races

* fix: token-fence actor admissions

* fix: preserve mixed-version actor fences

* fix: harden actor rollout and metrics

* fix: gate durable actor receipt rollout

* fix: enforce base actor receipt rollout

* fix: align experimental actor rollout
2026-08-27 06:00:43 -04:00
Danny Avila
68fc46a055
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks

* fix: fence event actor checkpoint uncertainty

* fix: satisfy event actor type contracts

* fix: make actor reconciliation recoverable

* fix: preserve event actor lifecycle transitions

* fix: fence event actor lifecycle outcomes

* fix: enforce event actor lifecycle ownership

* fix: retain event actor settlement proof

* 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal

Repair and compensation deleted the reconciliation row they resolved, which
was the only durable proof that the invocation had already applied an external
action. A delayed duplicate owner could then reacquire the same invocation id
and repeat that action. Both resolutions now retire their receipt to `settled`
and record how it settled, so the same-id tombstone survives; `history_repaired`
and `action_compensated` still force a cold rebuild. Compensation undoes the
effect without re-authorizing the delivery, so a legitimate retry must arrive
under a new invocation id. A retried repair converges on its own receipt.

Bound the journal so a long-lived actor cannot grow its conversation document
without limit: a new fence is admitted only when no active lifecycle row
exists, so a capped push can evict nothing but the oldest settled receipts.

Stop shipping the unbounded source payload on every bound-child continuation.
It rode the delivery body regardless of the feature flag while the sibling
`fire` body deliberately sends event identity alone, so a large webhook payload
could push a previously working delivery past the chat route's body limit. The
actor binds an invocation from identity and never builds the prompt from it.

Blank the positional token map on warm continuations. It is derived from the
full DB history, while a warm run executes on checkpoint-restored state, so its
indices address different messages and the pruner never recounts them —
misattributing cached counts to the wrong messages in both directions.

Keep the replaced-claim exit on its cleanup path when preserving reconciliation
fails: the committing CAS already left a blocking row, so the failed status
upgrade costs provenance, not safety.

* 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry

Give the warm-continuation client test a populated token map and a real
cross-run summary so its assertions bite: the positional map must arrive
blank (checkpoint-restored state no longer matches DB-derived indices, and
the pruner never recounts a populated entry) while `initialSummary` must
pass through unchanged — it rides the system tail and summarizes
pre-boundary turns that were excluded from the very history the committed
checkpoint was built from, so blanking it would silently drop context no
warm run can recover.

* ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal

A compensated receipt still tombstones its invocation id, but its external
effect was explicitly undone — the terminal handler nonetheless replayed
every settled lifecycle's stored action as authoritative and settled the
public outcome as applied, telling an action-aware source the operation
stands and suppressing the new-invocation retry compensation requires. The
handler now settles a compensated invocation as failed with an explicit
compensation error, overriding even fresh applied run evidence from a
replayed generation; verified and repaired receipts continue to replay
applied.

Receipt eviction is now primarily age-based: a stale same-id owner is
bounded by time, not by how many newer invocations settle, so the previous
count-only slice let a high-rate actor evict a tombstone while its delayed
duplicate owner could still wake and repeat the action. Admission prunes
only settled receipts older than a retention window that dwarfs every
generation, job, and delivery-retry lifetime, and the count cap is demoted
to a raised document-size backstop.

* 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts

The terminal handler read its lifecycle snapshot, verified history, and then
settled the public outcome — so a compensation resolving the same receipt
during that window lost: the handler settled applied from its stale snapshot
and no retry could ever change the replay-identity-locked outcome. The
receipt's status CAS is now the serialization point: verification resolves
the receipt BEFORE settling, whichever transition wins determines the public
outcome, and a crash between resolve and settle converges through the
retained receipt's replay. The verified-replay probe requires the receipt's
own resolution, so a compensated receipt can never satisfy a verification
retry. This inverts the settle-before-receipt ordering deliberately: that
ordering guarded proof that resolution used to delete, and the receipt now
retains its full action proof through resolution.

The document-size cap is no longer an eviction quota. A receipt inside its
retention window is never discarded: when the journal holds a full cap of
unexpired receipts, new invocations are refused fail-closed until receipts
age out, making duplicate protection and document integrity simultaneous
invariants instead of a rate-dependent trade.

* 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path

Skill primes are spliced into the message list directly ahead of the newest
message, and a warm continuation forwards only that newest message — so a
checkpoint-restored actor would keep serving the prime bodies baked in at
its last cold start and never observe an edited or newly attached skill.
Until the actor head carries a context fingerprint that forces a cold
rebuild when the agent's skill context changes, agents with always-apply or
manual skill primes stay on the legacy path, which re-primes fresh bodies
every turn: correct on every event, just never warm.

* 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes

History-derived re-priming was a third path into the same staleness class:
an actor that previously invoked a skill carries no request-time prime
arrays, yet primeInvokedSkills re-resolves that skill's current body from
history each turn and the warm slice drops the reconstruction — leaving the
checkpoint's old body active after edits. The fork gate now keys on the
priming hook itself (present exactly when the skills capability is enabled)
alongside the request-time arrays, so every skill-body path routes to the
legacy rebuild until #15235's context fingerprint restores warm
continuation for skill-bearing actors.

* 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage

The executor read applied-action evidence from the run-step collection the
instant sendMessage resolved, but that collection is populated
asynchronously — an applied invocation could classify as actionless
(runSteps still empty while the tool result already streamed), discarding
its fork and stranding the actor cold while the terminal handler later
settled the same delivery as applied from the persisted evidence.

Authoritative proof is now recorded in graph context the moment the
expected tool executes: the request-owned recorder observes the tool-end
chain (which ToolNode dispatches synchronously with both input and output)
and applies the same fences as run-step evidence — exact tool name with the
MCP-suffixed form, the declared argument subset against the execution
input, an error-free result, and the background non-execution receipt
exclusion. readAppliedAction consults the receipt first; run-step
inspection remains the fallback for paths that bypass the tool-end chain.

Regression coverage reproduces the observed ordering: the real executor
commits the head from the receipt while run steps are empty, warm-continues
the next event, and never re-executes the action; recorder fences and the
receipt-first controller wiring are covered separately.

* 🎯 fix: Supply Execution Arguments to the Tool End Callback

The live Vertex + MCP canary exposed a contract mismatch the synthetic
fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end
callback with output only, while the action recorder must verify the
declared argument subset against the execution input. The receipt never
qualified, every turn fell back to cold history rebuilds, and the
tournament advanced with zero actor heads and zero retained checkpoints
while looking successful.

The execution handler owns both halves at the same moment, so the fix is
at the source rather than a correlation store: ToolEndCallbackData gains
the executed call's input and every callback site passes tc.args. A
handler-level regression drives the real createToolExecuteHandler and
asserts the callback receives both fields; recorder regressions pin the
production shapes — an output-only tool end must starve an
argument-fenced receipt rather than trust an unfenced match, and still
qualifies a name-only expected action.

* 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions

The background-claim callback reports the ORIGINAL tool's name for artifact
attribution on the poll turn that harvests a completed task. A name-only
expected action could therefore be impersonated by work some earlier turn
dispatched: the recorder would attribute that delivery to the current
invocation and commit a head whose state never contained the invocation's
own action. The run-step evidence path never had this hole — it sees the
poll tool's name — so the recorder must match its provenance discipline.

Delivery callbacks now carry an explicit backgroundDelivery marker set at
the one site that rewrites the name, and the recorder ignores marked
deliveries outright. Regressions pin both halves of the contract: the
delivery callback must carry the marker with the poll call's arguments,
and a marked delivery can never qualify even a name-only expected action.

* 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions

Three closeout-round findings, each converted into an invariant.

Every legacy-path invalidation now advances a durable epoch — including for
headless and already cold-marked actors, where the marker alone leaves no
CAS-visible trace — and the actor-head CAS requires the epoch observed at
preparation. A concurrently prepared fork whose history predates an
intervening legacy turn can no longer commit past it; the commit reports an
ordinary conflict and journals.

Execution identity is now emitted before post-execution output policy: when
a side-effecting tool succeeds but its returned content is withheld by the
output filter, the callback delivers an outputFiltered receipt with blank
content — the recorder accepts it as proof (rejecting model-detached calls
it cannot distinguish through the blank shape), the artifact path never
sees it, and an applied action is no longer reclassified actionless and
re-executed on retry.

Background-capable expected actions stay off the fork path: dispatch
returns a launch handle every evidence fence correctly rejects, and the
completion is provenance-marked as another turn's work, so a fork would
settle actionless before the external effect lands with nothing to stop a
retry from dispatching it again. The gate mirrors the MCP-suffix name
matching of the evidence path.

* 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance

The epoch fenced only the legacy turn's start: a fork preparing after the
begin invalidation but before the turn's message persistence observed the
new epoch and cold marker, rebuilt from history that did not yet contain
the turn, and committed cleanly because nothing advanced the epoch again —
making the incomplete rebuild authoritative and clearing the marker.

Every legacy event turn now seals its invalidation at terminal persistence
with a second epoch advance, on the success, replaced-claim, and error
exits alike. Sealing deliberately carries no quiescence requirement — it
must succeed while a fork fence is active, because that is exactly the
mid-turn race it defeats — and a fork that already committed against the
begin epoch is healed the same way: the seal re-marks the head cold, so
the next event rebuilds with complete history. Seal failure never diverts
the turn's own exit; the begin bump still fences everything prepared
before the turn.

* 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence

The second epoch advance could not make a legacy turn atomic, and three
findings shared that root cause: two conditional updates left a headless
gap a fork could create the head inside; the error exit sealed before
saveErrorTurn made the error history durable; and any crash or failure
between persistence and sealing left an incomplete fork authoritative,
because the seal was best-effort and its failure was swallowed.

A legacy turn now carries one durable fence. A token is written before
execution by a single update-pipeline write — no two-write gap, and the
cold marker is applied only where a head exists via $cond/$$REMOVE. While
the token is present no fork may prepare (the adapter refuses) or commit
(the CAS requires its absence), because the turn's messages are not yet
durable. One atomic write clears the exact token and advances the epoch
once history is persisted — after saveErrorTurn on the error exit — and
success, replacement, and error exits all route through it.

Failure is now fail-closed rather than silent: a failed seal leaves the
token set, which keeps blocking forks and is logged as such, and a fence
abandoned by a crashed turn is reclaimed only once stale, advancing the
epoch and marking any head cold so the next event rebuilds from whatever
history actually survived.

* fix: serialize legacy event actor turns

* fix: close legacy actor fence ownership gaps

* fix: preserve legacy actor persistence fences
2026-08-26 08:39:00 -04:00
Danny Avila
290b8664d9
🧾 feat: Track Authoritative Agent Event Outcomes (#15213)
* feat: track authoritative agent event outcomes

* fix: isolate agent event outcome types

* fix: declare agent event handler result

* fix: simplify agent event status selection

* fix: preserve authoritative event outcomes

* fix: preserve terminal event evidence

* test: use completed run-step envelope

* test: scope deferred HITL question locator

* fix: settle every agent event terminal path

* style: sort terminal host action imports

* fix: fence agent event terminal evidence

* fix: recover agent event terminal settlement

* fix: scope terminal retry hints by generation

* fix: settle terminal host actions exactly
2026-08-25 18:20:57 -04:00
Danny Avila
862ebf3235
🪢 fix: Persist Failed Agent Turns Before Error Publication (#14118) 2026-08-25 06:50:46 -04:00
Danny Avila
ac2aef00f6
🫗 fix: Drain Quoted Excerpts Into Mid-Run Steering (#15175)
* 🧭 fix: Carry Quoted Excerpts Through Mid-Run Steering

"Add to chat" quote chips were dropped by every during-run steer path: the
steer POST had no quotes concept, so a composer-origin steer left the chip
staged (gluing onto the NEXT send) and a queued item steered into the live
run lost its quotes silently.

Quotes now ride the steer protocol end to end:
- POST + admission: `quotes` on the steer body, normalized like the chat
  route's (getReferencedQuotes caps), part of the idempotency fingerprint
  only when present so pre-existing receipts still replay.
- Injection: merged into the model-bound turn as Markdown blockquotes at
  both boundaries (text-only and media paths), mirroring prependQuotes.
- Persistence + replay: the STEER content part stores `quotes` separately
  from the typed text; stampSteerPartMedia re-merges them per turn (even
  with resendFiles off) via the SDK's transient media stamp, with the quote
  block folded into the token budget.
- UI: composer steers/interrupt-steers drain the chips (skill picks stay
  staged — they configure a NEW turn's run); SteerPart and the in-flight
  bubble render the same MessageQuotes reference blocks as user bubbles;
  queued/failed rows show a quote count; reconnect reseeds fall back to the
  server item's quotes when no local chip survives.
- buildMessages keeps its zero-await path to the parallel context kickoff
  via a synchronous stamp-target probe.

* 🧭 fix: Keep Quotes in the Client-Safe Steer Projection

toPendingSteer is the projection behind resume-state pendingSteers, abort
responses, and terminal leftover claims — dropping quotes there would lose
them on exactly the recovery paths the reconnect reseed's server fallback
relies on.

* 🧪 test: In-Flight Steer Bubble Renders Carried Quotes

* 🔁 fix: Re-Stage Quotes When a Pre-Quotes Replica Accepts the Steer

Codex flagged the rolling-deploy window: an old replica 202s a quoted steer
while dropping the excerpts, so the client cleared the chips for context the
model never received.

The 202 (fresh and receipt replay) now echoes quotesAccepted from the
DURABLE item; a missing echo on a quote-bearing composer-origin steer
re-stages the excerpts as composer chips — the pre-steer behavior, so they
ride the next send instead of vanishing — and strips them from the surviving
chip so a later terminal conversion cannot duplicate them. Queued-origin
steers keep quotes on the item, whose restore paths already return it
intact. The residual cross-version lost-ACK retry stays fail-closed as a
409 idempotency conflict (failed chip with retry controls).

* 🔁 fix: Close the Remaining Cross-Version Quote-Loss Windows

Codex round 2:
- Send now of a quoted queued item against a pre-quotes replica now
  re-stages the excerpts too (the row is consumed and the words inject
  bare, so the composer is their only remaining home); the strip clears the
  chip's captured origin copy so reclaims and terminal conversions cannot
  duplicate them.
- A quoted retry whose lost first ACK was accepted by a pre-quotes replica
  now REPLAYS that legacy receipt instead of 409ing: the stored fingerprint
  matching the quote-less hash of the same words proves the cross-version
  case, and the replayed 202's missing echo drives the re-stage. Different
  quotes against a quote-bearing receipt still conflict.
- TSteerAppliedEvent.part gains the quotes field (typed SSE consumers).

* 🧪 test: Drop the Stale Narrow SteerDrainOutput Alias

The spec's local intersection re-declared injectedMessages with
content: string, predating the SDK pin that declares the field natively
(content: string | MessageContentComplex[]). Under CI's clean install the
hook's BaseHookOutput is no longer assignable to that narrower alias; the
plain PostToolBatchHookOutput is the correct type for every drain/boundary
assertion. Verified against the published 3.6.16 dist and the local one.

* 🔁 fix: Honor the Generation Owner's Quote Capability End to End

Codex round 4:
- steerQuotesCapable rides job metadata (createJob + HITL resume rewrite),
  mirroring preemptCapable's owner-recorded pattern: an upgraded admission
  replica no longer stores quotes — or claims them accepted — for a
  generation whose older owning drain would silently drop them at
  injection. The missing echo drives the client re-stage, and a later
  capable handover cannot double-deliver restored context.
- Applied events reconcile dropped quotes: when a quote-less applied part
  settles a quote-bearing chip (the lost-202 ordering the ACK-echo path
  cannot see), resolveSteerChip and both reconnect settle paths re-stage
  the chip's excerpts before removing their only copy. mergeRestagedQuotes
  dedupe keeps every trigger idempotent for the same excerpts.

* 🔁 fix: Re-Read Quote Capability at the Last Moment and Cap Restaged Chips

Codex round 5:
- A HITL resume rewrites steerQuotesCapable without changing the
  generation's createdAt, so the enqueue fence cannot see a
  capable-to-legacy handover landing during admission's awaits. Re-read
  the owner's flag immediately before item construction (paid for only by
  quote-bearing requests); the residual between re-read and enqueue commit
  matches preemptCapable's documented race.
- mergeRestagedQuotes now respects the 10-quote contract with the staged
  chips winning: a restored tail that cannot ride the next send is dropped
  explicitly instead of rendering as a chip the submission would silently
  discard. MAX_QUOTE_COUNT moves to utils/steer as the single client
  source; QuoteButton imports it.

* 🔁 fix: Steer Quote Coverage for Preflights, Memory, and Single-Scan Stamping

Codex round 6:
- Stored-message policy inspection now extracts steer-part quotes as quote
  fragments (path /content/N/quotes/M), so conversation import and shared
  link preflights inspect the newly persisted field exactly like top-level
  message.quotes.
- The memory copy gets its own quote-merge stamp (text only, resendFiles
  false): formatAgentMessages ignores part.quotes, so without it a steer
  whose substance lives in its excerpt reached the chat model but never
  memory extraction.
- collectSteerStampTargets replaces the boolean probe: buildMessages
  collects once and hands the targets to stampSteerPartMedia, keeping the
  zero-await fast path without scanning the history twice.

* 🔁 fix: Redis Quote Plumbing, Conversion-Race Guard, and Quote-Bound Recovery Proof

Codex round 7:
- RedisJobStore.deserializeJob now restores steerQuotesCapable (the explicit
  mapper otherwise dropped it on every read, leaving quote steering inert in
  Redis deployments), with the round-trip spec extended.
- Both Lua parked-steer projections (terminal close + generation
  replacement) forward item.quotes, matching toPendingSteer — a lost final
  no longer strips excerpts from durable recovery in Redis mode.
- The no-echo restage reads the SURVIVING chip (reclaimRejectedChipQuotes):
  a terminal conversion that beat the delayed 202 already moved the quotes
  onto the queued follow-up, and re-staging them again double-delivered.
  Regression-tested with the conversion-before-ACK ordering.
- RecoveredSteerPayload binds normalized, order-significant quotes (builder,
  validator, TS matcher, and the Lua decode+matcher): a stale client
  presenting the same recoverySteerId with altered or missing quotes cannot
  consume the parked source. Quote-less sources keep matching quote-less
  recoveries.

* 🔁 fix: Execution-Bound Quote Capability with an Atomic Enqueue Predicate

Codex round 8:
- steerQuotesCapable becomes a transient assertion translated (at createJob
  and in ApprovalLifecycle.resolve) into steerQuotesExecutionId, valid only
  while it equals the LIVE providerExecutionId. A legacy replica winning a
  HITL resume rewrites the execution id without knowing the marker, so its
  stale assertion self-invalidates — a bare boolean could not be cleared by
  code that predates it.
- The fenced enqueue evaluates that equality atomically (all three Redis
  scripts decode-and-strip like the existing preemptCapable normalization;
  both InMemory sites mirror it) and returns the persisted item, so the
  quotesAccepted echo reflects exactly what was stored even when a handover
  lands between admission's read and the commit. The last-moment re-read is
  gone — the transaction is the authority.
- Tests: capable-resume re-binding, legacy-resume omit-not-clear
  invalidation, the admission-vs-handover race (capability read true, then
  execution rewritten before enqueue), and the Redis round-trip of the
  marker.

* 🔁 fix: Full Redis Parking Coverage and Loss-Moment Quote Restaging

Codex round 9:
- The two remaining Redis parking projections (terminal status CAS and
  stale-running cleanup) forward item.quotes — every field-picked steer
  projection now carries them (audited: 2 Lua 'projected' + 2 Lua
  'clientItem' + toPendingSteer).
- The ordinary no-echo ACK no longer re-stages: the steer has not injected
  yet, so the quotes stay carried on the pending chip. A quote-less applied
  event re-stages them at the actual loss; a terminal leftover conversion
  carries them onto the recovered row, whose normal send delivers quotes on
  any server — re-staging at the ACK let that leftover auto-send bare text
  while the excerpts glued onto an unrelated draft. Only the settled
  receipt replay (already injected, no future event) reclaims immediately.

* 🔁 fix: Legacy-Replayable Receipts with Separate Quote Identity

Codex round 10: an upgraded-first receipt stored a quote-inclusive
fingerprint no pre-quotes replica could recompute, so a lost-ACK retry
routed through one 409'd already-accepted words with duplicate-send
controls.

The durable fingerprint reverts to the quote-independent 3-field hash —
the one shape EVERY deployed version computes, replayable across a rolling
deploy in both directions — and quote identity moves beside it as
requestedQuotesFingerprint (of the REQUESTED quotes, pre any capability
strip, so an incapable-owner acceptance still replays its own retries).
Absent records (legacy-written or quote-less) accept any same-words retry,
preserving the round-5 rule; present records must match exactly, keeping
different-quotes clientSteerId reuse a 409 on quote-aware readers. Under
the keep-on-chip client contract a legacy replay's missing echo is
harmless — the excerpts stay carried on the pending chip.

* 🧪 chore: Re-Trigger CI After Dropped Workflow Events
2026-08-24 22:29:13 -04:00
Danny Avila
fc2b8584c4
📇 feat: Surface Event Child Activity Through a Bounded Parent Index (#15142)
* feat: surface event-driven child activity

* fix: keep child task aggregation documentdb-compatible

* fix: address event activity review findings

* test: provide markdown message context defaults

* fix: report bounded child history truncation

* fix: preserve current child activity state

* fix: preserve durable event child activity

* fix: handle missing task timestamps

* fix: keep active event snapshots live

* fix: preserve event activity across valid anchors

* fix: close event child activity gaps

* fix: preserve event activity across resume
2026-08-23 18:50:00 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Danny Avila
8ae94afa91
🪡 fix: Thread Parent Message ID Through MCP Request-Scoped Bodies (#15095)
* fix: Unify MCP request-scoped headers

* fix: address request-scoped MCP review findings

* test: preserve request scope on status errors

* fix: treat authorized on-demand MCP servers as ready

* refactor: separate MCP readiness from connection state

* fix: preserve on-demand MCP readiness labels

* test: satisfy OpenAI conversation ownership guard

* fix: keep MCP action predicates boolean

* fix: close deferred MCP request context gaps

* fix: preserve on-demand MCP configuration actions

* fix: fail closed on unavailable MCP parent context

* test: complete MCP connecting-state mocks

* fix: preserve missing MCP parent on continuations

* fix: align native MCP request identities

* fix: preserve edited MCP parent identity

* test: use scoped Agent initializer fixture

* test: expose MCP request body helper

* fix: preserve MCP turn identity across resume

* style: sort stream metadata imports

* fix: carry normalized MCP identity to execution
2026-08-21 16:33:01 -04:00
Yorgos K
9f8d71a3c5
🪢 fix: Preserve Response Identity and Branch During Resumable SSE Sync (#14788)
* fix(client): preserve resumable response identity

Fixes #14787

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(client): align resumable sync regressions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(client): clarify resumable response ownership

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(client): preserve resumed regeneration ordering

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(client): cover missing resumed response row

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(client): preserve resume identity on page reload

* fix(client): replace reassigned resume placeholder

* fix(client): preserve content during response id handoff

* fix(client): limit resume placeholder handoff

* fix(client): preserve resume display metadata

* fix(client): reconcile resume metadata in one pass

* fix(client): reconcile preliminary resume user

* fix(client): restore regenerated branch on early abort

* test(client): cover external regeneration resume

* fix(client): preserve regeneration history on errors

* fix(client): replace reused regeneration error ids

* fix(client): preserve exact-id regeneration rollback

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-21 14:25:03 -04:00
Danny Avila
8c14f03432
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects

Adds an optional chat-project destination to a schedule, plus the operator
config to require one — or to pin every scheduled run to a specific project.

Feature:
- `chatProjectId` on the schedule row, accepted on create/update, projected on
  the wire, and carried into the run's conversation through the durable trigger
  envelope's `run` context.
- `interface.schedules.requireProject` refuses schedules that are not filed
  under a project; `interface.schedules.projectId` pins every run to one
  project and implies the requirement.
- Dialog gains a project picker (required when configured, a read-only row when
  pinned); the card shows the destination and the new disabled reasons.

Invariants:
- ONE resolver (`resolveScheduleProjectId`) decides the destination for the
  write handler, the fire path, and the wire projection alike, and an operator
  pin OUTRANKS the stored id in all three. Tightening the config therefore
  redirects — or stops — existing schedules instead of grandfathering where
  their runs land. A pin implies `requireProject` for the same reason: without
  it, a row created before the pin would keep firing with no project at all.
- Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write
  this handler accepts is one the next fire also accepts. Any edit leaving a
  schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like
  the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips
  the requirement, or a schedule auto-disabled for `project_required` could
  never be turned off.
- Fire-time enforcement auto-disables rather than filing runs loose, matching
  agent_deleted: new `project_required` (requirement raised after creation) and
  `project_deleted` (gone, or pinned to a project this owner does not have)
  reasons, both refused BEFORE a billed generation is dispatched, and both
  advancing so a schedule can never wedge on the occurrence.
- `computeCreateDigest` appends the field only when present, so a payload
  without a project digests byte-identically to one from before this change —
  a create retried across the upgrade still matches its own row instead of
  reading as key reuse.
- Project reads are scoped to the owner, so ownership and existence are the
  same lookup; a read ERROR propagates instead of failing closed, so a Mongo
  blip retries the fire rather than auto-disabling the schedule.

The trigger idempotency key hashes principal/event/target and never
`envelope.run`, so the added run field cannot destabilize delivery identity.

* 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget

The project picker landed as a new ROW in the schedule dialog, which broke the
e2e edit spec: `md:overflow-visible` turns off the template's scrolling from
`md` up, so the dialog's content must fit the viewport. The extra row pushed the
footer's Save button below a 720x1280 window, where Playwright reported a
visible, enabled button it could never click — 226 scroll-into-view retries and
a 2-minute timeout, on all three attempts.

Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720):
  dev              688   (32px slack)
  project row      ~790  (off-screen, CI failure)
  3-column row     704   (16px slack — half the budget spent)
  this commit      686   (34px slack, 2px better than dev)

The identity row is now three columns — name, agent, project — and its caption
moved out of the agent cell to sit full width beneath the row: at a third of the
dialog that sentence wraps an extra line, and the row is the tallest thing
competing for the budget. The caption is grouped with the row rather than left
to the form's own 4-unit rhythm, which spent more height on the gap than the
caption occupies.

The e2e spec now asserts the button is in the viewport before clicking it, so
the next field that overflows this dialog says so in one line instead of a
two-minute timeout on a visible element. FOLLOWUPS.md records what the planned
dialog controls (multi-day weekly, timezone, attachments) need first: give
`ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the
popovers into the dialog content, and let the form scroll again.

* 🧹 fix: Address Codex Review on Scheduled Chat Project Scope

Four P2 findings, all real.

Unreachable clearing path (ScheduleDialog). The picker only held live projects, so
`com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a
schedule had a project the owner could never take it away, leaving the server's
`chatProjectId: null` path reachable only by API. The picker now carries a real
"No project" option whenever a project is optional, and omits it when one is
required, where there is nothing valid to select.

Placeholder shown for a real project (ScheduleDialog). A stored or pinned project
outside the first loaded page had no name in the paged map, and the combobox
renders its placeholder for an empty display value — telling the owner a scoped
schedule had no project. That one project is now read by id, with the raw id as a
last resort: a poor label, but an honest one.

Project policy skipped at the resume boundary (service.ts). `claimScheduleResume`
re-applied the schedules gate, the revision fence, the kill switch and
SCHEDULES:USE, but not the project policy this PR added. Approving a paused run
whose project was deleted — or whose owner now sits under a requirement or a pin
it no longer satisfies — billed a continuation the very next scheduled fire would
refuse and auto-disable the schedule for. The effective-project resolution now
runs there too, refused before the lease and the capacity slot so a policy refusal
costs nothing and leaves no state to unwind. NOTE: agent access and balance are
still not rechecked on resume; that gap predates this PR and is left alone.

Per-card project derivation (ScheduleCard). Every card ran the projects hook and
rebuilt the full option array, name map, and one icon element per project, to use
a single name — O(schedules x projects) per render and per project-list refresh.
The hook is split: `useChatProjectNames` (map only, for the panel, which resolves
every card's name once and passes it down) and `useChatProjectPicker` (options and
pagination, for the dialog's one combobox). The panel skips the query entirely
until some schedule actually has a scope.

Tests: four at the resume boundary (verified failing without the gate) and four in
the dialog spec. The picker selection in one existing test now goes through the
search field — the popover's VIRTUALIZED renderer sizes its window from a scroll
height jsdom always reports as 0, so with three options it materialized only two.
Full schedules e2e re-run green against the rebuilt client.

* 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent

Second Codex round, four P2s. Three were consequences of the resume gate added in
the previous commit, which was half-built: it admitted where it should not and
stranded the run where it refused.

Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy`
branch. Both entry points consult that branch FIRST, and both already route its
refusal through abort-and-settle — so a policy stop now settles the occurrence
instead of answering a bare 409 while the job stays `requires_action`, the card
keeps reading "Needs approval", and every retry repeats the same 409 until expiry.
No change to resume.js: the existing branch does the work.

The rule is deliberately NARROW. It refuses only where no valid destination is
left — the requirement is on with nothing satisfying it, or the schedule's own
project is gone (which also unset it on the conversation). It does NOT refuse
because an operator's pin moved: the paused conversation cannot be rebound
(`chatProjectId` is excluded from the resume context and the continuation reuses
the same conversationId), so refusing would strand a pending approval over a
destination it can never reach, for a pin that governs only where the NEXT run
lands — which the fire path already redirects.

Create retries are idempotent again. Project policy had been applied BEFORE the
`clientRequestId` replay lookup, so a raised requirement, a deleted project, or a
moved pin could answer 400 for a create that already committed — pushing the
client to rotate its key and create a DUPLICATE schedule, the exact failure the
key exists to prevent. Policy now applies only to a genuinely new insert, and the
digest is computed from the CLIENT's payload rather than the resolved destination,
so today's policy can no longer re-digest a genuine retry into a mismatch.

An explicit `chatProjectId: null` under a pin is refused rather than silently
resolved to the pin. The payload contract defines `null` as clearing the scope;
answering 201 while filing under the pin reported success for the opposite of what
was asked. Only an OMITTED field takes the pin silently.

Tests: five on the policy branch (both refusals verified failing without it, plus
guards that a moved pin and a live project still admit) and two on the handlers
(the pinned explicit clear, and a committed create recovered by retry after the
policy tightened — verified answering 400 without the reordering).

* 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved

Third Codex round, four P2s.

The root confusion behind the resume findings: an operator pin outranks the stored
id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row
keeps its old value. The row therefore LIED about where that occurrence's
conversation went, and every later re-validation — the resume boundary above all —
checked a project the conversation was never filed under. A schedule storing A,
pinned to B, with B later deleted and A still live, was admitted for resume into a
conversation that had just been unscoped.

Fixed at the source rather than at each reader: a fire that resolves a destination
different from the stored one writes it back, claim-token fenced like every other
worker-side write and deliberately WITHOUT a configRevision bump — this is the
server reconciling itself to policy, not an owner edit, and a bump would fence an
in-flight occurrence off its own run. Written only AFTER the destination validates,
so an unusable pin never lands in the row, and best-effort: the envelope already
carries the right destination, so a failed write costs accuracy on a later recheck,
never the run itself. The wire projection already reported the pin, so this also
stops the row and the UI disagreeing.

An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit
path too. The pin check and the requirement are independent rules, and folding them
together let `{enabled: false, chatProjectId: null}` skip the pin check entirely,
unset the row, and answer with a wire projection still naming the pin. Only the
requirement is waived for a disabling edit.

The dialog no longer requires a project for an edit that leaves a schedule DISABLED.
The server waives the requirement there precisely so a row auto-disabled for
`project_required` can still be renamed or tidied up; requiring it in the form made
that unreachable, and an owner with no projects could not edit the stopped schedule
at all.

FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second
deletion race inside the resume claim window (which needs the effective project
persisted per OCCURRENCE plus a distinct policy conflict routed through
abort-and-settle), and the fact that convergence happens only when a schedule fires.

Tests: four on convergence (pin written, no write when unchanged, no write for a
destination that failed validation, fire survives a failed write), two on the
disabling-edit pin rules, two in the dialog. Schedules e2e re-run green.

* 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest

`computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and
an explicit `null` produced the same digest. Because the replay lookup and
`matchesCreateIntent` deliberately run before project policy, a request could reuse a
pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and
receive 201 describing the pinned row — success reported for the opposite of what it
asked, and the pinned-clear refusal the normal create path applies never reached.

Now `!== undefined`: an omitted field still digests byte-identically to a payload from
before project scope existed, so a create in flight across the upgrade still matches
its own row, while an explicit clear is a distinct intent and digests differently. A
pre-scope client never sent the field at all, so nothing legacy can carry an explicit
null.

* 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used

The schedule-wide convergence from the previous commit was not enough, and the
reason is the single-active run index: it covers `status: 'started'` only, so a
PAUSED run does not block the next occurrence. While run 1 sat paused in project A,
a pin move plus one later fire rewrote the schedule row to B — and the resume
policy then validated B while run 1's conversation was still filed under A. Delete
A and the continuation was admitted into a conversation that had just been unscoped.
That window lasts as long as the pause, not the sub-second race the previous commit
documented.

The reservation now records the destination THIS occurrence used, and
`isScheduleLive` validates that record when given the occurrence's `scheduledFor` —
which `resume.js` already reads two lines above the call. No new conflict type and
no settlement-path surgery: the refusal rides the abort-and-settle branch that check
already has.

An ABSENT record falls back to the schedule-level resolution rather than reading as
unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as
evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as
they do today.

Schedule-wide convergence stays: it keeps the row honest for the UI and for every
check that has no occurrence in hand.

FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the
claim window, which needs a distinct policy conflict routed through abort-and-settle
rather than the bare 409 an `inactive` conflict produces.

Tests: three on occurrence-vs-row precedence (the decisive one verified failing
without the lookup), two on what the reservation records, and the resume controller
spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green.

* 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One

The occurrence fallback added in the previous commit conflated two different
absences. A post-upgrade run that deliberately went unscoped omitted the field
exactly like a row written before the field existed, so both took the fallback —
and a paused unscoped run was then validated against the schedule's CURRENT
project. Under a requirement or a pin added while it sat paused, that admitted a
billed continuation into a conversation satisfying no present policy.

The reservation now ALWAYS records its decision, `null` for unscoped, and the read
reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record
— a pre-scope row, or no row at all — falls back to the schedule-level resolution;
a recorded null is the genuinely unscoped occurrence it says it is, and is refused
once a project becomes required.

The distinction rests entirely on a stored `null` surviving as a present key while a
never-written field stays absent, so that is asserted against real Mongo rather than
assumed: if it ever stopped holding, unscoped runs would silently start being
validated against the schedule's current project again.

Tests: two against mongodb-memory-server (recorded null vs never-written vs missing
row), plus refusal of a recorded-unscoped occurrence under a new requirement and the
preserved fallback for a pre-scope one. Schedules e2e green.

* 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence

The initial-start policy check in `request.js` called `isScheduleLive(..., { policy:
true })` without `scheduledFor`, so it fell back to the schedule-level resolution
even though the run row already carries the occurrence's recorded scope. An
occurrence reserved unscoped, with a pin introduced while its loopback request sat
queued, was therefore admitted against the new pin — producing a billed unscoped
conversation under a requirement it does not satisfy, from an envelope already built
without a project.

`scheduledFor` was already in scope there. Passing it makes the initial start and the
resume validate the same way: against the destination the occurrence itself recorded.
2026-08-21 11:09:04 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
5e3c680761
🪃 feat: Wake Parent Agents on Child Completion (#14975)
* feat: wake parent agents on child completion

* wip: harden child completion wakeup lifecycle

* fix: close the completion-wakeup static failures

Type the durable-claim store fixture, the continue-envelope test helper,
and the terminal message's task metadata so the wakeup suites compile
against the shapes they actually exercise. Replace `Array.prototype.at`,
which the package target library does not provide.

Capture the prepared child thread in a non-optional local before the
provider callback closes over it, and narrow the trigger envelope itself
on `mode === 'continue'` rather than a separately copied mode, so reading
the continue target is sound. Lift the parent-message fallback out of a
nested ternary into a named resolver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the active-predecessor admission fence

The Redis job-creation call gained a thirteenth scalar argument, so the
spec helper reconstructed the HSET pairs one slot early and rebuilt an
invalid job hash; three creation tests failed on that alone.

Give the fence itself direct coverage in both store adapters, which it
had none of despite deciding whether an automatic continuation may
replace a live parent turn. Each proves a running and a requires_action
predecessor are refused with the state a controller needs for a finite
409, that an absent or settled predecessor is admitted, and that an
ordinary user turn without the policy still replaces its predecessor.
The Redis case also asserts a refused continuation leaves the parent's
durable job and chunks untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: harden completion wakeup rollout and claims

* fix: close completion wakeup race windows

* test: keep the child store fixture exact

* fix: close final subagent wakeup gaps

* fix: preserve ambiguous completion claims

* fix: release pre-admission wakeup claims

* fix: stabilize subagent completion recovery

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 12:18:31 -04:00
Danny Avila
547bd8c4bf
🧵 feat: Persist View-Only Subagent Threads (#14957) 2026-08-18 07:41:42 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
57ea1137f6
🛡️ feat: Let Admins Restrict Stateful Workspace Scopes (#14910)
* feat: let admins restrict stateful workspace scopes

* fix: enforce stateful scope policy across agent paths

* fix: close stateful scope policy activation gaps
2026-08-17 01:29:19 -04:00
Anubhav Anand
a2ad0aa0c8
🤐 feat: Allow Promptless Sends When Files Are Attached (#13717)
* ✨ feat: Allow sending file attachments without a text message

When an agent asks the user to upload a document, the user could attach
the file but still had to type a placeholder message ("OK", "Here is
the file") before the send button enabled and the submit guard let the
message through.

Attachments now count as submittable content:

- New isSubmittableMessage(text, fileCount) util: non-whitespace text
  OR at least one attached file.
- ask() in useChatFunctions uses it instead of bailing on empty text,
  so an empty draft with attached files submits.
- SendButton receives the attached file count and enables accordingly.
- ChatForm only marks the text field as required when no files are
  attached, so react-hook-form validation no longer blocks handleSubmit.

Submitting an empty draft with no attachments is still rejected at all
three layers.

Fixes #13646

* Address review: support replayed file-only turns + drop empty vision text

- ask(): count replayed attachments (overrideFiles) in the submittable
  check and skip it entirely for regenerate, so a file-only message can
  be regenerated or saved-and-resubmitted instead of being rejected as
  empty.
- formatVisionMessage(): omit the text content part when the message text
  is empty. Anthropic rejects empty text content blocks with HTTP 400,
  and an empty block adds nothing for other providers; image-only sends
  now format cleanly. Added formatMessages tests for with-text and
  image-only (Anthropic + other) cases.

* Address review: keep attachment-only turns valid for providers, answer mode, and titles

- formatMessage: substitute minimal text when a user turn carries files but no
  inline content, so Anthropic does not reject an empty user message for RAG or
  code-environment attachments.
- assistants chatV1: send the same stand-in for attachment-only Threads
  messages, which reject an empty body. The persisted message keeps empty text.
- ChatForm: attachments no longer make an empty draft submittable in answer
  mode, where submitText consumes the click without answering or sending.
- agents request: seed title generation from attachment filenames when the turn
  has no text, so immediate-mode titles are not invented from an empty string.
- useChatFunctions.regenerate.spec: mock the utils barrel over the real module
  so new exports resolve.

* Cover the agents path for attachment-only turns

AgentClient formats its payload with the SDK's formatMessage, not the local
one, so the earlier guard missed the endpoint the feature actually targets: an
attachment-only turn still reached Anthropic as an empty user message. Apply
the same stand-in after the file-context and quote merges, so a turn that
already gained inline content is untouched.

* Carry filenames on freshly attached files

The fresh-file submission mapping copied only file_id, filepath, type, and
dimensions, so the attachment-only title fallback read an undefined filename
and produced nothing. Include filename, and cover it with a test that submits
an empty draft with one attachment.

* Address review: cover assistants v2, fresh agent attachments, editor, and title fallback

- agents client: the current turn has no files during buildMessages, so read
  the resolved attachments from message_file_map instead. The previous guard
  only ever fired for persisted historical turns.
- assistants chatV2: the default assistants endpoint routes here, so it needs
  the same stand-in body chatV1 got.
- assistants title: fall back to filenames, then the response, and keep the
  default title rather than saving an empty one.
- EditMessage: retained attachments make an empty edit submittable, matching
  the composer, so the overrideFiles replay path is reachable from the UI.

---------

Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com>
2026-08-15 12:47:31 -04:00
Danny Avila
abc669ab58
🩹 fix: Restore the @librechat/api Build and Remove Legacy Code (#14808)
* 🧹 chore: Remove Dead Legacy Agent Controller

`_LegacyAgentController` has been unreachable since the resumable path became
the only route: it is unreferenced, unexported, and untested. It had also
drifted out of compilability against the live file — line 2009 called
`attachConversationCreatedAt(req, { userId, conversationId, isNewConvo })`
against the 3-argument signature declared at line 97, which would await
`undefined` and then throw dereferencing `resolved.createdAt`.

Keeping it was not free. It carried a third independent copy of the response
message-id wiring (`getReqData`, `onStart`, four `updateMetadata` calls), so
every change to how a generation identifies its response row had a dead third
site to keep in step, and no test to say whether it had been kept in step.

Removing the block leaves `createCloseHandler` and the `sendEvent`,
`clientRegistry`, `requestDataMap` and `handleAbortError` imports with no
remaining callers, so those go too. `AgentController` was a three-line
passthrough to `ResumableAgentController`; the real controller is now exported
directly, which also matches the `[ResumableAgentController]` prefix every log
line in the file already uses. `server/routes/agents/chat.js` binds the export
to its own local name and passes the same five arguments, so the route is
unchanged.

No behavior change: 379 lines removed, 2 added.

* 🩹 fix: Remove Duplicated Anchor Block Breaking the `@librechat/api` Build

`dev` does not build. `packages/api/src/agents/activityPhases/runtime.ts`
carries two byte-identical 98-line copies of the same block (former lines
516-613 and 614-711), so rolldown fails to parse it:

    [PARSE_ERROR] Identifier `AnchorFields` has already been declared

The duplicated block is the anchor-construction work from #14805:
`AnchorFields`, `laterDefinedIndex`, `foldedAgentIds`, `boundedAnchor` and
`mergeAnchors`. #14807 was squashed from a branch that predated #14805 and
re-included that commit, so both copies landed. Only the `type` produced an
error — the four function declarations simply redeclare.

This removes the first copy. The two blocks were verified byte-identical
before the cut, and the resulting file has no duplicate top-level
declarations, is missing nothing that #14805 introduced, and retains
everything new to #14807 (`ResolvedPosition`, `resolvePosition`).

Verified: `tsdown` builds, `tsc --noEmit` clean, `config/circular-deps.mjs`
green across all five graphs (it was reporting `✗ @librechat/api` purely
because the build it shells out to was failing), and the 68 tests in
`activityPhases/runtime.spec.ts` pass.

Carried here rather than in a separate PR because this PR's checks cannot go
green until it lands: the failed `packages/api` build cascades into e2e, MCP
list_changed, bombadil and the Docker image jobs.
2026-08-13 19:57:32 -04:00
Danny Avila
5ff46d8c67
🛟 fix: Stop Agents When Code Resources Cannot Recover (#14651)
* fix: Block Agents When Code Resources Cannot Recover

* fix: Preserve Resource Recovery Failures Across Agent Paths

* fix: Centralize Fatal Agent Initialization

* chore: sort agent imports
2026-08-07 00:33:28 -04:00
Danny Avila
3191f6975a
🏷️ fix: Skip Title Generation for Preempt-Incomplete Turns (#14571)
Some checks failed
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
2026-08-01 09:01:13 -04:00
Danny Avila
3dc5532111
🏢 fix: Preserve Tenant Context for Partial Response Saves on Disconnect (#14567) 2026-08-01 08:18:32 -04:00
Danny Avila
e7f1838515
⚡ feat: Reliable Interrupt & Steer Escalation and Recovery (#14558)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: surface interrupt-steer escalation on waiting messages

The interrupt & steer feature shipped reachable only through the
composer chord, the send-button hovercard, and the composer button; a
message already waiting (queued for after the run, or steered and
parked at the next tool boundary) had no path to it. Both waiting
surfaces now carry one:

- Queued rows get an icon-only ZapOff escalation button beside the
  existing Steer primary. It routes through sendQueuedNow, which now
  takes a preempt option on its live-run path. The tooltip teaches the
  composer chord, derived through resolveComposerKeyDown so a rebound
  or yielded chord is never advertised.
- In-flight steer bubbles get an "Interrupt now" overflow entry with
  the same race rules as Edit: reclaim first, and only a `reclaimed`
  outcome resubmits (via retrySteer with preempt, swapping the chip
  for an interrupting one). `applied` and run-ended-mid-reclaim
  outcomes stop at the existing informational toasts, so the words can
  never land twice. Not offered on a steer already preempting.
- Every during-run overflow menu gains an "Always interrupt instead"
  toggle for steerInterruptsByDefault, next to the existing steer/queue
  default toggle. MenuEntry supports disabled for the new entries.

Only one interrupt can be unresolved at a time: while one preempt is
pending (or the run is paused on approval, where the server 409s),
every escalation control disables instead of racing the same seal.

Ten new tests across both surfaces; 381 green in the affected suites.

* fix: lock escalation across its reclaim window, keep the paused control visible, label as steer

Codex round 1, all three findings.

P2, escalation race. The single-interrupt invariant had a window between
clicking "Interrupt now" and the reclaim resolving, where no preempt
chip existed for the chip-derived gate to see: two bubbles escalated
back-to-back could both resubmit. A shared escalating flag (Jotai,
per-conversation) now covers the window and disables every escalation
control on both surfaces, and a fresh recheck before resubmitting
catches an interrupt armed elsewhere meanwhile (composer chord, queued
row); those words re-home to the queue with an informational toast
instead of breaking the invariant.

P2, unreachable paused state. canSteer is defined as
hasRealConvoId && !pausedOnApproval, so gating the button on canSteer
removed it exactly when it was meant to render disabled; the test only
passed on an impossible stub combination. The render gate is now
duringRunActive && (canSteer || pausedOnApproval), and the test uses the
real invariant.

P2, label semantics. "Interrupt & send now" borrowed the name of the
hard-abort action; this one preserves the partial answer and steers.
Renamed to "Interrupt & steer now" (com_ui_interrupt_steer_now).

Both behavior fixes counterfactually verified; 384 tests green across
the affected suites.

* fix: disable bubble escalation while the run cannot accept a steer

Codex round 2, one P2. Answer mode (ask_user_question) sets
duringRunActive false while pausedOnApproval stays false, since that
flag only detects approval-bearing tool calls. The bubble's escalation
entry stayed enabled there, so clicking it cancelled a healthy waiting
steer and the preempt resubmission bounced off RUN_PAUSED, degrading
the words to the queue. The entry now also disables on
!duringRunActive, matching the queued-row control's gate.

Counterfactually verified: reverting the gate fails the new
answer-mode test.

* fix: recheck live run state after the reclaim, not just at the click

Codex round 3, one P2, and it is the round-1 recheck principle applied
one level deeper: the entry-time disable cannot see a run that pauses
(tool approval, answer mode) while the reclaim round-trip is in flight,
and the .then closure held the render's stale steering controls, so the
resubmit would fire into a RUN_PAUSED rejection after the reclaim had
already surrendered the steer's boundary slot.

The escalation continuation now reads the LIVE controls through a
latest-ref: if the run can no longer accept a steer, the words re-home
to the queue with an informational toast instead of resubmitting, and
the resubmit itself also goes through the live controls.

Counterfactually verified: reading the stale closure instead of the ref
fails the new mid-reclaim pause test.

* refactor: make escalation one atomic server-side arm, in place

Codex round 4: four P2s, every one an interleaving of the same window —
escalation as reclaim-then-repost is a compound, non-atomic operation
whose continuation must revalidate the world (FIFO position lost, ref
assigned too late, no run fence, competing bubble actions). Rounds 1-3
patched that window with a lock and rechecks; round 4 shows the window
itself is the defect, so this removes it instead of guarding it again.

Escalation is now POST /chat/steer/arm: the server flips preempt on the
EXISTING queued item in one atomic store op (new IJobStore.armSteer; a
decode-patch-encode LSET Lua on Redis, an in-place mutation in memory),
fenced to the validated generation and refused once the queue closes.
The handler mirrors the steer POST's preempt contract exactly: durable
flag gated on the owner's recorded capability, volatile requestPreempt
fire-and-forget because the durable flag is the truth resume/handover
re-arm from.

By construction this resolves all four findings: FIFO survives (the
item never moves; the whole queue still drains in instruction order at
the seal), there is no continuation to hold stale controls, the store
op is fenced to the original run, and a competing Edit/Queue/Cancel
either beats the arm (armed:false, chip untouched) or operates on the
armed item, whose cancel already disarms.

The client escalation entry becomes one mutation: armed:true relabels
the chip in place (same steerId, same position), PREEMPT_UNSUPPORTED
and lost races toast honestly, and the round 1-3 machinery — the
escalating lock atom, the latest-ref, the post-reclaim rechecks and
their two toast strings — is deleted rather than extended.

Verified: 7 new handler tests on the real in-memory manager (including
FIFO preservation and the stale-generation fence), 2 Redis integration
tests against real Redis (in-place arm keeps order and every field;
missing/stale/closed all refuse), client suites 396 green.

* fix: decide capability inside the atomic arm, neutralize the lost-race toast

Codex round 5, both findings, both edges of the new arm design rather
than its mechanism.

P2, capability TOCTOU. A HITL resume on a rolling deploy rewrites
preemptCapable for the SAME generation, so the handler's read could go
stale between validation and the flag flip, arming a steer the live
owner cannot seal. armSteer now returns armed | missing | incapable,
with the owner's live capability part of the same atomic predicate as
the generation fence (HGET preemptCapable inside the Lua; the flat job
field, not a metadata blob — the in-memory store reads the same field).
The handler's pre-check is deleted rather than kept alongside; the
store predicate is the single source. New handler test rewrites the
capability after queueing and expects PREEMPT_UNSUPPORTED with the item
left unflagged; the Redis guards test now asserts the incapable refusal
against real Redis.

P2, ambiguous toast. armed:false covers injected, cancelled, re-homed,
and run-over alike, so telling the user the message "already reached
the agent" claimed one specific outcome. The lost-race branch now uses
a neutral message (com_ui_steer_arm_lost_race) and defers to the events
for what actually happened.

* fix: flip the escalation lock synchronously before the arm request

Codex round 6, one P2. Round 4 deleted the escalating flag along with
the reclaim continuation it guarded, but that left the one-interrupt
gate blind during the arm request's own round trip: the chip-derived
check cannot see an arm until its response relabels the chip, so on a
slow connection two bubbles could both arm before either response
landed. Double-arm is harmless server-side now (the run seals once and
drains the whole queue in order), but every escalation control
advertises "one interrupt at a time" by disabling, and the controls
must tell the truth.

The per-conversation escalating flag returns as a pure UX gate: set
synchronously at click, before the mutation, cleared on settlement, and
folded into interruptPending on both surfaces. Unlike its round 1-3
ancestor there is no continuation behind it to guard and no recheck to
pair with it.

Counterfactually verified: without the synchronous set, the two-bubble
race test arms twice. 207 tests green across the Chat Input suites.

* test(e2e): cover escalation of waiting messages through the real seal

Three mock-harness tests on E2E_SLOW_REPLY, a 160-chunk stream with no
tool boundary, so an in-thread steer part can ONLY come from a genuine
mid-stream seal — which makes each test a behavioral proof rather than
a UI check:

- Queued row escalation: the ZapOff button turns a waiting queued
  message into a preempt-armed steer (202 echoes preempt: true) that
  seals and injects, where the sibling steering.spec test proves the
  unescalated path waits for run end instead.
- Bubble in-place arm: an ordinary steer (202 with no preempt echo)
  waits as a bubble, POST /chat/steer/arm answers armed: true, the
  bubble relabels in place (same single bubble, same text, escalation
  no longer offered on reopen), and the armed steer seals mid-stream.
- Always-interrupt toggle: flipped from a waiting row's overflow menu,
  plain Enter now produces a preempt: true steer that seals in the SAME
  run, and the menu offers the way back. An afterEach clears the
  localStorage preference so a mid-test failure cannot leak
  preempt-by-default into the rest of the serial suite.

All three verified locally through the full harness (real backend, mock
LLM, seeded DB): 3 passed in 27s.

* feat: dedicated escalation arrow + shortcut, menu split into actions and preferences

The escalation was still half-hidden: the bubble only offered it inside
the overflow menu, and the tooltip taught the composer chord, which does
a different thing (interrupts with typed text, not this chip). Three
changes make it a first-class command:

- A shared EscalateNowButton (circular arrow, ghost-bordered like the
  composer's interrupt control) is always visible on BOTH surfaces:
  beside each queued row's Steer primary and on every waiting steer
  bubble next to its menu. It disappears once a steer is interrupting.
- A dedicated registry shortcut, escalateSteer (Cmd/Ctrl+Shift+.),
  editing-allowed and rebindable like every other action. Deliberately
  NOT an Enter chord: the composer owns every Enter chord, and the
  yield design rests on no default binding using Enter besides submit.
  Its handler clicks the newest enabled arrow control (bubbles beat
  queued rows), so the shortcut can never diverge from the button, and
  the arrow's tooltip teaches THIS command via the registry display.
- The overflow menus separate one-off actions from sticky behavior
  changes: Edit, Cancel, Queue, then a smaller "Preferences" section
  holding the queueing and always-interrupt toggles, each with the
  standard InfoHoverCard reusing the Settings panel's descriptions.
  "Interrupt & steer now" leaves the menu entirely.

386 client tests green, including a menu-structure test locking the
order and the absence of the escalation entry; bubble escalation tests
drive the visible arrow. The e2e spec's bubble test now clicks the
arrow, and a fourth test drives the dedicated shortcut end to end
through a real mid-stream seal.

* style: bind the escalation arrow to its message (variant A anatomy)

Two same-weight circles in a row read as one control group, leaving the
arrow's ownership ambiguous, and a floating arrow stops meaning anything
once several messages stack. The shared control now carries variant A's
anatomy: a thin divider binds a small SOLID arrow (filled, inverted) to
the message region on its left, and the menu ellipsis stays a bare
glyph, so the two affordances can no longer blur together — and the
divider+arrow pairing repeats cleanly per chip at N messages.

* chore: drop the unused within import CI lint caught

* fix: advertise the escalation shortcut only while the control is live

Codex on the e2e head, one P2: the tooltip appended the chord hint even
while the button was disabled, advertising a shortcut that does nothing
during an approval pause. The flagged control (InterruptNowButton) was
since replaced by the shared EscalateNowButton, which inherited the
pattern; the successor now omits the chord whenever the control is
disabled, matching the rule the during-run hovercard already follows.

* fix: harden steer escalation lifecycle and recovery

* test(e2e): disambiguate accessible steer preferences

* test: align abort persistence coverage with prerequisites

* chore(i18n): remove obsolete steer race message

* chore: normalize imports across steering changes

* test: exercise stream integration on Redis Cluster

* test: scope HITL checkpoints to generation

* test: fix cluster cleanup and locale policy

* fix: keep escalation visible during ask pauses

* fix: fence recovery downgrade and stale predecessors

* fix: require generation owner abort acknowledgement

* fix: validate delayed preempt arms

* test: align final escalation fixtures

* fix: preserve in-memory predecessor abort handoff

* fix: restore controls for recovered queued messages

* test: cover recovered queue controls

* fix: close final steering review gaps
2026-07-31 20:07:56 -04:00
Danny Avila
7bb6651883
🛑 feat: Preemptive Steer - Backend Interrupt & Steer (#14518)
* 🛑 feat: Preemptive Steer — server half (Interrupt & Steer, PR 2 of 3)

Lets the steer route ask the generating replica to seal its live model
stream at the next provider-safe boundary instead of waiting for a tool
step. The run is never aborted, job status never changes, the partial
answer is kept, and generation resumes in the same assistant message
after the injected steer. Consumes the SDK seam in @librechat/agents
(danny-avila/agents#335, #346).

Transport: IEventTransport gains a fenced emitPreempt/onPreempt pair
beside abort. RedisEventTransport fans PREEMPT out on the SAME events
channel and subscription (no new connection, key, or subscribe call);
onPreempt returns a registration-scoped unsubscribe with the same
replacement-safe state-identity guard onAbort uses. InMemory implements
neither — single-process preempt lives entirely in the runtime set.

Runtime state: RuntimeJobState carries the per-generation request set,
createdAt-fenced and capped at STEER_QUEUE_MAX_DEPTH, plus a bounded
`cleared` tombstone so a late cross-replica arm cannot resurrect a
request whose steer already drained. registerPreemptSubscription
mirrors the abort registration's double fence (runtime identity +
generation createdAt); releaseAbortSubscription retires BOTH listeners
and the armed set, so every terminal path drops preempt state for free.
Public surface: requestPreempt (arm + fenced publish, never a rejection
surface, never touches job status), isPreemptRequested (O(1)
level-triggered poll), noteSteersRemoved (drain/cancel bookkeeping +
fenced clear), clearPreemptRequests (empty-boundary disarm).

One drain body, two boundaries: createSteerDrainHook (PostToolBatch)
and createSteerPreemptBoundaryHook (PreemptBoundary) share
drainAndBuildInjections, so the two injection sites cannot drift — the
SDK's provider-safety argument rests on identical HumanMessage shapes.
The shared body builds injections incrementally under a swallow-all
catch (a mid-loop throw still injects what was applied — those parts
are already persisted), clears preempt requests in finally, and
disarms the generation when a boundary drains nothing.

Request path: POST /chat/steer accepts preempt: true. The guard ladder
is unchanged in order and in every status code. A preempt request is
NEVER a rejection reason — without the capability the steer still
enqueues and the 202 echoes preempt: false. Armed strictly after a
successful enqueue; cancel disarms. The capability is read from the
OWNING replica's recorded `preemptCapable` rather than the route
replica's own SDK probe, so a rolling deploy cannot label a steer
"interrupting" that the older owner will only inject at a tool step.

Durable label: SteerQueueItem.preempt → TPendingSteer.preempt, so a
parked/claimed/replayed chip keeps its wording.

Run wiring: createRun registers the PreemptBoundary hook and threads
RunConfig.preemption, both gated on isSteerPreemptSupported() — a
separate probe from isSteeringSupported(), so the client affordance can
never arm against an SDK that only injects at tool boundaries.
buildSteerWiring builds both hooks from one shared closures object, so
preemption survives HITL pause/resume for free.

Honest finalization: an empty preempt boundary persists and emits with
unfinished: true — the same contract an abort gets — re-marked
explicitly because BaseClient has already saved the row as
unfinished: false by that point.

Not changed: no new job status, store method, Lua, SSE event type,
endpoint, or authorization surface. abortJob, completeJob,
transitionStatus, closeAndDrainSteers, getResumeState, emitChunk,
applySteerPart and the whole abort path are untouched.

Tests: 120 packages/api steering specs (preempt lifecycle, tombstone,
fences, caps, terminal release, both-boundary drain parity,
level-triggered poll, request/cancel arming, owner-capability
degradation) plus 5 in api for buildSteerWiring gating, and 2
Redis-gated cross-replica transport specs.

* 🔒 fix: Codex round 2 — evict tombstones, scope the empty-boundary disarm, honest resumes

All four server findings were fresh consequences of the round-1 fixes,
which is the review doing exactly what it should.

- Tombstone cap refused new entries instead of evicting. Every drained
  or cancelled steer is tombstoned, not just preempting ones, so a
  generation that processed 20 steers exhausted the set and the
  late-arm race resurfaced silently. Now evicts oldest-first (Set
  iteration is insertion-ordered), with the budget named
  PREEMPT_TOMBSTONE_MAX rather than an inline expression.

- The empty-boundary disarm I added in round 1 wiped the generation's
  ENTIRE armed set. A second steer can enqueue and arm between the
  atomic drain returning empty and the disarm running — that arm is
  backed by a live, uninjected queue item and must survive. The drain
  now snapshots the armed ids BEFORE draining
  (getArmedPreemptIds) and clearPreemptRequests takes an explicit id
  list instead of clearing everything.

- HITL resume finalized with a hardcoded unfinished: false. The
  boundary hook is re-registered on resume via buildSteerWiring, so a
  resumed segment can end on an empty preempt boundary exactly like a
  fresh one; finalizeResumedTurn now reads getPreemptStats() and the
  halt reason, matching the normal request path.

- Ownership moves on resume, so the job's recorded preemptCapable must
  describe the replica that will actually generate. Refreshed before
  resumeCompletion; a job created on a capable replica that resumes on
  an older one during a rolling deploy no longer acknowledges steers as
  interrupting.

Tests: +3 (scoped disarm sparing a post-snapshot arm, oldest-first
tombstone eviction, id-list disarm). 122 packages/api steering specs
green.

* 🚨 fix: Codex round 3 — deserialize preemptCapable from Redis (feature was dead under Redis)

The P1 here is the most consequential defect in the whole feature, and
it was introduced by round 1's own capability fix.

- `RedisJobStore.serializeJob` writes booleans generically, so
  `preemptCapable` reached Redis — but `deserializeJob` is an EXPLICIT
  field map and had no line for it. Every `getJob()` therefore dropped
  the flag, `job.metadata.preemptCapable` was always undefined, and
  `handleSteerRequest` computed `preemptArmed: false` unconditionally.
  Interrupt & steer would have silently degraded to ordinary
  tool-boundary steering in EVERY Redis deployment — i.e. the feature
  shipping as a no-op in production while passing every in-memory test.
  Now deserialized, with a round-trip assertion in the metadata spec
  that fails (`Received: undefined`) against the unfixed store.

- The resume capability refresh moved from just-before
  `resumeCompletion` to immediately after `approvals.resolve` claims
  the run. That call already flips the job back to `running`, so the
  steer route accepts requests from that instant; leaving the refresh
  135 lines later (across the whole client reconstruction) left a real
  window where a steer read the PREVIOUS owner's capability. Not the
  fully atomic transition Codex suggested — that reaches into the
  approvals Lua — but it shrinks the window from seconds to one await,
  which is proportionate for a label-accuracy issue.

Refuted: "avoid triggering preemption inside subagents". The premise —
that the run-wide poll can seal a subagent stream — does not hold
against the shipped SDK. Child graphs are constructed with
`subagentScope: true` (SubagentExecutor) and `preemption` is NOT
propagated into child inputs, while `canClaimPreemptSeal()` requires
`!subagentScope && preemption != null`. Both conditions fail
independently, so a subagent can never claim a seal and the boundary
cannot fire with `agentId` set. The `input.agentId != null` guard in
the hook is defensive depth, not the thing standing between us and the
described failure.

140 packages/api specs green.

* 🔁 fix: Codex round 4 — re-arm durable interrupt steers when resume moves owners

- An arm lives only in the owning replica's runtime plus a transient
  pub/sub message, while the steer's `preempt` flag is durable on the
  queue item. A HITL resume landing on a different replica therefore
  started with an empty armed set and a poll stuck false, so an
  interrupt the user had already been ACKed for silently waited for an
  ordinary tool boundary. New `GenerationJobManager.rearmQueuedPreempts`
  rebuilds the armed set by peeking the durable queue (fenced on the
  generation) and re-arming every item flagged `preempt`; resume calls
  it right after claiming. Safe by construction: every item peeked is
  still queued, so no drained steer can be resurrected.

- Capability-refresh failure now logs at error rather than warn, but
  deliberately does NOT fail the resume — see the reply on that thread.

Tests: +2 (rebuild from queue arms only the flagged item and reports
the count; a stale generation arms nothing). 124 packages/api steering
specs green.

* 📡 fix: Codex round 5 — acknowledge only what was actually armed

- A cross-replica arm was fire-and-forget: `emitPreempt` logged its own
  publish failure and `requestPreempt` returned void, so the route
  answered `preempt: true` even when the owner never armed a poll. The
  steer still injected at the next tool boundary, but the chip claimed
  an interrupt that could not happen — and unlike HITL resume, an
  ordinary running generation had no durable reconciliation to recover
  it.

  `emitPreempt` now resolves to the subscriber count and rejects on
  failure; `requestPreempt` is async and returns whether the arm truly
  landed (owned locally, or delivered to at least one subscriber). The
  202 reports THAT rather than what was asked for, so the chip relabels
  to ordinary steering exactly as it does for a capability-degraded
  deployment. Errors are swallowed into `false` — an unarmed interrupt
  is a downgrade, never a failed steer.

- The owner capability is re-read immediately before enqueue rather
  than reused from the top of the guard ladder. `checkAgentAccess` and
  file resolution are awaits, so a request can span an entire HITL
  pause/resume that moves ownership to a replica with different
  capability and rewrites that very flag. Only paid for by requests
  that actually asked to interrupt.

Tests: +3 (not-armed when the publish reaches nobody; armed when this
replica owns the generation; a throwing publish downgrades instead of
propagating). 127 packages/api steering specs green.

* 🎯 fix: Codex round 6 — real ownership, confirmed disarms, and a CI regression of my own

Three review findings plus three CI failures the round-5 commit caused.

Review:
- Ownership came from `runtimeState`, which a cross-replica `getJob`
  populates with a FACADE runtime on any replica that merely read the
  job. Matching `createdAt` therefore proved only "we looked at this
  job", so a non-owner could arm nothing and report success. Ownership
  now comes from `ownedJobs`, the actual owner map.
- `armPreemptIds` returns how many ids it accepted, and a local arm is
  only reported as armed when one was. A tombstoned id (its steer
  drained at an ordinary boundary mid-request) no longer answers
  `preempt: true` for an interrupt that cannot happen.
- The cancel disarm is awaited. A dropped clear is worse than a dropped
  arm: the owner keeps a level-triggered request for a steer that no
  longer exists, seals its next chunk and truncates an unrelated
  answer. The boundary drain's own call stays non-blocking — there the
  owner is local, so the disarm is already effective and awaiting the
  informational publish would only delay injection.
- Subscriber count is NOT read as proof of owner receipt: the count
  includes this replica's own facade subscription. A successful publish
  reports armed, a rejected one does not. Documented rather than
  papered over — see the acknowledgement-semantics note on the PR.

CI regressions from round 5, all mine:
- `registerPreemptSubscription` was AWAITED at both runtime-init sites,
  so job creation blocked on a second Redis channel subscription and
  hung when that subscribe was slow. Abort is awaited because a missed
  abort strands a run; a missed preempt only degrades that steer to the
  next tool boundary, so it now registers without gating createJob.
- Two api specs mocked `@librechat/api` without the newly imported
  `isSteerPreemptSupported`, so the call threw before createJob; and one
  exact-match assertion needed the new `preemptCapable` metadata field.
- My own Redis integration spec asserted arm-before-clear ordering,
  which two publishes carry no guarantee of — the receiving tombstone
  exists precisely because of that. Now asserts delivery and payload
  fidelity, order-independent.

158 packages/api specs, 27 api specs green.

* 🧭 fix: Codex round 7 — settle the acknowledgement semantics (Option A)

Round 7's second finding is the incoherence I flagged on the PR: the
route persisted `preempt: true` on the durable queue item while
returning `preempt: false` when delivery could not be confirmed. Those
two then disagreed, and `rearmQueuedPreempts` trusts the DURABLE one —
so a resumed owner would honour an interrupt the client had explicitly
been told degraded to ordinary steering.

Rather than patch the disagreement, this settles the meaning:

`preempt` in the 202 means "queued as an interrupt request", NOT "a
seal is guaranteed". It mirrors `SteerQueueItem.preempt` exactly, so
the response, the durable record, and the resume-time re-arm can never
disagree. The gates that ARE knowable stay — the owner's recorded
capability and a successful enqueue. Everything past that degrades to
the documented fallback of injecting at the next tool boundary.

A route cannot synchronously know whether another replica will seal:
proving it needs a correlated request/response over pub-sub, and even
that only proves the owner heard, not that it is still streaming when
the arm lands. Four rounds of tightening this boolean each surfaced a
narrower case; the sequence does not converge, so the invariant is now
"the flag describes the durable decision" and an unconfirmed arm logs a
warning instead of rewriting the answer.

Also from this round: a failed disarm publish is retried once and its
outcome reported. `handleSteerCancel` keeps `removed: true` — the steer
really did leave the queue, and saying otherwise would make the client
re-show a chip for a steer that can never arrive — and adds
`disarmed: false` so the residual risk is visible rather than swallowed.
Damage stays bounded regardless: the empty-boundary self-clear disarms
the generation after a single seal.

Tests: +1 pinning the response/durable-flag invariant. 159
packages/api specs green.

* 🧹 fix: Codex round 8 — remove the unverifiable disarm signal

Round 8 found the same over-promise on the disarm side that round 7
corrected on the arm side, so this applies the same answer rather than
patching around it.

The `disarmed: false` field added in round 7 was both unreliable and
unused: a resolved publish is not proof the owner heard it (the
delivery count includes this replica's own facade subscription), and it
was never threaded into `CancelSteerResponse` or read by any client. A
signal that claims a certainty the transport cannot provide is worse
than no signal — it invites callers to trust it.

Removed from the response. The retry stays, because it genuinely
reduces the failure rate, and `noteSteersRemoved` still returns whether
the publish succeeded FOR LOGGING, now documented explicitly as
"published without error", not "the owner disarmed".

Disarm is best effort with a bounded, self-healing failure: if the
clear is lost the owner seals once, the empty-boundary self-clear
disarms the generation, and the turn is persisted `unfinished: true`
rather than silently truncated. Tightening that further needs a
correlated request/response over pub-sub with a timeout — noted on the
PR as the deliberate boundary of this design rather than an oversight.

130 packages/api steering specs green.

* 🧽 fix: Codex round 9 — spend snapshot arms on nonempty drains too

The round-6 scoping fix only cleared the pre-drain snapshot when the
drain came back EMPTY. On a nonempty drain the `finally` cleared just
the drained ids, so a stale arm — typically a cancel whose
cross-replica clear was lost — survived the boundary. It would then
immediately seal the continuation meant to answer the steer that had
just been injected, and land on an empty boundary as
`preempt_incomplete`: the interrupt appears to work, and the answer to
it is truncated.

A boundary that runs has spent its seal, so everything armed at
snapshot time is spent whether or not it came back from the drain. The
`finally` now clears the union of the snapshot and the drained ids.
Arms that land AFTER the snapshot are still spared — their queue items
are live and uninjected, which is the property round 6 added.

Also fixes an api-workspace CI failure of mine: `resume.spec.js` stubs
`GenerationJobManager` wholesale, and the round-3/4 resume work added
two calls (`updateMetadata`, `rearmQueuedPreempts`) the stub did not
define, so 34 specs threw. Stub extended.

Tests: +2 (a nonempty drain clears a stale snapshot arm; a nonempty
drain spares an arm that landed mid-drain). Counterfactually verified —
the stale-arm spec fails against the unfixed drain. 132 packages/api
specs, 60 resume specs green.

* fix: never let a failed preempt subscription reject into the void

registerPreemptSubscription is called detached at both sites, so a
rejected Redis SUBSCRIBE became an unhandled rejection — process-fatal
under Node's default --unhandled-rejections=throw. The comment already
promised this path merely degrades steering; it now does.

Swallowed and logged inside the registration rather than at each call
site, so a future third caller cannot reintroduce the trap. Losing the
channel costs this generation's cross-replica preempts, not the server:
same-replica arming is runtime state and still works, and remote arms
fall back to the next tool boundary.

Verified counterfactually — the new spec surfaces SUBSCRIBE failed as an
unhandled rejection against the unfixed registration.

* docs: state the real blast radius of a failed preempt subscription

LibreChat's own entrypoints install a global unhandledRejection handler
that logs and keeps serving, so the escaping rejection this guards was
never fatal to this server — only to another consumer of @librechat/api
that installs no handler. The fix stands either way; the comment just
should not overstate what it prevents.

* test: cover the cross-replica preempt hop with two manager instances

Every other preempt test runs against a single manager, so the hop that
actually carries an interrupt in production had no coverage: the steer POST
lands on whichever replica the balancer picks, which is usually not the one
generating. Non-owner publishes, owner arms, owner's level-triggered poll
flips — none of that was exercised end to end.

Two GenerationJobManagerClass instances are a faithful replica pair here.
runtimeState and ownedJobs are private instance fields, there is no
module-level mutable state between them, and createStreamServices duplicates
a dedicated subscriber connection per call, so separate OS processes would
exercise the same objects over the same Redis.

Both assertions verified counterfactually against real Redis:
- Deleting the preemptCapable deserialization in RedisJobStore fails this
  with 'Expected: true, Received: undefined' — the exact P1 that shipped past
  every in-memory test and would have made the feature a silent no-op on
  every Redis deployment.
- Dropping the non-owner arm publish fails it with 'Received: false'.

* test: remove the fixed sleeps and vacuity from the cross-replica preempt test

Codex round 11, both findings, both on the test I added last commit.

P2 — the 300ms waits were load-bearing. Redis pub/sub never replays and the
owner's SUBSCRIBE is detached, so on a slow CI worker the publish could land
before anyone was listening and the test would fail against correct code.
Now it republishes until the owner's state converges, which is safe because
arms and clears are idempotent set writes keyed by steerId. Side effect: the
tests got ~10x faster (85ms/57ms vs 929ms/606ms) since they finish on
delivery rather than on a timer.

P3 — afterEach destroyed only the transports, leaving each manager alive in
its own cleanup-interval closure, still working against a dead transport.
Now tracks the managers and awaits destroy(), which disposes the job store
and its timer too. Matches how the rest of this file cleans up.

Fixing the sleeps exposed a third problem codex did not flag: the stale-arm
test could pass vacuously, because an undelivered arm and a fenced one look
identical. It now brackets the stale publish between two control arms — the
first proves the owner is listening before the stale one is sent, the second
proves it has had its chance to arrive.

Verified counterfactually against real Redis, and stable over 5 runs:
- dropping the preemptCapable deserialization fails with 'Received: undefined'
- dropping the non-owner arm publish times out both tests
- removing the generation fence fails the stale test with
  ["control-before", "steer-stale", "control-after"] — which also confirms
  the bracketing orders as intended rather than by luck

* fix: gate interrupt on the OWNER's capability alone, not the route's

Codex round 12. The comment above this gate already said 'the OWNER's
recorded capability, not this replica's probe' — and then the code ANDed in
isSteerPreemptSupported(), which is exactly this replica's probe. The
contradiction dates to the original commit; round 6 made the gate
owner-scoped and wrote that comment without removing the local conjunct.

The route never seals. It enqueues and publishes an arm, neither of which
touches the SDK, so during a rolling deploy a steer landing on an
un-upgraded replica silently lost its interrupt even though the owner could
seal. When the route IS the owner the probe is redundant anyway: the flag it
would consult is the one this process wrote at createJob.

The real degradation path is unchanged and still tested — an owner that
recorded no capability relabels to an ordinary steer. The test that pinned
the local probe asserted an impossible same-replica state (capable metadata
plus an incapable local SDK, when the metadata is written from that probe);
it now pins the mixed-SDK direction instead, and fails with
'Expected: true, Received: false' if the probe is put back.

* fix: reconcile arms at handover, and stop holding the 202 on a publish

Codex round 13, two of three findings.

P2 — rearmQueuedPreempts only ever ADDED. A replica that merely read the job
still installs a facade runtime and subscribes, so it can accept an arm and
then miss the best-effort clear that follows the drain. HITL resume promotes
that facade to owner, the union keeps the orphan, and the first resumed
stream seals on a steer no longer in the queue, drains nothing, and
truncates the resumed answer as preempt_incomplete. acquireResumedJobOwnership
only sets ownedJobs, so nothing else was clearing it. The durable queue is
the sole authority at a handover: arms it does not back are now disarmed and
tombstoned, so an in-flight publish cannot revive them either.

Worth recording that my own independent review raised this and my verifier
refuted it. Codex found it separately; two reviewers converging should have
outweighed one refutation.

P2 — the route awaited the arm publish before answering. The 202 reports
capability, not delivery, so the await could not change the response; it only
exposed the caller to Redis latency after the queue item was already durable.
A client that times out and retries mints a second steer while the first
stays queued, injecting the same instruction twice, whereas a lost publish
merely takes the tool-boundary fallback. Detached, with both outcomes logged.

All three tests verified counterfactually: union-only rearm fails the two new
handover specs, and re-awaiting the publish hangs the stalled-publish spec
until jest kills it.

* fix: snapshot arms before reading the queue at handover

Codex round 14 — a regression from my own round-13 fix, and a worse failure
than the one it corrected.

Round 13 read the durable queue first, then tombstoned any armed id the
snapshot did not back. But approvals.resolve reopens steering before
reconciliation runs, so another replica can commit a preempt steer and
publish its arm while the peek is in flight. That arm is then present locally
but absent from a snapshot taken before the steer existed, so a LIVE
interrupt the route already acknowledged got dropped — and tombstoned, which
blocks the re-arm, making it unrecoverable rather than merely late.

Fixed by inverting the two reads rather than by locking or paying a second
round trip. A steer is durably enqueued BEFORE its arm is published, so any
id in an arms-first snapshot was already queued when it was armed, and the
later peek must observe it unless it has since drained — which is exactly the
orphan this reconciliation exists to drop. Arms landing after the snapshot
are simply not candidates.

Also re-checks runtime identity across the await, since the generation can be
replaced while the queue read is in flight.

New spec injects a steer + arm during the peek and verifies it survives;
against the round-13 ordering it fails with Received array: [].

* fix: bound the cancel disarm wait and fence enqueue to its generation

Codex round 15.

P2 — the cancel awaited its disarm publish unbounded. ioredis queues
commands during an outage rather than rejecting, so that await could hang for
the length of the outage with the steer ALREADY durably cancelled; a client
that gives up then treats the cancel as failed and restores a chip for a
steer that can never produce an applied event. Every successful cancel
publishes, so ordinary steers were exposed too, not only preemptive ones.
Now bounded at 1s, with the publish continuing behind it — its retry and
logging are unchanged, it is just no longer in front of the response. This is
the sibling of round 13's arm-publish finding; I fixed one path and left this
one.

P3 — enqueue was not fenced to the generation the capability decision was
made against. The access checks, file resolution and owner re-read are all
awaits, so the run can be replaced before the enqueue: the item then lands on
the REPLACEMENT queue while the durable preempt flag and the arm still name
the previous epoch, the arm is fenced out at the owner, and the 202 promises
an interrupt that cannot happen. enqueueSteer now takes an expected
generation, mirroring drain/peek, and the Redis path enforces it inside
STEER_ENQUEUE_LUA so the check is atomic with the push rather than racing it.

All three new specs verified counterfactually, including the Lua guard
against real Redis (removing it returns 1 where -1 is required).

* fix: fence the steer to its authorized generation, bound resume setup, keep preempt when Redis parks

Codex round 16, all three findings.

P2 — round 15 fenced the enqueue to owner.createdAt, the RE-READ job. Every
guard above it (ownership, tenant, paused-state, agent ACL) ran against the
job read at the top, so if the run was replaced during those awaits the fence
happily accepted the steer into a generation the request was never authorized
against, carrying the wrong agent's metadata. Now rejects on any mismatch
between the validated job and the re-read.

P2 — resume awaited its steering bookkeeping unbounded, after
approvals.resolve had consumed the action and flipped the job to running, and
outside the resume lifecycle's own try/finally. .catch does not fire on a
promise that never settles, which is what ioredis produces during an outage,
so the client times out, its retry gets a 409 for a spent action, and no
cleanup runs. Bounded at 1s with the writes finishing in the background.

P3 — Redis parks leftover steers inside its terminal-transition Lua, which
projects item fields one by one, so preempt was silently dropped and a steer
recovered from /chat/status lost its interrupting label. Added to both
projections.

All three verified counterfactually, two against real Redis. Worth recording
that my first version of the generation-mismatch test was VACUOUS — it faked
a createdAt matching no live job, so the round-15 enqueue fence rejected it
for the wrong reason and the test passed with the guard removed. Rewritten to
replace the run for real; it now fails with 'Expected 404, Received 202'.
2026-07-30 12:36:13 -04:00
Danny Avila
d8427ffc5e
🛂 test: Cover Tool Approval Workflows End to End (#14427)
* test: cover tool approval workflows end to end

* fix: preserve tool approval state across resume

* fix: preserve agent context in mock stream responses

* fix: preserve nested approvals in collapsed groups
2026-07-26 21:58:25 -04:00
Danny Avila
73699b5c25
⚡ perf: Reduce Agent Chat Startup Latency (#14423)
* perf: reduce agent chat startup latency

* test: align Redis stream readiness assertions

* perf: overlap remaining agent startup work

* perf: persist initial agent job metadata atomically

* test: add agent startup latency benchmark

* fix: harden resumable agent stream lifecycle

* fix: isolate replacement stream lifecycles

* fix: preserve terminal stream epochs
2026-07-25 07:58:20 -04:00