mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 23:13:35 +00:00
* 💬 refactor: ChatForm UI redesign with unified palette
Replace the scattered composer controls (badge row, tools dropdown, attach
menu, floating steer chips) with one system: a searchable palette with
favourites, a composer bar holding active tools plus effort and speech
controls, a tray staging files and manual context, and a queue rail that
buffers messages while a reply is running.
Collapses the 94 development and review-fix commits of PR #14473 into one
commit so the branch could be rebased onto dev once; the upstream merge of
this pull request is a squash merge either way. The pre-rebase history is
kept locally at tag prerebase-backup (afde3e4b9a).
* fix: finish the dev rebase of the composer redesign
The rebase onto dev left four defects the merge itself produced, plus one
review finding the branch had never answered:
- ChatForm's `min-w-0` JSX comment lost its closing brace, so the file did
not parse at all.
- `useSteering` carried both dev's local draft consumer and the host-injected
one under the same name; the host's `consumeDraft` owns the storage key
autosave actually parked on, so the local copy is gone.
- `useUploadOptions` kept dev's `awaitingAgentProvider` gate after the merge
replaced the block that derived `isSavedAgent`; the predicate is back, from
the same `isEphemeralAgentId` the shared allowance helper uses.
- `MessagesView` still reserved a band for the in-flight steer overlay and
read `steerOverlayHeightFamily`, which nothing writes now that pending
steers render inside the streaming reply. The reservation is gone and
`ScrollButton`'s lift is optional; composer panels keep standing the button
down through `composerOverlayCountFamily`.
- The aggregate MCP chip keyed on `mcp:pinned`, which a server literally
named `pinned` also produces (`mcp:${serverName}`), so both entries shared
one React key and one packing slot. It uses a key outside that namespace.
Also drops the locale keys whose only consumers the redesign deleted, ports
dev's extracted-text delivery path into `useAttachExisting` (every
attach-existing surface routes through it now), repoints the mock e2e
built-in helpers at the palette, and repairs test fixtures the merge left
stale (a query mock that blanked agent metadata, a Jotai store mismatch, an
ambiguous tray query, an ephemeral-agent fixture standing in for a saved one).
* test: cover the composer redesign's acceptance scenarios end to end
Twelve mock-harness scenarios, one `@scenario:` tagged test each, over the
behaviours this change is accountable for: the single searchable palette
(open, search, favourite across a reload, keyboard-only operation), the
composer bar and tray (active tool chip, a staged file riding with the
message, a per-turn thinking level that applies to the next turn only), the
queue rail (Enter buffers during a run and drains after it, keyboard reorder
changes the send order, the draft survives run completion), and the composer's
portaled surfaces (Stop reaches the run from inside a portal, a pending tool
approval in the bar resumes the run).
Two contract scenarios are deliberately absent, with reasons: split-pane Stop
isolation is not runtime-reachable because `ChatRoute` mounts one pane, and
code-approval mode mid-run needs an attached stateful code environment the
mock lab does not configure, so dev's `ChatForm.approvalMode` unit spec stays
its coverage.
* fix: make the composer's acceptance run and the whole client suite green
The first verification run of the twelve scenarios passed six and failed six,
and the full client jest suite (which CI runs sharded, unlike the diff-scoped
local precheck) failed fourteen tests. One of each group was a real defect; the
rest were queries and fixtures still aimed at surfaces the redesign replaced.
Defects:
- `useQueueDrain` held its send lock after a revealed queued turn settled, so
the next server-owned row could never be selected. The reveal path now owns
its own lock and releases it when the revealed turn clears.
- `ChatForm` recomputed the during-run slot without `consumeComposer` in its
dependency list, and still carried a landing flag the host prop replaced.
Tests and lab:
- The five `ChatForm.attachments` focus cases now drive the palette dialog and
the tray's quote chips instead of the deleted attach menu and quote popup,
with the same strict focus assertions; the harness gains the `main` landmark
the palette portals into, and the touch fixture a complete `matchMedia`.
- `Palette.spec` no longer closes a hoisted `jest.mock` factory over a later
`const`; `useSteering.spec` follows dev's toast wording.
- The six failing scenarios were repointed at what this lab actually renders:
uploads go through the palette's local-computer row, the reasoning control
needs `interface.parameters`, the queued-row contract is the row and its
drain rather than a control the submit consumes, the palette's keyboard path
uses a built-in the lab grants, the slow-reply wait uses the fixture's chunk
marker, and the approval scenario selects its agent through the builder flow
the passing approval specs use.
* fix: close the eight findings Codex raised on the rebased head
Codex reviewed the rebased heads and reported eight issues; all eight were
real. Two of them concern code this session added.
Behaviour:
- A custom or OpenRouter endpoint that declares neither `paramDefinitions`
nor a `reasoningFormat` no longer gets a reasoning control inferred from the
generic settings list. Capability now has to be declared (a reasoning-key
parameter definition, a non-disabled `reasoningFormat`, or a known reasoning
provider as the default params endpoint); a `disabled` format always
resolves nothing, and every consumer — the composer control, the pre-send
validation and the server — asks the same resolver, so a gateway backed by a
non-reasoning model can no longer offer a level the turn would fail on.
- Restoring a queued attachment keeps its `llmDeliveryPath`, so an extracted
document is routed as its text rather than as raw provider content.
- The placeholder-to-durable reasoning migration now requires this pane's own
live submission, so opening another conversation from a new chat no longer
carries the pending level into it. The created/sync case it was written for
still carries.
- The queue rail treats an acknowledged server-owned row as an ordering
boundary: local rows still reorder among themselves, a server-owned row
cannot move, and a move that would cross one is refused out loud — the
queued-turn API has enqueue, list and cancel only, so a client-side reorder
across that boundary could not be made authoritative.
- The palette resolves an agent's skill scope with `resolveAgentSkillsScope`
instead of treating an empty list as an open catalog, and the attach target
resolves `useResponsesApi` with `resolveUseResponsesApi`, matching what the
run will actually do.
Policy:
- The reasoning-override behaviour moved out of the CJS middleware into
`packages/api/src/modelSpecs/reasoningOverride.ts`, which returns a
discriminated result; the middleware is wiring again, and the behavioural
cases moved with it.
- `showComposerTips` and `showFilesDialog` are Jotai atoms now, the tips
preference keeping its storage key and reaching the composer as a prop.
* fix: make the redesigned composer reachable from the mock e2e helpers
CI's Playwright shard failed four `file-provisioning` tests on this branch
because the shared upload helpers still clicked `#attach-file-menu-button` and
`#attach-file-button`, entry points the redesign removed — attaching now goes
through the palette, which also owns the file input. Both helpers drive the
palette now, `uploadViaLegacyOption` mapping its two destination names onto the
palette rows that route to the code environment and the vector store, and the
specs that drove the old surfaces directly (`code-upload`, `thread-renderers`,
`unified-upload`) follow.
Also closes the last client/server disagreement the acceptance run exposed: a
custom endpoint that declares a reasoning key in `paramDefinitions` had that
declaration merged into the provider defaults and then discarded by model-name
inference, so the composer offered a level for `mock-model-a` that the server
rejected with "Invalid reasoning override". An explicitly declared key now
selects the merged setting before inference, while an undeclared generic custom
or OpenRouter target still resolves nothing.
The twelfth acceptance scenario is rescoped: a paused approval keeps the
composer usable and resumes to completion, but the composer-hosted approval
panel and chip are not asserted, because the pending action is keyed by the
stream's conversation id while the composer stays bound to the new-chat
sentinel — in code byte-identical to dev, and mirroring the write onto the
sentinel did not surface the panel. Tracked at berry-13/LibreChat#39.
* test: finish the acceptance run and keep the client typecheck green
- The thinking scenario read `reasoningOverride` from a `userMessage` wrapper
the chat request does not have; the trace shows the request carrying
`{ key: 'effort', value: 'max' }` at the root, so the assertion reads it
there. Source behaviour was already correct.
- The approval scenario timed out on mobile because the Agent Builder drawer
stayed open after selecting the agent and swallowed the pointer events meant
for the composer; it now closes the sidebar and waits for the drawer to go
inert before sending.
- Two specs typed a named custom endpoint's label as an `EModelEndpoint`, which
CI's client typecheck rejects (the label is exactly what the case is about).
All twelve acceptance scenarios now pass in desktop light, desktop dark and
mobile emulation.
* fix: close the nine findings Codex raised on the verified head
Two were defects in code this session added while closing earlier findings:
- A reasoning key declared in `paramDefinitions` but absent from the generic
settings list was accepted as capability and then never inserted, so the
resolver exposed and validated the generic `reasoning_effort` instead. A
declared key is now appended to the effective settings and resolved before
any fallback, for `effort`, `thinkingLevel` and `thinkingBudget` alike.
- The queue's reorder guard validated from the destination row's own index, so
`target === index` rejected every pointer drag and only keyboard reordering
worked. It validates the dragged row's identity now; the server-owned
boundary, its announcement and the disabled grips are unchanged.
The rest:
- Stop no longer falls through to the document-wide control once a focused
pane has been resolved: focus in an idle pane while another generates used
to abort the other pane's run.
- `PendingSteers` is mounted once in `MessagesView`, above the renderer
switch, so a sent or failed steer keeps its in-flight, retry and
send-as-new surface with `FLAT_THREAD` off, where the recursive renderer
had none and the composer no longer carries a fallback.
- The file-list policy moved out of the CJS route into
`packages/api/src/files/list.ts`, and its cap is `interface`-level config
(`fileListLimit`, default 100) instead of a literal in the route.
- The steer-arm confirmation window is `interface.steerArmConfirmationTimeoutMs`
with its current 10s as the default, travelling the same path as the other
client-consumed timings.
- The pause predicates judge the active branch's tail rather than the last
assistant in array order, so an inactive sibling branch can no longer
enable or disable the inline escalation control.
* fix: move persisted reasoning validation into api package
* fix: restore composer entry points and defaults
* style: normalize reasoning helper formatting
* test: open mobile sidebar before account menu
* test: use settled sidebar layout for mobile account menu
* test: handle empty mobile sidebar in account scenario
* test: target account menu in mobile drawer
* fix: keep composer surfaces and mobile drawer reachable
* fix: let thread approvals receive pointer events
* fix: stabilize composer steering and shortcuts
* fix: commit queue drags and centralize file modal
* fix: return a canceled steer's text to the composer
Canceling a pending steer dropped the words the user had written: the
bubble disappeared and nothing carried the text back, so a reclaim that
the server confirmed still lost the message.
Hand the reclaimed text to `pendingComposerTextByConvoId`, which the
composer already drains on its first render, and append rather than
overwrite when the composer holds a draft the user typed while the cancel
request was in flight.
* fix: close the six findings Codex raised on the pre-rebase head
Pending steers keyed off the Recoil conversation id, which lags the route
during warm-cache navigation, so the surface could sit at the destination
thread's tail while its Cancel and Escalate acted on the source run. It
now keys off the rendered tree, as the mount window already does.
Arming a steer removes the control that was activated, dropping keyboard
focus on the document body. `useSteerEscalate` already exposes `onArmed`
for exactly that; the bubble now passes one that moves focus to the row's
Cancel action.
Opening the file manager from the account menu left its dialog with no
opener: the modal rejects an active element inside a menu, and nothing
supplied a fallback, so closing it returned focus to the body. The opener
is published alongside the open flag and cleared on close, which keeps the
shortcut path's composer-focus behaviour intact.
A steer the server rejected was swept into the queue with no marking, and
the run-end drain then sent those words as a fresh turn without the user
touching Retry or Send as new. Converted failures now carry
`needsExplicitSend`, which the drain skips the same way it skips a row the
rail is mid-edit on.
The queued-turn reconciliation window was a hard-coded 60s, unreachable by
an operator behind a slow proxy or a delayed read replica. It ships as
`interface.queuedTurnReconciliationTimeoutMs` with that value as default,
beside the steer arm timeout it belongs with.
Reasoning-override shape validation moved to
`parseReasoningOverrideRequest` in packages/api, leaving the middleware to
map a rejection onto its own error response.
* fix: keep the composer's reasoning level in step with the parameters panel
`stableConversation` memoized the conversation without any reasoning field
in its key, so editing the level in the Parameters panel produced a new
conversation object that every listed dependency ignored. The composer
went on showing the old level while the next ordinary send already used
the new one.
* test: stop the sidebar helper waiting for a drawer desktop never renders
`ensureSidebarOnScreen` measured `#mobile-drawer` whenever the sidebar had
no Pinned region. Desktop renders no drawer, and `boundingBox()` on a
locator that matches nothing waits for it, so every scenario that reached
that branch at a desktop viewport timed out instead of continuing. Check
for the drawer's absence rather than measuring it.
* test: open the mobile drawer from its inert state, not its geometry
A closed drawer stays mounted and keeps its layout box, so measuring the
Pinned region reported it on screen while every control inside was still
untappable. `inert` is the state the component actually sets when it
closes, so the helper reads that and clicks the header opener before
anything else measures.
* fix: return a canceled steer whole, through one recovery boundary
A steer that stops being a steer was handed back three different ways,
and each one lost something. The immediate cancel wrote `steer.text`
into the composer handoff atom, dropping the attachments, quoted
excerpts, manual skill picks and reasoning override that `settleReclaimed`
then deleted with the pending object. The cancel that won its race
against its own POST cleared the pending marker and restored nothing at
all, so those words disappeared. A definitively rejected steer could
only be resent unchanged, because the row offering to edit it was gone.
Give the recovery one boundary. `useSteerRehome` hands the whole steer
to the composer through the guarded restore the queue rail already uses,
and converts it into an ordinary queued follow-up when the composer
refuses: a draft in the box, another chat on screen, a paused question,
or a composer that unmounted while the reclaim was in flight. The chip
goes either way, since a pending row claiming the server still holds the
words is a lie once they live somewhere else.
The composer is a sibling of the message list rather than its ancestor,
so the thread had no way to reach that restore. The pane's host now
holds the reference and both sides read it there, which also means a
reclaim resolving after the composer is gone refuses instead of writing
into a dead form.
Because the restore refuses rather than overwrites, a draft typed while
a cancel is in flight is no longer glued to the returned steer; the
steer becomes its own queued message and the draft is left as typed.
* test: drive the surfaces these scenarios actually reach
Four scenarios were failing against surfaces the user never touches.
The approval scenario clicked Approve on the thread's copy of the
pending card. The composer's review panel floats over the tail of the
thread and owns the batch submit, so that copy sits behind it and the
click retried until the test timed out. Drive the panel instead; the
thread card stays asserted as the record of the request.
The palette scenario counted rows the moment the dialog painted, then
expected clearing the filter to restore that number. The catalog arrives
in waves: the MCP server list is released by the app-wide warmup timer
and the skills catalog only starts fetching once the dialog's open
effect has run, so the count kept climbing under the test. Wait for the
last wave and for the count to hold still, then snapshot.
The sidebar helper opened the mobile drawer twice through two signals
that disagree. It waited for `inert` to clear, which happens on a state
commit several frames ahead of the slide, then fell through to a
bounding box still reading off-screen and clicked the opener again, into
a machine already opening. One open, through `inert`, then wait for the
drawer to settle in the viewport.
Splitting the cancel scenario follows the recovery change: an empty
composer takes the steer back, and a staged draft keeps the box while
the steer becomes a queued message of its own.
* fix: hold a server-rejected steer for an explicit send
Editing a definitively failed steer falls back to the queue when the
composer will not take it. Queued like any other follow-up, the run-end
drain would send it the moment the reply finished, which turns the
server's refusal into an automatic new turn carrying the payload it
refused.
Mark that fallback the way the rest of the rejected-steer path is
marked, so the row waits for the user to send it.
* test: restore two assertions this branch broke while rewriting specs
Both specs were updated for the redesigned surfaces and both lost a step
they needed, and both fail deterministically in CI as a result.
thread-fold pressed Enter mid-run and waited for a queued row, under a
comment claiming Enter queues by default. It does not: the default
during-run action is steer, so Enter produced a pending steer and the
queued row the next line waits for could never appear. Ctrl/Cmd+Enter is
what takes the other action, which is the one this test needs.
mcp-oauth-readiness asserted on the server card's Cancel button straight
after clicking Connect. Connect opens the OAuth dialog over the panel,
and it is modal, so while it is open the card underneath is marked
aria-hidden and Cancel is not in the accessibility tree at all, which is
why the failure reads "element(s) not found" rather than "not visible".
The rewrite dropped the dismissal the previous flow performed; restore it
as a helper both call sites share.
* test: reach the account menu without the pinned section
The two account-menu tests borrowed `ensureSidebarOnScreen`, which also
settles the Pinned section's placement for the pinned specs. They never
touch that region, and on a run where an earlier test left a pinned
conversation the section exists but has not been laid out yet, so both
tests spent fifteen seconds polling it and failed on the wait rather than
on anything they assert.
Give them a helper that does only what they need: open the mobile drawer
from `inert`, the one signal that distinguishes a closed drawer from an
open one while it stays mounted, then wait for the slide to reach the
viewport. Desktop renders no drawer and needs nothing.
* test: open the drawer by what it shows, not one early read
The helper decided once, from a single read of `inert` taken right after
navigation. On a fresh mobile load the drawer can still be mounting, so
that read reports a state the app has not settled into, the helper
returns believing the drawer is open, and the account menu inside it is
never shown.
Retry against the thing the caller actually needs, the account menu being
visible, re-checking before each attempt. While the drawer is open its
control is labelled "Close sidebar", so the retry cannot toggle it shut.
* fix: keep the composer observing MCP authorization where its tools apply
The old badge row mounted its provider with observeToolAuthorization set
whenever the composer showed ephemeral tools, so on a plain endpoint the
chat kept MCP connection status and discovered tools in the query cache.
The redesigned composer mounts the same provider without the prop, so it
defaults to off and nothing on the chat side loads that state any more.
The Agent Builder's tool library reads that cache. With it empty, every
server reads as Unknown with no tools, and clicking one opens its detail
dialog offering to connect instead of adding it to the agent.
Pass the same gate the badge row used: showTools is the old
showEphemeralBadges expression.
* test: let the palette close before reopening it, and name what is asserted
enableBuiltinTool pressed Escape and returned while the palette was still
animating out. A caller that reopened it straight away resumed the same
popover, scroll offset included, so the Attach rows at the top sat outside
the virtualized window and upload helpers timed out looking for them. Wait
for the dialog to be gone before returning.
The staged-file assertions matched the bare file name, which the palette
now also lists under recent uploads while it closes; scope them to the
tray they are about.
The OAuth dialog dismissal waited for a title built from the server's
display name, but the dialog is titled with the server name.
* feat: make the queued-send lock expiry configurable
The per-pane lock that stops the run-end drain and an explicit Send now
from submitting the same queued message twice gives up a broken claim
after a fixed 60 seconds. Until then every drain and every Send now for
that pane is refused, and a deployment had no way to tune that window.
Expose it as interface.queuedSendLockTimeoutMs beside the queued-turn
reconciliation timeout, defaulting to the same 60 seconds so nothing
changes unless an operator sets it. The interface loader copies each
field explicitly, so the value is added there too; without that line a
configured value would never reach the client.
* fix: decline interrupt-and-steer while reasoning is staged
With a reasoning choice staged, the Interrupt & steer row is disabled,
but its keyboard chord still reached interruptSteer, which fell back to
interruptAndSend: it aborted the run and threw away the partial answer
the action promises to keep.
A staged reasoning choice belongs to a new generation that a steer into
the live one cannot carry, so decline the chord exactly as the row does,
the same way the function already declines while paused on an approval.
The words stay in the composer, where Enter queues them as a turn of
their own.
* fix: wake the queue drain when a late steer retry is queued
Retry marks a failed steer sending, and a sending steer is kept out of
the run-end sweep so its own POST settles it. When the run finishes
while that POST is in flight, the drain spends its one-shot signal on a
queue the steer is not in yet; the late ack or NO_ACTIVE_RUN then moves
it into the queue, where it sat until the user sent it by hand or
another run ended.
Every other late conversion re-posts the last completed run end through
the composer's rewakeDrain. The retry lives in the thread, so the
composer now publishes that function through the same host it already
uses to hand a reclaimed steer back to the draft, and the retry calls it
after converting. It stays a no-op unless the run completed cleanly and
no newer signal is armed, or when no composer is mounted.
* fix: honour thinking-budget ranges wider than 200,000
The request-scoped reasoning override capped thinkingBudget at 200,000
in the shared schema. An operator's paramDefinitions may advertise a
wider range, and the composer renders it, but every value above the cap
was dropped by the control's parse and rejected by the server's before
the range-aware check could run.
That check already exists: resolveReasoningOverride validates the value
against the endpoint's resolved range. Drop the fixed ceiling so the
resolved range is the bound, keeping the integer and lower-bound rules.
The new case covers both directions on a custom endpoint whose range
goes to 500,000.
* refactor: persist the preliminary reasoning override through the shared helper
The agents controller validated the request's reasoning override with
the schema itself before writing the preliminary user message. That is
behavior in the CJS layer, and it duplicated resolvePersistedReasoningOverride,
which @librechat/api already exports and BaseClient already uses for the
same field.
Call the helper instead. A preliminary message is always a fresh turn,
so it is never an edit or a compaction. The two controller specs that
replace @librechat/api with a hand-built mock now take the real helper.
* fix: clear the file manager's state at the session boundary
The file manager's open flag and its opener live in Jotai's default
store, which outlives the authenticated route. A file manager left open
at logout reopened for the next account, still holding a ref to the
previous session's unmounted opener. Reset both in useClearStates, next
to the other composer state this branch already clears there.
* fix: decline interrupt-and-steer on the first turn while reasoning is staged
* test: read the phone layout from the viewport and keep the drawer open
* fix: name both reasons a queued message cannot be edited yet
* test: reuse a published file through the composer palette
* style: move the composer onto the design-system rules
* style: move the file row's new layout styles into classes
* fix: keep the composer's reused thumbnails, focus ring and disabled dimming intact
* fix: bring a steer back when its deferred cancel does not remove it
* fix: treat a run as over only when every pane holding it is idle
* refactor: pass enter-to-send into the composer from its host
* feat: make the palette's recent-file count configurable
* fix: let a file already in the composer be reattached at the limit
* test: hide the chip before replaying a deferred cancel
* refactor: restore the resume request through one packages/api facade
* fix: keep a conversation's staged reasoning across a visit to another model
* fix: keep a connected MCP server's variables configurable from the palette
* fix: resolve every pane-scoped shortcut through the pane holding focus
* refactor: keep every reasoning-override decision out of the /api wiring
* perf: skip paging the skill catalog for an agent that can show no skill
* test: type the reasoning snapshot the override helper records
* test: pin the applied budget and the untouched parameters in the override specs
* style: give the stop button the send button's shared submit recipe
* fix: name only the newline key when no chord sends
* fix: name the stop key only once the stop control can act
* fix: clear every staged reasoning selection at sign-out
* fix: stop advertising send-now while staged reasoning forces a queue
* fix: resolve the reasoning target of an ephemeral agent in the composer and the request
* test: type the staged reasoning the sign-out spec seeds
* fix: check the composer will take a queued row before discarding its parked copy
* style: carry the composer's classes onto Tailwind v4
* style: sort the composer's classes in Tailwind v4 order
* fix: return a refused queued row to the durable queue instead of memory
* i18n: localize the whole attachment-limit message
* refactor: validate the reasoning override inside the packages/api facade
* style: give the composer's submit slot its own IconButton variant
* test: pin the submit slot to IconButton's submit variant
* test: stub the seen-state hook the rebased MessagesView now calls
* fix: turn file search on for an embedded file attached through the shared hook
* chore: record the composer's design-rule counts against the rebased base
* fix: keep the composer's container query inert under Tailwind v4
* test: point the v4 focus scenarios at the redesigned composer's controls
* fix: keep a re-queued row on its own lineage after the run ends
* test: assert a re-queued row carries its lineage
* refactor: pass the dictation preferences into the composer from its host
* test: read the usage atom from the store the ask hook renders under
* fix: read a pending steer's pause from its own pane's branch
* a11y: keep focus on the quote chips until the last one is removed
* fix: keep staged reasoning for the next message across a manual compaction
* test: expect focus on the remaining quote chip after removing one of two
* style: sort the imports the canary rebase merged
* refactor: remove the side-panel files table the account menu replaced
* fix: settle a retried steer's late ack against a racing cancel
A cancel clicked while the retry POST was in flight left the optimistic
marker unread: the ack re-minted a live chip and injected words the user
had asked back, and a failed retry after that cancel lost them entirely,
because the failure path maps chips the optimistic cancel had removed.
The ack path now defers to the marker the way the composer's does, the
failure path re-adds the chip before settling it, and the queued row a
late ack produces binds its recovery identity only under protocol v2,
the same default the composer's degradation path trusts.
* fix: strip a replayed reasoning override that no longer validates on resume
A paused turn's reasoning override is replayed onto its resume request
and then validated as if it were fresh client input, so an operator
changing the endpoint's reasoning config between pause and approve made
the checkpoint permanently unresumable with a 400. The replay now marks
the request, and a replayed override that fails validation is stripped
so the resume proceeds on defaults; a fresh override that fails is still
rejected.
* refactor: derive the enforced spec fields inside the modelSpecs facade
The set of keys an enforced model spec locks is the input contract of
the reasoning-override refusal rule, and deriving it in the route
middleware left the policy untestable from packages/api and free to
diverge from the facade that owns the rest of the preset application.
applyModelSpecPreset now returns the enforced fields beside the private
ones it applies, and the middleware consumes them.
* fix: restore the pre-override parameters when a resume degrades
Stripping a replayed reasoning override that no longer validates removed
only the metadata field, while the resume had already replayed the paused
turn's model parameters with the override applied into the endpoint
option, so the provider still received what validation had refused. The
strip now inverts the application from the replayed base: the recorded
pre-override value returns and the thinking flag the override forced on
comes off.
* fix: resolve the effective responses route for upload targets
The attach target returned the conversation or agent's saved
useResponsesApi flag, so an Azure chat auto-routed to the Responses API
by responsesApiRouting preflighted its uploads against the chat
completions route and lost provider-document uploads. The resolver now
applies resolveEffectiveUseResponsesApi with the endpoint, model,
web-search state and routing policy, the same resolution the during-run
upload target makes.
* fix: name the interrupt chord in the hint only while it still interrupts
The during-run hint advertised Alt+Enter Interrupt & send even when the
chord had been claimed: a submitMessage rebound to Alt+Enter, a chord
yielded to a global shortcut, or shortcuts disabled altogether each make
the keydown resolver return something else. The hint now reads the same
resolver verdict the during-run send button does and omits the chord it
no longer owns.
* test: assert the unseen count the scenario adds rather than a zero baseline
The tab title counts every unseen conversation, and a shared
verification run leaves legitimately-unseen conversations from earlier
scenarios behind, so the three title-count scenarios failed on other
scenarios' leftovers instead of the behavior they seed. Each now reads
its baseline and asserts the count it adds and clears.
* test: hold the unseen count to its direction on the mobile project
The away poll that lights a dot also discovers conversations the client
had not counted yet, so on the mobile project the title count can jump
past any baseline taken mid-test. Opening still has to clear this
conversation's contribution and an away reply still has to raise the
count, which is what the assertions now hold.
* test: enable the MCP Apps server through the redesigned palette
The MCP Apps integration spec still drove the old composer's MCP
Servers menu, which the redesign replaced with toggleable rows in the
attach-and-tools palette; the lane failed on the missing menu on every
run. The helper now opens the palette, waits for the server wave, and
flips the row's pressed state.
* fix: preserve numeric reasoning defaults on blur
* test: reopen persisted code cards before checking highlights
* test: follow the current file manager navigation
* fix: offer only reasoning options a request override can carry
* fix: keep the favorited copy when skills share a name
* fix: restore composer focus by the interaction that closed the palette
* test: expect a keyboard close to refocus the composer on every project
* chore: re-record the design-rule backlog on the rebased composer
* chore: drop the files table and label the rebase brought back
* test: tolerate other projects' unseen conversations when marking one unread
* test: open the file table header scenario from the account menu
* fix: let the stop shortcut reach a hidden stop control from a dialog
* fix: keep staged reasoning out of tool-limit recovery and MCP App turns
* fix: degrade a queued turn's stale reasoning override instead of dead-lettering it
* fix: refresh the attach target when web search changes the route
* fix: re-arm the run-end drain after moving a steer to the queue
* test: hover the MCP row before reaching its revealed Connect action
* test: pick the parts scenario's MCP server from the tools palette
* fix: cancel a recovered row's parked steer only after the composer takes it
A recovered row's only durable copy is the steer parked on the server, and
once its run has ended nothing can re-create it, so a composer that refused
the row after the cancel left the words in memory until the next reload.
The composer now takes the words first; a cancel that then fails holds the
row out of the run-end drain instead of sending it a second time.
* fix: name only the idle composer affordances that act
* fix: keep steer escalation off until the pane owns the rendered conversation
The branch tail that decides whether a run is paused is keyed by the pane's
conversation, which can still name the previous chat while a cached one
renders. Until the pane catches up, a pause on any cached branch now keeps
Escalate disabled instead of reading a missing or foreign tail as unpaused.
* fix: route the host's Enter and auto-send preferences into the composer's nested hooks
ChatForm takes enterToSend and autoSendText from its host, but useTextarea,
DuringRunSendButton and both speech engines still read the store atoms, so a
host whose values differ advertised one behavior while the handlers ran
another. Pass the host values through instead.
* fix: settle a browser dictation take only after its final transcript lands
stopListening resolves on the recognizer's end event, after the final result
reached useSpeechRecognition's reducer but before React committed it, so the
take settled while the composer still held the interim text. Settle from an
effect declared after the transcript effect, off a state update queued behind
the result.
* fix: drop a restored reasoning override the current setting no longer offers
The staged override was only checked for a matching parameter key, so a value
restored after a capability change on the same target stayed staged, blocked
live steering, and was dropped at send time. Validate it against the setting's
options and range.
* test: scan the attach and tools palette for accessibility
The a11y suite still opened the removed tools dropdown. Open the palette
instead, and add a scenario that runs Axe over the open palette.
* fix: make the palette's virtualized list the grid that owns its rows
The rows sat under a wrapper grid behind react-virtualized's scroller, which
always renders aria-label and aria-readonly and so could not be made
presentational. Axe reported both the disallowed attribute and the grid's
invalid child. The scroller is now the labelled grid, with a rowgroup holding
the rows.
* fix: state the palette grid's row total and each row's position
Only the rows in view are mounted, so without aria-rowcount and aria-rowindex
assistive technology reads the rendered window as the whole grid.
* fix: key an empty destination's pending steers to the route
An empty tree has no message naming its conversation, so the pending-steer
surface fell back to the chat context, which still names the source chat
during warm navigation, and its controls acted on the source run. Fall back
to the chat's messages key, which follows the route.
* fix: keep assistant uploads available when the file config fails
Assistants upload directly and never needed the destination policy, which is
why the replaced AttachFileChat rendered their uploader outside the policy
gate. Restore that exception so a failed or paused file config request does
not remove every attachment entry.
(cherry picked from commit f568a4ba30)
Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14473
6555 lines
232 KiB
JavaScript
6555 lines
232 KiB
JavaScript
const { EventEmitter } = require('events');
|
|
|
|
const mockLogger = {
|
|
debug: jest.fn(),
|
|
warn: jest.fn(),
|
|
error: jest.fn(),
|
|
info: jest.fn(),
|
|
};
|
|
|
|
const mockGenerationJobManager = {
|
|
isRedis: false,
|
|
detachedAgentEventActionStoreMode: 'process_local',
|
|
supportsDetachedAgentEventActions: true,
|
|
createJob: jest.fn(),
|
|
getJob: jest.fn(),
|
|
emitError: jest.fn(),
|
|
emitChunk: jest.fn(),
|
|
emitDone: jest.fn(),
|
|
claimTerminalJob: jest.fn(),
|
|
publishTerminalClaim: jest.fn(),
|
|
finishTerminalJob: jest.fn(),
|
|
completeJob: jest.fn(),
|
|
beginProviderExecution: jest.fn(),
|
|
markProviderExecutionDrained: jest.fn(),
|
|
failPausePersistence: jest.fn(),
|
|
getResumeState: jest.fn(),
|
|
getJobStore: jest.fn(),
|
|
updateMetadata: jest.fn(),
|
|
persistAgentEventDetachedTerminalEvidence: jest.fn(),
|
|
claimGeneration: jest.fn(),
|
|
resumeClaimedGeneration: jest.fn(),
|
|
takeoverGeneration: jest.fn(),
|
|
releaseGeneration: jest.fn(),
|
|
hasJob: jest.fn(),
|
|
approvals: {
|
|
ownsPausePersistence: jest.fn(),
|
|
finishPausePersistence: jest.fn(),
|
|
},
|
|
steering: {
|
|
closeAndDrain: jest.fn(),
|
|
park: jest.fn(),
|
|
consumeRecovered: jest.fn(),
|
|
},
|
|
};
|
|
|
|
const DEFAULT_OWNED_CLAIM = Object.freeze({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimedAt: 100,
|
|
claimToken: 'claim-token',
|
|
});
|
|
|
|
function wonGenerationClaim(overrides = {}) {
|
|
return { claimed: true, existing: { ...DEFAULT_OWNED_CLAIM, ...overrides } };
|
|
}
|
|
|
|
const mockCheckAndIncrementPendingRequest = jest.fn();
|
|
const mockGetFailedTurnTraceFields = jest.fn();
|
|
const mockDecrementPendingRequest = jest.fn();
|
|
const mockGetViolationInfo = jest.fn(() => ({
|
|
type: 'concurrent',
|
|
limit: 2,
|
|
pendingRequests: 3,
|
|
score: 1,
|
|
}));
|
|
const mockFilterPersistableAbortContent = jest.fn((content) =>
|
|
content.filter((part) => part?.type !== 'tool_call'),
|
|
);
|
|
const mockGetConvo = jest.fn();
|
|
const mockGetMessages = jest.fn();
|
|
const mockStampConvoLastResponse = jest.fn().mockResolvedValue(undefined);
|
|
const mockSaveMessage = jest.fn();
|
|
const mockSaveConvo = jest.fn();
|
|
const mockAppendConvoMessageReference = jest.fn();
|
|
const mockIsAgentTriggerPrincipalActive = jest.fn();
|
|
const mockIsSubagentOwnerAdmissible = jest.fn();
|
|
const mockAcquireEventChildGenerationLease = jest.fn();
|
|
const mockIsScheduleFireRequest = jest.fn();
|
|
const mockExemptFromConcurrencyLimiter = jest.fn();
|
|
const mockRecordScheduleOutcome = jest.fn();
|
|
const mockIsScheduleLive = jest.fn();
|
|
const mockDeleteAgentCheckpoint = jest.fn();
|
|
const mockSettleAgentQueuedTurnExecutionAdmission = jest.fn();
|
|
const mockVerifyAgentQueuedTurnExecutionAdmission = jest.fn();
|
|
const mockExecuteAgentEventActor = jest.fn();
|
|
const mockResumeAgentEventActor = jest.fn();
|
|
const mockCreateAgentEventActorTurn = jest.fn((input, dependencies) => {
|
|
let historyToken;
|
|
let historyOwner;
|
|
return {
|
|
run: async () => {
|
|
if (input.strategy === 'checkpoint') {
|
|
const result =
|
|
input.checkpoint.kind === 'resume'
|
|
? await mockResumeAgentEventActor(input.checkpoint.input, dependencies.actor)
|
|
: await mockExecuteAgentEventActor(input.checkpoint.input, dependencies.actor);
|
|
return { adapter: 'checkpoint', ...result };
|
|
}
|
|
const token = 'event-actor-history-token';
|
|
const acquired = await dependencies.history.begin({ ...input.history.owner, token });
|
|
if (!acquired) {
|
|
throw Object.assign(new Error('The event actor is temporarily unavailable'), {
|
|
code: 'EVENT_ACTOR_NOT_READY',
|
|
status: 409,
|
|
});
|
|
}
|
|
historyToken = token;
|
|
historyOwner = input.history.owner;
|
|
try {
|
|
await input.history.persistToken(token);
|
|
} catch (error) {
|
|
historyToken = undefined;
|
|
historyOwner = undefined;
|
|
await dependencies.history.complete({ ...input.history.owner, token });
|
|
throw error;
|
|
}
|
|
return { adapter: 'history', value: await input.history.invoke() };
|
|
},
|
|
historyPersisted: async () => {
|
|
if (historyToken == null) {
|
|
return;
|
|
}
|
|
const token = historyToken;
|
|
historyToken = undefined;
|
|
await dependencies.history.complete({ ...historyOwner, token });
|
|
},
|
|
};
|
|
});
|
|
const mockCreateAgentEventActorDetachedActionLifecycle = jest.fn(() => undefined);
|
|
const mockFindAgentEventAppliedAction = jest.fn();
|
|
const mockResolveAgentTurnExecutionPlan = jest.fn((input) => {
|
|
let origin = 'user';
|
|
if (input.isSchedule) {
|
|
origin = 'schedule';
|
|
} else if (input.isEvent || input.event) {
|
|
origin = 'event';
|
|
}
|
|
const checkpointCompatible =
|
|
input.event?.binding != null &&
|
|
input.event?.expectedAction != null &&
|
|
input.checkpointerType !== 'memory' &&
|
|
(!input.canPause || input.durableEventActorSuspensions);
|
|
let strategy = 'history';
|
|
if (input.isNewConversation) {
|
|
strategy = 'fresh';
|
|
} else if (checkpointCompatible) {
|
|
strategy = 'checkpoint';
|
|
}
|
|
return {
|
|
origin,
|
|
strategy,
|
|
conversationId: input.conversationId,
|
|
parentMessageId: input.parentMessageId,
|
|
canPause: input.canPause,
|
|
expectedAction: input.event?.expectedAction,
|
|
binding: input.event?.binding,
|
|
};
|
|
});
|
|
/** Faithful stand-in for the graph-context action recorder: first observed
|
|
* tool end with a name becomes the receipt. Matching fences are unit-tested
|
|
* against the real implementation in packages/api. */
|
|
const mockCreateAgentEventActionRecorder = jest.fn(() => {
|
|
let receipt;
|
|
return {
|
|
observeToolEnd: (data) => {
|
|
if (receipt == null && data?.output?.name != null) {
|
|
receipt = {
|
|
toolName: data.output.name,
|
|
...(data.output.tool_call_id == null ? {} : { toolCallId: data.output.tool_call_id }),
|
|
};
|
|
}
|
|
},
|
|
read: () => receipt,
|
|
};
|
|
});
|
|
const mockGetAgentEventActorSnapshot = jest.fn();
|
|
const mockCommitAgentEventActorState = jest.fn();
|
|
const mockBeginAgentEventActorLegacyTurn = jest.fn();
|
|
const mockCompleteAgentEventActorLegacyTurn = jest.fn();
|
|
const mockRecordAgentEventActorReconciliation = jest.fn();
|
|
const mockResolveAgentEventActorReconciliation = jest.fn();
|
|
const mockClearAgentEventActorReconciliation = jest.fn();
|
|
const mockAdmitAgentEventActorAction = jest.fn();
|
|
const mockReleaseAgentEventActorAction = jest.fn();
|
|
const mockHasAgentEventActorActionAdmission = jest.fn();
|
|
const mockGetAgentEventActorReceipt = jest.fn();
|
|
const mockGetAgentEventActorDetachedAction = jest.fn();
|
|
const mockClaimAgentEventActorSuspension = jest.fn();
|
|
const mockSettleAgentEventActorSuspension = jest.fn();
|
|
const mockStartupTelemetry = {
|
|
mark: jest.fn(),
|
|
setStreamId: jest.fn(),
|
|
recordGenerationEvent: jest.fn(),
|
|
end: jest.fn(),
|
|
};
|
|
const mockGetAgentStartupTelemetry = jest.fn(() => mockStartupTelemetry);
|
|
const mockAcceptAgentStartupTelemetry = jest.fn();
|
|
let mockMCPContexts = new WeakMap();
|
|
|
|
const mockCreateMCPRequestContext = jest.fn(() => ({
|
|
connections: new Map(),
|
|
pending: new Map(),
|
|
cleanupStarted: false,
|
|
cleanupOnResponse: false,
|
|
responseCleanupAttached: false,
|
|
}));
|
|
const mockGetMCPRequestContext = jest.fn((req) => {
|
|
if (!req) {
|
|
return undefined;
|
|
}
|
|
|
|
let context = mockMCPContexts.get(req);
|
|
if (!context) {
|
|
context = mockCreateMCPRequestContext();
|
|
mockMCPContexts.set(req, context);
|
|
}
|
|
|
|
return context.cleanupStarted ? undefined : context;
|
|
});
|
|
const mockCleanupMCPRequestContext = jest.fn(async (context) => {
|
|
if (!context || context.cleanupStarted) {
|
|
return;
|
|
}
|
|
|
|
context.cleanupStarted = true;
|
|
const connections = new Set(context.connections.values());
|
|
const settled = await Promise.allSettled(context.pending.values());
|
|
for (const result of settled) {
|
|
if (result.status === 'fulfilled' && result.value) {
|
|
connections.add(result.value);
|
|
}
|
|
}
|
|
|
|
await Promise.allSettled(Array.from(connections).map((connection) => connection.disconnect?.()));
|
|
context.connections.clear();
|
|
context.pending.clear();
|
|
});
|
|
const mockCleanupMCPRequestContextForReq = jest.fn(async (req) => {
|
|
const context = mockMCPContexts.get(req);
|
|
if (!context) {
|
|
return;
|
|
}
|
|
|
|
try {
|
|
await mockCleanupMCPRequestContext(context);
|
|
} finally {
|
|
mockMCPContexts.delete(req);
|
|
}
|
|
});
|
|
|
|
jest.mock('@librechat/data-schemas', () => ({
|
|
logger: mockLogger,
|
|
}));
|
|
|
|
jest.mock('@librechat/api', () => ({
|
|
getSteerRecoveryFailure: jest.requireActual(
|
|
'../../../../../packages/api/src/stream/SteerRecovery',
|
|
).getSteerRecoveryFailure,
|
|
getAgentErrorMetadata: (...args) =>
|
|
jest.requireActual('@librechat/api').getAgentErrorMetadata(...args),
|
|
sendEvent: jest.fn(),
|
|
/** Real, because whether a skipped-persistence turn may raise an indicator is under test. */
|
|
isAnnounceableReply: jest.requireActual('@librechat/api').isAnnounceableReply,
|
|
announceReply: jest.requireActual('@librechat/api').announceReply,
|
|
announceErrorTurn: jest.requireActual('@librechat/api').announceErrorTurn,
|
|
persistedReasoningOverrideFields:
|
|
jest.requireActual('@librechat/api').persistedReasoningOverrideFields,
|
|
logAgentMemorySnapshot: jest.fn(),
|
|
isScheduleFireRequest: (...args) => mockIsScheduleFireRequest(...args),
|
|
exemptFromConcurrencyLimiter: (...args) => mockExemptFromConcurrencyLimiter(...args),
|
|
toPendingSteer: jest.fn((item) => item),
|
|
/** Recorded onto the job so the steer route can honour the OWNING replica's
|
|
* seal capability rather than its own probe. */
|
|
isSteerPreemptSupported: jest.fn(() => true),
|
|
isSteerTerminalContinuationSupported: jest.fn(() => false),
|
|
buildRecoveredSteerPayload: jest.fn((text, files) => {
|
|
if (typeof text !== 'string' || (files != null && !Array.isArray(files))) {
|
|
return null;
|
|
}
|
|
const fileIds = [...new Set((files ?? []).map((file) => file?.file_id))];
|
|
if (fileIds.some((id) => typeof id !== 'string' || id.length === 0)) {
|
|
return null;
|
|
}
|
|
return { text, fileIds: fileIds.sort() };
|
|
}),
|
|
getViolationInfo: (...args) => mockGetViolationInfo(...args),
|
|
buildMessageFiles: jest.fn(() => []),
|
|
resolveTitleTiming: jest.fn(() => 'immediate'),
|
|
createConvoPersistenceSignal: jest.requireActual('@librechat/api').createConvoPersistenceSignal,
|
|
recoverTurnMessageReference: jest.requireActual('@librechat/api').recoverTurnMessageReference,
|
|
resolveConversationAnchor: jest.requireActual('@librechat/api').resolveConversationAnchor,
|
|
resolveRunCodeWorkspaces: jest.requireActual('@librechat/api').resolveRunCodeWorkspaces,
|
|
AttachmentStorageError: jest.requireActual('@librechat/api').AttachmentStorageError,
|
|
encodeAndFormatImages: jest.requireActual('@librechat/api').encodeAndFormatImages,
|
|
getCodeWorkspaceSelectionErrorDetails:
|
|
jest.requireActual('@librechat/api').getCodeWorkspaceSelectionErrorDetails,
|
|
shouldPersistCodeWorkspaceInitializationError:
|
|
jest.requireActual('@librechat/api').shouldPersistCodeWorkspaceInitializationError,
|
|
resolvePersistableCodeEnvironmentDecision: (...args) =>
|
|
jest.requireActual('@librechat/api').resolvePersistableCodeEnvironmentDecision(...args),
|
|
getSafeErrorMetadata: jest.requireActual('@librechat/api').getSafeErrorMetadata,
|
|
getSafeErrorText: jest.requireActual('@librechat/api').getSafeErrorText,
|
|
resolveFailedTurnContent: jest.requireActual('@librechat/api').resolveFailedTurnContent,
|
|
getFailedTurnTraceFields: (...args) => mockGetFailedTurnTraceFields(...args),
|
|
GenerationJobManager: mockGenerationJobManager,
|
|
getReferencedQuotes: jest.fn((quotes) => {
|
|
if (!Array.isArray(quotes)) {
|
|
return null;
|
|
}
|
|
const normalized = quotes
|
|
.filter((quote) => typeof quote === 'string' && quote.trim().length > 0)
|
|
.map((quote) => quote.trim());
|
|
return normalized.length > 0 ? normalized : null;
|
|
}),
|
|
cleanupMCPRequestContext: (...args) => mockCleanupMCPRequestContext(...args),
|
|
createMCPRequestContext: (...args) => mockCreateMCPRequestContext(...args),
|
|
getMCPRequestContext: (...args) => mockGetMCPRequestContext(...args),
|
|
filterPersistableAbortContent: (...args) => mockFilterPersistableAbortContent(...args),
|
|
cleanupMCPRequestContextForReq: (...args) => mockCleanupMCPRequestContextForReq(...args),
|
|
decrementPendingRequest: (...args) => mockDecrementPendingRequest(...args),
|
|
sanitizeMessageForTransmit: jest.fn((message) => message),
|
|
checkAndIncrementPendingRequest: (...args) => mockCheckAndIncrementPendingRequest(...args),
|
|
getAgentStartupTelemetry: (...args) => mockGetAgentStartupTelemetry(...args),
|
|
acceptAgentStartupTelemetry: (...args) => mockAcceptAgentStartupTelemetry(...args),
|
|
isUnpersistedPreliminaryParent: async ({
|
|
userId,
|
|
conversationId,
|
|
parentMessageId,
|
|
getMessages,
|
|
}) => {
|
|
if (typeof parentMessageId !== 'string' || !parentMessageId.endsWith('_')) {
|
|
return false;
|
|
}
|
|
|
|
const filter = { user: userId, messageId: parentMessageId };
|
|
if (conversationId && conversationId !== 'new') {
|
|
filter.conversationId = conversationId;
|
|
}
|
|
|
|
const messages = await getMessages(filter, '_id');
|
|
return messages.length === 0;
|
|
},
|
|
deleteAgentCheckpoint: (...args) => mockDeleteAgentCheckpoint(...args),
|
|
createAgentEventActorTurn: (...args) => mockCreateAgentEventActorTurn(...args),
|
|
createAgentEventActorDetachedActionLifecycle: (...args) =>
|
|
mockCreateAgentEventActorDetachedActionLifecycle(...args),
|
|
parseAgentEventActorDetachedCompletion: (value) => (value?.version === 1 ? value : undefined),
|
|
EVENT_ACTOR_DETACHED_COMPLETION_SOURCE: 'librechat-event-actor',
|
|
EVENT_ACTOR_DETACHED_COMPLETION_TYPE: 'librechat.event_actor.detached_completion',
|
|
findAgentEventAppliedAction: (...args) => mockFindAgentEventAppliedAction(...args),
|
|
createAgentEventActionRecorder: (...args) => mockCreateAgentEventActionRecorder(...args),
|
|
resolveAgentTurnExecutionPlan: (...args) => mockResolveAgentTurnExecutionPlan(...args),
|
|
isHITLEnabled: (policy) => policy?.enabled === true,
|
|
agentRequestsAskUserQuestion: (agent) =>
|
|
agent?.toolDefinitions?.some((tool) => tool?.name === 'ask_user_question') === true,
|
|
isAgentEventRetentionActive: (expiredAt) =>
|
|
expiredAt == null || new Date(expiredAt).getTime() > Date.now(),
|
|
createMCPRuntimeRequestBody: ({
|
|
messageId,
|
|
conversationId,
|
|
parentMessageId,
|
|
codeEnvironmentMode,
|
|
codeWorkspaces,
|
|
}) => ({
|
|
messageId,
|
|
conversationId,
|
|
parentMessageId,
|
|
...(codeEnvironmentMode !== undefined ? { codeEnvironmentMode } : {}),
|
|
...(codeWorkspaces !== undefined ? { codeWorkspaces } : {}),
|
|
}),
|
|
}));
|
|
|
|
jest.mock('~/server/cleanup', () => ({
|
|
disposeClient: jest.fn(),
|
|
clientRegistry: null,
|
|
requestDataMap: {
|
|
set: jest.fn(),
|
|
},
|
|
}));
|
|
|
|
jest.mock('~/server/middleware', () => ({
|
|
handleAbortError: jest.fn(() => Promise.resolve()),
|
|
}));
|
|
|
|
jest.mock('~/cache', () => ({
|
|
logViolation: jest.fn(),
|
|
}));
|
|
|
|
jest.mock('~/models', () => ({
|
|
saveMessage: (...args) => mockSaveMessage(...args),
|
|
saveConvo: (...args) => mockSaveConvo(...args),
|
|
appendConvoMessageReference: (...args) => mockAppendConvoMessageReference(...args),
|
|
getMessages: (...args) => mockGetMessages(...args),
|
|
getConvo: (...args) => mockGetConvo(...args),
|
|
getAgentEventActorSnapshot: (...args) => mockGetAgentEventActorSnapshot(...args),
|
|
commitAgentEventActorState: (...args) => mockCommitAgentEventActorState(...args),
|
|
beginAgentEventActorLegacyTurn: (...args) => mockBeginAgentEventActorLegacyTurn(...args),
|
|
completeAgentEventActorLegacyTurn: (...args) => mockCompleteAgentEventActorLegacyTurn(...args),
|
|
recordAgentEventActorReconciliation: (...args) =>
|
|
mockRecordAgentEventActorReconciliation(...args),
|
|
resolveAgentEventActorReconciliation: (...args) =>
|
|
mockResolveAgentEventActorReconciliation(...args),
|
|
clearAgentEventActorReconciliation: (...args) => mockClearAgentEventActorReconciliation(...args),
|
|
admitAgentEventActorAction: (...args) => mockAdmitAgentEventActorAction(...args),
|
|
releaseAgentEventActorAction: (...args) => mockReleaseAgentEventActorAction(...args),
|
|
hasAgentEventActorActionAdmission: (...args) => mockHasAgentEventActorActionAdmission(...args),
|
|
getAgentEventActorReceipt: (...args) => mockGetAgentEventActorReceipt(...args),
|
|
getAgentEventActorDetachedAction: (...args) => mockGetAgentEventActorDetachedAction(...args),
|
|
claimAgentEventActorSuspension: (...args) => mockClaimAgentEventActorSuspension(...args),
|
|
settleAgentEventActorSuspension: (...args) => mockSettleAgentEventActorSuspension(...args),
|
|
stampConvoLastResponse: (...args) => mockStampConvoLastResponse(...args),
|
|
isAgentTriggerPrincipalActive: (...args) => mockIsAgentTriggerPrincipalActive(...args),
|
|
isSubagentOwnerAdmissible: (...args) => mockIsSubagentOwnerAdmissible(...args),
|
|
}));
|
|
|
|
jest.mock('~/server/services/Endpoints/agents/eventChildLease', () => ({
|
|
acquireEventChildGenerationLease: (...args) => mockAcquireEventChildGenerationLease(...args),
|
|
}));
|
|
|
|
jest.mock('~/server/services/Schedules', () => ({
|
|
recordScheduleOutcome: (...args) => mockRecordScheduleOutcome(...args),
|
|
isScheduleLive: (...args) => mockIsScheduleLive(...args),
|
|
}));
|
|
|
|
jest.mock('~/server/services/Agents/triggers', () => ({
|
|
settleAgentQueuedTurnExecutionAdmission: (...args) =>
|
|
mockSettleAgentQueuedTurnExecutionAdmission(...args),
|
|
verifyAgentQueuedTurnExecutionAdmission: (...args) =>
|
|
mockVerifyAgentQueuedTurnExecutionAdmission(...args),
|
|
}));
|
|
|
|
const AgentController = require('../request');
|
|
const { AttachmentStorageError, encodeAndFormatImages } = require('@librechat/api');
|
|
const { ErrorTypes } = require('librechat-data-provider');
|
|
const { disposeClient: mockDisposeClient } = require('~/server/cleanup');
|
|
const { getMCPRequestContext } = require('~/server/services/MCPRequestContext');
|
|
|
|
function createResumableResponse() {
|
|
const res = new EventEmitter();
|
|
res.headersSent = false;
|
|
res.writableEnded = false;
|
|
res.finished = false;
|
|
res.destroyed = false;
|
|
res.json = jest.fn(() => {
|
|
res.headersSent = true;
|
|
res.writableEnded = true;
|
|
res.finished = true;
|
|
res.emit('finish');
|
|
return res;
|
|
});
|
|
res.status = jest.fn(() => res);
|
|
res.set = jest.fn(() => res);
|
|
return res;
|
|
}
|
|
|
|
function nextTick() {
|
|
return new Promise((resolve) => setImmediate(resolve));
|
|
}
|
|
|
|
describe('ResumableAgentController resume metadata', () => {
|
|
beforeEach(() => {
|
|
jest.clearAllMocks();
|
|
mockMCPContexts = new WeakMap();
|
|
mockCheckAndIncrementPendingRequest.mockResolvedValue({ allowed: true });
|
|
mockDecrementPendingRequest.mockResolvedValue(undefined);
|
|
mockGetConvo.mockResolvedValue({ createdAt: '2026-06-07T00:00:00.000Z' });
|
|
mockGetMessages.mockResolvedValue([]);
|
|
mockIsAgentTriggerPrincipalActive.mockResolvedValue(true);
|
|
mockIsSubagentOwnerAdmissible.mockResolvedValue(true);
|
|
mockAcquireEventChildGenerationLease.mockResolvedValue(jest.fn());
|
|
mockIsScheduleFireRequest.mockImplementation((req) => req?._isScheduledFire === true);
|
|
mockExemptFromConcurrencyLimiter.mockImplementation(
|
|
(req) => req?._isScheduledFire === true && req?._isManualScheduledFire !== true,
|
|
);
|
|
mockRecordScheduleOutcome.mockResolvedValue(true);
|
|
mockIsScheduleLive.mockResolvedValue(true);
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
metadata: {
|
|
checkpointNamespace: '1000',
|
|
providerExecutionId: 'provider-segment-1',
|
|
providerDrained: true,
|
|
},
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
mockGenerationJobManager.getResumeState.mockResolvedValue(null);
|
|
mockGenerationJobManager.getJobStore.mockReturnValue({
|
|
getJob: jest.fn().mockResolvedValue(null),
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.updateMetadata.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.isRedis = false;
|
|
mockGenerationJobManager.detachedAgentEventActionStoreMode = 'process_local';
|
|
mockGenerationJobManager.supportsDetachedAgentEventActions = true;
|
|
mockGenerationJobManager.persistAgentEventDetachedTerminalEvidence.mockResolvedValue(true);
|
|
mockGenerationJobManager.emitChunk.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.emitDone.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.emitError.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.claimTerminalJob.mockResolvedValue({
|
|
streamId: 'conversation-123',
|
|
createdAt: 1000,
|
|
status: 'complete',
|
|
persistencePending: true,
|
|
drainedSteers: [],
|
|
});
|
|
mockGenerationJobManager.publishTerminalClaim.mockImplementation(
|
|
async (_claim, finalEvent) => ({
|
|
finalEvent: finalEvent ?? {
|
|
final: true,
|
|
reconcile: true,
|
|
reconcileReason: 'terminal_payload_missing',
|
|
terminalStatus: 'error',
|
|
},
|
|
persistenceFailed: finalEvent == null,
|
|
}),
|
|
);
|
|
mockGenerationJobManager.finishTerminalJob.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.completeJob.mockImplementation(
|
|
async (_streamId, _error, _createdAt, options) => {
|
|
await options?.beforeErrorPublication?.();
|
|
return true;
|
|
},
|
|
);
|
|
mockGenerationJobManager.beginProviderExecution.mockResolvedValue(true);
|
|
mockGenerationJobManager.markProviderExecutionDrained.mockResolvedValue(true);
|
|
mockGenerationJobManager.failPausePersistence.mockResolvedValue(true);
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue(null);
|
|
mockGenerationJobManager.takeoverGeneration.mockResolvedValue({ claimed: false });
|
|
mockGenerationJobManager.releaseGeneration.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.hasJob.mockResolvedValue(true);
|
|
mockGenerationJobManager.approvals.ownsPausePersistence.mockResolvedValue(true);
|
|
mockGenerationJobManager.approvals.finishPausePersistence.mockResolvedValue(true);
|
|
mockGenerationJobManager.steering.closeAndDrain.mockResolvedValue([]);
|
|
mockGenerationJobManager.steering.park.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.steering.consumeRecovered.mockResolvedValue(true);
|
|
mockSaveMessage.mockResolvedValue({});
|
|
mockSaveConvo.mockResolvedValue({});
|
|
mockAppendConvoMessageReference.mockResolvedValue({});
|
|
mockDeleteAgentCheckpoint.mockResolvedValue(undefined);
|
|
mockSettleAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
|
|
mockVerifyAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
|
|
mockGetAgentEventActorSnapshot.mockResolvedValue({ state: null, reconciliations: [] });
|
|
mockCommitAgentEventActorState.mockResolvedValue({ status: 'stale' });
|
|
mockBeginAgentEventActorLegacyTurn.mockResolvedValue(true);
|
|
mockCompleteAgentEventActorLegacyTurn.mockResolvedValue(true);
|
|
mockRecordAgentEventActorReconciliation.mockResolvedValue(true);
|
|
mockResolveAgentEventActorReconciliation.mockResolvedValue(true);
|
|
mockClearAgentEventActorReconciliation.mockResolvedValue(true);
|
|
mockAdmitAgentEventActorAction.mockResolvedValue(true);
|
|
mockReleaseAgentEventActorAction.mockResolvedValue(true);
|
|
mockHasAgentEventActorActionAdmission.mockResolvedValue(false);
|
|
mockGetAgentEventActorReceipt.mockResolvedValue(null);
|
|
mockGetAgentEventActorDetachedAction.mockResolvedValue(null);
|
|
mockClaimAgentEventActorSuspension.mockResolvedValue({ status: 'claimed' });
|
|
mockSettleAgentEventActorSuspension.mockResolvedValue({ status: 'settled' });
|
|
});
|
|
|
|
it.each([
|
|
['non-string', { arbitrary: true }],
|
|
['empty', ''],
|
|
['oversized', 'a'.repeat(129)],
|
|
['unsafe characters', 'request id with spaces'],
|
|
])(
|
|
'rejects a %s clientRequestId before creating durable state',
|
|
async (_label, clientRequestId) => {
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Invalid request identity',
|
|
messageId: 'user-message',
|
|
clientRequestId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res) };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(400);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'INVALID_CLIENT_REQUEST_ID' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it.each([-1, 1.5, Number.MAX_SAFE_INTEGER + 1, '1000'])(
|
|
'rejects invalid expected predecessor epoch %p before creating durable state',
|
|
async (expectedPredecessorCreatedAt) => {
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Conditional queued follow-up',
|
|
messageId: 'user-message',
|
|
clientRequestId: 'conditional-request',
|
|
expectedPredecessorCreatedAt,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(400);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'INVALID_GENERATION_PREDECESSOR' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it.each(['overrideUserMessageId', 'overrideConvoId'])(
|
|
'rejects a non-string %s before admission',
|
|
async (field) => {
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Invalid override identity',
|
|
messageId: 'user-message',
|
|
clientRequestId: 'override-request',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
[field]: { malformed: true },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res) };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(400);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'INVALID_OVERRIDE_ID' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
['empty recovery id', { clientRequestId: 'steer-recovery:' }],
|
|
['regenerate', { isRegenerate: true }],
|
|
['continued response', { isContinued: true }],
|
|
['content edit', { editedContent: { index: 0, type: 'text', text: 'edited' } }],
|
|
['response reuse', { responseMessageId: 'existing-response' }],
|
|
['mismatched user-row override', { overrideUserMessageId: 'existing-user__1' }],
|
|
['malformed files', { files: [{}] }],
|
|
])('rejects a recovered steer submitted as an incompatible %s shape', async (_label, shape) => {
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Recovered words',
|
|
messageId: 'recovered-user-message',
|
|
clientRequestId: 'steer-recovery:server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
...shape,
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res) };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(400);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'INVALID_RECOVERY_REQUEST' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
['missing attempt id', { clientRequestId: undefined, recoverySteerId: 'server-steer-1' }],
|
|
['invalid source id', { clientRequestId: 'attempt-1', recoverySteerId: 'bad source id' }],
|
|
[
|
|
'mismatched legacy and explicit ids',
|
|
{ clientRequestId: 'steer-recovery:legacy-source', recoverySteerId: 'explicit-source' },
|
|
],
|
|
])('rejects an explicit recovery with %s', async (_label, recoveryFields) => {
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Recovered words',
|
|
messageId: 'recovered-user-message',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
...recoveryFields,
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res) };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(400);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'INVALID_RECOVERY_REQUEST' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('rejects an underscore-suffixed parent that is not persisted', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const initializeClient = jest.fn();
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Follow up too early.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'pending-response_',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
modelOptions: { model: 'gpt-3.5-turbo' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
json: jest.fn(),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGetMessages).toHaveBeenCalledWith(
|
|
{ user: 'user-123', messageId: 'pending-response_', conversationId },
|
|
'_id',
|
|
);
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
error: expect.stringContaining('selected parent response is still being saved'),
|
|
}),
|
|
);
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('allows an underscore-suffixed parent when it is already persisted', async () => {
|
|
const conversationId = 'conversation-123';
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Follow up to persisted underscore id.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
modelOptions: { model: 'gpt-3.5-turbo' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGetMessages).toHaveBeenCalledWith(
|
|
{ user: 'user-123', messageId: 'persisted-response_', conversationId },
|
|
'_id',
|
|
);
|
|
expect(res.status).not.toHaveBeenCalledWith(409);
|
|
expect(mockCheckAndIncrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
expect.objectContaining({
|
|
startupTelemetry: mockStartupTelemetry,
|
|
initialMetadata: expect.objectContaining({
|
|
conversationId,
|
|
endpoint: 'agents',
|
|
}),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('defers a trusted trigger resume while its parent generation is still active', async () => {
|
|
const conversationId = 'conversation-123';
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
status: 'running',
|
|
metadata: { userId: 'user-123' },
|
|
});
|
|
const initializeClient = jest.fn();
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Collect the completed child.',
|
|
messageId: 'wakeup-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId,
|
|
clientRequestId: 'trigger_resume_1',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'PARENT_NOT_READY' }));
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('labels a trigger parent-state lookup failure as provably pre-admission', async () => {
|
|
const conversationId = 'conversation-123';
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
mockGenerationJobManager.getJob.mockRejectedValue(new Error('redis unavailable'));
|
|
const initializeClient = jest.fn();
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Collect the completed child.',
|
|
messageId: 'wakeup-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId,
|
|
clientRequestId: 'trigger_resume_1',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.set).toHaveBeenCalledWith('Retry-After', '1');
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'PARENT_STATE_UNAVAILABLE' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('deduplicates the active continuation whose admission response was lost', async () => {
|
|
const conversationId = 'conversation-123';
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'requires_action',
|
|
metadata: {
|
|
userId: 'user-123',
|
|
idempotencyClientRequestId: 'trigger_resume_1',
|
|
},
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: conversationId,
|
|
conversationId,
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Collect the completed child.',
|
|
messageId: 'wakeup-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId,
|
|
clientRequestId: 'trigger_resume_1',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).not.toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: conversationId,
|
|
conversationId,
|
|
generationCreatedAt: 1000,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('creates the job with the in-flight turn before MCP initialization can emit OAuth', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Check Google Workspace availability.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'original-response',
|
|
conversationId,
|
|
isTemporary: true,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
iconURL: 'https://example.com/spec-icon.png',
|
|
modelOptions: { model: 'gpt-3.5-turbo' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
{
|
|
startupTelemetry: mockStartupTelemetry,
|
|
initialMetadata: {
|
|
conversationId,
|
|
generationProtocolVersion: 1,
|
|
endpoint: 'agents',
|
|
iconURL: 'https://example.com/spec-icon.png',
|
|
model: 'gpt-3.5-turbo',
|
|
/** The OWNING replica's seal capability, read by the steer route. */
|
|
preemptCapable: true,
|
|
steerQuotesCapable: true,
|
|
agent_id: undefined,
|
|
isTemporary: true,
|
|
responseMessageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
mcpRequestBody: {
|
|
messageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
conversationId,
|
|
parentMessageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
},
|
|
userMessage: {
|
|
messageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
parentMessageId: 'original-response',
|
|
conversationId,
|
|
text: 'Check Google Workspace availability.',
|
|
},
|
|
},
|
|
},
|
|
);
|
|
expect(mockGenerationJobManager.createJob.mock.invocationCallOrder[0]).toBeLessThan(
|
|
initializeClient.mock.invocationCallOrder[0],
|
|
);
|
|
expect(initializeClient).toHaveBeenCalledWith(
|
|
expect.objectContaining({ checkpointNamespace: '1000', jobCreatedAt: 1000 }),
|
|
);
|
|
const [{ foregroundRunId, requestBody }] = initializeClient.mock.calls[0];
|
|
expect(foregroundRunId).toBe(requestBody.messageId);
|
|
expect(foregroundRunId).not.toBe(req.body.messageId);
|
|
expect(req.turnStartedAt).toBe(1000);
|
|
expect(mockGenerationJobManager.updateMetadata).not.toHaveBeenCalled();
|
|
const startupMilestones = mockStartupTelemetry.mark.mock.calls.map(([milestone]) => milestone);
|
|
expect(startupMilestones.slice(0, 2)).toEqual(['request_admitted', 'job_created']);
|
|
expect(new Set(startupMilestones.slice(2))).toEqual(
|
|
new Set(['conversation_resolved', 'metadata_persisted']),
|
|
);
|
|
expect(mockAcceptAgentStartupTelemetry).toHaveBeenCalledWith(req, conversationId);
|
|
expect(mockStartupTelemetry.end).toHaveBeenCalledWith('error', expect.any(Error));
|
|
});
|
|
|
|
it('persists and exactly echoes protocol v2 on a newly created generation', async () => {
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
metadata: { checkpointNamespace: '1000', generationProtocolVersion: 2 },
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Negotiate the rollout protocol.',
|
|
messageId: 'user-message',
|
|
conversationId: 'conversation-123',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after negotiation'));
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'user-123',
|
|
'conversation-123',
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({ generationProtocolVersion: 2 }),
|
|
}),
|
|
);
|
|
expect(res.set).toHaveBeenCalledWith('x-librechat-generation-protocol', '2');
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 1000,
|
|
status: 'started',
|
|
generationProtocolVersion: 2,
|
|
});
|
|
});
|
|
|
|
it('rejects any session fenced after authentication before generation execution starts', async () => {
|
|
mockIsAgentTriggerPrincipalActive.mockResolvedValue(false);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Run after a slow trigger admission.',
|
|
messageId: 'user-message',
|
|
clientRequestId: 'trigger-request',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
const initializeClient = jest.fn();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledTimes(1);
|
|
expect(mockIsAgentTriggerPrincipalActive).toHaveBeenCalledWith('user-123');
|
|
expect(mockGenerationJobManager.createJob.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockIsAgentTriggerPrincipalActive.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockGenerationJobManager.beginProviderExecution).not.toHaveBeenCalled();
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
status: 409,
|
|
code: 'ACCOUNT_DELETION_IN_PROGRESS',
|
|
error: 'Account deletion is in progress',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
expect(mockAcceptAgentStartupTelemetry).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
JSON.stringify({
|
|
status: 409,
|
|
code: 'ACCOUNT_DELETION_IN_PROGRESS',
|
|
error: 'Account deletion is in progress',
|
|
}),
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'trigger-request',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
});
|
|
|
|
it('rejects a superseded automatic occurrence after durable job creation and preserves its outcome', async () => {
|
|
const conversationId = 'scheduled-conversation-123';
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: conversationId, conversationId }),
|
|
);
|
|
mockIsScheduleLive.mockResolvedValue(false);
|
|
const req = {
|
|
_isScheduledFire: true,
|
|
_isManualScheduledFire: false,
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Run the scheduled digest.',
|
|
messageId: 'scheduled-user-message',
|
|
clientRequestId: 'sched:schedule-1:2026-08-17T12:00:00-000Z',
|
|
conversationId: 'new',
|
|
newConversationId: conversationId,
|
|
scheduleId: 'schedule-1',
|
|
scheduledFor: '2026-08-17T12:00:00.000Z',
|
|
scheduleConfigRevision: 7,
|
|
endpointOption: { endpoint: 'agents', agent_id: 'agent-1' },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
const initializeClient = jest.fn();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({
|
|
scheduleId: 'schedule-1',
|
|
scheduledFor: '2026-08-17T12:00:00.000Z',
|
|
scheduleConfigRevision: 7,
|
|
preserveForScheduleReconcile: true,
|
|
}),
|
|
}),
|
|
);
|
|
// `scheduledFor` identifies the OCCURRENCE, matching the resume path: the run row
|
|
// is reserved before this loopback request is dispatched, so a pin introduced while
|
|
// it sat queued must not be validated in place of the destination the envelope was
|
|
// already built with.
|
|
expect(mockIsScheduleLive).toHaveBeenCalledWith('schedule-1', 7, {
|
|
automatic: true,
|
|
policy: true,
|
|
scheduledFor: '2026-08-17T12:00:00.000Z',
|
|
});
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
status: 409,
|
|
code: 'SCHEDULE_NO_LONGER_ACTIVE',
|
|
error: 'This scheduled occurrence is no longer active',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockRecordScheduleOutcome).toHaveBeenCalledWith({
|
|
scheduleId: 'schedule-1',
|
|
scheduledFor: '2026-08-17T12:00:00.000Z',
|
|
streamId: conversationId,
|
|
jobCreatedAt: 1000,
|
|
status: 'interrupted',
|
|
conversationId,
|
|
clearConversationId: false,
|
|
error: 'This scheduled occurrence is no longer active',
|
|
});
|
|
expect(mockDecrementPendingRequest).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('does not start a provider when account deletion or replacement wins the startup CAS', async () => {
|
|
mockGenerationJobManager.beginProviderExecution.mockResolvedValue(false);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Race destructive cleanup.',
|
|
messageId: 'user-message',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
const initializeClient = jest.fn();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockIsAgentTriggerPrincipalActive).toHaveBeenCalledWith('user-123');
|
|
expect(mockIsAgentTriggerPrincipalActive.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.beginProviderExecution.mock.invocationCallOrder[0],
|
|
);
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
expect.stringContaining('Generation stopped before provider startup'),
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.markProviderExecutionDrained).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
1000,
|
|
'provider-segment-1',
|
|
);
|
|
});
|
|
|
|
it('keeps a queued-turn source nonterminal when provider initialization fails', async () => {
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
const admissionSource = {
|
|
source: 'agent-queued-turn',
|
|
sourceId: 'queued-turn-1',
|
|
claimId: 'queued-delivery-1',
|
|
claimBy: 'queued-worker-1',
|
|
};
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Run the queued turn.',
|
|
messageId: 'queued-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId: 'conversation-123',
|
|
clientRequestId: 'queued-delivery-1',
|
|
agentContinuationAdmission: admissionSource,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before provider'));
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(initializeClient).toHaveBeenCalledTimes(1);
|
|
expect(mockSettleAgentQueuedTurnExecutionAdmission).not.toHaveBeenCalled();
|
|
expect(res.status).toHaveBeenCalledWith(500);
|
|
});
|
|
|
|
it('commits a queued-turn receipt after provider invocation and before accepting HTTP', async () => {
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
const admissionSource = {
|
|
source: 'agent-queued-turn',
|
|
sourceId: 'queued-turn-1',
|
|
claimId: 'queued-delivery-1',
|
|
claimBy: 'queued-worker-1',
|
|
};
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Run the queued turn.',
|
|
messageId: 'queued-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId: 'conversation-123',
|
|
clientRequestId: 'queued-delivery-1',
|
|
agentContinuationAdmission: admissionSource,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const sendMessage = jest.fn(() => new Promise(() => {}));
|
|
const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
await nextTick();
|
|
|
|
expect(mockSettleAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
|
|
userId: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'conversation-123',
|
|
clientRequestId: 'queued-delivery-1',
|
|
generationId: 'conversation-123',
|
|
generationCreatedAt: 1000,
|
|
});
|
|
expect(initializeClient.mock.invocationCallOrder[0]).toBeLessThan(
|
|
sendMessage.mock.invocationCallOrder[0],
|
|
);
|
|
expect(sendMessage.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0]).toBeLessThan(
|
|
res.json.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it('fails closed after provider invocation when the queued-turn receipt is unavailable', async () => {
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
|
mockSettleAgentQueuedTurnExecutionAdmission.mockRejectedValue(new Error('mongo unavailable'));
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Run the queued turn.',
|
|
messageId: 'queued-user-message',
|
|
parentMessageId: 'persisted-response_',
|
|
conversationId: 'conversation-123',
|
|
clientRequestId: 'queued-delivery-1',
|
|
agentContinuationAdmission: {
|
|
source: 'agent-queued-turn',
|
|
sourceId: 'queued-turn-1',
|
|
claimId: 'queued-delivery-1',
|
|
claimBy: 'queued-worker-1',
|
|
},
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const sendMessage = jest.fn(() => new Promise(() => {}));
|
|
const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.beginProviderExecution).toHaveBeenCalledTimes(1);
|
|
expect(sendMessage).toHaveBeenCalledTimes(1);
|
|
expect(res.status).toHaveBeenCalledWith(500);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'mongo unavailable',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
expect(mockGenerationJobManager.markProviderExecutionDrained).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
1000,
|
|
'provider-segment-1',
|
|
);
|
|
});
|
|
|
|
it('prefetches conversation state before admission and joins it with job metadata', async () => {
|
|
let resolveConversation;
|
|
let signalMetadataStarted;
|
|
const conversationPromise = new Promise((resolve) => {
|
|
resolveConversation = resolve;
|
|
});
|
|
const metadataStarted = new Promise((resolve) => {
|
|
signalMetadataStarted = resolve;
|
|
});
|
|
mockGetConvo.mockReturnValue(conversationPromise);
|
|
mockGenerationJobManager.createJob.mockImplementation(() => {
|
|
signalMetadataStarted();
|
|
return Promise.resolve({
|
|
createdAt: 1000,
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
});
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after startup reads'));
|
|
const conversationId = 'conversation-123';
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Run independent startup work together.',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
modelOptions: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
const controllerPromise = AgentController(req, res, jest.fn(), initializeClient, null);
|
|
expect(mockGetConvo).toHaveBeenCalledWith('user-123', conversationId);
|
|
await metadataStarted;
|
|
await nextTick();
|
|
|
|
expect(mockGetConvo.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockCheckAndIncrementPendingRequest.mock.invocationCallOrder[0],
|
|
);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: conversationId,
|
|
conversationId,
|
|
generationCreatedAt: 1000,
|
|
status: 'started',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
|
|
resolveConversation({ createdAt: '2026-06-07T00:00:00.000Z' });
|
|
await controllerPromise;
|
|
|
|
expect(initializeClient).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it('keeps request-scoped MCP connections until resumable initialization finishes', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const disconnect = jest.fn().mockResolvedValue(undefined);
|
|
const initializeClient = jest.fn(async ({ req, res }) => {
|
|
const context = getMCPRequestContext(req, res);
|
|
context.connections.set('mcp-server', { disconnect });
|
|
|
|
await nextTick();
|
|
expect(disconnect).not.toHaveBeenCalled();
|
|
|
|
throw new Error('stop after request-scoped MCP connection');
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use a BODY-scoped MCP server.',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
modelOptions: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: conversationId,
|
|
conversationId,
|
|
generationCreatedAt: 1000,
|
|
status: 'started',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(disconnect).toHaveBeenCalledTimes(1);
|
|
expect(disconnect.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockDecrementPendingRequest.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it('preallocates response-scoped MCP identities before native Agent initialization', async () => {
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after MCP discovery'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use request-scoped headers.',
|
|
messageId: 'incoming-client-message',
|
|
parentMessageId: 'previous-response',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
expect(initializeClient).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
requestBody: {
|
|
messageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
conversationId: 'conversation-123',
|
|
parentMessageId: expect.stringMatching(/^[0-9a-f-]{36}$/),
|
|
},
|
|
}),
|
|
);
|
|
const [{ requestBody }] = initializeClient.mock.calls[0];
|
|
const jobOptions = mockGenerationJobManager.createJob.mock.calls[0][3];
|
|
expect(jobOptions.initialMetadata.responseMessageId).toBe(requestBody.messageId);
|
|
expect(jobOptions.initialMetadata.userMessage.messageId).toBe(requestBody.parentMessageId);
|
|
expect(jobOptions.initialMetadata.mcpRequestBody).toBe(requestBody);
|
|
expect(requestBody.messageId).not.toBe(req.body.messageId);
|
|
});
|
|
|
|
it('pins a normalized code-environment decision before provider execution', async () => {
|
|
let signalProviderStarted;
|
|
const providerStarted = new Promise((resolve) => {
|
|
signalProviderStarted = resolve;
|
|
});
|
|
const sendMessage = jest.fn(() => {
|
|
signalProviderStarted();
|
|
return new Promise(() => {});
|
|
});
|
|
const initializeClient = jest.fn(async ({ req }) => {
|
|
req.body.codeEnvironmentMode = 'without_attached';
|
|
delete req.body.codeWorkspaces;
|
|
return { client: { options: {}, sendMessage } };
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Continue without my attached machine.',
|
|
messageId: 'incoming-client-message',
|
|
parentMessageId: 'previous-response',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
await providerStarted;
|
|
|
|
const initialRequestBody =
|
|
mockGenerationJobManager.createJob.mock.calls[0][3].initialMetadata.mcpRequestBody;
|
|
expect(initialRequestBody).not.toHaveProperty('codeEnvironmentMode');
|
|
expect(mockGenerationJobManager.updateMetadata).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
{
|
|
mcpRequestBody: {
|
|
...initialRequestBody,
|
|
codeEnvironmentMode: 'without_attached',
|
|
},
|
|
},
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.updateMetadata.mock.invocationCallOrder[0]).toBeLessThan(
|
|
sendMessage.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it('uses the effective overridden conversation in the MCP request body', async () => {
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after MCP discovery'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Continue in the overridden conversation.',
|
|
messageId: 'incoming-client-message',
|
|
parentMessageId: 'previous-response',
|
|
conversationId: 'source-conversation',
|
|
overrideConvoId: 'overridden-conversation__0',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
const [{ requestBody }] = initializeClient.mock.calls[0];
|
|
const jobOptions = mockGenerationJobManager.createJob.mock.calls[0][3];
|
|
expect(requestBody.conversationId).toBe('overridden-conversation');
|
|
expect(jobOptions.initialMetadata.mcpRequestBody).toBe(requestBody);
|
|
});
|
|
|
|
it('preallocates the replacement response as the MCP parent for edited content', async () => {
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after MCP discovery'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Edited response text.',
|
|
messageId: 'existing-user-message',
|
|
responseMessageId: 'existing-response-message',
|
|
parentMessageId: 'previous-response',
|
|
overrideParentMessageId: 'existing-user-message',
|
|
editedContent: { index: 0, type: 'text', text: 'Edited response text.' },
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
const [{ requestBody }] = initializeClient.mock.calls[0];
|
|
const jobOptions = mockGenerationJobManager.createJob.mock.calls[0][3];
|
|
expect(requestBody.messageId).toMatch(/^[0-9a-f-]{36}$/);
|
|
expect(requestBody.parentMessageId).toBe(requestBody.messageId);
|
|
expect(requestBody.messageId).not.toBe('existing-response-message');
|
|
expect(jobOptions.initialMetadata.mcpRequestBody).toBe(requestBody);
|
|
});
|
|
|
|
it('stores model spec icon fallbacks and agent ids in early resume metadata', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use the resume spec.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'original-response',
|
|
conversationId,
|
|
isTemporary: true,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
spec: 'agent-spec',
|
|
agent_id: 'agent_resume_spec',
|
|
model_parameters: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {
|
|
modelSpecs: {
|
|
list: [
|
|
{
|
|
name: 'agent-spec',
|
|
preset: {
|
|
endpoint: 'openAI',
|
|
iconURL: 'https://example.com/preset-icon.png',
|
|
},
|
|
},
|
|
],
|
|
},
|
|
},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({
|
|
iconURL: 'https://example.com/preset-icon.png',
|
|
model: 'agent_resume_spec',
|
|
agent_id: 'agent_resume_spec',
|
|
isTemporary: true,
|
|
}),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('records regeneration ownership for exact-ID resume reconstruction', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Regenerate the edited response.',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
responseMessageId: 'edited-response',
|
|
isRegenerate: true,
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
modelOptions: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({
|
|
responseMessageId: 'edited-response',
|
|
isRegenerate: true,
|
|
}),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('falls back to the model spec preset endpoint when no icon URL is configured', async () => {
|
|
const conversationId = 'conversation-123';
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use the endpoint icon.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'original-response',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
spec: 'endpoint-icon-spec',
|
|
model_parameters: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {
|
|
modelSpecs: {
|
|
list: [
|
|
{
|
|
name: 'endpoint-icon-spec',
|
|
preset: {
|
|
endpoint: 'anthropic',
|
|
},
|
|
},
|
|
],
|
|
},
|
|
},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'user-123',
|
|
conversationId,
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({
|
|
iconURL: 'anthropic',
|
|
model: 'gpt-4.1',
|
|
}),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('filters OAuth prompts before saving partial responses on disconnect', async () => {
|
|
const conversationId = 'conversation-123';
|
|
let allSubscribersLeftHandler;
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: {
|
|
on: jest.fn((event, handler) => {
|
|
if (event === 'allSubscribersLeft') {
|
|
allSubscribersLeftHandler = handler;
|
|
}
|
|
}),
|
|
},
|
|
});
|
|
mockGenerationJobManager.getResumeState.mockResolvedValue({
|
|
conversationId,
|
|
responseMessageId: 'response-message',
|
|
iconURL: 'https://example.com/spec-icon.png',
|
|
model: 'gpt-4.1',
|
|
userMessage: {
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
text: 'Use Google Workspace',
|
|
},
|
|
});
|
|
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after setup'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use Google Workspace',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
iconURL: 'https://example.com/fallback-icon.png',
|
|
modelOptions: { model: 'gpt-3.5-turbo' },
|
|
},
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
expect(allSubscribersLeftHandler).toEqual(expect.any(Function));
|
|
mockSaveMessage.mockClear();
|
|
mockSaveConvo.mockClear();
|
|
|
|
const oauthPart = {
|
|
type: 'tool_call',
|
|
tool_call: {
|
|
name: 'oauth_mcp_Google-Workspace',
|
|
auth: 'https://auth.example.com/oauth',
|
|
},
|
|
};
|
|
const textPart = { type: 'text', text: 'Partial response...' };
|
|
|
|
mockSaveMessage.mockResolvedValueOnce(undefined).mockResolvedValueOnce({});
|
|
await allSubscribersLeftHandler([oauthPart, textPart]);
|
|
await allSubscribersLeftHandler([oauthPart, textPart]);
|
|
|
|
expect(mockFilterPersistableAbortContent).toHaveBeenCalledWith([oauthPart, textPart]);
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
content: [textPart],
|
|
iconURL: 'https://example.com/spec-icon.png',
|
|
model: 'gpt-4.1',
|
|
messageId: 'response-message',
|
|
parentMessageId: 'user-message',
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockSaveMessage).toHaveBeenCalledTimes(2);
|
|
expect(mockLogger.error).toHaveBeenCalledWith(
|
|
'[ResumableAgentController] Error saving partial response:',
|
|
expect.objectContaining({
|
|
message: 'Partial response could not be persisted after disconnect',
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('uses model spec and agent fallbacks when saving partial responses on disconnect', async () => {
|
|
const conversationId = 'conversation-123';
|
|
let allSubscribersLeftHandler;
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: {
|
|
on: jest.fn((event, handler) => {
|
|
if (event === 'allSubscribersLeft') {
|
|
allSubscribersLeftHandler = handler;
|
|
}
|
|
}),
|
|
},
|
|
});
|
|
mockGenerationJobManager.getResumeState.mockResolvedValue({
|
|
conversationId,
|
|
responseMessageId: 'response-message',
|
|
userMessage: {
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
text: 'Use fallback metadata',
|
|
},
|
|
});
|
|
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after setup'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use fallback metadata',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'parent-message',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'agents',
|
|
spec: 'agent-spec',
|
|
agent_id: 'agent_resume_spec',
|
|
model_parameters: { model: 'gpt-4.1' },
|
|
},
|
|
},
|
|
config: {
|
|
modelSpecs: {
|
|
list: [
|
|
{
|
|
name: 'agent-spec',
|
|
preset: {
|
|
endpoint: 'openAI',
|
|
iconURL: 'https://example.com/preset-icon.png',
|
|
},
|
|
},
|
|
],
|
|
},
|
|
},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
expect(allSubscribersLeftHandler).toEqual(expect.any(Function));
|
|
|
|
const textPart = { type: 'text', text: 'Partial response...' };
|
|
await allSubscribersLeftHandler([textPart]);
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
content: [textPart],
|
|
iconURL: 'https://example.com/preset-icon.png',
|
|
model: 'agent_resume_spec',
|
|
messageId: 'response-message',
|
|
parentMessageId: 'user-message',
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
});
|
|
|
|
it('dedups a retried start-generation request to the original stream', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'running',
|
|
metadata: { userId: 'user-123', idempotencyClientRequestId: 'req-abc' },
|
|
});
|
|
const initializeClient = jest.fn();
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retried after a lost response.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
generationCreatedAt: 1000,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(initializeClient).not.toHaveBeenCalled();
|
|
expect(mockStartupTelemetry.end).toHaveBeenCalledWith('deduplicated');
|
|
});
|
|
|
|
it('attaches a new-chat retry to a tokenless legacy random stream after owner validation', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
source: 'legacy',
|
|
existing: {
|
|
streamId: 'legacy-random-stream',
|
|
conversationId: 'legacy-random-stream',
|
|
claimedAt: Date.now() - 100,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'running',
|
|
metadata: { userId: 'user-123' },
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry an old-server new chat.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'new',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'legacy-random-stream',
|
|
conversationId: 'legacy-random-stream',
|
|
generationCreatedAt: 1000,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.takeoverGeneration).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('never takes over a tokenless legacy claim after its job has disappeared', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
source: 'legacy',
|
|
existing: {
|
|
streamId: 'legacy-random-stream',
|
|
conversationId: 'legacy-random-stream',
|
|
claimedAt: Date.now() - 60_000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry after an old generation was cleaned.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'new',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'legacy-random-stream',
|
|
conversationId: 'legacy-random-stream',
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.takeoverGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('rejects a tokenless legacy claim that miscorrelates an existing conversation', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
source: 'legacy',
|
|
existing: {
|
|
streamId: 'different-conversation',
|
|
conversationId: 'different-conversation',
|
|
claimedAt: Date.now() - 100,
|
|
},
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Do not cross-wire this retry.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(mockGenerationJobManager.getJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
['a different stream', { streamId: 'foreign-stream' }],
|
|
['a different conversation', { conversationId: 'foreign-conversation' }],
|
|
['a missing claim token', { claimToken: undefined }],
|
|
['an invalid claim timestamp', { claimedAt: 'not-a-timestamp' }],
|
|
['an invalid started timestamp', { startedAt: -1 }],
|
|
])('fails closed when the idempotency claim contains %s', async (_name, override) => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
...override,
|
|
},
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry against corrupt claim state.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockGenerationJobManager.getJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.takeoverGeneration).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
['another user', { userId: 'someone-else' }, { id: 'user-123' }],
|
|
['a missing owner', {}, { id: 'user-123' }],
|
|
[
|
|
'another tenant',
|
|
{ userId: 'user-123', tenantId: 'tenant-b' },
|
|
{ id: 'user-123', tenantId: 'tenant-a' },
|
|
],
|
|
])(
|
|
'fails closed when a valid idempotency claim resolves to %s',
|
|
async (_name, jobMetadata, requestUser) => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'running',
|
|
metadata: { ...jobMetadata, idempotencyClientRequestId: 'req-abc' },
|
|
});
|
|
const req = {
|
|
user: requestUser,
|
|
body: {
|
|
text: 'Do not attach this retry to a foreign run.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it('keeps a terminal duplicate without a durable payload on the readiness path', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'aborted',
|
|
metadata: {
|
|
userId: 'user-123',
|
|
idempotencyClientRequestId: 'req-abc',
|
|
terminalPersistencePending: true,
|
|
},
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry while abort persistence is still in flight.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.set).toHaveBeenCalledWith('Retry-After', '1');
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('allows a terminal duplicate with a stored FINAL to replay through SSE', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: 100,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'complete',
|
|
finalEvent: JSON.stringify({ final: true }),
|
|
metadata: { userId: 'user-123', idempotencyClientRequestId: 'req-abc' },
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry after the terminal payload was persisted.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
generationCreatedAt: 1000,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('derives the same new-conversation stream for a lost-response retry', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockImplementation(
|
|
async (_userId, _requestId, streamId, conversationId) =>
|
|
wonGenerationClaim({ streamId, conversationId }),
|
|
);
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after deterministic id'));
|
|
const makeReq = () => ({
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Create a stable new chat.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: '4ea9fc40-f28f-4f89-a575-aa5854d10c19',
|
|
conversationId: 'new',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
});
|
|
|
|
await AgentController(makeReq(), createResumableResponse(), jest.fn(), initializeClient, null);
|
|
await AgentController(makeReq(), createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
const firstStreamId = mockGenerationJobManager.claimGeneration.mock.calls[0][2];
|
|
const retryStreamId = mockGenerationJobManager.claimGeneration.mock.calls[1][2];
|
|
expect(firstStreamId).toMatch(
|
|
/^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/,
|
|
);
|
|
expect(retryStreamId).toBe(firstStreamId);
|
|
});
|
|
|
|
it('resumes a matching live generation when its fixed claim lease was reacquired', async () => {
|
|
const reacquired = wonGenerationClaim({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimToken: 'reacquired-token',
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(reacquired);
|
|
mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue({
|
|
...reacquired.existing,
|
|
startedAt: 42,
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry after sleeping through a long approval.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(mockGenerationJobManager.resumeClaimedGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
reacquired.existing,
|
|
);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 42,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('fails closed when a deduplicated queued generation lacks its source receipt', async () => {
|
|
const reacquired = wonGenerationClaim({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimToken: 'reacquired-token',
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(reacquired);
|
|
mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue({
|
|
...reacquired.existing,
|
|
startedAt: 42,
|
|
});
|
|
mockVerifyAgentQueuedTurnExecutionAdmission.mockRejectedValue(
|
|
new Error('source receipt missing'),
|
|
);
|
|
const admissionSource = {
|
|
source: 'agent-queued-turn',
|
|
sourceId: 'queued-turn-1',
|
|
claimId: 'queued-delivery-1',
|
|
claimBy: 'queued-worker-1',
|
|
};
|
|
const req = {
|
|
_isAgentTrigger: true,
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Retry a queued generation.',
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'assistant-1',
|
|
clientRequestId: 'queued-delivery-1',
|
|
conversationId: 'conversation-123',
|
|
agentContinuationAdmission: admissionSource,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(mockVerifyAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
|
|
userId: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'conversation-123',
|
|
clientRequestId: 'queued-delivery-1',
|
|
generationId: 'conversation-123',
|
|
generationCreatedAt: 42,
|
|
});
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
code: 'SERVER_NOT_READY',
|
|
}),
|
|
);
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('caps a reacquired v2 lease to the immutable v1 live-job protocol', async () => {
|
|
const reacquired = wonGenerationClaim({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimToken: 'reacquired-v2-token',
|
|
generationProtocolVersion: 2,
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(reacquired);
|
|
mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue({
|
|
...reacquired.existing,
|
|
startedAt: 42,
|
|
generationProtocolVersion: 1,
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
headers: { 'x-librechat-generation-protocol': '2' },
|
|
body: {
|
|
text: 'Retry a v1 job from an upgraded client.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 42,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('returns a settled response when the original job completed and was cleaned up', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: Date.now() - 60000,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry after a fast, already-cleaned-up generation.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
generationCreatedAt: 1000,
|
|
status: 'resumed',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(res.status).not.toHaveBeenCalledWith(503);
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('uses the v2 settled control only when the durable claim confirms protocol v2', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: Date.now() - 60_000,
|
|
claimToken: 'existing-token',
|
|
startedAt: 2000,
|
|
generationProtocolVersion: 2,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry a completed v2 generation.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
conversationId: 'orig-stream',
|
|
generationCreatedAt: 2000,
|
|
status: 'settled',
|
|
generationProtocolVersion: 2,
|
|
});
|
|
});
|
|
|
|
it('does not attach a stale retry to a newer generation on the same conversation stream', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimedAt: Date.now() - 60_000,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 2000,
|
|
status: 'running',
|
|
metadata: { userId: 'user-123', idempotencyClientRequestId: 'newer-request' },
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry the earlier turn after its response was lost.',
|
|
messageId: 'old-user-msg',
|
|
clientRequestId: 'older-request',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
code: 'RUN_REPLACED',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('uses the v2 replacement handoff only when both claim and live job confirm v2', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
claimedAt: Date.now() - 60_000,
|
|
claimToken: 'existing-token',
|
|
startedAt: 1000,
|
|
generationProtocolVersion: 2,
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 2000,
|
|
status: 'running',
|
|
metadata: {
|
|
userId: 'user-123',
|
|
idempotencyClientRequestId: 'newer-request',
|
|
generationProtocolVersion: 2,
|
|
},
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Retry the earlier v2 turn.',
|
|
messageId: 'old-user-msg',
|
|
clientRequestId: 'older-request',
|
|
conversationId: 'conversation-123',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(200);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 2000,
|
|
status: 'replaced',
|
|
generationProtocolVersion: 2,
|
|
});
|
|
});
|
|
|
|
it('takes over an abandoned old pre-create claim and fences the prior owner', async () => {
|
|
const abandonedClaim = {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: Date.now() - 60000,
|
|
claimToken: 'abandoned-token',
|
|
};
|
|
const takeover = wonGenerationClaim({
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimToken: 'takeover-token',
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: abandonedClaim,
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.takeoverGeneration.mockResolvedValue(takeover);
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop after takeover'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Recover the abandoned request.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.takeoverGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'orig-stream',
|
|
abandonedClaim,
|
|
);
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'orig-stream',
|
|
'user-123',
|
|
'orig-stream',
|
|
expect.objectContaining({
|
|
idempotencyClientRequestId: 'req-abc',
|
|
idempotencyClaimToken: 'takeover-token',
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('returns 503 SERVER_NOT_READY when a fresh claim still has no job (winner is between claim and createJob)', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: Date.now(),
|
|
claimToken: 'existing-token',
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Concurrent duplicate before the winner wrote its job.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.set).toHaveBeenCalledWith('Retry-After', '1');
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('never starts a second generation when the job lookup fails for a confirmed duplicate', async () => {
|
|
// A store hiccup while checking an existing claim must not fail open into createJob.
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
|
claimed: false,
|
|
existing: {
|
|
streamId: 'orig-stream',
|
|
conversationId: 'orig-stream',
|
|
claimedAt: Date.now(),
|
|
claimToken: 'existing-token',
|
|
},
|
|
});
|
|
mockGenerationJobManager.getJob.mockRejectedValue(new Error('redis down'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Duplicate during a Redis hiccup.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'orig-stream',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('does not finalize an unscoped generation when job creation rejects before returning', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
mockGenerationJobManager.createJob.mockRejectedValue(new Error('create failed before return'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Fail before receiving a job epoch.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(500);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
error: 'create failed before return',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.emitError).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
});
|
|
|
|
it('names the failure class at the initialization boundary', async () => {
|
|
const initializeClient = jest.fn().mockRejectedValue(new AttachmentStorageError());
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Describe the attached image.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
const [message, ...metadata] = mockLogger.error.mock.calls.find((call) =>
|
|
String(call[0]).startsWith('[ResumableAgentController] Initialization error:'),
|
|
);
|
|
expect(metadata).toEqual([]);
|
|
expect(message).toContain(
|
|
'AttachmentStorageError: An attached file could not be read from storage.',
|
|
);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'An attached file could not be read from storage. Try again or upload it again.',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it('redacts a signed storage URL an initialization failure carries into the log', async () => {
|
|
const signedUrl =
|
|
'https://minio.example.com/bucket/image.png?X-Amz-Credential=secret&X-Amz-Signature=signed';
|
|
const initializeClient = jest
|
|
.fn()
|
|
.mockRejectedValue(new Error(`AccessDenied reading ${signedUrl}`));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Describe the attached image.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
const [message, ...metadata] = mockLogger.error.mock.calls.find((call) =>
|
|
String(call[0]).startsWith('[ResumableAgentController] Initialization error:'),
|
|
);
|
|
expect(metadata).toEqual([]);
|
|
expect(message).toContain('AccessDenied reading https://minio.example.com/[redacted]');
|
|
expect(message).not.toContain('X-Amz-Signature');
|
|
expect(JSON.stringify(mockLogger.error.mock.calls)).not.toContain(signedUrl);
|
|
});
|
|
|
|
it('publishes a safe image storage failure during generation and releases the request', async () => {
|
|
const signedUrl =
|
|
'https://minio.example.com/bucket/image.png?X-Amz-Credential=secret&X-Amz-Signature=signed';
|
|
const storageError = Object.assign(new Error(`Access denied for ${signedUrl}`), {
|
|
code: 'AccessDenied',
|
|
statusCode: 403,
|
|
});
|
|
const getDownloadStream = jest.fn().mockRejectedValue(storageError);
|
|
const file = {
|
|
file_id: 'image-1',
|
|
source: 's3',
|
|
filepath: signedUrl,
|
|
storageKey: 'images/user/image.png',
|
|
height: 10,
|
|
width: 10,
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Describe the attached image.',
|
|
messageId: 'user-msg',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(() =>
|
|
encodeAndFormatImages(
|
|
req,
|
|
[file],
|
|
{},
|
|
{ getStrategyFunctions: () => ({ getDownloadStream }) },
|
|
),
|
|
),
|
|
};
|
|
const initializeClient = jest.fn().mockResolvedValue({ client });
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
await nextTick();
|
|
|
|
const safeError = new AttachmentStorageError();
|
|
expect(getDownloadStream).toHaveBeenCalledWith(req, file.storageKey);
|
|
expect(mockLogger.error).toHaveBeenCalledWith(
|
|
'[ResumableAgentController] Generation error for conversation-123:',
|
|
safeError,
|
|
);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
safeError.message,
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({ error: true, text: safeError.message }),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockDisposeClient).toHaveBeenCalledWith(client);
|
|
expect(JSON.stringify(mockGenerationJobManager.completeJob.mock.calls)).not.toContain(
|
|
signedUrl,
|
|
);
|
|
expect(JSON.stringify(mockSaveMessage.mock.calls)).not.toContain(signedUrl);
|
|
});
|
|
|
|
it('returns a typed recovery conflict before acknowledging generation startup', async () => {
|
|
const recoveryError = new Error('Attached resources could not be restored');
|
|
recoveryError.code = ErrorTypes.RESOURCE_RECOVERY_REQUIRED;
|
|
mockGenerationJobManager.createJob.mockRejectedValue(recoveryError);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Describe the attached image.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
status: 409,
|
|
code: ErrorTypes.RESOURCE_RECOVERY_REQUIRED,
|
|
error: 'Attached resources could not be restored',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('preserves the recovery code in the durable error after acknowledging startup', async () => {
|
|
const recoveryError = new Error('Attached resources could not be restored');
|
|
recoveryError.code = ErrorTypes.RESOURCE_RECOVERY_REQUIRED;
|
|
const initializeClient = jest.fn().mockRejectedValue(recoveryError);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Describe the attached image.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(200);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 1000,
|
|
status: 'started',
|
|
generationProtocolVersion: 1,
|
|
});
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
JSON.stringify({
|
|
status: 409,
|
|
code: ErrorTypes.RESOURCE_RECOVERY_REQUIRED,
|
|
error: 'Attached resources could not be restored',
|
|
}),
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it.each([
|
|
['required', 'required'],
|
|
['future_reason', undefined],
|
|
])('serializes only allowlisted workspace reason %s', async (reason, expectedReason) => {
|
|
const workspaceError = Object.assign(new Error('Choose an attached workspace'), {
|
|
code: ErrorTypes.CODE_WORKSPACE_UNAVAILABLE,
|
|
reason,
|
|
status: 409,
|
|
statusCode: 409,
|
|
});
|
|
const initializeClient = jest.fn().mockRejectedValue(workspaceError);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Edit the project.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
JSON.stringify({
|
|
status: 409,
|
|
code: ErrorTypes.CODE_WORKSPACE_UNAVAILABLE,
|
|
...(expectedReason == null ? {} : { reason: expectedReason }),
|
|
error: 'Choose an attached workspace',
|
|
}),
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it('preserves a stateful scope policy denial in the durable initialization error', async () => {
|
|
const policyError = Object.assign(
|
|
new Error('Stateful code environment is not allowed by this deployment: conversation'),
|
|
{
|
|
code: ErrorTypes.STATEFUL_CODE_ENVIRONMENT_NOT_ALLOWED,
|
|
status: 403,
|
|
statusCode: 403,
|
|
},
|
|
);
|
|
const initializeClient = jest.fn().mockRejectedValue(policyError);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Run code.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(200);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
JSON.stringify({
|
|
status: 403,
|
|
code: ErrorTypes.STATEFUL_CODE_ENVIRONMENT_NOT_ALLOWED,
|
|
error: 'Stateful code environment is not allowed by this deployment: conversation',
|
|
}),
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it.each(['source_missing', 'protocol_mismatch', 'payload_mismatch'])(
|
|
'returns a recovery conflict with its actual reason (%s)',
|
|
async (reason) => {
|
|
const mismatch = new Error('recovery mismatch');
|
|
mismatch.code = 'RECOVERY_PAYLOAD_MISMATCH';
|
|
mismatch.reason = reason;
|
|
mockGenerationJobManager.createJob.mockRejectedValue(mismatch);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Changed words',
|
|
messageId: 'recovered-user-msg',
|
|
clientRequestId: 'steer-recovery:server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'user-123',
|
|
'conversation-123',
|
|
expect.objectContaining({
|
|
recoveredSteerId: 'server-steer-1',
|
|
recoveredSteerPayload: { text: 'Changed words', fileIds: [] },
|
|
}),
|
|
);
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'RECOVERY_PAYLOAD_MISMATCH', reason }),
|
|
);
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it('restores a conditional queued send when a newer generation wins the create CAS', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ generationProtocolVersion: 2 }),
|
|
);
|
|
const mismatch = new Error('predecessor changed');
|
|
mismatch.code = 'GENERATION_PREDECESSOR_MISMATCH';
|
|
mismatch.currentJob = {
|
|
createdAt: 2000,
|
|
status: 'running',
|
|
conversationId: 'conversation-123',
|
|
verified: true,
|
|
};
|
|
mockGenerationJobManager.createJob.mockRejectedValue(mismatch);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
headers: { 'x-librechat-generation-protocol': '2' },
|
|
body: {
|
|
text: 'Queued follow-up C',
|
|
messageId: 'queued-message-c',
|
|
clientRequestId: 'queued-attempt-c',
|
|
expectedPredecessorCreatedAt: 1000,
|
|
conversationId: 'conversation-123',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'user-123',
|
|
'conversation-123',
|
|
expect.objectContaining({ expectedPredecessorCreatedAt: 1000 }),
|
|
);
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
status: 'predecessor_mismatch',
|
|
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
|
error: 'A newer generation became current before this request could start.',
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 2000,
|
|
predecessorVerified: true,
|
|
active: true,
|
|
generationProtocolVersion: 2,
|
|
});
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'queued-attempt-c',
|
|
'conversation-123',
|
|
expect.objectContaining(DEFAULT_OWNED_CLAIM),
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
});
|
|
|
|
it('returns a finite fail-closed mismatch when predecessor evidence expired', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ generationProtocolVersion: 2 }),
|
|
);
|
|
const mismatch = new Error('predecessor evidence expired');
|
|
mismatch.code = 'GENERATION_PREDECESSOR_MISMATCH';
|
|
mismatch.currentJob = {
|
|
createdAt: 1000,
|
|
active: false,
|
|
verified: false,
|
|
};
|
|
mockGenerationJobManager.createJob.mockRejectedValue(mismatch);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
headers: { 'x-librechat-generation-protocol': '2' },
|
|
body: {
|
|
text: 'Queued follow-up C',
|
|
messageId: 'queued-message-c',
|
|
clientRequestId: 'queued-attempt-c',
|
|
expectedPredecessorCreatedAt: 1000,
|
|
conversationId: 'conversation-123',
|
|
generationProtocolVersion: 2,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith({
|
|
status: 'predecessor_mismatch',
|
|
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
|
error: 'The prior generation could not be verified. Please retry.',
|
|
streamId: 'conversation-123',
|
|
conversationId: 'conversation-123',
|
|
generationCreatedAt: 1000,
|
|
predecessorVerified: false,
|
|
active: false,
|
|
generationProtocolVersion: 2,
|
|
});
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'queued-attempt-c',
|
|
'conversation-123',
|
|
expect.objectContaining(DEFAULT_OWNED_CLAIM),
|
|
);
|
|
});
|
|
|
|
it('does not consume a recovered steer when client initialization fails before persistence', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ claimToken: 'recovery-claim-token' }),
|
|
);
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('provider init failed'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Recovered words',
|
|
messageId: 'recovered-user-msg',
|
|
clientRequestId: 'recovery-attempt-1',
|
|
recoverySteerId: 'server-steer-1',
|
|
overrideUserMessageId: 'server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'user-123',
|
|
'conversation-123',
|
|
expect.objectContaining({ recoveredSteerId: 'server-steer-1' }),
|
|
);
|
|
expect(mockGenerationJobManager.claimGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'recovery-attempt-1',
|
|
'conversation-123',
|
|
'conversation-123',
|
|
1,
|
|
);
|
|
expect(req.body.overrideUserMessageId).toBe('server-steer-1__0');
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'provider init failed',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it('fails closed when a recovered-turn client derives skipSaveUserMessage', async () => {
|
|
const client = {
|
|
options: {},
|
|
skipSaveUserMessage: true,
|
|
sendMessage: jest.fn(),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Recovered words',
|
|
messageId: 'recovered-user-msg',
|
|
clientRequestId: 'steer-recovery:server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
|
|
expect(client.sendMessage).not.toHaveBeenCalled();
|
|
expect(mockSaveMessage).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'Recovered steer cannot skip user message persistence',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it('does not consume a recovered steer when the explicit normal-flow user save returns no row', async () => {
|
|
const userMessage = {
|
|
messageId: 'recovered-user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Recovered words',
|
|
};
|
|
mockSaveMessage.mockImplementation(async (_reqCtx, message) => {
|
|
if (message?.messageId === userMessage.messageId) {
|
|
return undefined;
|
|
}
|
|
return {};
|
|
});
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async (...args) => {
|
|
signalFinished(args);
|
|
});
|
|
const client = {
|
|
options: {},
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
clientRequestId: 'steer-recovery:server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
const finishArgs = await finished;
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
userMessage,
|
|
expect.objectContaining({ context: expect.stringContaining('resumable user message') }),
|
|
);
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.publishTerminalClaim).toHaveBeenCalledWith(
|
|
expect.objectContaining({ streamId: 'conversation-123', status: 'complete' }),
|
|
null,
|
|
);
|
|
expect(finishArgs).toEqual([
|
|
expect.objectContaining({ streamId: 'conversation-123', status: 'complete' }),
|
|
]);
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('does not consume a recovered steer when the explicit HITL user save fails', async () => {
|
|
const userMessage = {
|
|
messageId: 'recovered-user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Recovered words before approval',
|
|
};
|
|
const userSaveError = new Error('paused user row unavailable');
|
|
mockSaveMessage.mockRejectedValue(userSaveError);
|
|
let signalPauseFailed;
|
|
const pauseFailed = new Promise((resolve) => {
|
|
signalPauseFailed = resolve;
|
|
});
|
|
mockGenerationJobManager.failPausePersistence.mockImplementation(async (...args) => {
|
|
signalPauseFailed(args);
|
|
return true;
|
|
});
|
|
const client = {
|
|
options: {},
|
|
pendingApproval: { actionId: 'action-paused-save-fails' },
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
clientRequestId: 'steer-recovery:server-steer-1',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
const failArgs = await pauseFailed;
|
|
await nextTick();
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
userMessage,
|
|
expect.objectContaining({ context: expect.stringContaining('before HITL pause') }),
|
|
);
|
|
expect(mockGenerationJobManager.steering.consumeRecovered).not.toHaveBeenCalled();
|
|
expect(failArgs).toEqual([
|
|
'conversation-123',
|
|
'action-paused-save-fails',
|
|
userSaveError.message,
|
|
1000,
|
|
]);
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockDeleteAgentCheckpoint).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
undefined,
|
|
undefined,
|
|
{ checkpointNamespace: '1000' },
|
|
);
|
|
});
|
|
|
|
it('fails the pause when a normal user-row retry resolves without a durable row', async () => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Pause only after my row is durable.',
|
|
};
|
|
mockSaveMessage.mockResolvedValueOnce(undefined);
|
|
let signalPauseFailed;
|
|
const pauseFailed = new Promise((resolve) => {
|
|
signalPauseFailed = resolve;
|
|
});
|
|
mockGenerationJobManager.failPausePersistence.mockImplementation(async (...args) => {
|
|
signalPauseFailed(args);
|
|
return true;
|
|
});
|
|
const client = {
|
|
options: {},
|
|
pendingApproval: { actionId: 'action-normal-user-falsy' },
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
const failArgs = await pauseFailed;
|
|
await nextTick();
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
userMessage,
|
|
expect.objectContaining({ context: expect.stringContaining('before HITL pause') }),
|
|
);
|
|
expect(failArgs).toEqual([
|
|
'conversation-123',
|
|
'action-normal-user-falsy',
|
|
'User message could not be persisted before HITL pause',
|
|
1000,
|
|
]);
|
|
expect(mockSaveMessage).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
[
|
|
'message row',
|
|
{ message: undefined, conversation: { conversationId: 'conversation-123' } },
|
|
'User message could not be persisted before HITL pause',
|
|
],
|
|
[
|
|
'conversation row',
|
|
{ message: { messageId: 'user-msg' }, conversation: undefined },
|
|
'Conversation could not be persisted before HITL pause',
|
|
],
|
|
])(
|
|
'fails the pause when the BaseClient retry returns no %s',
|
|
async (_label, retryResult, expectedError) => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Repair the whole user turn before approval.',
|
|
};
|
|
let signalPauseFailed;
|
|
const pauseFailed = new Promise((resolve) => {
|
|
signalPauseFailed = resolve;
|
|
});
|
|
mockGenerationJobManager.failPausePersistence.mockImplementation(async (...args) => {
|
|
signalPauseFailed(args);
|
|
return true;
|
|
});
|
|
const client = {
|
|
options: {},
|
|
pendingApproval: { actionId: 'action-base-client-retry-falsy' },
|
|
skipSaveUserMessage: false,
|
|
skipSaveConvo: false,
|
|
getSaveOptions: jest.fn(() => ({ endpoint: 'agents' })),
|
|
saveMessageToDatabase: jest.fn().mockResolvedValue(retryResult),
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
const failArgs = await pauseFailed;
|
|
await nextTick();
|
|
|
|
expect(client.saveMessageToDatabase).toHaveBeenCalledWith(
|
|
userMessage,
|
|
{ endpoint: 'agents' },
|
|
'user-123',
|
|
);
|
|
expect(failArgs).toEqual([
|
|
'conversation-123',
|
|
'action-base-client-retry-falsy',
|
|
expectedError,
|
|
1000,
|
|
]);
|
|
expect(mockSaveMessage).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it('does not let generic cleanup touch a newer generation after paused persistence loses ownership', async () => {
|
|
let signalPauseFailed;
|
|
const pauseFailed = new Promise((resolve) => {
|
|
signalPauseFailed = resolve;
|
|
});
|
|
mockSaveMessage.mockResolvedValue(undefined);
|
|
mockGenerationJobManager.failPausePersistence.mockImplementation(async (...args) => {
|
|
signalPauseFailed(args);
|
|
return false;
|
|
});
|
|
const client = {
|
|
options: {},
|
|
jobCreatedAt: 1000,
|
|
pendingApproval: { actionId: 'action-replaced-before-pause-save' },
|
|
skipSaveUserMessage: true,
|
|
sendMessage: jest.fn(async () => ({
|
|
messageId: 'response-msg',
|
|
content: [{ type: 'text', text: 'Waiting for approval.' }],
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
})),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Pause before the tool runs.',
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
const failArgs = await pauseFailed;
|
|
await nextTick();
|
|
|
|
expect(failArgs).toEqual([
|
|
'conversation-123',
|
|
'action-replaced-before-pause-save',
|
|
'Paused response could not be persisted as unfinished',
|
|
1000,
|
|
]);
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockDeleteAgentCheckpoint).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('persists a paused response as unfinished before releasing its Stop/resume barrier', async () => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Pause before the tool runs.',
|
|
};
|
|
const response = {
|
|
messageId: 'response-msg',
|
|
parentMessageId: userMessage.messageId,
|
|
conversationId: 'conversation-123',
|
|
content: [{ type: 'text', text: 'Waiting for approval.' }],
|
|
};
|
|
let signalPauseSaveStarted;
|
|
const pauseSaveStarted = new Promise((resolve) => {
|
|
signalPauseSaveStarted = resolve;
|
|
});
|
|
let releasePauseSave;
|
|
const pauseSaveGate = new Promise((resolve) => {
|
|
releasePauseSave = resolve;
|
|
});
|
|
mockSaveMessage.mockImplementation(async () => {
|
|
signalPauseSaveStarted();
|
|
await pauseSaveGate;
|
|
return {};
|
|
});
|
|
let observedHookResult;
|
|
const completedResponseWrite = jest.fn();
|
|
const exposePendingApproval = jest.fn().mockResolvedValue(undefined);
|
|
const client = {
|
|
options: {},
|
|
jobCreatedAt: 1000,
|
|
pendingApproval: { actionId: 'action-pause-barrier' },
|
|
exposePendingApproval,
|
|
skipSaveUserMessage: false,
|
|
skipSaveConvo: false,
|
|
getSaveOptions: jest.fn(() => ({ endpoint: 'agents' })),
|
|
saveMessageToDatabase: jest.fn().mockResolvedValue({
|
|
message: userMessage,
|
|
conversation: { conversationId: 'conversation-123' },
|
|
}),
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, response.messageId);
|
|
observedHookResult = await options.beforeResponsePersistence(response);
|
|
response.databasePromise = observedHookResult
|
|
? completedResponseWrite()
|
|
: Promise.resolve({
|
|
persistenceSkipped: true,
|
|
conversation: { conversationId: 'conversation-123' },
|
|
});
|
|
return response;
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await pauseSaveStarted;
|
|
|
|
expect(observedHookResult).toBe(false);
|
|
expect(completedResponseWrite).not.toHaveBeenCalled();
|
|
expect(client.saveMessageToDatabase).toHaveBeenCalledWith(
|
|
userMessage,
|
|
{ endpoint: 'agents' },
|
|
'user-123',
|
|
);
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
messageId: response.messageId,
|
|
content: response.content,
|
|
unfinished: true,
|
|
}),
|
|
expect.objectContaining({ context: expect.stringContaining('persist unfinished') }),
|
|
);
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).not.toHaveBeenCalled();
|
|
|
|
releasePauseSave();
|
|
await nextTick();
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.approvals.ownsPausePersistence).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'action-pause-barrier',
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.approvals.finishPausePersistence).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'action-pause-barrier',
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.failPausePersistence).not.toHaveBeenCalled();
|
|
expect(client.saveMessageToDatabase.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockSaveMessage.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockSaveMessage.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.approvals.finishPausePersistence.mock.invocationCallOrder[0],
|
|
);
|
|
expect(exposePendingApproval.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.approvals.finishPausePersistence.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockGenerationJobManager.claimTerminalJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
describe('failed-turn persistence', () => {
|
|
const conversationId = 'conversation-123';
|
|
|
|
const createFailedRequest = (bodyOverrides = {}) => ({
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Hello with a removed model.',
|
|
messageId: 'user-message',
|
|
parentMessageId: 'prior-response',
|
|
conversationId,
|
|
endpointOption: {
|
|
endpoint: 'azureOpenAI',
|
|
modelOptions: { model: 'gpt-4o' },
|
|
},
|
|
...bodyOverrides,
|
|
},
|
|
config: {},
|
|
});
|
|
|
|
async function flushBackgroundGeneration() {
|
|
for (let i = 0; i < 10; i++) {
|
|
await nextTick();
|
|
}
|
|
}
|
|
|
|
it('persists an initialization failure before terminal error publication', async () => {
|
|
const events = [];
|
|
mockSaveConvo.mockImplementation(async () => {
|
|
events.push('turn-persisted');
|
|
return {};
|
|
});
|
|
mockGenerationJobManager.completeJob.mockImplementation(
|
|
async (_streamId, _error, _createdAt, options) => {
|
|
await options.beforeErrorPublication();
|
|
events.push('error-published');
|
|
return true;
|
|
},
|
|
);
|
|
const initializeClient = jest
|
|
.fn()
|
|
.mockRejectedValue(new Error('The model "gpt-4o" is not available.'));
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
initializeClient,
|
|
null,
|
|
);
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
messageId: 'user-message',
|
|
parentMessageId: 'prior-response',
|
|
conversationId,
|
|
text: 'Hello with a removed model.',
|
|
isCreatedByUser: true,
|
|
error: false,
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
messageId: 'user-message_',
|
|
parentMessageId: 'user-message',
|
|
conversationId,
|
|
endpoint: 'azureOpenAI',
|
|
model: 'gpt-4o',
|
|
text: 'The model "gpt-4o" is not available.',
|
|
error: true,
|
|
isCreatedByUser: false,
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
expect(events).toEqual(['turn-persisted', 'error-published']);
|
|
expect(mockSaveConvo).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
{ conversationId },
|
|
expect.objectContaining({ noUpsert: true }),
|
|
);
|
|
});
|
|
|
|
it('allows a follow-up to chain from the persisted failed response', async () => {
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('model unavailable'));
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
initializeClient,
|
|
null,
|
|
);
|
|
|
|
expect(mockSaveMessage.mock.calls.map(([, message]) => message.messageId)).toContain(
|
|
'user-message_',
|
|
);
|
|
mockGetMessages.mockResolvedValue([{ _id: 'persisted-error-turn' }]);
|
|
const followUpRes = createResumableResponse();
|
|
|
|
await AgentController(
|
|
createFailedRequest({
|
|
text: 'Retry with a valid model.',
|
|
messageId: 'follow-up-user',
|
|
parentMessageId: 'user-message_',
|
|
}),
|
|
followUpRes,
|
|
jest.fn(),
|
|
initializeClient,
|
|
null,
|
|
);
|
|
|
|
expect(followUpRes.status).not.toHaveBeenCalledWith(409);
|
|
expect(mockCheckAndIncrementPendingRequest).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it('persists failures raised before generation saves any message', async () => {
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn().mockRejectedValue(new Error('provider exploded')),
|
|
};
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await flushBackgroundGeneration();
|
|
|
|
expect(mockResolveAgentTurnExecutionPlan).toHaveBeenCalledTimes(1);
|
|
expect(mockResolveAgentTurnExecutionPlan).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
conversationId,
|
|
isEvent: false,
|
|
event: undefined,
|
|
}),
|
|
);
|
|
expect(mockExecuteAgentEventActor).not.toHaveBeenCalled();
|
|
expect(mockCreateAgentEventActionRecorder).not.toHaveBeenCalled();
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
messageId: 'user-message_',
|
|
text: 'provider exploded',
|
|
error: true,
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'provider exploded',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
});
|
|
|
|
it('uses the live user identity after generation starts', async () => {
|
|
const serverUserMessage = {
|
|
messageId: 'server-user',
|
|
parentMessageId: 'prior-response',
|
|
conversationId,
|
|
sender: 'User',
|
|
text: 'Hello with a removed model.',
|
|
isCreatedByUser: true,
|
|
};
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(serverUserMessage, 'server-response-uuid');
|
|
throw new Error('failed after onStart');
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await flushBackgroundGeneration();
|
|
|
|
const savedIds = mockSaveMessage.mock.calls.map(([, message]) => message.messageId);
|
|
expect(savedIds).toEqual(expect.arrayContaining(['server-user', 'server-user_']));
|
|
expect(savedIds).not.toContain('user-message_');
|
|
});
|
|
|
|
it('points a failed turn at the trace of the run that failed', async () => {
|
|
const traceFields = {
|
|
langfuseSampled: true,
|
|
langfuseDestinationIds: ['destination-1'],
|
|
langfuseRunId: 'server-response-uuid',
|
|
};
|
|
mockGetFailedTurnTraceFields.mockResolvedValue(traceFields);
|
|
const serverUserMessage = {
|
|
messageId: 'server-user',
|
|
parentMessageId: 'prior-response',
|
|
conversationId,
|
|
sender: 'User',
|
|
text: 'Hello with a removed model.',
|
|
isCreatedByUser: true,
|
|
};
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(serverUserMessage, 'server-response-uuid');
|
|
client.run = {};
|
|
throw new Error('failed inside the run');
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await flushBackgroundGeneration();
|
|
|
|
const errorRow = mockSaveMessage.mock.calls
|
|
.map(([, message]) => message)
|
|
.find((message) => message.messageId === 'server-user_');
|
|
expect(mockGetFailedTurnTraceFields).toHaveBeenCalledWith(expect.anything(), {
|
|
messageId: 'server-user_',
|
|
runId: 'server-response-uuid',
|
|
runCreated: true,
|
|
});
|
|
expect(errorRow).toMatchObject({ error: true, isCreatedByUser: false, ...traceFields });
|
|
});
|
|
|
|
it('tells the trace lookup when a failure came before the run was created', async () => {
|
|
mockGetFailedTurnTraceFields.mockResolvedValue({});
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(
|
|
{ messageId: 'server-user', conversationId, isCreatedByUser: true, text: 'Hi' },
|
|
'server-response-uuid',
|
|
);
|
|
throw new Error('failed before the run');
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await flushBackgroundGeneration();
|
|
|
|
const errorRow = mockSaveMessage.mock.calls
|
|
.map(([, message]) => message)
|
|
.find((message) => message.messageId === 'server-user_');
|
|
expect(mockGetFailedTurnTraceFields).toHaveBeenCalledWith(
|
|
expect.anything(),
|
|
expect.objectContaining({ runCreated: false }),
|
|
);
|
|
expect(errorRow).toMatchObject({ error: true });
|
|
expect(errorRow).not.toHaveProperty('langfuseSampled');
|
|
});
|
|
|
|
it('does not overwrite an existing response row', async () => {
|
|
mockGetMessages.mockResolvedValue([{ _id: 'already-saved' }]);
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockRejectedValue(new Error('late failure')),
|
|
null,
|
|
);
|
|
|
|
expect(mockSaveMessage).not.toHaveBeenCalled();
|
|
expect(mockSaveConvo).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('does not persist a first turn whose code environment decision was rejected', async () => {
|
|
const res = createResumableResponse();
|
|
mockGenerationJobManager.claimGeneration.mockImplementation(
|
|
async (_userId, _clientRequestId, streamId, claimedConversationId) =>
|
|
wonGenerationClaim({ streamId, conversationId: claimedConversationId }),
|
|
);
|
|
const req = createFailedRequest({
|
|
conversationId: undefined,
|
|
clientRequestId: 'invalid-code-decision',
|
|
codeEnvironmentMode: 'attached',
|
|
codeWorkspaces: undefined,
|
|
});
|
|
const workspaceError = Object.assign(new Error('Choose an attached workspace'), {
|
|
code: ErrorTypes.CODE_WORKSPACE_UNAVAILABLE,
|
|
reason: 'required',
|
|
status: 409,
|
|
statusCode: 409,
|
|
});
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn().mockRejectedValue(workspaceError), null);
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
expect.any(String),
|
|
expect.stringContaining(ErrorTypes.CODE_WORKSPACE_UNAVAILABLE),
|
|
1000,
|
|
);
|
|
expect(mockSaveMessage).not.toHaveBeenCalled();
|
|
expect(mockSaveConvo).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('creates the conversation row for a failed first turn', async () => {
|
|
const res = createResumableResponse();
|
|
mockGenerationJobManager.claimGeneration.mockImplementation(
|
|
async (_userId, _clientRequestId, streamId, claimedConversationId) =>
|
|
wonGenerationClaim({ streamId, conversationId: claimedConversationId }),
|
|
);
|
|
const req = createFailedRequest({
|
|
conversationId: undefined,
|
|
clientRequestId: 'failed-new-conversation',
|
|
agent_id: 'unverified-agent',
|
|
parentMessageId: '00000000-0000-0000-0000-000000000000',
|
|
endpointOption: {
|
|
endpoint: 'azureOpenAI',
|
|
agent_id: 'unverified-agent',
|
|
modelOptions: { model: 'gpt-4o' },
|
|
chatProjectId: '507f1f77bcf86cd799439011',
|
|
},
|
|
codeEnvironmentMode: 'attached',
|
|
codeWorkspaces: [{ environmentId: 'personal-vm', workspaceId: 'project-a' }],
|
|
});
|
|
const initializeClient = jest.fn().mockImplementation(async ({ req: request }) => {
|
|
request._codeEnvironmentDecision = {
|
|
mode: 'attached',
|
|
codeWorkspaces: [{ environmentId: 'personal-vm', workspaceId: 'project-a' }],
|
|
};
|
|
throw new Error('model unavailable');
|
|
});
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
const mintedConversationId = res.json.mock.calls[0][0].conversationId;
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.any(Object),
|
|
expect.objectContaining({
|
|
messageId: 'user-message_',
|
|
conversationId: mintedConversationId,
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockSaveConvo).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
expect.objectContaining({
|
|
conversationId: mintedConversationId,
|
|
endpoint: 'azureOpenAI',
|
|
model: 'gpt-4o',
|
|
chatProjectId: '507f1f77bcf86cd799439011',
|
|
codeEnvironmentMode: 'attached',
|
|
codeWorkspaces: [{ environmentId: 'personal-vm', workspaceId: 'project-a' }],
|
|
}),
|
|
expect.objectContaining({ initialAgentId: null }),
|
|
);
|
|
});
|
|
|
|
it('records the decision a failed turn of a saved chat ran under', async () => {
|
|
const codeWorkspaces = [{ environmentId: 'personal-vm', workspaceId: 'project-a' }];
|
|
const initializeClient = jest.fn().mockImplementation(async ({ req: request }) => {
|
|
request._codeEnvironmentDecision = { mode: 'attached', codeWorkspaces };
|
|
throw new Error('model unavailable');
|
|
});
|
|
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
initializeClient,
|
|
null,
|
|
);
|
|
|
|
expect(mockSaveConvo).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
{ conversationId, codeEnvironmentMode: 'attached', codeWorkspaces },
|
|
expect.objectContaining({ noUpsert: true }),
|
|
);
|
|
});
|
|
|
|
it('does not rewrite the decision a chat already holds', async () => {
|
|
const req = createFailedRequest();
|
|
req.resolvedConversation = {
|
|
conversationId,
|
|
codeEnvironmentMode: 'attached',
|
|
codeWorkspaces: [{ environmentId: 'personal-vm', workspaceId: 'project-a' }],
|
|
};
|
|
const initializeClient = jest.fn().mockImplementation(async ({ req: request }) => {
|
|
request._codeEnvironmentDecision = {
|
|
mode: 'attached',
|
|
codeWorkspaces: [{ environmentId: 'personal-vm', workspaceId: 'project-b' }],
|
|
};
|
|
throw new Error('model unavailable');
|
|
});
|
|
|
|
await AgentController(req, createResumableResponse(), jest.fn(), initializeClient, null);
|
|
|
|
expect(mockSaveConvo).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
{ conversationId },
|
|
expect.objectContaining({ noUpsert: true }),
|
|
);
|
|
});
|
|
|
|
it('leaves a saved chat undecided when the turn resolved no decision', async () => {
|
|
await AgentController(
|
|
createFailedRequest(),
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockRejectedValue(new Error('model unavailable')),
|
|
null,
|
|
);
|
|
|
|
expect(mockSaveConvo).toHaveBeenCalledWith(
|
|
expect.objectContaining({ userId: 'user-123' }),
|
|
{ conversationId },
|
|
expect.objectContaining({ noUpsert: true }),
|
|
);
|
|
});
|
|
});
|
|
|
|
it.each([
|
|
new (jest.requireActual('@librechat/api').OpenIDReauthRequiredError)(
|
|
'Please sign in again to continue using this MCP server.',
|
|
),
|
|
new (jest.requireActual('@librechat/api').MCPAuthenticationRejectedError)('private-mcp', false),
|
|
new (jest.requireActual('@librechat/api').MCPAuthenticationRefreshError)(
|
|
new Error('upstream refresh unavailable'),
|
|
),
|
|
new (jest.requireActual('@librechat/api').OboTokenResolutionError)(
|
|
'session_refresh_failed',
|
|
'Please sign in again',
|
|
false,
|
|
),
|
|
new (jest.requireActual('@librechat/api').OboTokenResolutionError)(
|
|
'exchange_failed',
|
|
'Temporary exchange failure',
|
|
true,
|
|
),
|
|
])(
|
|
'publishes an actionable MCP initialization failure without an HTTP 401: %s',
|
|
async (error) => {
|
|
const initializeClient = jest.fn().mockRejectedValue(error);
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Use the private tool.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(200);
|
|
expect(res.status).not.toHaveBeenCalledWith(401);
|
|
expect(initializeClient).toHaveBeenCalledTimes(1);
|
|
let metadata = {
|
|
status: error.status ?? error.statusCode,
|
|
...(error.code ? { code: error.code } : {}),
|
|
};
|
|
if (error.name === 'OboTokenResolutionError') {
|
|
metadata = {
|
|
status: error.retryable ? 503 : 403,
|
|
code: error.retryable
|
|
? 'MCP_AUTHENTICATION_REFRESH_FAILED'
|
|
: 'MCP_AUTHENTICATION_REJECTED',
|
|
retryable: error.retryable,
|
|
};
|
|
}
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
JSON.stringify({
|
|
...metadata,
|
|
error: error.message,
|
|
}),
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
},
|
|
);
|
|
|
|
it('finalizes the failed job before releasing the idempotency claim', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('init boom after res.json'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Start fails after the initial JSON.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
expect.any(String),
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
// completeJob must finalize the failed job BEFORE the claim is released, or a racing
|
|
// retry could win the key, createJob the same streamId, and be aborted by this completeJob.
|
|
expect(mockGenerationJobManager.completeJob.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.releaseGeneration.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it('still releases the claim and pending slot when completeJob fails during init-error cleanup', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
mockGenerationJobManager.completeJob.mockRejectedValue(new Error('store hiccup'));
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('init boom after res.json'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Start fails while the store is degraded.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
// A completeJob rejection must not wedge the retry behind the claim or leak the slot.
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
});
|
|
|
|
it('still finalizes and releases when streaming the initialization error fails', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
mockGenerationJobManager.emitError.mockRejectedValue(new Error('publish failed'));
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('init boom after res.json'));
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Start fails while Redis publish is degraded.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'init boom after res.json',
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockStartupTelemetry.end).toHaveBeenCalledWith('error', expect.any(Error));
|
|
});
|
|
|
|
it('finalizes and disposes a client aborted during initialization before releasing the slot', async () => {
|
|
const abortController = new AbortController();
|
|
let resolveCompletion;
|
|
let signalCompletionStarted;
|
|
const completionStarted = new Promise((resolve) => {
|
|
signalCompletionStarted = resolve;
|
|
});
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
readyPromise: Promise.resolve(),
|
|
abortController,
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
mockGenerationJobManager.completeJob.mockImplementation(() => {
|
|
signalCompletionStarted();
|
|
return new Promise((resolve) => {
|
|
resolveCompletion = resolve;
|
|
});
|
|
});
|
|
const client = { options: {} };
|
|
const initializeClient = jest.fn(async ({ signal }) => {
|
|
expect(signal).toBe(abortController.signal);
|
|
abortController.abort();
|
|
return { client };
|
|
});
|
|
const conversationId = 'conversation-123';
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Stop during initialization.',
|
|
messageId: 'user-msg',
|
|
conversationId,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
const controllerPromise = AgentController(req, res, jest.fn(), initializeClient, null);
|
|
await completionStarted;
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
conversationId,
|
|
'Request aborted during initialization',
|
|
1000,
|
|
);
|
|
expect(mockDecrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockDisposeClient).not.toHaveBeenCalled();
|
|
|
|
resolveCompletion();
|
|
await controllerPromise;
|
|
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockDisposeClient).toHaveBeenCalledTimes(1);
|
|
expect(mockDisposeClient).toHaveBeenCalledWith(client);
|
|
expect(mockStartupTelemetry.end).toHaveBeenCalledWith('aborted');
|
|
});
|
|
|
|
it('awaits background error finalization before releasing the slot and always disposes', async () => {
|
|
const generationError = new Error('generation failed');
|
|
let rejectCompletion;
|
|
let signalCompletionStarted;
|
|
const completionStarted = new Promise((resolve) => {
|
|
signalCompletionStarted = resolve;
|
|
});
|
|
mockGenerationJobManager.completeJob.mockImplementation(() => {
|
|
signalCompletionStarted();
|
|
return new Promise((_, reject) => {
|
|
rejectCompletion = reject;
|
|
});
|
|
});
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn().mockRejectedValue(generationError),
|
|
};
|
|
const initializeClient = jest.fn().mockResolvedValue({ client });
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Fail after initialization.',
|
|
messageId: 'user-msg',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = createResumableResponse();
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
await completionStarted;
|
|
|
|
// completeJob owns both the terminal CAS and error publication; the
|
|
// controller must not publish before terminal ownership is established.
|
|
expect(mockGenerationJobManager.emitError).not.toHaveBeenCalled();
|
|
expect(mockDecrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockDisposeClient).not.toHaveBeenCalled();
|
|
|
|
rejectCompletion(new Error('store failed'));
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
generationError.message,
|
|
1000,
|
|
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
|
|
);
|
|
expect(mockGenerationJobManager.completeJob.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockDecrementPendingRequest.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockDecrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockDisposeClient).toHaveBeenCalledWith(client);
|
|
});
|
|
|
|
it('claims terminal ownership before FINAL and always finishes the winning claim', async () => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Finish normally.',
|
|
};
|
|
const terminalClaim = {
|
|
streamId: 'conversation-123',
|
|
createdAt: 1000,
|
|
status: 'complete',
|
|
persistencePending: true,
|
|
drainedSteers: [],
|
|
};
|
|
mockGenerationJobManager.getJob.mockResolvedValue({
|
|
createdAt: 1000,
|
|
status: 'running',
|
|
});
|
|
mockGenerationJobManager.claimTerminalJob.mockResolvedValue(terminalClaim);
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async () => signalFinished());
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
parentMessageId: userMessage.messageId,
|
|
conversationId: 'conversation-123',
|
|
content: [{ type: 'text', text: 'Done.' }],
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123', title: 'Existing' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await finished;
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.claimTerminalJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'complete',
|
|
undefined,
|
|
1000,
|
|
{ persistencePending: true },
|
|
);
|
|
expect(mockGenerationJobManager.publishTerminalClaim).toHaveBeenCalledWith(
|
|
terminalClaim,
|
|
expect.objectContaining({ final: true }),
|
|
);
|
|
expect(mockGenerationJobManager.finishTerminalJob).toHaveBeenCalledWith(terminalClaim);
|
|
expect(mockGenerationJobManager.claimTerminalJob.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.publishTerminalClaim.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockGenerationJobManager.publishTerminalClaim.mock.invocationCallOrder[0]).toBeLessThan(
|
|
mockGenerationJobManager.finishTerminalJob.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.steering.closeAndDrain).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
[
|
|
'completed',
|
|
undefined,
|
|
'api/server/controllers/agents/request.js - resumable response end',
|
|
'Response message could not be persisted before terminal publication',
|
|
],
|
|
[
|
|
'unfinished preempt',
|
|
{
|
|
getPreemptStats: () => ({ emptyBoundaries: 1 }),
|
|
getHaltReason: () => 'preempt_incomplete',
|
|
},
|
|
'api/server/controllers/agents/request.js - terminal response unfinished',
|
|
'Terminal response could not be persisted as unfinished',
|
|
],
|
|
])(
|
|
'publishes reconciliation when a BaseClient-marked %s response save returned no row',
|
|
async (_label, run, expectedContext, expectedError) => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Verify terminal persistence.',
|
|
};
|
|
const terminalClaim = {
|
|
streamId: 'conversation-123',
|
|
createdAt: 1000,
|
|
status: 'complete',
|
|
persistencePending: true,
|
|
drainedSteers: [],
|
|
};
|
|
mockGenerationJobManager.claimTerminalJob.mockResolvedValue(terminalClaim);
|
|
mockSaveMessage.mockImplementation(async (_reqCtx, message) =>
|
|
message?.messageId === userMessage.messageId ? {} : undefined,
|
|
);
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async () => signalFinished());
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(['response-msg']),
|
|
skipSaveUserMessage: false,
|
|
...(run && { run }),
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
parentMessageId: userMessage.messageId,
|
|
conversationId: 'conversation-123',
|
|
content: [{ type: 'text', text: 'Done.' }],
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123', title: 'Existing' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await finished;
|
|
await nextTick();
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.anything(),
|
|
expect.objectContaining({
|
|
messageId: 'response-msg',
|
|
unfinished: _label === 'unfinished preempt',
|
|
}),
|
|
{ context: expectedContext },
|
|
);
|
|
expect(mockGenerationJobManager.publishTerminalClaim).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.publishTerminalClaim).toHaveBeenCalledWith(
|
|
terminalClaim,
|
|
null,
|
|
);
|
|
expect(mockGenerationJobManager.finishTerminalJob).toHaveBeenCalledWith(terminalClaim);
|
|
expect(mockGenerationJobManager.completeJob).not.toHaveBeenCalled();
|
|
expect(mockLogger.error).toHaveBeenCalledWith(
|
|
expect.stringContaining('Terminal persistence failed'),
|
|
expect.objectContaining({ message: expectedError }),
|
|
);
|
|
},
|
|
);
|
|
|
|
it('suppresses the completed response write when Stop already won terminal ownership', async () => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Race Stop against completion.',
|
|
};
|
|
let signalClaimStarted;
|
|
const claimStarted = new Promise((resolve) => {
|
|
signalClaimStarted = resolve;
|
|
});
|
|
let resolveTerminalClaim;
|
|
const terminalClaimGate = new Promise((resolve) => {
|
|
resolveTerminalClaim = resolve;
|
|
});
|
|
mockGenerationJobManager.claimTerminalJob.mockImplementation(async () => {
|
|
signalClaimStarted();
|
|
return terminalClaimGate;
|
|
});
|
|
const completedResponseWrite = jest.fn(() =>
|
|
Promise.resolve({ conversation: { conversationId: 'conversation-123' } }),
|
|
);
|
|
let observedHookResult;
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
const response = {
|
|
messageId: 'response-msg',
|
|
parentMessageId: userMessage.messageId,
|
|
conversationId: 'conversation-123',
|
|
content: [{ type: 'text', text: 'Stale completion.' }],
|
|
};
|
|
observedHookResult = await options.beforeResponsePersistence(response);
|
|
response.databasePromise = observedHookResult
|
|
? completedResponseWrite()
|
|
: Promise.resolve({ persistenceSkipped: true });
|
|
return response;
|
|
}),
|
|
};
|
|
let signalSettled;
|
|
const settled = new Promise((resolve) => {
|
|
signalSettled = resolve;
|
|
});
|
|
mockDecrementPendingRequest.mockImplementationOnce(async () => signalSettled());
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await claimStarted;
|
|
expect(completedResponseWrite).not.toHaveBeenCalled();
|
|
|
|
// Model the Stop endpoint winning the durable CAS while completion is
|
|
// blocked at the exact pre-write ownership hook.
|
|
resolveTerminalClaim(null);
|
|
await settled;
|
|
await nextTick();
|
|
|
|
expect(observedHookResult).toBe(false);
|
|
expect(mockGenerationJobManager.claimTerminalJob).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.claimTerminalJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'complete',
|
|
undefined,
|
|
1000,
|
|
{ persistencePending: true },
|
|
);
|
|
expect(completedResponseWrite).not.toHaveBeenCalled();
|
|
expect(mockSaveMessage).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.publishTerminalClaim).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.finishTerminalJob).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('finishes a claimed request job when FINAL publication throws', async () => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: 'parent-msg',
|
|
conversationId: 'conversation-123',
|
|
text: 'Finish despite transport failure.',
|
|
};
|
|
mockGenerationJobManager.getJob.mockResolvedValue({ createdAt: 1000, status: 'running' });
|
|
mockGenerationJobManager.publishTerminalClaim.mockRejectedValue(
|
|
new Error('done publish failed'),
|
|
);
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async () => signalFinished());
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'conversation-123', title: 'Existing' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: userMessage.text,
|
|
messageId: userMessage.messageId,
|
|
parentMessageId: userMessage.parentMessageId,
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await finished;
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.finishTerminalJob).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.finishTerminalJob.mock.invocationCallOrder[0]).toBeGreaterThan(
|
|
mockGenerationJobManager.publishTerminalClaim.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it.each(['submitted', 'persisted', 'overridden'])(
|
|
'preserves %s workspace selections in the runtime envelope',
|
|
async (source) => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
|
const codeWorkspaces = [{ environmentId: 'machine-a', workspaceId: 'project-b' }];
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
...(source !== 'submitted'
|
|
? { resolvedConversation: { conversationId: 'conversation-123', codeWorkspaces } }
|
|
: {}),
|
|
body: {
|
|
...(source === 'submitted' ? { codeWorkspaces } : {}),
|
|
...(source === 'overridden' ? { overrideConvoId: 'target-conversation' } : {}),
|
|
text: 'Fresh submission.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = {
|
|
headersSent: true,
|
|
json: jest.fn(() => {
|
|
res.headersSent = true;
|
|
}),
|
|
status: jest.fn(() => res),
|
|
set: jest.fn(),
|
|
};
|
|
|
|
await AgentController(req, res, jest.fn(), initializeClient, null);
|
|
|
|
expect(initializeClient).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
requestBody: expect.objectContaining({
|
|
conversationId: source === 'overridden' ? 'target-conversation' : 'conversation-123',
|
|
}),
|
|
}),
|
|
);
|
|
expect(initializeClient.mock.calls[0][0].requestBody.codeWorkspaces).toEqual(
|
|
source === 'overridden' ? undefined : codeWorkspaces,
|
|
);
|
|
expect(mockCheckAndIncrementPendingRequest).toHaveBeenCalledWith('user-123');
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'conversation-123',
|
|
'user-123',
|
|
'conversation-123',
|
|
expect.objectContaining({
|
|
startupTelemetry: mockStartupTelemetry,
|
|
initialMetadata: expect.objectContaining({
|
|
conversationId: 'conversation-123',
|
|
endpoint: 'agents',
|
|
}),
|
|
}),
|
|
);
|
|
},
|
|
);
|
|
|
|
it('retains terminal metadata for the first event in an empty bound actor thread', async () => {
|
|
const expiredAt = new Date(Date.now() + 60_000);
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({
|
|
streamId: 'child-conversation',
|
|
conversationId: 'child-conversation',
|
|
}),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
mockIsSubagentOwnerAdmissible.mockResolvedValue(false);
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Continue from event.',
|
|
messageId: 'user-msg',
|
|
parentMessageId: '00000000-0000-0000-0000-000000000000',
|
|
clientRequestId: 'req-event',
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event',
|
|
expectedAction: {
|
|
toolName: 'submit_move',
|
|
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
|
|
},
|
|
},
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { isTemporary: true, expiredAt },
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'EVENT_ACTOR_NOT_READY' }),
|
|
);
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledTimes(1);
|
|
expect(mockAcquireEventChildGenerationLease).toHaveBeenCalledWith(
|
|
expect.objectContaining({ taskId: 'req-event', retentionExpiresAt: expiredAt }),
|
|
);
|
|
const eventJobOptions = mockGenerationJobManager.createJob.mock.calls[0][3];
|
|
expect(eventJobOptions.initialMetadata).toEqual(
|
|
expect.objectContaining({
|
|
responseMessageId: 'req-event:assistant',
|
|
userMessage: expect.objectContaining({ messageId: 'req-event:user' }),
|
|
agentEventDeliveryKey: 'req-event',
|
|
agentEventBindingId: 'binding-1',
|
|
retentionExpiresAt: expiredAt.toISOString(),
|
|
agentEventExpectedAction: {
|
|
toolName: 'submit_move',
|
|
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
|
|
},
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('routes an enabled authenticated event through the checkpoint-fork executor', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
});
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(async () => {
|
|
throw new Error('stop after event actor invocation started');
|
|
}),
|
|
};
|
|
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => {
|
|
await input.invoke({
|
|
checkpointNamespace: 'event-actor/fork',
|
|
checkpointId: 'checkpoint-base',
|
|
invocationId: 'req-event-fork',
|
|
continuation: 'warm',
|
|
signal: input.signal,
|
|
});
|
|
});
|
|
const event = {
|
|
id: 'game-1:ply-8',
|
|
type: 'chess.\u202Eturn',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'speed-chess', type: 'm\u2066cp' },
|
|
payload: { gameId: 'game-1', expectedPly: 8 },
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: '' },
|
|
body: {
|
|
text: 'Play the next move.',
|
|
clientRequestId: 'req-event-fork',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-fork',
|
|
target: { bindingId: 'binding-1' },
|
|
event,
|
|
expectedAction: { toolName: 'submit_\u200Fmove', argumentSubset: { expectedPly: 8 } },
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: {} },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: undefined,
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await nextTick();
|
|
|
|
expect(mockResolveAgentTurnExecutionPlan).toHaveBeenCalledTimes(1);
|
|
expect(mockResolveAgentTurnExecutionPlan).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
conversationId: 'child-conversation',
|
|
isNewConversation: false,
|
|
canPause: false,
|
|
}),
|
|
);
|
|
expect(mockExecuteAgentEventActor).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
conversationId: 'child-conversation',
|
|
invocationId: 'req-event-fork',
|
|
event,
|
|
expectedAction: { toolName: 'submit_\u200Fmove', argumentSubset: { expectedPly: 8 } },
|
|
}),
|
|
expect.objectContaining({
|
|
getSnapshot: expect.any(Function),
|
|
commitState: expect.any(Function),
|
|
recordReconciliation: expect.any(Function),
|
|
resolveReconciliation: expect.any(Function),
|
|
clearReconciliation: expect.any(Function),
|
|
admitAction: expect.any(Function),
|
|
releaseAction: expect.any(Function),
|
|
hasActionAdmission: expect.any(Function),
|
|
getReceipt: expect.any(Function),
|
|
}),
|
|
);
|
|
expect(mockExecuteAgentEventActor.mock.calls[0][0]).not.toHaveProperty('tenantId');
|
|
expect(req._agentEventTriggerProjection).toEqual({
|
|
version: 1,
|
|
eventType: 'chess. turn',
|
|
sourceType: 'm cp',
|
|
occurredAt: new Date(event.occurredAt),
|
|
expectedActionToolName: 'submit_ move',
|
|
});
|
|
expect(client).toMatchObject({
|
|
checkpointNamespace: 'event-actor/fork',
|
|
eventActorCheckpointId: 'checkpoint-base',
|
|
eventActorInvocationId: 'req-event-fork',
|
|
eventActorContinuation: 'warm',
|
|
});
|
|
});
|
|
|
|
it('prefers the execution-time action receipt over lagging run steps', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
});
|
|
mockFindAgentEventAppliedAction.mockReturnValue(undefined);
|
|
let observedRead;
|
|
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => {
|
|
await input.invoke({
|
|
checkpointNamespace: 'event-actor/fork',
|
|
checkpointId: 'checkpoint-base',
|
|
invocationId: 'req-event-receipt',
|
|
continuation: 'warm',
|
|
signal: input.signal,
|
|
});
|
|
observedRead = input.readAppliedAction();
|
|
throw new Error('stop after evidence read');
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: '' },
|
|
body: {
|
|
text: 'Play the next move.',
|
|
clientRequestId: 'req-event-receipt',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-receipt',
|
|
target: { bindingId: 'binding-1' },
|
|
event: {
|
|
id: 'game-1:ply-9',
|
|
type: 'chess.turn',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'speed-chess', type: 'mcp' },
|
|
},
|
|
expectedAction: { toolName: 'submit_move', argumentSubset: { expectedPly: 9 } },
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: {} },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: undefined,
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
/** Reproduces the observed race: the tool executed (graph context fires
|
|
* the observer during sendMessage) but the run-step collection is still
|
|
* empty when the executor reads evidence right after resolution. */
|
|
const client = {
|
|
options: {},
|
|
sendMessage: jest.fn(async () => {
|
|
req._agentEventActionObserver({
|
|
input: { expectedPly: 9 },
|
|
output: { name: 'submit_move', tool_call_id: 'call-9', content: '{"ok":true}' },
|
|
});
|
|
return {};
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await nextTick();
|
|
|
|
expect(mockExecuteAgentEventActor).toHaveBeenCalledTimes(1);
|
|
expect(mockCreateAgentEventActionRecorder).toHaveBeenCalledWith({
|
|
toolName: 'submit_move',
|
|
argumentSubset: { expectedPly: 9 },
|
|
});
|
|
expect(typeof req._agentEventActionObserver).toBe('function');
|
|
expect(observedRead).toEqual({ toolName: 'submit_move', toolCallId: 'call-9' });
|
|
expect(mockFindAgentEventAppliedAction).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('resumes the original actor suspension from exact detached terminal evidence', async () => {
|
|
mockGenerationJobManager.isRedis = true;
|
|
mockGenerationJobManager.detachedAgentEventActionStoreMode = 'distributed';
|
|
mockGenerationJobManager.createJob.mockResolvedValue({
|
|
createdAt: 2000,
|
|
metadata: {
|
|
checkpointNamespace: '2000',
|
|
providerExecutionId: 'provider-segment-2',
|
|
providerDrained: true,
|
|
},
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
const suspension = {
|
|
version: 1,
|
|
suspensionId: 'suspension-detached-1',
|
|
attempt: 0,
|
|
invocation: { invocationId: 'original-delivery-1' },
|
|
};
|
|
mockGetAgentEventActorSnapshot.mockResolvedValue({
|
|
state: null,
|
|
reconciliations: [],
|
|
suspension: {
|
|
kind: 'internal_completion',
|
|
suspension,
|
|
actionId: 'task-detached-1',
|
|
status: 'pending',
|
|
},
|
|
});
|
|
mockGetAgentEventActorDetachedAction.mockResolvedValue({
|
|
invocationId: 'original-delivery-1',
|
|
expectedToolName: 'submit_move',
|
|
toolName: 'submit_move_mcp_chess',
|
|
toolCallId: 'call-detached-1',
|
|
taskId: 'task-detached-1',
|
|
idempotencyKey: 'a'.repeat(64),
|
|
status: 'succeeded',
|
|
result: 'move accepted',
|
|
});
|
|
const repaused = { kind: 'internal_completion', actionId: 'task-detached-2' };
|
|
mockCreateAgentEventActorDetachedActionLifecycle.mockReturnValueOnce({
|
|
readSuspension: () => repaused,
|
|
});
|
|
mockResumeAgentEventActor.mockImplementationOnce(async (input) => {
|
|
expect(input.readSuspension()).toBe(repaused);
|
|
throw new Error('stop after resume contract');
|
|
});
|
|
const client = { options: { agent: {} }, sendMessage: jest.fn() };
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Trusted detached completion.',
|
|
clientRequestId: 'detached-resume-generation-1',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'detached-resume-generation-1',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'task-detached-1',
|
|
type: 'librechat.event_actor.detached_completion',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'librechat-event-actor', type: 'internal' },
|
|
},
|
|
internalCompletion: {
|
|
version: 1,
|
|
invocationId: 'original-delivery-1',
|
|
generationCreatedAt: 1000,
|
|
wakeGenerationCreatedAt: 2000,
|
|
taskId: 'task-detached-1',
|
|
idempotencyKey: 'a'.repeat(64),
|
|
},
|
|
},
|
|
},
|
|
config: { endpoints: { agents: {} } },
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await nextTick();
|
|
|
|
expect(mockExecuteAgentEventActor).not.toHaveBeenCalled();
|
|
expect(mockResumeAgentEventActor).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
conversationId: 'child-conversation',
|
|
bindingId: 'binding-1',
|
|
suspension,
|
|
resumeAttemptId: 'detached-resume-generation-1',
|
|
resumeValue: expect.objectContaining({
|
|
taskId: 'task-detached-1',
|
|
status: 'succeeded',
|
|
result: 'move accepted',
|
|
}),
|
|
}),
|
|
expect.objectContaining({
|
|
claimSuspension: expect.any(Function),
|
|
settleSuspension: expect.any(Function),
|
|
}),
|
|
);
|
|
expect(mockGetAgentEventActorDetachedAction).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
deliveryKey: 'original-delivery-1',
|
|
generationCreatedAt: 1000,
|
|
}),
|
|
);
|
|
expect(mockCreateAgentEventActorDetachedActionLifecycle).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
generationCreatedAt: 1000,
|
|
turnCreatedAt: 2000,
|
|
}),
|
|
expect.any(Object),
|
|
);
|
|
const lifecycleDependencies =
|
|
mockCreateAgentEventActorDetachedActionLifecycle.mock.calls.at(-1)[1];
|
|
expect(lifecycleDependencies.storeMode()).toBe('distributed');
|
|
mockGenerationJobManager.detachedAgentEventActionStoreMode = 'process_local';
|
|
expect(lifecycleDependencies.storeMode()).toBe('process_local');
|
|
expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith(
|
|
'child-conversation',
|
|
'user-123',
|
|
'child-conversation',
|
|
expect.objectContaining({
|
|
initialMetadata: expect.objectContaining({
|
|
agentEventDeliveryKey: 'detached-resume-generation-1',
|
|
agentEventInvocationKey: 'original-delivery-1',
|
|
agentEventInvocationGenerationCreatedAt: 1000,
|
|
agentEventDetachedActionProducerRequired: true,
|
|
}),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('records reconciliation when persistence fails after an event action commits', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
const checkpoint = {
|
|
threadId: 'child-conversation',
|
|
checkpointId: 'checkpoint-applied',
|
|
checkpointNs: 'event-actor/applied',
|
|
};
|
|
mockExecuteAgentEventActor.mockResolvedValueOnce({
|
|
value: {
|
|
messageId: 'req-event-persistence:assistant',
|
|
databasePromise: Promise.resolve().then(() => {
|
|
throw new Error('response persistence unavailable');
|
|
}),
|
|
},
|
|
execution: {
|
|
status: 'applied',
|
|
continuation: 'warm',
|
|
head: { actorThreadId: 'child-conversation', generation: 2, checkpoint },
|
|
result: { action: { toolName: 'submit_move', toolCallId: 'call-move' } },
|
|
},
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Play the next move.',
|
|
clientRequestId: 'req-event-persistence',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-persistence',
|
|
target: { bindingId: 'binding-1' },
|
|
event: {
|
|
id: 'event-persistence',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
expectedAction: { toolName: 'submit_move' },
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: {} },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client: { options: {} } }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockRecordAgentEventActorReconciliation.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
|
|
expect(mockExecuteAgentEventActor).toHaveBeenCalledTimes(1);
|
|
expect(mockLogger.info).toHaveBeenCalledWith(
|
|
'[event-actor] Bound child event completed',
|
|
expect.any(Object),
|
|
);
|
|
expect(mockRecordAgentEventActorReconciliation).toHaveBeenCalledWith({
|
|
user: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'child-conversation',
|
|
reconciliation: expect.objectContaining({
|
|
invocationId: 'req-event-persistence',
|
|
status: 'persistence_failed',
|
|
checkpoint,
|
|
action: { toolName: 'submit_move', toolCallId: 'call-move' },
|
|
}),
|
|
});
|
|
});
|
|
|
|
it('records the applied actor history barrier only after both deterministic messages persist', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
const checkpoint = {
|
|
threadId: 'child-conversation',
|
|
checkpointId: 'checkpoint-applied',
|
|
checkpointNs: 'event-actor/applied',
|
|
};
|
|
const userMessage = {
|
|
messageId: 'req-event-success:user',
|
|
parentMessageId: 'parent-message',
|
|
conversationId: 'child-conversation',
|
|
text: 'Play the next move.',
|
|
};
|
|
const client = {
|
|
options: {},
|
|
skipSaveUserMessage: false,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(userMessage, 'req-event-success:assistant');
|
|
return {
|
|
messageId: 'req-event-success:assistant',
|
|
text: 'Move submitted.',
|
|
databasePromise: Promise.resolve({
|
|
conversation: { conversationId: 'child-conversation' },
|
|
}),
|
|
};
|
|
}),
|
|
};
|
|
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => ({
|
|
value: await input.invoke({
|
|
checkpointNamespace: checkpoint.checkpointNs,
|
|
checkpointId: checkpoint.checkpointId,
|
|
invocationId: 'req-event-success',
|
|
continuation: 'warm',
|
|
signal: input.signal,
|
|
}),
|
|
execution: {
|
|
status: 'applied',
|
|
continuation: 'warm',
|
|
head: { actorThreadId: 'child-conversation', generation: 2, checkpoint },
|
|
result: { action: { toolName: 'submit_move', toolCallId: 'call-move' } },
|
|
},
|
|
}));
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: userMessage.text,
|
|
clientRequestId: 'req-event-success',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-success',
|
|
target: { bindingId: 'binding-1' },
|
|
event: {
|
|
id: 'event-success',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
expectedAction: { toolName: 'submit_move' },
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: {} },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockRecordAgentEventActorReconciliation.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.any(Object),
|
|
expect.objectContaining({ messageId: 'req-event-success:user' }),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockSaveMessage).toHaveBeenCalledWith(
|
|
expect.any(Object),
|
|
expect.objectContaining({ messageId: 'req-event-success:assistant' }),
|
|
expect.any(Object),
|
|
);
|
|
expect(mockRecordAgentEventActorReconciliation).toHaveBeenCalledWith({
|
|
user: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'child-conversation',
|
|
reconciliation: {
|
|
invocationId: 'req-event-success',
|
|
status: 'history_persisted',
|
|
checkpoint,
|
|
action: { toolName: 'submit_move', toolCallId: 'call-move' },
|
|
actionAdmitted: true,
|
|
observedAt: expect.any(Date),
|
|
},
|
|
});
|
|
const lastMessageWrite = Math.max(...mockSaveMessage.mock.invocationCallOrder);
|
|
expect(lastMessageWrite).toBeLessThan(
|
|
mockRecordAgentEventActorReconciliation.mock.invocationCallOrder[0],
|
|
);
|
|
});
|
|
|
|
it.each([
|
|
[
|
|
'tool approval',
|
|
{ toolDefinitions: [] },
|
|
{
|
|
toolApproval: { enabled: true },
|
|
},
|
|
undefined,
|
|
undefined,
|
|
],
|
|
[
|
|
'primary ask_user_question',
|
|
{ toolDefinitions: [{ name: 'ask_user_question' }] },
|
|
{},
|
|
undefined,
|
|
undefined,
|
|
],
|
|
[
|
|
'added-agent ask_user_question',
|
|
{ toolDefinitions: [] },
|
|
{},
|
|
new Map([['added-agent', { toolDefinitions: [{ name: 'ask_user_question' }] }]]),
|
|
undefined,
|
|
],
|
|
[
|
|
'memory-checkpointer',
|
|
{ toolDefinitions: [] },
|
|
{
|
|
checkpointer: { type: 'memory' },
|
|
},
|
|
undefined,
|
|
undefined,
|
|
],
|
|
[
|
|
'background-capable expected action',
|
|
{ toolDefinitions: [], backgroundToolNames: ['submit_move_mcp_chess'] },
|
|
{},
|
|
undefined,
|
|
undefined,
|
|
],
|
|
[
|
|
'pre-capability pause consumer fleet',
|
|
{ toolDefinitions: [] },
|
|
{ toolApproval: { enabled: true } },
|
|
undefined,
|
|
undefined,
|
|
1,
|
|
],
|
|
])(
|
|
'routes %s event actors through the compatible continuation path',
|
|
async (_label, agent, config, agentConfigs, clientOptions, generationProtocolVersion = 2) => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({
|
|
streamId: 'child-conversation',
|
|
conversationId: 'child-conversation',
|
|
generationProtocolVersion,
|
|
}),
|
|
);
|
|
mockGenerationJobManager.createJob.mockResolvedValueOnce({
|
|
createdAt: 1000,
|
|
metadata: {
|
|
checkpointNamespace: '1000',
|
|
providerExecutionId: 'provider-segment-1',
|
|
providerDrained: true,
|
|
generationProtocolVersion,
|
|
},
|
|
readyPromise: Promise.resolve(),
|
|
abortController: new AbortController(),
|
|
emitter: { on: jest.fn() },
|
|
});
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
const client = {
|
|
options: { agent, ...(clientOptions ?? {}) },
|
|
agentConfigs,
|
|
sendMessage: jest.fn(async () => {
|
|
throw new Error('stop after legacy event invocation started');
|
|
}),
|
|
};
|
|
const shouldCheckpoint =
|
|
_label !== 'memory-checkpointer' && _label !== 'pre-capability pause consumer fleet';
|
|
if (shouldCheckpoint) {
|
|
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => {
|
|
await input.invoke({
|
|
checkpointNamespace: 'event-actor/pause-capable',
|
|
checkpointId: 'checkpoint-pause-capable',
|
|
invocationId: 'req-event-hitl',
|
|
continuation: 'warm',
|
|
signal: input.signal,
|
|
});
|
|
});
|
|
}
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Continue with a pause-capable actor.',
|
|
clientRequestId: 'req-event-hitl',
|
|
conversationId: 'child-conversation',
|
|
generationProtocolVersion,
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-hitl',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'event-hitl',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
},
|
|
},
|
|
config: { endpoints: { agents: config } },
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
await nextTick();
|
|
|
|
if (shouldCheckpoint) {
|
|
expect(mockExecuteAgentEventActor).toHaveBeenCalled();
|
|
expect(mockGetMessages).not.toHaveBeenCalled();
|
|
expect(mockBeginAgentEventActorLegacyTurn).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.updateMetadata).not.toHaveBeenCalledWith(
|
|
'child-conversation',
|
|
expect.objectContaining({ agentEventLegacyTurnToken: expect.any(String) }),
|
|
1000,
|
|
);
|
|
expect(client.sendMessage).toHaveBeenCalledTimes(1);
|
|
return;
|
|
}
|
|
expect(mockExecuteAgentEventActor).not.toHaveBeenCalled();
|
|
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
|
user: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'child-conversation',
|
|
token: expect.any(String),
|
|
});
|
|
expect(client.sendMessage).toHaveBeenCalledTimes(1);
|
|
/** The fence must open before execution and close only after this
|
|
* turn's history is durable, under the same token. */
|
|
const fenceToken = mockBeginAgentEventActorLegacyTurn.mock.calls[0][0].token;
|
|
expect(mockBeginAgentEventActorLegacyTurn.mock.invocationCallOrder[0]).toBeLessThan(
|
|
client.sendMessage.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockGenerationJobManager.updateMetadata).toHaveBeenCalledWith(
|
|
'child-conversation',
|
|
{ agentEventLegacyTurnToken: fenceToken },
|
|
1000,
|
|
);
|
|
expect(mockGenerationJobManager.updateMetadata.mock.invocationCallOrder[0]).toBeLessThan(
|
|
client.sendMessage.mock.invocationCallOrder[0],
|
|
);
|
|
expect(mockCompleteAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
|
user: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'child-conversation',
|
|
token: fenceToken,
|
|
});
|
|
expect(mockCompleteAgentEventActorLegacyTurn.mock.invocationCallOrder[0]).toBeGreaterThan(
|
|
client.sendMessage.mock.invocationCallOrder[0],
|
|
);
|
|
},
|
|
);
|
|
|
|
it('retains local fence ownership when Redis metadata persistence fails', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
mockGenerationJobManager.updateMetadata.mockRejectedValueOnce(
|
|
new Error('redis metadata unavailable'),
|
|
);
|
|
const client = {
|
|
options: {
|
|
agent: {
|
|
toolDefinitions: [],
|
|
backgroundToolNames: ['submit_move'],
|
|
},
|
|
},
|
|
sendMessage: jest.fn(),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Persist ownership before Redis metadata.',
|
|
clientRequestId: 'req-event-metadata-failure',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-metadata-failure',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'event-metadata-failure',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: { checkpointer: { type: 'memory' } } },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockCompleteAgentEventActorLegacyTurn.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
|
|
const fenceToken = mockBeginAgentEventActorLegacyTurn.mock.calls[0][0].token;
|
|
expect(client.sendMessage).not.toHaveBeenCalled();
|
|
expect(mockCompleteAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
|
user: 'user-123',
|
|
tenantId: 'tenant-1',
|
|
conversationId: 'child-conversation',
|
|
token: fenceToken,
|
|
});
|
|
});
|
|
|
|
it('does not execute a legacy event turn when its durable fence is not acquired', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
mockBeginAgentEventActorLegacyTurn.mockResolvedValueOnce(false);
|
|
const client = {
|
|
options: {
|
|
agent: {
|
|
toolDefinitions: [],
|
|
backgroundToolNames: ['submit_move'],
|
|
},
|
|
},
|
|
sendMessage: jest.fn(async () => ({ messageId: 'must-not-run' })),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Do not run without the fence.',
|
|
clientRequestId: 'req-event-fence-lost',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-fence-lost',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'event-fence-lost',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: { checkpointer: { type: 'memory' } } },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockBeginAgentEventActorLegacyTurn.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
|
|
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
|
expect(client.sendMessage).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('keeps the legacy event fence open when failed-turn persistence cannot complete', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
mockGenerationJobManager.completeJob.mockRejectedValueOnce(new Error('job store unavailable'));
|
|
const client = {
|
|
options: {
|
|
agent: {
|
|
toolDefinitions: [],
|
|
backgroundToolNames: ['submit_move'],
|
|
},
|
|
},
|
|
sendMessage: jest.fn(async () => {
|
|
throw new Error('provider failed');
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Persist the failed turn before releasing the fence.',
|
|
clientRequestId: 'req-event-error-persistence',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-error-persistence',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'event-error-persistence',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: { checkpointer: { type: 'memory' } } },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockGenerationJobManager.completeJob.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
await nextTick();
|
|
|
|
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledTimes(1);
|
|
expect(mockCompleteAgentEventActorLegacyTurn).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('does not replay a stale ambiguous legacy fence based only on age', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
tenantId: 'tenant-1',
|
|
});
|
|
mockBeginAgentEventActorLegacyTurn.mockResolvedValueOnce(false);
|
|
mockGetAgentEventActorSnapshot.mockResolvedValueOnce({
|
|
state: null,
|
|
reconciliations: [],
|
|
epoch: 0,
|
|
legacyTurn: {
|
|
token: 'crashed-legacy-turn',
|
|
startedAt: new Date(Date.now() - 2 * 60 * 60 * 1000),
|
|
},
|
|
});
|
|
const client = {
|
|
options: {
|
|
agent: {
|
|
toolDefinitions: [],
|
|
backgroundToolNames: ['submit_move'],
|
|
},
|
|
},
|
|
sendMessage: jest.fn(async () => {
|
|
throw new Error('stop after recovered execution starts');
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'Recover the abandoned legacy fence.',
|
|
clientRequestId: 'req-event-stale-legacy',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
agentEventDelivery: {
|
|
deliveryKey: 'req-event-stale-legacy',
|
|
target: { bindingId: 'binding-1' },
|
|
expectedAction: { toolName: 'submit_move' },
|
|
event: {
|
|
id: 'event-stale-legacy',
|
|
type: 'test.event',
|
|
occurredAt: Date.now(),
|
|
source: { id: 'test', type: 'api' },
|
|
payload: {},
|
|
},
|
|
},
|
|
},
|
|
config: {
|
|
endpoints: { agents: { checkpointer: { type: 'memory' } } },
|
|
},
|
|
_isAgentTrigger: true,
|
|
_agentEventBindingId: 'binding-1',
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
null,
|
|
);
|
|
for (
|
|
let attempt = 0;
|
|
attempt < 20 && mockBeginAgentEventActorLegacyTurn.mock.calls.length === 0;
|
|
attempt++
|
|
) {
|
|
await nextTick();
|
|
}
|
|
|
|
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
|
expect(client.sendMessage).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('uses the guard-normalized tenant for a legacy untenanted event actor', async () => {
|
|
const expiredAt = new Date(Date.now() + 60_000);
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({
|
|
streamId: 'child-conversation',
|
|
conversationId: 'child-conversation',
|
|
}),
|
|
);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'parent-conversation',
|
|
agent_id: 'parent-agent',
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: '' },
|
|
body: {
|
|
text: 'Continue from an old untenanted binding.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-event-legacy',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: undefined,
|
|
_agentEventBindingRetention: { isTemporary: true, expiredAt },
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(mockAcquireEventChildGenerationLease).toHaveBeenCalledWith(
|
|
expect.objectContaining({ tenantId: undefined, retentionExpiresAt: expiredAt }),
|
|
);
|
|
});
|
|
|
|
it('does not start an event actor whose inherited binding expired after the guard', async () => {
|
|
const expiredAt = new Date(Date.now() - 1);
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
|
wonGenerationClaim({
|
|
streamId: 'child-conversation',
|
|
conversationId: 'child-conversation',
|
|
}),
|
|
);
|
|
mockAcquireEventChildGenerationLease.mockResolvedValue(null);
|
|
const req = {
|
|
user: { id: 'user-123', tenantId: 'tenant-1' },
|
|
body: {
|
|
text: 'This event arrived too late.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-event-expired',
|
|
conversationId: 'child-conversation',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
_agentEventBindingParentConversationId: 'parent-conversation',
|
|
_agentEventBindingParentAgentId: 'parent-agent',
|
|
_agentEventBindingTenantId: 'tenant-1',
|
|
_agentEventBindingRetention: { isTemporary: true, expiredAt },
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(409);
|
|
expect(res.json).toHaveBeenCalledWith(
|
|
expect.objectContaining({ code: 'EVENT_BINDING_PARENT_ENDED' }),
|
|
);
|
|
expect(mockIsSubagentOwnerAdmissible).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('releases the idempotency claim on a 429 only when it won the claim', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockResolvedValue(wonGenerationClaim());
|
|
mockCheckAndIncrementPendingRequest.mockResolvedValue({
|
|
allowed: false,
|
|
pendingRequests: 3,
|
|
limit: 2,
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Over the limit.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(429);
|
|
expect(mockGenerationJobManager.releaseGeneration).toHaveBeenCalledWith(
|
|
'user-123',
|
|
'req-abc',
|
|
'conversation-123',
|
|
DEFAULT_OWNED_CLAIM,
|
|
);
|
|
expect(mockStartupTelemetry.end).toHaveBeenCalledWith('rejected');
|
|
});
|
|
|
|
it('fails closed before the limiter when generation ownership is ambiguous', async () => {
|
|
mockGenerationJobManager.claimGeneration.mockRejectedValue(new Error('redis down'));
|
|
mockCheckAndIncrementPendingRequest.mockResolvedValue({
|
|
allowed: false,
|
|
pendingRequests: 3,
|
|
limit: 2,
|
|
});
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'Duplicate while the original runs.',
|
|
messageId: 'user-msg',
|
|
clientRequestId: 'req-abc',
|
|
conversationId: 'conversation-123',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
const res = { json: jest.fn(), status: jest.fn(() => res), set: jest.fn() };
|
|
|
|
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
|
|
|
expect(res.status).toHaveBeenCalledWith(503);
|
|
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'SERVER_NOT_READY' }));
|
|
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
|
expect(mockGenerationJobManager.releaseGeneration).not.toHaveBeenCalled();
|
|
});
|
|
|
|
describe('immediate title persistence gate', () => {
|
|
const { Constants } = require('librechat-data-provider');
|
|
|
|
/**
|
|
* Starts a first turn that stays mid-run until the returned `release` is
|
|
* called, so the gate an immediate title waits on can be observed while the
|
|
* generation is still going. `userMessageWrite` is what the client hands back
|
|
* through `getReqData`, exactly as BaseClient does once it has started the
|
|
* user-message write.
|
|
*/
|
|
const startHeldFirstTurn = async ({ userMessageWrite, clientOverrides, responseWrite }) => {
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async () => signalFinished());
|
|
|
|
let release;
|
|
const held = new Promise((resolve) => {
|
|
release = resolve;
|
|
});
|
|
|
|
let convoReadyResolved = false;
|
|
const addTitle = jest.fn(async (_req, options) => {
|
|
void options?.convoReady?.then(() => {
|
|
convoReadyResolved = true;
|
|
});
|
|
});
|
|
|
|
let turnConversationId;
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
...clientOverrides,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
turnConversationId = options.conversationId;
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: options.conversationId,
|
|
text: 'First message',
|
|
};
|
|
options.onStart(userMessage, 'response-msg');
|
|
options.getReqData({ userMessagePromise: userMessageWrite(options.conversationId) });
|
|
await held;
|
|
return {
|
|
messageId: 'response-msg',
|
|
parentMessageId: 'user-msg',
|
|
conversationId: options.conversationId,
|
|
content: [{ type: 'text', text: 'Answer' }],
|
|
databasePromise: Promise.resolve(
|
|
responseWrite
|
|
? responseWrite(options.conversationId)
|
|
: {
|
|
message: { _id: 'response-row-id' },
|
|
conversation: { conversationId: options.conversationId, title: null },
|
|
},
|
|
),
|
|
};
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
{
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'First message',
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: 'new',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
},
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
addTitle,
|
|
);
|
|
await nextTick();
|
|
await nextTick();
|
|
|
|
return {
|
|
addTitle,
|
|
conversationId: () => turnConversationId,
|
|
isConvoReadyResolved: () => convoReadyResolved,
|
|
finish: async () => {
|
|
release();
|
|
await finished;
|
|
await nextTick();
|
|
await nextTick();
|
|
},
|
|
};
|
|
};
|
|
|
|
/** The bug this covers: gating the title's save on the end of the turn left
|
|
* the row on "New Chat" for the whole run, so every reader without the live
|
|
* stream — a reloaded tab, the sidebar on another device — read the
|
|
* placeholder until the turn finished. */
|
|
it('lets an immediate title persist as soon as the user message write creates the row', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: (conversationId) =>
|
|
Promise.resolve({ message: { messageId: 'user-msg' }, conversation: { conversationId } }),
|
|
});
|
|
|
|
expect(turn.addTitle).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.finishTerminalJob).not.toHaveBeenCalled();
|
|
expect(turn.isConvoReadyResolved()).toBe(true);
|
|
|
|
await turn.finish();
|
|
});
|
|
|
|
it('keeps the title waiting until the turn ends when no conversation was persisted', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: () => Promise.resolve({}),
|
|
});
|
|
|
|
expect(turn.addTitle).toHaveBeenCalledTimes(1);
|
|
expect(turn.isConvoReadyResolved()).toBe(false);
|
|
|
|
await turn.finish();
|
|
|
|
expect(turn.isConvoReadyResolved()).toBe(true);
|
|
});
|
|
|
|
/** The reference repair the title's save used to perform as a side effect of
|
|
* rebuilding the whole `messages` array. The title now writes metadata only, so
|
|
* the recovered user row has to carry its own reference. */
|
|
describe('recovered user message reference', () => {
|
|
/** Distinct rows, so one reference says nothing about the other. */
|
|
beforeEach(() => {
|
|
mockSaveMessage.mockImplementation(async (_ctx, message) => ({
|
|
_id: message.messageId === 'response-msg' ? 'response-row-id' : 'user-row-id',
|
|
}));
|
|
});
|
|
|
|
const appendedIdsFor = (conversationId) =>
|
|
mockAppendConvoMessageReference.mock.calls
|
|
.filter((call) => call[1] === conversationId)
|
|
.map((call) => call[2]);
|
|
|
|
it('appends the recovered reference when no write reported the conversation', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
/** BaseClient swallows a failed user-message save and resolves with `{}`. */
|
|
userMessageWrite: () => Promise.resolve({}),
|
|
});
|
|
|
|
await turn.finish();
|
|
|
|
expect(appendedIdsFor(turn.conversationId())).toEqual(['user-row-id']);
|
|
expect(mockAppendConvoMessageReference).toHaveBeenCalledWith(
|
|
'user-123',
|
|
turn.conversationId(),
|
|
'user-row-id',
|
|
);
|
|
});
|
|
|
|
it('does not append again when the write already recorded the reference', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: (conversationId) =>
|
|
Promise.resolve({
|
|
message: { _id: 'user-row-id' },
|
|
conversation: { conversationId },
|
|
}),
|
|
});
|
|
|
|
await turn.finish();
|
|
|
|
expect(appendedIdsFor(turn.conversationId())).toEqual([]);
|
|
});
|
|
|
|
/** `saveMessage` can resolve falsy without throwing, and the conversation is still
|
|
* written and reported with nothing appended. The row existing is not evidence
|
|
* that this turn is referenced by it. */
|
|
it('appends when the write reported the row but appended no message', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: (conversationId) =>
|
|
Promise.resolve({ message: undefined, conversation: { conversationId } }),
|
|
});
|
|
|
|
await turn.finish();
|
|
|
|
expect(appendedIdsFor(turn.conversationId())).toEqual(['user-row-id']);
|
|
});
|
|
|
|
/** The response row has the same exposure: its write can report the conversation
|
|
* while appending nothing, and its terminal retry is also a bare `saveMessage`. */
|
|
it('appends the recovered response reference when its write appended nothing', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: (conversationId) =>
|
|
Promise.resolve({
|
|
message: { _id: 'user-row-id' },
|
|
conversation: { conversationId },
|
|
}),
|
|
/** The response write reports the row without appending its id. */
|
|
responseWrite: (conversationId) => ({
|
|
conversation: { conversationId, title: null },
|
|
}),
|
|
});
|
|
|
|
await turn.finish();
|
|
|
|
expect(appendedIdsFor(turn.conversationId())).toEqual(['response-row-id']);
|
|
});
|
|
|
|
/** A first turn that pauses for approval never reaches the terminal, so its rows
|
|
* cannot wait for one. The paused response is saved bare, and the title write that
|
|
* used to rebuild the array in passing no longer does. */
|
|
describe('a first turn paused for approval', () => {
|
|
const runPausedFirstTurn = async ({ clientOverrides } = {}) => {
|
|
mockGenerationJobManager.approvals.ownsPausePersistence.mockResolvedValue(true);
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
pendingApproval: { actionId: 'action-paused-reference' },
|
|
...clientOverrides,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
options.onStart(
|
|
{
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: options.conversationId,
|
|
text: 'First message',
|
|
},
|
|
'response-msg',
|
|
);
|
|
options.getReqData({
|
|
userMessagePromise: Promise.resolve({
|
|
message: { _id: 'user-row-id' },
|
|
conversation: { conversationId: options.conversationId },
|
|
}),
|
|
});
|
|
return { messageId: 'response-msg', conversationId: options.conversationId };
|
|
}),
|
|
};
|
|
|
|
await AgentController(
|
|
{
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'First message',
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: 'new',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
},
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
jest.fn(async () => {}),
|
|
);
|
|
await nextTick();
|
|
await nextTick();
|
|
};
|
|
|
|
it('appends the paused response reference', async () => {
|
|
await runPausedFirstTurn();
|
|
|
|
expect(mockAppendConvoMessageReference).toHaveBeenCalledTimes(1);
|
|
expect(mockAppendConvoMessageReference).toHaveBeenCalledWith(
|
|
'user-123',
|
|
expect.any(String),
|
|
'response-row-id',
|
|
);
|
|
});
|
|
|
|
/** The BaseClient re-save reports its own conversation write, which the signal
|
|
* never saw through `getReqData`, so it must not be appended twice. */
|
|
it('does not append the user reference its re-save already recorded', async () => {
|
|
await runPausedFirstTurn({
|
|
clientOverrides: {
|
|
getSaveOptions: () => ({}),
|
|
saveMessageToDatabase: jest.fn().mockResolvedValue({
|
|
message: { _id: 'user-row-id' },
|
|
conversation: { conversationId: 'convo-paused' },
|
|
}),
|
|
},
|
|
});
|
|
|
|
const appended = mockAppendConvoMessageReference.mock.calls.map((call) => call[2]);
|
|
expect(appended).toEqual(['response-row-id']);
|
|
});
|
|
});
|
|
|
|
it('leaves the conversation alone for a turn that does not save one', async () => {
|
|
const turn = await startHeldFirstTurn({
|
|
userMessageWrite: () => Promise.resolve({ message: {} }),
|
|
clientOverrides: { skipSaveConvo: true },
|
|
});
|
|
|
|
await turn.finish();
|
|
|
|
expect(appendedIdsFor(turn.conversationId())).toEqual([]);
|
|
});
|
|
});
|
|
});
|
|
|
|
describe('preempt-incomplete title gating', () => {
|
|
const { Constants } = require('librechat-data-provider');
|
|
const { resolveTitleTiming } = require('@librechat/api');
|
|
|
|
/** An empty preempt boundary ends the turn truncated — the response is
|
|
* persisted `unfinished`, so it must follow the abort title contract. */
|
|
const preemptIncompleteRun = {
|
|
getPreemptStats: () => ({ emptyBoundaries: 1 }),
|
|
getHaltReason: () => 'preempt_incomplete',
|
|
};
|
|
|
|
const runFirstTurn = async ({
|
|
run,
|
|
addTitle: suppliedAddTitle,
|
|
clientOverrides,
|
|
databaseResult,
|
|
responseContent = [{ type: 'text', text: 'Truncated answer' }],
|
|
} = {}) => {
|
|
let signalFinished;
|
|
const finished = new Promise((resolve) => {
|
|
signalFinished = resolve;
|
|
});
|
|
mockGenerationJobManager.finishTerminalJob.mockImplementation(async () => signalFinished());
|
|
|
|
let titleSignal;
|
|
const addTitle =
|
|
suppliedAddTitle ??
|
|
jest.fn(async (_req, options) => {
|
|
titleSignal = options?.signal;
|
|
});
|
|
|
|
const client = {
|
|
options: {},
|
|
savedMessageIds: new Set(),
|
|
skipSaveUserMessage: false,
|
|
...(run && { run }),
|
|
...clientOverrides,
|
|
sendMessage: jest.fn(async (_text, options) => {
|
|
const userMessage = {
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: options.conversationId,
|
|
text: 'First message',
|
|
};
|
|
options.onStart(userMessage, 'response-msg');
|
|
return {
|
|
messageId: 'response-msg',
|
|
parentMessageId: 'user-msg',
|
|
conversationId: options.conversationId,
|
|
content: responseContent,
|
|
databasePromise: Promise.resolve(
|
|
databaseResult ?? {
|
|
conversation: { conversationId: options.conversationId, title: null },
|
|
},
|
|
),
|
|
};
|
|
}),
|
|
};
|
|
const req = {
|
|
user: { id: 'user-123' },
|
|
body: {
|
|
text: 'First message',
|
|
messageId: 'user-msg',
|
|
parentMessageId: Constants.NO_PARENT,
|
|
conversationId: 'new',
|
|
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
|
},
|
|
config: {},
|
|
};
|
|
|
|
await AgentController(
|
|
req,
|
|
createResumableResponse(),
|
|
jest.fn(),
|
|
jest.fn().mockResolvedValue({ client }),
|
|
addTitle,
|
|
);
|
|
await finished;
|
|
await nextTick();
|
|
await nextTick();
|
|
|
|
return { addTitle, getTitleSignal: () => titleSignal };
|
|
};
|
|
|
|
it('skips deferred title generation when an empty preempt boundary truncates the first turn', async () => {
|
|
resolveTitleTiming.mockReturnValueOnce('final');
|
|
|
|
const { addTitle } = await runFirstTurn({ run: preemptIncompleteRun });
|
|
|
|
expect(addTitle).not.toHaveBeenCalled();
|
|
});
|
|
it('does not re-stamp an unfinished response already persisted by BaseClient', async () => {
|
|
const stampedAt = new Date('2026-09-07T12:00:00.000Z');
|
|
|
|
await runFirstTurn({
|
|
run: preemptIncompleteRun,
|
|
databaseResult: {
|
|
conversation: {
|
|
conversationId: 'new',
|
|
title: null,
|
|
lastResponseAt: stampedAt,
|
|
},
|
|
},
|
|
});
|
|
|
|
expect(mockStampConvoLastResponse).not.toHaveBeenCalled();
|
|
const finalEvent = mockGenerationJobManager.publishTerminalClaim.mock.calls.at(-1)[1];
|
|
expect(finalEvent.conversation.lastResponseAt).toEqual(stampedAt);
|
|
});
|
|
|
|
it('stamps and acknowledges the fallback conversation when BaseClient persistence is skipped', async () => {
|
|
const stampedAt = new Date('2026-09-07T12:00:00.000Z');
|
|
mockSaveMessage.mockImplementation(async (_ctx, message) => message);
|
|
mockGetConvo.mockResolvedValue({
|
|
conversationId: 'new',
|
|
title: null,
|
|
createdAt: '2026-06-07T00:00:00.000Z',
|
|
lastResponseAt: stampedAt,
|
|
});
|
|
|
|
await runFirstTurn({
|
|
run: preemptIncompleteRun,
|
|
databaseResult: {
|
|
persistenceSkipped: true,
|
|
conversation: {
|
|
conversationId: 'new',
|
|
title: null,
|
|
lastResponseAt: new Date('2026-09-07T11:59:00.000Z'),
|
|
},
|
|
},
|
|
});
|
|
|
|
expect(mockStampConvoLastResponse).toHaveBeenCalledTimes(1);
|
|
const finalEvent = mockGenerationJobManager.publishTerminalClaim.mock.calls.at(-1)[1];
|
|
expect(finalEvent.conversation.lastResponseAt).toEqual(stampedAt);
|
|
});
|
|
|
|
/* A preempted turn can persist its row before the model produced anything readable. A dot
|
|
raised for that row could never be acknowledged, because acknowledgement requires the
|
|
stamped reply to be on screen. */
|
|
it('does not stamp a skipped-persistence turn that persisted nothing to read', async () => {
|
|
mockSaveMessage.mockImplementation(async (_ctx, message) => message);
|
|
|
|
await runFirstTurn({
|
|
run: preemptIncompleteRun,
|
|
responseContent: [],
|
|
databaseResult: {
|
|
persistenceSkipped: true,
|
|
conversation: { conversationId: 'new', title: null },
|
|
},
|
|
});
|
|
|
|
expect(mockStampConvoLastResponse).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('still generates a deferred title for a completed first turn', async () => {
|
|
resolveTitleTiming.mockReturnValueOnce('final');
|
|
|
|
const { addTitle } = await runFirstTurn();
|
|
|
|
expect(addTitle).toHaveBeenCalledTimes(1);
|
|
expect(addTitle).toHaveBeenCalledWith(
|
|
expect.anything(),
|
|
expect.objectContaining({ response: expect.anything() }),
|
|
);
|
|
});
|
|
|
|
it('does not acknowledge provider drain until deferred title persistence settles', async () => {
|
|
resolveTitleTiming.mockReturnValueOnce('final');
|
|
let resolveTitle;
|
|
const titlePending = new Promise((resolve) => {
|
|
resolveTitle = resolve;
|
|
});
|
|
const addTitle = jest.fn(() => titlePending);
|
|
|
|
await runFirstTurn({ addTitle });
|
|
|
|
expect(addTitle).toHaveBeenCalledTimes(1);
|
|
expect(mockGenerationJobManager.markProviderExecutionDrained).not.toHaveBeenCalled();
|
|
|
|
resolveTitle();
|
|
await nextTick();
|
|
await nextTick();
|
|
|
|
expect(mockGenerationJobManager.markProviderExecutionDrained).toHaveBeenCalledWith(
|
|
expect.any(String),
|
|
1000,
|
|
'provider-segment-1',
|
|
);
|
|
});
|
|
|
|
it('cancels an in-flight immediate title when the turn ends preempt-incomplete', async () => {
|
|
const { addTitle, getTitleSignal } = await runFirstTurn({ run: preemptIncompleteRun });
|
|
|
|
expect(addTitle).toHaveBeenCalledTimes(1);
|
|
expect(getTitleSignal().aborted).toBe(true);
|
|
});
|
|
|
|
it('lets an immediate title proceed for a completed first turn', async () => {
|
|
const { addTitle, getTitleSignal } = await runFirstTurn();
|
|
|
|
expect(addTitle).toHaveBeenCalledTimes(1);
|
|
expect(getTitleSignal().aborted).toBe(false);
|
|
});
|
|
|
|
/**
|
|
* The hop that makes the client notice possible: `AgentClient` swallows the
|
|
* graph's step-limit error and raises `stepLimitReached`, and the controller
|
|
* must turn that into an `unfinished` row stamped with the tool-call-limit
|
|
* finish reason on BOTH the durable write and the terminal SSE event.
|
|
*/
|
|
const stepLimitClient = { stepLimitReached: true };
|
|
|
|
const savedResponseRow = () =>
|
|
mockSaveMessage.mock.calls
|
|
.map(([, message]) => message)
|
|
.find((message) => message?.messageId === 'response-msg');
|
|
|
|
it('persists a step-limited turn as unfinished with the tool-call-limit finish reason', async () => {
|
|
await runFirstTurn({ clientOverrides: stepLimitClient });
|
|
|
|
expect(savedResponseRow()).toEqual(
|
|
expect.objectContaining({
|
|
unfinished: true,
|
|
finish_reason: Constants.TOOL_CALL_LIMIT_FINISH_REASON,
|
|
}),
|
|
);
|
|
});
|
|
|
|
it('keeps the partial content on a step-limited turn instead of replacing it with an error', async () => {
|
|
await runFirstTurn({ clientOverrides: stepLimitClient });
|
|
|
|
const saved = savedResponseRow();
|
|
expect(saved.content).toEqual([{ type: 'text', text: 'Truncated answer' }]);
|
|
expect(saved.error).not.toBe(true);
|
|
});
|
|
|
|
it('publishes the finish reason on the terminal event so the client needs no refetch', async () => {
|
|
await runFirstTurn({ clientOverrides: stepLimitClient });
|
|
|
|
const published = mockGenerationJobManager.publishTerminalClaim.mock.calls.at(-1);
|
|
expect(published).toBeDefined();
|
|
expect(JSON.stringify(published)).toContain(Constants.TOOL_CALL_LIMIT_FINISH_REASON);
|
|
});
|
|
|
|
it('leaves an ordinary completed turn finished and unstamped', async () => {
|
|
await runFirstTurn();
|
|
|
|
const saved = savedResponseRow();
|
|
expect(saved.unfinished).toBe(false);
|
|
expect(saved.finish_reason).toBeUndefined();
|
|
});
|
|
});
|
|
});
|