mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-09-05 05:59:14 +00:00
3f02efdef9
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🛑 feat: Preemptive Steer — server half (Interrupt & Steer, PR 2 of 3)
Lets the steer route ask the generating replica to seal its live model
stream at the next provider-safe boundary instead of waiting for a tool
step. The run is never aborted, job status never changes, the partial
answer is kept, and generation resumes in the same assistant message
after the injected steer. Consumes the SDK seam in @librechat/agents
(danny-avila/agents#335, #346).
Transport: IEventTransport gains a fenced emitPreempt/onPreempt pair
beside abort. RedisEventTransport fans PREEMPT out on the SAME events
channel and subscription (no new connection, key, or subscribe call);
onPreempt returns a registration-scoped unsubscribe with the same
replacement-safe state-identity guard onAbort uses. InMemory implements
neither — single-process preempt lives entirely in the runtime set.
Runtime state: RuntimeJobState carries the per-generation request set,
createdAt-fenced and capped at STEER_QUEUE_MAX_DEPTH, plus a bounded
`cleared` tombstone so a late cross-replica arm cannot resurrect a
request whose steer already drained. registerPreemptSubscription
mirrors the abort registration's double fence (runtime identity +
generation createdAt); releaseAbortSubscription retires BOTH listeners
and the armed set, so every terminal path drops preempt state for free.
Public surface: requestPreempt (arm + fenced publish, never a rejection
surface, never touches job status), isPreemptRequested (O(1)
level-triggered poll), noteSteersRemoved (drain/cancel bookkeeping +
fenced clear), clearPreemptRequests (empty-boundary disarm).
One drain body, two boundaries: createSteerDrainHook (PostToolBatch)
and createSteerPreemptBoundaryHook (PreemptBoundary) share
drainAndBuildInjections, so the two injection sites cannot drift — the
SDK's provider-safety argument rests on identical HumanMessage shapes.
The shared body builds injections incrementally under a swallow-all
catch (a mid-loop throw still injects what was applied — those parts
are already persisted), clears preempt requests in finally, and
disarms the generation when a boundary drains nothing.
Request path: POST /chat/steer accepts preempt: true. The guard ladder
is unchanged in order and in every status code. A preempt request is
NEVER a rejection reason — without the capability the steer still
enqueues and the 202 echoes preempt: false. Armed strictly after a
successful enqueue; cancel disarms. The capability is read from the
OWNING replica's recorded `preemptCapable` rather than the route
replica's own SDK probe, so a rolling deploy cannot label a steer
"interrupting" that the older owner will only inject at a tool step.
Durable label: SteerQueueItem.preempt → TPendingSteer.preempt, so a
parked/claimed/replayed chip keeps its wording.
Run wiring: createRun registers the PreemptBoundary hook and threads
RunConfig.preemption, both gated on isSteerPreemptSupported() — a
separate probe from isSteeringSupported(), so the client affordance can
never arm against an SDK that only injects at tool boundaries.
buildSteerWiring builds both hooks from one shared closures object, so
preemption survives HITL pause/resume for free.
Honest finalization: an empty preempt boundary persists and emits with
unfinished: true — the same contract an abort gets — re-marked
explicitly because BaseClient has already saved the row as
unfinished: false by that point.
Not changed: no new job status, store method, Lua, SSE event type,
endpoint, or authorization surface. abortJob, completeJob,
transitionStatus, closeAndDrainSteers, getResumeState, emitChunk,
applySteerPart and the whole abort path are untouched.
Tests: 120 packages/api steering specs (preempt lifecycle, tombstone,
fences, caps, terminal release, both-boundary drain parity,
level-triggered poll, request/cancel arming, owner-capability
degradation) plus 5 in api for buildSteerWiring gating, and 2
Redis-gated cross-replica transport specs.
* 🔒 fix: Codex round 2 — evict tombstones, scope the empty-boundary disarm, honest resumes
All four server findings were fresh consequences of the round-1 fixes,
which is the review doing exactly what it should.
- Tombstone cap refused new entries instead of evicting. Every drained
or cancelled steer is tombstoned, not just preempting ones, so a
generation that processed 20 steers exhausted the set and the
late-arm race resurfaced silently. Now evicts oldest-first (Set
iteration is insertion-ordered), with the budget named
PREEMPT_TOMBSTONE_MAX rather than an inline expression.
- The empty-boundary disarm I added in round 1 wiped the generation's
ENTIRE armed set. A second steer can enqueue and arm between the
atomic drain returning empty and the disarm running — that arm is
backed by a live, uninjected queue item and must survive. The drain
now snapshots the armed ids BEFORE draining
(getArmedPreemptIds) and clearPreemptRequests takes an explicit id
list instead of clearing everything.
- HITL resume finalized with a hardcoded unfinished: false. The
boundary hook is re-registered on resume via buildSteerWiring, so a
resumed segment can end on an empty preempt boundary exactly like a
fresh one; finalizeResumedTurn now reads getPreemptStats() and the
halt reason, matching the normal request path.
- Ownership moves on resume, so the job's recorded preemptCapable must
describe the replica that will actually generate. Refreshed before
resumeCompletion; a job created on a capable replica that resumes on
an older one during a rolling deploy no longer acknowledges steers as
interrupting.
Tests: +3 (scoped disarm sparing a post-snapshot arm, oldest-first
tombstone eviction, id-list disarm). 122 packages/api steering specs
green.
* 🚨 fix: Codex round 3 — deserialize preemptCapable from Redis (feature was dead under Redis)
The P1 here is the most consequential defect in the whole feature, and
it was introduced by round 1's own capability fix.
- `RedisJobStore.serializeJob` writes booleans generically, so
`preemptCapable` reached Redis — but `deserializeJob` is an EXPLICIT
field map and had no line for it. Every `getJob()` therefore dropped
the flag, `job.metadata.preemptCapable` was always undefined, and
`handleSteerRequest` computed `preemptArmed: false` unconditionally.
Interrupt & steer would have silently degraded to ordinary
tool-boundary steering in EVERY Redis deployment — i.e. the feature
shipping as a no-op in production while passing every in-memory test.
Now deserialized, with a round-trip assertion in the metadata spec
that fails (`Received: undefined`) against the unfixed store.
- The resume capability refresh moved from just-before
`resumeCompletion` to immediately after `approvals.resolve` claims
the run. That call already flips the job back to `running`, so the
steer route accepts requests from that instant; leaving the refresh
135 lines later (across the whole client reconstruction) left a real
window where a steer read the PREVIOUS owner's capability. Not the
fully atomic transition Codex suggested — that reaches into the
approvals Lua — but it shrinks the window from seconds to one await,
which is proportionate for a label-accuracy issue.
Refuted: "avoid triggering preemption inside subagents". The premise —
that the run-wide poll can seal a subagent stream — does not hold
against the shipped SDK. Child graphs are constructed with
`subagentScope: true` (SubagentExecutor) and `preemption` is NOT
propagated into child inputs, while `canClaimPreemptSeal()` requires
`!subagentScope && preemption != null`. Both conditions fail
independently, so a subagent can never claim a seal and the boundary
cannot fire with `agentId` set. The `input.agentId != null` guard in
the hook is defensive depth, not the thing standing between us and the
described failure.
140 packages/api specs green.
* 🔁 fix: Codex round 4 — re-arm durable interrupt steers when resume moves owners
- An arm lives only in the owning replica's runtime plus a transient
pub/sub message, while the steer's `preempt` flag is durable on the
queue item. A HITL resume landing on a different replica therefore
started with an empty armed set and a poll stuck false, so an
interrupt the user had already been ACKed for silently waited for an
ordinary tool boundary. New `GenerationJobManager.rearmQueuedPreempts`
rebuilds the armed set by peeking the durable queue (fenced on the
generation) and re-arming every item flagged `preempt`; resume calls
it right after claiming. Safe by construction: every item peeked is
still queued, so no drained steer can be resurrected.
- Capability-refresh failure now logs at error rather than warn, but
deliberately does NOT fail the resume — see the reply on that thread.
Tests: +2 (rebuild from queue arms only the flagged item and reports
the count; a stale generation arms nothing). 124 packages/api steering
specs green.
* 📡 fix: Codex round 5 — acknowledge only what was actually armed
- A cross-replica arm was fire-and-forget: `emitPreempt` logged its own
publish failure and `requestPreempt` returned void, so the route
answered `preempt: true` even when the owner never armed a poll. The
steer still injected at the next tool boundary, but the chip claimed
an interrupt that could not happen — and unlike HITL resume, an
ordinary running generation had no durable reconciliation to recover
it.
`emitPreempt` now resolves to the subscriber count and rejects on
failure; `requestPreempt` is async and returns whether the arm truly
landed (owned locally, or delivered to at least one subscriber). The
202 reports THAT rather than what was asked for, so the chip relabels
to ordinary steering exactly as it does for a capability-degraded
deployment. Errors are swallowed into `false` — an unarmed interrupt
is a downgrade, never a failed steer.
- The owner capability is re-read immediately before enqueue rather
than reused from the top of the guard ladder. `checkAgentAccess` and
file resolution are awaits, so a request can span an entire HITL
pause/resume that moves ownership to a replica with different
capability and rewrites that very flag. Only paid for by requests
that actually asked to interrupt.
Tests: +3 (not-armed when the publish reaches nobody; armed when this
replica owns the generation; a throwing publish downgrades instead of
propagating). 127 packages/api steering specs green.
* 🎯 fix: Codex round 6 — real ownership, confirmed disarms, and a CI regression of my own
Three review findings plus three CI failures the round-5 commit caused.
Review:
- Ownership came from `runtimeState`, which a cross-replica `getJob`
populates with a FACADE runtime on any replica that merely read the
job. Matching `createdAt` therefore proved only "we looked at this
job", so a non-owner could arm nothing and report success. Ownership
now comes from `ownedJobs`, the actual owner map.
- `armPreemptIds` returns how many ids it accepted, and a local arm is
only reported as armed when one was. A tombstoned id (its steer
drained at an ordinary boundary mid-request) no longer answers
`preempt: true` for an interrupt that cannot happen.
- The cancel disarm is awaited. A dropped clear is worse than a dropped
arm: the owner keeps a level-triggered request for a steer that no
longer exists, seals its next chunk and truncates an unrelated
answer. The boundary drain's own call stays non-blocking — there the
owner is local, so the disarm is already effective and awaiting the
informational publish would only delay injection.
- Subscriber count is NOT read as proof of owner receipt: the count
includes this replica's own facade subscription. A successful publish
reports armed, a rejected one does not. Documented rather than
papered over — see the acknowledgement-semantics note on the PR.
CI regressions from round 5, all mine:
- `registerPreemptSubscription` was AWAITED at both runtime-init sites,
so job creation blocked on a second Redis channel subscription and
hung when that subscribe was slow. Abort is awaited because a missed
abort strands a run; a missed preempt only degrades that steer to the
next tool boundary, so it now registers without gating createJob.
- Two api specs mocked `@librechat/api` without the newly imported
`isSteerPreemptSupported`, so the call threw before createJob; and one
exact-match assertion needed the new `preemptCapable` metadata field.
- My own Redis integration spec asserted arm-before-clear ordering,
which two publishes carry no guarantee of — the receiving tombstone
exists precisely because of that. Now asserts delivery and payload
fidelity, order-independent.
158 packages/api specs, 27 api specs green.
* 🧭 fix: Codex round 7 — settle the acknowledgement semantics (Option A)
Round 7's second finding is the incoherence I flagged on the PR: the
route persisted `preempt: true` on the durable queue item while
returning `preempt: false` when delivery could not be confirmed. Those
two then disagreed, and `rearmQueuedPreempts` trusts the DURABLE one —
so a resumed owner would honour an interrupt the client had explicitly
been told degraded to ordinary steering.
Rather than patch the disagreement, this settles the meaning:
`preempt` in the 202 means "queued as an interrupt request", NOT "a
seal is guaranteed". It mirrors `SteerQueueItem.preempt` exactly, so
the response, the durable record, and the resume-time re-arm can never
disagree. The gates that ARE knowable stay — the owner's recorded
capability and a successful enqueue. Everything past that degrades to
the documented fallback of injecting at the next tool boundary.
A route cannot synchronously know whether another replica will seal:
proving it needs a correlated request/response over pub-sub, and even
that only proves the owner heard, not that it is still streaming when
the arm lands. Four rounds of tightening this boolean each surfaced a
narrower case; the sequence does not converge, so the invariant is now
"the flag describes the durable decision" and an unconfirmed arm logs a
warning instead of rewriting the answer.
Also from this round: a failed disarm publish is retried once and its
outcome reported. `handleSteerCancel` keeps `removed: true` — the steer
really did leave the queue, and saying otherwise would make the client
re-show a chip for a steer that can never arrive — and adds
`disarmed: false` so the residual risk is visible rather than swallowed.
Damage stays bounded regardless: the empty-boundary self-clear disarms
the generation after a single seal.
Tests: +1 pinning the response/durable-flag invariant. 159
packages/api specs green.
* 🧹 fix: Codex round 8 — remove the unverifiable disarm signal
Round 8 found the same over-promise on the disarm side that round 7
corrected on the arm side, so this applies the same answer rather than
patching around it.
The `disarmed: false` field added in round 7 was both unreliable and
unused: a resolved publish is not proof the owner heard it (the
delivery count includes this replica's own facade subscription), and it
was never threaded into `CancelSteerResponse` or read by any client. A
signal that claims a certainty the transport cannot provide is worse
than no signal — it invites callers to trust it.
Removed from the response. The retry stays, because it genuinely
reduces the failure rate, and `noteSteersRemoved` still returns whether
the publish succeeded FOR LOGGING, now documented explicitly as
"published without error", not "the owner disarmed".
Disarm is best effort with a bounded, self-healing failure: if the
clear is lost the owner seals once, the empty-boundary self-clear
disarms the generation, and the turn is persisted `unfinished: true`
rather than silently truncated. Tightening that further needs a
correlated request/response over pub-sub with a timeout — noted on the
PR as the deliberate boundary of this design rather than an oversight.
130 packages/api steering specs green.
* 🧽 fix: Codex round 9 — spend snapshot arms on nonempty drains too
The round-6 scoping fix only cleared the pre-drain snapshot when the
drain came back EMPTY. On a nonempty drain the `finally` cleared just
the drained ids, so a stale arm — typically a cancel whose
cross-replica clear was lost — survived the boundary. It would then
immediately seal the continuation meant to answer the steer that had
just been injected, and land on an empty boundary as
`preempt_incomplete`: the interrupt appears to work, and the answer to
it is truncated.
A boundary that runs has spent its seal, so everything armed at
snapshot time is spent whether or not it came back from the drain. The
`finally` now clears the union of the snapshot and the drained ids.
Arms that land AFTER the snapshot are still spared — their queue items
are live and uninjected, which is the property round 6 added.
Also fixes an api-workspace CI failure of mine: `resume.spec.js` stubs
`GenerationJobManager` wholesale, and the round-3/4 resume work added
two calls (`updateMetadata`, `rearmQueuedPreempts`) the stub did not
define, so 34 specs threw. Stub extended.
Tests: +2 (a nonempty drain clears a stale snapshot arm; a nonempty
drain spares an arm that landed mid-drain). Counterfactually verified —
the stale-arm spec fails against the unfixed drain. 132 packages/api
specs, 60 resume specs green.
* fix: never let a failed preempt subscription reject into the void
registerPreemptSubscription is called detached at both sites, so a
rejected Redis SUBSCRIBE became an unhandled rejection — process-fatal
under Node's default --unhandled-rejections=throw. The comment already
promised this path merely degrades steering; it now does.
Swallowed and logged inside the registration rather than at each call
site, so a future third caller cannot reintroduce the trap. Losing the
channel costs this generation's cross-replica preempts, not the server:
same-replica arming is runtime state and still works, and remote arms
fall back to the next tool boundary.
Verified counterfactually — the new spec surfaces SUBSCRIBE failed as an
unhandled rejection against the unfixed registration.
* docs: state the real blast radius of a failed preempt subscription
LibreChat's own entrypoints install a global unhandledRejection handler
that logs and keeps serving, so the escaping rejection this guards was
never fatal to this server — only to another consumer of @librechat/api
that installs no handler. The fix stands either way; the comment just
should not overstate what it prevents.
* test: cover the cross-replica preempt hop with two manager instances
Every other preempt test runs against a single manager, so the hop that
actually carries an interrupt in production had no coverage: the steer POST
lands on whichever replica the balancer picks, which is usually not the one
generating. Non-owner publishes, owner arms, owner's level-triggered poll
flips — none of that was exercised end to end.
Two GenerationJobManagerClass instances are a faithful replica pair here.
runtimeState and ownedJobs are private instance fields, there is no
module-level mutable state between them, and createStreamServices duplicates
a dedicated subscriber connection per call, so separate OS processes would
exercise the same objects over the same Redis.
Both assertions verified counterfactually against real Redis:
- Deleting the preemptCapable deserialization in RedisJobStore fails this
with 'Expected: true, Received: undefined' — the exact P1 that shipped past
every in-memory test and would have made the feature a silent no-op on
every Redis deployment.
- Dropping the non-owner arm publish fails it with 'Received: false'.
* test: remove the fixed sleeps and vacuity from the cross-replica preempt test
Codex round 11, both findings, both on the test I added last commit.
P2 — the 300ms waits were load-bearing. Redis pub/sub never replays and the
owner's SUBSCRIBE is detached, so on a slow CI worker the publish could land
before anyone was listening and the test would fail against correct code.
Now it republishes until the owner's state converges, which is safe because
arms and clears are idempotent set writes keyed by steerId. Side effect: the
tests got ~10x faster (85ms/57ms vs 929ms/606ms) since they finish on
delivery rather than on a timer.
P3 — afterEach destroyed only the transports, leaving each manager alive in
its own cleanup-interval closure, still working against a dead transport.
Now tracks the managers and awaits destroy(), which disposes the job store
and its timer too. Matches how the rest of this file cleans up.
Fixing the sleeps exposed a third problem codex did not flag: the stale-arm
test could pass vacuously, because an undelivered arm and a fenced one look
identical. It now brackets the stale publish between two control arms — the
first proves the owner is listening before the stale one is sent, the second
proves it has had its chance to arrive.
Verified counterfactually against real Redis, and stable over 5 runs:
- dropping the preemptCapable deserialization fails with 'Received: undefined'
- dropping the non-owner arm publish times out both tests
- removing the generation fence fails the stale test with
["control-before", "steer-stale", "control-after"] — which also confirms
the bracketing orders as intended rather than by luck
* fix: gate interrupt on the OWNER's capability alone, not the route's
Codex round 12. The comment above this gate already said 'the OWNER's
recorded capability, not this replica's probe' — and then the code ANDed in
isSteerPreemptSupported(), which is exactly this replica's probe. The
contradiction dates to the original commit; round 6 made the gate
owner-scoped and wrote that comment without removing the local conjunct.
The route never seals. It enqueues and publishes an arm, neither of which
touches the SDK, so during a rolling deploy a steer landing on an
un-upgraded replica silently lost its interrupt even though the owner could
seal. When the route IS the owner the probe is redundant anyway: the flag it
would consult is the one this process wrote at createJob.
The real degradation path is unchanged and still tested — an owner that
recorded no capability relabels to an ordinary steer. The test that pinned
the local probe asserted an impossible same-replica state (capable metadata
plus an incapable local SDK, when the metadata is written from that probe);
it now pins the mixed-SDK direction instead, and fails with
'Expected: true, Received: false' if the probe is put back.
* fix: reconcile arms at handover, and stop holding the 202 on a publish
Codex round 13, two of three findings.
P2 — rearmQueuedPreempts only ever ADDED. A replica that merely read the job
still installs a facade runtime and subscribes, so it can accept an arm and
then miss the best-effort clear that follows the drain. HITL resume promotes
that facade to owner, the union keeps the orphan, and the first resumed
stream seals on a steer no longer in the queue, drains nothing, and
truncates the resumed answer as preempt_incomplete. acquireResumedJobOwnership
only sets ownedJobs, so nothing else was clearing it. The durable queue is
the sole authority at a handover: arms it does not back are now disarmed and
tombstoned, so an in-flight publish cannot revive them either.
Worth recording that my own independent review raised this and my verifier
refuted it. Codex found it separately; two reviewers converging should have
outweighed one refutation.
P2 — the route awaited the arm publish before answering. The 202 reports
capability, not delivery, so the await could not change the response; it only
exposed the caller to Redis latency after the queue item was already durable.
A client that times out and retries mints a second steer while the first
stays queued, injecting the same instruction twice, whereas a lost publish
merely takes the tool-boundary fallback. Detached, with both outcomes logged.
All three tests verified counterfactually: union-only rearm fails the two new
handover specs, and re-awaiting the publish hangs the stalled-publish spec
until jest kills it.
* fix: snapshot arms before reading the queue at handover
Codex round 14 — a regression from my own round-13 fix, and a worse failure
than the one it corrected.
Round 13 read the durable queue first, then tombstoned any armed id the
snapshot did not back. But approvals.resolve reopens steering before
reconciliation runs, so another replica can commit a preempt steer and
publish its arm while the peek is in flight. That arm is then present locally
but absent from a snapshot taken before the steer existed, so a LIVE
interrupt the route already acknowledged got dropped — and tombstoned, which
blocks the re-arm, making it unrecoverable rather than merely late.
Fixed by inverting the two reads rather than by locking or paying a second
round trip. A steer is durably enqueued BEFORE its arm is published, so any
id in an arms-first snapshot was already queued when it was armed, and the
later peek must observe it unless it has since drained — which is exactly the
orphan this reconciliation exists to drop. Arms landing after the snapshot
are simply not candidates.
Also re-checks runtime identity across the await, since the generation can be
replaced while the queue read is in flight.
New spec injects a steer + arm during the peek and verifies it survives;
against the round-13 ordering it fails with Received array: [].
* fix: bound the cancel disarm wait and fence enqueue to its generation
Codex round 15.
P2 — the cancel awaited its disarm publish unbounded. ioredis queues
commands during an outage rather than rejecting, so that await could hang for
the length of the outage with the steer ALREADY durably cancelled; a client
that gives up then treats the cancel as failed and restores a chip for a
steer that can never produce an applied event. Every successful cancel
publishes, so ordinary steers were exposed too, not only preemptive ones.
Now bounded at 1s, with the publish continuing behind it — its retry and
logging are unchanged, it is just no longer in front of the response. This is
the sibling of round 13's arm-publish finding; I fixed one path and left this
one.
P3 — enqueue was not fenced to the generation the capability decision was
made against. The access checks, file resolution and owner re-read are all
awaits, so the run can be replaced before the enqueue: the item then lands on
the REPLACEMENT queue while the durable preempt flag and the arm still name
the previous epoch, the arm is fenced out at the owner, and the 202 promises
an interrupt that cannot happen. enqueueSteer now takes an expected
generation, mirroring drain/peek, and the Redis path enforces it inside
STEER_ENQUEUE_LUA so the check is atomic with the push rather than racing it.
All three new specs verified counterfactually, including the Lua guard
against real Redis (removing it returns 1 where -1 is required).
* ⚡ feat: Interrupt & Steer — client half (PR 3 of 3)
Makes preemptive steering reachable. Consumes the server contract from
PR 2 (POST /chat/steer `preempt`, echoed on the 202) and the SDK seam
in @librechat/agents 3.3.5.
Settings shape follows the agreed correction, NOT the earlier plan
draft: `steerInterruptsByDefault` is a boolean ORTHOGONAL to
`duringRunDefaultAction` — that enum still chooses steer-vs-queue, the
new boolean chooses how soon a steer lands. This deliberately avoids
widening the enum to three values, which would have silently broken two
hard-coded binary TOGGLES (`DuringRunAction.tsx`'s setter and
`SteerMenu`'s `useDefaultToggleEntry`, both `prev === 'steer' ? … : …`)
where a third value collapses to the wrong branch and one click erases
the setting.
- useSteering: `submitSteer` takes an opts bag and threads `preempt`
into the POST, the optimistic chip, and the failure chip. The ACK
relabels from the SERVER's echo, so a deployment that cannot seal
mid-stream downgrades the chip's wording instead of erroring — the
entire UX surface of capability degradation. New `interruptSteer`
reuses the whole chip lifecycle and degradation ladder, and falls
back to `interruptAndSend` when `!canSteer`, because steering needs a
server-side job and an always-visible button would otherwise be dead
for the whole first turn. `steerFromComposer` honours the new
preference.
- Composer: always-visible `InterruptSteerButton` with one fixed
meaning (stop now, keep what's written), disabled on a paused run to
pre-empt the server's 409, `type="button"` so it never steals the
form's Enter submit, RTL-correct margins. A fourth hovercard row on
the during-run send button, and ⌘/Ctrl+Shift+Enter routed AHEAD of
the bare ⌘/Ctrl+Enter branch that would otherwise swallow it.
- Chips: an in-flight preempt chip reads "Interrupting" with a ZapOff
glyph; `preempt` survives reconnect through `seedSteerChips`.
- `RunEnd.interruptArmed`, `drainAfterAbortByIndex`, `useQueueDrain`,
`stopGenerating` and `interruptAndSend` are untouched — the preempt
path deliberately shares none of the abort machinery.
Tests: 7 new specs (posts preempt, turn-1 fallback, empty-text refusal,
default route with and without the preference, server-echo relabel,
double-click). 66 useSteering specs green; tsc and lint clean.
Round-1 review fixes folded in:
- P1: interrupt & steer no longer hard-aborts a run paused on tool
approval. `canSteer` is false there, so the fallback was routing the
keyboard and hovercard paths into `interruptAndSend` — discarding the
partial answer, the exact opposite of what the action promises. The
fallback is now scoped to the missing-conversation case only, and a
paused run refuses outright (the standalone button was already
disabled; the guard now lives where all three paths reach it).
- The preference no longer leaks into the explicit Steer action.
`steerFromComposer` backs both the default Enter route AND the
explicit hovercard row / Ctrl+Enter alternate; applying
`steerInterruptsByDefault` inside it made ordinary Steer interrupt and
the two rows indistinguishable. It now takes an explicit argument that
only `submitDuringRun`'s default route sets.
- Retry preserves preemption: a failed interrupt-steer chip keeps
`preempt: true`, and `retrySteer` now forwards it rather than silently
resubmitting as an ordinary tool-boundary steer.
- ⌘/Ctrl+Shift+Enter defers to a rebound submit shortcut, mirroring the
bare ⌘/Ctrl+Enter branch — a user who bound submit to that chord keeps
getting submit.
Round-2 fix: the preempt label now survives the page-reload resume path
too. `seedSteerChips` (useResumableSSE) and `restoreSteerChips`
(useResumeOnLoad) are two independent TPendingSteer→PendingSteer
mappers with near-identical bodies; the first carried the flag and the
second silently dropped it, so an armed interrupt reverted to plain
"Steering" after a reload. Swept: those are the only two in production
code. The reclaim/convert paths deliberately omit it — a queued
follow-up starts its own turn, so there is nothing to interrupt.
* fix: yield the interrupt-steer chord only to a submit shortcut bound to it
The previous guard skipped the Ctrl/Cmd+Shift+Enter branch whenever ANY
submitMessage override existed. Rebinding submit to something unrelated
(Ctrl+J) or unbinding it entirely then fell through to the override
resolver, which returns 'none' for shifted Enter — silently removing the
shortcut the hovercard still advertises.
Compare the pressed chord against the configured one instead. The
adjacent bare Ctrl/Cmd+Enter branch keeps its any-override guard on
purpose: that chord IS the default submit chord, so once submit moves
the resolver should own it.
The predicate already existed inside resolveSubmitOverrideAction; pulled
it out as bindingsMatch so both sites compare chords the same way. That
call is behavior-preserving — eventBinding.key is 'Enter' by the early
return, and equal hashes imply equal keys, so the dropped explicit key
check was redundant.
* fix: disable the Interrupt & steer menu row while paused on approval
interruptSteer hard-refuses when the run is paused for tool approval, but
the hovercard row was never gated, so it rendered enabled and clicking it
did nothing at all — no chip, no queue entry, no toast — at exactly the
moment a user is trying to say "stop, don't run that command". The
standalone button already gates on pausedOnApproval; the row contradicted
it.
Gated on pausedOnApproval rather than !canSteer like the steer row above,
because canSteer is also false before a conversation exists, where
interruptSteer deliberately falls back to interruptAndSend and the row
must stay live for the whole first turn.
Tests pin both directions and were verified counterfactually: removing the
gate fails the paused case, and using !canSteer fails the first-turn case.
* test: render the during-run hovercard eagerly instead of driving Ariakit
The new spec passed locally and failed all four cases on CI's Ubuntu and
Windows shards: Ariakit's show path keys off pointer geometry, which jsdom
reports as zeros, so whether a synthetic mouseEnter opens the hovercard is
environment-dependent. Driving it was testing Ariakit's hover behavior, not
which rows this component disables.
Mocking the three Ariakit primitives renders the rows unconditionally and
drops the fake timers. Both counterfactuals still fail as they should:
removing the gate fails the paused case, !canSteer fails the first-turn case.
* test(e2e): cover interrupt & steer sealing mid-stream
The mock Playwright suite covered every sibling during-run action — steer at
a tool boundary, steer degrading to a queued follow-up, queue, and interrupt
& send — but not interrupt & steer, the one this stack adds.
Uses E2E_SLOW_REPLY, which streams pure text with no tools, so the scenario
is the same one where an ordinary steer provably degrades to a queued
follow-up turn. Injecting in-thread there is something only a mid-stream
seal can do, which makes the assertion discriminating rather than incidental:
the steer part lands in the response, the final chunk never arrives, the text
written before the seal survives, and no follow-up turn pair is created.
* test(e2e): assert the run resumes after the seal, not just that it sealed
The other four assertions are all satisfied by a seal that killed the run:
the steer part is persisted by applySteer during the drain, before the
continuation starts, so 'sealed and resumed' and 'sealed and died' were
indistinguishable — and resuming is the whole difference from interrupt &
send.
The continuation answers the injected steer, whose text carries no
fake-model marker, so getLatestUserText falls through to the default reply.
That string ('E2E mock reply') is distinct from the setup turn's
('E2E reply <label>'), so seeing it proves generation restarted rather than
matching text that was already on screen.
The test itself is confirmed working: it ran as 104/121 in the Playwright
job on
|
||
|---|---|---|
| .. | ||
| benchmarks | ||
| benchmarks-reasoning | ||
| config | ||
| fixtures/deployment-skills/e2e-deployment-skill | ||
| recordings | ||
| setup | ||
| specs | ||
| config.local.example.ts | ||
| jestSetup.js | ||
| playwright.config.a11y.ts | ||
| playwright.config.benchmark.ts | ||
| playwright.config.local.ts | ||
| playwright.config.mock.ts | ||
| playwright.config.real.ts | ||
| playwright.config.reasoning-perf.ts | ||
| playwright.config.ts | ||
| README.md | ||
| types.ts | ||
LibreChat e2e
The mock e2e profile is the safest default for generated tests. It starts LibreChat with e2e/config/librechat.e2e.yaml, injects an in-process fake LLM (via LIBRECHAT_TEST_RUN_HOOK), creates an authenticated e2e user, and avoids real provider credentials.
Recording Tests
Use Playwright codegen when you want to turn an exploratory browser session into a draft test:
npm run e2e:record
That command builds the app, starts the LibreChat test server (with an in-process fake LLM) when needed, writes e2e/storageState.json, and opens Playwright codegen at /c/new. The npm script uses http://localhost:3333 so it does not collide with a normal dev server on 3080. Raw recordings are written to e2e/recordings/ and ignored by git.
For a real local LibreChat config instead of the mock profile:
npm run e2e:record:local
Useful direct options:
node e2e/setup/record.js --url=http://localhost:3080/c/new
node e2e/setup/record.js --profile=local --no-output
node e2e/setup/record.js --auth-only
node e2e/setup/record.js --output=e2e/recordings/settings-draft.spec.ts
LLM-Assisted Loop
- Start
npm run e2e:record. - Let the LLM use Computer Use to operate the headed Playwright browser.
- Stop codegen after the workflow is captured.
- Move the useful parts from
e2e/recordings/into a committed spec undere2e/specs/mock/. - Replace brittle generated selectors with role, label, text, or
data-testidlocators. - Add assertions that prove the behavior, not just the clicked path.
- Run the finished spec with
npm run e2e:mock -- <spec name>.
Generated recordings are a draft, not the final test. The committed version should use the shared helpers in e2e/specs/mock/helpers.ts where possible, wait on network or visible UI state instead of fixed sleeps, and keep test data deterministic.