* 🎭 ci: Gate Playwright Lanes and Docker Smokes on Codegraph Selection (Stage 2)
* ci: surface the fail-open reason in the stage-2 select summaries
* ci: log the stage-2 select decision for harvesting
* ci: fail open on fetch failure or truncated file list; type-strict skip decisions (Codex)
* ci: check curl's exit status before honoring a selection (Codex r2)
Every Playwright job spent a flat 90s on `npx playwright install ffmpeg`,
and none of them ended up with a usable ffmpeg.
The 2.3MB download finishes in under a second; extraction then hangs until
`timeout -k 10 90` reaps it (exit 124, masked by `continue-on-error`). That
is a Node 24.16.0 readable-stream change (nodejs/node#62557) colliding with
yauzl/fd-slicer never firing `close` after EOF, which hangs extract-zip.
It leaves a truncated `ffmpeg-linux` — 5,055,201 bytes against the zip's
declared 5,101,056, segfaulting on exec — and no INSTALLATION_COMPLETE
marker, so Playwright treated ffmpeg as uninstalled. `video: 'on-first-retry'`
has therefore never worked in CI, and every first retry of a flaky test died
in browserContext.newPage: exactly the failure the step existed to prevent.
Upstream fixed it in Playwright 1.60.0 (microsoft/playwright#40747) and Node
reverted it in 24.18.0 (nodejs/node#63834). Node 24.16.0 is pinned in 17
places including the Dockerfiles, so bump Playwright instead — it is a dev
dependency, and `^1.56.1` already permitted 1.62.1; only the lockfile pinned
it. Staying at or above 1.62.1 also avoids the tsconfig-resolution
regressions in 1.62.0.
Caching alone could not have fixed this: a cold cache still hangs, and what
would have been cached is the corrupt binary. So the ffmpeg download is now
restored from cache keyed on the resolved playwright-core version, the
install is skipped outright on a hit, and the cache is only saved once the
binary is verified to actually execute — a partial extraction can never be
promoted into a cache that every later job restores.
Per job: 90s to ~0s on a hit, ~2s on a miss.
* refactor: presets, skills motion and model selector polish
Four surfaces that had drifted from the rest of the app, plus the CI
fragility that surfaced while getting them green.
Two were functional bugs rather than styling:
Keyboard focus was invisible in the model selector. The highlight rule
existed and the background was painted, but it used surface-secondary and
the menu sits on bg-presentation, which resolve to the same value in dark
and to within 3/255 in light, so only the thin indicator bar ever showed.
Keyboard focus now uses the same surface a pointer gets.
Importing a malformed preset raised com_ui_upload_invalid, which talks
about image size limits, and FileUpload's JSON.parse had nothing catching
it at that call site. The overflow menu owns the input and reports the
existing preset import error instead.
The rest is polish: preset surfaces use the theme radius roles rather than
raw values; the edit dialog stops nesting a fixed 350px scroll box inside
an already scrolling dialog and pins its title and actions, with the
endpoint picker moved to ControlCombobox and kept out of any clipping
ancestor; Clear all and Import move into a three-dots menu matching the
conversation row; the Skills sections and pinned chats adopt the Collapse
that Projects already used; the rendered/source toggle slides between
states, is extracted rather than duplicated, and gains the accessible name
and RTL mirroring it lacked; the header toggle loses its fill and the
mobile new chat button hides when you are already in a new chat.
The CI changes are unrelated to the UI but blocked it: the MCP and Redis
cache jobs installed Redis with a bare apt-get and lost a race against the
runner's own apt-daily work, failing four times and once hanging for 30
minutes. They now stop that background work and wait for the lock.
DPkg::Lock::Timeout alone does not help, since it covers the dpkg frontend
lock and not the lists lock.
* refactor: move the section label appearance into the Label primitive
The preset dialog reached into the agent panel's private `Advanced/ui` for
its field eyebrow, so an agent-only refactor could change the dialog.
Give the shared `Label` a `section` variant and export the recipe for the
agent id row, which heads its value on a span and must not inherit the
label's block layout. Each variant carries its own size, leading and color:
the recipe output reaches that span unmerged, and a font size declared after
`leading-none` drops it.
* fix: derive the mobile new chat action from the route
The context conversation still holds the previous chat for a render after a
history or link navigation, a lag ChatView already guards against, so the
action could show on /c/new or hide while an existing chat loaded.
* style: sort imports in the touched files
* fix: return focus to the menu item after the clear dialog
The dialog is controlled and has no trigger, so Radix restored focus to
whatever held it when the content mounted, the menu's own focus trap, and a
keyboard user was left on the document. The menu stays open behind the
dialog, so the invoking item is still there to take focus back.
* fix: fall back to the trigger when clearing removes the invoking item
Confirming empties the presets optimistically, so React commits the removed
menu item together with the dialog close and the saved invoker is already
disconnected when focus is handed back.
---------
Co-authored-by: Danny Avila <danny@librechat.ai>
* feat: route live subagent controls across replicas
* fix: initialize task routing in cluster workers
* fix: harden cross-replica task routing
* fix: expire routed task owners independently
* fix: close cross-replica routing edge cases
* fix: bound owner refresh and close routed cancellation gaps
Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.
Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.
Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.
Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: retain claimed results apart from control replays
A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.
Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: never consume a result the owner cannot replay
Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.
Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: own a claimed result until it is acknowledged
A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.
Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.
Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: treat acknowledgement as part of delivering a result
Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.
An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.
Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* style: sort the widened node:crypto import
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: own control invocations and cancellation plans at the task seam
Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.
Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* style: separate the control invocation map from the next member
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: close subagent deletion, claim, and control invocation gaps
Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.
A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.
The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: assert the drain's calls before restoring its spies
Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: close the follow-on gaps in the deletion fence and result claim
The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.
The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.
Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: hold every deletion fence and keep live idempotency records
An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.
A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.
Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: keep the deletion fence portable and never drop a live record
The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.
An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.
Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: hold the fence, bound recovered results, and expire stale commands
The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.
Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.
A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.
A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: size the fence window so a renewal can be observed
The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix: wire the durable claim method and close the fence follow-ons
The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.
The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.
Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.
A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: wait on observed lease renewal instead of a fixed delay
The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): close the routing, fence, and cache gaps found in review
Five separate seams, each with its own failure:
`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.
The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.
`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.
Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.
The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): cap the merged task list the poll tool actually reads
Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: admit every task the merged-list cap test starts
The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): let a deletion notice its admission fence lapsing
Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: raise both concurrency caps the merged-list test trips
Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): close the fence renewal gap and keep running tasks listed
A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.
The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): finish the cap and the fence at the seams they still missed
The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.
A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.
A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* test: cover the lease lapse and the post-deletion re-drain
The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ
* fix(agents): close live-task lifecycle gaps
* test(redis): exercise cluster node discovery
* fix(test): type cluster discovery seam
* fix(ci): wait for orphaned apt processes
* fix(ci): reserve time for apt drain
* fix(ci): skip optional fonts in MCP jobs
* fix(agents): recover tasks after owner loss
* fix(agents): preserve local task discovery
* fix(agents): initialize fail-fast cluster publisher
* style(agents): sort routing test imports
---------
Co-authored-by: Claude <noreply@anthropic.com>
* ci: codegraph test-selection probe (observe-only)
Asks the codegraph service which test files and matrix jobs the PR
needs and writes the decision to the job summary. Gates nothing —
every path exits 0, forks without secrets no-op. Companion to the
shadow-mode evaluation: the decision CI would act on, made visible
next to the runs it would have replaced.
* chore: remove GitNexus CI and deployment configs
Superseded by the codegraph service: the index workflow spent ~45min
per invocation building an artifact the PR flow never served, while
the replacement indexes incrementally in ~1.4s per commit server-side.
Removes the four workflows (index, deploy, cleanup-pr, pr-command) and
the .do/gitnexus deployment bundle. No remaining references.
* ci: render playwright spec tiers in the codegraph probe summary
* test(e2e): grant MULTI_CONVO.USE in the mock e2e config
`agent-skills-added.spec.ts` drives the composer's `+` command, which opens the
added-model popover. That path is gated on MULTI_CONVO.USE:
if (!hasMultiConvoAccess || !plusCommandEnabled || isAssistantsEndpoint(endpoint)) return;
The mock config never sets `interface.multiConvo`, so the permission falls
through to the seeded role default and `handlePlusCommand` returns before
opening the popover. The spec then fails on a popover that is absent from the
DOM entirely, which reads as a selector or timing problem rather than a missing
permission.
Set it explicitly, the same way `contextCost` is set just above for the usage
gauge — the mock config's job is to make each exercised feature's gate explicit
rather than inherit a default.
* test(e2e): wait for the reply to settle before the double-click quote
`quotes.spec.ts` › 'summons the popup from a native double-click word
selection' double-clicks a word as soon as `mockReply` becomes visible. But
`sendMessage` resolves on the stream *response*, not on the final render, so the
reply can still be re-rendering.
A streaming markdown re-render swaps out the text node the selection points at,
which collapses the selection — the same mechanism the sibling
`selectionchange` test documents deliberately. A double-click landing mid-stream
therefore loses its selection before the popup can be clicked, and because the
whole gesture is wrapped in `toPass`, every retry re-runs into the same
still-streaming reply rather than recovering from a one-off.
This is a different race from the one #14777 fixed. That one is the *selection*
still settling (touch long-press, native handle drags, block-granularity
gestures) and is handled inside QuoteButton. This one is the *reply* still
streaming, which no amount of component-side settling can absorb.
Observed on a downstream fork running this suite on slower hardware: the test
fails all three attempts, deterministically on the in-memory stream store while
the Redis lane passes the same shard — the in-memory store's final re-renders
land late enough to outlive the gesture. Four separate runs, same split.
Wait for the reply text to hold steady before selecting.
* ci(e2e): install ffmpeg so first-retry video actually records (#14841)
`playwright.config.mock.ts` sets `video: 'on-first-retry'`, but the runner only
installs `install-deps chrome`, which does not include ffmpeg. Without it the
first retry fails inside `browserContext.newPage` while setting up video
recording — before the test body runs.
The cost is the retry itself: a genuinely flaky test loses the attempt that
would have recovered it, and the reported failure is a video-setup error rather
than the original symptom.
Bounded and non-fatal on purpose. The CLI has been observed hanging after the
download completes on these runners, so the step is wrapped in `timeout` and
its failure is swallowed — if ffmpeg cannot be installed the job proceeds
exactly as it does today, and no lane is blocked on it.
Applied to both jobs that run Playwright (`e2e_shards` and
`mcp_tool_list_changed`), since both configure retries.
`npx playwright install-deps chrome` is the third-most-common e2e failure:
three of the last twenty-five Playwright runs died on it, taking the whole
aggregate gate with them. The step is not installing anything CI needs.
The runner's Chrome is an apt package, so apt has already satisfied every
library Playwright lists — the log shows each one "already the newest version".
All `install-deps` adds are decorative CJK/Thai/Cyrillic font packages, ~21MB
pulled from azure.archive.ubuntu.com by seven jobs on every PR. No CI assertion
depends on them: the only spec that screenshots gates its comparison behind
`E2E_VISUAL_SNAPSHOTS`, which no workflow sets, and no baselines are committed.
Keep the install, but demote it. `google-chrome --version` becomes its own
fatal step so a genuinely missing browser still fails loudly and immediately,
while the font install retries with a per-attempt cap and degrades to a warning.
The Redis install in the list_changed job stays fatal — that one is required.
The full-sweep regression gate lints api+client+packages twice — once under
the PR's config and once under the base ref's — inside the same job as ~20
later steps (data-provider/data-schemas/api builds, config migration tests,
unused-i18n scan, and four depcheck passes), all sharing one 30-minute budget.
Two type-aware sweeps of the whole tree cost more than everything else in that
job combined. When they run long the job hits its timeout mid-sweep, so every
step behind the gate never executes and Static checks reports no result at all
— strictly worse than not running the gate. continue-on-error: true hides this,
because the step never fails; it simply never finishes.
Move the gate to its own job with its own budget so it cannot starve the other
checks, and bound each sweep so an over-budget run degrades to a notice rather
than a failure — an unfinished sweep is no evidence of a regression, and the
gate is advisory about config scope. Behaviour on a sweep that completes is
unchanged: coverage loss and new (file, rule, severity) diagnostics still fail.
* 🧩 ci: Close Workflow Path-Filter Gaps
Six trigger-filter gaps found by reading all workflows against the live
dependency graph (AI-1755, codegraph FINDINGS §6l):
- backend-review/frontend-review: root package.json/package-lock.json now
trigger unit tests — a lockfile-only dependency bump previously ran zero
backend or frontend unit tests while every test job installs from it
- agents-integration-tests: widen to the three package src trees it builds
and imports (was only src/agents/**)
- cache-integration-tests: same shape — verified live that
packages/api/src/flow/manager.ts (imported by mcp/oauth) matched neither
integration filter
- docker-smoke: plain Dockerfile had no PR-time validation despite shipping
via dev-images/tag-images; new node-image-smoke job builds it, gated by
paths-filter to Dockerfile/.dockerignore changes
- dev-images/dev-branch-images: add config/**, skill/**, .dockerignore —
the single-stage image COPYs the full build context
- static-checks: eslint.config.mjs now re-triggers the lint job (gap carried
over from eslint-ci.yml in the #14716 consolidation)
- delete generate_embeddings.yml: fired on docs/**, which no longer exists,
and its docs-root-path pointed at the same missing directory
* 🧩 ci: Address Codex Review Findings
- Build caches: all 26 build-* keys across 8 workflows now lead with
root package.json + package-lock.json so manifest-only bumps cannot
restore stale dists (data-provider embeds the root version); unifies
the split key families (playwright already hashed the lockfile)
- static-checks: config changes now gate on the ESLint config loading
and applying to representative files, plus a report-only full-tree
sweep (70 pre-existing errors at dev HEAD block a hard gate for now)
- docker-smoke: the workflow file itself triggers the plain-Dockerfile
build so job edits are validated
- dev-images/dev-branch-images: re-include skill/**/*.md after !**.md
so shipped deployment-skill Markdown rebuilds images
* 🧩 ci: Gate Config Lint Sweep on Regression vs Base Config
Second-round codex finding: the report-only sweep swallowed config-wide
breakage in scoped blocks the representative files don't exercise. The
sweep now lints the same tree under the PR's config and the base ref's
config and fails only when the PR's config produces more diagnostics for
some (file, rule) pair — pre-existing debt never fails the gate, and
fixes are never penalized. Base-config unavailability degrades to the
load gate with a notice. Outcome surfaced in the failure summary.
* 🧩 ci: Harden Config Lint Gate per External Review
- Coverage direction: fail when the PR config stops linting files the
base config covered (set difference on linted files) — a mis-scoped
ignores previously only removed diagnostics and passed both gates
- Severity-aware fingerprints: (file, rule, severity) so warn->error
escalations gate on a clean tree for that rule; downgrades still free
- Hard-fail when the base commit is missing so a future shallow-checkout
change cannot silently disable the gate; annotate fetch-depth: 0
- Tab-separated fingerprint keys (space-in-path proof), --config on
both sweeps, EXIT trap for the base config copy, comment on why it
must live at the repo root (flat-config pattern base paths)
- Narrow skill md re-include with !skill/README.md: top-level README is
documentation-only; runtime skill Markdown still rebuilds images
* ci: bump GitHub Actions to Node.js 24 runtimes
Clear Node 20 deprecation warnings on runners by moving workflow
actions to majors that declare node24 (checkout, cache, setup-node,
artifacts, Docker buildx/build-push/qemu/login, github-script,
setup-go, Azure login/helm, create-pull-request, axe-linter).
* fix: release leader lock via ioredis on Redis Cluster
@keyv/redis EVAL can surface unhandled MOVED redirects on cluster,
so resign() logged failure and left LeadingServerUUID set. Use ioredis
for leader election SET NX / GET / Lua (same pattern as principals and
concurrency locks) so cluster redirects are retried and resignation
clears the lock.
* 🍃 fix: Amazon DocumentDB Compatibility for Pipeline-Form Updates
- Rewrite acceptTerms without aggregation-pipeline update + $$NOW (null-guarded first-acceptance claim preserves the original timestamp under concurrent and repeat requests)
- Rewrite decrementTagCounts clamp-at-zero decrement as ordered two-op bulkWrite (clamp before guarded $inc)
- Rewrite extendFilesTTL TTL hold as projected read + per-doc guarded $set via tenantSafeBulkWrite, preserving only-widens/ceiling/cleared-stays-permanent semantics
- Log background index-build failures via Model 'index' listeners (previously swallowed silently, e.g. partialFilterExpression rejection on DocumentDB <5.0)
- Add misc/documentdb live-compatibility harness + assessment (AWS-cited)
- Fix latent file.spec helper bug: createdAt backdating was silently stripped by mongoose immutability
Closes#14488
* 🍃 fix: Harden DocumentDB-Safe Updates Against Cross-Call Races
Addresses Codex review on #14495:
- decrementTagCounts: normalize-null / $inc / clamp-negative op triple so
interleaved decrements of the same tag converge on max(0, ...) exactly as
the serialized pipeline did (clamp keys on count < 0, not count < amount)
- acceptTerms: guard the repeat-acceptance fallback with a non-null timestamp
(exact complement of the claim guard) and retry the claim when a
config/reset-terms.js reset races between the two updates, so acceptance
never resurrects a reset cycle without a fresh audit timestamp
* 👷 ci: Suppress Ignored-File Warnings in Changed-File ESLint Run
Changed files under config-ignored paths (packages/data-schemas/misc/**)
emit "File ignored" warnings that fail --max-warnings=0.
* 🛡️ fix: Guard HITL checkpoint size against MongoDB 16MB limit
A LangGraph HITL checkpoint embeds the whole serialized message history
in a single BSON document, so a large conversation (inlined base64 media,
big tool outputs, long history) can serialize past MongoDB's 16MB
document ceiling. `MongoDBSaver.put` would then throw a raw
`BSONObjectTooLarge` at pause time and the pause would be lost with no
legible error.
`LazyMongoSaver` now measures the serialized checkpoint on the persist
path (rare HITL pauses only — the clean-exit common path is untouched):
debug-logs the size, warns past a soft 8MB threshold, and throws a typed
`CheckpointTooLargeError` before the doomed write past a 15MB hard limit
(16MB minus headroom for the document's other fields). Thresholds are
overridable via the constructor for testing.
Adds integration coverage (real serde + mongodb-memory-server) for the
under-threshold, soft-warn, and hard-reject cases.
* 🧱 fix: Add explicit types for isolatedDeclarations build
The production build (tsdown + rolldown-plugin-dts) compiles with
--isolatedDeclarations, which requires explicit type annotations on
exported bindings whose initializers it can't infer syntactically.
`CHECKPOINT_HARD_LIMIT_BYTES` (arithmetic over two consts) tripped
TS9010; annotate it and `CHECKPOINT_WARN_BYTES` as `number`. Verified
with `tsc --isolatedDeclarations` over the package.
* fix: Codex review — include metadata in the size guard + flush parked bookkeeping
P1 (lost bookkeeping): `put` consumes the write anchor, then AWAITS
assertCheckpointFitsDocument (checkpoint serialization). A bookkeeping-only
putWrites dispatched in that window sees neither the anchor nor persistedIds,
so it parks — and `put` never flushed it, dropping the marker (e.g. a
completed Send-sibling's __no_writes__) so a resume re-executed the sibling.
Extract flushBufferedBookkeeping (shared with the anchoring putWrites) and call
it after super.put in the persist path.
P2 (metadata ignored by guard): MongoDBSaver.put stores the serialized
checkpoint AND metadata (plus metadata_search) in the SAME document, but the
guard measured only the checkpoint — a just-under-limit checkpoint with large
metadata fell through to a raw BSONObjectTooLarge. Measure checkpoint +
metadata; the fixed headroom now only covers metadata_search/ids/framing.
Two integration regressions added (both fail without the fix, pass with it):
metadata-pushes-over-the-ceiling, and flush-during-the-serialization-window.
* fix: count metadata_search (raw metadata copy) in the checkpoint size guard
Codex follow-up: MongoDBSaver.put stores metadata a SECOND time as
`metadata_search: metadata` — the whole raw metadata object as a queryable BSON
subdocument in the same agent_checkpoints document. Measuring only checkpoint +
serialized metadata undercounted by that raw copy, so a large metadata.writes
payload could pass the 15 MB preflight while metadata_search pushed the actual
BSON past 16 MB — the raw BSONObjectTooLarge the guard exists to prevent.
Add mongoose.mongo.BSON.calculateObjectSize(metadata) for the metadata_search
contribution (mongoose already imported; no new dep). Headroom now only covers
ids + BSON framing. New integration test sizes a case where checkpoint +
serialized metadata is under the limit but the raw metadata_search copy pushes
it over — green (24/24).
* ci: run packages/api agents integration specs (checkpointer) in CI
The checkpointer.integration.spec.ts (durable HITL checkpointer vs a real
in-process MongoDB) is a *.integration.spec.ts, which test:ci deliberately
excludes — and cache-integration-tests.yml only covers cache/cluster/mcp/
stream, not src/agents/**. So it ran nowhere and its regressions guarded
nothing. Add:
- test:agents-integration script (jest over src/agents/*.integration.spec.ts,
runInBand — mongodb-memory-server is in-process, no external service);
- a dedicated agents-integration-tests.yml (mirrors the proven build setup,
no Redis) triggered on packages/api/src/agents/** changes;
- babel-plugin-replace-ts-export-assignment as a packages/api devDep: the spec
imports @langchain/langgraph-checkpoint (whitelisted for babel transform,
uses `export =`), whose transform needs this plugin — it was only present
under client/node_modules, unresolvable from packages/api, so the suite
couldn't load. 24/24 pass locally.
DataTable.spec failed with "Too many re-renders" (35 tests). Root cause: @tanstack/react-virtual is measurement-driven, and jsdom has no real layout, so its re-render loop never converges. This went unnoticed because packages/client had no jest CI job (only the client workspace runs jest in frontend-review.yml).
- DataTable: only read the virtualizer (getVirtualItems/getTotalSize) when virtualization is active; the non-virtualized branch renders rows directly, so engaging it for small tables was wasted render-phase work.
- Spec: mock @tanstack/react-virtual, since jsdom can't exercise real virtualization layout.
- Add a test:ci script to @librechat/client and a Tests: @librechat/client CI job so packages/client specs run on every frontend PR.
The gitnexus droplet is ~8.7GB usable, not the 60GB the disk-cleanup
comment assumed. With /usr (~2.8GB), the in-use docker images (~2.1GB),
and the growing /opt/gitnexus/indexes (~1.2GB), deploys were aborting at
the `AVAIL_MB < 2048` guard ("Disk critically low").
Two fixes:
- Reclaim the previous gitnexus image after force-recreate. The pre-pull
`docker system prune -af` cannot remove it while the old container is
still running, so a stale ~700MB generation accumulated every deploy.
A post-recreate `docker image prune -f` makes the box self-cleaning.
- Lower the abort threshold 2048 -> 1536MB. The image is ~700MB and
shares most layers with the running one, so an incremental pull needs
well under 1GB; the old guard was sized for the 60GB assumption.
Also corrects the stale 60GB comment to reflect the actual disk.
* 🗜️ ci: Cache Dependencies and Builds in Cache Integration Tests
Port the node_modules and package-dist caching pattern from
backend-review.yml to cache-integration-tests.yml, which ran a full
npm ci (~72s) and rebuilt data-provider, data-schemas, and api on
every run. Cache keys are identical to backend-review.yml so the two
workflows share entries. Drops setup-node's npm tarball cache,
superseded by the node_modules restore, matching backend-review.yml.
* 🗜️ ci: Exercise Warm-Cache Path
* 🪟 ci: Shard Windows Frontend Unit Tests
Mirror the 4-way jest sharding the Ubuntu frontend test job already
uses onto the Windows job, which currently runs the whole client suite
in a single 20-minute job. Also drops the `--verbose` flag, which npm
consumed itself (it preceded `--`) and only raised npm's own log level.
* 🪟 ci: Trigger Frontend Tests on Workflow Changes
* 👷 ci: Add API runtime smoke (boot the production image) to docker-smoke
The docker-smoke workflow only built the `client-package-build` stage and
never booted the runtime, so it couldn't catch the class of regression that
recently took production down: the api tsdown bundle externalizes runtime
deps that, after `npm ci --omit=dev`, were missing from the image
(`Cannot find module 'get-stream'`).
- Add an `api-runtime-smoke` job that builds the real production image
(final `api-build` stage, `npm ci --omit=dev`), then:
1. loads the @librechat/api bundle's full require graph in the pruned
image (deterministic, no DB) — fails on any missing/ESM-incompatible
runtime dependency.
2. boots the actual entrypoint and asserts no module-load crash (the
server loads its require graph before connecting to Mongo, so this
surfaces without a database).
- Expand triggers to include `packages/api/**`, `packages/data-schemas/**`,
and `api/package.json` (previously a packages/api change only triggered
this via a root lockfile change, and even then only built the client stage).
- Add gha build cache + concurrency cancellation to bound CI cost.
* 👷 ci: Address Codex review — boot smoke against real Mongo + crash detection
- Boot the production image against a real MongoDB container with the env
the server needs, so the *entire* require graph loads. `api/db/connect.js`
throws at module scope without `MONGO_URI` and is imported before
models/services/routes, so the previous no-env boot exercised almost none
of the legacy API graph. (Codex finding 2)
- Gate on `/health` returning 200 AND the container staying alive, failing on
any container exit. A non-module startup crash (ReferenceError, SyntaxError,
bad config) now fails the smoke instead of slipping past a missing-module
grep. (Codex finding 3)
- Expand trigger from `api/package.json` to `api/**`, since the image copies
the whole `api/` tree and runs `node server/index.js`. (Codex finding 1)
* 👷 ci: Address Codex round 2 — poll /readyz + cover all image inputs
- Poll /readyz instead of /health. /health returns 200 at app.listen, but
initializeMCPs() and checkMigrations() run *after* listen and process.exit(1)
on failure; /readyz only returns 200 once serverReady is set after those
complete. So post-listen startup crashes now fail the smoke too. (finding A)
- Expand triggers to every source tree copied into the production image:
client/**, config/**, skill/** (the final stage copies client/dist, config,
and skill). (finding B)
