LibreChat/e2e
Marco Beretta 16faadb998
🪧 feat: Unify Artifact Triggers as Tool Rows (#15915)
* 🪧 feat: Unify Artifact Triggers as Tool Rows

Every chat-side artifact trigger — code-execution output, a model-authored
`:::artifact` block, and a mermaid diagram — now renders through one
`ArtifactRow` that sits on the tool-row axis instead of three card variants
in a wrapped chip strip. The row carries the file type twice: the language's
brand glyph where one exists, and a format badge for titles that are not
filenames. `kind.rendersPreview` tints the glyph and folds the same fact into
the accessible name, so the row says whether opening it yields a rendered view
or source to read.

The mermaid panel routes every download through its export menu — SVG, PNG and
the diagram source — rather than parking a second, unlabelled button beside it.
The menu turns the pressed format into its own loading row, defers the
synchronous SVG export by a macrotask so the spinner paints before the frame
blocks, and no longer disappears on the code tab. A mermaid download keeps its
real filename: its content *is* the stored `.mmd`, so it is not suffixed
`.preview`.

`useArtifactDownload` holds the original-file vs. edited-content decision that
both the generic download button and the mermaid menu need.

* 🩹 fix: Save the Stored Artifact File, Not Its Cached Text

The panel serialized a mermaid diagram from `artifact.content`, which is the
backend cache `extractUtf8` truncates at 512 KB, so "Download source" saved a
corrupt `.mmd` for any larger stored diagram. An unedited mermaid artifact now
takes the same original-file route as every other text-backed artifact,
`handleDownload` reports whether bytes were actually delivered, and the export
menu turns a failed delivery into its failure status and toast instead of
"Diagram download started.".

A `text/plain` row announced "Opens as source" while the panel opened it on the
rendered markdown preview; it now announces the preview. A pending preview
placeholder keeps the row slot it arrived in rather than sinking to the end of
the artifact group and jumping up once it resolves.

* 🧪 test: Exercise Artifact Rows and Mermaid Exports End to End

Seven acceptance scenarios in the mock harness cover what the unit suites
cannot: that a message which writes five artifacts shows five trigger rows on
one axis, that a row's accessible name matches what the panel actually does
with the artifact (rendered preview vs. source, including `text/plain`), that
the mermaid export menu offers SVG, PNG and the diagram source on both the
preview and the code tab, that "Download source" delivers the stored file and
announces a failure when the stored file is unreachable, and that a pending
preview placeholder keeps the row slot it arrived in.

Each test carries its `@scenario:<id>` tag and is self-contained: fixtures come
from the existing fake-model markers, Mongo seed helpers, or page-scoped route
stubs, and every seeded conversation is deleted again.

* 🏷️ fix: Name a Mermaid Blob After the Preview It Is

A share with snapshots disabled strips the download route from a
code-execution `.mmd` but keeps the cached `attachment.text`, and that text is
what `extractUtf8` truncated at 512 KB. With no route to the stored file the
panel can only serialize that cached copy, and the mermaid exception in
`getArtifactDownloadFilename` handed those partial bytes over as `flow.mmd` —
the stored file's own name, from a menu item labelled "Download source".

Mermaid now follows every other file-backed artifact: a serialized blob is named
`flow.preview.mmd`, and only the original-file route produces `flow.mmd`. A
model-authored diagram has no stored file to be a preview of, so it keeps its
own name.

* 🎹 test: Reach the Export Menu's First Item From Either Start

`supports keyboard export` opened the menu with Enter and pressed ArrowDown
once, assuming no item was active yet. CI observed the opposite — the first item
already active — so the single ArrowDown walked past SVG onto PNG and the test
failed there while passing locally (it failed the same way on the previous head).
`Home` lands on the first item from either starting point.

* 🔧 test: Fix the Fixtures Three Artifact Scenarios Ran On

The first verification run failed five of the seven scenarios on fixture defects,
not on product behavior:

- `artifact-trigger-rows` drove a fake-model marker
  (`E2E_ARTIFACT_SHOWCASE_REPLY`) that does not exist, so the reply carried no
  artifacts and no trigger rows. Both tests now seed the assistant message text
  through Mongo, the way the passing plain-text scenario does.
- `mermaid-source-download` waited for an `ArtifactRow` that a code-execution
  `.mmd` never renders — it renders an inline diagram whose trigger is "Open as
  artifact" — and its cached diagram embedded the literal truncation marker,
  which mermaid cannot parse, so the diagram failed to render at all. The cached
  copy is now a valid shorter diagram and the stored file keeps the sentinel that
  proves which bytes were saved.
- `pending-artifact-row-slot` gave the tool call's id to only one of its two
  attachments. Attachments reach a renderer through `mapAttachments` keyed by
  tool-call id, so the other one rendered nowhere; both now carry the same id and
  land in the one `artifact-row-group` whose ordering the scenario is about.

* 🪜 test: Count Four Authored Rows, Then the Mermaid One\n\nA mermaid fence is not a trigger row while it is inline: it renders the diagram\nwith an "Open as artifact" button and only becomes an ArtifactRow once opened.\nThe axis scenario now asserts the four authored artifacts stack as rows, then\nopens the diagram and asserts its row joins the same column.

* 📐 test: Measure the Row Axis in One Layout State\n\nOpening an artifact narrows the chat column, so comparing the mermaid row's x\nagainst a box measured before the panel opened reported an 8px offset that no\nuser sees. All five rows are now measured after the open, and the panel is\nlocated at page level, where it renders \u2014 not inside the message list.

* 🧩 fix: Give the Mermaid Row Its Own Filename and Download

A code-execution `.mmd` arrives wrapped in a header that prints the filename and
a download button, and that header stayed put when the diagram collapsed into
its trigger row — so the chat showed `flow.mmd` twice and left the download
outside the row that every other file-backed artifact keeps it in.

`Mermaid` now takes the file's download action and hands it to `ArtifactRow`,
and reports whether it is currently rendered as that row; `ToolMermaidArtifact`
drops its header while the row owns both. Opening and closing the diagram
therefore moves the filename and the download between exactly one place each.

* 🔖 test: Match a File-Backed Diagram's Own Trigger Id\n\nThe row ownership scenario looked for a \ trigger id, which\nonly a model-authored fence gets; a code-execution diagram's row carries the\nattachment's artifact id.

* 📱 test: Dismiss the Artifact Sheet Before Using the Row\n\nOn a phone the artifact panel opens as a sheet over the message list, so the\nrow's download button sits behind it. The scenario now closes the panel with its\nown control first, which also shows the trigger stays a row after closing.

* 🔔 fix: Raise One Toast per Failed Source Download\n\n\ already toasts "Error downloading file" before reporting a\nfailed delivery, and the export menu then raised its own — two notifications for\none press, on every ordinary 4xx/5xx, because React Query's refetch resolves\nwith an error result rather than throwing.\n\nA reported failure now only updates the menu's live region, which keeps the\nexport's own phrase for a screen reader; a thrown task still raises the toast,\nsince nothing else has spoken for it.

* 🔕 fix: Let the Download Layer Own Every Failure It Reports

The previous round split the failure notification by cause and left a hole in
the other direction: `useAttachmentLink` toasted only its empty-response
branch, so a thrown fetch — and the artifact hook's own blob failure — returned
`false` with nobody having spoken, and the export menu's live-region-only
announcement reached no sighted user.

One rule instead of two branches: whichever layer returns `false` has already
told the user. `useAttachmentLink` toasts its thrown branch as well, and
`useArtifactDownload` toasts when the blob path throws or there is nothing to
serialize. The mermaid export menu keeps announcing the export's own phrase in
its live region, so one press still raises exactly one toast — this time for
every failure, not just the polite ones.

* 🪄 fix: Hold the Row Slot and the Frame Before Blocking Work

Two fixes on the same theme — a row or a control that moves at exactly the
moment it is meant to stay put.

Writing salience-sorted resolved entries back into their arrival slots still
moved a pending row once it resolved: the placeholder joined the sort and could
pass an empty artifact that had been sitting above it. `attachmentSalience`
reads only `bytes`, which resolution does not change, so one sort over the whole
panel row — pending entries included — is stable across the transition and is
also less code.

The export deferral used `setTimeout(…, 0)`, which only moves the blocking SVG
work to another task; the timer can still run before the browser paints the
loading row, which is the whole point of deferring it. It now waits for an
animation frame and then a task, falling back to a timer when the tab is hidden
and serves no frames.

* 🏷️ fix: Keep the Recovered Filename on the Mermaid Row\n\n\ turns a sandbox-generated \ back into\n\, but that name lived only in the wrapper header. The moment the\ndiagram collapsed into its trigger row, the row title and its download button's\naccessible name fell back to \ — the internal filename.\n\n\ now takes the caller's display name for the row, and\n\ passes the one it already computed.

* 🗂️ fix: Open Each Artifact on the Tab Its Row Promises

A code-only artifact forces the panel onto Code, and that constraint was written
back into the panel's single `activeTab`. The next HTML, Markdown, plain-text,
React or diagram artifact then opened on Code while its trigger row announced
"Opens as a rendered preview" — the row was telling the truth about the artifact
and a lie about the click.

The constrained tab no longer leaks: the panel resets to the preview tab when a
different, unconstrained artifact is opened, so a manual Code choice lasts as
long as that artifact stays open and never follows the user to the next one.
`displayedTab` already derived the forced tab for rendering, so nothing else
depended on the write-back.
2026-09-14 15:08:07 -04:00
..
benchmarks ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
benchmarks-mobile-chat 📴 fix: Stop Hidden Code Spinners (#15846) 2026-09-11 21:37:49 -04:00
benchmarks-navigation 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
benchmarks-reasoning 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
benchmarks-tree 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
bombadil 🤖 feat: make Event Actor HITL durable (#15305) 2026-08-28 08:05:11 -04:00
byom 🛫 feat: Enable Full Access for BYOM Coding (#15718) 2026-09-07 22:46:01 -04:00
config 🦘 feat: Share Run-Scoped Files With Subagents (#15848) 2026-09-12 20:14:22 -04:00
fixtures 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210) 2026-08-25 18:29:55 -04:00
lighthouse 💠 fix: Paint Native Tool Verification Badges (#15862) 2026-09-12 16:03:03 -04:00
perf 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
recordings 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
setup 🧷 fix: Keep a Failed Summary From Bounding the Conversation History (#15913) 2026-09-14 10:22:47 -04:00
specs 🪧 feat: Unify Artifact Triggers as Tool Rows (#15915) 2026-09-14 15:08:07 -04:00
client-build.test.mjs 🔭 feat: Tag Browser Diagnostics With Client Build IDs (#15656) 2026-09-06 12:47:34 -04:00
config.local.example.ts
jestSetup.js 🚀 v0.8.8-rc3 (#15859) 2026-09-13 02:17:48 -04:00
playwright.config.a11y.ts
playwright.config.benchmark.ts ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
playwright.config.bombadil.ts 🧪 test(e2e): add Bombadil property exploration (#14462) 2026-08-10 02:11:06 +02:00
playwright.config.lighthouse.ts 🎈 ci: Gate Page Loads Under Database Latency (#15646) 2026-09-06 12:01:44 -04:00
playwright.config.local.ts
playwright.config.mermaid.ts 🧜 feat: Open Mermaid Diagrams as Artifacts with SVG/PNG Export (#14713) 2026-08-09 09:02:42 -04:00
playwright.config.mobile-chat-perf.ts 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
playwright.config.mock.ts 📎 test: Verify Stateful Sandbox Upload Continuity (#14363) 2026-09-09 00:35:28 -04:00
playwright.config.navigation-perf.ts 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
playwright.config.real.ts 🎯 feat: Tool Intent Label Capability (tool_intents) (#14499) 2026-07-29 15:40:52 -04:00
playwright.config.reasoning-perf.ts 🧪 test: Reasoning-Stream Render Perf Benchmark via react-scan (#14494) 2026-07-28 22:18:24 -04:00
playwright.config.redis.ts 🎬 test: Cover Detached Subagent Activity Lifecycle (#15117) 2026-08-23 01:16:25 -04:00
playwright.config.tree-parity.ts 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
playwright.config.tree-perf-prod.ts 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
playwright.config.tree-perf.ts 🧶 feat: Flatten Active Message Threads Behind a Flag (#15633) 2026-09-06 14:30:50 -04:00
playwright.config.ts 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
README.md 🏎️ ci: Focus Redis E2E Coverage (#14842) 2026-08-14 12:54:30 -04:00
types.ts

LibreChat e2e

The mock e2e profile is the safest default for generated tests. It starts LibreChat with e2e/config/librechat.e2e.yaml, injects an in-process fake LLM (via LIBRECHAT_TEST_RUN_HOOK), creates an authenticated e2e user, and avoids real provider credentials.

Stream Stores and Shards

The mock profile uses the in-memory generation stream store by default. To exercise the same browser scenarios through a real Redis job store and pub/sub transport, start Redis on port 6379 and run:

npm run e2e:mock:redis

Memory mode explicitly disables Redis. Redis mode defaults to database 15 with a LibreChatE2E key prefix, and fails closed: the test server pings Redis and verifies that the generation job manager did not silently fall back to memory. Override REDIS_URI or E2E_REDIS_KEY_PREFIX when needed.

Pull request CI runs the complete mock suite in memory mode across three shards, plus a focused Redis transport suite. The Redis suite covers streaming fidelity, steering, interrupts, resumptions, HITL approvals, completion, thread folding, model icons, and usage:

npx playwright test --config=e2e/playwright.config.mock.ts --shard=1/3
npm run e2e:mock:redis:transport

The nightly schedule and manual workflow dispatch run the complete mock suite in both stream modes across two shards per mode. Every shard keeps one worker so tests do not contend for its authenticated user and database.

Property-based browser testing

Bombadil explores randomized sequences across the core chat loop, message branches, parallel multi-conversation responses, model changes, reloads, and sidebar conversation lifecycle operations:

npm run e2e:bombadil

Set BOMBADIL_TIME_LIMIT for longer local or scheduled runs. Failures leave a reproducible trace under e2e/.generated/bombadil-output; rerun it with:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

Reproducing a real violation is expected to fail the Playwright test. Before a new run overwrites the active output, the harness archives it under e2e/.generated/bombadil-history/. Reproduction can diverge when streaming timing changes; Bombadil reports that explicitly.

The harness uses the credential-free mock-LLM profile, so exploration never sends billable provider requests.

CI runs the broad property exploration for five minutes in the non-blocking Bombadil Property Exploration workflow. If a property fails, download the bombadil-reproduction-* artifact into e2e/.generated/bombadil-output/, then reproduce it locally:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

The accompanying bombadil-diagnostics-* artifact contains the captured CI log, Playwright HTML report, and Playwright test results. A Bombadil failure produces a workflow warning but does not block merge.

The default instruments inline JavaScript only because instrumenting LibreChat's full Vite bundle can exceed Bombadil's driver timeout during stateful runs. Set BOMBADIL_INSTRUMENT_JAVASCRIPT=files,inline for shorter coverage-guided experiments.

The branch reload, fork submission, model/conversation, HITL pause/resume, and mid-run steering lifecycle properties can be run independently:

npm run e2e:bombadil:branch-reload
npm run e2e:bombadil:fork-lifecycle
npm run e2e:bombadil:model-lifecycle
npm run e2e:bombadil:hitl
npm run e2e:bombadil:steering

These focused commands are diagnostic properties: they exit nonzero when they reproduce a product invariant violation. Reproduce a focused trace with its matching :run script and output directory, for example:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output-hitl npm run e2e:bombadil:hitl:run

HITL drives a real ask_user_question checkpoint through the answer/resume controller, reloads while the question is paused, answers it once, and reloads the completed conversation. Steering submits an in-flight steer during a slow MCP-backed run, checks that it moves exactly once from the composer anchor into the response at the tool boundary, and reloads the applied state. The model lifecycle property is the passing control. The branch reload and fork properties preserve their minimal failing traces.

Recording Tests

Use Playwright codegen when you want to turn an exploratory browser session into a draft test:

npm run e2e:record

That command builds the app, starts the LibreChat test server (with an in-process fake LLM) when needed, writes e2e/storageState.json, and opens Playwright codegen at /c/new. The npm script uses http://localhost:3333 so it does not collide with a normal dev server on 3080. Raw recordings are written to e2e/recordings/ and ignored by git.

For a real local LibreChat config instead of the mock profile:

npm run e2e:record:local

Useful direct options:

node e2e/setup/record.js --url=http://localhost:3080/c/new
node e2e/setup/record.js --profile=local --no-output
node e2e/setup/record.js --auth-only
node e2e/setup/record.js --output=e2e/recordings/settings-draft.spec.ts

LLM-Assisted Loop

  1. Start npm run e2e:record.
  2. Let the LLM use Computer Use to operate the headed Playwright browser.
  3. Stop codegen after the workflow is captured.
  4. Move the useful parts from e2e/recordings/ into a committed spec under e2e/specs/mock/.
  5. Replace brittle generated selectors with role, label, text, or data-testid locators.
  6. Add assertions that prove the behavior, not just the clicked path.
  7. Run the finished spec with npm run e2e:mock -- <spec name>.

Generated recordings are a draft, not the final test. The committed version should use the shared helpers in e2e/specs/mock/helpers.ts where possible, wait on network or visible UI state instead of fixed sleeps, and keep test data deterministic.