LibreChat/e2e
Marco Beretta 246e7e5de7
💠 fix: Paint Native Tool Verification Badges (#15862)
* 🪪 style: Give Native Tools a Painted Verified Mark

The native badge on a tool card was lucide's BadgeCheck filled with status-success-strong: green, the same hue the card already uses for selected, and stroked in currentColor, so the white check color also outlined the badge and left a white halo on any surface that is not the card's own white — hover, selection, dark mode.\n\nVerifiedIcon paints the badge instead of stroking it and carries the check as the only stroked path, and status-verified gives provenance its own blue across all four themes.

* 🩹 fix: Hold the Verified Mark at 3:1 on Every Card State

The dark fill was tuned against surface-secondary, but ToolCard repaints to surface-tertiary (#2f2f2f) on hover, where #0b74d4 fell to 2.85:1 — below the non-text floor, and invisible to the spec, which asserted the panel only. The fill moves to #1a7fd8 and the contract now covers every background the card takes. Both of the mark's relationships are graphical objects under WCAG 1.4.11, so the check is held at 3:1 rather than AA: no hue clears 3:1 on the hover surface and 4.5:1 under a white check at once.\n\nA theme that predates the token names status-success-strong and not status-verified, and would otherwise wear LibreChat's stock blue beside its own text-on-status and card surfaces. resolveTheme and the legacy mapColors adapter now carry the mark onto the fill it had before the token existed, following the six derived-token fallbacks already there.

* ✅ test: Exercise the Verified Mark in the Tool Library

Four scenarios in the mock harness, each named by the behavior it observes: the mark a native card wears, its silhouette under the dark hover repaint that the retune fixes, a pre-token deployment palette carrying it on the theme's own success fill, and a third-party MCP server card carrying no mark at all. The contrast assertions read the rendered colors rather than the tokens, so a regression in the component or in theme application fails them.

* 🧪 fix: Teach the Marketplace Dialog Spec About VerifiedIcon

The suite hand-lists the @librechat/client surface it mocks, so ToolCard's new icon resolved to undefined and every render in the file threw.

* 🧪 fix: Reach the Agent Builder on a Narrow Viewport

The scenario run covers mobile emulation, where the side rail is collapsed and the panel opens from the header's Control Panel toggle; the helper only knew the rail. The MCP scenario also minted its access token before the app had loaded, so the refresh request had no origin to resolve against.

* 🧪 fix: Run the Verified-Mark Scenarios on a Touch Viewport

Mobile emulation has no side rail: the panel switcher lives inside a sidebar that starts off-canvas, so the helper opens the sidebar and picks the Agent Builder entry from the switcher's menu. A touch viewport also has no hover state, so the hover scenario measures whatever background the card actually takes there and still holds the mark to 3:1 against it.

* 🩹 fix: Keep Pre-Token Themes on the Fill the Mark Inherited

The first fallback only fired when a theme restated status-success-strong, so a partial palette that customizes surfaces or text-on-status and inherits the green still landed on the stock blue. resolveTheme now derives the omission for any custom theme from the resolved success fill, and the legacy adapter takes the mode's bundled palette from ThemeProvider so it can do the same for a theme that names neither.\n\nThe scenario helper picked its path from a one-second probe for the desktop rail, which turns a slow first paint into the mobile branch; it now decides on viewport width and keeps the suite's full assertion timeout in each branch. The mock project is Desktop Chrome, so a touch scenario asks for a touch context explicitly rather than leaving that layout unexercised in CI.

* 🧪 fix: Match the App's Inclusive Layout Breakpoint

useSidebarState switches on (max-width: 768px), so a 768px-wide context renders the sidebar switcher while the helper's strict comparison went looking for the desktop rail.

* 🩹 fix: Scope the Verified Fallback to Themes That Painted the Mark

Two rounds pulled the same predicate in opposite directions: keying it on a theme restating status-success-strong missed palettes that only repaint the check or the card, and keying it on any custom color at all gave a new partial theme a green mark identical to the selected cue. Both are answered by asking what the theme actually painted — the fill the mark wore, the text-on-status check it carries, or the surfaces ToolCard takes — which is the same shape as the series-scale fallback above it. A theme that repaints none of those keeps the bundled blue.

* 🧪 fix: Wait for the Shell Before Reading the Sidebar State

isVisible() answers immediately, so a slow first paint read as an already-open drawer and the helper then waited on a switcher sealed inside it. It now waits for whichever of the opener and the switcher the shell paints first, which is the same assertion timeout the desktop branch gets.

* 📝 fix: Keep the Token Catalog Entry Catalog-Sized

The entry restated the icon's construction, the contrast rule and the pre-token fallback, all of which live on the token's own doc comment and in semanticTokens.spec.ts; the catalog around it is one line per role.

* 🔦 fix: Let a Crashed Lighthouse Run Be Retried

The audit shells out to Lighthouse three times and fails the lane if any single process exits nonzero, so a NO_NAVSTART — Lighthouse losing the trace of a page load, whose own message is "run Lighthouse again" — sinks a pull request that has nothing wrong with it. That is what happened on this branch: runs 1 and 2 wrote reports, run 3 crashed, and the identical head passed on rerun.\n\nEach run now gets a second attempt before the failure propagates. The repeat is per run, so the median is still taken over three completed reports, and a page that genuinely cannot be traced fails both attempts and still fails the lane.

* 🧪 fix: Read the Drawer State From Its Inert Marker

The readiness wait accepted the panel switcher, which is mounted whether or not the drawer is open — translated off-canvas and inert — so a shell that painted the sidebar before the chat header could satisfy it with the drawer closed and leave the helper clicking an off-viewport control. The drawer's own inert attribute is the state, so the helper waits for the drawer to attach and reads it there, with no timing probe on either side.

* 🧪 fix: Open the Drawer Before Trusting the Restored Form

A narrow context that had already selected the builder restores the form inside the closed drawer, and a translated element passes a visibility check, so the early probe accepted a form sitting under an inert drawer and handed it to the caller. The layout decision now comes first and the drawer is opened before anything inside it is read.
2026-09-12 16:03:03 -04:00
..
benchmarks ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
benchmarks-mobile-chat 📴 fix: Stop Hidden Code Spinners (#15846) 2026-09-11 21:37:49 -04:00
benchmarks-navigation 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
benchmarks-reasoning 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
benchmarks-tree 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
bombadil 🤖 feat: make Event Actor HITL durable (#15305) 2026-08-28 08:05:11 -04:00
byom 🛫 feat: Enable Full Access for BYOM Coding (#15718) 2026-09-07 22:46:01 -04:00
config 📎 test: Verify Stateful Sandbox Upload Continuity (#14363) 2026-09-09 00:35:28 -04:00
fixtures 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210) 2026-08-25 18:29:55 -04:00
lighthouse 💠 fix: Paint Native Tool Verification Badges (#15862) 2026-09-12 16:03:03 -04:00
perf 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
recordings 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
setup 📎 test: Verify Stateful Sandbox Upload Continuity (#14363) 2026-09-09 00:35:28 -04:00
specs 💠 fix: Paint Native Tool Verification Badges (#15862) 2026-09-12 16:03:03 -04:00
client-build.test.mjs 🔭 feat: Tag Browser Diagnostics With Client Build IDs (#15656) 2026-09-06 12:47:34 -04:00
config.local.example.ts
jestSetup.js ✨ v0.8.8-rc2 (#14845) 2026-09-02 22:18:48 -04:00
playwright.config.a11y.ts
playwright.config.benchmark.ts ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
playwright.config.bombadil.ts 🧪 test(e2e): add Bombadil property exploration (#14462) 2026-08-10 02:11:06 +02:00
playwright.config.lighthouse.ts 🎈 ci: Gate Page Loads Under Database Latency (#15646) 2026-09-06 12:01:44 -04:00
playwright.config.local.ts
playwright.config.mermaid.ts 🧜 feat: Open Mermaid Diagrams as Artifacts with SVG/PNG Export (#14713) 2026-08-09 09:02:42 -04:00
playwright.config.mobile-chat-perf.ts 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
playwright.config.mock.ts 📎 test: Verify Stateful Sandbox Upload Continuity (#14363) 2026-09-09 00:35:28 -04:00
playwright.config.navigation-perf.ts 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
playwright.config.real.ts 🎯 feat: Tool Intent Label Capability (tool_intents) (#14499) 2026-07-29 15:40:52 -04:00
playwright.config.reasoning-perf.ts 🧪 test: Reasoning-Stream Render Perf Benchmark via react-scan (#14494) 2026-07-28 22:18:24 -04:00
playwright.config.redis.ts 🎬 test: Cover Detached Subagent Activity Lifecycle (#15117) 2026-08-23 01:16:25 -04:00
playwright.config.tree-parity.ts 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
playwright.config.tree-perf-prod.ts 🎖️ perf: Promote Flat Thread Rendering to Default (#15660) 2026-09-06 18:31:08 +00:00
playwright.config.tree-perf.ts 🧶 feat: Flatten Active Message Threads Behind a Flag (#15633) 2026-09-06 14:30:50 -04:00
playwright.config.ts 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
README.md 🏎️ ci: Focus Redis E2E Coverage (#14842) 2026-08-14 12:54:30 -04:00
types.ts

LibreChat e2e

The mock e2e profile is the safest default for generated tests. It starts LibreChat with e2e/config/librechat.e2e.yaml, injects an in-process fake LLM (via LIBRECHAT_TEST_RUN_HOOK), creates an authenticated e2e user, and avoids real provider credentials.

Stream Stores and Shards

The mock profile uses the in-memory generation stream store by default. To exercise the same browser scenarios through a real Redis job store and pub/sub transport, start Redis on port 6379 and run:

npm run e2e:mock:redis

Memory mode explicitly disables Redis. Redis mode defaults to database 15 with a LibreChatE2E key prefix, and fails closed: the test server pings Redis and verifies that the generation job manager did not silently fall back to memory. Override REDIS_URI or E2E_REDIS_KEY_PREFIX when needed.

Pull request CI runs the complete mock suite in memory mode across three shards, plus a focused Redis transport suite. The Redis suite covers streaming fidelity, steering, interrupts, resumptions, HITL approvals, completion, thread folding, model icons, and usage:

npx playwright test --config=e2e/playwright.config.mock.ts --shard=1/3
npm run e2e:mock:redis:transport

The nightly schedule and manual workflow dispatch run the complete mock suite in both stream modes across two shards per mode. Every shard keeps one worker so tests do not contend for its authenticated user and database.

Property-based browser testing

Bombadil explores randomized sequences across the core chat loop, message branches, parallel multi-conversation responses, model changes, reloads, and sidebar conversation lifecycle operations:

npm run e2e:bombadil

Set BOMBADIL_TIME_LIMIT for longer local or scheduled runs. Failures leave a reproducible trace under e2e/.generated/bombadil-output; rerun it with:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

Reproducing a real violation is expected to fail the Playwright test. Before a new run overwrites the active output, the harness archives it under e2e/.generated/bombadil-history/. Reproduction can diverge when streaming timing changes; Bombadil reports that explicitly.

The harness uses the credential-free mock-LLM profile, so exploration never sends billable provider requests.

CI runs the broad property exploration for five minutes in the non-blocking Bombadil Property Exploration workflow. If a property fails, download the bombadil-reproduction-* artifact into e2e/.generated/bombadil-output/, then reproduce it locally:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

The accompanying bombadil-diagnostics-* artifact contains the captured CI log, Playwright HTML report, and Playwright test results. A Bombadil failure produces a workflow warning but does not block merge.

The default instruments inline JavaScript only because instrumenting LibreChat's full Vite bundle can exceed Bombadil's driver timeout during stateful runs. Set BOMBADIL_INSTRUMENT_JAVASCRIPT=files,inline for shorter coverage-guided experiments.

The branch reload, fork submission, model/conversation, HITL pause/resume, and mid-run steering lifecycle properties can be run independently:

npm run e2e:bombadil:branch-reload
npm run e2e:bombadil:fork-lifecycle
npm run e2e:bombadil:model-lifecycle
npm run e2e:bombadil:hitl
npm run e2e:bombadil:steering

These focused commands are diagnostic properties: they exit nonzero when they reproduce a product invariant violation. Reproduce a focused trace with its matching :run script and output directory, for example:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output-hitl npm run e2e:bombadil:hitl:run

HITL drives a real ask_user_question checkpoint through the answer/resume controller, reloads while the question is paused, answers it once, and reloads the completed conversation. Steering submits an in-flight steer during a slow MCP-backed run, checks that it moves exactly once from the composer anchor into the response at the tool boundary, and reloads the applied state. The model lifecycle property is the passing control. The branch reload and fork properties preserve their minimal failing traces.

Recording Tests

Use Playwright codegen when you want to turn an exploratory browser session into a draft test:

npm run e2e:record

That command builds the app, starts the LibreChat test server (with an in-process fake LLM) when needed, writes e2e/storageState.json, and opens Playwright codegen at /c/new. The npm script uses http://localhost:3333 so it does not collide with a normal dev server on 3080. Raw recordings are written to e2e/recordings/ and ignored by git.

For a real local LibreChat config instead of the mock profile:

npm run e2e:record:local

Useful direct options:

node e2e/setup/record.js --url=http://localhost:3080/c/new
node e2e/setup/record.js --profile=local --no-output
node e2e/setup/record.js --auth-only
node e2e/setup/record.js --output=e2e/recordings/settings-draft.spec.ts

LLM-Assisted Loop

  1. Start npm run e2e:record.
  2. Let the LLM use Computer Use to operate the headed Playwright browser.
  3. Stop codegen after the workflow is captured.
  4. Move the useful parts from e2e/recordings/ into a committed spec under e2e/specs/mock/.
  5. Replace brittle generated selectors with role, label, text, or data-testid locators.
  6. Add assertions that prove the behavior, not just the clicked path.
  7. Run the finished spec with npm run e2e:mock -- <spec name>.

Generated recordings are a draft, not the final test. The committed version should use the shared helpers in e2e/specs/mock/helpers.ts where possible, wait on network or visible UI state instead of fixed sleeps, and keep test data deterministic.