diff --git a/e2e/benchmarks-mobile-chat/README.md b/e2e/benchmarks-mobile-chat/README.md new file mode 100644 index 0000000000..f1ad245840 --- /dev/null +++ b/e2e/benchmarks-mobile-chat/README.md @@ -0,0 +1,43 @@ +# Mobile Chat Performance Benchmark + +Measures long-conversation behavior with Playwright's iPhone 13 descriptor: a +`390x664` browser viewport on a `390x844` screen. It uses a local production +build and the mock-model pipeline; no provider credentials or external model +calls are used. + +The benchmark seeds 150 user/assistant turns (300 rendered message rows), then +captures six phases: + +- sitting idle on an empty chat as a control; +- loading and progressively mounting the long transcript; +- sitting idle for three seconds after the transcript settles; +- repeated full-history scrolling; +- three streamed continuation turns with long Markdown responses; +- typing after the transcript grows to 306 rows. + +Each scenario records Chromium `Performance` domain counters (main-thread task, +script, layout and style time), long tasks, heap size, and DOM node count. It +runs twice: first without react-scan for lower-overhead browser measurements, +then with react-scan to attribute render counts and duration to components. + +This is a desktop-hosted Chromium measurement at an iPhone viewport, not a +physical-device battery or thermal measurement. Use it to reproduce and compare +frontend changes locally; confirm energy impact separately on a real iPhone. + +## Run + +Extract the pinned instrumentation bundle outside the repository so npm cannot +re-resolve LibreChat's dependencies, prepare the production client, then run +the benchmark: + +```bash +mkdir -p /tmp/librechat-react-scan +npm pack react-scan@0.5.7 --pack-destination /tmp/librechat-react-scan +tar -xzf /tmp/librechat-react-scan/react-scan-0.5.7.tgz -C /tmp/librechat-react-scan +npm run e2e:prepare +REACT_SCAN_PATH=/tmp/librechat-react-scan/package/dist/auto.global.js \ + npx playwright test --config=e2e/playwright.config.mobile-chat-perf.ts +``` + +JSON snapshots are attached to the Playwright results under +`e2e/benchmarks/.test-results/mobile-chat`. diff --git a/e2e/benchmarks-mobile-chat/RESULTS.md b/e2e/benchmarks-mobile-chat/RESULTS.md new file mode 100644 index 0000000000..47f88b0f7e --- /dev/null +++ b/e2e/benchmarks-mobile-chat/RESULTS.md @@ -0,0 +1,56 @@ +# Mobile Chat Performance Baseline + +Captured September 2, 2026 from LibreChat `519e64769` on an Apple M4 Max with +macOS 26.6, Node.js 24.16.0, Playwright 1.62.1, and Chromium at the iPhone 13 +descriptor's `390x664` viewport on a `390x844` screen. The production client +was used. + +The stress conversation contained 150 user/assistant turns (300 rows) with +prose, lists, code blocks, and tables. The continuation phase streamed three +additional rich Markdown replies. `busy` is Chromium main-thread task time +divided by elapsed wall time; it is a useful comparison proxy, not device CPU +utilization. + +## Raw browser measurements + +| Phase | Wall | Main-thread busy | Task | Script | Style | Long tasks (total/worst) | Heap at end | DOM nodes | +| ----------------------- | --------: | ---------------: | --------: | -------: | -----: | -----------------------: | ----------: | --------: | +| Empty chat idle | 3,007 ms | 15.9% | 479 ms | 273 ms | 64 ms | 102/102 ms | 52.6 MB | 2,063 | +| Load 300 rows | 2,909 ms | 84.4% | 2,455 ms | 1,717 ms | 119 ms | 347/250 ms | 271.2 MB | 121,371 | +| Long chat idle | 3,030 ms | 14.5% | 439 ms | 5 ms | 132 ms | 0/0 ms | 349.5 MB | 123,427 | +| Four full scroll cycles | 3,787 ms | 90.9% | 3,441 ms | 1,107 ms | 343 ms | 0/0 ms | 348.4 MB | 123,587 | +| Three continuations | 11,182 ms | 95.2% | 10,646 ms | 4,208 ms | 872 ms | 2,791/325 ms | 511.6 MB | 126,234 | +| Typing after stress | 1,357 ms | 71.1% | 965 ms | 206 ms | 145 ms | 0/0 ms | 541.7 MB | 127,013 | + +Heap figures are point-in-time values without forced garbage collection, so +they do not establish a memory leak. + +## React Scan diagnostic + +React Scan observed 44,521 render records while loading, 2,852 while scrolling, +201,013 during the three streamed continuations, and 1,334 while typing. The +settled long conversation produced only 26 render records during its +three-second idle phase. This points away from a continuous React rerender loop +as the source of settled-idle work, but shows a very large amount of React work +during streaming. + +React Scan materially perturbed the workload. During continuation, Chromium +script time increased from 4,208 ms to 12,332 ms and task time increased from +10,646 ms to 14,991 ms. Its instrumented production-build timings were reported +as zero and many component names were minified. Therefore the raw run is the +performance baseline; React Scan is useful here only for comparative render +counts and for narrowing a follow-up development-build profile. + +## Interpretation + +This run supports the report that long LibreChat conversations can sustain high +frontend work while loading, scrolling, and streaming. It does not prove that +ordinary LibreChat use consumes more energy than other apps, and it cannot +measure iPhone thermals or Mobile Safari behavior. + +The largest scaling signal is the document size: after progressive mounting +settles, all 300 rows remain mounted at roughly 120,000 DOM nodes. The current +progressive-row strategy optimizes the first commit but deliberately converges +to the complete DOM. A real-device Safari energy trace and a controlled A/B +against a bounded/virtualized message DOM are the next tests needed to connect +this browser result to phone heating. diff --git a/e2e/benchmarks-mobile-chat/mobile-chat.perf.spec.ts b/e2e/benchmarks-mobile-chat/mobile-chat.perf.spec.ts new file mode 100644 index 0000000000..8ec07f2bad --- /dev/null +++ b/e2e/benchmarks-mobile-chat/mobile-chat.perf.spec.ts @@ -0,0 +1,222 @@ +import { randomUUID } from 'node:crypto'; +import { expect, test } from '@playwright/test'; +import type { Page, TestInfo } from '@playwright/test'; +import { + clearUserConversations, + deleteConversations, + deleteMessagesByConversation, + seedConversations, + seedMessages, +} from '../specs/mock/db'; +import { + MOCK_ENDPOINTS, + NEW_CHAT_PATH, + messagesView, + selectMockEndpoint, + sendMessage, +} from '../specs/mock/helpers'; +import { getE2EUser } from '../setup/user'; +import { + attachBrowserPhases, + createBrowserProbe, + formatBrowserPhase, + installBrowserPerf, +} from '../perf/browser'; +import type { BrowserPhase } from '../perf/browser'; +import { + attachSnapshot, + installReactScan, + resetPerf, + snapshotPerf, + topComponents, + totals, +} from '../perf/scan'; +import type { PerfSnapshot } from '../perf/scan'; +import { buildStressMessages, ROWS, STREAM_END_MARKER, TURNS } from './payload'; + +const userEmail = getE2EUser().email; +const FIXTURES = [ + { id: randomUUID(), label: 'raw', title: 'Mobile performance raw metrics' }, + { id: randomUUID(), label: 'scan', title: 'Mobile performance react scan' }, +] as const; +const CONTINUATIONS = 3; + +interface ScenarioResults { + browser: Record; + scan: Record; +} + +function messageRows(page: Page) { + return messagesView(page).locator('.message-render'); +} + +async function stressScroll(page: Page): Promise { + const scroller = messagesView(page).locator('.scrollbar-gutter-stable'); + await scroller.evaluate(async (element) => { + const scrollTo = (target: number) => + new Promise((resolve) => { + const startedAt = performance.now(); + const from = element.scrollTop; + const duration = 450; + const step = (now: number) => { + const progress = Math.min(1, (now - startedAt) / duration); + element.scrollTop = from + (target - from) * progress; + if (progress < 1) { + requestAnimationFrame(step); + return; + } + resolve(); + }; + requestAnimationFrame(step); + }); + for (let pass = 0; pass < 4; pass += 1) { + await scrollTo(0); + await scrollTo(element.scrollHeight); + } + }); +} + +async function runPhase( + name: string, + page: Page, + results: ScenarioResults, + reactScan: boolean, + action: () => Promise, +): Promise { + const probe = await createBrowserProbe(page); + if (reactScan) { + await resetPerf(page); + } + await probe.start(); + await action(); + results.browser[name] = await probe.finish(); + if (reactScan) { + results.scan[name] = await snapshotPerf(page); + } +} + +async function runScenario( + page: Page, + testInfo: TestInfo, + fixture: (typeof FIXTURES)[number], + reactScan: boolean, +): Promise { + await installBrowserPerf(page); + if (reactScan) { + await installReactScan(page); + } + + await page.goto(NEW_CHAT_PATH, { timeout: 120_000 }); + const results: ScenarioResults = { browser: {}, scan: {} }; + + await runPhase('idle-empty', page, results, reactScan, async () => { + await page.waitForTimeout(3_000); + }); + + await runPhase('load', page, results, reactScan, async () => { + await page.goto(`/c/${fixture.id}`, { timeout: 120_000 }); + await expect(messageRows(page)).toHaveCount(ROWS, { timeout: 120_000 }); + }); + + await selectMockEndpoint(page, MOCK_ENDPOINTS[0]); + + await runPhase('idle-long', page, results, reactScan, async () => { + await page.waitForTimeout(3_000); + }); + + await runPhase('scroll', page, results, reactScan, async () => { + await stressScroll(page); + }); + + await runPhase('continue', page, results, reactScan, async () => { + for (let turn = 1; turn <= CONTINUATIONS; turn += 1) { + await sendMessage(page, `Continue mobile stress turn ${turn}`); + await expect(messagesView(page).getByText(STREAM_END_MARKER, { exact: true })).toHaveCount( + turn, + { timeout: 90_000 }, + ); + await expect(page.getByRole('button', { name: 'Stop generating' })).toBeHidden({ + timeout: 30_000, + }); + } + }); + await expect(messageRows(page)).toHaveCount(ROWS + CONTINUATIONS * 2, { timeout: 30_000 }); + + await runPhase('typing', page, results, reactScan, async () => { + const input = page.getByRole('textbox', { name: 'Message input' }); + await input.click(); + await input.pressSequentially( + 'Typing after a three-hundred-message transcript should keep the transcript quiet.', + { delay: 10 }, + ); + }); + + console.log( + `\n=== Mobile chat stress: ${reactScan ? 'react-scan diagnostic' : 'raw browser metrics'} ` + + `(390x664 viewport, 390x844 screen, ${TURNS} seeded turns / ${ROWS} rows, ` + + `${CONTINUATIONS} continuations) ===`, + ); + for (const [name, phase] of Object.entries(results.browser)) { + console.log(formatBrowserPhase(name, phase)); + } + if (reactScan) { + let capturedRenders = 0; + for (const [name, snapshot] of Object.entries(results.scan)) { + const phaseTotals = totals(snapshot); + capturedRenders += phaseTotals.renders; + console.log( + `${name} react renders=${phaseTotals.renders} render-time=${phaseTotals.time.toFixed(0)}ms`, + ); + for (const line of topComponents(snapshot, 12)) { + console.log(` ${line}`); + } + await attachSnapshot(testInfo, `${name}-react-scan.json`, snapshot, { + seededRows: ROWS, + continuations: CONTINUATIONS, + }); + } + expect(capturedRenders).toBeGreaterThan(0); + } + await attachBrowserPhases(testInfo, 'browser-metrics.json', results.browser); + + for (const phase of Object.values(results.browser)) { + expect(phase.elapsedMs).toBeGreaterThan(0); + expect(phase.taskMs).toBeGreaterThanOrEqual(0); + expect(Number.isFinite(phase.busyPercent)).toBe(true); + } +} + +test.describe.configure({ mode: 'serial' }); + +test.describe('iPhone-sized long-chat performance', () => { + test.beforeAll(async () => { + await clearUserConversations(userEmail); + await seedConversations( + userEmail, + FIXTURES.map((fixture, index) => ({ + conversationId: fixture.id, + title: fixture.title, + updatedAt: new Date(Date.now() - index * 60_000), + })), + ); + for (const fixture of FIXTURES) { + await seedMessages(userEmail, fixture.id, buildStressMessages(fixture.label)); + } + }); + + test.afterAll(async () => { + const ids = FIXTURES.map((fixture) => fixture.id); + await deleteMessagesByConversation(ids); + await deleteConversations(ids); + }); + + test('captures browser CPU proxies without react-scan overhead', async ({ page }, testInfo) => { + test.setTimeout(10 * 60 * 1000); + await runScenario(page, testInfo, FIXTURES[0], false); + }); + + test('attributes renders with react-scan enabled', async ({ page }, testInfo) => { + test.setTimeout(10 * 60 * 1000); + await runScenario(page, testInfo, FIXTURES[1], true); + }); +}); diff --git a/e2e/benchmarks-mobile-chat/payload.ts b/e2e/benchmarks-mobile-chat/payload.ts new file mode 100644 index 0000000000..883f7258f6 --- /dev/null +++ b/e2e/benchmarks-mobile-chat/payload.ts @@ -0,0 +1,62 @@ +import type { SeedMessage } from '../specs/mock/db'; + +export const TURNS = 150; +export const ROWS = TURNS * 2; +export const STREAM_END_MARKER = 'MOBILE-PERF-STREAM-END'; + +const PROSE = + 'This mobile performance transcript deliberately combines prose, lists, tables, and code so each mounted message exercises realistic chat rendering. '; + +function assistantBody(turn: number): string { + let body = `## Mobile stress turn ${turn}\n\n${PROSE}${PROSE}\n\n`; + body += `- First observation for turn ${turn}\n`; + body += `- Second observation for turn ${turn}\n\n`; + if (turn % 5 === 0) { + body += + '```ts\nexport function mobileSample(value: number): number {\n return value * 2;\n}\n```\n\n'; + } + if (turn % 7 === 0) { + body += + '| Metric | Value |\n| --- | ---: |\n' + `| messages | ${turn * 2} |\n| turn | ${turn} |\n\n`; + } + return body; +} + +export function buildStressMessages(label: string): SeedMessage[] { + const messages: SeedMessage[] = []; + let parentMessageId = '00000000-0000-0000-0000-000000000000'; + for (let turn = 1; turn <= TURNS; turn += 1) { + const userMessageId = `${label}-user-${turn}`; + messages.push({ + messageId: userMessageId, + parentMessageId, + text: `Mobile stress prompt ${turn}: summarize the measurements for this turn.`, + isCreatedByUser: true, + sender: 'User', + }); + const assistantMessageId = `${label}-assistant-${turn}`; + messages.push({ + messageId: assistantMessageId, + parentMessageId: userMessageId, + text: assistantBody(turn), + isCreatedByUser: false, + sender: 'Assistant', + }); + parentMessageId = assistantMessageId; + } + return messages; +} + +export function buildContinuationReply(): string { + let body = '# Continued mobile stress response\n\n'; + for (let section = 1; section <= 16; section += 1) { + body += `## Stream section ${section}\n\n${PROSE}${PROSE}\n\n`; + body += `1. Render the streamed section ${section}.\n`; + body += `2. Keep the long transcript mounted behind it.\n\n`; + if (section % 4 === 0) { + body += + '```ts\nexport function streamedValue(input: number): number {\n return input + 1;\n}\n```\n\n'; + } + } + return `${body}${STREAM_END_MARKER}\n`; +} diff --git a/e2e/perf/browser.ts b/e2e/perf/browser.ts new file mode 100644 index 0000000000..a04cb434f9 --- /dev/null +++ b/e2e/perf/browser.ts @@ -0,0 +1,153 @@ +import type { CDPSession, Page, TestInfo } from '@playwright/test'; + +const METRIC_NAMES = [ + 'Timestamp', + 'TaskDuration', + 'ScriptDuration', + 'LayoutDuration', + 'RecalcStyleDuration', + 'LayoutCount', + 'RecalcStyleCount', + 'JSHeapUsedSize', + 'Nodes', +] as const; + +type MetricName = (typeof METRIC_NAMES)[number]; +type MetricMap = Record; + +export interface BrowserPhase { + elapsedMs: number; + taskMs: number; + scriptMs: number; + layoutMs: number; + styleMs: number; + busyPercent: number; + layoutCount: number; + styleCount: number; + heapDeltaBytes: number; + heapEndBytes: number; + nodesEnd: number; + longTasks: number[]; +} + +interface BrowserPerfGlobal { + longTasks: number[]; + reset(): void; + drain(): void; +} + +declare global { + interface Window { + __BROWSER_PERF__: BrowserPerfGlobal; + } +} + +const LONG_TASK_OBSERVER = `(() => { + const perf = { + longTasks: [], + observer: null, + drain() { + if (!this.observer) { + return; + } + for (const entry of this.observer.takeRecords()) { + this.longTasks.push(entry.duration); + } + }, + reset() { + this.drain(); + this.longTasks = []; + }, + }; + window.__BROWSER_PERF__ = perf; + try { + perf.observer = new PerformanceObserver((list) => { + for (const entry of list.getEntries()) { + perf.longTasks.push(entry.duration); + } + }); + perf.observer.observe({ type: 'longtask', buffered: true }); + } catch (_error) { + /* longtask unsupported: totals stay empty */ + } +})();`; + +async function readMetrics(session: CDPSession): Promise { + const response = await session.send('Performance.getMetrics'); + const values = new Map(response.metrics.map(({ name, value }) => [name, value])); + return Object.fromEntries(METRIC_NAMES.map((name) => [name, values.get(name) ?? 0])) as MetricMap; +} + +export async function installBrowserPerf(page: Page): Promise { + await page.addInitScript({ content: LONG_TASK_OBSERVER }); +} + +export async function createBrowserProbe(page: Page): Promise<{ + start(): Promise; + finish(): Promise; +}> { + const session = await page.context().newCDPSession(page); + await session.send('Performance.enable'); + let startMetrics: MetricMap | null = null; + + return { + async start() { + await page.evaluate(() => window.__BROWSER_PERF__.reset()); + startMetrics = await readMetrics(session); + }, + async finish() { + if (!startMetrics) { + throw new Error('Browser performance probe must be started before it is finished'); + } + const endMetrics = await readMetrics(session); + const longTasks = await page.evaluate(() => { + window.__BROWSER_PERF__.drain(); + return window.__BROWSER_PERF__.longTasks.slice(); + }); + const secondsToMs = (name: MetricName) => + (endMetrics[name] - (startMetrics?.[name] ?? 0)) * 1000; + const elapsedMs = secondsToMs('Timestamp'); + const taskMs = secondsToMs('TaskDuration'); + return { + elapsedMs, + taskMs, + scriptMs: secondsToMs('ScriptDuration'), + layoutMs: secondsToMs('LayoutDuration'), + styleMs: secondsToMs('RecalcStyleDuration'), + busyPercent: elapsedMs > 0 ? (taskMs / elapsedMs) * 100 : 0, + layoutCount: endMetrics.LayoutCount - startMetrics.LayoutCount, + styleCount: endMetrics.RecalcStyleCount - startMetrics.RecalcStyleCount, + heapDeltaBytes: endMetrics.JSHeapUsedSize - startMetrics.JSHeapUsedSize, + heapEndBytes: endMetrics.JSHeapUsedSize, + nodesEnd: endMetrics.Nodes, + longTasks, + }; + }, + }; +} + +export function formatBrowserPhase(name: string, phase: BrowserPhase): string { + const longTaskTotal = phase.longTasks.reduce((sum, duration) => sum + duration, 0); + const worstLongTask = phase.longTasks.reduce((max, duration) => Math.max(max, duration), 0); + return ( + `${name.padEnd(18)} wall=${phase.elapsedMs.toFixed(0).padStart(6)}ms ` + + `busy=${phase.busyPercent.toFixed(1).padStart(5)}% ` + + `task=${phase.taskMs.toFixed(0).padStart(6)}ms ` + + `script=${phase.scriptMs.toFixed(0).padStart(6)}ms ` + + `layout=${phase.layoutMs.toFixed(0).padStart(5)}ms ` + + `style=${phase.styleMs.toFixed(0).padStart(5)}ms ` + + `longtasks=${longTaskTotal.toFixed(0)}/${worstLongTask.toFixed(0)}ms ` + + `heap=${(phase.heapEndBytes / 1024 / 1024).toFixed(1)}MB nodes=${phase.nodesEnd}` + ); +} + +export async function attachBrowserPhases( + testInfo: TestInfo, + name: string, + phases: Record, +): Promise { + await testInfo.attach(name, { + body: JSON.stringify(phases, null, 2), + contentType: 'application/json', + }); +} diff --git a/e2e/playwright.config.mobile-chat-perf.ts b/e2e/playwright.config.mobile-chat-perf.ts new file mode 100644 index 0000000000..a8e4b8f2e2 --- /dev/null +++ b/e2e/playwright.config.mobile-chat-perf.ts @@ -0,0 +1,36 @@ +import { defineConfig, devices } from '@playwright/test'; +import mockConfig from './playwright.config.mock'; +import { buildContinuationReply } from './benchmarks-mobile-chat/payload'; + +const mockServers = Array.isArray(mockConfig.webServer) ? mockConfig.webServer : []; +if (mockConfig.webServer && !Array.isArray(mockConfig.webServer)) { + mockServers.push(mockConfig.webServer); +} +const benchmarkModelEnv = { + MOCK_LLM_REPLY: buildContinuationReply(), + MOCK_LLM_CHUNK_DELAY_MS: '1', +}; + +export default defineConfig({ + ...mockConfig, + testDir: 'benchmarks-mobile-chat', + outputDir: 'benchmarks/.test-results/mobile-chat', + fullyParallel: false, + timeout: 10 * 60 * 1000, + retries: 0, + workers: 1, + reporter: [['line']], + webServer: mockServers.map((server) => ({ + ...server, + env: { ...server.env, ...benchmarkModelEnv }, + })), + projects: [ + { + name: 'iPhone 13 (Chromium)', + use: { + ...devices['iPhone 13'], + browserName: 'chromium', + }, + }, + ], +}); diff --git a/package.json b/package.json index f4f5962c79..a60c89de17 100644 --- a/package.json +++ b/package.json @@ -72,6 +72,7 @@ "e2e:benchmark:agents": "npm run e2e:prepare && playwright test --config=e2e/playwright.config.benchmark.ts agent-startup.latency.spec.ts", "e2e:benchmark:navigation": "npm run e2e:prepare && playwright test --config=e2e/playwright.config.navigation-perf.ts", "e2e:benchmark:reasoning": "npm run e2e:prepare && playwright test --config=e2e/playwright.config.reasoning-perf.ts", + "e2e:benchmark:mobile-chat": "npm run e2e:prepare && playwright test --config=e2e/playwright.config.mobile-chat-perf.ts", "e2e:bombadil": "npm run e2e:prepare && playwright test --config=e2e/playwright.config.bombadil.ts", "e2e:bombadil:run": "playwright test --config=e2e/playwright.config.bombadil.ts", "e2e:bombadil:branch-reload": "npm run e2e:prepare && cross-env BOMBADIL_SPECIFICATION=branch-reload.specification.ts BOMBADIL_TIME_LIMIT=30s playwright test --config=e2e/playwright.config.bombadil.ts",