mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 06:51:59 +00:00
🔦 chore: Drop @lhci/cli and Run Lighthouse 13 Directly (#15687)
`@lhci/cli@0.15.1` pins `lighthouse` to an exact `12.6.1`, so the Lighthouse
lane could never move off a release whose `puppeteer-core` still depends on
`extract-zip` — an advisory with no patched version. That pin, not Lighthouse
itself, was holding 8 of the repository's 16 `npm audit` findings open.
LHCI was only ever a subprocess wrapper here: `audit.ts` shelled out to
`@lhci/cli/src/cli.js` for `collect` and `assert`, and LHCI's own node runner
shells out to `lighthouse/cli/index.js` in turn. Call that CLI directly, run
the three navigations in a loop, and assert the median budgets in TypeScript
next to the assertions `load.spec.ts` already makes.
- `npm audit`: 16 findings (7 high, 1 moderate, 8 low) -> 6 low. The remainder
is the pre-existing `elliptic` chain under `vite-plugin-node-polyfills`,
which has no fixed version and is unrelated to this lane.
- Lighthouse 13 removed `largest-contentful-paint-element` and moved the LCP
node into `lcp-breakdown-insight`. Reading it through a `lcpElement` helper
keeps the "the transcript must be the LCP element" guard working and makes
the next rename fail loudly in one place.
- Reports move to `.lighthouse/` as `lhr-N.report.{json,html}`; cookie
redaction, the API timing table and the desktop/provided-throttling settings
are unchanged. Budgets are now printed as a table immediately before they are
asserted, so the workflow's 80-line failure comment always contains them.
- `CHROME_PATH` and `LIGHTHOUSE_CHROME_FLAGS` are documented: chrome-launcher
prefers the Windows Chrome under WSL, whose debugging port Linux cannot
reach.
This commit is contained in:
parent
644e33ff37
commit
c883e9ef32
8 changed files with 699 additions and 1833 deletions
2
.github/workflows/lighthouse.yml
vendored
2
.github/workflows/lighthouse.yml
vendored
|
|
@ -65,7 +65,7 @@ jobs:
|
|||
with:
|
||||
name: lighthouse
|
||||
path: |
|
||||
.lighthouseci/
|
||||
.lighthouse/
|
||||
lighthouse-ci.log
|
||||
include-hidden-files: true
|
||||
retention-days: 7
|
||||
|
|
|
|||
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -192,5 +192,5 @@ codedb.snapshot
|
|||
scripts/activity-labels/results/
|
||||
|
||||
!e2e/lighthouse/librechat.yaml
|
||||
/.lighthouseci/
|
||||
/.lighthouse/
|
||||
/e2e/lighthouse/.test-results/
|
||||
|
|
|
|||
|
|
@ -11,10 +11,13 @@ E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:run
|
|||
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:regression
|
||||
```
|
||||
|
||||
`E2E_BASE_URL=http://localhost:3098` selects another local port. Each run starts a
|
||||
disposable MongoDB and the real Express server, registers a local user, and seeds
|
||||
a conversation. No model inference is needed. Do not point this test at a deployed
|
||||
service. Playwright refuses to reuse an existing server.
|
||||
`E2E_BASE_URL=http://localhost:3098` selects another local port. Set `CHROME_PATH` if
|
||||
chrome-launcher picks the wrong browser — on WSL it prefers the Windows install, whose
|
||||
debugging port is unreachable from Linux — and `LIGHTHOUSE_CHROME_FLAGS` to append Chrome
|
||||
flags such as `--no-sandbox`. Each run starts a disposable MongoDB and the real Express
|
||||
server, registers a local user, and seeds a conversation. No model inference is needed.
|
||||
Do not point this test at a deployed service. Playwright refuses to reuse an existing
|
||||
server.
|
||||
|
||||
The existing `benchmarks/mongoose-latency-hook.cjs` adds **250 ms per Mongoose
|
||||
Query/Aggregate execution** in the server process. Independent queries can overlap;
|
||||
|
|
@ -23,20 +26,21 @@ latency, not a replica topology or network emulator. It also delays query-based
|
|||
writes; native driver calls, bulk operations and cursor batches are outside its
|
||||
coverage. Use a TCP latency proxy if those paths need coverage.
|
||||
|
||||
Lighthouse CI makes three cold browser navigations to a populated conversation,
|
||||
The runner makes three cold browser navigations to a populated conversation,
|
||||
using the production client build and real authentication, config, file and message
|
||||
routes. It uses desktop settings with `throttlingMethod: provided` so Lighthouse
|
||||
does not replace the measured server delays with simulated network timing.
|
||||
Median budgets are LCP **4,500 ms**, CLS **0.1**, and TBT **500 ms**. These are lab
|
||||
regression budgets, not field web-vitals percentiles; Lighthouse does not measure INP.
|
||||
Median budgets are LCP **4,500 ms**, CLS **0.1**, and TBT **500 ms**, asserted against
|
||||
the median of the three runs. These are lab regression budgets, not field web-vitals
|
||||
percentiles; Lighthouse does not measure INP.
|
||||
The test also requires the seeded transcript to be the LCP element, so a fast
|
||||
login page, spinner, or empty shell cannot pass.
|
||||
|
||||
## When the gate fails
|
||||
|
||||
1. Read the failed audit's actual value and limit in the job log or
|
||||
`.lighthouseci/assertion-results.json`.
|
||||
2. Open a `.lighthouseci/lhr-*.html` report. The console also prints API request
|
||||
1. Read the failed audit's measured median and limit in the budget table the runner
|
||||
prints immediately before asserting.
|
||||
2. Open a `.lighthouse/lhr-*.report.html` report. The console also prints API request
|
||||
start/end times. A late request start suggests a browser dependency; a long
|
||||
request suggests server work or serial database reads.
|
||||
3. Inspect the relevant path before changing the budget:
|
||||
|
|
@ -54,15 +58,17 @@ before returning data. Do not raise a threshold to hide added round trips.
|
|||
|
||||
## Adding another scenario
|
||||
|
||||
`auditPage({ url, cookies, configPath })` in `audit.ts` collects the reports,
|
||||
redacts authentication headers, prints API timings and enforces the selected LHCI
|
||||
budgets. It returns the reports so each scenario can assert its expected final
|
||||
URL and LCP content. `load.spec.ts` owns local login state, conversation seeding
|
||||
and transcript assertions; the runner does not depend on them.
|
||||
`auditPage({ url, cookies, runs, budgets })` in `audit.ts` invokes the Lighthouse CLI
|
||||
once per run, redacts authentication headers, prints API timings and asserts the median
|
||||
budgets. It returns the reports so each scenario can assert its expected final URL and
|
||||
LCP content; read the LCP element through the exported `lcpElement(report)` helper rather
|
||||
than an audit id, because Lighthouse renames those between majors. `load.spec.ts` owns
|
||||
local login state, conversation seeding and transcript assertions; the runner does not
|
||||
depend on them.
|
||||
|
||||
A downstream fork can import this runner from a separate launch spec, supply
|
||||
cookies from its own authentication fixture and use a separate LHCI config for
|
||||
launch budgets. The database delay hook can be preloaded by that fixture's server
|
||||
cookies from its own authentication fixture and pass its own `budgets` for
|
||||
launch thresholds. The database delay hook can be preloaded by that fixture's server
|
||||
configuration. Keep provider-specific authentication and provisioning in that spec.
|
||||
For launch flows that navigate across documents, also measure entry-to-ready time
|
||||
with Playwright: LCP resets on a new document, so final-page LCP alone does not
|
||||
|
|
@ -70,7 +76,7 @@ cover the entire launch. This lane does not exercise OpenID/Redis cache priming.
|
|||
|
||||
`lighthouse:regression` preloads a test-only hook that adds 16 real, sequential
|
||||
user reads before message retrieval (at least four extra seconds). It uses the
|
||||
same page and LHCI assertions, and never modifies production source. Confirm
|
||||
same page and budget assertions, and never modifies production source. Confirm
|
||||
that `largest-contentful-paint` fails, rather than treating any process error as
|
||||
proof. Run the normal command again to restore baseline reports. Reports stay
|
||||
local or in GitHub job artifacts; session cookie values are redacted before upload.
|
||||
|
|
|
|||
|
|
@ -3,45 +3,100 @@ import path from 'node:path';
|
|||
import { promisify } from 'node:util';
|
||||
import { execFile } from 'node:child_process';
|
||||
import { expect } from '@playwright/test';
|
||||
import type { Cookie } from '@playwright/test';
|
||||
import type Result from 'lighthouse/types/lhr/lhr';
|
||||
import type { Cookie } from '@playwright/test';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
|
||||
/** Median budgets keyed by Lighthouse audit id. */
|
||||
export type MedianBudgets = Record<string, number>;
|
||||
|
||||
const REPORTS_DIRECTORY = '.lighthouse';
|
||||
|
||||
const DEFAULT_RUNS = 3;
|
||||
|
||||
const DEFAULT_BUDGETS: MedianBudgets = {
|
||||
'largest-contentful-paint': 4500,
|
||||
'cumulative-layout-shift': 0.1,
|
||||
'total-blocking-time': 500,
|
||||
};
|
||||
|
||||
/**
|
||||
* Lighthouse 13 removed `largest-contentful-paint-element` and moved the LCP node into the
|
||||
* breakdown insight. Scenarios read the element through here so a future rename fails once,
|
||||
* loudly, instead of silently passing an assertion against an absent audit.
|
||||
*/
|
||||
export function lcpElement(report: Result): string {
|
||||
const details = report.audits['lcp-breakdown-insight']?.details;
|
||||
if (!details) {
|
||||
throw new Error('Lighthouse reported no LCP element; check the `lcp-breakdown-insight` audit.');
|
||||
}
|
||||
return JSON.stringify(details);
|
||||
}
|
||||
|
||||
function medianOf(results: Result[], audit: string): number {
|
||||
const values = results
|
||||
.map((report) => report.audits[audit]?.numericValue)
|
||||
.filter((value): value is number => typeof value === 'number')
|
||||
.sort((a, b) => a - b);
|
||||
expect(values, `Lighthouse must report a numeric ${audit} for every run`).toHaveLength(
|
||||
results.length,
|
||||
);
|
||||
const middle = values.length >> 1;
|
||||
return values.length % 2 ? values[middle] : (values[middle - 1] + values[middle]) / 2;
|
||||
}
|
||||
|
||||
export async function auditPage({
|
||||
url,
|
||||
cookies,
|
||||
configPath = path.resolve(__dirname, 'lighthouserc.cjs'),
|
||||
runs = DEFAULT_RUNS,
|
||||
budgets = DEFAULT_BUDGETS,
|
||||
}: {
|
||||
url: string;
|
||||
cookies: Cookie[];
|
||||
configPath?: string;
|
||||
runs?: number;
|
||||
budgets?: MedianBudgets;
|
||||
}): Promise<Result[]> {
|
||||
const env = {
|
||||
...process.env,
|
||||
LIGHTHOUSE_URL: url,
|
||||
LIGHTHOUSE_COOKIE: cookies.map(({ name, value }) => `${name}=${value}`).join('; '),
|
||||
};
|
||||
const cli = require.resolve('@lhci/cli/src/cli.js');
|
||||
const config = `--config=${configPath}`;
|
||||
const run = (command: string) => exec(process.execPath, [cli, command, config], { env });
|
||||
const directory = path.resolve('.lighthouseci');
|
||||
const cookie = cookies.map(({ name, value }) => `${name}=${value}`).join('; ');
|
||||
const directory = path.resolve(REPORTS_DIRECTORY);
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
fs.mkdirSync(directory, { recursive: true });
|
||||
|
||||
const cli = require.resolve('lighthouse/cli/index.js');
|
||||
const chromeFlags = `--headless=new ${process.env.LIGHTHOUSE_CHROME_FLAGS ?? ''}`.trim();
|
||||
const flags = [
|
||||
url,
|
||||
'--quiet',
|
||||
'--preset=desktop',
|
||||
'--throttling-method=provided',
|
||||
'--only-categories=performance',
|
||||
`--chrome-flags=${chromeFlags}`,
|
||||
`--extra-headers=${JSON.stringify({ Cookie: cookie })}`,
|
||||
'--output=json',
|
||||
'--output=html',
|
||||
];
|
||||
|
||||
try {
|
||||
console.log((await run('collect')).stdout);
|
||||
for (let run = 1; run <= runs; run++) {
|
||||
const output = path.join(directory, `lhr-${run}`);
|
||||
const { stdout } = await exec(process.execPath, [cli, ...flags, `--output-path=${output}`]);
|
||||
console.log(`Lighthouse run ${run}/${runs} wrote ${output}.report.json`);
|
||||
if (stdout.trim()) {
|
||||
console.log(stdout);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
// LHCI embeds extraHeaders in reports; keep disposable session cookies out of artifacts.
|
||||
// Lighthouse embeds extraHeaders in reports; keep disposable session cookies out of artifacts.
|
||||
for (const file of fs.existsSync(directory) ? fs.readdirSync(directory) : []) {
|
||||
if (!/\.(json|html)$/.test(file)) continue;
|
||||
if (!/\.report\.(json|html)$/.test(file)) continue;
|
||||
const filename = path.join(directory, file);
|
||||
const content = fs.readFileSync(filename, 'utf8');
|
||||
fs.writeFileSync(
|
||||
filename,
|
||||
env.LIGHTHOUSE_COOKIE ? content.replaceAll(env.LIGHTHOUSE_COOKIE, '[redacted]') : content,
|
||||
);
|
||||
fs.writeFileSync(filename, cookie ? content.replaceAll(cookie, '[redacted]') : content);
|
||||
}
|
||||
}
|
||||
const reports = fs.readdirSync(directory).filter((file) => /^lhr-.*\.json$/.test(file));
|
||||
expect(reports.length, 'Lighthouse must produce reports').toBeGreaterThan(0);
|
||||
|
||||
const reports = fs.readdirSync(directory).filter((file) => /^lhr-.*\.report\.json$/.test(file));
|
||||
expect(reports.length, 'Lighthouse must produce one report per run').toBe(runs);
|
||||
const results = reports.map(
|
||||
(file) => JSON.parse(fs.readFileSync(path.join(directory, file), 'utf8')) as Result,
|
||||
);
|
||||
|
|
@ -65,8 +120,17 @@ export async function auditPage({
|
|||
expect(report.runtimeError, 'Lighthouse navigation must succeed').toBeUndefined();
|
||||
}
|
||||
console.log(
|
||||
'Inspect .lighthouseci HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.',
|
||||
`Inspect ${REPORTS_DIRECTORY} HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.`,
|
||||
);
|
||||
console.log((await run('assert')).stdout);
|
||||
|
||||
const measured = Object.entries(budgets).map(([audit, limit]) => ({
|
||||
audit,
|
||||
median: medianOf(results, audit),
|
||||
limit,
|
||||
}));
|
||||
console.table(measured);
|
||||
for (const { audit, median, limit } of measured) {
|
||||
expect(median, `Median ${audit} must stay within ${limit}`).toBeLessThanOrEqual(limit);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,24 +0,0 @@
|
|||
module.exports = {
|
||||
ci: {
|
||||
collect: {
|
||||
url: [process.env.LIGHTHOUSE_URL],
|
||||
numberOfRuns: 3,
|
||||
settings: {
|
||||
preset: 'desktop',
|
||||
throttlingMethod: 'provided',
|
||||
onlyCategories: ['performance'],
|
||||
extraHeaders: { Cookie: process.env.LIGHTHOUSE_COOKIE },
|
||||
},
|
||||
},
|
||||
assert: {
|
||||
assertions: {
|
||||
'largest-contentful-paint': [
|
||||
'error',
|
||||
{ maxNumericValue: 4500, aggregationMethod: 'median' },
|
||||
],
|
||||
'cumulative-layout-shift': ['error', { maxNumericValue: 0.1, aggregationMethod: 'median' }],
|
||||
'total-blocking-time': ['error', { maxNumericValue: 500, aggregationMethod: 'median' }],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { expect, test } from '@playwright/test';
|
||||
import { seedConversations, seedMessages } from '../specs/mock/db';
|
||||
import { getE2EUser } from '../setup/user';
|
||||
import { auditPage } from './audit';
|
||||
import { auditPage, lcpElement } from './audit';
|
||||
|
||||
const conversationId = '16390000-0000-4000-8000-000000000001';
|
||||
const title = 'Lighthouse latency audit';
|
||||
|
|
@ -35,7 +35,7 @@ test('serial database latency stays within web-vitals budgets', async ({ page, b
|
|||
for (const report of reports) {
|
||||
expect(report.finalDisplayedUrl, 'Do not measure a login redirect').toBe(url);
|
||||
expect(
|
||||
JSON.stringify(report.audits['largest-contentful-paint-element'].details),
|
||||
lcpElement(report),
|
||||
'The measured LCP must be the seeded transcript, not the shell or a spinner',
|
||||
).toContain(transcriptMarker);
|
||||
}
|
||||
|
|
|
|||
2345
package-lock.json
generated
2345
package-lock.json
generated
File diff suppressed because it is too large
Load diff
|
|
@ -162,7 +162,6 @@
|
|||
"@eslint/compat": "^1.2.6",
|
||||
"@eslint/eslintrc": "^3.3.4",
|
||||
"@eslint/js": "^9.20.0",
|
||||
"@lhci/cli": "0.15.1",
|
||||
"@playwright/test": "^1.62.1",
|
||||
"@types/react-virtualized": "^9.22.0",
|
||||
"brace-expansion": "^2.1.2",
|
||||
|
|
@ -183,7 +182,7 @@
|
|||
"globals": "^15.14.0",
|
||||
"husky": "^9.1.7",
|
||||
"jest": "^30.2.0",
|
||||
"lighthouse": "12.6.1",
|
||||
"lighthouse": "13.4.1",
|
||||
"lint-staged": "^15.4.3",
|
||||
"prettier": "^3.5.0",
|
||||
"prettier-plugin-tailwindcss": "^0.6.11",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue