🔦 chore: Drop @lhci/cli and Run Lighthouse 13 Directly (#15687)

`@lhci/cli@0.15.1` pins `lighthouse` to an exact `12.6.1`, so the Lighthouse
lane could never move off a release whose `puppeteer-core` still depends on
`extract-zip` — an advisory with no patched version. That pin, not Lighthouse
itself, was holding 8 of the repository's 16 `npm audit` findings open.

LHCI was only ever a subprocess wrapper here: `audit.ts` shelled out to
`@lhci/cli/src/cli.js` for `collect` and `assert`, and LHCI's own node runner
shells out to `lighthouse/cli/index.js` in turn. Call that CLI directly, run
the three navigations in a loop, and assert the median budgets in TypeScript
next to the assertions `load.spec.ts` already makes.

- `npm audit`: 16 findings (7 high, 1 moderate, 8 low) -> 6 low. The remainder
  is the pre-existing `elliptic` chain under `vite-plugin-node-polyfills`,
  which has no fixed version and is unrelated to this lane.
- Lighthouse 13 removed `largest-contentful-paint-element` and moved the LCP
  node into `lcp-breakdown-insight`. Reading it through a `lcpElement` helper
  keeps the "the transcript must be the LCP element" guard working and makes
  the next rename fail loudly in one place.
- Reports move to `.lighthouse/` as `lhr-N.report.{json,html}`; cookie
  redaction, the API timing table and the desktop/provided-throttling settings
  are unchanged. Budgets are now printed as a table immediately before they are
  asserted, so the workflow's 80-line failure comment always contains them.
- `CHROME_PATH` and `LIGHTHOUSE_CHROME_FLAGS` are documented: chrome-launcher
  prefers the Windows Chrome under WSL, whose debugging port Linux cannot
  reach.
This commit is contained in:
Danny Avila 2026-09-06 23:20:22 -04:00 • committed by GitHub
parent 644e33ff37
commit c883e9ef32
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 699 additions and 1833 deletions

View file

@ -65,7 +65,7 @@ jobs:
with:
name: lighthouse
path: |
.lighthouseci/
.lighthouse/
lighthouse-ci.log
include-hidden-files: true
retention-days: 7

2
.gitignore vendored
View file

@ -192,5 +192,5 @@ codedb.snapshot
scripts/activity-labels/results/
!e2e/lighthouse/librechat.yaml
/.lighthouseci/
/.lighthouse/
/e2e/lighthouse/.test-results/

View file

@ -11,10 +11,13 @@ E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:run
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:regression
```
`E2E_BASE_URL=http://localhost:3098` selects another local port. Each run starts a
disposable MongoDB and the real Express server, registers a local user, and seeds
a conversation. No model inference is needed. Do not point this test at a deployed
service. Playwright refuses to reuse an existing server.
`E2E_BASE_URL=http://localhost:3098` selects another local port. Set `CHROME_PATH` if
chrome-launcher picks the wrong browser — on WSL it prefers the Windows install, whose
debugging port is unreachable from Linux — and `LIGHTHOUSE_CHROME_FLAGS` to append Chrome
flags such as `--no-sandbox`. Each run starts a disposable MongoDB and the real Express
server, registers a local user, and seeds a conversation. No model inference is needed.
Do not point this test at a deployed service. Playwright refuses to reuse an existing
server.
The existing `benchmarks/mongoose-latency-hook.cjs` adds **250 ms per Mongoose
Query/Aggregate execution** in the server process. Independent queries can overlap;
@ -23,20 +26,21 @@ latency, not a replica topology or network emulator. It also delays query-based
writes; native driver calls, bulk operations and cursor batches are outside its
coverage. Use a TCP latency proxy if those paths need coverage.
Lighthouse CI makes three cold browser navigations to a populated conversation,
The runner makes three cold browser navigations to a populated conversation,
using the production client build and real authentication, config, file and message
routes. It uses desktop settings with `throttlingMethod: provided` so Lighthouse
does not replace the measured server delays with simulated network timing.
Median budgets are LCP **4,500 ms**, CLS **0.1**, and TBT **500 ms**. These are lab
regression budgets, not field web-vitals percentiles; Lighthouse does not measure INP.
Median budgets are LCP **4,500 ms**, CLS **0.1**, and TBT **500 ms**, asserted against
the median of the three runs. These are lab regression budgets, not field web-vitals
percentiles; Lighthouse does not measure INP.
The test also requires the seeded transcript to be the LCP element, so a fast
login page, spinner, or empty shell cannot pass.
## When the gate fails
1. Read the failed audit's actual value and limit in the job log or
`.lighthouseci/assertion-results.json`.
2. Open a `.lighthouseci/lhr-*.html` report. The console also prints API request
1. Read the failed audit's measured median and limit in the budget table the runner
prints immediately before asserting.
2. Open a `.lighthouse/lhr-*.report.html` report. The console also prints API request
start/end times. A late request start suggests a browser dependency; a long
request suggests server work or serial database reads.
3. Inspect the relevant path before changing the budget:
@ -54,15 +58,17 @@ before returning data. Do not raise a threshold to hide added round trips.
## Adding another scenario
`auditPage({ url, cookies, configPath })` in `audit.ts` collects the reports,
redacts authentication headers, prints API timings and enforces the selected LHCI
budgets. It returns the reports so each scenario can assert its expected final
URL and LCP content. `load.spec.ts` owns local login state, conversation seeding
and transcript assertions; the runner does not depend on them.
`auditPage({ url, cookies, runs, budgets })` in `audit.ts` invokes the Lighthouse CLI
once per run, redacts authentication headers, prints API timings and asserts the median
budgets. It returns the reports so each scenario can assert its expected final URL and
LCP content; read the LCP element through the exported `lcpElement(report)` helper rather
than an audit id, because Lighthouse renames those between majors. `load.spec.ts` owns
local login state, conversation seeding and transcript assertions; the runner does not
depend on them.
A downstream fork can import this runner from a separate launch spec, supply
cookies from its own authentication fixture and use a separate LHCI config for
launch budgets. The database delay hook can be preloaded by that fixture's server
cookies from its own authentication fixture and pass its own `budgets` for
launch thresholds. The database delay hook can be preloaded by that fixture's server
configuration. Keep provider-specific authentication and provisioning in that spec.
For launch flows that navigate across documents, also measure entry-to-ready time
with Playwright: LCP resets on a new document, so final-page LCP alone does not
@ -70,7 +76,7 @@ cover the entire launch. This lane does not exercise OpenID/Redis cache priming.
`lighthouse:regression` preloads a test-only hook that adds 16 real, sequential
user reads before message retrieval (at least four extra seconds). It uses the
same page and LHCI assertions, and never modifies production source. Confirm
same page and budget assertions, and never modifies production source. Confirm
that `largest-contentful-paint` fails, rather than treating any process error as
proof. Run the normal command again to restore baseline reports. Reports stay
local or in GitHub job artifacts; session cookie values are redacted before upload.

View file

@ -3,45 +3,100 @@ import path from 'node:path';
import { promisify } from 'node:util';
import { execFile } from 'node:child_process';
import { expect } from '@playwright/test';
import type { Cookie } from '@playwright/test';
import type Result from 'lighthouse/types/lhr/lhr';
import type { Cookie } from '@playwright/test';
const exec = promisify(execFile);
/** Median budgets keyed by Lighthouse audit id. */
export type MedianBudgets = Record<string, number>;
const REPORTS_DIRECTORY = '.lighthouse';
const DEFAULT_RUNS = 3;
const DEFAULT_BUDGETS: MedianBudgets = {
'largest-contentful-paint': 4500,
'cumulative-layout-shift': 0.1,
'total-blocking-time': 500,
};
/**
* Lighthouse 13 removed `largest-contentful-paint-element` and moved the LCP node into the
* breakdown insight. Scenarios read the element through here so a future rename fails once,
* loudly, instead of silently passing an assertion against an absent audit.
*/
export function lcpElement(report: Result): string {
const details = report.audits['lcp-breakdown-insight']?.details;
if (!details) {
throw new Error('Lighthouse reported no LCP element; check the `lcp-breakdown-insight` audit.');
}
return JSON.stringify(details);
}
function medianOf(results: Result[], audit: string): number {
const values = results
.map((report) => report.audits[audit]?.numericValue)
.filter((value): value is number => typeof value === 'number')
.sort((a, b) => a - b);
expect(values, `Lighthouse must report a numeric ${audit} for every run`).toHaveLength(
results.length,
);
const middle = values.length >> 1;
return values.length % 2 ? values[middle] : (values[middle - 1] + values[middle]) / 2;
}
export async function auditPage({
url,
cookies,
configPath = path.resolve(__dirname, 'lighthouserc.cjs'),
runs = DEFAULT_RUNS,
budgets = DEFAULT_BUDGETS,
}: {
url: string;
cookies: Cookie[];
configPath?: string;
runs?: number;
budgets?: MedianBudgets;
}): Promise<Result[]> {
const env = {
...process.env,
LIGHTHOUSE_URL: url,
LIGHTHOUSE_COOKIE: cookies.map(({ name, value }) => `${name}=${value}`).join('; '),
};
const cli = require.resolve('@lhci/cli/src/cli.js');
const config = `--config=${configPath}`;
const run = (command: string) => exec(process.execPath, [cli, command, config], { env });
const directory = path.resolve('.lighthouseci');
const cookie = cookies.map(({ name, value }) => `${name}=${value}`).join('; ');
const directory = path.resolve(REPORTS_DIRECTORY);
fs.rmSync(directory, { recursive: true, force: true });
fs.mkdirSync(directory, { recursive: true });
const cli = require.resolve('lighthouse/cli/index.js');
const chromeFlags = `--headless=new ${process.env.LIGHTHOUSE_CHROME_FLAGS ?? ''}`.trim();
const flags = [
url,
'--quiet',
'--preset=desktop',
'--throttling-method=provided',
'--only-categories=performance',
`--chrome-flags=${chromeFlags}`,
`--extra-headers=${JSON.stringify({ Cookie: cookie })}`,
'--output=json',
'--output=html',
];
try {
console.log((await run('collect')).stdout);
for (let run = 1; run <= runs; run++) {
const output = path.join(directory, `lhr-${run}`);
const { stdout } = await exec(process.execPath, [cli, ...flags, `--output-path=${output}`]);
console.log(`Lighthouse run ${run}/${runs} wrote ${output}.report.json`);
if (stdout.trim()) {
console.log(stdout);
}
}
} finally {
// LHCI embeds extraHeaders in reports; keep disposable session cookies out of artifacts.
// Lighthouse embeds extraHeaders in reports; keep disposable session cookies out of artifacts.
for (const file of fs.existsSync(directory) ? fs.readdirSync(directory) : []) {
if (!/\.(json|html)$/.test(file)) continue;
if (!/\.report\.(json|html)$/.test(file)) continue;
const filename = path.join(directory, file);
const content = fs.readFileSync(filename, 'utf8');
fs.writeFileSync(
filename,
env.LIGHTHOUSE_COOKIE ? content.replaceAll(env.LIGHTHOUSE_COOKIE, '[redacted]') : content,
);
fs.writeFileSync(filename, cookie ? content.replaceAll(cookie, '[redacted]') : content);
}
}
const reports = fs.readdirSync(directory).filter((file) => /^lhr-.*\.json$/.test(file));
expect(reports.length, 'Lighthouse must produce reports').toBeGreaterThan(0);
const reports = fs.readdirSync(directory).filter((file) => /^lhr-.*\.report\.json$/.test(file));
expect(reports.length, 'Lighthouse must produce one report per run').toBe(runs);
const results = reports.map(
(file) => JSON.parse(fs.readFileSync(path.join(directory, file), 'utf8')) as Result,
);
@ -65,8 +120,17 @@ export async function auditPage({
expect(report.runtimeError, 'Lighthouse navigation must succeed').toBeUndefined();
}
console.log(
'Inspect .lighthouseci HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.',
`Inspect ${REPORTS_DIRECTORY} HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.`,
);
console.log((await run('assert')).stdout);
const measured = Object.entries(budgets).map(([audit, limit]) => ({
audit,
median: medianOf(results, audit),
limit,
}));
console.table(measured);
for (const { audit, median, limit } of measured) {
expect(median, `Median ${audit} must stay within ${limit}`).toBeLessThanOrEqual(limit);
}
return results;
}

View file

@ -1,24 +0,0 @@
module.exports = {
ci: {
collect: {
url: [process.env.LIGHTHOUSE_URL],
numberOfRuns: 3,
settings: {
preset: 'desktop',
throttlingMethod: 'provided',
onlyCategories: ['performance'],
extraHeaders: { Cookie: process.env.LIGHTHOUSE_COOKIE },
},
},
assert: {
assertions: {
'largest-contentful-paint': [
'error',
{ maxNumericValue: 4500, aggregationMethod: 'median' },
],
'cumulative-layout-shift': ['error', { maxNumericValue: 0.1, aggregationMethod: 'median' }],
'total-blocking-time': ['error', { maxNumericValue: 500, aggregationMethod: 'median' }],
},
},
},
};

View file

@ -1,7 +1,7 @@
import { expect, test } from '@playwright/test';
import { seedConversations, seedMessages } from '../specs/mock/db';
import { getE2EUser } from '../setup/user';
import { auditPage } from './audit';
import { auditPage, lcpElement } from './audit';
const conversationId = '16390000-0000-4000-8000-000000000001';
const title = 'Lighthouse latency audit';
@ -35,7 +35,7 @@ test('serial database latency stays within web-vitals budgets', async ({ page, b
for (const report of reports) {
expect(report.finalDisplayedUrl, 'Do not measure a login redirect').toBe(url);
expect(
JSON.stringify(report.audits['largest-contentful-paint-element'].details),
lcpElement(report),
'The measured LCP must be the seeded transcript, not the shell or a spinner',
).toContain(transcriptMarker);
}

2345
package-lock.json generated

File diff suppressed because it is too large Load diff

View file

@ -162,7 +162,6 @@
"@eslint/compat": "^1.2.6",
"@eslint/eslintrc": "^3.3.4",
"@eslint/js": "^9.20.0",
"@lhci/cli": "0.15.1",
"@playwright/test": "^1.62.1",
"@types/react-virtualized": "^9.22.0",
"brace-expansion": "^2.1.2",
@ -183,7 +182,7 @@
"globals": "^15.14.0",
"husky": "^9.1.7",
"jest": "^30.2.0",
"lighthouse": "12.6.1",
"lighthouse": "13.4.1",
"lint-staged": "^15.4.3",
"prettier": "^3.5.0",
"prettier-plugin-tailwindcss": "^0.6.11",