mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 15:02:02 +00:00
🎲 ci: Fail Playwright Runs When a Test Passes Only on a Retry (#18051)
Playwright reports a test that fails and then passes on a retry as "flaky" and exits 0, so a regression that fails two attempts in three merges green. Set `failOnFlakyTests` in CI on the mock profile (inherited by the Redis transport, registered-email and other derived profiles) and on the base profile. Retry counts are unchanged: the later attempts still record the trace and video and show whether a failure repeats. A run failed this way still prints "N flaky" beside a passed count, so add a reporter that names each retried pass in the job log and the job summary and says that it is what failed the run.
This commit is contained in:
parent
97f7fcdbac
commit
c6b7a5ac12
8 changed files with 335 additions and 7 deletions
7
.github/workflows/playwright-mock.yml
vendored
7
.github/workflows/playwright-mock.yml
vendored
|
|
@ -245,8 +245,8 @@ jobs:
|
|||
if: steps.cache-node-modules.outputs.cache-hit != 'true'
|
||||
run: npm ci
|
||||
|
||||
- name: Test Playwright matrix selection and the vote gate
|
||||
run: node --test .github/scripts/playwright-matrix.test.mjs .github/scripts/playwright-shards.test.mjs .github/scripts/vote-gate.test.mjs
|
||||
- name: Test Playwright matrix selection, the vote gate and the retried-pass reporter
|
||||
run: node --test .github/scripts/playwright-matrix.test.mjs .github/scripts/playwright-shards.test.mjs .github/scripts/vote-gate.test.mjs e2e/reporters/retried.test.mjs
|
||||
|
||||
- name: Restore data-provider build cache
|
||||
id: cache-data-provider
|
||||
|
|
@ -440,7 +440,8 @@ jobs:
|
|||
E2E_TITLE_CONVO: 'true'
|
||||
run: >-
|
||||
npx playwright test --config=e2e/playwright.config.mock.ts
|
||||
conversation-management.spec.ts --grep 'first-turn|final-title polling' --reporter=line
|
||||
conversation-management.spec.ts --grep 'first-turn|final-title polling'
|
||||
--reporter=line,e2e/reporters/retried.mjs
|
||||
--output=e2e/specs/.test-results/title-ownership
|
||||
|
||||
- name: Upload title ownership diagnostics
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue