🎲 ci: Fail Playwright Runs When a Test Passes Only on a Retry (#18051)

Playwright reports a test that fails and then passes on a retry as "flaky" and exits 0, so a
regression that fails two attempts in three merges green. Set `failOnFlakyTests` in CI on the
mock profile (inherited by the Redis transport, registered-email and other derived profiles) and
on the base profile. Retry counts are unchanged: the later attempts still record the trace and
video and show whether a failure repeats.

A run failed this way still prints "N flaky" beside a passed count, so add a reporter that names
each retried pass in the job log and the job summary and says that it is what failed the run.
This commit is contained in:
Danny Avila 2026-11-11 10:31:06 -05:00 • committed by GitHub
parent 97f7fcdbac
commit c6b7a5ac12
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 335 additions and 7 deletions

View file

@ -245,8 +245,8 @@ jobs:
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Test Playwright matrix selection and the vote gate
run: node --test .github/scripts/playwright-matrix.test.mjs .github/scripts/playwright-shards.test.mjs .github/scripts/vote-gate.test.mjs
- name: Test Playwright matrix selection, the vote gate and the retried-pass reporter
run: node --test .github/scripts/playwright-matrix.test.mjs .github/scripts/playwright-shards.test.mjs .github/scripts/vote-gate.test.mjs e2e/reporters/retried.test.mjs
- name: Restore data-provider build cache
id: cache-data-provider
@ -440,7 +440,8 @@ jobs:
E2E_TITLE_CONVO: 'true'
run: >-
npx playwright test --config=e2e/playwright.config.mock.ts
conversation-management.spec.ts --grep 'first-turn|final-title polling' --reporter=line
conversation-management.spec.ts --grep 'first-turn|final-title polling'
--reporter=line,e2e/reporters/retried.mjs
--output=e2e/specs/.test-results/title-ownership
- name: Upload title ownership diagnostics