CERBERUS/speedtest
1
0
Fork 0
mirror of https://github.com/librespeed/speedtest.git synced 2026-06-28 21:02:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
speedtest/tests/PLAYWRIGHT_MODES_PLAN.md
sstidl 3bd9e1b7af
chore/playwright mode test plan (#778) 
* ci: make standalone playwright manual and document e2e gating

* docs(test): refine playwright plan with implemented CI strategy

---------

Co-authored-by: Stefan Stidl <stefan.stidl@ffg.at>
2026-04-12 15:15:14 +02:00

5 KiB
Raw Blame History

Playwright Plan: Test All Runtime and UI Modes

Objective

Build a deterministic Playwright test suite that validates LibreSpeed behavior across all supported deployment modes and UI design modes, without asserting real network throughput values.

Current Status

  • Phase 1 is implemented and passing in Chromium.
  • Added regression coverage for the classic standalone "No servers available" issue path.
  • Docker image workflow is hard-gated by e2e (build depends on e2e).
  • Standalone Playwright workflow is manual-only to avoid duplicate e2e runs.

Modes to Cover

Docker runtime modes

  • standalone
  • backend
  • frontend
  • dual

UI design modes

  • Classic (index-classic.html)
  • Modern (index-modern.html)
  • Switcher behavior from index.html:
    • default from config.json (useNewDesign)
    • ?design=new override
    • ?design=old override

Test Strategy

1. Keep assertions deterministic

Do not assert real bandwidth numbers. Focus on:

  • HTTP availability of expected files/endpoints
  • correct redirects/switching behavior
  • expected UI controls rendered
  • expected server-list loading behavior
  • ability to initiate/abort flow at UI level

2. Separate test types

  • Mode smoke tests (fast, always-run): verify each runtime mode serves the right surfaces.
  • UI mode tests: verify classic/modern pages and switcher rules.
  • Optional flow tests (later): mock Speedtest in browser to simulate state changes and verify UI updates.

3. Use Docker Compose as the environment contract

Run Playwright against containers started with explicit MODE values to mirror production entrypoint behavior.

Proposed Test Matrix

A) standalone

Expectations:

  • GET / responds and serves UI (classic by default unless overridden)
  • GET /backend/empty.php, GET /backend/garbage.php, GET /backend/getIP.php available
  • GET /results/telemetry.php reachable (even if telemetry disabled behavior differs)
  • GET /index.html?design=new resolves to modern page
  • GET /index.html?design=old resolves to classic page

B) backend

Expectations:

  • backend endpoints return success (/backend/empty.php, /backend/garbage.php, /backend/getIP.php)
  • tests only assert local backend endpoint contracts in this mode

C) frontend

Expectations:

  • UI entrypoint available
  • server list loads from /servers.json (or SERVER_LIST_URL if set)
  • backend test endpoints should not be treated as local testpoint contract in this mode
  • selecting server and pressing start does not crash UI shell

D) dual

Expectations:

  • combines frontend + local backend availability
  • UI can load multi-server list
  • backend endpoints available locally

Playwright Architecture

Files

  • playwright.config.js
  • tests/e2e/modes.spec.js (runtime-mode smoke)
  • tests/e2e/design-switch.spec.js (classic/modern/switch overrides)
  • tests/e2e/classic-standalone-regression.spec.js (revert regression guard)
  • tests/e2e/helpers/env.js (base URLs + mode metadata)
  • tests/e2e/helpers/ui.js (shared selectors, start/abort helpers)

Environment boot

  • docker compose -f tests/docker-compose-playwright.yml up -d --build
  • dedicate one service per runtime mode on separate ports
  • for frontend and dual, mount a stable servers.json

Selector policy

Use role/text selectors anchored on stable labels and IDs already in pages; avoid brittle CSS-path selectors.

Phased Rollout

Phase 1 (recommended first PR)

  • Add Playwright scaffolding and CI job
  • Add smoke coverage for 4 Docker runtime modes
  • Add design switch tests (index.html, ?design=new, ?design=old)
  • No full speed measurement assertions

Phase 2

  • Add deterministic UI flow tests with mocked Speedtest state updates
  • Validate button states (Start -> running -> abort/end)
  • Validate result widgets receive simulated values

Phase 3

  • Add telemetry-enabled scenario tests (TELEMETRY=true) for link visibility and stats exposure
  • Add negative tests (missing/invalid servers.json in frontend/dual)

Risks and Mitigations

  • Flaky speed measurements due to host/network variance
    • Mitigation: avoid throughput assertions; use mocked state for UI behavior.
  • Divergence between local static run and Docker entrypoint behavior
    • Mitigation: run all mode tests against Docker services.
  • Selector drift between classic and modern UIs
    • Mitigation: maintain per-design helper selectors with minimal coupling.

CI Proposal

  • Docker workflow runs e2e first, then build/push only if e2e passes
  • Standalone Playwright workflow is workflow_dispatch only for manual branch runs
  • Keep a single automatic e2e path to avoid duplicate runs
  • Playwright retries: 1 in CI, 0 locally
  • Upload traces/screenshots on failure only
  • Browser scope for v1: Chromium only

Confirmed Decisions

  1. Browser scope for v1: Chromium only.
  2. Telemetry checks are deferred to Phase 3.
  3. backend mode tests assert backend endpoint contracts only.
  4. Automatic e2e gating lives in Docker workflow; standalone Playwright workflow is manual.