compose/pkg/e2e/SCENARIO.md
Nicolas De Loof b2e3deb18e test(e2e): testdata/<TestName>/ anchors each scenario's project files
Each scenario resolves its project files from testdata/<TestName>/ by
convention: standalone compose files in their native format, directly
runnable with docker compose -f, copied to a temporary directory so the
committed files are never mutated. Subtests map to nested directories.

Ownership is strictly one test per directory: a check fails the suite on
any testdata directory no test owns, which is what keeps this from
degrading into a catch-all fixtures directory — no sharing, no
leftovers after a rename.

The inline Compose() declaration is removed; the demonstrator scenarios
move their models to testdata, including the multi-file build context
of TestUpBuildUnchangedContext (compose.yaml + Dockerfile + marker).

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00

102 lines
4.7 KiB
Markdown

# E2E scenarios: the contract
With coding agents writing most of the production code, e2e tests are the
document humans actually read and review. A scenario must state an intent, a
compose model, and a sequence of steps `(command → expected observables)` —
and nothing else. Everything operational (project naming, cleanup, failure
diagnostics) belongs to the framework, not to the test.
The DSL lives in [`scenario.go`](scenario.go) (execution, actions,
requirements) and [`checks.go`](checks.go) (the vocabulary of observables).
## Writing a scenario
The project files live in `testdata/<TestName>/` — standalone compose files,
directly runnable with `docker compose -f testdata/TestRestart/compose.yaml`:
```yaml
# testdata/TestRestart/compose.yaml
services:
app:
image: alpine
init: true
command: ash -c "if [[ -f /tmp/restart.lock ]] ; then sleep infinity; else touch /tmp/restart.lock; fi"
```
```go
func TestRestart(t *testing.T) {
NewScenario(t, "restart must bring an exited service back up, restarting the same container").
Step("up starts the service, whose first run exits at once",
ComposeCmd("up", "-d"),
Eventually(ServiceState("app", "exited"), 10*time.Second)).
Step("restart brings the service back up, reusing the container",
ComposeCmd("restart"),
Eventually(ServiceState("app", "running"), 10*time.Second),
NotRecreated("app"))
}
```
Rules:
- **New e2e tests use `NewScenario`.** The legacy `NewCLI` style remains for
existing tests, converted opportunistically; don't add to it.
- **One intent = one invariant.** The intent is a one-line statement of the
behavior being locked, phrased as an obligation ("X must Y"). If you need
two intents, write two scenarios.
- **Step names are behavior sentences**, not command echoes: "an unchanged
create is a no-op", not "run create again". The transcript of step names
should read as the specification.
- **The project files live in `testdata/<TestName>/`.** The scenario resolves
that directory by convention (subtests map to nested directories, following
`t.Name()`) and copies it to a temporary directory, so the committed files
are never mutated. The directory holds a `compose.yaml` plus whatever the
project needs (Dockerfile, env or config files) in their native format,
directly runnable outside the test. Ownership is strictly one test per
directory — no shared fixtures — and a check fails the suite on any
`testdata` directory no test owns. Interpolate runtime values via `Env`.
- **Regression tests link the issue** in a comment above the test, with a
sentence on the failure mode being locked.
## Checks: observe real state
Checks are the shared vocabulary between scenarios; their discipline is what
keeps the contract meaningful.
- **Prefer state-based checks** (`ServiceState`, `NotRecreated`, `LabelSet`,
`RunsOnPlatform`, …): they observe containers, labels and image manifests —
what the user actually gets — not what the CLI printed.
- **`OutputContains` is a last resort**, legitimate only when the CLI's
reported decision is itself the observable (e.g. "Skipped" vs "Pulled").
- **Never poll by hand**: wrap a state check in `Eventually(check, timeout)`.
No `time.Sleep` in scenarios.
- **A new check must be generic** — no test-specific logic — **and named
after the observable it asserts**, not after the test that needed it. It
goes in `checks.go`, where the whole vocabulary is reviewed as one file.
Before adding one, verify the observable isn't already expressible.
- A check should also fail loudly on a broken precondition (e.g.
`NotRecreated` errors if the service had no container before the step)
rather than pass vacuously.
## When a scenario fails
The report opens with everything needed to diagnose without re-running:
- **`artifacts: <dir>`** — a stable per-project directory holding the
untruncated material: `compose.yaml`, `failure.txt`, each step's full
command and output (`step-NN-*.txt`), `containers.txt`, `events.txt`, full
container logs (`logs-*.txt`) and the per-step state snapshots
(`snapshots.json`). Read these before re-running anything.
- **`E2E_KEEP_FAILED=1`** — rerun with this set to skip teardown of failed
scenarios: containers, volumes and networks stay alive for `docker
inspect`/`exec`. Clean up afterwards with
`docker compose --project-name <project> down -v --remove-orphans`.
- The inline report shows the transcript (every step, exit code, duration),
the failing step's output, project containers, engine events since the
scenario started, and container log tails — truncated for readability; the
artifacts have the full versions.
Run a single scenario with:
```sh
go test -tags e2e ./pkg/e2e/ -run TestRestart -v
```