mirror of
https://github.com/docker/compose.git
synced 2026-11-18 06:07:53 +00:00
Each scenario resolves its project files from testdata/<TestName>/ by convention: standalone compose files in their native format, directly runnable with docker compose -f, copied to a temporary directory so the committed files are never mutated. Subtests map to nested directories. Ownership is strictly one test per directory: a check fails the suite on any testdata directory no test owns, which is what keeps this from degrading into a catch-all fixtures directory — no sharing, no leftovers after a rename. The inline Compose() declaration is removed; the demonstrator scenarios move their models to testdata, including the multi-file build context of TestUpBuildUnchangedContext (compose.yaml + Dockerfile + marker). Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
102 lines
4.7 KiB
Markdown
102 lines
4.7 KiB
Markdown
# E2E scenarios: the contract
|
|
|
|
With coding agents writing most of the production code, e2e tests are the
|
|
document humans actually read and review. A scenario must state an intent, a
|
|
compose model, and a sequence of steps `(command → expected observables)` —
|
|
and nothing else. Everything operational (project naming, cleanup, failure
|
|
diagnostics) belongs to the framework, not to the test.
|
|
|
|
The DSL lives in [`scenario.go`](scenario.go) (execution, actions,
|
|
requirements) and [`checks.go`](checks.go) (the vocabulary of observables).
|
|
|
|
## Writing a scenario
|
|
|
|
The project files live in `testdata/<TestName>/` — standalone compose files,
|
|
directly runnable with `docker compose -f testdata/TestRestart/compose.yaml`:
|
|
|
|
```yaml
|
|
# testdata/TestRestart/compose.yaml
|
|
services:
|
|
app:
|
|
image: alpine
|
|
init: true
|
|
command: ash -c "if [[ -f /tmp/restart.lock ]] ; then sleep infinity; else touch /tmp/restart.lock; fi"
|
|
```
|
|
|
|
```go
|
|
func TestRestart(t *testing.T) {
|
|
NewScenario(t, "restart must bring an exited service back up, restarting the same container").
|
|
Step("up starts the service, whose first run exits at once",
|
|
ComposeCmd("up", "-d"),
|
|
Eventually(ServiceState("app", "exited"), 10*time.Second)).
|
|
Step("restart brings the service back up, reusing the container",
|
|
ComposeCmd("restart"),
|
|
Eventually(ServiceState("app", "running"), 10*time.Second),
|
|
NotRecreated("app"))
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
|
|
- **New e2e tests use `NewScenario`.** The legacy `NewCLI` style remains for
|
|
existing tests, converted opportunistically; don't add to it.
|
|
- **One intent = one invariant.** The intent is a one-line statement of the
|
|
behavior being locked, phrased as an obligation ("X must Y"). If you need
|
|
two intents, write two scenarios.
|
|
- **Step names are behavior sentences**, not command echoes: "an unchanged
|
|
create is a no-op", not "run create again". The transcript of step names
|
|
should read as the specification.
|
|
- **The project files live in `testdata/<TestName>/`.** The scenario resolves
|
|
that directory by convention (subtests map to nested directories, following
|
|
`t.Name()`) and copies it to a temporary directory, so the committed files
|
|
are never mutated. The directory holds a `compose.yaml` plus whatever the
|
|
project needs (Dockerfile, env or config files) in their native format,
|
|
directly runnable outside the test. Ownership is strictly one test per
|
|
directory — no shared fixtures — and a check fails the suite on any
|
|
`testdata` directory no test owns. Interpolate runtime values via `Env`.
|
|
- **Regression tests link the issue** in a comment above the test, with a
|
|
sentence on the failure mode being locked.
|
|
|
|
## Checks: observe real state
|
|
|
|
Checks are the shared vocabulary between scenarios; their discipline is what
|
|
keeps the contract meaningful.
|
|
|
|
- **Prefer state-based checks** (`ServiceState`, `NotRecreated`, `LabelSet`,
|
|
`RunsOnPlatform`, …): they observe containers, labels and image manifests —
|
|
what the user actually gets — not what the CLI printed.
|
|
- **`OutputContains` is a last resort**, legitimate only when the CLI's
|
|
reported decision is itself the observable (e.g. "Skipped" vs "Pulled").
|
|
- **Never poll by hand**: wrap a state check in `Eventually(check, timeout)`.
|
|
No `time.Sleep` in scenarios.
|
|
- **A new check must be generic** — no test-specific logic — **and named
|
|
after the observable it asserts**, not after the test that needed it. It
|
|
goes in `checks.go`, where the whole vocabulary is reviewed as one file.
|
|
Before adding one, verify the observable isn't already expressible.
|
|
- A check should also fail loudly on a broken precondition (e.g.
|
|
`NotRecreated` errors if the service had no container before the step)
|
|
rather than pass vacuously.
|
|
|
|
## When a scenario fails
|
|
|
|
The report opens with everything needed to diagnose without re-running:
|
|
|
|
- **`artifacts: <dir>`** — a stable per-project directory holding the
|
|
untruncated material: `compose.yaml`, `failure.txt`, each step's full
|
|
command and output (`step-NN-*.txt`), `containers.txt`, `events.txt`, full
|
|
container logs (`logs-*.txt`) and the per-step state snapshots
|
|
(`snapshots.json`). Read these before re-running anything.
|
|
- **`E2E_KEEP_FAILED=1`** — rerun with this set to skip teardown of failed
|
|
scenarios: containers, volumes and networks stay alive for `docker
|
|
inspect`/`exec`. Clean up afterwards with
|
|
`docker compose --project-name <project> down -v --remove-orphans`.
|
|
- The inline report shows the transcript (every step, exit code, duration),
|
|
the failing step's output, project containers, engine events since the
|
|
scenario started, and container log tails — truncated for readability; the
|
|
artifacts have the full versions.
|
|
|
|
Run a single scenario with:
|
|
|
|
```sh
|
|
go test -tags e2e ./pkg/e2e/ -run TestRestart -v
|
|
```
|