mirror of
https://github.com/docker/compose.git
synced 2026-11-17 13:45:39 +00:00
Checks are the shared vocabulary between scenarios: move them to a dedicated checks.go so the whole vocabulary is read — and reviewed — as one file, independently of the tests using it. SCENARIO.md codifies the rules that so far only lived in the PR description: one intent = one invariant, step names as behavior sentences, inline compose model, state-based checks first with OutputContains as an explicit last resort, new checks generic and named after their observable — plus how to exploit the failure artifacts and E2E_KEEP_FAILED. AGENTS.md points new e2e work at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
4 KiB
4 KiB
E2E scenarios: the contract
With coding agents writing most of the production code, e2e tests are the
document humans actually read and review. A scenario must state an intent, a
compose model, and a sequence of steps (command → expected observables) —
and nothing else. Everything operational (project naming, cleanup, failure
diagnostics) belongs to the framework, not to the test.
The DSL lives in scenario.go (execution, actions,
requirements) and checks.go (the vocabulary of observables).
Writing a scenario
func TestRestart(t *testing.T) {
NewScenario(t, "restart must bring an exited service back up, restarting the same container").
Compose(`
services:
app:
image: alpine
init: true
command: ash -c "if [[ -f /tmp/restart.lock ]] ; then sleep infinity; else touch /tmp/restart.lock; fi"
`).
Step("up starts the service, whose first run exits at once",
ComposeCmd("up", "-d"),
Eventually(ServiceState("app", "exited"), 10*time.Second)).
Step("restart brings the service back up, reusing the container",
ComposeCmd("restart"),
Eventually(ServiceState("app", "running"), 10*time.Second),
NotRecreated("app"))
}
Rules:
- New e2e tests use
NewScenario. The legacyNewCLIstyle remains for existing tests, converted opportunistically; don't add to it. - One intent = one invariant. The intent is a one-line statement of the behavior being locked, phrased as an obligation ("X must Y"). If you need two intents, write two scenarios.
- Step names are behavior sentences, not command echoes: "an unchanged create is a no-op", not "run create again". The transcript of step names should read as the specification.
- The compose model is inline. A scenario is self-contained in the test
source; no fixture directories. Interpolate runtime values via
Env. - Regression tests link the issue in a comment above the test, with a sentence on the failure mode being locked.
Checks: observe real state
Checks are the shared vocabulary between scenarios; their discipline is what keeps the contract meaningful.
- Prefer state-based checks (
ServiceState,NotRecreated,LabelSet,RunsOnPlatform, …): they observe containers, labels and image manifests — what the user actually gets — not what the CLI printed. OutputContainsis a last resort, legitimate only when the CLI's reported decision is itself the observable (e.g. "Skipped" vs "Pulled").- Never poll by hand: wrap a state check in
Eventually(check, timeout). Notime.Sleepin scenarios. - A new check must be generic — no test-specific logic — and named
after the observable it asserts, not after the test that needed it. It
goes in
checks.go, where the whole vocabulary is reviewed as one file. Before adding one, verify the observable isn't already expressible. - A check should also fail loudly on a broken precondition (e.g.
NotRecreatederrors if the service had no container before the step) rather than pass vacuously.
When a scenario fails
The report opens with everything needed to diagnose without re-running:
artifacts: <dir>— a stable per-project directory holding the untruncated material:compose.yaml,failure.txt, each step's full command and output (step-NN-*.txt),containers.txt,events.txt, full container logs (logs-*.txt) and the per-step state snapshots (snapshots.json). Read these before re-running anything.E2E_KEEP_FAILED=1— rerun with this set to skip teardown of failed scenarios: containers, volumes and networks stay alive fordocker inspect/exec. Clean up afterwards withdocker compose --project-name <project> down -v --remove-orphans.- The inline report shows the transcript (every step, exit code, duration), the failing step's output, project containers, engine events since the scenario started, and container log tails — truncated for readability; the artifacts have the full versions.
Run a single scenario with:
go test -tags e2e ./pkg/e2e/ -run TestRestart -v