Commit graph

64 commits

Author SHA1 Message Date
Nicolas De Loof
cc5c4ef50f test(e2e): migrate start/stop legacy tests to scenarios
TestStartAlreadyRunning, TestStopAlreadyStopped, TestStartStopMultipleServices
and TestStartStopMultipleFiles now state their intent and observe container
state (running/exited, not recreated) instead of matching progress lines.
The shared fixtures/start-stop had no other user and is removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-03 15:01:50 +01:00
Nicolas De Loof
3b2f42df17 test(e2e): migrate TestUnusedMissingEnvFile to a scenario
State what the test locks (a missing env_file only breaks the commands that
need the service referencing it) as a sequence of steps, and observe the
containers (running, scaled, not created) rather than only exit codes.
Completion is covered in both plugin and standalone modes through CompleteCmd.
The fixtures/env_file directory had no other user and is removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-03 15:01:50 +01:00
Nicolas De Loof
813238a7c7 test(e2e): migrate TestEventsUnknownService to a scenario
Express the intent (events refuses a service outside the project) with the
Scenario DSL; the project now lives in testdata/TestEventsUnknownService.
The shared fixtures/simple-composefile is still used by another test.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-03 15:01:50 +01:00
Nicolas De Loof
82d8bcb288 fix(use_api_socket): hand the registry credentials over like docker run does
Align the credentials use_api_socket gives a service with what
`docker run --use-api-socket` does in the Docker CLI:

- the config file is created with mode 0600 instead of the 0444 default
  of a config file, so that only the container's root reads it;
- a service which sets DOCKER_CONFIG itself gets the socket but no
  credentials, as the comment of the function always said;
- nothing is injected, and DOCKER_CONFIG is not set, when the host has no
  registry credentials, and they are not looked up when no service is to
  get them.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-02 15:41:36 +01:00
hiroto.toyoda
385ee60741 fix(up): check a depends_on condition before the first poll tick
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
2026-10-27 11:06:08 +01:00
hiroto.toyoda
4d61d5a1f0 fix(up): honor -t when an attached up stops the application
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
2026-10-27 08:58:05 +01:00
Nicolas De Loof
821153b2e4 fix(start): give a started use_api_socket service its config file
A container created beforehand and only started by `compose start`
(`create` then `start`, `up --no-start` then `start`, or a dependency named
to `start`) came up with DOCKER_CONFIG pointing at /run/secrets/docker and no
file there: use_api_socket is applied by rewriting the project, and the
config file it adds is injected when a container starts, from the project the
start works on. Create and run applied the rewriting, start did not.

start() now applies it too, on the copy it already makes of the project, so
the caller's is left alone, and a project already rewritten (a dependency
started by run) is not rewritten twice. The rewriting tolerates a project
without environment or configs maps, as the one `compose start` loads.

up -d, a foreground up, scale and run already provided the file.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 17:16:24 +01:00
Nicolas De Loof
b08c543d23 test(e2e): a foreground up ignores the exit of a pre_start runner
A pre_start runner carries the project and service labels but not the one-off
label the monitor's event filter requires, so its exit never reaches the
session. Nothing locked that: a runner exiting successfully would otherwise
win over the service's own exit code under --exit-code-from, or abort the
application under --abort-on-container-exit.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 16:28:51 +01:00
Nicolas De Loof
8011e8a921 feat(up): interactive up runs on a single plan
Interactive up used to create() the project, then start() it: two
daemon snapshots, the second built from the project as the caller gave
it rather than the canonical one the Create phase ran on.

It now prepares one Create+Start plan, like detached up. The Create
phase runs under the "up" progress scope, then the attach/printer/
monitor session is set up exactly as before, and the Start phase is
dispatched last, outside that scope, on a context that survives
Ctrl+C, with the printer as the hook-log listener. A failed Create
phase reaches neither the session nor the Start phase, and a dry run
only ever runs the Create phase.

The monitor's starting point is the daemon's time, which is read right
before the Start phase: a daemon that answers with a time that can't be
used fails the up with an explicit error, with no container started. One
that can't be asked at all (a read-only socket proxy without the info
endpoint) doesn't: the monitor then follows live events, with a warning.

The session, the watcher and the teardown now use the plan's own
project, which carries what preparation resolved. That makes
use_api_socket work in a foreground up: its config file lives only in
that project. A scenario locks it.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 16:28:51 +01:00
hiroto.toyoda
fab1bb43a1 test(e2e): pin that down -v keeps the volumes the project does not own
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
2026-10-26 12:08:23 +01:00
Nicolas De Loof
34cae9461c test(e2e): lock pre_start's streamed output and tighten the foreground up scenarios
A foreground up does stream a pre_start hook's output into the session since
the hook's log stream is opened once it is started. The hook scenario had
stopped asserting it on the belief it never did; its fixture only wrote to a
file, so nothing could reach stdout. The pre_start now also prints a marker
the service never logs, asserted on stdout, and the scenario states a single
intent covering both hooks.

Also tighten the scenarios after review:
- assert a container's printed output on stdout rather than on the whole
  output, and add the state or negative checks that were missing
  (ContainerEnv after a recreation, web exited once its dependency was
  healthy, no output of the replaced container);
- the dependency timeout scenario carries the "deadline exceeded" guard of
  its detached counterpart, with a fixture aligned on it;
- drop the abort message and the useless wait from the immediate-exit
  scenario: up only returns once the application is stopped;
- rename the dry-run scenario after the others, minimize its fixture and stop
  coupling it to the dry-run end-of-output wording;
- explain the sleeps of the fixtures, and give the hook one a larger margin;
- build tag e2e only: nothing here is Unix-specific.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 10:52:50 +01:00
Nicolas De Loof
0dd199257f test(e2e): stop asserting pre_start output the foreground up never streams
The hook scenario matched "pre-start-hook-ran" in the output, but the
service's own line "app-read:pre-start-hook-ran" contains it, so the
assertion passed without any pre_start output reaching the session. A
foreground up streams post_start's output but not pre_start's today
(checked on main too). Lock what actually holds: pre_start's effect, read
by the service through a value distinct from anything it logs.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 10:52:50 +01:00
Nicolas De Loof
07306856b4 test(e2e): lock the behavior of the foreground up
Most e2e coverage of the start phase runs detached; the foreground
`up`, whose create and start phases are driven by separate code paths,
had none for hooks, health-gated dependencies, a container that exits as
soon as it starts, restarts of exited containers, recreation, dependency
timeouts or dry-run. Add scenarios that fix these observables before the
machinery behind it changes.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 10:52:50 +01:00
Nicolas De Loof
d099a67bb8 fix(executor): name the failed wait and close every row a failed start leaves
Follow-up to the report of the services a failed start interrupts:

- When it is the wait itself that fails (the dependency is unhealthy, or
  exited with an error), the held service reads which condition was not met
  instead of an interruption by the internal name of the wait node.
- Only the start phase reports what it gives up: the create phase stays
  silent as before.
- An inspection cut short by the cancellation ends the wait as interrupted,
  and no longer marks the healthy dependency as failed. A caller canceled
  (Ctrl-C) at that point ends the wait as it always did.
- Two nodes failing together each end their own row with their own error,
  whichever one cancels the phase, instead of leaving one on "Starting" or
  reading "interrupted" for a failure that was its own.
- A replica whose start is behind it reads that only its post_start hooks
  were interrupted.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 09:54:25 +01:00
Nicolas De Loof
9a2d0d94ca fix(executor): report the services a failed start interrupts
When a node of a plan phase fails, the executor cancels the whole phase.
The nodes it interrupts, or had not dispatched yet, got no final event:
a service recreated by the command and held by a depends_on wait stayed on
"Recreated", as if it were up, and the row of the dependency it was
waiting for stayed on "Waiting". Nothing said the service was left created
and never started.

Such a node now ends on an Error event, "not started" when its group never
reported Working, which names the depends_on condition it was held by and
the resource whose failure interrupted the phase. A wait interrupted that
way is a failure of its own (the condition was not reached) and closes the
"Waiting" rows of its dependency with a Skipped event. A phase canceled by
its caller (Ctrl-C) is reported as before: silently.

docker/compose#14372

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-26 09:54:25 +01:00
Nicolas De Loof
2ee07f440a feat: support the umask service attribute
Pass `umask` to the engine as HostConfig.Umask when creating a
container, so that the entrypoint, exec and healthcheck processes run
with the requested file mode creation mask. The attribute is part of the
container specification, so pre_start init containers, created the same
way, get the umask of their service unless they declare their own.

Engines older than API 1.56 (Docker Engine 29.8) silently ignore the
field, which would run the container with the runtime's default umask
without any notice: creating a container with a umask against such an
engine now fails with an explicit error, like interface_name or image
volumes do.

The umask is part of the service configuration hash, so changing it
recreates the container; the hash of every other configuration is
untouched.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-21 14:17:44 +02:00
hiroto.toyoda
412515250a fix(run): keep a recreated dependency's anonymous volumes
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
2026-10-20 11:54:18 +02:00
Nicolas De Loof
ea2284d712 fix(pre_start): capture a hook's output by opening its log stream once started
A pre_start hook's output never reached anyone. The follow log stream was
opened on the runner while it was still in "created" state, before
ContainerStart; the engine ends such a stream at once, empty. So a foreground
up never showed the hook's output, and the error of a failing hook carried no
tail, though both are built from that stream.

The stream is now opened once the hook is started. The runner is retained
after it exits (AutoRemove is off), so a fast-exiting hook cannot make the
stream hit a 404, which is what opening it first was meant to prevent. Once the
hook has exited, the stream gets a short, bounded delay to deliver what the
daemon buffered before it is cut, instead of being cut right away.

The unit tests that pinned the old call order now pin the new one, and a
scenario locks that a failing hook reports what it printed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-16 15:11:33 +02:00
Nicolas De Loof
42c2b0d85d test(e2e): lock --wait-timeout on a depends_on service_healthy dependency
The wiring of --wait-timeout into the plan's dependency waits was only
covered by executor tests that set the timeout by hand, and by an e2e
without depends_on. The scenario has a dependency that never gets
healthy and checks that up -d --wait-timeout fails with the dependency
timeout message (without the raw deadline error) and leaves the
dependent service created but not started.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-15 14:52:20 +02:00
Nicolas De Loof
77f57a9c8a feat(run): --no-recreate reuses existing dependencies as-is
The dependencies a one-off starts go through the regular create path
and its config-hash divergence check, with no way to opt out: any
resolution difference with the command that created them (environment,
model edit) recreates the dependency — destroying state a previous run
built up (issue #13769's database losing its users). up already offers
--no-recreate for exactly this; run now aligns: the flag maps to the
"never" recreate policy, propagated through the RunOptions.CreateOptions
fields that startDependencies previously dropped. Default behavior is
unchanged (empty policy = "diverged").

Closes #13769

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-14 09:34:19 +02:00
Nicolas De Loof
eef29d10c3 test(e2e): pre_start hooks must receive the service's env_file and environment
Reproduces docker/compose#14287: a hook inheriting the service's
env_file (or declaring its own) runs without those variables.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-12 18:15:14 +02:00
Nicolas De Loof
eb0fb65e0c docs(provider), e2e: image distribution documented and exercised
extension.md gains the Image distribution section: invocation contract
(--image/--digest/--created/--source/--policy), the get-image request
and its chunked image-stream answer (zero-chunk terminated, stdin
exclusivity and drain-before-next-answer contract), the two-branch
decision guidance (digest as identity, created as ordering fallback,
reproducible builds caveat), and the support-by-presence metadata
convention is now stated as the general rule. The example provider
implements pull end to end — request, stock chunked reader, hash
recorded at PROVIDER_PULL_MARKER — and the e2e scenario locks the
whole path: up builds the service image and streams it to the provider
under the local verdict, compose pull re-invokes it under the
freshness contract.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-10-02 17:35:20 +02:00
Nicolas De Loof
0a64ff09a8 fix: address copilot-pull-request-reviewer's 3 findings on the hook-runner plan
A GitHub Copilot PR review (docker/compose#14221#pullrequestreview-5300897216)
found three real issues in the reconciliation-plan-based hook runners, all
confirmed by direct reproduction against a live daemon before being fixed:

- **Hook runners excluded from discovery, stale runners persist.**
  collectObservedState's single getContainers call requires ConfigHashLabel
  (getDefaultFilters), which hook runners deliberately never carry -- so they
  never reached the classification loop that populates HookContainers, and
  planPreStartHookRunners never saw a stale runner to purge. Reproduced: a
  failed hook run followed by a second `up` failed with "container name ...
  already in use" instead of recovering. Fixed by listing hook containers
  separately, by project+hook label alone (new getHookContainers), and
  merging them in before classification. Locked in by a new e2e scenario,
  TestPreStartHookRetryAfterFailureSucceeds, that reproduces the exact
  failure-then-retry sequence.

- **Prepared hook runners misclassified as service replicas.** start's
  containers.filter(isService(name), isNotOneOff) has no reason to exclude a
  hook runner: it carries the same project/service labels as a real replica
  and no one-off label either. Added isNotHookContainer to the filter chain
  in startService, so a hook runner is never counted as (or started as) a
  surviving replica.

- **Per-replica hooks create runners before unsupported configuration is
  rejected.** per_replica is not yet supported (#14259): runPreStart
  validates every hook up front and rejects the whole service before
  executing any of them, but planPreStartHookRunners had no equivalent
  check, so create/up would plan (and leave behind) runners for a service
  whose hooks could never run. Mirrored the same up-front, all-or-nothing
  validation in planPreStartHookRunners.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-30 12:03:10 +02:00
Nicolas De Loof
eaf172eab2 feat: reconciliation plan creates pre_start hook runners
pre_start hook containers become first-class resources of the
reconciliation plan. The reconciler plans one OpCreateHookContainer per
declared hook — gated on the same predicate the start phase uses to run
the hooks: a replica to start and no running replica SURVIVING the plan
(a recreated replica is not running at start time, so its runners are
planned too). Runner creation depends on every replica node (the
executor resolves the VolumesFrom target against a final live view),
on the infrastructure, and on the purge of previously observed runners:
names are deterministic (<project>-<service>-pre_start-<i>) so repeated
plans converge on the same container instead of accumulating anonymous
ones, and a new HookIndexLabel ties each runner to its hook.

runPreStart becomes pure execution: it looks up the created-state
runner for each hook by label and never creates one. A declared hook
without a prepared runner is an actionable error naming the
reconciliation command (docker compose up <service>) — the accepted
consequence is that stop-then-start of a hooked service errors, since
runners are consumed on success (locked by e2e). Success removal,
failure retention and cancellation cleanup are unchanged.

Runners stay invisible to the start/ps container listings — they carry
no ConfigHashLabel, which getDefaultFilters requires — and down keeps
removing them by hook label.

Rebased onto main, which diverged substantially since this branch was
opened (the unrelated container-spec-layering refactor, #14093,
restructured compose-go's ServiceConfig/PreStartHook types this PR also
touches). Reconciling pre_start.go/pre_start_test.go against that
refactor surfaced a real bug: createPreStartContainer builds the hook's
config via the shared getCreateConfigs, which unconditionally stamps a
ConfigHashLabel — silently contradicting this same commit's own
"invisible to ps/start" guarantee above, since that label is exactly
what getDefaultFilters requires. Fixed by deleting the label after
getCreateConfigs returns, with a test asserting its absence (an
independent local review, via the GitHub Copilot CLI, and a
fresh-context review agent both converged on catching this during
rebase verification). Also restored two security/label tests that only
existed on one side of the rebase (SecuritySensitiveFieldsHonored,
HookLabelsMerged), ported to call createPreStartContainer directly.

Also addresses glours' three review comments on this PR: hook-container
create operations for a service no longer chain serially (each depends
on the same shared containerNodes+infraDeps+purges set, letting the
executor create them in parallel), the corresponding plan-string test
assertion, and a clearer missing-runner error message hinting this
isn't necessarily a planning bug.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-30 12:03:10 +02:00
Nicolas De Loof
32bddfc4c6 fix: provider migration condemns the service's stale replicas
Some checks failed
ci / validate (lint) (push) Has been cancelled
ci / validate (relay-lint) (push) Has been cancelled
ci / validate (relay-test) (push) Has been cancelled
ci / validate (validate-docs) (push) Has been cancelled
ci / validate (validate-go-mod) (push) Has been cancelled
ci / validate (validate-headers) (push) Has been cancelled
ci / validate (validate-mocks) (push) Has been cancelled
ci / binary (push) Has been cancelled
ci / bin-image-test (push) Has been cancelled
ci / relay-image-test (push) Has been cancelled
ci / test (push) Has been cancelled
ci / e2e (plugin, oldstable, graphdriver) (push) Has been cancelled
ci / e2e (standalone, oldstable, graphdriver) (push) Has been cancelled
ci / e2e (plugin, stable, containerd) (push) Has been cancelled
ci / e2e (plugin, stable, graphdriver) (push) Has been cancelled
ci / e2e (standalone, stable, containerd) (push) Has been cancelled
ci / e2e (standalone, stable, graphdriver) (push) Has been cancelled
merge / bin-image-prepare (push) Has been cancelled
merge / relay-image (push) Has been cancelled
merge / module-image (push) Has been cancelled
Scorecards supply-chain security / Scorecards analysis (push) Has been cancelled
zizmor / zizmor (push) Has been cancelled
ci / binary-finalize (push) Has been cancelled
ci / coverage (push) Has been cancelled
ci / release (push) Has been cancelled
merge / bin-image (push) Has been cancelled
A service migrated from regular replicas to a provider left its old
containers standing: not converged (a provider service has no replicas
to converge), not orphaned (the service is still in the model). The
first up after the migration then failed creating the relay, because
the leftover replica still held the service's canonical container name
the relay takes over. And even without the name conflict (replicas
2..n of a scaled service), stale replicas kept the service's network
alias, competing with the relay on DNS and serving outdated traffic.

The reconciler now plans a stop+remove for every observed non-relay
container of a provider service, and the RunProvider node depends on
those removals: the canonical name is free by the time
ensureServiceRelay creates the relay. The relay itself (RelayLabel) is
the provider service's legitimate container and is left alone —
converged by ensureServiceRelay, not by the plan.

The e2e scenario migrates a deployed service to a provider and locks
the replacement (Recreated, relay reachable at the compose-native
address); the Scenario DSL gains FromFile — a local mirror of
FromRemote — to switch the model between steps.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-25 15:58:18 +02:00
Nicolas De Loof
207a3eb6f7 test(e2e): reproduce --project-directory ignored on an oci:// project
TestOciRemoteProjectDirectory publishes a project whose only service
mounts a relative volume, then runs `up` on the oci:// artifact with an
explicit --project-directory pointing elsewhere. The relative volume
must resolve against that directory.

It currently fails against the pinned compose-go: LoadConfigFiles
defaults the working dir to the downloaded artifact's own cache
directory whenever a remote resource loader is involved, silently
overriding the explicit --project-directory the same way it would a
mere default -- docker/compose#14224. On Docker Desktop this surfaces
as a hard failure (the cache directory isn't a shared mount); on Linux
it silently mounts the wrong directory, matching the original report.

New BindMountSource check (pkg/e2e/checks.go) pins a service's bind
mount source to an exact expected path, for tests that need to verify
which working directory a relative volume path resolved against.

Needs the compose-go fix (github.com/ndeloof/compose-go@a626c70,
branch 14224-working-dir) to pass -- not yet reflected in go.mod here.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-23 10:56:23 +02:00
Nicolas De Loof
7e23119934 test(e2e): reproduce #14223 -- additional_contexts to a disabled service
A service reachable only as another service's build additional_contexts
must still build even when its own profile is inactive: it's referenced
for its image, not started as a workload -- the same tolerance the
depends_on consistency check already grants a disabled, non-required
dependency, and the same case addBuildDependencies already special-cases
on the docker/compose side.

compose-go's own consistency check doesn't grant that tolerance yet: it
rejects the reference outright, failing the whole project load before
addBuildDependencies (or anything else, up included) ever runs -- for
up, build and up --no-build alike, exactly as reported.

This is expected to fail until compose-go bumps to include
compose-spec/compose-go#931.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 18:01:16 +02:00
Nicolas De Loof
0d206e6704 fix(compose): move the job-target refusal into projectOrName itself
The previous fix only covered create/start. projectOrName backs every
other service-targeting command too (stop, kill, pause, unpause, logs,
rm, down, ps, events), so they still surfaced the same two problems: a
raw "no such service" error for a job name, and -- when
COMPOSE_PROJECT_NAME is set -- a silent no-op, since projectOrName's
label-driven fallback swallows the load error entirely.

Fold the check into projectOrName so every caller is fixed at once,
instead of duplicating the wiring per command. start.go's own
job-handling code, now redundant, is removed; jobTargetErr and
unselectedJobs (cmd/compose/run.go) are unchanged, just called from one
more place.

Adds unit coverage for projectOrName itself (including the
COMPOSE_PROJECT_NAME path and the typo-correlation edge case) and e2e
coverage for two more callers (stop, down) to confirm the fix lives in
the shared helper rather than being re-implemented per command.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
039413cc77 fix(start): still refuse a job when COMPOSE_PROJECT_NAME masks the error
projectOrName falls back to a label-driven, file-less project on any
load failure when COMPOSE_PROJECT_NAME is set -- swallowing the load
error entirely, including "no such service" for a target that's
actually a declared job. Since that branch returns err == nil,
jobTargetErr (gated on err != nil) and rejectScheduledJobs (gated on
project != nil) were both skipped: `docker compose start migrate`
fell through to a label-driven start that finds no container for a
job that was never run, and exited 0 having silently done nothing.

Refuse a named job explicitly on that fallback path too, reusing
unselectedJobs from the previous commit.

Also adds the test coverage this and the previous fix (job-target
error messages on create/start) were entirely missing: create.go and
start.go had no unit tests at all, and pkg/e2e/jobs_test.go only
exercised up/run.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
56bc854f37 fix(run): manual: false also blocks a job pulled in transitively
materializeJobClosure walked job-typed depends_on edges without
re-checking triggers.manual: false on anything but the top-level run
target, so `docker compose run A` where A depends_on job B (manual:
false) silently ran B anyway.

manual: false declares a job harmful to trigger outside its
schedule; depends_on doesn't change who caused the execution or when
-- pulling B in to satisfy A's dependency is still the run command
causing that out-of-schedule execution, one hop removed. Refuse it
the same way the top-level check already does, before anything in the
closure is created.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
80cae9a7a5 pre_start: consume the load-time-resolved hook specification
compose-go resolves each hook against its service at load time: the
model itself carries the full container specification a hook runs with,
and the runtime consumes it as-is through the standard create path —
the runtime-side merge helper goes away.

Consuming the full spec means honoring ALL of it:
- service references it may carry — volumes_from entries, and
  service:-scoped network_mode/ipc/pid, inherited or declared — resolve
  to live container IDs exactly like the service create path does; the
  daemon knows nothing about service names and rejected them, failing
  up for any service combining volumes_from with a pre_start hook
  (locked by an e2e scenario).
- hook labels — declared or inherited — merge into the container's
  labels, the runtime identification set winning on conflicts.
- the <API 1.44 network-connect fallback joins the HOOK's networks,
  not the parent service's: a hook overriding networks was connected
  to the wrong ones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
2804f9d4ae run: honor the explicit manual: false opt-out on jobs
Per the spec, any job can be triggered manually by an explicit run
command, its automated triggers notwithstanding — so run now accepts
scheduled jobs too. The exception is a job explicitly declared with
triggers.manual: false, which run rejects: meant for scheduled jobs
whose out-of-schedule execution would be harmful.

compose-go is bumped to the jobs-branch commit making Manual tri-state
(*bool) and allowing manual and schedule to be combined.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
135fb0ed3a e2e: jobs scenarios — scheduled rejection, manual run, env_file, job chain
Scenario-DSL coverage for the jobs entry points:
- up refuses a project declaring an active scheduled job before creating
  any resource;
- run executes a manual job like a service, starting its depends_on
  services first, and refuses a schedule-only job;
- a job's own env_file feeds its environment through run — the
  materialization happens before environment resolution;
- a job depending on another job runs the dependency to completion
  first, through the exact machinery a service dependency uses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
ef277d266b pre_start: inherit the whole container specification generically
pre_start hooks carry the full container specification, but the
init-container runner only consumed a handful of attributes (image,
command, user, env, workdir) — everything else was silently dropped.

Instead of wiring attributes one by one, the hook's specification is
merged over the service's ContainerSpec through compose-go's own file
merge machinery (override.Merge on the canonical yaml tree): command
and entrypoint replace, environment merges per key with the hook
winning, extra_hosts and dns accumulate entries, ulimits merge — the
exact per-attribute rules of multi-file compose, maintained in one
place. Every ContainerSpec attribute inherits this way, current and
future, with zero attribute-specific code.

The merged spec then runs through the standard create path
(getCreateConfigs) as a service-shaped one-off, so resources,
capabilities, dns, sysctls, logging... materialize exactly as they
would for a service container. Hook containers keep their minimal
labels and carry no container-number, so tooling telling replicas
apart does not count them.

The only deliberate exception is volumes: mounts inherit at runtime
through volumes_from — the only mechanism that shares the service's
anonymous and image volumes — and the hook's own volume declarations,
materialized by the standard path, take precedence per target. This is
what lets an init container get read-write access to a volume the
service mounts read-only (fixes: see PR).

e2e scenarios lock extra_hosts inheritance and accumulation, volume
override and completion, and the unit tests pin the merge rules.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-21 16:35:20 +02:00
Nicolas De Loof
aab5819147 feat(provider): publish-endpoint deploys a network relay for provider services
A provider's resource lives outside the compose network: consumers could
only reach it through injected variables carrying a host-published
address — nothing like the compose-native experience of addressing a
service by name at its well-known port.

A provider may now publish where each endpoint of its resource actually
listens:

    {"type": "publish-endpoint", "message": "80=localhost:49152"}

The endpoint is announced as seen from the provider's host: the relay —
the component that knows it runs inside a container — rewrites loopback
or unspecified upstream hosts to host.docker.internal (resolved through
its injected host-gateway extra_host); routable addresses pass through.

When at least one endpoint is published, compose deploys a relay
container in place of the service: a minimal TCP forwarder (new relay/
directory, published as docker/compose-relay, overridable with
COMPOSE_RELAY_IMAGE for internal registries) joining the networks of the
services that depend on the provider service, aliased with the service
name. Consumers then use http://<service>:<port> as if the service were
a regular container.

The relay is a first-class project container — canonical name, standard
compose labels including config-hash (label-driven commands run without
the compose file keep seeing the service: ps, logs, stop, down) — plus
the com.docker.compose.relay label declaring its role:

- the reconciler already leaves provider services' containers alone, and
  the relay's identity hash (image + routes) makes up idempotent: kept
  when routes are unchanged, recreated otherwise;
- process-level commands (exec, cp) refuse a relay — there is no service
  process in it to act on;
- the up monitor excludes relays from the containers whose termination
  ends an attached up: they are long-lived infrastructure and would
  otherwise keep 'up' waiting forever.

The example provider demonstrates the flow behind PROVIDER_DEMO_ENDPOINT
(a detached helper serving a fixed HTTP response), backed by an e2e
scenario asserting the compose-native address works and exec is refused.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-16 17:03:23 +02:00
Nicolas De Loof
3a5353d4ec e2e: cover git and OCI remote stacks
The remote loaders had almost no end-to-end coverage: one test consumed
an OCI artifact (config only, plus an up declined at the prompt) and
nothing exercised the git loader at all.

Git scenarios run against a throwaway repository served over the smart
HTTP protocol by an in-process server — git http-backend as a CGI per
httptest request: no daemon, no container, no fixed port. Smart HTTP is
a hard requirement: the loader ls-remotes the ref then shallow-fetches
the raw commit, which the dumb protocol cannot serve (no shallow
capability) and which needs uploadpack.allowAnySHA1InWant. Covered:
deploying from the default branch with repository-relative files,
selecting a #branch, selecting a #ref:subdir project.

OCI scenarios publish the fixture to a throwaway local registry (the
TestPublish pattern) then deploy from oci://: a full up consuming the
bundled env-file layer, and tag selection between two published
revisions.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-16 16:53:33 +02:00
Guillaume Lours
22d170f807 test(e2e): add attached up regression test for #13985
Unit tests alone didn't cover the exact failure: an attached up
hanging when the project is stopped/removed externally while a
service sits in restart backoff. The new test reproduces that
sequence end to end, so a future regression on this path breaks
loudly instead of slipping through as a passing unit suite. A
RestartCount guard narrows the race with the backoff expiring, so a
missed window fails loudly instead of passing silently through the
pre-existing die-based path.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-09-16 07:54:38 +02:00
Guillaume Lours
8018a96fde feat(compose): warn on unsupported compose-file attributes
Compose silently ignored several spec-valid attributes it doesn't
honor outside Swarm mode: deploy.update_config, credential_spec,
ports[].mode: host, and others had no visible effect on a plain
`docker compose up`.

Detection is wired into compose-go's loader as a fail-closed
allowlist: every attribute path the schema declares, minus the
handful this runtime deliberately doesn't implement, plus a few
value-conditional checks for attributes only unsupported for one
specific value. A newly-specified attribute gets flagged
automatically, with no corresponding code change needed here.

Depends on an unreleased compose-go change; go.mod carries a
temporary replace to a fork branch until it's released.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-09-09 16:32:04 +02:00
Nicolas De Loof
a255af9368 feat(provider): get-service-config provider request
Providers could not see the definition of the service they manage:
options had to be duplicated between the compose file and the provider,
or the provider had to re-resolve the model on its own.

A provider may now emit {"type": "get-service-config"} on stdout;
compose answers on the provider's stdin with one JSON line holding the
resolved canonical configuration of the provider's own service,
straight from the in-memory model. The message can be repeated; each
occurrence is answered with one line. Detection is by construction: a
compose that predates the message aborts on it and never writes to
stdin, so the provider treats EOF as 'unsupported, upgrade compose'.

The example provider demonstrates the round trip, backed by an e2e
scenario; unit tests drive executePlugin against a helper-process
provider and cover the injection.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-09-08 14:53:14 +02:00
Guillaume Lours
a896529193 fix(bridge): confirm before wiping non-empty convert output dir
`bridge convert -o <dir>` unconditionally ran `os.RemoveAll` on the
output directory and discarded any error, so a typo (`-o .`, `-o
$PWD`, `-o ~`) silently destroyed unrelated user data.

An empty or missing output directory is still created silently, but a
non-empty one now requires explicit confirmation (prompted
interactively, or via the new `--yes` flag for scripts/CI) before
being wiped.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-09-03 14:38:41 +02:00
Nicolas De Loof
06f4f9d622 fix: dependency wait timeout surfaced only by luck
waitDependency returned nil whenever its context ended, so an expired
wait deadline was reported only when the 500ms poll ticker happened to
win the select race and the condition check then failed on the expired
context. Whether `up --wait --wait-timeout` failed or silently
succeeded after the timeout was a coin toss, and the "timeout waiting
for dependencies" mapping in waitDependencies was unreachable on the
direct path.

Return the context error when the deadline expired; a plain
cancellation (Ctrl-C) is still not reported as a dependency failure.
The unit test pins both sides deterministically (a deadline shorter
than the first tick used to be swallowed with certainty); the e2e
scenario documents the user-visible contract: `up --wait
--wait-timeout` on a never-healthy service fails with "application not
healthy after Xs" while the container keeps running.

Part of #14074 (section C) / #14081 (lot 0).

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-20 15:16:57 +02:00
Guillaume Lours
3a62ae63ce fix(ps): honor psFormat from Docker CLI config when no --format flag given
The --format flag defaulted to "table" instead of "", so the
opts.Format == "" guard in runPs never fired for a bare compose ps.
Setting the default to "" lets the existing config-fallback path work,
matching the behaviour of docker ps.

Fixes #13643
Supersedes #14032

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-08-20 14:05:34 +02:00
Guillaume Lours
8bd79da4bc e2e: fix test resource isolation and make standalone parallelism configurable
Several e2e tests used fixed host ports (8070, 8080, 8090, etc.), global
buildx builders registered with --use, and a hardcoded IPC container name.
Under parallel execution these shared daemon-level resources caused port
conflicts, buildkitd container-name collisions, and intermittent failures.
Separately, the standalone make target hardcoded -parallel=1, forcing the
entire standalone suite to run serially even though the same tests run in
parallel in plugin mode.

Changes:
- Convert fixed host ports in four fixtures (network-test, sentences,
  build-test, volume-test) to ephemeral bindings; add ServicePublishedPort
  helper to framework.go to resolve the actual mapped port at runtime.
- Give each buildx-builder test a unique daemon-scoped name via BuilderName
  helper, preventing container-name collisions between parallel goroutines.
  Applies to TestBuildPlatformsWithCorrectBuildxConfig, TestBuildPrivileged,
  TestBuildBuilder, TestBuildEntitlements, TestBuildTLS.
- Rename the fixed ipc_mode_container to a project-scoped name
  (ipc_e2e-src) and update the ipc-test fixture accordingly.
- Replace the hardcoded -parallel=1 in make e2e-compose-standalone with
  E2E_PARALLEL_PLUGIN?=4 and E2E_STANDALONE_PARALLEL?=4, aligning both
  modes with the effective GOMAXPROCS on 4-core GitHub Actions runners.
  Both variables are overridable (E2E_STANDALONE_PARALLEL=1 restores
  serial local debugging).
- Raise the poll delay in watch_test.go long-running polls (90s/120s
  timeout) from the default 100ms to 1s; bump cascade_test.go and
  publish_test.go from 100ms to 500ms. Reduces CPU pressure without
  affecting effective test duration.

The logging-driver fixture retains its fixed port 24224: the Docker fluentd
log driver encodes fluentd-address at service-start time, making ephemeral
resolution non-trivial. TestLoggingDriver is serial so there is no
collision risk in practice.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-08-19 17:41:02 +02:00
Nicolas De Loof
aef01ae584 test(e2e): batch snapshot inspect with per-container fallback, explicit suite timeout
The per-container inspect of the previous fix doubled the suite's
wall-clock on the containerd CI job — one docker CLI process per
container per snapshot — pushing it past go test's default 10m and
killing the run mid-flight. Snapshots go back to one batch inspect,
falling back to individual inspects only when the batch fails because
a helper container vanished. The e2e make targets also pass an
explicit -timeout 20m so a slow job fails on its own results, not on
the default timeout.

TestUpStopWithLogsMixed becomes deterministic: with a bounded ping the
service could exit on its own before the abort reached it and the
pre_stop hook never ran; it now pings forever so the abort always
interrupts it.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
1fa9ae160a test(e2e): image-identity, images and model scenarios
- TestUpIdempotentContainerdStore joins the scenario style of its
  corner-case siblings: Serial + ContainerdImageStore requirement, the
  regression story in a comment, NotRecreated as the invariant. The
  requireContainerdStore helper and the last identity fixture go away.
- TestImagesAfterImageRemoved scopes its throwaway tag to the project
  and observes container state before pulling the image record away.
- TestComposeModel inlines its model; verified in CI where the
  docker-model plugin is installed (the local runner cannot pull the
  model, as with the legacy test).

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
447157274e test(e2e): build scenarios — the 'Built' family inlines its models
TestBuildDependsOn, Subset, DependentImage, SubDependencies,
LongOutputLine, DependentImageWithProfile, Escaped, Tags and Secrets
become scenarios: models inline (the long-line Dockerfile generated
with strings.Repeat instead of 34 copy-pasted lines), image names
project-scoped with Defer cleanup, and the secrets build asserts
ImageExists on top of the Dockerfile's own diff checks.

Kept legacy: TestLocalComposeBuild (DOCKER_BUILDKIT=0/1 matrix over a
shared fixture with fixed names and host port), the buildx-builder
family (SSH, platforms, privileged, entitlements, builder, TLS) which
provisions builders and dind daemons, and TestBuildImageDependencies
(classic/bake matrix). Their fixtures under build-test/ stay; the
single-purpose subdirectories are removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
a7293130c6 test(e2e): compose command scenarios — rm, compatibility, config, orphan scoping
- TestRm, TestRemoveOrphaned and TestUnnecessaryResources observe which
  containers exist instead of grepping docker ps.
- TestCompatibility drops the heavyweight sentences fixture: underscore
  naming shows on any model.
- TestConfig/TestConfigInterpolate build their expected rendering from
  the scenario's own project name and directory; OutputMatches joins
  the vocabulary for TestInitContainer's ordering assertion.
- TestStopWithDependenciesAttached inlines its model.
- Kept legacy: TestLocalComposeUp (fixed port, HTTP, shared sentences
  fixture), TestDownComposefileInParentFolder (project name inferred
  from a fixture-relative directory), TestAttachRestart (long-running
  attached up), the COMPOSE_FILE-from-.env family and TestNestedDotEnv
  (they exercise invocation without -f, which ComposeCmd always sets).
- Fixtures init-container/, external/ and dependencies/ are removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
37f57744de test(e2e): exec and provider scenarios
- TestExec locks the container-selection order (service container over
  one-off) with one-off state observed at each step.
- providers: a shared providerScenario helper wires the
  example-provider binary into PATH; the env assertions anchor on the
  'test-1  | ' log prefix, replacing the hand-rolled env parser, and
  the stop-hook sentinel file is observed with FileExists.
- Fixtures exec/ and providers/ are removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
f4f6ac5324 test(e2e): volume and build-once scenarios
- volumes: external-volume switching, volume recreation on definition
  change and image volumes become scenarios; the two-compose-file
  fixtures collapse into one model with an interpolated volume name or
  label, so the changed definition is visible in the test. The rebuild
  scenario locks the Recreated expectation and scopes the source image
  to the project. TestLocalComposeVolume (fixed host port, HTTP) and
  TestProjectVolumeBind (host file written mid-test) stay legacy.
- build-once: the three run --build regression cases each get a
  scenario; OutputMatchesCount joins the vocabulary to count the build
  reports, replacing the hand-rolled regex counter, and image names are
  project-scoped instead of random suffixes.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00
Nicolas De Loof
ee7f323fed test(e2e): run and up scenarios
- compose run splits into seven scenarios (one-off basics, ports, deps,
  optional deps, quiet pull, --pull always, chained build contexts).
  The orphan-warning cases fold into the basic scenario: an exited
  one-off IS the orphan the next run warns about, no second compose
  file needed. Piped-stdin and stop-signal tests stay legacy.
- up/compose_up: unhealthy service, dependency exit (both now also
  locking that dependents stay in created state), build dependencies
  (project-scoped image name instead of the fixture's global one),
  optional dependency, --all-resources, profile targeting (dropping the
  fixture's global container_names), image-by-ID, exit-code-from (42
  and 143), port ranges, stdout/stderr separation (StderrContains joins
  the vocabulary) and logging-driver reconfiguration (now locking the
  Recreated expectation). The Ctrl-C dependency test stays legacy.
- Fixtures start-fail/, resources/, stop/, port-range/, stdout-stderr/,
  logging-driver/, profiles/ and most of dependencies/ and run-test/
  are removed.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 16:49:56 +02:00