The header of service_containers.go still listed the dependency waits among
what it covers.
Comments only.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
waitDependencies, the polling of a depends_on condition (waitDependency,
waitDependencyReached and the per-condition checks), shouldWaitForDependency
and the container health probes they use leave service_containers.go for
dependency_wait.go, and their tests for dependency_wait_test.go. The
remaining clients of the waits are the plan's OpWaitCondition nodes,
restart, run and the --wait verification of up and start.
A pure move: every declaration, its comments included, is unchanged.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Say that a relay restarted by a start plan does take startMx while the
deployment of the relay does not, that the lock was unlikely (rather than
unable) to help against #12846, and name isReplicaRelay, which is what
gates pre_start. Rewrap the test comment.
Comments only.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
docker/compose#12846 was not a race between container starts but a flaw of
the daemon's port allocator, fixed in Engine 28.3.0: the mutex added as its
workaround could not help, and the comment said the opposite. Say what the
mutex does and does not provide, and that the sequential start of a
service's replicas (#8530) is the plan's replica chain, not this mutex.
Also fix the comment of the test of a stale non-relay container of a
provider-backed service, which still described a special case the start
path no longer has: planServiceStart routes every provider to
planProviderStart, which decides bare-vs-enriched per container.
Comments only: the token stream of every Go file, comments excluded, is
unchanged.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Review follow-ups of the removal of waitDependencies' timeout parameter:
the unsupported-condition test lost the 2s bound the parameter gave it, so
a regression that waits would hang instead of failing; and the test
comparing a timeout that expires during the inspection with one that
expires between polls now also compares whether they are a deadline.
Also reword the comment of execWaitCondition, which still said that
waitDependencies threads a timeout through its waits.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Now that waitDependencies has no timeout parameter, the plan's wait nodes
are the only ones to wrap the context of a wait: tell so in the comment of
waitDependency, which compares the wrapped context with the original.
Comments only.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Its four callers (up --wait, start --wait, run, restart) all pass 0, so the
wrapper it built around the context was dead: a wait only gets a timeout of
its own through the plan's OpWaitCondition nodes, which bound it with
exec.waitTimeout in execBoundedWaitCondition and never go through
waitDependencies. A deadline on the caller's context, which is how up and
start --wait bound their wait, is untouched.
The tests that passed a timeout now exercise the same semantics through
the live paths: a deadline on the caller's context for waitDependencies,
and execBoundedWaitCondition with exec.waitTimeout for the wait's own
timeout (tolerated on an optional dependency, an error on a required one).
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
startMx works around duplicate published host ports on Engine 28.0 to
28.2 (docker/compose#12846), a daemon port-allocator flaw fixed in Engine
28.3.0 (moby/moby#50054). Say so in its comment, state which start paths
take it, and add a TODO to remove it once Engine 28.x is out of support.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Follow-up of the two previous commits, from their review: rewrap the
comments they left with an orphan line, give two of them back a subject,
point one at the plan's wait nodes, and drop the "imperative engine"
wording of the restart tests, restart being the one command that still
walks the dependency order itself.
Comments only: the token stream of every Go file, comments excluded, is
unchanged.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Same cleanup as the previous commit, for the comments of the unit tests:
they cite startService, isNotRunning and the "old imperative engine" as
the source of a behavior the plan now owns.
Comments only: the token stream of every Go file, comments excluded, is
unchanged.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
startService and startServiceContainer were removed by the migration of
compose start to the plan engine, yet many comments still cite them, or
isNotRunning, or describe the start as walked by InDependencyOrder.
Rewrite them to state the behavior itself, and fix the ones that say
which commands still walk the dependency order.
Comments only: the token stream of every Go file, comments excluded, is
unchanged.
Part of #14081
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
These four operations act from the outside on the container of a service.
They only need the Docker client, the concurrency bound, the event bus, the
dry-run switch, the Docker CLI (which owns the terminal streams) and the two
container lookups, so they move out of the flat pkg/compose package into
internal/containerops behind a Backend interface declaring exactly that.
composeService implements the interface in a new file and keeps Exec, Copy,
Export and Commit as one-line forwarders (the progress wrapper of the last
three stays there), so no caller changes. The now unused checkRelayTarget
forwarder is dropped. Behavior is unchanged: the cp test still runs through
the forwarders.
Part of #14396
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Epic #14081 moved the start phase into the plan engine. Since #14302 and
#14305, a caller that streams pre_start/post_start hook logs into an attached
session hands its listener to planRun.runStart when it enters the Start phase,
because its printer only exists once the Create phase is done. As a result
every production caller builds the executor with a nil listener, and only a
few tests still passed one at construction; runStart kept a "nil keeps the
construction-time listener" branch just for them.
Drop the listener parameter of newPlanExecutor and make runStart the only way
to give the executor a listener (nil means hook logs are not streamed). The
doc comments that described a construction-time listener now describe that
single path. Production behavior is unchanged.
In tests, the one that fed a listener at construction sets it on the executor
it exercises, the two that only pinned "construction-time listener kept or
replaced" are removed, and a test pins that a nil listener leaves hooks
running without one.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- a project that already went through use_api_socket, as the one `run` hands
to the dependencies it starts, is injected once by start(): rewriting it
again would duplicate the config entry, and the file would be copied twice.
The first test also checks that the option was applied;
- the comments say which maps the rewriting may find nil: the service
Environment in the project `compose start` loads, the Configs of a
programmatic one, and that a project rebuilt from the labels knows nothing of
the option.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The project `compose start` loads has no environment nor configs map, which
the rewriting used to write into; a project built with empty maps is the
other shape. Both are started with the config file in place, and neither
caller's project is written to.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A container created beforehand and only started by `compose start`
(`create` then `start`, `up --no-start` then `start`, or a dependency named
to `start`) came up with DOCKER_CONFIG pointing at /run/secrets/docker and no
file there: use_api_socket is applied by rewriting the project, and the
config file it adds is injected when a container starts, from the project the
start works on. Create and run applied the rewriting, start did not.
start() now applies it too, on the copy it already makes of the project, so
the caller's is left alone, and a project already rewritten (a dependency
started by run) is not rewritten twice. The rewriting tolerates a project
without environment or configs maps, as the one `compose start` loads.
up -d, a foreground up, scale and run already provided the file.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- the plan's own project, not the caller's, is what the session, the Start
phase, the watcher and the teardown work on: a test checks that the Create
phase hands it over, with use_api_socket's config only the plan's project
has. It was found by mutation to survive the other tests;
- the interactive up's doc no longer over-claims: an interruption is only
honored while the session attaches or reads the daemon's time, and one
arriving later does not stop the Start phase;
- readDaemonTime's doc says the error is the client's, which carries the
context's;
- two test comments say what they rely on: that only the monitor uses the
events API, and that the fake engine reports an interrupted call.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Tests:
- the monitor goroutine runs before the Start phase: a fake container start
holds until the events subscription is opened, so a Start phase run before
it would show;
- an interruption met by the Create phase ends the up with it and sets up no
session: the phase runs on the caller's context.
Both were found by mutation to survive the existing tests.
Comments and scenarios:
- useAPISocket's guard says what a second pass really does: duplicate volume
and config entries that also change the service's config hash;
- the interactive up's doc says an early interruption starts nothing, the
readDaemonTime's that callers decide how to report an interruption, and a
test comment on the listener no longer announces what is already wired;
- the use_api_socket scenario links docker/compose#14386, as SCENARIO.md asks
of a regression test, and the hook-runner scenario drops an output assertion
its ExitCode already covers.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
UpOptions.Validate now refuses the combination, so the field's doc says that
instead of describing a foreground Up that silently skips the verification.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Production:
- detached and interactive up prepare their plan the same way: prepareUp
holds the preparation and the --wait-timeout rationale both share, so the
two cannot diverge;
- useAPISocket leaves alone a project it already transformed. The session,
the watcher and the teardown now work on the plan's own project, which a
watch rebuild hands to a later plan: transforming it again mounted the
socket and the config file twice;
- the docs say what the code does: an interruption met while reading the
daemon's time is returned wrapping the context's error (the CLI maps it to
its interruption exit code), Up ignores StartOptions.Project and does not
verify StartOptions.Wait in the foreground, and the Start phase's listener
goes through runStart.
Tests:
- the Start phase is shown to survive the caller's cancellation, and an
interruption before it to start nothing;
- the dry-run test no longer claims to exercise the dry-run client: it pins
the gate of Up;
- the phase order test also checks the "up" scope opens before the create;
- useAPISocket's idempotence is tested.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A pre_start runner carries the project and service labels but not the one-off
label the monitor's event filter requires, so its exit never reaches the
session. Nothing locked that: a runner exiting successfully would otherwise
win over the service's own exit code under --exit-code-from, or abort the
application under --abort-on-container-exit.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Interactive up used to create() the project, then start() it: two
daemon snapshots, the second built from the project as the caller gave
it rather than the canonical one the Create phase ran on.
It now prepares one Create+Start plan, like detached up. The Create
phase runs under the "up" progress scope, then the attach/printer/
monitor session is set up exactly as before, and the Start phase is
dispatched last, outside that scope, on a context that survives
Ctrl+C, with the printer as the hook-log listener. A failed Create
phase reaches neither the session nor the Start phase, and a dry run
only ever runs the Create phase.
The monitor's starting point is the daemon's time, which is read right
before the Start phase: a daemon that answers with a time that can't be
used fails the up with an explicit error, with no container started. One
that can't be asked at all (a read-only socket proxy without the info
endpoint) doesn't: the monitor then follows live events, with a warning.
The session, the watcher and the teardown now use the plan's own
project, which carries what preparation resolved. That makes
use_api_socket work in a foreground up: its config file lives only in
that project. A scenario locks it.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The CLI refuses combinations such as --wait with --attach, or --detach with
--abort-on-container-exit, --exit-code-from or --watch. The library did not:
an integrator setting Start.Wait together with Start.Attach got an
interactive session in which the wait was silently dropped, and the
session-only options were silently ignored when Attach is nil.
Add UpOptions.Validate, called first thing by Up, which returns an
*IncompatibleOptionsError matching api.ErrIncompatibleOptions. The CLI is
unchanged.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
run() walked both phases in one call, keeping the state they share local
and unreachable. Split it into begin, which prepares the run without
dispatching anything, and a planRun whose runCreate and runStart each
dispatch their own phase, so a caller can act between the two -- the
interactive up will set up its attach/printer/monitor session there.
run() is now begin followed by both phases, so every existing caller
behaves as before. The start phase takes its hook-log listener when it is
entered, since an attached caller only has one once its session exists,
and refuses to run before a successful create phase or twice, keeping the
create-phase barrier intact for callers that drive the phases themselves.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
envResolverWithCase unconditionally built a lower-cased copy of the whole
environment map before any lookup, even when every lookup hits a key
directly. Fall back to a case-insensitive scan (strings.EqualFold) of the
original map only when the exact lookup misses, so the common path
allocates nothing.
Folded-key conflicts (e.g. FOO and foo both present) were already
documented as indefinite and remain so.
Signed-off-by: Aucoto <tdnorthgarden@gmail.com>
monitor.Start lists the project's containers, then opens the event
stream. Both are round trips: a container that starts and terminates
in between has its die emitted to nobody, the monitor tracks it
forever, and --abort-on-container-exit / --exit-code-from never fire.
The window is narrow, but a plan running Create and Start back to back
makes the first start follow the monitor's attachment closely.
Ask the engine for events since a point taken before the listing, and
take that point from the daemon (its system time, one extra round trip),
not from the client: events are stamped by the daemon, so a client clock
that differs from it, as against a remote daemon, would either miss part
of the window or replay events from before the monitor was started.
There is no falling back on the client's clock. A daemon that answers
with a time that is empty or can't be parsed makes Start fail with an
explicit error, and so the command. A daemon that can't be asked at all
(a read-only socket proxy that doesn't expose the info endpoint) doesn't:
`logs --follow` never needed that endpoint, and failing it would be a
regression. The monitor then warns that the window can't be closed and
subscribes to live events, as it did before the replay. A caller that
wants to know before it starts any container can read the time ahead of
Start (readDaemonTime), which then doesn't ask, or warn, again.
Replayed events already reflected in the listing are harmless: the
listing notifies nothing, so each is reported once, and the tracked set
only gets an ID added or removed again. Unit tests drive the monitor
against a fake event log that, like the engine, only delivers
unsubscribed events to a subscription asking for them with since, and
stamps events with a clock of its own. They pin that replay for a die, a
start, a destroy and a restart policy, the daemon's time being used
whatever its offset from the client's, the failure on an unusable time
(also through `logs --follow`), and the live subscription, with a single
warning, when the engine can't be asked.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A foreground up does stream a pre_start hook's output into the session since
the hook's log stream is opened once it is started. The hook scenario had
stopped asserting it on the belief it never did; its fixture only wrote to a
file, so nothing could reach stdout. The pre_start now also prints a marker
the service never logs, asserted on stdout, and the scenario states a single
intent covering both hooks.
Also tighten the scenarios after review:
- assert a container's printed output on stdout rather than on the whole
output, and add the state or negative checks that were missing
(ContainerEnv after a recreation, web exited once its dependency was
healthy, no output of the replaced container);
- the dependency timeout scenario carries the "deadline exceeded" guard of
its detached counterpart, with a fixture aligned on it;
- drop the abort message and the useless wait from the immediate-exit
scenario: up only returns once the application is stopped;
- rename the dry-run scenario after the others, minimize its fixture and stop
coupling it to the dry-run end-of-output wording;
- explain the sleeps of the fixtures, and give the hook one a larger margin;
- build tag e2e only: nothing here is Unix-specific.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Wait for the long-running service's final state after an abort instead of
reading it the instant the command returns, document why the dependent is
expected in the created state on a dependency timeout, and narrow the
dry-run "no start" check to the service's own lines.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The hook scenario matched "pre-start-hook-ran" in the output, but the
service's own line "app-read:pre-start-hook-ran" contains it, so the
assertion passed without any pre_start output reaching the session. A
foreground up streams post_start's output but not pre_start's today
(checked on main too). Lock what actually holds: pre_start's effect, read
by the service through a value distinct from anything it logs.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Most e2e coverage of the start phase runs detached; the foreground
`up`, whose create and start phases are driven by separate code paths,
had none for hooks, health-gated dependencies, a container that exits as
soon as it starts, restarts of exited containers, recreation, dependency
timeouts or dry-run. Add scenarios that fix these observables before the
machinery behind it changes.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
checkDependencyHealthy stayed silent whenever the phase context was
canceled, so a real failure (unhealthy, exited) found just as another node
cancelled the phase left the dependency's rows on Waiting. Test the error
itself instead of the context: only a failure caused by the cancellation is
silent.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A start node whose group another node gave up while it was getting a slot
no longer goes on: it would emit a Working event after the group's final one,
and leave the row on it. The reason a node reads when it is given up now
follows its prerequisites through the chain (wait, start, post_start), so
that it does not depend on which node of the chain reports first, and a
post_start is only reported as started when its start succeeded.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Follow-up to the report of the services a failed start interrupts:
- When it is the wait itself that fails (the dependency is unhealthy, or
exited with an error), the held service reads which condition was not met
instead of an interruption by the internal name of the wait node.
- Only the start phase reports what it gives up: the create phase stays
silent as before.
- An inspection cut short by the cancellation ends the wait as interrupted,
and no longer marks the healthy dependency as failed. A caller canceled
(Ctrl-C) at that point ends the wait as it always did.
- Two nodes failing together each end their own row with their own error,
whichever one cancels the phase, instead of leaving one on "Starting" or
reading "interrupted" for a failure that was its own.
- A replica whose start is behind it reads that only its post_start hooks
were interrupted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
When a node of a plan phase fails, the executor cancels the whole phase.
The nodes it interrupts, or had not dispatched yet, got no final event:
a service recreated by the command and held by a depends_on wait stayed on
"Recreated", as if it were up, and the row of the dependency it was
waiting for stayed on "Waiting". Nothing said the service was left created
and never started.
Such a node now ends on an Error event, "not started" when its group never
reported Working, which names the depends_on condition it was held by and
the resource whose failure interrupted the phase. A wait interrupted that
way is a failure of its own (the condition was not reached) and closes the
"Waiting" rows of its dependency with a Skipped event. A phase canceled by
its caller (Ctrl-C) is reported as before: silently.
docker/compose#14372
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Pass `umask` to the engine as HostConfig.Umask when creating a
container, so that the entrypoint, exec and healthcheck processes run
with the requested file mode creation mask. The attribute is part of the
container specification, so pre_start init containers, created the same
way, get the umask of their service unless they declare their own.
Engines older than API 1.56 (Docker Engine 29.8) silently ignore the
field, which would run the container with the runtime's default umask
without any notice: creating a container with a umask against such an
engine now fails with an explicit error, like interface_name or image
volumes do.
The umask is part of the service configuration hash, so changing it
recreates the container; the hash of every other configuration is
untouched.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Switch from the v2.16.2 pre-release pseudo-version to the tagged release.
It brings the umask service attribute, which must be pinned at the end of
serviceHashKeyOrder, as the hash coverage test requires.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A pre_start hook's output never reached anyone. The follow log stream was
opened on the runner while it was still in "created" state, before
ContainerStart; the engine ends such a stream at once, empty. So a foreground
up never showed the hook's output, and the error of a failing hook carried no
tail, though both are built from that stream.
The stream is now opened once the hook is started. The runner is retained
after it exits (AutoRemove is off), so a fast-exiting hook cannot make the
stream hit a 404, which is what opening it first was meant to prevent. Once the
hook has exited, the stream gets a short, bounded delay to deliver what the
daemon buffered before it is cut, instead of being cut right away.
The unit tests that pinned the old call order now pin the new one, and a
scenario locks that a failing hook reports what it printed.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Cover what the removed imperative tests used to: a no-op start is silent and
only lists containers, --wait-timeout bounds the dependency waits, secrets and
configs are injected before ContainerStart, a failing post_start fails the
start, pre_start is skipped on a stale relay, deploy.replicas: 0 starts
nothing, and the project-wide no-container check. On the provider path: the
restart is a chain end, a relay gets no post_start, pre_start is gated on
running replicas and runs once, before the chain, after the dependencies.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
planProviderRelayRestart duplicated the replica chain of planServiceStart
(sort, pre_start gate, bare vs enriched start, post_start), so a fix to one
did not reach the other. Extract it as chainStartReplicas and let the
provider path feed it the replicas startPhaseReplicas already collects
(under pure ScopeStart, the sorted not-running observed containers).
No behavior change: the plans are node-for-node identical.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
startService received the whole project's container list, so "no container
to start" held only for a project without any container. A service without
a container next to existing ones was a no-op and the others still started;
the plan failed the whole start instead.
Plan the error only when nothing is observed in the project, orphans
included.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
start() translated any DeadlineExceeded from the plan into a
dependency-readiness message when --wait-timeout was set, including one
from an engine call or a hook. Use the marker a wait node's own timeout
now carries, as upDetached does.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The final readiness check opened its --wait-timeout window before
re-listing the project's containers, so a slow daemon listing could
exhaust the budget and report "application not healthy" before
waitDependencies had polled once. List first on the caller's context,
then derive the timed one, as the imperative start path did.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>