The previous commit gated the deadline on options.Start.Wait &&
WaitTimeout > 0, describing it as matching start()'s own gating -- it
doesn't. start()'s InDependencyOrder passes options.WaitTimeout into
every startService call unconditionally, not only when options.Wait is
set, and the CLI never requires --wait alongside --wait-timeout
(`up -d --wait-timeout 30` with no --wait is accepted as-is). Under the
previous gating, that exact combination would still hang on a stuck
OpWaitCondition node -- the same bug the deadline was just added to
fix, reintroduced for one flag combination.
Drop the options.Start.Wait condition: the deadline now applies
whenever WaitTimeout > 0, matching start()'s actual unconditional
behavior.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review (High): the plan's own OpWaitCondition nodes
(service_healthy / service_completed_successfully dependency waits,
executed as part of executePlan) have no timeout of their own -- they
block on whatever ctx they're given, unlike the imperative engine's
waitDependencies, which already threaded WaitTimeout through every
per-dependency wait, not just the final --wait check. upDetached only
wrapped ctx with WaitTimeout around that final check, established
after executePlan already returned -- so a condition that's never
satisfied hung `up -d --wait --wait-timeout N` forever instead of
failing after N seconds.
Move the deadline to wrap ctx before executePlan runs (gated on
Wait && WaitTimeout > 0, matching start()'s own gating), and reuse the
same ctx for the final verification instead of opening a second,
independent WaitTimeout window -- one overall budget for "becoming
healthy" covering both the plan's in-flight waits and the final check.
preparePlan is deliberately left outside the deadline: image pulls and
the rest of create()'s own work were never bounded by --wait-timeout
in the old create()+start() sequence either.
Regression-tested at two levels:
- TestExecutePlanWaitConditionRespectsContextDeadline drives a real
OpWaitCondition node through executePlan/run() under a short
deadline and asserts it returns promptly with ctx.Err(), proving the
deadline propagates into the plan's blocking nodes.
- The existing e2e TestUpWaitTimeout (up --wait --wait-timeout on a
service that never turns healthy) is what actually exercises
upDetached's own wiring: verified by ablation that reverting this
fix makes it hang (test timeout) instead of failing after 3s.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Detached `up` used to run create() then start() back to back: two
separate daemon snapshots, the second blind to what the first just
created or recreated. preparePlan already supports planning both
phases in one pass (ScopeCreateStart, landed across lot 1) -- this
wires it in for the one caller that can use it outright: `up -d`.
preparePlan gains an explicit scope parameter (ScopeCreate for
create(), unchanged); Up()'s detached branch now calls a new
upDetached, which runs preparePlan(..., ScopeCreateStart) +
executePlan once instead of create() + start(). --wait stays a
post-plan verification: it re-lists containers after the plan runs
(the plan may have created or recreated some, so the pre-execution
snapshot can't be reused for this) and runs the same waitDependencies
check start() already does.
Interactive up is untouched: it still runs create() then start() with
its printer/monitor listener wired in between, converging onto the
plan engine separately (a riskier, later item of the same lot).
ScopeCreateStart landed in lot 1 as inert code with no real caller;
this is the first one, and it surfaced two latent bugs in the
container-recreate path that nothing had exercised end to end before:
- planRecreateContainer plans a container's recreation as Create
(under a temporary "<shortID>_<name>" to avoid colliding with the
old container still holding the final name) -> Stop -> Remove ->
Rename(final name). The start phase's event naming and container-ID
resolution (groupEventName, resolveContainerID) read the create
node's published result by design (plannedReplica keeps every
start-phase reference pointed at CreateNodeID, not the rename node),
but execRenameContainer never updated that result after renaming --
so a recreated container's Starting/Started events kept reporting
the temporary name forever.
- execCreateContainer also publishes the new container into
containersByService (the live view OpWaitCondition reads for
depends_on health waits) under that same temporary name, with
nothing refreshing it after rename either -- a dependent waiting on
the recreated container's health reported Waiting/Healthy under the
temporary name too.
Both fixed in execRenameContainer: it now updates the create node's
pctx result in place after a successful rename, and (via a new
Service field on the rename Operation) patches the matching
containersByService entry's name too. Each pinned by a regression
test, verified by reverting the fix locally and confirming the test
catches it.
Verified with the full e2e suite (`go test -tags e2e ./pkg/e2e/...`)
against a locally built binary -- including TestRestartWithDependencies,
which is what caught both bugs in the first place -- plus the existing
unit suite and lint.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Running a provider plugin (finding its binary, executing it, serving the
JSON messages it sends, applying the variables and endpoints it publishes,
streaming it an image) needs from the engine the Docker API, the event bus,
the Docker CLI that locates the plugin, the preparation of the environment
of a shelled-out command, and the lock that guards the project model every
concurrent provider run mutates.
It now lives in internal/provider behind a Backend interface that
composeService implements in pkg/compose/plugins.go, next to runPlugin, which
stays as a one-line forwarder so none of its callers (up, down, stop, pull)
changes. The relay it deploys goes through internal/relay, whose Backend the
new one extends. The public types and constants of the protocol (JsonMessage,
ProviderMetadata, the *Type message constants...) stay exported from
pkg/compose as aliases of the new package's.
The tests of the plugin protocol move with the code, assertions unchanged;
those of the provider image phase and of the relay link network removal
stay in pkg/compose. The forwarders to the relay that only the plugin code
used are dropped.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The relay container and the relay link networks that stand in for a
provider service need three things from the engine: the Docker API, the
event bus and whether Docker Desktop is the engine.
They now live in internal/relay behind a Backend interface that
composeService implements in pkg/compose/relay.go, which keeps the names
the lifecycle code calls (ensureServiceRelay, isRelayContainer,
checkRelayTarget...) as one-line forwarders, so none of its callers change.
The one down-specific piece, turning the relay link networks into down
operations gated on the shared limiter, stays in pkg/compose together with
its tests; the other relay tests move with the code, assertions unchanged.
The progress events and the container name the package needs come from two
small shared packages, internal/progress and internal/naming, which the
following moves reuse. pkg/compose keeps its own copies until the shared
kernel is extracted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The dry-run client fakes ImageInspect for some callers, picked by the
name of the calling function, and fakes the push of an image. Nothing
tied --ignore-missing to the real state of the local image store in
dry-run.
Check that a missing image is skipped from what the engine really
holds, that the other images get the faked push, and that nothing is
pushed nor looked up in a registry for the skipped ones.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
`push --ignore-push-failures` ignores every push error, including
authentication and registry failures, which can hide real problems when
the only expectation is that some services have not been built.
Add `--ignore-missing`, which skips an image that is not in the local
image store and still fails on any other error. It is independent of
`--ignore-push-failures`, whose behavior is unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The sibling's start now only returns once the wait has polled its
dependency, so without the fix the plan deadlocks whichever node takes the
single slot first, instead of depending on the scheduler. It also uses the
platform-neutral newTestService helper, so the file builds on Windows.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Every plan node acquired a concurrency slot before running, including
OpWaitCondition, which then kept it for the whole health polling. A
dependency awaited as healthy can itself need a sibling container the plan
has yet to start, so with a bounded --parallel the wait could take the slot
that sibling's start needed and only end at its timeout.
Wait nodes now skip the limiter: they only poll the daemon on a ticker, as
the imperative waitDependencies does outside the cap.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The dependencies a one-off starts go through the regular create path
and its config-hash divergence check, with no way to opt out: any
resolution difference with the command that created them (environment,
model edit) recreates the dependency — destroying state a previous run
built up (issue #13769's database losing its users). up already offers
--no-recreate for exactly this; run now aligns: the flag maps to the
"never" recreate policy, propagated through the RunOptions.CreateOptions
fields that startDependencies previously dropped. Default behavior is
unchanged (empty policy = "diverged").
Closes#13769
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
doBuildBake waited for the buildx process in a goroutine while reading its
rawjson stderr stream. os/exec closes the pipe as soon as Wait sees the
process exit, dropping whatever was not read yet: when the consumer of the
solve statuses (the progress display) lags behind the process, the end of the
stream is lost. That is where bake tells why it failed (the Dockerfile
excerpt, then "failed to solve: ..."), so the build failed with exit code 1
but without its reason, or with it truncated.
Read the stream to EOF first, as the os/exec documentation requires, then
Wait. The os.ErrClosed case of the reader, which only existed to absorb the
pipe being closed under it, goes away with the race.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The docker model plugin is an external process the dry-run API client
cannot intercept, so up, create and run --dry-run really pulled the
models and changed the runner configuration.
Skip pull and configure when dry-run is set while still emitting the
progress events. The read-only ls and status commands still run, so
pulling is only reported for a model that is actually missing and the
model variables are still computed.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Dry-run only wraps the Docker API client, so provider services, which
are external processes, were still executed: up and create ran the
provider's up command, down and stop ran theirs.
Skip the process in executePlugin when dry-run is set while still
emitting the progress events, and return early from runPlugin so no
relay is deployed or removed for a provider that never ran. The
metadata command is still invoked to keep validating the binary and
its required parameters.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
LoadProject and Generate (the api.ProjectLoader operations) need two things
from the engine: the Docker API, to read the containers a project is
generated from, and the Docker CLI, which resolves remote Compose files.
They now live in internal/projectloader, along with the unsupported
attributes detection only the loader uses, behind a Backend interface that
composeService implements in a new file of pkg/compose, like internal/inspect.
The tests move with the code; their assertions are unchanged, only the way
the service is built differs.
The canonical container name, which both this package and internal/inspect
need, moves to a tiny internal/naming package instead of being copied a
third time. pkg/compose keeps its own copy until the shared kernel is
extracted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ps, ls, top, port, volumes, events and viz only read the state of a
project: they need the Docker API, the list of the project's containers
and the concurrency bound, nothing else of composeService.
They now live in internal/inspect, which declares that need as a Backend
interface. composeService implements it in a new file of pkg/compose and
forwards the seven api.Inspector methods, so no other file of pkg/compose
changes, and neither does the behavior: the service-level tests stay in
pkg/compose and run unchanged through the forwarders.
The few tiny helpers the operations share with the rest of pkg/compose
(canonical container name, concurrency-limited errgroup, project filter)
are copied for now: the shared kernel they belong to is a later step.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
compose run only wrapped the start of its dependencies in the "run"
progress operation. The pull of the target service's image, and the
creation of the one-off container, happened once that operation was
closed, so the TTY display fell back to one plain line per event,
including every layer, instead of the compact in-place block.
Wrap the whole preparation in the operation.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Reproduces docker/compose#14287: a hook inheriting the service's
env_file (or declaring its own) runs without those variables.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Pure extraction, no behavior change: create() resolved the project
model, ensured images/models and external networks/volumes, observed
the daemon, and reconciled a Plan, all inline before calling
executePlan. That decision-making half is now its own function,
preparePlan, returning the canonical project, the daemon snapshot, and
the Plan — nothing is created, started, or otherwise touched by it.
This is Lot 1's last item (epic #14081): interactive up's eventual
create/start phase boundary needs to prepare the plan once, run the
Create phase, set up attach/printer/monitor, then run the Start phase
— which needs the plan and snapshot available between those steps,
not buried inside create()'s single call chain. create() itself is
unchanged in behavior, just now a two-line call into preparePlan
followed by the existing emitRunningEvents + executePlan.
Verified with the existing unit suite plus the create/up/down e2e
scenarios (TestLocalComposeUp, TestCreateIdempotentDefaultPlatform,
TestCreateIdempotentSharedImageMixedPlatforms, TestCreateRefusesJob,
TestCreateLocalMultiPlatformImage, TestUpWithAllResources) against a
locally built binary.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A sync into a path that is also bind-mounted writes back to the host,
which the watcher sees as a new change and syncs again, looping
forever. isSync, which gates the bind mount check, missed sync+exec,
so only sync and sync+restart triggers were skipped with a warning.
Include sync+exec in isSync. This also makes initial_sync apply to
sync+exec triggers, as it does for the other sync+x actions.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Grant access to SSH keys and sockets from build configuration and CLI
options, and authorize default agent forwarding, including implicit SSH
Git contexts. Serialize pathless SSH entries without an empty source so
Bake applies the agent entitlement check.
Grant read access to local cache imports and OCI layout contexts, and
write access to local cache exports. Match Bake's path and cache syntax
while excluding service, target and image references from filesystem grants.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Buildx 0.37.2 enforces entitlements with rawjson progress, exposing missing
permissions in Compose's Bake invocation. Grant read access to Dockerfiles
and file secrets, including local Dockerfiles used with remote contexts,
and forward requested entitlements and host network access.
Deduplicate grants and preserve environment secrets and privileged builds.
Add regression coverage for the generated Bake configuration and arguments.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Compose is now composed by embedding Lifecycle, Inspector, ImageManager,
Runner, Watcher and ProjectLoader. The method set of Compose is unchanged.
ProjectOptions.ToProject and configOptions.ToProject only need to load a
project, so they now depend on api.ProjectLoader.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
TestBuildTLS slept a fixed second after the dind daemon logged that it
listens, hoping the client certificates were written; poll for them
instead. TestForwardKeepsStreamingAfterHalfClose had 150ms of margin
between each chunk and the 300ms idle grace, which a loaded CI runner can
eat; use 100ms chunks against a 1s grace.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
pkg/compose still imported the keyboard library: it grabbed the terminal,
forwarded raw key events to the menu and released the terminal on exit.
The NavigationMenu now takes care of all of it through Open, Run and Close,
so the library no longer knows about keys or the terminal, and the menu
interface no longer exposes a third-party event type.
As a side effect, key handling no longer shares a goroutine with the
signal handling loop of `up`.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
pkg/compose imported cmd/formatter for the interactive keyboard menu of a
foreground up, so the library depended on the CLI. Define a NavigationMenu
interface (and a factory) in pkg/compose, satisfied by formatter.LogKeyboard,
and have the CLI inject it with the new WithNavigationMenu option. The terminal
code stays in cmd/formatter. Without the option no menu is displayed.
formatter.Feature moves to api.Feature so that both sides share the type, and
a depguard rule now forbids pkg/ and internal/ from importing cmd/.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The daemon's swarm mode cache, the provider variable-injection lock and
the log-stream drain bound now live on composeService, so several
services in one process (SDK users, tests) no longer share them.
startMx stays global on purpose: it protects port allocation in the
daemon, which instances sharing a daemon must still serialize.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Start and Stop guard a Watcher's own stopFn/errCh, yet the lock was a
package-level mutex shared by every Watcher in the process.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The log journal of a container is shared by all its runs, and a followed
log stream has no upper bound: it ends with the run in progress when it
opened. After a restart, compose opens a stream starting at the end of
the previous run (d834777e1), but a stream opened late, once the
container has restarted again (the restart delay is only 100ms, and the
event monitor or the daemon can be slower), also returns the next run's
lines, which that run's own stream then returns a second time.
The streams of one container now run one at a time, in start-event
order, and each starts right after the last line the previous ones
relayed. Line timestamps are the daemon's own, so no client clock is
involved. A stream opened in time still starts at the previous run's
end, as before. TTY containers, whose stream is not multiplexed, are
unchanged.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
When the inspection of an optional (required: false) dependency fails, the
dependency is reported as skipped and the wait succeeds. That also happened
when the failure came from the wait's context being done: a cancellation
was reported as a skipped dependency, and an expired deadline (the wait
timeout, or one inherited from the caller) was turned into a successful
wait, unlike the same deadline expiring between two polls.
A check failing while its context is done now reports "not done yet" for an
optional dependency, so that waitDependency decides what the end of the wait
means. Genuine inspection errors still skip the dependency, and a required
dependency is handled as before.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Shorten the doc comments added for the Docker Desktop credentials provider to what the code does not already say, and drop them from the tests.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Docker Hub credentials are now read first from the session of the account signed in to Docker Desktop, through the Docker Desktop secrets engine (github.com/docker/secrets-engine/client), and from the Docker CLI config file and its credential helpers otherwise. Other registries keep using the config file only.
The OAuth access token is passed as the password of the signed-in account, never as an IdentityToken, which containerd sends as a refresh token. It flows unchanged to the engine in X-Registry-Auth and through the OCI resolver as an OAuth password grant.
Compose falls back to the config file when Desktop is not running, nobody is signed in, the session is expired or about to expire, or the lookup fails (the last logged as a warning). The session is shared process wide, cached until shortly before it expires, and an unusable session is not asked for again. Lookups are bounded by a 5s timeout, as the SDK applies none by default.
Used by pull, push, publish, config --resolve-image-digests, oci:// remote resources and dry-run. The classic builder and use_api_socket keep config file credentials, as a short-lived token copied into a container would go stale.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Two independent reviews (Copilot CLI + a fresh-context agent) of the
previous commit converged on the same finding: execStartContainer's,
execRunPreStart's, and execRunPostStart's ctx.Err() guards still
claimed the TOCTOU window "cannot be closed" and deferred the real fix
to #14081 -- stale now that run()'s failedDependency check closes it
for a direct dependency before executeNode is ever reached. Reworded
to say what's actually still open: defense in depth for ctx canceled
by something other than a direct dependency (an unrelated branch of
the same plan, or an external cancellation).
Also: recordFailure stored the already-wrapped "dependency failed: %w"
error, compounding the prefix at every hop of a multi-node chain.
Store the original error instead, wrapping only the value returned
from this node's own goroutine.
And a stale reference to a test name that was never the one actually
added (TestExecutePlanFailedPreStartGatesStart, not ...GroupDoesNotRestart).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Two review findings on the same root cause:
- docker-agent: a failing OpRunPreStart emitted a bare Error with no
preceding Working/Starting event, since triggersWorking only fires
on OpStartContainer. onNodeError now emits the group's Working event
first when it hasn't fired yet, keeping the progression well-formed.
- glours (following up on the docker-agent thread): worse than a
missing event -- OpStartContainer's own goroutine was unblocked by
<-done[dep.ID] and ran anyway, since close(done[node.ID]) races
ahead of errgroup's cancel() (same TOCTOU class as the ctx.Err()
guards already tracked under #14081, but deterministic here with the
default unset --parallel limiter: no semaphore wait gives cancel()
time to land first). It fired a second, spurious Starting right
after the sibling's Error, with no further event ever closing it.
Reproduced 143/143 iterations against this PR's own regression test.
run() now records each node's terminal error in a shared, mutex-guarded
map as it closes done[node.ID], and a dependent checks that map right
after its dependency wait, treating a genuine dependency failure
exactly like a context cancellation: no slot acquired, no event
emitted, its own failure recorded and propagated to its own
dependents the same way. This closes the event-ordering race outright
for this specific chain, rather than narrowing it like the existing
ctx.Err() guards -- those stay in place and still guard actual engine
calls, unrelated to event emission.
TestExecutePlanFailedPreStartGatesStart now asserts the full event
sequence (Creating, Created, Starting, Error — nothing after),
verified by reverting the executor.go fix locally and confirming the
test catches the reintroduced spurious Starting.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Same interim narrowing as execStartContainer and execRunPostStart: the
node hangs off the replica's create node, so when that create fails the
pre_start hooks could still fire in the window before errgroup cancels
the context. The structural fix stays tracked in #14081.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
docker-agent's review of the previous commit's fix correctly pointed
out that the ctx.Err() guard added to execStartContainer/execRunPostStart
narrows the race it targets but does not eliminate it: close(done[...])
still runs inside the failing node's own goroutine, strictly before
errgroup's cancel() on that goroutine's returned error. A dependent
unblocked from <-done[dep.ID] can observe ctx.Err() == nil for a brief
window even though its dependency just failed for a genuine reason.
This is exactly the structural gap the epic (#14081) already flagged
in its "failed-dependency semantics" comment before this PR existed:
the real fix carries each node's success/failure through what
dependents wait on (e.g. a chan error) instead of inferring it from
ctx.Err(), uniformly across every operation type - including the
already-merged execCreateHookContainer, which has the same limit.
That is a dedicated brick of the executor lot, not squeezed into this
PR's diff.
No behavior change: this documents the residual window explicitly at
both guard sites and in the regression test, so the next reader (and
the next PR) doesn't mistake "narrows" for "closes".
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A node's done-channel closes strictly before errgroup cancels its
derived context on that same error - the close happens synchronously
inside the failing node's goroutine, ctx cancellation only after it
returns. A dependent blocked in run()'s
select { case <-done[dep.ID]: ...; case <-ctx.Done(): ... } therefore
always sees its dependency's done-channel ready first, regardless of
whether that dependency failed for a genuine reason or was cancelled.
execCreateHookContainer already guarded against exactly this. The two
other new dispatch functions from the previous commit did not:
execStartContainer's enriched branch could call ContainerStart right
after its OpRunPreStart failed for real (e.g. "no hook runner
container found"), contradicting pre_start.go's documented guarantee
that a non-zero hook exit gates service start - and symmetrically for
execRunPostStart after a failed OpStartContainer.
Both now bail out on ctx.Err() first, mirroring the existing guard.
Pinned by a regression test reproducing the race deterministically: no
ContainerStart expectation is registered, so the unguarded code calls
it from inside the errgroup worker goroutine, where gomock's missing-
expectation failure hangs the test binary instead of failing it
cleanly - a stronger tell than a plain assertion failure would have
been.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Wire the plan executor to the start-phase vocabulary the reconciler
already produces (#14200): OpWaitCondition, OpRunPreStart, the enriched
OpStartContainer, and OpRunPostStart all had no case in executeNode
and hit "unknown operation type".
Every operation reuses the exact imperative primitive it mirrors —
waitDependency, runPreStart, runHook, injectSecrets/injectConfigs — so
behavior and events cannot drift between the two engines: execWaitCondition
resolves the dependency's live containers and hands off to the same
polling/condition-check functions waitDependencies uses; the enriched
OpStartContainer resolves its target either from the observed container
or, for a replica the plan itself creates, from the CreateContainer
node's result (the mechanism OpRenameContainer already relies on), then
injects secrets/configs before ContainerStart; OpRunPreStart re-checks
the daemon first so a replica that started in the observe-to-execute
window skips a redundant hook run.
A replica's pre_start/start/post_start chain shares one event group,
reporting a single Starting→Started progression exactly like
startServiceContainer does today — Starting fires specifically at
StartContainer, not at the (silent) pre_start hooks preceding it.
pkg/api/event.go documents this per-resource progression contract.
Still inert in production: create() (executePlan's only caller never
builds a plan with start-phase scope, so none of this runs outside
tests until a later brick in epic #14081 wires a real caller.
EOF
)
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
down() validates options.Images upfront, but ImagePruner is exported
and can be driven directly with a hand-built ImagePruneOptions that
bypasses that check. Add a direct test for the Valid() guard added in
the previous commit, so the two checks can't silently drift without a
test catching it.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ImagesToPrune's own check duplicated the same set of legal
ImagePruneMode values down() already validates upfront, risking silent
drift if a mode is ever added. Reuse Valid() instead.
Suggested-by: glours https://github.com/docker/compose/pull/14149#discussion_r4069721340
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The legal values of DownOptions.Images lived in pkg/compose
(image_pruner.go) while the field itself was a bare string in pkg/api,
and the only validation fired inside ImagesToPrune — after down had
already removed the project's containers, leaving the teardown half
done on a typo. The CLI validated its --rmi flag separately, so only
SDK callers were exposed.
ImagePruneMode and its three values now live in pkg/api next to the
field they constrain (type-aliased in pkg/compose for existing
consumers), down() rejects any other value before touching a single
resource, and the CLI check reuses the same definition. A unit test
runs Down with expectation-free mocks: one daemon call would fail it.
Epic #14074, B.6.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
docker-agent found that streamImageTo returned a plain error (channel
intact) when the announce write itself failed -- for the success
announce, and for the error announce on a platform-parse/ImageSave
failure. In both cases the provider never received anything to read
and was left blocked on ReadString, while the caller, seeing a plain
error, never closed stdin either -- a deadlock.
Both paths now return streamAbortedError when the announce write
fails, matching the existing mid-stream/close failure handling so the
caller's errors.As check closes stdin and unblocks the provider.
Pinned by two new regression tests using a writer that always errors,
verified by reverting the fix locally and confirming both fail with a
plain error instead of streamAbortedError.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Three fixes from glours' review:
- ensureProviderImages now skips a provider service declaring neither
image: nor build: (legal for a pure resource provider, e.g. one
backing a remote Amazon RDS instance). Without the guard, such a
service reached pull with a fabricated "<project>-<service>"
reference (GetImageNameOrDefault) under source=registry
(providerImageSource treats Build == nil as registry-authoritative)
-- a name that was never real. Already reachable through existing
e2e fixtures (TestDependsOnMultipleProviders' provider1/provider2),
only a no-op today because the demo pull() short-circuits on an
unset env var, not because of any guard here.
- localImageFacts now resolves --digest through the manifest-aware
localContentDigest instead of the raw ImageInspect ID. Under
BuildKit provenance attestations, the raw ID covers the attestation
manifest too and changes on every build even when the runnable image
content didn't (#13636), which would make the provider re-sync on
every up for no reason -- defeating docs/extension.md's own
"identity test" contract for --digest. Threads service.Platform
through from runProviderPull so the resolved manifest matches what
the provider will actually run.
- streamImageTo now distinguishes a failure before the success
announce (invalid platform, ImageSave error -- a clean, well-formed
error line, channel intact) from one after it (the chunked body
truncated, channel unresynchronizable) via a new streamAbortedError
type. Only the latter closes stdin; the provider protocol supports
several sequential get-image requests per pull invocation, and
closing stdin on a clean pre-stream error was killing any other
independent request in the same invocation for no reason.
Each fix is pinned by a regression test, verified by reverting the
fix locally and confirming the new test catches it.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A service declaring only build: (no image:) answered get-service-config
with an empty "image" field: service.Image is the YAML-declared value,
and compose-go never fills it in for this case (every other caller of
GetImageNameOrDefault in this package exists precisely because of that).
The provider only ever sees this response, never the model compose
builds internally, so an sbx-style provider building its own runtime
image rejected the service outright with "defines no image".
The response now carries the resolved name (api.GetImageNameOrDefault)
instead - the same one the image phase already built and tagged by the
time a provider asks. The build directive itself is cleared before
marshaling: a provider has no builder to run it against, only an image
identity to run.
Pinned by a regression test using the existing fake-provider harness,
verified by ablation to fail on either half of the fix reverted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
extension.md gains the Image distribution section: invocation contract
(--image/--digest/--created/--source/--policy), the get-image request
and its chunked image-stream answer (zero-chunk terminated, stdin
exclusivity and drain-before-next-answer contract), the two-branch
decision guidance (digest as identity, created as ordering fallback,
reproducible builds caveat), and the support-by-presence metadata
convention is now stated as the general rule. The example provider
implements pull end to end — request, stock chunked reader, hash
recorded at PROVIDER_PULL_MARKER — and the e2e scenario locks the
whole path: up builds the service image and streams it to the provider
under the local verdict, compose pull re-invokes it under the
freshness contract.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
startService errored with 'no container to start' whenever the
project-wide container listing came back empty, before the per-service
filtering that lets provider services pass through. A project whose
services are all provider-backed legitimately reaches the start phase
with zero containers. The empty-list error is now softened for
provider services only — a relay container deployed for published
endpoints still goes through the regular start path.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ensureImagesExists and docker compose pull now give provider-backed
services their turn: compose invokes the provider's pull command with
the image identity and the state of the local daemon cache
(--image/--digest/--created), plus two verdicts compose alone can
compute so providers never re-implement that arbitration:
- --source: the authority for this invocation — local (the daemon's
image is the desired state: build-only service, pull_policy build,
or an image the current run just built) or registry (resolve the
reference upstream — including the CI workflow where build is only
the recipe used to publish the image consumers pull);
- --policy: missing on the up path (a usable version suffices) or
always on compose pull (ensure freshness of the authority).
digest/created describe the cache, they are not instructions: the
provider persists them as the bookkeeping keys of what it ingested —
digest as identity test, created as the ordering fallback for backends
that cannot preserve digests. Providers without the metadata block are
skipped: behavior unchanged.
Providers are independent, so their pulls run without fail-fast
cancellation — the first failure no longer kills its siblings mid-run
and mangles their reports into context-canceled noise; every provider
runs to completion and every failure is reported, prefixed by service.
On the pull path the provider phase runs unconditionally — skipping it
because an unrelated service failed to pull would silently leave
provider runtimes stale — and its error is a per-service pull error,
following their exact regime: reported with them, suppressed by
IgnoreFailures. A fatal errgroup error is reported joined with the
accumulated per-service context instead of dropping it. The provider
runs under the root context: the errgroup context is canceled once
Wait returns and would kill the provider process on the spot.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Providers opt into image distribution by declaring a pull block in
their metadata — the presence of the command block is the declaration
of support, generalizing the stop precedent. During its pull command a
provider can request the service image from the local daemon with a
get-image message (optionally platform-narrowed); compose answers on
stdin with one image-stream JSON line then the tar as HTTP/1.1 chunked
data (RFC 9112): length-prefixed blocks need no in-band delimiter in
binary data, the zero-length chunk marks a COMPLETE transfer, and any
stock chunked reader consumes it. There is deliberately no trailer nor
final CRLF after the zero chunk — a stock reader stops there without
consuming further bytes, so the next stdin answer starts clean for a
provider requesting several images. An export failure is announced
in-band through the error field so the provider is never left waiting;
a failure after the announce closes the answer channel so the
truncation is observable as EOF mid-chunk instead of a stream nobody
will finish. The stream is exclusive on stdin for its whole duration —
the framing invariant behind the lock.
get-image is scoped to the pull command: emitted during up, down or
stop it is a protocol error rather than served, so an image export can
never stall another lifecycle command.
Message dispatch moves out of executePlugin into handlePluginMessage,
which also keeps the function under the complexity threshold.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Binding the announced gateway alone restricts by destination address, not
by the interface traffic arrives on, so another bridge can still reach it
by IP. A provider that pins the socket to the relay-link bridge's own
interface (SO_BINDTODEVICE on Linux) keeps the endpoint relay-only.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>