A job with a schedule is registered through the engine's jobs extension,
which fires it on its own clock; create and up register it, up stays
re-runnable since Create is idempotent on the spec hash, and a full down
removes the jobs. The extension ships disabled by default and cannot be
listed, so Compose probes it with a List scoped to the project; when the
engine cannot run jobs, up, create and start refuse the project before
creating anything, and down ignores the jobs. A job the file no longer
declares is reported by up and removed by --remove-orphans, unless its
run is in progress. The spec is stable: sorted entries, no volatile labels,
services referenced by container name.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Once every attached container is gone the monitor ends the session, and
with it the event loop that handles Ctrl+C: later interruptions are lost.
During --wait that left the stack running on exit 130: end the wait
without canceling, and stop the stack from there when the loop is gone.
During the start phase, blocked on dependencies of unattached services,
the signal was swallowed and the command hung: keep the session open.
Add tests for both and document that --wait leaves the services up.
Keep an api test valid now that Wait is accepted together with Attach.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Without --detach, `up --wait` now attaches to the containers and streams
their logs until the services are running|healthy, then ends the
session and leaves them up. Until now --wait silently forced --detach,
so a service failing to become healthy left no clue about why.
--detach --wait keeps the previous silent behavior, and --dry-run
stays on the detached plan. The library Validate accepts Wait with
Attach, and still rejects OnExit, ExitCodeFrom and Watch with it. The
navigation menu is disabled while waiting.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A pre_start hook container that fails to start (an executable missing from
the image of the hook, a bad mount, ...) was reported with the bare error of
the engine. It names neither the hook nor its image, and it surfaces on the row
of the service container, so it reads as the service failing to start.
Wrap it with the service, the index of the hook and its image, like the other
pre_start errors: service "app" pre_start[0] (image hello-world): failed to
start the hook container: <engine error>. The wrapped error is unchanged for
errors.Is and errors.As.
Refs #14513
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A stopped container still refers to the networks it was attached to. When
one of them was removed, up created the network again but started the old
container, and the engine refused: "network <id> not found".
The network check of mustRecreate only ran for running containers, so a
stopped one was never compared with the network of the project. It now runs
whatever the state of the container, like the check of the volumes.
A network that no longer exists is not in the observed state, so a container
attached to a network of that name refers to the removed one and is
recreated. A container that was created but never started has no network ID
yet: it is left alone, rather than taken for one attached to another network.
Fixes#14516
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Several exported symbols describe a contract Compose never honors, and a
reader of the package cannot tell: the stack status constants (Stack.Status is
never one of them), ConfigOptions and ServiceStatus (no operation takes or
returns them), the error sentinels ErrAlreadyExists, ErrForbidden, ErrUnknown
and ErrUnsupportedFlag (never returned, with their Is helpers), and the
ExitCodeLoginRequired exit code of the cloud login Compose no longer has.
Mark them Deprecated, with no change of behavior, so that they can be
removed with the next breaking release. Nothing in the repository uses them.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Exec, port, export, commit and attach fail with "service X is not running"
when they find no container to act on. A program driving Compose as a library
could only recognise that failure by matching the message.
Return a ServiceNotRunningError, which matches the new ErrNotRunning and
carries the service and the replica index. The message is unchanged.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
CreateOptions.Recreate and RecreateDependencies are strings, and any value
other than force or never silently behaved like diverged: a typo such as
"forced" was accepted and did the opposite of what the caller meant.
Add CreateOptions.Validate, which rejects such values with an
InvalidOptionError matching the new ErrInvalidOption, and call it from
Create and, through UpOptions.Validate, from Up, before anything is
touched. The empty string keeps meaning diverged, and the Recreate*
constants are unchanged, so existing callers are not affected.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
An options struct left at its zero value does not always behave like the
matching command run without flags, because the CLI fills in some defaults
before calling the API (Inherit is the sharpest case: its zero value is the
opposite of the CLI default). Index 0 and All also mean different things
depending on the operation.
Describe them in a package comment, and on the fields they concern. No code
change.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The credentials assertion in TestImageDigestResolverUsesDistributionDigest
was vacuous: EncodedAuth encodes even an empty AuthConfig, so
EncodedRegistryAuth != "" always holds. Decode the header and check the
username and password instead.
Build the digest resolver once in generateImageDigestsOverride rather than
once for the services and once for the jobs.
Move the "registry descriptor digest, not the local content digest"
rationale from the pkg/compose wrapper onto registry.ImageDigestResolver,
where the code now lives.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
fakeBackend.ImageDigestResolver in the publish tests was a verbatim copy of
imageDigestResolver in pkg/compose, and Backend.ImageDigestResolver was
redundant with APIClient() + AuthProvider().
Move the resolver to registry.ImageDigestResolver, call it from pull and from
internal/publish, and drop the method from the Backend interface, from
publishBackend and from the fake. The publish digest tests now run on the real
helper. The resolver test moves with it and also pins the caller's context and
a non-empty registry auth, which the publishBackend wiring test used to cover.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Publish already runs inside the "publish" progress bracket, and the public
Push would open a second one. The comment in internal/publish only covers
the other half of that invariant.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The publish tests run against their own fake backend, so nothing checked
the adapter between a composeService and internal/publish. Pin it the way
TestImagesBackendWiring pins the images one: the accessors follow the
service, dry run follows s.dryRun, a refused prompt stays refused, Push
reaches the unexported push with its options (the public Push would open a
second "push" bracket inside the "publish" one), and the image digest
resolver uses the caller's context and the service's credentials.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The e2e framework (framework.go, scenario.go, checks.go, assert.go) and
the e2e tests lived under pkg/e2e, so the framework was published as part
of the public /v5 API although it is only test support code. Move it to
internal/e2e.
This is a pure move: the files keep their content, only the references to
the old path (Makefile, .golangci.yml, codecov.yml, .gitignore, AGENTS.md
and SCENARIO.md) are updated. internal/e2e is at the same depth as
pkg/e2e, so the relative paths to bin/, relay/ and testdata keep working.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Rebuild only forwarded to rebuild, and its sole consumer was the
composeService.rebuild forwarder, itself called by a single test. Keep one
exported Rebuild and let that test reach it through the devwatch service.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The watch loop, its rebuild scheduler and their tests move to
internal/devwatch. The package declares the Backend it expects from the
engine; composeService implements it and keeps one-line forwarders.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The classic and bake builders, with the dispatch between them, become
internal/build, which declares what it needs from the engine as a small
Backend. composeService implements it and keeps a one-line build
forwarder, so Build, watch and the image phase of up and run do not change.
The environment variable resolver, shared with run, becomes its own
internal/envresolver package, and the Building/Built events move to
internal/progress next to the other events.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The documented contract of EventProcessor.Done is success == true when
the operation succeeded, but Run was passing err != nil, so custom
processors received an inverted value. No built-in writer uses the
argument, which is why this went unnoticed.
Pass err == nil, document the meaning of success on the interface, and
add a test covering both the success and the failure case.
Fixes#14498
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Hooks are skipped silently under --dry-run. Log each skipped hook at
debug level so --verbose shows which one would have run.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
WithDryRun re-initializes the Docker CLI with fresh client options,
whose empty log level makes Initialize reset logrus to info. Under
--dry-run, --verbose was therefore dropped as soon as the backend was
created, hiding every debug log of the operation.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The dry-run client cannot attach to an exec, so any lifecycle hook
reached under --dry-run (pre_stop on down/stop/restart, post_start on
up/restart) made the command fail with "interactive exec is not
supported in dry-run mode".
A dry-run must have no side effects, so skip the hook in runHook, which
every hook phase goes through, rather than guarding each call site.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
A Copilot review finding on TestProviderScaleReachesTheProvider: the
scenario left both central guarantees unobserved -- the second `compose
scale` call only checked a successful exit, and no relay container ever
existed to assert singularity against (PROVIDER_DEMO_ENDPOINT was unset).
It would have passed even if a later scale silently lost the requested
value, or if relay creation mishandled it.
The scenario now enables endpoint publication (as in
TestProviderPublishEndpoint) so a real relay container exists, includes
the dependent service in each scale call's targets so it is (re)created
in the same call, and asserts against real state per the project's own
e2e conventions (pkg/e2e/SCENARIO.md: state-based checks over output
matching) -- ContainerEnv for the exact value the provider received,
ServiceScale("db", 1) for the relay staying a single container across
two different scale requests.
Verified: the full provider e2e subset re-run clean against a locally
built binary, golangci-lint clean.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A provider's relay is a singleton network stand-in, never itself a
replica: reconcileService's provider branch always plans exactly one
OpRunProvider node, regardless of the scale requested through
`compose scale` or `scale:`. That value is neither rejected nor acted
on by Compose -- it is already passed through unmodified as part of
the full service config a provider receives over its control channel
(GetServiceConfigType marshals the whole resolved ServiceConfig).
Interpreting it -- scaling the real resource the provider manages,
ignoring it, or rejecting it itself -- is entirely the provider's own
call.
The example provider used by the e2e suite is extended to read that
response into the real types.ServiceConfig (the same struct Compose
marshaled) and call GetScale() on it -- the canonical accessor, which
already reconciles the scale: and deploy.replicas: spellings -- then
surfaces the result as CONFIG_SCALE, exercising the path end to end
the same way it already does for CONFIG_TYPE.
Two regression tests:
- TestReconcileService_ProviderScaleNeverMultipliesTheRelay locks the
plan-level invariant: a declared scale of 3 still produces exactly
one OpRunProvider node.
- TestProviderScaleReachesTheProvider (e2e) locks the value actually
reaching the provider through a real daemon, that `compose scale`
itself converges without error on a provider-backed service, and
that the relay stays a single container across two different scale
requests.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; the full provider e2e subset re-run clean against a locally
built binary.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A manifest with no media type was taken for an image index when it had a
"manifests" key, even when its value is null. It is now an index only when
that value is a list.
writeCacheEntryManifest reported every failure to create the directory
holding the manifest as an invalid OCI artifact. Only the collision with a
file of the artifact of that name is one; any other failure (missing
directory, permission, full disk) is the host's and is now returned as it is.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The media type a registry reports with a manifest is not authenticated, only
the content is: a registry could serve the authentic bytes of an image index
under the media type of a manifest, which was then read as one, extracted as
nothing and cached as a complete entry that holds no compose file.
The document is now read as what it says it is, and a registry that reports
another type for it is refused. An entry is only put in place once it holds the
compose file that is loaded from it.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
An entry of the cache of extracted projects was taken as complete as soon as
its directory, or a marker file in it, was there. An artifact chooses, through
the annotations of its layers, the files it writes in that directory, so it can
write a marker itself, and so did what releases that did not verify the
manifests they fetched had cached there: neither proves that the directory is
the extraction of the artifact it is named after, or that the extraction ended.
The entries of OCI artifacts, in the oci directory of the cache, now hold the
manifest they were extracted from, as the registry served it (the index when
the reference is an image index), in a .manifest directory of their own. An
artifact names the files of an entry without a path separator, so it can't
write in a directory of it: what is in .manifest was put there by Compose,
which refuses an artifact that has a file of that name. A directory is an entry
for a digest only when it is named after it and holds the manifest that has it,
a check which needs no registry and holds offline. A directory without one, or
with the manifest of another digest, is not an entry.
To make the manifest mean that the entry is complete, the entry is extracted
aside, the manifest is written last, and the directory is moved in place in one
step: nothing is in place which is not complete, so a run never finds an entry
without its manifest. What is in place and is not an entry is set aside and
replaced, never completed, and a run that finds the entry in place after another
run uses it. What earlier releases cached is at the root of the cache, where
nothing looks for it any more.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Get returned the manifest the registry served without checking it against
the digest of the descriptor: the one of the reference when it is digested,
the one the registry announces in Docker-Content-Digest when it is a tag.
Neither is checked by containerd on this path, which doesn't go through its
content store, and GetBlob already checks what it fetches for that reason.
The extracted project cache is keyed by that digest alone, whatever the
registry, so a registry could have its content cached as the one of another
manifest, which a later load of the pinned digest, from the registry that
holds it, then returned.
Get now fails when the manifest doesn't hash to its digest, and shares the
check with GetBlob.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The cache of remote resources now has one directory for each kind of resource:
the checkouts of Git repositories are in "git", and the extracted OCI artifacts
are in "oci" rather than at the root of the cache.
What earlier releases left at the root of the cache is no longer looked for: an
artifact is pulled again the first time it is used.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A commit was checked out directly into its directory of the cache, which was
then taken for complete as soon as it existed: a run interrupted during the
fetch or the checkout left a partial directory that every later run used.
The checkout is now made in a directory of its own and renamed to the one of the
commit once it is complete, so the directory of a commit is never a partial
checkout, and no file beside it is needed to tell. Checkouts live in a "git"
directory of the cache, and what earlier releases left at the root of the cache
is no longer looked for.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A run whose replacement of a directory that is not an entry failed put
that directory back unconditionally. Another run having set aside the
entry that took its place in the meantime, the directory put back stood
in the way of the final rename of that run, whose deferred removal then
deleted the complete entry it had set aside: the cache ended with the
incomplete directory after a run returned success.
Only a complete entry is worth restoring, so the restore is guarded.
A run that loses to another may still return an error: the concurrency
test now pins the path of the entry for the runs that succeed, and the
state of the cache once all of them are over.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
When a run replaces a directory of the cache that is not a complete entry, it
sets it aside before removing it, so that it does not remove the entry that
another run has put in place in the meantime. The place it is set aside in
is now a temporary directory of its own, so that two runs replacing the same
entry at the same time never share one.
If the extracted entry can't be moved in place after that, and nothing else
took the place of what was set aside, it is put back instead of being removed
with the rest.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
An entry of the cache of extracted projects was taken as complete as soon as
its directory existed, while the files were written into it one after the
other. A run interrupted in the middle of an extraction, or another run that
loads the same artifact at the same time, left a directory that was then used
as it was, with some of the files missing.
The project is now extracted in a temporary directory of the cache, a
.complete marker is written last, and the directory is moved in place in one
step. A directory of the cache without the marker is not an entry: it is
extracted again, and replaced, never completed. A run that finds the entry
in place after another run uses it.
The entries written by the previous releases have no marker, so each one is
extracted again the first time it is used.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Networks and volumes are created by plan nodes, yet createNetwork and
createVolume still emitted their own events on top of the executor's, so
each line showed twice.
A failing node also reported its error under the plan's internal ID
(service:web:1, network:front), not under the row it had opened. Container
rows stayed on Creating and consumers keying on the ID saw rows nothing
opened.
Make the executor the only emitter and name the row through one helper.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Scope image pruning to selected services and their dependents. Keep
images of declared but unselected services when removing orphans.
Fixes#14453
Signed-off-by: Seyed Noorollah Aghayenoor <noorollah.aghayenoor@gmail.com>
run forces the scale of its target to one, since a container is created
whatever the project says, but the warning looked the service up again in
the project and skipped it when declared with `scale: 0` or
`deploy.replicas: 0`. The per-service check now takes the service
itself, and run hands it the copy it actually creates the container
from. create and up keep skipping services scaled to zero.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
uid, gid and mode on a service-level configs:/secrets: reference are applied
when the object is copied into the container (content: and environment:
sources) and dropped when it is bind-mounted (file: sources). That is a
property of how the container is created, not of the compose file, and an
engine able to set ownership on bind mounts would change the answer.
Report them from create(), next to the other creation-time warnings, once
per requested service, instead of from the load-time unsupported-attribute
check. The false positive on inline and environment sources goes away
without post-filtering findings against the loaded project.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Add TestImagesBackendWiring: the images tests now run against their own fake
backend, so nothing checked that the adapter follows the service's concurrency
bound and the negotiated API version.
Comments only otherwise: say why Backend is narrow rather than embedding
engine.Backend, cross-reference the API version constants declared in both
packages, name down instead of the pkg/compose symbols, point provider_images
to LocalContentDigest, document Summarize, and use the Pruner and PruneOptions
names.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Move the images command, the identity of a local image and the removal of
project images out of pkg/compose into internal/images, behind a small
Backend interface. composeService keeps one-line forwarders so its callers
(build, pull, down, watch) don't change, and ImagePruner, ImagePruneOptions
and NewImagePruner stay exported as aliases.
No behavior change.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The e2e job installed Docker through get-docker.sh, which resolves
packages from the distribution's apt repository. ubuntu-latest moved to
26.04, whose repository has no Docker 28, so the oldstable rows could
no longer install their engine.
Install it with docker/setup-docker-action instead, with binaries that
do not depend on the distribution: stable follows the latest release,
oldstable follows the 28 image tag. The action starts its engine on its
own socket, selected with set-host, so the e2e framework now forwards
DOCKER_HOST, and TestBuildTLS clears it to keep reaching its dind
builder.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
newMonitorTestFixtureListing still returned the unexported monitor type
that moved to internal/logstream, so the compose package tests no longer
compiled. Return *logstream.Monitor like its sibling fixture,
newMonitorTestFixture.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The rebuild forces the recreation of the service it rebuilds and leaves the
providers alone, and the test passed without either option. Give it an up to
date container of the rebuilt service, expected to be removed and created
anyway, and a provider-backed service depending on it, whose provider must not
run.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>