The error message produced by containerd changed in [containerd@7f0c0b8];
"reading blob sha256:536319e163bc3cc681d5ce1a5c88aab6e4afb4b7315ce62acfca619ad36717ee: httpReadSeeker: read past expected size 35: failed precondition
[containerd@7f0c0b8]: 7f0c0b822e
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Running a provider plugin (finding its binary, executing it, serving the
JSON messages it sends, applying the variables and endpoints it publishes,
streaming it an image) needs from the engine the Docker API, the event bus,
the Docker CLI that locates the plugin, the preparation of the environment
of a shelled-out command, and the lock that guards the project model every
concurrent provider run mutates.
It now lives in internal/provider behind a Backend interface that
composeService implements in pkg/compose/plugins.go, next to runPlugin, which
stays as a one-line forwarder so none of its callers (up, down, stop, pull)
changes. The relay it deploys goes through internal/relay, whose Backend the
new one extends. The public types and constants of the protocol (JsonMessage,
ProviderMetadata, the *Type message constants...) stay exported from
pkg/compose as aliases of the new package's.
The tests of the plugin protocol move with the code, assertions unchanged;
those of the provider image phase and of the relay link network removal
stay in pkg/compose. The forwarders to the relay that only the plugin code
used are dropped.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The relay container and the relay link networks that stand in for a
provider service need three things from the engine: the Docker API, the
event bus and whether Docker Desktop is the engine.
They now live in internal/relay behind a Backend interface that
composeService implements in pkg/compose/relay.go, which keeps the names
the lifecycle code calls (ensureServiceRelay, isRelayContainer,
checkRelayTarget...) as one-line forwarders, so none of its callers change.
The one down-specific piece, turning the relay link networks into down
operations gated on the shared limiter, stays in pkg/compose together with
its tests; the other relay tests move with the code, assertions unchanged.
The progress events and the container name the package needs come from two
small shared packages, internal/progress and internal/naming, which the
following moves reuse. pkg/compose keeps its own copies until the shared
kernel is extracted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
LoadProject and Generate (the api.ProjectLoader operations) need two things
from the engine: the Docker API, to read the containers a project is
generated from, and the Docker CLI, which resolves remote Compose files.
They now live in internal/projectloader, along with the unsupported
attributes detection only the loader uses, behind a Backend interface that
composeService implements in a new file of pkg/compose, like internal/inspect.
The tests move with the code; their assertions are unchanged, only the way
the service is built differs.
The canonical container name, which both this package and internal/inspect
need, moves to a tiny internal/naming package instead of being copied a
third time. pkg/compose keeps its own copy until the shared kernel is
extracted.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ps, ls, top, port, volumes, events and viz only read the state of a
project: they need the Docker API, the list of the project's containers
and the concurrency bound, nothing else of composeService.
They now live in internal/inspect, which declares that need as a Backend
interface. composeService implements it in a new file of pkg/compose and
forwards the seven api.Inspector methods, so no other file of pkg/compose
changes, and neither does the behavior: the service-level tests stay in
pkg/compose and run unchanged through the forwarders.
The few tiny helpers the operations share with the rest of pkg/compose
(canonical container name, concurrency-limited errgroup, project filter)
are copied for now: the shared kernel they belong to is a later step.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The client replaced ErrSecretsEngineNotAvailable with a typed ConnectError. Any failure to reach the secrets engine now falls back to the Docker CLI credentials and is logged at debug level.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Docker Desktop stores Docker Hardened Images credentials under https://dhi.io, so return that as the server address of dhi.io credentials, as StagingIndexServer does for the staging registry.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Docker Hardened Images on dhi.io authenticate with Docker Hub accounts (the registry advertises service=registry.docker.io), so resolve dhi.io credentials from the production Docker Hub session as well, with dhi.io as the server address. Other registries, including look-alikes of dhi.io, still use the Docker CLI credentials.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Rename isExpectedMiss to isSessionUnavailable, after what it reports, and shorten or drop doc comments in the Docker Desktop credentials provider.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
A lookup that found no usable session disabled Desktop credentials for the rest of the process. That is right within one command, where retrying a hung engine would add the lookup timeout to every image, but the sessions are process wide and pkg/compose is also used as a library: in a long-running process a transient timeout, or signing in to Desktop later, was never picked up. Hold lookups off for a minute instead of for good.
Also name the expected-miss classification and test it, including dockerhub.ErrNoDefaultProfile, which wraps dockerhub.ErrNoSession.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
It covered the resolver credential mapping and containerd grant selection, neither changed by this PR. The token-as-password contract is asserted on the provider output and on the encoded X-Registry-Auth in internal/registry.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Shorten the doc comments added for the Docker Desktop credentials provider to what the code does not already say, and drop them from the tests.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Docker Desktop keeps the Docker Hub staging session in separate secrets engine realms, read with dockerhub.Staging(). The environment is picked per registry host rather than with a global stage flag: the engine resolves docker.io to the production registry even when Desktop runs in stage mode, so the staging session is only used for registry-1-stage.docker.io and the production session only for the production Hub hosts. A token is never sent to the registry of the other environment.
Each environment has its own process-wide session and cache, and falls back to the Docker CLI config file independently. Staging credentials use https://registry-1-stage.docker.io/ as server address.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
The session payload may carry no claims, in which case the access token was reused for a minute without any expiry check, so an expired token could reach the registry and fail the pull instead of falling back. Read the exp claim from the JWT itself in that case. An expiry that cannot be read is still tolerated.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
Docker Hub credentials are now read first from the session of the account signed in to Docker Desktop, through the Docker Desktop secrets engine (github.com/docker/secrets-engine/client), and from the Docker CLI config file and its credential helpers otherwise. Other registries keep using the config file only.
The OAuth access token is passed as the password of the signed-in account, never as an IdentityToken, which containerd sends as a refresh token. It flows unchanged to the engine in X-Registry-Auth and through the OCI resolver as an OAuth password grant.
Compose falls back to the config file when Desktop is not running, nobody is signed in, the session is expired or about to expire, or the lookup fails (the last logged as a warning). The session is shared process wide, cached until shortly before it expires, and an unusable session is not asked for again. Lookups are bounded by a 5s timeout, as the SDK applies none by default.
Used by pull, push, publish, config --resolve-image-digests, oci:// remote resources and dry-run. The classic builder and use_api_socket keep config file credentials, as a short-lived token copied into a container would go stale.
Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
down's outer ops dispatch bounded only how many ops ran concurrently,
separate from the semaphore already shared by image removal -- a
single network plus --rmi could spend up to 2x the configured
budget. Network and volume removal now share that same semaphore.
up's post-attach log streaming called ContainerInspect directly and
passed no limiter to doLogContainer, unlike `compose logs --follow`'s
equivalent path, so it never shared --parallel's budget either.
The plan executor also held a node's slot for its whole dependency
wait instead of just its own work, and internal/sync/tar.go's file
sync used a hardcoded cap of 16 instead of --parallel.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
compose-spec/compose-go#866 is merged: jobs top-level element, container
specification layered as ContainerSpec/WorkloadSpec, pre_start hooks
carrying the full container specification resolved at load time. Bump
to the merged head and adapt in the same movement — composite literals
setting moved fields wrap them into the embedded ContainerSpec /
WorkloadSpec (promoted field access was already source-compatible, so
this is literal-only, no behavior change), and pre_start handling is
typed against PreStartHook.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
OTel SDK/exporter errors already go through docker/cli's own
otel.SetErrorHandler(debug.OTELErrorHandler), installed by the plugin
framework at startup. Our own handler, installed at package init time,
was always clobbered before any command ran (plugin.Run() sets its handler
after all inits execute), so it never had any effect - verified by tracing
the exact call order and reproducing the flag behavior against a built
binary.
Drop that dead handler along with the COMPOSE_OTEL_DEBUG env var it
introduced. The one genuinely-discarded error (the tracer shutdown/flush
error in wrapRunE) now goes through logrus like everything else in the
codebase: it's already gated by --debug/-D via logrus's own level, so no
explicit check is needed.
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
EncodedAuth now delegates the X-Registry-Auth wire encoding to
moby/api/pkg/authconfig (same module, already required) instead of
hand-rolling the JSON+base64url pair, and push.go reuses it instead of
its own copy. The CLI and moby AuthConfig structs are field-for-field
identical, so a direct type conversion bridges them.
The explicit GetAuthConfigKey normalization drops out of EncodedAuth:
docker/cli's configfile.GetAuthConfig normalizes the Docker Hub domain
itself. It remains for the OCI resolver, whose credential callback
receives the network host actually contacted (registry-1.docker.io),
which the configfile does not map.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
buildx was imported for two narrow purposes, both replaced by
docker/cli machinery compose already uses elsewhere:
- storeutil.GetProxyConfig -> configfile.ParseProxyConfig, the same
lookup docker build uses (also picks up ALL_PROXY and lowercase
variants)
- the dry-run image resolver (buildx builder + imagetools) -> a
DistributionInspect through the daemon with credentials from the CLI
config file, i.e. the same path a real pull/push would take; the
encoded-auth helper moves to internal/registry, shared with pull.go
bake is unaffected: it already execs the buildx CLI plugin. Dropping
the module also drops the transitive aws-sdk-go-v2 footprint
(-328 lines of go.sum).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Pulling an oci:// resource resolved each layer digest through the
registry manifests endpoint, which answers 500 when the digest points
to a non-manifest blob. containerd v2.3.0+ (pulled in by buildx v0.36
and buildkit v0.32) no longer falls back to the blobs endpoint unless
manifests returned 404, so publish/pull of compose artifacts broke.
Fetch layers directly with the descriptors already listed in the
manifest instead of resolving them again.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The transport passed to NewResolver was applied only via
docker.WithClient. Pass it via docker.WithAuthClient too so the
authorizer's OAuth token fetches use it instead of http.DefaultClient.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
`docker compose publish` routed all registry traffic through Docker
Desktop's HTTP proxy. Publishing to a registry on localhost therefore
failed on Windows with:
proxyconnect tcp: open ./pipe/dockerHttpProxy: The system cannot
find the path specified.
even though `docker push`/`docker pull` worked against the same registry.
Two bugs in internal/desktop/proxy.go:
1. No loopback bypass. ProxyTransport forced every request through the
Docker Desktop proxy and its DialContext always dialed the proxy
socket, so loopback targets could never connect directly. Proxy
selection now bypasses the proxy only for loopback targets
(localhost, 127.0.0.0/8, ::1); all other registry traffic stays
routed through Docker Desktop's PAC-aware proxy so Desktop keeps
ownership of proxy decisions (e.g. enterprise-managed proxies). The
local process NO_PROXY/no_proxy is deliberately not honored, so a
broad value such as * or .corp cannot bypass centrally managed
proxy policy.
2. Malformed Windows pipe path. The proxy named-pipe endpoint was
hardcoded as npipe://./pipe/..., yielding the relative path
./pipe/dockerHttpProxy. It is now derived from the engine endpoint,
preserving its namespace. Docker Desktop reports the backslash form
npipe://\\.\pipe\docker_cli, so the derivation uses LastIndexAny to
handle both backslash and forward-slash forms.
Publishing to localhost now connects directly like `docker push`, while
every non-loopback registry still goes through the Docker Desktop proxy.
Fixes#13824
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Domantas Petrauskas <dom.petrauskas@gmail.com>
Pass the active project name as the appId query parameter on the
docker-desktop://dashboard/logs deep link, both from the post-command
hint (compose up -d, compose logs) and the interactive nav menu
('l' key during compose up). The hook subprocess re-runs compose-go's
project loader so the name matches what the parent computed; it skips
the appId when -p, -f, --project-directory, --workdir, or --env-file
is set, since the hook payload does not carry their values. docker
logs stays unfiltered: the CLI hook contract does not expose the
positional container id.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Docker Desktop is removing the "Enable Logs view" beta setting, so drop
the /app/settings check and rely on /features alone. With the setting
gate gone, the compose hook subprocess would print the Logs view hint
regardless of LogsTab; add a flag check in handleHook. Consolidate
engine-label discovery and feature-flag evaluation into internal/desktop.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The compose process performs OCI artifact fetches in-process via
containerd's docker resolver, whose default transport only honors
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars. Users behind PAC-only
corporate proxies hit i/o timeouts on `oci://` includes and on
`compose publish`.
When Docker Desktop is the active engine and exposes httpproxy.sock,
route the resolver through it (PAC-aware). Falls back to the default
transport when DD is unavailable or the socket is missing. Modeled on
docker/mcp-gateway PR #354.
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Add CLI hooks handler to show "What's next:" hints pointing to the
Docker Desktop Logs view after `docker logs`, `docker compose logs`,
and `docker compose up -d`.
Add `l` keyboard shortcut in the `compose up` navigation menu to
open the Logs view, gated on Docker Desktop feature flag and settings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Previously, Sync() only checked for fs.ErrNotExist when classifying
paths into copy vs delete. Non-NotExist stat errors (e.g. EACCES,
EIO) caused the condition to be false, falling through to the else
clause which incorrectly treated the path as copyable. This masked
real errors and led to cryptic failures downstream.
Restructure the condition into a three-way branch:
- err == nil → copy
- ErrNotExist → delete
- other errors → return immediately with context
This follows the pattern already used by entriesForPath() in the
same file.
Fixes#13654
Signed-off-by: Lidang Jiang <lidangjiang@gmail.com>
Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
Before this, assertion libraries were mixed, sometimes
even in the same file.
git grep -l '"gotest.tools/v3/' | wc -l
75
git grep -l '"github.com/stretchr/testify' | wc -l
24
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Results of running the modernize command, with some minor changes
afterwards (removing the `contains` and `hasStatus` helper functions);
go install golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest
modernize -fix ./...
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also update TestDefaultNetworkSettings:
Test that the network with the highest priority is returned as
"primary" network, and other networks as extra networks.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Replace manual context creation with t.Context() which is automatically
cancelled when the test completes.
Go 1.24 modernization pattern.
Assisted-By: cagent
Signed-off-by: David Gageot <david.gageot@docker.com>
When using --env-file=~/.env, the tilde was not expanded to the user's
home directory. Instead, it was treated as a literal character and
resolved relative to the current working directory, resulting in errors
like "couldn't find env file: /current/dir/~/.env".
This adds an ExpandUser function that expands ~ to the home directory
before converting relative paths to absolute paths.
Fixes#13508
Signed-off-by: tensorworker <tensorworker@proton.me>
Most files already grouped imports into "stdlib -> other -> local",
but some files didn't. The gci formatter is similar to goimports, but
has better options to make sure imports are grouped in the expected
order (and to make sure no additional groups are present).
This formatter has a 'fix' function, so code can be re-formatted auto-
matically;
golangci-lint run -v --fix
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>