Commit graph

53 commits

Author SHA1 Message Date
Nicolas De Loof
7e5cddad37 fix(oci): tell a null "manifests" from a list, and report a host failure as such
A manifest with no media type was taken for an image index when it had a
"manifests" key, even when its value is null. It is now an index only when
that value is a list.

writeCacheEntryManifest reported every failure to create the directory
holding the manifest as an invalid OCI artifact. Only the collision with a
file of the artifact of that name is one; any other failure (missing
directory, permission, full disk) is the host's and is now returned as it is.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-12 10:43:27 +01:00
Nicolas De Loof
fb1fd129f8 fix(oci): read a manifest as the document it says it is, and keep nothing of one with no compose file
Some checks are pending
ci / validate (lint) (push) Waiting to run
ci / validate (relay-lint) (push) Waiting to run
ci / validate (relay-test) (push) Waiting to run
ci / validate (validate-docs) (push) Waiting to run
ci / validate (validate-go-mod) (push) Waiting to run
ci / validate (validate-headers) (push) Waiting to run
ci / validate (validate-mocks) (push) Waiting to run
ci / binary (push) Waiting to run
ci / binary-finalize (push) Blocked by required conditions
ci / bin-image-test (push) Waiting to run
ci / relay-image-test (push) Waiting to run
ci / test (push) Waiting to run
ci / e2e (plugin, oldstable, graphdriver) (push) Waiting to run
ci / e2e (standalone, oldstable, graphdriver) (push) Waiting to run
ci / e2e (plugin, stable, containerd) (push) Waiting to run
ci / e2e (plugin, stable, graphdriver) (push) Waiting to run
ci / e2e (standalone, stable, containerd) (push) Waiting to run
ci / e2e (standalone, stable, graphdriver) (push) Waiting to run
ci / coverage (push) Blocked by required conditions
ci / ci-ok (push) Blocked by required conditions
ci / release (push) Blocked by required conditions
merge / bin-image-prepare (push) Waiting to run
merge / bin-image (push) Blocked by required conditions
merge / relay-image (push) Waiting to run
merge / module-image (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
zizmor / zizmor (push) Waiting to run
The media type a registry reports with a manifest is not authenticated, only
the content is: a registry could serve the authentic bytes of an image index
under the media type of a manifest, which was then read as one, extracted as
nothing and cached as a complete entry that holds no compose file.

The document is now read as what it says it is, and a registry that reports
another type for it is refused. An entry is only put in place once it holds the
compose file that is loaded from it.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-11 08:52:55 +01:00
Nicolas De Loof
c470af4063 fix(oci): take a directory of the cache for an entry only when it holds the manifest of its digest
An entry of the cache of extracted projects was taken as complete as soon as
its directory, or a marker file in it, was there. An artifact chooses, through
the annotations of its layers, the files it writes in that directory, so it can
write a marker itself, and so did what releases that did not verify the
manifests they fetched had cached there: neither proves that the directory is
the extraction of the artifact it is named after, or that the extraction ended.

The entries of OCI artifacts, in the oci directory of the cache, now hold the
manifest they were extracted from, as the registry served it (the index when
the reference is an image index), in a .manifest directory of their own. An
artifact names the files of an entry without a path separator, so it can't
write in a directory of it: what is in .manifest was put there by Compose,
which refuses an artifact that has a file of that name. A directory is an entry
for a digest only when it is named after it and holds the manifest that has it,
a check which needs no registry and holds offline. A directory without one, or
with the manifest of another digest, is not an entry.

To make the manifest mean that the entry is complete, the entry is extracted
aside, the manifest is written last, and the directory is moved in place in one
step: nothing is in place which is not complete, so a run never finds an entry
without its manifest. What is in place and is not an entry is set aside and
replaced, never completed, and a run that finds the entry in place after another
run uses it. What earlier releases cached is at the root of the cache, where
nothing looks for it any more.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-11 08:52:55 +01:00
Nicolas De Loof
2ae22f6e29 fix(oci): verify the manifests fetched against their digest
Get returned the manifest the registry served without checking it against
the digest of the descriptor: the one of the reference when it is digested,
the one the registry announces in Docker-Content-Digest when it is a tag.
Neither is checked by containerd on this path, which doesn't go through its
content store, and GetBlob already checks what it fetches for that reason.

The extracted project cache is keyed by that digest alone, whatever the
registry, so a registry could have its content cached as the one of another
manifest, which a later load of the pinned digest, from the registry that
holds it, then returned.

Get now fails when the manifest doesn't hash to its digest, and shares the
check with GetBlob.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-11 08:52:55 +01:00
Nicolas De Loof
83c50295cb refactor(oci): keep the extracted artifacts in an "oci" directory of the cache
Some checks are pending
ci / validate (lint) (push) Waiting to run
ci / validate (relay-lint) (push) Waiting to run
ci / validate (relay-test) (push) Waiting to run
ci / validate (validate-docs) (push) Waiting to run
ci / validate (validate-go-mod) (push) Waiting to run
ci / validate (validate-headers) (push) Waiting to run
ci / validate (validate-mocks) (push) Waiting to run
ci / binary (push) Waiting to run
ci / binary-finalize (push) Blocked by required conditions
ci / bin-image-test (push) Waiting to run
ci / relay-image-test (push) Waiting to run
ci / test (push) Waiting to run
ci / e2e (plugin, oldstable, graphdriver) (push) Waiting to run
ci / e2e (standalone, oldstable, graphdriver) (push) Waiting to run
ci / e2e (plugin, stable, containerd) (push) Waiting to run
ci / e2e (plugin, stable, graphdriver) (push) Waiting to run
ci / e2e (standalone, stable, containerd) (push) Waiting to run
ci / e2e (standalone, stable, graphdriver) (push) Waiting to run
ci / coverage (push) Blocked by required conditions
ci / ci-ok (push) Blocked by required conditions
ci / release (push) Blocked by required conditions
merge / bin-image-prepare (push) Waiting to run
merge / bin-image (push) Blocked by required conditions
merge / relay-image (push) Waiting to run
merge / module-image (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
zizmor / zizmor (push) Waiting to run
The cache of remote resources now has one directory for each kind of resource:
the checkouts of Git repositories are in "git", and the extracted OCI artifacts
are in "oci" rather than at the root of the cache.

What earlier releases left at the root of the cache is no longer looked for: an
artifact is pulled again the first time it is used.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-09 14:38:36 +01:00
Nicolas De Loof
b5709cd309 fix(git): check a commit out aside, and keep the checkouts in a directory of the cache for Git
A commit was checked out directly into its directory of the cache, which was
then taken for complete as soon as it existed: a run interrupted during the
fetch or the checkout left a partial directory that every later run used.

The checkout is now made in a directory of its own and renamed to the one of the
commit once it is complete, so the directory of a commit is never a partial
checkout, and no file beside it is needed to tell. Checkouts live in a "git"
directory of the cache, and what earlier releases left at the root of the cache
is no longer looked for.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-09 14:38:36 +01:00
Nicolas De Loof
2fcf79b3c1 fix(oci): put back only a complete entry when the replacement of a cache entry fails
A run whose replacement of a directory that is not an entry failed put
that directory back unconditionally. Another run having set aside the
entry that took its place in the meantime, the directory put back stood
in the way of the final rename of that run, whose deferred removal then
deleted the complete entry it had set aside: the cache ended with the
incomplete directory after a run returned success.

Only a complete entry is worth restoring, so the restore is guarded.

A run that loses to another may still return an error: the concurrency
test now pins the path of the entry for the runs that succeed, and the
state of the cache once all of them are over.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-05 17:58:58 +01:00
Nicolas De Loof
ff9622c5c5 fix(oci): set aside a replaced cache entry in a place of its own
When a run replaces a directory of the cache that is not a complete entry, it
sets it aside before removing it, so that it does not remove the entry that
another run has put in place in the meantime. The place it is set aside in
is now a temporary directory of its own, so that two runs replacing the same
entry at the same time never share one.

If the extracted entry can't be moved in place after that, and nothing else
took the place of what was set aside, it is put back instead of being removed
with the rest.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-05 17:58:58 +01:00
Nicolas De Loof
4c34acd35f fix(oci): extract a project into the cache aside and mark the entry as complete
An entry of the cache of extracted projects was taken as complete as soon as
its directory existed, while the files were written into it one after the
other. A run interrupted in the middle of an extraction, or another run that
loads the same artifact at the same time, left a directory that was then used
as it was, with some of the files missing.

The project is now extracted in a temporary directory of the cache, a
.complete marker is written last, and the directory is moved in place in one
step. A directory of the cache without the marker is not an entry: it is
extracted again, and replaced, never completed. A run that finds the entry
in place after another run uses it.

The entries written by the previous releases have no marker, so each one is
extracted again the first time it is used.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-11-05 17:58:58 +01:00
Alano Terblanche
171f9d0767 docs(registry): trim comments
Shorten the doc comments added for the Docker Desktop credentials provider to what the code does not already say, and drop them from the tests.

Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
2026-10-07 14:24:28 +02:00
Alano Terblanche
fd4dae8351 feat(registry): use the Docker Desktop Hub session for Docker Hub credentials
Docker Hub credentials are now read first from the session of the account signed in to Docker Desktop, through the Docker Desktop secrets engine (github.com/docker/secrets-engine/client), and from the Docker CLI config file and its credential helpers otherwise. Other registries keep using the config file only.

The OAuth access token is passed as the password of the signed-in account, never as an IdentityToken, which containerd sends as a refresh token. It flows unchanged to the engine in X-Registry-Auth and through the OCI resolver as an OAuth password grant.

Compose falls back to the config file when Desktop is not running, nobody is signed in, the session is expired or about to expire, or the lookup fails (the last logged as a warning). The session is shared process wide, cached until shortly before it expires, and an unusable session is not asked for again. Lookups are bounded by a 5s timeout, as the SDK applies none by default.

Used by pull, push, publish, config --resolve-image-digests, oci:// remote resources and dry-run. The classic builder and use_api_socket keep config file credentials, as a short-lived token copied into a container would go stale.

Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
2026-10-07 14:24:28 +02:00
Sebastiaan van Stijn
063ef45da1 golangci-lint: enable perfsprint linter
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2026-09-10 10:25:34 +02:00
Nicolas De Loof
d6e26f427f refactor: split ociRemoteLoader.Load pull and index resolution
Load (cognitive complexity 48) mixed the enable/offline guards with
the artifact pull, the cache check and the image-index indirection.
The pull path moves to pullComposeArtifact and the index-to-manifest
resolution to resolveComposeManifest. Load reads as: guards, cache
lookup, pull on miss.

No behavior change, including the last-matching-manifest-wins loop
and the silent cache hit on a non-NotExist stat error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 13:37:48 +02:00
Nicolas De Loof
2340121f0e lint: replace gocyclo with gocognit
Cognitive complexity (gocognit, threshold 30 — the linter default) fits
this codebase better than raw cyclomatic complexity: it barely charges
guard clauses and early returns, and penalizes nesting instead. As a
result 8 of the 21 //nolint:gocyclo suppressions become unnecessary,
while deeply nested functions gocyclo never flagged are now caught.

The 19 functions still above the threshold keep a suppression, each
marked FIXME to complete the migration by restructuring them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2026-08-18 10:42:37 +02:00
Guillaume Lours
cda9f2044f fix(oci): fetch artifact layers from blobs endpoint, not via Resolve
Pulling an oci:// resource resolved each layer digest through the
registry manifests endpoint, which answers 500 when the digest points
to a non-manifest blob. containerd v2.3.0+ (pulled in by buildx v0.36
and buildkit v0.32) no longer falls back to the blobs endpoint unless
manifests returned 404, so publish/pull of compose artifacts broke.
Fetch layers directly with the descriptors already listed in the
manifest instead of resolving them again.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-08-03 14:56:34 +02:00
solunolab
be0c713ca5 chore: fix some comments to improve readability
Signed-off-by: solunolab <solunolab@outlook.com>
2026-06-09 11:45:13 +02:00
Guillaume Lours
66c21c3b6c fix: route OCI artifact pulls through Docker Desktop HTTP proxy
The compose process performs OCI artifact fetches in-process via
containerd's docker resolver, whose default transport only honors
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars. Users behind PAC-only
corporate proxies hit i/o timeouts on `oci://` includes and on
`compose publish`.

When Docker Desktop is the active engine and exposes httpproxy.sock,
route the resolver through it (PAC-aware). Falls back to the default
transport when DD is unavailable or the socket is missing. Modeled on
docker/mcp-gateway PR #354.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
2026-05-18 10:53:23 +02:00
Sebastiaan van Stijn
dc66e6bad1 golangci-lint: use gci formatter instead of goimports
Most files already grouped imports into "stdlib -> other -> local",
but some files didn't. The gci formatter is similar to goimports, but
has better options to make sure imports are grouped in the expected
order (and to make sure no additional groups are present).

This formatter has a 'fix' function, so code can be re-formatted auto-
matically;

    golangci-lint run -v --fix

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2025-12-01 12:21:50 +01:00
Nicolas De Loof
2b4543935c next release will be major version v5.x
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-11-13 09:32:43 +01:00
Nicolas De Loof
7eb5adeef6 introduce --insecure-registry, reserved for testing purpose
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-11-06 16:17:21 +01:00
Suleiman Dibirov
0d396bbacb fix(git): Add validation for Git subdirectory paths to prevent traversal
Signed-off-by: Suleiman Dibirov <idsulik@gmail.com>
2025-10-30 10:08:29 +01:00
Nicolas De Loof
e59150baa8 fix OCI compose override support
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-10-23 12:04:37 +02:00
Nicolas De Loof
6a90742ef2 Test to check writeComposeFile detects invalid OCI artifact
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-10-22 20:09:32 +02:00
Nicolas De Loof
6007d4c7e7 publish env_file references as opaque hash to prevent paths conflicts
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-10-22 19:32:16 +02:00
Guillaume Lours
69bcb962bf Enforce compose files from OCI artifact all get into the same target (cache) folder
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-10-22 19:32:16 +02:00
Nicolas De Loof
07602f2070 publish Compose application as compose.yaml + images
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-10-03 10:59:57 +02:00
Nicolas De Loof
fa08127456 use containerd client for OCI operations
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-09-30 12:03:46 +02:00
Sebastiaan van Stijn
b33ecf65e8 go.mod: bump buildx v0.28.0-rc2, buildkit v0.24.0-rc2
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2025-09-02 07:28:06 +02:00
Sebastiaan van Stijn
1d52012b82 go.mod: bump buildkit v0.24.0-rc1, buildx v0.28.0-rc1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2025-08-29 08:16:23 +02:00
Nicolas De Loof
d62e21025c forward git command error to user
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-07-23 12:10:25 +02:00
atagtm
f7ee9c8a0c feat(os): add FreeBSD support
Signed-off-by: atagtm <donisos1146@gmail.com>
2025-07-11 10:44:45 +02:00
Nicolas De Loof
ee33143026 capture git fetch output when debug output is enabled
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-04-08 12:03:41 +02:00
Nicolas De Loof
66a47169d5 Publish compose file with required siblings used by extends
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2025-03-07 13:58:10 +01:00
Guillaume Lours
b6c8a2b9fc display the location of OCI or GIT Compose stack download
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2025-03-03 22:09:06 +01:00
Guillaume Lours
840288895e add --with-env flag to publish command
this flag allow publishing env variables in the Compose OCI artifact

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2025-01-29 15:45:00 +01:00
Guillaume Lours
26064d4b60 allow usage of -f flag with oci Compose artifact
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2024-10-25 17:36:04 +02:00
Kirill A. Korinsky
fb25e88a03 Backport OpenBSD patches
Here a trivial patch whcih I've used inside OpenBSD port to build it

Signed-off-by: Kirill A. Korinsky <kirill@korins.ky>
2024-05-22 11:35:37 +02:00
Nicolas De Loof
eb4249ece4
bump compose-go to v2.0.0-rc.2
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2024-01-22 12:27:38 +01:00
Nicolas De Loof
cda04f288e adopt compose-go/v2
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-12-05 18:54:31 +01:00
Nicolas De Loof
d646d757a2 lint
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-11-02 11:16:35 +01:00
Nicolas De Loof
71237ef62b do not resolve cache dir until remote resource is in use
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-11-02 11:16:35 +01:00
Nicolas De Loof
0d905a896d add a copyright notice with original author
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-11-02 11:16:35 +01:00
Nicolas De Loof
5e3d8f671d re-implement cache folder detection
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-11-02 11:16:35 +01:00
Guillaume Lours
3d0207ebc8
remove uncessary return value of pullComposeFiles function
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-10-18 12:42:11 +02:00
Nicolas De Loof
599e4b242a
extract method to reduce cyclomatic complexity
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-29 10:18:11 +02:00
Nicolas De Loof
fe8c2780c8 warn user remote resource is disabled
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-29 09:30:38 +02:00
Matthieu MOREL
4f694919ff deps: remove deprecated github.com/pkg/errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-09-29 06:28:58 +02:00
Bjorn Neergaard
a6b7d78575 pkg/remote/oci: check artifactType instead of config.mediaType
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-29 05:27:14 +02:00
Bjorn Neergaard
991901f2ef pkg/remote/oci: refer to the manifest as manifest
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-29 05:27:14 +02:00
Nicolas De Loof
5ca35c88be implement publish
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-20 18:14:58 +02:00