Cover what the removed imperative tests used to: a no-op start is silent and
only lists containers, --wait-timeout bounds the dependency waits, secrets and
configs are injected before ContainerStart, a failing post_start fails the
start, pre_start is skipped on a stale relay, deploy.replicas: 0 starts
nothing, and the project-wide no-container check. On the provider path: the
restart is a chain end, a relay gets no post_start, pre_start is gated on
running replicas and runs once, before the chain, after the dependencies.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
planProviderRelayRestart duplicated the replica chain of planServiceStart
(sort, pre_start gate, bare vs enriched start, post_start), so a fix to one
did not reach the other. Extract it as chainStartReplicas and let the
provider path feed it the replicas startPhaseReplicas already collects
(under pure ScopeStart, the sorted not-running observed containers).
No behavior change: the plans are node-for-node identical.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
startService received the whole project's container list, so "no container
to start" held only for a project without any container. A service without
a container next to existing ones was a no-op and the others still started;
the plan failed the whole start instead.
Plan the error only when nothing is observed in the project, orphans
included.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
start() translated any DeadlineExceeded from the plan into a
dependency-readiness message when --wait-timeout was set, including one
from an engine call or a hook. Use the marker a wait node's own timeout
now carries, as upDetached does.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The final readiness check opened its --wait-timeout window before
re-listing the project's containers, so a slow daemon listing could
exhaust the budget and report "application not healthy" before
waitDependencies had polled once. List first on the caller's context,
then derive the timed one, as the imperative start path did.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A Copilot review finding on planProviderRelayRestart: its chain-building
loop re-checked "is this replica a relay?" on every iteration to decide
whether to run pre_start, so when the lowest-numbered stale container
was a relay (correctly skipped) but a later one was an ordinary stale
replica, pre_start still ran against that later one. The old imperative
engine's own gate -- lowestNumberedContainer(toStart), checked once --
never falls through like that: if its single candidate is a relay,
pre_start is skipped entirely for the whole chain, with no fallback to
a later non-relay container.
Eligibility is now decided once, from the first non-running container
only, exactly mirroring planServiceStart's own !isReplicaRelay(replicas[0])
gate (already reviewed and confirmed as faithful parity). The per-replica
relay check inside the loop is unchanged and still correct for its own,
separate purpose: deciding whether to skip injection/post_start for that
specific replica.
TestPlanStart_ProviderPathPreStartGateDoesNotRetryPastARelay reproduces
the exact shape the finding describes (relay at the lowest number,
ordinary stale replica after it) and is verified by ablation (reverting
the fix reproduces the bug deterministically, 3/3 runs).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; the full provider e2e subset re-run clean against a locally
built binary.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on planProviderRelayRestart: it iterated
r.observed.Containers[service.Name] directly, which preserves the
daemon's own container-list order, not replica-number order --
unlike startPhaseReplicas, which explicitly sorts before building its
chain (see its own sort comment) specifically to carry the plan's
determinism run to run. Two stale containers returned out of order by
the daemon could chain in list order instead of replica-number order.
Sorts a local clone by container number before the loop, matching
startPhaseReplicas' own approach.
TestPlanStart_ProviderPathChainsMultipleStaleContainers now lists its
two observed containers out of order (2 then 1) to exercise this;
verified by ablation (dropping the sort reproduces the wrong chain
order deterministically, 3/3 runs).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Two more review findings on planProviderRelayRestart's generalization
(previous commit), both confirmed real:
- docker-agent (100/100 confidence): the function returned after the
first not-running observed container, silently dropping every
subsequent one. The relay is uniquely named (at most one can exist),
but a service transitioning to provider-backed without an
intervening reconciliation can leave several stale non-relay
replicas behind (e.g. a scale>1 ordinary service whose provider:
declaration was just added) -- only the first ever got a plan node.
Restructured into a sequential chain, mirroring planServiceStart's
own replica-chain pattern: every not-running container gets a node,
each ordered after the previous one's chain end.
- Copilot: the post_start-only version of the enrichment never planned
pre_start for a stale non-relay container. Added, running at most
once, before the first non-relay candidate in observed
(container-number) order -- exactly the imperative engine's own
lowestNumberedContainer(toStart) + isRelayContainer gate, gated on
!anyRunning the same way planServiceStart's own pre_start is.
A third docker-agent finding (57/100 confidence) on planServiceStart's
own `!isReplicaRelay(replicas[0])` pre_start guard was investigated and
is a false positive: replicas is sorted ascending by container number
(see its own sort comment), so replicas[0] already IS the
lowest-numbered candidate -- exactly matching lowestNumberedContainer's
semantics, not an arbitrary index-0 pick. The suggested fix (skip
pre_start only when every replica is a relay) would have been a
behavior change from the old engine's actual parity target, not a bug
fix. Replied on the thread with this analysis; left unresolved for the
reviewer/maintainer to confirm rather than resolving unilaterally.
Two new regression tests, both verified by ablation (reverting each
fix separately reproduces its gap deterministically, 3/3 runs):
TestPlanStart_ProviderPathChainsMultipleStaleContainers and the
extended TestPlanStart_ProviderPathEnrichesStaleNonRelayContainer
(now also asserting pre_start).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Two Copilot review findings on #14296, both rooted in the same mistake:
relay-vs-ordinary handling was decided from the service's CURRENTLY
declared type (service.Provider != nil routes to planProviderStart or
not), not from the observed container itself. The deleted imperative
engine's startService/startServiceContainer always decided bare-vs-
enriched per container (isRelayContainer(ctr)), so it was naturally
robust to a service's provider: declaration changing without an
intervening `up`/create to converge the daemon to match -- under pure
ScopeStart, nothing ever converges, so the observed container can be
stale relative to the current model:
1. planServiceStart's ordinary (ex-provider) path: a service with
provider: removed still has its old relay observed under its name.
The replica loop unconditionally built an enriched OpStartContainer
(Service set), so execStartContainer would inject secrets/configs
and run post_start hooks against a shell-less scratch binary with
no process to act on. Fixed with isReplicaRelay, gating both the
pre_start and post_start/injection paths exactly like the old
startServiceContainer's own relay check did.
2. planProviderRelayRestart (newly-provider) path: a service with
provider: added still has its old normal replica observed under its
name, but the function filtered on isRelayContainer and silently
dropped anything that wasn't one -- `compose start` returned success
without starting the leftover container at all. Generalized to plan
a start for ANY observed, not-running container: bare if it's
actually a relay, enriched (injection + post_start) otherwise,
matching the old engine's per-container decision.
Two new regression tests, both verified by ablation (forcing the old,
declaration-based behavior reproduces each gap deterministically,
3/3 runs): TestPlanStart_OrdinaryPathBareStartsAStaleRelay and
TestPlanStart_ProviderPathEnrichesStaleNonRelayContainer.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Second Copilot review finding in the same vein as the previous
network/volume fix: collectObservedState's mergeHookContainers call
(getHookContainers, an extra ContainerList with a hook-specific label
filter) is consumed exclusively by create-phase runner reconciliation
(planPreStartHookRunners, reconcile.go) -- the start phase's own
RunPreStart never reads observed.HookContainers at all; it executes a
runner the create phase already prepared, or fails outright if none
exists. Performing this query unconditionally meant a no-op
`compose start` on a service declaring pre_start hooks could fail on
a daemon call the old imperative engine never made.
Gated behind the same scope.plansCreatePhase() check as the network/
volume listing.
New regression test TestCollectObservedState_ScopeStartSkipsHookContainerQuery
registers exactly one ContainerList expectation (the default Times(1)):
gomock fails the test if the extra hook-container listing fires under
ScopeStart. Verified by ablation (reverting the guard reproduces the
failure deterministically, 3/3 runs).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14296: collectObservedState unconditionally
listed networks and volumes (plus per-declared-resource Inspect calls
via discoverUnmanagedNetworks/Volumes) regardless of scope. ScopeStart's
planStartPhase never reads resolvedNetworks/resolvedVolumes -- only
reconcileNetworks/reconcileVolumes, which ScopeStart never runs, do --
so these calls were pure overhead for `compose start`, and a new daemon
dependency the old imperative engine never had (it only ever called
getContainers). An authorization plugin permitting container list/start
but denying network or volume listing would make `compose start` fail
before starting anything.
collectObservedState now takes the caller's ReconcileScope and skips
the network/volume listing (extracted into collectObservedNetworksAndVolumes
to keep cognitive complexity in check) when scope.plansCreatePhase() is
false. preparePlan (create.go) threads its own scope through unchanged;
start() passes ScopeStart.
New regression test TestCollectObservedState_ScopeStartSkipsNetworksAndVolumes
registers no NetworkList/VolumeList mock expectation, so gomock fails
the test if collectObservedState calls either under ScopeStart --
verified by ablation (reverting the scope guard reproduces the failure
deterministically, 3/3 runs).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14296: start() copied upDetached's
emitRunningEvents call without checking whether it fit start()'s own,
narrower UX contract. It doesn't -- the imperative engine's
startService emitted nothing at all for an already-running container
(locked by the now-deleted TestStartService_AlreadyRunningIsSilent),
matching compose start's long-standing silence-on-no-op behavior
(e2e's TestStartAlreadyRunning asserts empty stdout for exactly this
case). emitRunningEvents exists for up/create's full project-status
display, which accounts for containers the plan won't touch -- a
concern start() has never had.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel), including the
existing TestStartAlreadyRunning and the full start/stop e2e suite.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14296: planProviderRelayRestart's node was
added as a sibling of the provider's own startPhaseDependencies, not
their dependent. Under pure ScopeStart there is no Create->Start
barrier separating a provider from its siblings the way ScopeCreateStart's
OpRunProvider gets one for free (always a create-phase node, run to
completion before any start-phase node begins) -- so a stopped relay
with a required depends_on condition could start immediately,
concurrently with (or before) that condition's own wait resolving.
planProviderRelayRestart now takes the provider's own depNodes
(startPhaseDependencies' result, already computed by the caller) and
passes them through as the restart node's own DependsOn, exactly like
an ordinary service's bare-restart-under-ScopeStart already does via
its replica chain's `prev := depNodes`.
New regression test TestPlanStart_ProviderRelayRestartWaitsOnDependencies,
verified by ablation (dropping the deps argument reproduces the
missing edge deterministically, 3/3 runs).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Found by the e2e suite (TestProviderPublishEndpoint) the first time
ScopeStart became a real, exercised path (previous commit, `compose
start`): planProviderStart only ever ordered a provider's consumers
after serviceNodes[service.Name], the create-phase OpRunProvider node.
That node is never set under pure ScopeStart -- the create phase that
emits it doesn't run at all -- so a provider's own relay container, if
observed stopped, had no start node planned for it anywhere.
`compose start db` on a provider-backed service returned success
without actually starting its relay.
planProviderRelayRestart emits a bare OpStartContainer for the relay
when it is observed but not running, deliberately mirroring the old
imperative engine's own behavior for a provider's relay (the now-
deleted startService special-cased providers only for the zero-
container case, otherwise treating an existing relay container like
any other container -- a bare ContainerStart, never a redeploy through
ensureServiceRelay/OpRunProvider).
A local Copilot CLI review of the first version of this fix caught a
real bug before it shipped: "serviceNodes absent" is not a reliable
proxy for "the create phase never ran" on its own -- under
ScopeCreateStart with SkipProviders (watch's rebuild), reconcileService
deliberately skips the provider and leaves serviceNodes unset too, an
explicit request to leave the relay alone, not an invitation to
restart it regardless. planProviderRelayRestart is now also gated on
r.options.SkipProviders.
Three golden-plan regression tests, all verified by ablation (reverting
the relevant guard reproduces the bug deterministically, 3/3 runs):
TestPlanStart_ProviderRelayRestart, TestPlanStart_ProviderRelayAlreadyRunningPlansNothing,
and TestPlanStart_ProviderRelayRestartRespectsSkipProviders (the
Copilot-caught case).
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel) -- TestProviderPublishEndpoint
confirmed passing. Two rounds of local Copilot CLI review: the first
caught the SkipProviders gap, the second confirmed the ordering and
event-grouping of the new node are correct and no other call site
shares the same risk.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
`compose start` (and the two callers sharing its unexported start():
`compose run`'s dependency startup, and interactive up's own start
phase) switches from the imperative InDependencyOrder/startService/
startServiceContainer traversal to the plan-based reconciler/executor,
via the ScopeStart scope Lot 1 already built and golden-tested but
left with no real consumer (#14081, lot 2). ScopeStart's planStartPhase
works straight off the observed containers -- it never converges
anything, matching start()'s own historical contract of starting what
exists, never creating (see TestPlanStart_StartOnlyScope).
start() now mirrors upDetached's own established shape: collectObservedState
-> reconcile(ScopeStart) -> emitRunningEvents -> newPlanExecutor.run, then
the same post-plan --wait verification pattern (origCtx snapshot
distinguishing the wait's own fresh deadline from an inherited one,
exec.waitTimeout threaded unconditionally to match the old engine's
per-service-wait-regardless-of---wait semantics). The label-reconstructed
project path (no compose file, projectFromName) is unchanged -- ScopeStart
only reads project.Services/DependsOn, which that path already populates
the same way.
Interactive up's own start phase (up.go, listener non-nil for pre_start/
post_start hook log streaming) now runs on the plan engine too, as an
unavoidable consequence of start() being shared code -- epic #14081
originally scoped that switchover to a later, separate item (interactive
up), but there is no way to migrate `compose start` without also
affecting every other caller of the same function. This does NOT fix
interactive up's own two-snapshot problem (create() and start() still
each collect their own observed state) -- that remains a distinct,
not-yet-addressed follow-up; this PR only changes which engine runs the
second pass, not when it runs.
startService/startServiceContainer and the InDependencyOrder-based start
traversal are deleted: this migration makes them unreachable from every
production call site (confirmed via grep and golangci-lint's unparam/
unused catching the now-always-nil/zero listener and timeout parameters),
and the epic's own Lot 3 ("remove the imperative start path") rationale
for keeping that deletion separate -- making an interactive-up revert
trivial -- no longer applies: interactive up never used this code to
begin with once start() itself switched engines, so there is nothing
left for Lot 3 to revert around. Their dedicated characterization tests
(TestStartService_*, TestStartServiceContainer_*, the relay-hook-skip
test in service_containers_test.go) are removed with them; a coverage
spot-check (confirmed by local Copilot CLI review) found every behavior
they locked already has an equivalent at the executor/plan level
(executor_test.go's maxConcurrency tests, executor_start_test.go's event
and hook-ordering tests, reconcile_start_test.go's golden plan tests) --
nothing is left unverified. The now-fully-unused isNotRunning helper
is removed too.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary (same
3 pre-existing, environment-specific failures as the stack's baseline:
TestWatch, TestIPC, TestComposeModel). Local Copilot CLI review found
no correctness bugs and confirmed the dead-code deletion is complete
and safe.
A genuine regression this migration exposed in provider-backed
services' start-phase planning (planProviderStart never had a path to
restart an existing, stopped relay container under pure ScopeStart) is
fixed in the next commit, not folded in here, since it is a bug in
already-merged Lot 1 vocabulary that this PR's e2e run happened to be
the first to actually exercise -- not something introduced by this
diff.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The service name is always registered as a DNS alias on every network a
service is attached to, so two projects sharing a network and running a
service with the same name resolve it to either container.
Honor the use_service_name_alias attribute of a service's network
entry: when false, the service name is not registered as an alias on
that network only. The container name alias, the user's aliases and the
service's other networks are untouched, and --use-aliases still decides
whether any alias is set. Unset keeps the current behavior.
The attribute comes from compose-go, through its ServiceNameAliasEnabled
helper. go.mod temporarily replaces compose-go with its main branch,
which has it: drop the replace once a version containing it is tagged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
The error message produced by containerd changed in [containerd@7f0c0b8];
"reading blob sha256:536319e163bc3cc681d5ce1a5c88aab6e4afb4b7315ce62acfca619ad36717ee: httpReadSeeker: read past expected size 35: failed precondition
[containerd@7f0c0b8]: 7f0c0b822e
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
declaredServices duplicated (*types.Project).AllServices(), already in
the pinned compose-go: Services plus DisabledServices in a fresh map.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The rebuild now reports "Failed to recreate and restart service(s) after
update" instead of "Application failed to start after update", which
TestRebuildOnDotEnvWithExternalNetwork still asserted was absent: the
string no longer existed anywhere, so the check could never fail and the
test no longer detected a failed rebuild.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The narrowed-selection test only checked the selected service itself, so
a plan silently dropping its dependent would have passed. Also match the
disabled service by its resource prefix rather than a substring, and
drop the drifted image setup that force recreate made irrelevant.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
watch's rebuild narrows the project to the rebuilt services and their
dependents for the create phase too. Nothing locked that contract: a
future change of the selection or of the observed state could silently
bring back the convergence of the whole project, or report the services
left out as orphans.
Two tests: the plan of a narrowed project never touches a drifted
service outside the selection, nor removes it as an orphan under
--remove-orphans; and the observation of a narrowed project keeps the
containers of the disabled services out of the orphans, so the "Found
orphan containers" warning does not fire on every rebuild.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A project narrowed with WithSelectedServices (watch's rebuild, which
this PR moves to a narrowed Create phase, and `up`/`scale --no-deps`)
keeps every network and volume of the model but moves the services left
out to DisabledServices. The reconciler looked for the consumers of a
resource in project.Services only, so a drifted network or volume used
by a disabled service looked unused: the plan removed it without
detaching that service's container, and the removal failed with
"in use". For a volume, the user was first asked to confirm a
data-destroying recreation that could not succeed.
A same-name drifted network or volume that a disabled service still
holds through a live container is now kept as is, with a warning, and no
prompt is shown. A disabled service without a live container does not
block anything, and a rename is unaffected (it never removes the live
resource).
Also correct the comment in rebuild, which claimed narrowing left the
convergence of networks and volumes unaffected.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Same create() + start() two-snapshot pattern upDetached already
replaced for `up -d`, now shared via a new createAndStart helper for
the other two callers that had it:
- Scale: exact equivalent, same project object was already passed to
both calls.
- watch's rebuild(): the create() call used the full project (scoped
only via CreateOptions.Services/Recreate policy), but start() used a
project narrowed to services + their dependents
(WithSelectedServices). The narrowed project is now computed once
and used for both phases instead.
This does shrink the create phase's own scope a little, deliberately:
WithSelectedServices only trims the project's Services map
(unrelated services move to DisabledServices, not vanish -- orphan
detection still recognizes their containers via
DisabledServiceNames(), so nothing gets misflagged) -- Networks,
Volumes, Configs, Secrets are untouched, so external-resource
resolution is unaffected. What does narrow: config-drift checking
(RecreateDependencies policy) and the image pull/validate pre-pass no
longer consider services outside this rebuild's blast radius. For a
file-watch-triggered rebuild of specific services this is arguably
more correct than today (a watch rebuild of one service shouldn't be
re-validating or drift-checking the rest of the project on every
keystroke) -- not a literal byte-for-byte port, a deliberate,
narrower scope.
The old code also returned a create() failure immediately but logged
and swallowed a start() failure; a single combined call can't keep
that distinction. Traced as inert either way: the sole caller,
rebuildScheduler.run(), discards rebuild()'s return value
unconditionally -- only the logged message differs, not control flow.
AttachTo: services, also dropped from the old StartOptions, was
already dead in this call (only read by start() when Project is nil,
which it never was here).
Verified with the full e2e suite (TestScale*, TestWatch* in
particular) against a locally built binary, plus the existing unit
suite and lint. No new failures; the only 3 remaining are the same
pre-existing, environment-specific ones from the previous PR in this
stack (macOS Docker Desktop restricted-host-mount errors, a
model-pull/registry issue).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The wiring of --wait-timeout into the plan's dependency waits was only
covered by executor tests that set the timeout by hand, and by an e2e
without depends_on. The scenario has a dependency that never gets
healthy and checks that up -d --wait-timeout fails with the dependency
timeout message (without the raw deadline error) and leaves the
dependent service created but not started.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A node that leaves its dependency wait because its phase was canceled
closes its done channel (deferred) without recording any failure. In
X -> A -> B with X failing, B could select done[A] before it saw the
cancellation, find no failure recorded for A, emit its "Starting" event
and enter executeNode on a canceled context, leaving the row never
finalized.
Record the cancellation as the node's failure at that exit, so a
dependent that wins the race is skipped like any other dependent of a
failed node. The regression test pins GOMAXPROCS to 1, which makes the
bad interleaving the usual one.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14300: if a sibling Create-phase failure ever
stopped canceling the other node, VolumeCreate's mock blocks on its ctx
while exec.run waits for it, so the test's own five-second guard could
never fire and the regression would hang until the suite-wide timeout.
Give run a deadline of its own and assert the cancellation is the
sibling's (context.Canceled), not that fallback.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14296: planServiceStart returned before any
dependency planning for every service whose scale is zero. The imperative
startService only treats deploy.replicas: 0 as an unconditional no-op: a
service with scale: 0 still has its depends_on conditions evaluated first,
so an unhealthy or missing required dependency fails `up`/`start`, and
whatever is ordered after the service stays ordered after those
prerequisites. It is simply never an error for it to have no container to
start, and under scope Start the replicas it already owns are started
(CreateStart removes them all in the create phase).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14296: a wait node's own timeout is a
depends_on condition that did not become satisfied during the plan, not
the final --wait readiness check, which runs after the plan and has its
own "application not healthy" translation. The imperative path reported
"timeout waiting for dependencies" for the former regardless of --wait,
but translateWaitTimeout rewrote it as "application not healthy" when
--wait was set. It now always reports the dependency timeout.
Also mark the test nodes that stand for the Start phase as PhaseStart
(the recreate-then-start plan and the wait whose timeout survives a
skipped dependent), so they exercise the Create->Start barrier the way
the reconciler's plans do instead of running everything in one phase.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
up translated any DeadlineExceeded surfacing from the plan into "timeout
waiting for dependencies" or "application not healthy" whenever
--wait-timeout was set, even one coming from an engine call, a hook or
the secret injection, none of which is a dependency-readiness failure.
A wait node whose own --wait-timeout window expired now returns
errWaitTimeout (still satisfying DeadlineExceeded), and up keys its
message on that. A deadline inherited from the caller is never marked.
The test that checks --wait-timeout does not leak into non-wait nodes now
inspects the context those nodes receive, instead of a mock that would
have succeeded either way.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The final readiness check opened its --wait-timeout window before
re-listing the project's containers, so a slow daemon listing could
exhaust the budget and report "application not healthy" before
waitDependencies had polled once. List first on the caller's context,
then derive the timed one, as the imperative path did.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Reverts 365f0026f, a behavior-parity fix for a Copilot finding that the
plan-based start engine's per-(dependency,condition) wait node could
start its exec.waitTimeout clock earlier than the old imperative
engine's single waitDependencies(X) call -- which only begins once
EVERY direct dependency of X has itself finished starting, so a slow
sibling with no wait node of its own could silently extend another
dependency's effective grace period.
On reflection that old-engine coupling is itself the bug: a service's
declared --wait-timeout for a given dependency should bound how long
THAT dependency takes to satisfy its condition, period -- not be
inflated by how long an unrelated sibling dependency happens to take
to start. Letting db's health-check timeout silently grow because
migrate's own dependency chain is slow means --wait-timeout no longer
means what it says. Restoring the narrower, per-dependency gating is a
deliberate, accepted behavior change from the old engine, not a parity
regression to chase.
This also removes the dependency-cycle risk the sibling-widening
introduced (an ordinary diamond dependency, or a service whose own
chain end collapses onto a shared wait node, could merge a node back
into its own transitive dependents -- see 365f0026f's description) and
the wouldCreateCycle machinery and five regression tests that existed
solely to guard against it, now dead weight with no reachable scenario
to protect against.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot flagged a behavior-parity regression on #14290's Create->Start
barrier: a service's OpWaitCondition node only depended on its own
dependency's chain end, so its exec.waitTimeout clock could start
ticking earlier than the old imperative engine's single
waitDependencies(X) call. That call only begins once EVERY direct
dependency of X has itself finished starting (confirmed by reading
InDependencyOrder's graphTraversal.run: a service's visitorFn only
runs once none of its dependency children are still ServiceStopped).
A slow sibling with no wait node of its own (service_started is a
plain edge, nothing to poll) could silently eat into a health check's
effective grace period compared to the old engine.
startPhaseDependencies now computes allTargets, the union of every
declared dependency's own chain-end for the requesting service, and
passes that into waitConditionNode instead of just the one dependency
being checked.
Because wait nodes are shared/deduplicated across consuming services
by (dep, condition) alone, this widening reopened a second, more
serious finding from the same review pass: two consumers of the same
shared node can have different sibling sets, and merging a sibling
that already (transitively) depends on the shared node -- an entirely
ordinary diamond dependency, or a service whose own chain end
collapses onto the wait node itself when it has nothing left to start
-- creates a dependency cycle. The executor has no cycle detection and
no timeout on the dependency-wait phase itself (exec.waitTimeout only
applies once a node starts executing), so this was a guaranteed
permanent hang, not a correctness nicety.
Fixed with wouldCreateCycle: a wait node's merge now skips any
candidate that already (transitively) depends on it. This is provably
safe, not a workaround -- if a candidate already depends on the wait
node, the wait node is necessarily already done by the time the
candidate's own chain completes, so there is nothing to gain from also
adding the reverse edge; the candidate's own direct dependency on the
shared node (or on an intermediate sibling) already encodes everything
the consumer's depends_on requires.
Five new regression tests in reconcile_start_test.go, all verified by
ablation (reverting the relevant guard reliably reproduces the
original bug, 3-5/5 runs depending on the test):
- TestPlanStart_WaitNodeGatedOnAllSiblingDependencies: the original
parity fix (db:service_healthy + migrate:service_started siblings).
- TestPlanStart_DiamondDependencyNeverCreatesWaitCycle and
TestPlanStart_ExceptionalChainCollapseNeverCreatesWaitCycle: the two
concrete cycle shapes found in review.
- TestPlanStart_TransitiveChainNeverCreatesWaitCycle: exercises
wouldCreateCycle's BFS two hops deep, not just a direct edge.
- assertNoCycle, a plan-wide helper used by the cycle tests above.
Verified: go build, go test ./pkg/compose/..., and golangci-lint all
clean; full e2e suite re-run clean against a locally built binary
(same 3 pre-existing, environment-specific failures as the stack's
baseline: TestWatch, TestIPC, TestComposeModel). Two rounds of local
Copilot CLI review: the first caught the cycle risk in the initial
allTargets widening, the second confirmed wouldCreateCycle's BFS is
correct and complete, the "skipping is safe" reasoning holds in every
case, and no other call site in reconcile.go has the same unguarded
sharing risk.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding (low) on #14290: with --wait-timeout supplied but
not --wait, a dependency timeout in upDetached returned the raw "context
deadline exceeded" instead of the actionable "timeout waiting for
dependencies" the legacy start()+waitDependencies path already reports
for the exact same failure. The --wait-specific "application not healthy
after %s" translation only ever covered the Wait=true case, leaving the
Wait=false, WaitTimeout>0 case (a legal CLI combination -- --wait-timeout
alone is a hang guard) with no translation at all.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review finding on #14290: waitDependency decided whether a
DeadlineExceeded on an optional dependency wait was safe to tolerate
using a bare boolean -- whether the caller had configured a timeout
for this wait at all (exec.waitTimeout > 0, or legacy waitDependencies'
timeout > 0). That can't distinguish THIS call's own timeout actually
firing from an earlier deadline inherited from the caller's own ctx
also firing: context.WithTimeout(parent, waitTimeout) still makes both
parent and child report DeadlineExceeded once the parent's earlier
deadline elapses, even though this wait's own timer never got there.
An unsatisfied optional wait could therefore swallow a real external
deadline (the process's own context, a test harness timeout) and
report success instead of propagating it.
Replaced the bool with an origCtx snapshot -- ctx as it stood right
before this call's own conditional context.WithTimeout wrap -- threaded
from executor.go's OpWaitCondition dispatch through execWaitCondition
down to waitDependency, and from waitDependencies' own wrap in
service_containers.go. waitDependency now checks origCtx.Err() == nil
at the moment ctx.Done() fires: only when the pre-wrap context hasn't
failed on its own is this wait's own timeout the one responsible.
TestExecutePlanOptionalWaitDoesNotSwallowInheritedDeadline pins the
negative case (inherited deadline must still propagate even though
exec.waitTimeout is set and the dependency is optional) and
TestExecutePlanOptionalWaitTimeoutIsToleratedWithoutInheritedDeadline
pins the positive one (this wait's own timeout, no inherited deadline,
still tolerated as before) -- both verified by ablation.
Verified: go build, go test ./pkg/compose/... (full suite, not just the
new tests) and golangci-lint clean. Local Copilot CLI review confirmed
the origCtx threading is correct and race-free across all three call
sites; its one actionable finding (missing positive-path coverage for
the plan-executor route specifically) is addressed by the second new
test above.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
run() used to dispatch every node concurrently, with Create-phase and
Start-phase cancellation scoped separately via phaseCancel so a
Start-phase failure couldn't cancel unrelated in-flight Create work.
That stopped short of what the interactive up convergence (lot 2's
next step) actually needs: its attach/printer session takes exclusive
hold of the terminal for the Create phase's own progress display, and
hands it over to continuous container log streaming only once the
Create phase is entirely done project-wide -- a Start-phase node's log
output interleaving with Create's progress bars has nowhere consistent
to go. Phase-scoped cancellation alone still let an unrelated
Start-phase node begin (and start logging) while a sibling elsewhere
in the plan was still being created, or had just failed.
run() now splits plan.Nodes into createNodes/startNodes upfront and
runs them as two fully sequential runPhase() calls: Start-phase nodes
are never even dispatched until every Create-phase node has succeeded,
and a Create-phase failure returns immediately without calling
runPhase on startNodes at all. This matches the old create()-then-
start() sequence's own fail-fast behavior, which the single-plan
migration had implicitly relaxed.
Added a fast-fail validation: a Create-phase node depending on a
Start-phase node would otherwise deadlock forever (the Start phase
isn't dispatched until the Create phase blocked on it returns, so
nothing would ever close that dependency's done channel). Nothing in
the reconciler should produce that edge, but failing fast with a clear
error beats a silent hang if it ever does.
TestExecutePlanCreatePhaseIsABarrierBeforeStartPhase and
TestExecutePlanCreatePhaseFailureNeverDispatchesStartPhase cover both
directions of the barrier; both verified by ablation (reverting to a
single concurrent runPhase call reliably fails each, 5/5 runs).
Verified: go build, go test ./pkg/compose/... (incl. -race), and
golangci-lint all clean; full e2e suite re-run clean against a locally
built binary (same 3 pre-existing, environment-specific failures as
the stack's baseline: TestWatch, TestIPC, TestComposeModel). Local
Copilot CLI review found no correctness/deadlock issues in the barrier
logic itself; its two actionable findings (missing phase-ordering
guard, missing never-dispatched regression test) are both addressed
here.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review: waitDependency's optional-dependency tolerance (the
previous commit) treated ANY DeadlineExceeded on ctx as this call's own
timeout -- including one inherited from an ancestor context the caller
never gave this specific call a deadline of its own for. A plan whose
only outstanding work was an optional, never-satisfied wait could see
the API caller's own external deadline (a wrapper tool, a test
harness) expire, get silently converted to a Skipped event, and report
success despite the caller's context having actually timed out.
waitDependency gains a hasOwnDeadline bool: true only when the
specific call is responsible for ctx's deadline (waitDependencies
passes timeout > 0; execWaitCondition passes exec.waitTimeout > 0,
exactly the condition under which executor.go's dispatch wrapped ctx
with that same field). Only a DeadlineExceeded under hasOwnDeadline is
tolerated for an optional dependency; one without it still propagates,
regardless of Required.
TestWaitDependencyDeadline gains "an inherited parent deadline is not
swallowed for an optional dependency" (timeout=0, an already-expired
parent ctx) alongside the existing "own timeout is skipped" case --
verified by ablation that dropping hasOwnDeadline from the condition
makes the new test fail exactly as described.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review, following up on 374079a32's own fix: serviceNodes keeps
only the LAST node processed during a service's create-phase
convergence -- correct for a single replica, but a scaled service can
have more than one exceptional-state (paused, dead) replica, each with
its own bare-restart OpStartContainer node. Using serviceNodes alone
in planServiceStart's len(replicas) == 0 branch meant a downstream
service_started consumer only ended up depending on the highest-
numbered replica's restart, not every one of them.
exceptionalStateRestartNodes collects every OpStartContainer node from
containerNodes[service] (which tracks all of them, unlike serviceNodes),
in deterministic replica-number order. planServiceStart now merges all
of them into startChainEnds alongside the dependency prerequisites.
TestPlanStart_ScaledExceptionalStateConsumerWaitsForEveryRestart pins
it: two exceptional-state replicas, asserting the consumer depends on
both restart nodes -- verified by ablation to depend on only the last
one without the fix.
Two more small, independent fixes from the same review round:
- execWaitCondition's doc comment (executor_ops.go) described the
pre-waitTimeout "no deadline, caller wraps ctx" design; updated to
describe planExecutor.waitTimeout's actual per-node window.
- waitDependency's new optional-dependency-timeout tolerance (the
previous commit) gains its own direct regression test
(TestWaitDependencyDeadline/expired deadline on an optional
dependency is skipped, not an error) -- the existing tests covered a
required timeout, user cancellation, and an optional dependency's
definitive (non-timeout) failure, but not this specific path.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Three related findings from Copilot and docker-agent on the
phase-scoped cancellation commit, all stemming from the same root
cause: switching run() from errgroup.WithContext's own atomic
cancel-with-error-record to an explicit, separate cancelPhase() call
reopened timing windows that errgroup.WithContext used to close for
free.
- Copilot (high): cancelPhase() unblocks every sibling in that phase
before this goroutine's own error reaches errgroup.Group's internal
errOnce -- a sibling woken by the cancellation can return a bare
context.Canceled of its own and win that race, hiding the actual
daemon error or DeadlineExceeded (and making upDetached's own
DeadlineExceeded translation unreliable with it).
- docker-agent (medium): the "is this my own genuine failure"
nodeCtx.Err() == nil check and the cancellation it gates aren't
atomic -- two genuinely concurrent same-phase failures can both pass
it before either's cancelPhase() takes effect, both emitting a group
Error event for the same resource.
- docker-agent (medium): the dependency-wait loop's early return on
ctx cancellation skipped close(rs.done[node.ID]), the only exit path
that did. Not reachable today (every dependent in the same phase
shares that phase's ctx, already canceled by the time it would
matter), but latent: nothing in runNode enforces that no future
cross-phase dependency edge ever relies on this channel closing.
All three close around one new type, phaseCancel, bundling a phase's
context/cancel pair with a sync.Once: the first node in a phase to
fail records its error and emits the group event inside that Once,
cancels the phase's context, and every later caller -- genuine
concurrent failure or cancellation casualty alike -- is a silent
no-op. run() returns a phase's recorded error over whatever
eg.Wait() itself picked, so errgroup's own race can no longer surface
a cancellation casualty's error in its place. The done-channel close
moved to a single `defer` at the top of runNode, covering every exit
path uniformly instead of three separate explicit calls.
TestExecutePlanConcurrentPhaseFailureKeepsOriginatingError exercises
the first scenario under real concurrency (one immediately-failing
node alongside many siblings a dependency-wait unblocks via its
cancellation) -- see its own comment for why this specific race can't
be forced to fail by ablation in an in-process test (the Go scheduler
never actually lets a sibling win it here), so the fix's correctness
rests on reading errgroup's own errOnce.Do source directly, confirmed
independently by two reviewers, not on this test alone.
Verified: go test -race -count=10 ./pkg/compose/... clean; full e2e
suite re-run clean against a locally built binary (same 3
pre-existing, environment-specific failures as the stack's baseline).
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review ("previously missed"): the same either/or bug just
fixed for planProviderStart (e206f0566) also affects the ordinary
(non-provider) service branch of planServiceStart. A service whose
only container is in an exceptional state (paused, dead) gets a bare
create-phase restart (OpStartContainer, "not running") and takes the
len(replicas) == 0 path in the start-phase visit. When that service
also declares its own depends_on, startChainEnds used only the
dependency's wait node as ends, dropping the service's own restart
node entirely -- a downstream service_started consumer could then
start before the paused/dead service's only container was actually
running again.
ends now always includes the service's own create-phase node
(serviceNodes[service.Name]) alongside its dependency prerequisites
when it has any, instead of either/or -- the exact same fix shape as
planProviderStart.
TestPlanStart_ExceptionalStateConsumerWaitsForRestart pins it,
mirroring TestPlanStart_ProviderConsumerWaitsForDeployment: a
paused service with both an upstream dependency and a downstream
consumer, asserting the consumer's start node depends on the restart
node directly -- verified by ablation to fail, showing the exact
missing edge, without the fix.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review (High) on #14290: run() dispatched every node -- Create
and Start phase alike -- under one errgroup.WithContext(ctx), so any
node failing canceled the shared context for every other goroutine
still in flight. A required OpWaitCondition node hitting
--wait-timeout is a Start-phase failure that can now happen while
unrelated Create-phase work (a sibling service's container or network
create) is still legitimately running -- and that work got aborted
too, with nothing to do with the dependency that timed out. The old
create()-then-start() sequence never had this failure mode: the whole
create phase always finished completely before any dependency wait
even began.
run() now derives two contexts instead of one shared errgroup-context:
createCtx at the root, startCtx as its child. A node picks nodeCtx
(and the matching cancelPhase) from its own node.Phase and calls
cancelPhase() only on its own genuine failure. A Start-phase failure
cancels only startCtx, leaving createCtx and everything still running
under it untouched. A Create-phase failure still cancels the whole
plan, Start phase included, via startCtx's derivation from createCtx
-- matching the old sequence, where start() was never even invoked
once create() failed.
The node-goroutine body moved out of run()'s loop into a runNode
method (golangci-lint's cognitive-complexity limit after this change)
taking a new runState struct bundling the per-run bookkeeping every
node shares.
Two smaller, unrelated fixes bundled in:
- upDetached (up.go) keeps an origCtx snapshot so its two
DeadlineExceeded-to-friendly-message checks can tell its own
WaitTimeout window expiring apart from an inherited, unrelated
parent deadline propagating through the derived context -- both
report DeadlineExceeded on Err(), only the former earned the
"application not healthy" framing.
- waitDependency (service_containers.go), shared by the old
waitDependencies and the new execWaitCondition -- the latter can now
produce a real deadline for the first time via exec.waitTimeout --
tolerates a timeout on an optional (BestEffort) dependency the same
way it already tolerates every other definitive failure: skipped
with a warning, not a hard failure.
Two regression tests pin both cancellation directions:
TestExecutePlanStartPhaseFailureDoesNotCancelCreatePhase (a
Start-phase wait timing out must not cancel a concurrent, unrelated
Create-phase node) and TestExecutePlanCreatePhaseFailureCancelsCreatePhase
(a Create-phase failure must still fail-fast its Create-phase
siblings, same as before this change). Both verified by ablation.
Full e2e suite re-run clean against a locally built binary -- same 3
pre-existing, environment-specific failures as the stack's baseline,
no new ones.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
docker-agent review (moderate confidence): exec.run propagates any
context.DeadlineExceeded up, including one originating from ctx itself
(an outer caller's own deadline -- a test harness, a wrapper tool, a
client-side timeout) rather than from an OpWaitCondition node's own
derived timeout. The WaitTimeout > 0 guard alone doesn't distinguish
the two: a user who did configure --wait-timeout could still see
"application not healthy after Xs" when what actually happened was
the caller's own unrelated deadline firing during plan execution.
ctx at this point is still the caller-supplied, undecorated context
(the per-wait timeout is derived fresh inside executeNode, never
reassigned here) -- so ctx.Err() == nil confirms the DeadlineExceeded
didn't come from ctx's own deadline, leaving only an OpWaitCondition
node's derived timeout as the source.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review ("previously missed", code unchanged since an earlier
round): the prior fix (applyRemainingDeadline) replaced a confusing
raw DeadlineExceeded with a clearer message, but kept the underlying
design flaw it was reacting to -- a single absolute deadline
(time.Now().Add(WaitTimeout)) shared across the whole plan and the
final verification. Slow, unrelated create/start/hook work (never
meant to be bounded by --wait-timeout -- that's the entire point of
exec.waitDeadline existing) could still exhaust that shared budget on
its own, failing an otherwise healthy deployment as "not healthy" with
no health wait ever actually timing out.
start()'s real behavior never had this failure mode: waitDependencies
opens a fresh context.WithTimeout(ctx, timeout) every time it's
called, so each dependency wait gets its own full WaitTimeout window,
independent of how long anything before it took.
planExecutor.waitDeadline (an absolute time.Time) becomes waitTimeout
(a time.Duration): executeNode's OpWaitCondition case now derives a
fresh context.WithTimeout per wait node instead of a shared
context.WithDeadline. upDetached mirrors this for the final
verification -- its own fresh WaitTimeout window, not whatever's left
of one. applyRemainingDeadline's already-expired-deadline problem
disappears by construction: there's no longer a precomputed deadline
to have already expired.
Also fixes two sibling DeadlineExceeded-translation sites (the
post-plan container listing and waitDependencies' own check) that were
missing the WaitTimeout > 0 guard: without a --wait-timeout set, ctx
can still carry an external deadline (an upstream command timeout, a
cancellation) unrelated to --wait-timeout, and naming "application not
healthy after 0s" would blame a duration nobody configured.
TestExecutePlanWaitDeadlineOnlyBoundsWaitNodes and
TestExecutePlanWaitConditionRespectsContextDeadline updated for the
field rename; both still pin their original scenarios under the new
mechanism. Verified against the full e2e suite (TestUpWaitTimeout,
TestUpWait, TestRestartWithDependencies, the provider suite) on a
locally built binary.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copilot review: planProviderStart used a provider's own depends_on
wait nodes as its startChainEnds whenever it had any, entirely
dropping its own OpRunProvider (deployment) node from that set. A
downstream consumer of the provider -- ordered after startChainEnds
-- would then only wait for the provider's dependency to become
healthy, not for the provider plugin/relay deployment itself to
finish. The old create-then-start two-phase barrier guaranteed that
ordering for free; ScopeCreateStart (inert until this PR actually
exercised it end to end) didn't reproduce it.
ends now always includes the provider's own node alongside its
dependency prerequisites when it has any, instead of either/or.
TestPlanStart_ProviderConsumerWaitsForDeployment pins it: a provider
with both an upstream dependency and a downstream consumer, asserting
the consumer's start node depends on the provider's OpRunProvider node
directly -- verified by ablation to fail (showing the exact missing
edge) without the fix.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>