mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-01 19:46:50 +00:00
799 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Danny Avila
|
0219513115
|
🛑 fix: Forward Run Abort Signal to Foreground Tool Calls (#15517)
* 🛑 fix: Forward Run Abort Signal to Foreground Tool Calls Stopping a generation left in-flight tool calls running on the far side. For MCP over streamable-http the external server kept burning CPU until the tool finished or timed out, with no `notifications/cancelled` ever sent. The MCP layer was never the problem: `MCPManager.callTool` already spreads `options.signal` into `client.request`, and the SDK's `Protocol.request` sends `notifications/cancelled` the moment that signal aborts. The signal simply never arrived. Agent tools do not execute inside LangGraph's ToolNode — `eventDrivenMode` dispatches `on_tool_execute` and `createToolExecuteHandler` invokes them. The agents SDK puts the run's abort signal on the batch request (`ToolExecuteBatchRequest.signal`, documented as one handlers should forward), but the handler destructured every field except `signal` and built its invoke config without it. `config.signal` was therefore `undefined` in every foreground tool, so `createMCPTool` derived no signal and the SDK registered no abort listener. Forward it. This restores cancellation for all signal-aware foreground tools, not just MCP. The detached background invoke keeps its own controller, since that work deliberately outlives the turn. An aborted call now rejects promptly and resolves an error result rather than hanging, so the abort is logged at debug instead of as a tool error — a user pressing Stop should not spray the error log. * 🔇 fix: Keep Cancelled Tool Calls on the Filtered Path and Out of Error Logs Addresses both findings from the Codex review of |
||
|
Danny Avila
|
1c7a172321
|
🕹️ feat: Configure BYOM Tool Permissions (#15504)
* feat: add configurable BYOM permissions * test: cover BYOM permission settings live * fix: harden BYOM permission settings * perf: reuse principals for environment discovery * chore: normalize BYOM settings style * fix: Align BYOM Discovery and Admission * fix: Harden BYOM Permission Boundaries * fix: Align BYOM Skill Approval Routing * fix: Preserve Lazy BYOM Approval Capabilities * perf: Parallelize Cached Environment Reads |
||
|
Danny Avila
|
6372cb08ee
|
🧶 fix: Untangle Parallel Subagent Descriptor Paths (#15492)
* fix: isolate parallel subagent descriptor walks * fix: preserve canonical subagent config identity * fix: separate subagent graph assembly from path validation |
||
|
Danny Avila
|
a3c2277c38
|
🚧 fix: Narrow Chat MCP Selections to What the Picker May Offer (#15478)
* 🚧 fix: Narrow Chat MCP Selections to What the Picker May Offer The chat MCP picker offers a server only when the operator has not hidden it (`chatMenu: false`) and the user reaches it directly rather than through an agent (`consumeOnly`). Nothing enforced either condition on the request, so a stale client, a replayed body, or a hand-written one could name any server and get its tools — `consumeOnly` in particular is a permission the registry derives per user, so ignoring it let a user reach in plain chat a server they had only been granted through an agent. Narrow the picker's selection in the two ephemeral loaders, against the same predicate the client's `selectableServers` uses. The resolver is injected, so `packages/api` stays free of the registry and the loaders keep their existing `deps` shape. Provenance is why this lives in the loaders rather than at the tool-resolution chokepoint: only here is a server name known to have come from the picker. Servers a model spec pins are the operator's own choice and are added after the narrowing, so pinning a chat-hidden server to a spec keeps working. `chatMenu` only ever comes from the config tier, so the overlay settles it with no lookup at all. Lookups for the rest run concurrently and are cached per server and user. A name the registry cannot resolve — request-tier, declared on the body and never registered — is kept, as is one whose lookup fails: this narrows an already-authenticated selection and is not the authorization boundary. * 📝 docs: Say What chatMenu Now Bounds The schema still described `chatMenu` as dropdown visibility alone, which is misleading now that it also bars a server from the chat selection a request carries. Spell out both halves, and what it deliberately does not restrict: agents, and servers a model spec pins. * 🛡️ fix: Close Four Gaps in the Chat MCP Narrowing Dedupe before resolving. The selection is request-supplied and the loaders used to build a `Set` immediately; mapping the raw array first let one body repeat a name enough times to launch a resolver — registry, cache, and ACL reads, with no single-flight — per occurrence. Resolve names through the tool-loading aliases. A configured `private server` is addressable as `private_server` downstream, so an exact-key lookup let the normalized spelling walk past a `chatMenu: false` check and still load tools. Names are returned as sent; only the decision is canonicalized. Let the request overlay settle `chatMenu`. The config tier is where the flag lives and is what the picker itself was built from, so an overlay that re-enables a server now outranks the durable config the registry returns, which was silently dropping a valid selection. `consumeOnly` still comes from the registry. Narrow the request body, not just the loader's copy. `applyContextToAgent` reads `req.body.ephemeralAgent.mcp` directly and prefers it over the agent's tools, so a hidden server's `serverInstructions` reached the model context even after its tools were dropped. * 🧭 fix: Ask the Registry Which MCP Servers the Picker May Offer Three rounds of review found the same mistake in three disguises: this narrowing re-derived the registry's tier precedence instead of asking for it. A config entry was treated as authoritative over a user-DB or process-backed server of the same name; the alias map was built from config names only, so a DB server named `private_server` was misrouted to a config `private server`; and the overlay short-circuit ran before any of that could be consulted. Resolve the accessible set through `resolveAllMcpConfigs` instead — the same call behind `GET /api/mcp/servers`, so what the request may use and what the picker was offered come from one place and cannot drift. Precedence, cross-tier name collisions, and the config overlay are the registry's job again, and the selectability test is a single flag check over what it returns. The result is smaller: no overlay branch, no per-server lookups, one catalog read regardless of how long the selection is, and aliases built over the whole accessible set so an exact name always outranks another server's normalized form. Publish the servers actually in play back onto the request body, spec pins included. `applyContextToAgent` prefers that list over the agent's tools, so narrowing to the picker selection alone had started dropping a spec-pinned server's `serverInstructions` while suppressing a hidden one's. |
||
|
Danny Avila
|
51edb2a660
|
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing * fix: enforce code worker principal boundaries * fix: separate pairing control planes from execution * fix: preserve principal worker routing * fix: pin artifact traffic to code workers * fix: authenticate code worker selection * test: mock worker-aware artifact headers * fix: harden self-service code worker routing * fix: fail closed for principal code pairing * style: format code execution headers * fix: close self-service pairing gaps * fix: satisfy code environment type checks |
||
Oliver Faust
|
5e49868f0a
|
🔑 fix: Decode Action OAuth Credentials After Decryption (#14698)
* 🔑 fix: Decode Action OAuth Credentials After Decryption Action client credentials are URL-encoded before encryption, but the OAuth token exchange decrypted them with `decryptV2` alone, sending values like `abc%40domain.com` to the provider and causing 401s for secrets containing `@`, `+`, `=`, `/` or `:`. Decoding falls back to the raw decrypted value so legacy secrets stored before |
||
|
Danny Avila
|
aa7539909b
|
🪶 feat: Enable Standalone Skill Authoring (#15480)
* feat: enable standalone skill authoring * test: align skill e2e flow with enable toggle * fix: preserve legacy skill state on version restore |
||
|
Danny Avila
|
dccd7ae496
|
🧶 fix: Preserve Multi-Agent Tool Call History (#15468)
* 🧭 fix: Preserve Multi-Agent Tool History * 🧰 fix: Preserve Lazy Subagent Tool History * 🧪 test: Enable Lazy Code History Fixture * 🧭 fix: Derive Effective Lazy Tool History * 🧪 fix: Tolerate Empty MCP Name Audits * 🧪 test: Mock Lazy MCP Name Audit * 🧭 fix: Expand Lazy Historical Tool Names * 🧱 fix: Derive Durable Wildcard History * 🧬 fix: Traverse Nested Wildcard History * 🔭 fix: Resolve Wildcard Server Boundaries * 🔐 fix: Persist MCP Call Identity * 🪪 fix: Stamp Resolved MCP Execution Identity * 🕸️ fix: Resolve Graph Member MCP Identity * 🧭 fix: Fall Back From Empty Graph Member ID * 🧭 fix: Preserve Aggregated MCP Identity * 🧭 fix: Harden MCP Identity Resolution * 🧭 test: Assert Concurrent Graph Limit Invariant * 🧭 fix: Centralize Lazy History Policy * 📦 chore: Bump `@librechat/agents` to v3.7.13 * 🧪 test: Align Content Filter Fixture With SDK History |
||
|
Danny Avila
|
c8939c9730
|
🪄 fix: Reuse Downloaded Artifacts for Image Reads (#15454) | ||
|
Danny Avila
|
8f6228095f
|
🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format (#15453)
* 🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format `read_file` on a `/mnt/data` image collapsed every failure — rate limit, missing path, truncated transfer, transport error — into the binary-file hint, whose image branch reads as a permanent capability limit ("is an image file and cannot be read as text"). The model stopped retrying, and the operator had no way to tell which failure had actually occurred. The failure that surfaced it: the reader windows the bytes through `/exec` stdout, one call per window, against the Code API's per-user execution limiter (20 requests / 30s in the reference deployment). At a fixed 32KB window a 122KB chart cost 4 calls and a 1MB image cost 33 — more than a whole limiter window, so anything over ~640KB could never complete. A turn that generates several charts and reads them back exhausts the budget partway through and reports the 429s as unreadable images. - `handlers.ts`: failed reads name the real cause and the affordance that matches it — rate-limited waits and retries, a missing path lists the directory, a truncated transfer retries. The "cannot be read as text" phrasing is now reserved for bytes that are not a supported image. An image past the inline budget says to downscale and read the smaller copy rather than pointing at `bash_tool`, which cannot show an image. - `process.js`: the window is derived from the runner's stdout budget (48,960B vs 32,768B, so a 122KB chart is 3 calls and 1MB is 22), halves and re-reads the same offset when a runner truncates anyway (remembered per Code API, with the env remedy logged), waits out a rate limit it can afford instead of discarding the bytes already pulled, and stops at a 24-call ceiling so no read stalls a turn across limiter windows. - `utils/code.ts`: shared `getCodeApiRetryAfterMs` (header, then the `retry_after_seconds` body); `skillFiles.ts` drops its private copy. * ♻️ refactor: Move the Sandbox Image Reader Into TypeScript Addresses the review round on #15453. - P1: windowing, window sizing, the in-sandbox reader script, response parsing and byte assembly move to `packages/api/src/files/code/image.ts`, and the Code API rate-limit policy to `packages/api/src/utils/code.ts`. `process.js` keeps only the `/exec` transport it owns. Both new modules are tested against their real implementations (19 + 13 cases), including the round-trip ceiling, the changed-file guard and the shell-injection guard on the reader script. - P2: a configured stdout budget below the old 8KB window floor produced a window the runner could never emit, and narrowing was pinned to the same floor, so no retry could succeed. The window now aligns to the base64 group size instead, and the round-trip ceiling bounds what a small budget costs. - P2: `.env.example` documented a 32,768-byte default that no longer holds and never mentioned the stdout budget. It now describes `LIBRECHAT_CODE_SANDBOX_OUTPUT_MAX_SIZE` as the setting to tune, with the chunk override as the escape hatch. - P3: `process.spec.js` no longer reimplements the retry parser in its module mock. Behaviour is tested where it lives; the mock is a declared passthrough and the spec covers what the transport still owns — session forwarding, the profile header, parser hand-off and error logging. - P3: `getCodeApiRetryAfterMs` reads the response body through `in` narrowing rather than an `as` assertion. - P3: the window-sizing JSDoc no longer claims a multiple of 3 in the one case that is not — an explicit `LIBRECHAT_CODE_IMAGE_CHUNK_BYTES`. * 🎯 fix: Size Sandbox Image Reads to What the Limiter Can Deliver Addresses review round 2 on #15453. - Narrowing now halves the caller's own failing window instead of the shared learned value, and the learned value only moves down to the narrowest size any read needed. Reads run concurrently (one per tool call), so two reads failing at the same size were compounding into a window two steps smaller than the runner would have accepted, and persisting it for every later read. - Narrowing continues until a runner accepts a window or a floor below which no runner could serve an image at all. Two halvings stopped at 12,240 bytes — still ~16KB of base64 — so an 8-10KB stdout cap that was not configured up front stayed unreachable. - The round-trip ceiling is now one limiter window (20), and a read that cannot finish within it says so from the FIRST response rather than spending the whole budget rediscovering it: the first window reveals both the file size and what a call really delivers. A near-limit image no longer drains the turn's executions before failing, which is what made the wait budget look too small. - The round-trip message names the size this deployment can actually inline, so the model has a downscale target instead of a guess. - `python3: not found` from the sandbox reader was matching the generic "not found" in `isSandboxMissingFileError` and being reported as a missing image path, sending the model to `ls /mnt/data` while hiding a runner dependency. Image reads now classify only path-specific ENOENT forms as a missing file. * 📐 fix: Size a Truncated Read's Retry From the Cap It Revealed Addresses review round 3 on #15453. A ~1KB sandbox stdout cap is a real runner default, and the 1,536-byte narrowing floor sat above it: discovery stopped one step short and declared every image unreadable, even though small ones fit comfortably in the 576-byte window the configured path already derives. Narrowing now runs until no smaller window exists. It also no longer has to grope its way down: a truncated response emitted exactly as much as the runner allows, so its length measures the cap this deployment never declared, and the retry is derived from it directly — one round-trip instead of the eight blind halvings it takes to fall from ~49KB to ~1KB. Blind halving remains the fallback and bounds the result either way, so every retry is strictly smaller than the window that just failed. |
||
|
Danny Avila
|
224512851a
|
🗂️ fix: Resolve Code Interpreter Destination Collisions (#15449)
* 🗂️ fix: Resolve Code Interpreter Destination Collisions
Codeapi mounts each `/exec` input at a destination derived from its `name`
and rejects the whole request when two entries land on one. LibreChat never
enforced that: `primeFiles` sent `name: file.filename` verbatim, and the
graph seed deduped by `(storage_session_id, id)` — identity, not destination.
Two conversation files sharing a filename therefore killed code execution for
the rest of the conversation. The rejected request never reaches the sandbox,
so no result comes back to collapse the pair, and every later turn re-primes
both and fails identically. Only code-generated outputs are covered by the
`(filename, conversationId, context, tenantId)` partial unique index; uploads
carry `context: message_attachment`, so one conversation can hold several
records sharing a filename — two uploads of `image.png`, or an upload whose
name a later execution wrote back in place.
`packages/api/src/files/code/destinations.ts` now owns the name → destination
mapping for every contributor to an `/exec` file list, mirroring codeapi's
conflict rule including directory-prefix conflicts:
- `primeFiles` claims per pushed file, disambiguating with a `-2`, `-3`
counter and echoing the resolved path into the tool context, so the model
is told where each file actually mounts. Claim order comes from immutable
`createdAt`, not `getFiles`'s `updatedAt` sort, which usage accounting and
re-upload both bump — order-dependent destinations would silently repoint
a path an earlier turn had already handed the model. Newest wins the bare
name, matching `ToolNode.updateCodeSession` and giving an in-place rewrite
precedence over the original it superseded.
- `seedCodeFilesIntoSessions` drops a later file whose destination is already
claimed. Renaming there would invent a path nothing told the model about,
and a conflict at that layer means one logical file arrived twice under
different storage pointers.
Fixes #15443
* 🧹 chore: Tolerate Holes in the Destination Priority Sort
`primeFiles` sorts the raw `getFiles` result before its own per-entry guard
runs, so the comparator has to survive a null row rather than throw ahead of
the loop that was already defending against one.
Also tightens the tool-context assertion in the upload-vs-output case, which
was passing on a trailing space rather than on the path.
* 🔁 fix: Address Codex Review on Destination Assignment
Four findings from the review of `
|
||
|
Danny Avila
|
49e1e32f3a
|
🧺 fix: Funnel MCP Catalog Work Through a Shared Gate (#15429)
* 🧯 fix: Bound MCP Catalog Work Across Requests * 🛑 fix: Reject Saturated MCP Discovery * fix: fairly bound MCP catalog work * test: type catalog snapshot mock arguments * test: assert snapshot deadline forwarding * fix: harden MCP catalog scheduling |
||
|
Danny Avila
|
b7ae67307e
|
🕰️ fix: Anchor Temporal Variables to Turn Start (#15419) | ||
|
Danny Avila
|
413906abc3
|
🖍️ fix: Honor Agent Image Detail Settings (#15407)
* 🖼️ fix: Apply an Agent's Configured Image Detail The Agent Builder exposes an Image Detail slider and saves it to `model_parameters.imageDetail`, but nothing ever read it back: `encodeAndFormat` resolves the level from `req.body.imageDetail`, which the agents route never populates, so every image was encoded as `detail: "auto"`. Providers that reject `auto` — MiniMax's OpenAI-compatible API among them — fail every image upload through an agent, and the setting that would fix it has no effect. The root cause is a classification error rather than a missing argument. `extractLibreChatParams` splits LibreChat's own settings out of an agent's model parameters, and its key set is derived from the `librechat` group in `parameterSettings`. `imageDetail` sat in `baseDefinitions` instead, so it was neither extracted for LibreChat's use nor stripped from the options handed to the provider. It is now grouped with `resendFiles` and the other LibreChat-level parameters, which gives it the same path those already take: extracted during agent initialization, carried on the initialized agent, passed to the client, and read by `encodeAndFormat`. `AgentClient.getSaveOptions` already expected `options.imageDetail` — nothing had ever populated it. The request body still wins where it is set, so the classic endpoints keep resolving detail from conversation settings exactly as before. Fixes #15359 * 🔤 fix: Type Image Detail as Its Enum, Not a Bare String `TConversation.imageDetail` is the `ImageDetail` enum, so the extractor's return type and the initialized agent's field are now declared as that enum rather than widened to `string`, and the tests pass enum members instead of raw literals. Caught by CI's type check, which I had run before adding the tests and not after — jest compiles the specs through babel, so the suite stayed green over a type error. |
||
|
Danny Avila
|
c7f0a8f247
|
📬 feat: Move Agent Queued Turns Server-Side (#15370)
* feat: move agent queued turns server-side * test: update queued turn composition coverage * fix: harden queued-turn lifecycle ownership * fix: centralize queued-turn lifecycle ownership * fix: close queued-turn lifecycle gaps * fix: unify queued turn lifecycle transitions * fix: fence queued-turn deletion admission * fix: preserve queued-turn lifecycle ownership * fix: close queued-turn reconciliation races * fix: reconcile quarantined queued-turn admissions * fix: annotate admission evidence default * fix: deepen queued turn lifecycle admission * fix: fence queued turn provider admission |
||
|
Danny Avila
|
fcae1025c0
|
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments * fix: address code environment CI coverage * fix: expose principal code environments in endpoint config * fix: reject missing code environment bodies * fix: harden code environment lifecycle * fix: revalidate principal code environments * fix: synchronize code environment authorization * fix: bind code environments to current principals * fix: fail closed on code ACL cache errors * fix: prevent code environment override shadowing * fix: preserve principal environment defaults * fix: suppress revoked code environment aliases * fix: fail closed on environment augmentation * fix: narrow code environment defaults * fix: isolate code environment fallback * style: sort code config imports |
||
|
Danny Avila
|
30124f21b2
|
🎻 refactor: Orchestrate Agent Runs Through a Request-Free Host (#15366)
* refactor: decouple agent initialization from HTTP * refactor: centralize remote agent execution lifecycle * fix: preserve pre-settlement error rendering * fix: preserve request-backed tool loading * style: sort agent execution imports * fix: adapt public agent tool loaders |
||
|
Danny Avila
|
c1cb591d49
|
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments * fix: include code environment in lazy agent type * fix: harden stateful environment routing * fix: complete code environment route isolation * fix: declare code environment map type * fix: preserve configured code execution routes * fix: harden stateful environment updates * test: assert route-scoped sandbox readiness * fix: harden stateful environment lifecycle * fix: isolate migrated code sessions * test: assert route-qualified code sessions |
||
|
Danny Avila
|
2ae6c8aea9
|
🛎️ feat: Wake Agents on Background Tool Completion (#15350)
* feat: wake agents for background tool completion * fix: isolate background completion contracts * fix: anchor background completion identity * fix: close background completion delivery gaps * fix: preserve manual background polling * fix: preserve legacy tool group identity * fix: close background wakeup identity gaps * test: type background wakeup enqueue mock * fix: preserve background completion identity * test: type activity phase fixture * test: mock phase media query * fix: arbitrate background result ownership * fix: retain tool step routing helper * test: expand phase groups before identity checks * fix: preserve background completion ownership * fix: bound durable background results * fix: type wakeup input budget export * chore: sort background handler imports * fix: persist timed-out background completions * fix: reconcile background completion ownership * fix: type background completion capabilities * fix: harden background completion terminalization * test: type missing completion evidence * fix: require evidence for completion retirement * chore: satisfy background completion static checks * test: cover automatic background completion wakeups * test: assert background wakeup agent identity * fix: wake capability-fenced trigger deliveries * test: keep capability shield fixture public * test: assert capability worker wakeup * fix: close background completion ownership gaps * test: satisfy completion lease static checks * fix: preserve artifact completion wakeups * fix: close background delivery recovery gaps * fix: simplify background receipt guidance * fix: recover dead background completion batches * fix: fence background completion recovery * fix: recheck background recovery ownership * fix: fence unpublished background continuations * test: await background message hydration |
||
Dustin Healy
|
8fcab7e44f
|
🔄 fix: Recover Missing MCP Marketplace Catalogs (#15323)
* fix: recover missing MCP marketplace catalogs * fix: make MCP catalog recovery passive * test: type MCP catalog recovery fixtures * fix: bound and back off passive MCP catalog recovery Passive recovery runs inline on `GET /api/mcp/tools` and its results are request-local by design, so every list request re-dialed the same cold servers with the default connection timeout. Three limits keep that cost proportional to what recovery can actually recover: - Cap the discovery timeout at 5s instead of inheriting the connection default (`initTimeout ?? 30s`); a server configured to connect faster keeps its own shorter limit. - Skip a server the config tier already marked `inspectionFailed`, leaving it to that tier's retry window rather than re-dialing it per request. - Skip a server whose declared `customUserVars` are unset, matching the gate `reinitMCPServer` applies for issue #10969 — connecting without them fails auth, so the attempt is spent for nothing. Servers that still fail discovery enter a one-minute per-process cooldown, which is what stops an unreachable server from being re-dialed by every subsequent list request. A server that recovers clears its own entry, and expired entries are swept at most once per window so the map stays bounded. Skipped servers render exactly as they did before recovery existed: present in the catalog with an empty tool list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B * fix: bound passive MCP recovery by deadline, key cooldowns by config Both follow-ups address the same mistake: recovery expressed its own request-level constraints in terms borrowed from other layers. `connectionTimeout` bounds one connection attempt, and `MCPConnectionFactory.discoverToolsInternal` spends it twice — once on the authenticated connection, then again in `attemptUnauthenticatedToolListing` — so capping it bounded no total this layer could reason about. Recovery now enforces its own wall-clock deadline per server with `withTimeout`, which holds however many attempts the factory makes; `connectionTimeout` is left to do only its own job, still honouring a shorter operator `initTimeout`. An attempt abandoned by the deadline disposes its own connection when it settles, and `Promise.race` keeps a handler on it, so a late rejection is not unhandled. A per-request budget now caps total recovery regardless of server count. A server is dialed only if the remaining budget can fund a full deadline; never dialing one is not evidence against it, so a skipped server records no cooldown and a later request reaches it once those ahead are cached or cooling down. Cooldown identity now includes the publication generation — the same effective-config identity the tool caches fence on — instead of just user and server name. Correcting a server's URL or transport keys a new entry, so the refetch the client issues on update is no longer skipped for up to a minute by the previous configuration's failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B * refactor: keep passive MCP recovery stateless and bounded by its own work Reverts the cooldown, request budget and deadline race added in |
||
|
Danny Avila
|
16c2bb4149
|
📬 feat: Establish Background Continuation Admission (#15348)
* feat: establish background continuation admission * fix: preserve routed subagent wakeup controls * fix: probe configured subagent task store |
||
|
Danny Avila
|
fa913148fb
|
🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334)
* 🧊 fix: Inline-refresh OpenID session tokens at MCP OBO call time Resolves the walk-away failure mode where MCP tool calls using OBO auth fail with "No valid OpenID access token is available for OBO exchange" after a user idles past their access-token lifetime. The strategy-time snapshot on `user.federatedTokens` could expire mid-stream before `resolveOboToken` ran, while `req.session.openidTokens` carried a still- valid (or refreshable) token that nothing read. - New OpenIDSessionRefresh service: per-user single-flighted closure that reads `req.session.openidTokens` at OBO time and inline-refreshes via `openid-client.refreshTokenGrant` when expired (30s skew), persisting via `req.session.save()`. No cookie writes (headers already flushed). - `resolveOboToken` gains a required UpstreamTokenProvider parameter (typed as `() => Promise<OIDCTokens | null>`, reusing the shared shape from @librechat/data-schemas). Compile-time guarantee that every call site is updated. - New `session_refresh_failed` OboTokenResolutionReason distinguishes "session expired and IdP rejected refresh" from "no upstream token ever existed." - `req` threaded through createMCPTool/createMCPTools/createToolInstance to construct the closure with captured request, plus fail-closed guards in MCPConnectionFactory.getOboTokens and MCPManager.callTool when the closure isn't plumbed. - Startup warning in MCPServersInitializer when OBO is configured but OPENID_REUSE_TOKENS is unset (the strategy populating user.federatedTokens is only registered under reuse, so OBO would fail every call without it). Tests: 16 new in OpenIDSessionRefresh.spec.js; obo.spec.ts extended for the new param + error reason; wiring smoke tests in MCPManager, MCPConnectionFactory, MCPServersInitializer, and MCP.spec.js. * 🛡️ fix: Harden OBO inline-refresh against token type and session edge cases - Token-preference asymmetry: live-token reuse and expires_at derivation now strictly gate on the access_token, not the id_token. Added a required `tokenPreference` parameter on isLiveSessionTokenStillValid, buildOIDCTokensFromSession, and createOpenIDSessionTokenProvider so every call site is explicit. Dropped the bogus id_token-exp fallback in performIdpRefresh — id_token TTL is governed by IdP session policy and would mark a short-lived access_token reusable past its real lifetime. - Missing req in /reinitialize route: the manual reconnect endpoint now forwards req into reinitMCPServer, so OBO servers can build a session-aware upstream-token closure instead of failing with missing_upstream_token. - Single-flight key collisions: composed key as tenantId:openidIssuer:openidId:sessionId via getSingleFlightKey. Concurrent calls in the same session still coalesce; separate sessions never share an in-flight refresh, preventing refresh-token rotation from breaking sibling sessions and preventing cross-tenant token crossover when distinct users share an IdP sub. - Opaque access token reuse): persist accessTokenExpiresAt (unix seconds, from tokenset.expires_in) on each refresh AND on initial login / SPA refresh in setOpenIDAuthTokens. New getAccessTokenExp helper falls back to it when the access token isn't a JWT, avoiding redundant inline refreshes for Microsoft Graph and Auth0 default audiences. - Log hygiene: the single-flight key (containing sessionId, openidId, openidIssuer, tenantId) is now SHA-256-hashed in the "Joining in-flight refresh" debug log. Preserves cross-line correlation via a 12-char prefix without leaking credential or PII material. Documented req.session.openidTokens shape contract via JSDoc typedef so the new accessTokenExpiresAt field has a discoverable home alongside the existing accessToken/idToken/refreshToken/expiresAt/lastRefreshedAt. Tests: OpenIDSessionRefresh.spec.js up to 30 passing (added coverage for opaque-token reuse, JWT-access-token-exp fallback, no-id_token-fallback regression, cross-session no-coalesce, persistence on refresh, and a guard against stale accessTokenExpiresAt carryover). AuthService.spec.js adds two cases covering accessTokenExpiresAt persistence on login. mcp.spec.js (route) gains a regression test asserting req flows into reinitMCPServer. * 🔍 fix: Detect OBO-only MCP admin config overrides Admin Config overlays for YAML-defined MCP servers compare only ADMIN_CONFIGURABLE_FIELDS to decide whether to lazy-init a config-tier override. The OBO config field was added after that fingerprint list, so an override that only added or changed `obo` was treated as unchanged YAML and skipped. Include `obo` in the admin-configurable field list and add a regression test for an OBO-only override. * 🔊 fix: Mock MCP OAuth timeout in SDK integration test MCPConnectionFactory.attemptToConnect reads mcpConfig.OAUTH_HANDLING_TIMEOUT when building the OAuth connection timeout. The SDK OAuth integration test mocked mcpConfig without that field, which made the timeout calculation produce NaN and caused the test to fail before the OAuth refresh/start path completed. Add OAUTH_HANDLING_TIMEOUT to the test mock. * ♻️ refactor: Pass OBO upstream-token closure into MCP instead of req Build the OpenID upstream-token provider at the request boundary and thread only the closure through MCP handling, so the MCP service layer no longer receives the raw Express request. The closure still reads/refreshes the live session at tool-call time, preserving the walk-away recovery. - Drop `req`/`capturedReq` from createMCPTools, createMCPTool, reconnectServer, createToolInstance, and reinitMCPServer; forward `upstreamTokenProvider` instead. Closure is constructed in loadTools, loadToolDefinitionsWrapper, and the reinitialize route, where req/res are in scope. - OBO: fall back to user.federatedTokens when the provider yields no live session, so OIDC remote-agent calls (verified bearer, no session) still work. - Inline refresh: mirror a rotated refresh token to the refreshToken cookie via a shared setRefreshTokenCookie helper, guarded by !res.headersSent (no-op on the streaming path; session copy stays authoritative). - Single-flight: hydrate a joining request's own session from the resolved tokens so a later OBO call doesn't replay a rotated-away refresh token. Addresses owner feedback and three review findings. * 🔒 fix: Recover OIDC refresh-token rotation after SSE OBO refresh When an inline OBO refresh rotates the OpenID refresh token after SSE headers have already been sent, the browser refreshToken cookie cannot be updated. Store a short-lived encrypted bridge from the stale cookie token to the rotated token so /api/auth/refresh can recover after express-session loss. Use the signed openid_user_id cookie to load user context for bridge validation, retry only on invalid_grant, and delete the bridge only after the bridged refresh succeeds. * 🔨 fix: hydrate joined OIDC refresh sessions with stable refresh tokens Update single-flight OIDC refresh joiners whenever refreshed access token state changes, even if the IdP keeps the refresh token unchanged. This prevents joined requests from retaining stale accessToken or accessTokenExpiresAt values and redundantly refreshing later in the same run. * 🌉 Persist OIDC refresh-token recovery bridges in MongoDB Store SSE OBO refresh-token recovery bridges in MongoDB instead of process-local memory so /api/auth/refresh can recover after worker restarts or cross-worker routing. Derive bridge expiry from REFRESH_TOKEN_EXPIRY so the recovery window matches the stale refreshToken cookie it repairs, and delete bridges after successful recovery. * 🤝 Coordinate OIDC inline refreshes across workers Add a short-lived Mongo-backed refresh-flight record so concurrent OBO refreshes for the same OpenID session do not redeem the same rotating refresh token on different workers. The winning worker performs the IdP refresh and stores an encrypted result; joiners wait for that result, hydrate their request session, and return without calling the IdP. * ⚓ Keep OpenID marker cookies aligned on inline refresh Refresh token_provider and openid_user_id with the same expiry as the rotated refreshToken cookie when an inline OBO refresh can still write headers. Share the marker-cookie writer with the normal OpenID auth refresh path so the fallback /api/auth/refresh branch continues to recognize valid OpenID refresh tokens after session expiry. * 🔑 fix: include refresh token in OIDC local refresh flight key Key the process-local OIDC refresh coalescing by the current session refresh token, matching the Mongo-backed flight key. This prevents a request with a newly rotated token from joining an older pending refresh and inheriting its failure/result. * 🌉 fix: store OIDC refresh bridge without cookie response Treat missing or non-cookie responses like headers-sent streaming responses during inline OIDC refresh. When the IdP rotates the refresh token and cookies cannot be written, persist a recovery bridge so a later /auth/refresh can recover after session expiry. * 🫙 fix: preserve stale OIDC cookie bridge key Track the refresh token last written to the browser cookie separately from the current session refresh token. When inline OIDC refreshes rotate tokens without a writable response, keep bridging from the browser-stale token directly to the latest session token. * 🙌 fix: keep OIDC bridge recovery success on cleanup failure Make refresh-token bridge cleanup best-effort after a bridged OIDC refresh succeeds. A transient delete failure now logs a warning but does not convert the already-refreshed session and cookies into a 403 response. * 📦 test: Exclude RefreshTokenBridge from tenant-isolation coverage Add RefreshTokenBridge to the tenant-isolation coverage allowlist because refresh bridge lookups run during unauthenticated OpenID refresh recovery. The controller first recovers user context from the signed OpenID marker cookie, then the bridge methods apply explicit user and tenant filters. Ambient tenant isolation would bind this recovery path to request-local tenant context that is not available at the point the stale cookie is being resolved * ⚡ Fix OpenID refresh flight retry and marker hydration Allow failed OpenID refresh flights to be reclaimed immediately instead of pinning transient errors. Preserve the browser refresh-token marker when joined refreshes hydrate session tokens from a shared flight result. Stabilize AuthService tests by isolating mocked module imports from prior suites. * 🛠️ fix: centralize OBO identity scoping Add shared auth identity helpers for app user ids, OpenID subjects, tenant ids, and normalized OpenID issuers. Thread a non-placeholder-visible OBO identity context from the real request user through MCP connection, tool-call, reinit, and refresh paths. Keep tenantId and openidIssuer out of createSafeUser so MCP user placeholders do not expose those fields. Scope OBO token cache and in-flight exchange keys by tenant, issuer, OpenID subject, scopes, and a SHA-256 hash of the upstream assertion. This prevents cross-tenant/cross-issuer collisions and avoids reusing tokens minted from stale rotated assertions. Use the shared identity helpers for OpenID refresh-flight keys and refresh-token bridge recovery records so related OBO refresh paths share the same identity normalization rules. The helper is intended for auth-boundary and credential-cache code, not as a blanket replacement for ordinary app user id ownership checks. * 🛠️ fix: preserve OIDC refresh-token sync on save failures Sync OpenID refresh-token cookie/bridge state before persisting the session so a transient session-store failure cannot lose an IdP-rotated refresh token. Also trigger sync when the session refresh token differs from the browser refresh-token marker, not only when the current grant rotates the token. This lets later writable refreshes repair stale browser cookies left behind by SSE refreshes. Route refresh bridge identity through the shared identity helper with the threaded OBO identity context, falling back to request/user context when needed. Add regression coverage for session-save failures, stale browser cookie repair, non-writable bridge storage, and shared-helper identity fallback. * 🛠️ fix: keep OIDC refresh bridge during recovery grace After successful bridged refresh recovery, re-store the stale-cookie bridge with a short grace TTL instead of deleting it immediately. This lets parallel /api/auth/refresh requests that already sent the stale browser cookie recover before they can observe the first response's Set-Cookie. Retarget the bridge to the refresh token returned by the bridged retry so B-to-C refresh-token rotation remains recoverable. The grace TTL is parsed with math() and defaults to 60s, which shrinks the replay window from the original REFRESH_TOKEN_EXPIRY bridge lifetime to the short recovery grace period. Remove the now-unused explicit bridge delete path from the service and data-schemas method surface. Add regression coverage for grace re-store, identity symmetry, retry failure behavior, and same-key upsert replacement. * 🛠️ fix: fail closed on OBO MCP user identity mismatch Add an OBO-specific guard before MCP tool execution that requires the effective invocation user and captured request user to both have ids and to match. This prevents OBO tool calls from falling back to a separate configurable.user_id identity after request-bound OBO context has already been captured. Keep the existing user id fallback behavior for non-OBO MCP calls. Tests cover mismatched OBO users, missing user ids, and the matching-user path ignoring a conflicting configurable.user_id. * 🛠️ fix: Guard OpenID bridge retry user identity Extract the shared OpenID refresh/user-resolution flow in AuthController so the normal refresh path and bridge-recovery retry use the same grant, claims, issuer, user lookup, and diagnostic logging code. Preserve the existing path-specific behavior: the normal path still owns migration updates and 401 login redirects, while the bridge retry still falls through to the existing 403 invalid-token response. Add a bridge-recovery guard that rejects retry results whose resolved user id differs from the signed openid_user_id cookie before issuing tokens or re-storing the grace bridge. Cover both the successful matching-user recovery and the mismatched-user rejection. * 🛠️ fix: type-safety polish on OBO data layer Replace refresh token bridge query/update Record<string, unknown> usage with typed Mongoose FilterQuery and UpdateQuery definitions. Harden OpenID marker cookie JWT expiry handling by converting refresh expiry milliseconds to integer seconds and rejecting invalid or non-positive durations. Add focused CSRF tests for fractional refresh expiry values and invalid expiry configuration. * 🛠️ fix: Bind OpenID session tokens to authenticated identity Stamp OpenID session token state with the LibreChat user id, OpenID subject, tenant id, and normalized issuer when tokens are stored. Fail closed before OBO inline token reuse/refresh when the session token identity does not match the current authenticated identity, preventing a stale or mixed Express session from supplying another user's upstream assertion. Also validate the normal /api/auth/refresh session-token reuse shortcut against the signed marker-cookie user before returning cached session tokens. Note: sessions created before this change carry no identity stamp and are treated as a mismatch. This is self-healing — the reuse path forces a full IdP refresh (which re-stamps the session) and the OBO path throws, surfacing as a one-time re-authentication for active OBO users at deploy time. The session re-stamps within one session lifetime (SESSION_EXPIRY, default 15 min). * 🛠️ fix: Recover OpenID refresh token drift Prefer the browser refresh-token cookie when it differs from the server-side OpenID session state, and force a real IdP refresh in that case instead of reusing stale session tokens. Store a short-lived refresh-token bridge when inline OBO refresh writes a rotated browser cookie but session persistence fails, so follow-up refreshes can still recover from the old token. Keep the bridge grace TTL centralized in RefreshTokenBridge so both recovery paths use the same env-backed value. Note: drift is measured against the last-synced browserRefreshToken marker, so the SSE path (intentionally stale cookie, authoritative session) does not false-positive. Sessions predating the marker have no browserRefreshToken; for those, drift falls back to comparing the cookie against the session refresh token and prefers the cookie on difference. This is the same self-healing pre-change-session window as the identity binding fix and re-syncs within one session lifetime. Tests cover cookie/session drift selection, reusable-session bypass on drift, bridge storage after session-save failure, and the shared bridge constant wiring. * 🛠️ fix: Harden OBO token caching and expiry handling Reject malformed OBO grant responses before writing them to the exchanged-token cache so a missing access_token cannot poison the cache. Store absolute expires_at values with cached OBO tokens and ignore legacy cache entries without usable expiry metadata. This keeps cached-token freshness based on the token’s real remaining lifetime instead of reusing the original relative expires_in on cache hits. Move OBO expiry normalization and skew helpers into packages/api and use them from both the JS exchange service and the TS MCP resolver. Apply a 30-second safety margin with a one- second floor for short-lived tokens, covered by direct helper tests and caller-level regression tests. Tests: - packages/api: npm run build - packages/api: npx jest src/mcp/oauth/expiry.spec.ts src/mcp/oauth/obo.spec.ts - api: npx jest server/services/OboTokenService.spec.js * 🛠️ fix: Harden OBO refresh-token bridge lookup and indexing Reuse getValidOpenIDReuseUserId for the bridge-recovery user lookup in refreshController instead of re-verifying openid_user_id inline. The shared helper enforces the JWT_REFRESH_SECRET presence check and a strict typeof payload.id === 'string' guard, rejecting tokens whose id claim is present but not a string (e.g. a numeric id) that the inline check accepted. Fail closed on issuer mismatch in getRefreshTokenBridge. Both the stored and the expected issuer are now normalized and compared for equality, so a bridge is recovered only when both sides agree (both absent, or both present and equal after normalization). Previously the check was skipped whenever the stored issuer was absent, allowing recovery across mismatched issuer context. Drop the unused {oldRefreshTokenHash, userId, tenantId, openidIssuer} index and the openidIssuer field on RefreshTokenBridgeQuery. The data-layer filter only queries the 3-field {oldRefreshTokenHash, userId, tenantId} index; the issuer is verified in application code, not the query. Hoist the repeated model accessor into getRefreshTokenBridgeModel. Note: issuer is now load-bearing for recovery. A bridge stored with an issuer recovers only when the lookup supplies a matching issuer; the recovery lookup reads user.openidIssuer via AUTH_REFRESH_USER_PROJECTION (an exclusion projection that retains the field). If a user's persisted openidIssuer is empty while the stored bridge has one, recovery fails closed (falls through to normal re-authentication) until the bridge TTLs out — no security regression. Tests cover invalid signed-cookie payloads bypassing the bridge, both asymmetric issuer-presence cases, issuer normalization before comparison, and an index-alignment assertion guarding against re-adding the dropped index. * 🛠️ fix: Degrade OBO discovery on token resolution failures Catch expected OboTokenResolutionError failures during MCP tool discovery and fall back to unauthenticated tool listing instead of aborting discovery. This keeps discovery aligned with the existing unauthenticated listing behavior while preserving unexpected errors as real failures. Also correct OBO tool-call freshness comment and tighten the OBO trust-check permissions type to the existing role permission shape. Tests: - npx jest src/mcp/__tests__/MCPConnectionFactory.test.ts --runInBand --coverage=false - npx jest src/mcp/oauth/obo.spec.ts --runInBand --coverage=false * 🛠️ fix: tighten OBO tool-call errors, bridge logging, and flight typing Move resolveToolCallUserId inside the tool-call try/catch so an OBO identity mismatch surfaces with serverName/toolName context and the standard tool-call-failed message instead of an opaque bare Error. Raise the refresh-token bridge lookup failure log from debug to warn so transient infrastructure failures on the unauthenticated /api/auth/refresh path are observable, and guard the message access against non-Error values. Replace the unknown+cast in isDuplicateKeyError with a hasErrorCode type predicate so the duplicate-key check reads error.code without an assertion. Preserve real math/isEnabled in the MCPConnectionFactory test mock (mock only processMCPEnv) so mcpConfig timeouts no longer resolve to NaN, fixing the TimeoutNaNWarning that masked slow OAuth retry behavior. * 🧪 fix: Restore the Flight Uniqueness Index and Buffer the Graph Cache TTL Two CI failures on the merge, both in suites this environment cannot run (their MongoDB binary download is blocked). `GraphApiService.spec.js` still asserted the unbuffered TTL. Graph tokens route through the same `getTokenCacheTtlMs` as the OBO and openidStrategy caches, so the entry now expires 30s before the credential does. `openidRefreshFlight.spec.ts` dropped the database between tests, which takes the indexes with it, and Mongoose builds them only once when the model is compiled. Whether the unique `key` index survived into a test was a race with that one-time build. Without it a second `create` inserts instead of raising a duplicate-key error, so every worker believes it won the flight — the mutual exclusion the file exists to prove. Indexes are now rebuilt after each drop, which also makes the reclaim and complete cases reach those paths for the right reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F * fix: address OBO review findings * 🔐 fix: Install Bridge Indexes and Carry OBO Through Assistant Recovery Two findings from the Codex pass on |
||
|
Danny Avila
|
77c2a51cf3
|
🪧 fix: Advertise Detached Event Actor Support from the Generation Store (#15322)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
* fix: activate detached event actions automatically * fix: shield detached terminal generations * perf: share capability availability index * style: flatten capability status selection * fix: preserve mixed-version lifecycle shells * fix: complete detached action store adapters * fix: close mixed-version capability races * fix: honor legacy capability success |
||
|
Danny Avila
|
487984193a
|
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable * fix: align detached actor checks with merged base * fix: close detached actor lifecycle gaps * fix: break detached action outcome cycle * fix: deepen detached action ownership * fix: fence detached launch handoffs * fix: preserve detached rollout fencing * test: type detached recovery failure * fix: close detached ownership handoffs * fix: stage detached action activation * fix: persist detached terminal retries * fix: fence detached durability ownership * fix: fence trigger lane publication * fix: fence detached transition generations * fix: retire failed detached predecessors * fix: close detached rollout consumers * test: type detached retry turns * fix: close detached ownership gaps * fix: fence detached action resumes * fix: fence prior-head event actor resumes * fix: preserve legacy event actor resumes * fix: align detached resume boundaries |
||
|
Danny Avila
|
01f5391ee3
|
🕰️ fix: Guard expires_in So a Token Response Cannot Outlive Its Credential (#15321)
* 🕰️ fix: Guard `expires_in` So a Token Response Cannot Outlive Its Credential RFC 6749 §5.1 makes `expires_in` only RECOMMENDED, so a token response may legally omit it. Four sites derived a lifetime from the raw field, where `undefined * 1000` is `NaN`. `NaN` is not a short TTL, it is no TTL. `@keyv/redis` writes the key without `PX` because `NaN` is falsy, so the entry is stored in Redis with no expiration at all; the in-memory backend embeds `expires: NaN` and every check compares with `>`, always false against `NaN`. The namespace default does not stand in either, since Keyv applies it with `??=` and `NaN` is neither `null` nor `undefined`. The exchanged access token was therefore cached permanently at `openidStrategy.js` and `GraphApiService.js`, and once it genuinely expired the poisoned entry kept being served with no path to eviction. The same omission is sharper in `ActionService.js`, where `new Date(NaN).toISOString()` throws `RangeError: Invalid time value`. Both call sites are inside a `try`, so the failure surfaces as a generic "Failed to authenticate OAuth tool" that names nothing, and the refresh site falls through to `requestLogin()` on every attempt, looping with no exit. The rule had six hand-written homes and three of them were wrong, so it now has one. A new `packages/api/src/oauth/expiry.ts` normalizes `expires_in` to a positive finite number of seconds or nothing, and exposes the two shapes callers actually need: a cache TTL that falls back rather than returning `NaN`, and an absolute expiry that is absent rather than Invalid. The four unguarded sites adopt it, and the two ad-hoc guards in `openidStrategy.js` and `OboTokenService.js` are consolidated onto it. `createHandleOAuthToken` is folded in as well. Its guard already handled `null` and unparseable strings but admitted `NaN`, since `typeof NaN === 'number'` satisfied its first branch. The `mcp/oauth` sites are deliberately left alone: `tokens.ts` guards on truthiness and carries richer logic that reads a JWT access token's own expiry when the response omits one, and the file is being reworked in #13901. Closes #15318 Closes #15319 * 🕰️ fix: Address `expires_in` Guard Review Round 1 Preserve an explicitly elapsed lifetime instead of collapsing it into "unknown". `expires_in: 0` is the provider stating the credential is already dead, which is information; treating it as absent handed it the one-hour fallback in `createHandleOAuthToken` and dropped the expiry entirely in `ActionService`, so a credential declared expired could be used and retained for up to an hour. Both sites preserved that value before this branch, so the collapse was a regression introduced here. `normalizeExpiresIn` now returns any finite number, positive or not, and reports `undefined` only for a lifetime that is genuinely unusable. `getTokenExpiresAt` therefore yields a past timestamp for an elapsed lifetime, so callers refresh rather than guess. Cache TTLs cannot pass such a value through raw: Keyv reads a TTL of exactly `0` as "no expiry", turning a dead credential into the immortal entry this module exists to prevent. `getTokenCacheTtlMs` floors an elapsed lifetime at one millisecond, which expires immediately without ever writing an entry that outlives its credential. Parse numeric strings with `Number` rather than `parseInt`, which truncates a complete value such as `"3.6e3"` to `3` and would expire an hour-long credential after three seconds, re-exchanging against the identity provider on every request. An empty or blank string is rejected rather than read as zero, since `Number('')` is `0`. * 🕰️ fix: Bound `expires_in` to Lifetimes a Date Can Represent Parsing the complete numeric string last round made an overflow reachable that `parseInt` had been masking. `parseInt('1e13', 10)` was `1`; `Number('1e13')` is `1e13`, and `1e13` seconds is 1e16 ms, past the ECMAScript time value range of ±8.64e15. Every derived timestamp was therefore an Invalid Date whose `toISOString()` throws `RangeError: Invalid time value` — the exact failure this branch exists to remove, reintroduced by its own fix. The token model derives the same way at `packages/data-schemas/src/methods/token.ts:19`, so storage and authentication would fail with it. A lifetime is now reported as unusable unless it can still produce a valid `Date`. The bound is the time value range halved, leaving room for the `Date.now()` every derived timestamp adds. At roughly 137,000 years it rejects nothing a provider could mean: a one-year refresh token and even a hundred-year lifetime still pass through untouched, while `1e13`, `Number.MAX_SAFE_INTEGER` and `1e300` take the caller's fallback instead of poisoning a timestamp. The invariant tests now carry the overflow shapes rather than a fixed list of small ones, since a guard that only sees the inputs its author imagined is how the previous round's regression got in. |
||
|
Danny Avila
|
6fba51a35a
|
feat: Fingerprint Agent Context for Zero-Read Warm Continuation (#15299)
* feat: add context-fingerprinted warm continuation * fix: preserve manual skills across warm turns * perf: keep ordinary agent initialization neutral * fix: fingerprint complete agent topology * fix: close warm context replay gaps * fix: preserve lazy subagent memory scope * perf: fingerprint only model-bound memory * fix: project lazy memory capability * style: sort warm continuation imports * fix: make warm continuation reconstructable * fix: preserve lazy skill isolation * fix: retain scoped skill semantics |
||
|
Danny Avila
|
a5fd00cd7a
|
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning * fix: preserve automatic subagent activity handles * fix: gate subagent wakeups for ephemeral parents * fix: scope subagent wakeup guidance by agent |
||
|
Danny Avila
|
03fa98eb37
|
🕸️ feat: Complete Keenable Web Search Provider (#15288)
* feat: wire Keenable web-search provider into config, schema, and UI Keenable landed as a search provider in @librechat/agents (#285, shipped in 3.2.58+), but LibreChat did not yet expose it. This adds the config/schema/UI glue so it can be selected, mirroring the existing Tavily provider. - data-provider: add `keenable` to SearchProvider type + SearchProviders enum, keenableApiKey/keenableApiUrl schema fields, and a keenableSearchOptions block (maxResults, site, attributionTitle, timeout). - data-schemas: register keenable in webSearchAuth.providers and default the key/URL placeholders in loadWebSearchConfig. - api/web: pass keenableSearchOptions through to the provider and handle Keenable's keyless model. Unlike other providers it authenticates with no key (the public endpoint), picking up an optional key/URL when set; the URL override is SSRF-preflighted like other user-provided URLs. - client: add Keenable to the provider dropdown with an optional API-key input. - docs: document KEENABLE_API_KEY/KEENABLE_API_URL in .env.example and a webSearch example in librechat.example.yaml. - tests: keyless + keyed auth resolution, config defaults, and schema parsing. * fix: ESLint no-unused-vars and clarify Keenable yaml example - Remove the now-unused RerankerTypes import in data-schemas web.ts (the lint job runs with --max-warnings 0 on changed files, so this latent warning failed CI once the file was touched). - Note in the librechat.example.yaml Keenable stanza that a scraper (and reranker) is still required for web search to load, and include a Firecrawl scraper in the example. * chore: fix import order drift (sort-imports) * feat: add Keenable as a keyless scraper and select it without a pinned provider The Keenable scraper landed in @librechat/agents#337, so wire the scraper category the same way the search provider already is: `scraperProvider: keenable` reads pages through Keenable's public fetch endpoint with no key (a key only lifts rate limits, and the endpoint is overridden with KEENABLE_FETCH_URL). Paired with `rerankerType: none` this makes a fully keyless web-search stack possible for the first time. Also closes the Codex finding on this PR: because none of Keenable's auth fields are required, the generic auth loop skips it whenever it isn't pinned, so a key submitted through the API-key dialog (which cannot pin a provider) left the providers category unauthenticated. Keenable is now selected in that case, gated on one of its values actually being present so installs that configured nothing keep their current behavior. The scraper gets the same fallback, additionally gated on Keenable being the resolved search provider, so it never silently scrapes for another provider. * fix: select the Keenable scraper from a supplied key, not only for Keenable search The API-key dialog submits credentials and cannot pin a provider, so choosing Keenable as the scraper while search stays on Serper/SearXNG/Tavily had no effect: the unpinned-scraper fallback required Keenable to also be the resolved search provider. A supplied Keenable value now triggers it as well, which is the only signal the dialog can send. The fallback still runs only when no keyed scraper authenticated, and with neither trigger the category stays unauthenticated, so a deployment that never configured Keenable is unaffected. Note the fully keyless choice still cannot be expressed through the dialog: Keenable's key is optional, so picking it with no key submits nothing at all. librechat.example.yaml now documents pinning scraperProvider: keenable for that case. * style: Sort Keenable imports * fix: Harden Keenable auth resolution * fix: Preserve Keenable selection intent * fix: Fail closed on invalid web search auth * fix: close keenable auth gaps * style: sort web auth imports * fix: preserve web search selection integrity * fix: isolate web search auth ordering * fix: silence expected credential misses * chore: bump agents sdk * fix: preserve web search preference ownership * fix: forward cleared Keenable endpoint * style: sort web search hook imports * fix: require intent for credential clears --------- Co-authored-by: Ilya Bogin <ilya.bogin@keenable.ai> |
||
|
Danny Avila
|
de59da9636
|
🎟️ refactor: Require Credentials for Local Image Access by Default (#15252)
* 🔐 fix: Protect Local Image Access by Default * 🔐 fix: Scope Image Authorization to Active Sessions * 🧹 style: Format Image Authorization Checks * 🛡️ fix: Harden Image Avatar Authorization * 🧭 style: Sort Image Authorization Imports * 🔐 fix: Close Image Authorization Review Gaps * 🧭 fix: Normalize Stored Avatar Base Paths * 🏢 fix: Resolve Tenant Assistant Image Policy * 🛂 fix: Enforce Effective Image Access Policy * 🧹 style: Flatten Assistant Config Selection * 🧷 fix: Preserve Image Access Compatibility * 🪪 fix: Make Image Sessions Revocable * 🏗️ fix: Move Image Session Policy Into API |
||
|
Danny Avila
|
8b1fcc0fc2
|
🌐 fix: Expose Gemini Models to Vertex AI Agents (#15234)
* 🌐 fix: Expose Gemini Models to Vertex AI Agents * ♻️ refactor: Resolve Shared Vertex Model Catalogs * fix: Preserve Exact Vertex Model Catalogs * style: Format Agent Model Selection * test: Preserve Native FS in Stable Diffusion Spec |
||
|
Danny Avila
|
68fc46a055
|
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks * fix: fence event actor checkpoint uncertainty * fix: satisfy event actor type contracts * fix: make actor reconciliation recoverable * fix: preserve event actor lifecycle transitions * fix: fence event actor lifecycle outcomes * fix: enforce event actor lifecycle ownership * fix: retain event actor settlement proof * 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal Repair and compensation deleted the reconciliation row they resolved, which was the only durable proof that the invocation had already applied an external action. A delayed duplicate owner could then reacquire the same invocation id and repeat that action. Both resolutions now retire their receipt to `settled` and record how it settled, so the same-id tombstone survives; `history_repaired` and `action_compensated` still force a cold rebuild. Compensation undoes the effect without re-authorizing the delivery, so a legitimate retry must arrive under a new invocation id. A retried repair converges on its own receipt. Bound the journal so a long-lived actor cannot grow its conversation document without limit: a new fence is admitted only when no active lifecycle row exists, so a capped push can evict nothing but the oldest settled receipts. Stop shipping the unbounded source payload on every bound-child continuation. It rode the delivery body regardless of the feature flag while the sibling `fire` body deliberately sends event identity alone, so a large webhook payload could push a previously working delivery past the chat route's body limit. The actor binds an invocation from identity and never builds the prompt from it. Blank the positional token map on warm continuations. It is derived from the full DB history, while a warm run executes on checkpoint-restored state, so its indices address different messages and the pruner never recounts them — misattributing cached counts to the wrong messages in both directions. Keep the replaced-claim exit on its cleanup path when preserving reconciliation fails: the committing CAS already left a blocking row, so the failed status upgrade costs provenance, not safety. * 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry Give the warm-continuation client test a populated token map and a real cross-run summary so its assertions bite: the positional map must arrive blank (checkpoint-restored state no longer matches DB-derived indices, and the pruner never recounts a populated entry) while `initialSummary` must pass through unchanged — it rides the system tail and summarizes pre-boundary turns that were excluded from the very history the committed checkpoint was built from, so blanking it would silently drop context no warm run can recover. * ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal A compensated receipt still tombstones its invocation id, but its external effect was explicitly undone — the terminal handler nonetheless replayed every settled lifecycle's stored action as authoritative and settled the public outcome as applied, telling an action-aware source the operation stands and suppressing the new-invocation retry compensation requires. The handler now settles a compensated invocation as failed with an explicit compensation error, overriding even fresh applied run evidence from a replayed generation; verified and repaired receipts continue to replay applied. Receipt eviction is now primarily age-based: a stale same-id owner is bounded by time, not by how many newer invocations settle, so the previous count-only slice let a high-rate actor evict a tombstone while its delayed duplicate owner could still wake and repeat the action. Admission prunes only settled receipts older than a retention window that dwarfs every generation, job, and delivery-retry lifetime, and the count cap is demoted to a raised document-size backstop. * 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts The terminal handler read its lifecycle snapshot, verified history, and then settled the public outcome — so a compensation resolving the same receipt during that window lost: the handler settled applied from its stale snapshot and no retry could ever change the replay-identity-locked outcome. The receipt's status CAS is now the serialization point: verification resolves the receipt BEFORE settling, whichever transition wins determines the public outcome, and a crash between resolve and settle converges through the retained receipt's replay. The verified-replay probe requires the receipt's own resolution, so a compensated receipt can never satisfy a verification retry. This inverts the settle-before-receipt ordering deliberately: that ordering guarded proof that resolution used to delete, and the receipt now retains its full action proof through resolution. The document-size cap is no longer an eviction quota. A receipt inside its retention window is never discarded: when the journal holds a full cap of unexpired receipts, new invocations are refused fail-closed until receipts age out, making duplicate protection and document integrity simultaneous invariants instead of a rate-dependent trade. * 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path Skill primes are spliced into the message list directly ahead of the newest message, and a warm continuation forwards only that newest message — so a checkpoint-restored actor would keep serving the prime bodies baked in at its last cold start and never observe an edited or newly attached skill. Until the actor head carries a context fingerprint that forces a cold rebuild when the agent's skill context changes, agents with always-apply or manual skill primes stay on the legacy path, which re-primes fresh bodies every turn: correct on every event, just never warm. * 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes History-derived re-priming was a third path into the same staleness class: an actor that previously invoked a skill carries no request-time prime arrays, yet primeInvokedSkills re-resolves that skill's current body from history each turn and the warm slice drops the reconstruction — leaving the checkpoint's old body active after edits. The fork gate now keys on the priming hook itself (present exactly when the skills capability is enabled) alongside the request-time arrays, so every skill-body path routes to the legacy rebuild until #15235's context fingerprint restores warm continuation for skill-bearing actors. * 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage The executor read applied-action evidence from the run-step collection the instant sendMessage resolved, but that collection is populated asynchronously — an applied invocation could classify as actionless (runSteps still empty while the tool result already streamed), discarding its fork and stranding the actor cold while the terminal handler later settled the same delivery as applied from the persisted evidence. Authoritative proof is now recorded in graph context the moment the expected tool executes: the request-owned recorder observes the tool-end chain (which ToolNode dispatches synchronously with both input and output) and applies the same fences as run-step evidence — exact tool name with the MCP-suffixed form, the declared argument subset against the execution input, an error-free result, and the background non-execution receipt exclusion. readAppliedAction consults the receipt first; run-step inspection remains the fallback for paths that bypass the tool-end chain. Regression coverage reproduces the observed ordering: the real executor commits the head from the receipt while run steps are empty, warm-continues the next event, and never re-executes the action; recorder fences and the receipt-first controller wiring are covered separately. * 🎯 fix: Supply Execution Arguments to the Tool End Callback The live Vertex + MCP canary exposed a contract mismatch the synthetic fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end callback with output only, while the action recorder must verify the declared argument subset against the execution input. The receipt never qualified, every turn fell back to cold history rebuilds, and the tournament advanced with zero actor heads and zero retained checkpoints while looking successful. The execution handler owns both halves at the same moment, so the fix is at the source rather than a correlation store: ToolEndCallbackData gains the executed call's input and every callback site passes tc.args. A handler-level regression drives the real createToolExecuteHandler and asserts the callback receives both fields; recorder regressions pin the production shapes — an output-only tool end must starve an argument-fenced receipt rather than trust an unfenced match, and still qualifies a name-only expected action. * 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions The background-claim callback reports the ORIGINAL tool's name for artifact attribution on the poll turn that harvests a completed task. A name-only expected action could therefore be impersonated by work some earlier turn dispatched: the recorder would attribute that delivery to the current invocation and commit a head whose state never contained the invocation's own action. The run-step evidence path never had this hole — it sees the poll tool's name — so the recorder must match its provenance discipline. Delivery callbacks now carry an explicit backgroundDelivery marker set at the one site that rewrites the name, and the recorder ignores marked deliveries outright. Regressions pin both halves of the contract: the delivery callback must carry the marker with the poll call's arguments, and a marked delivery can never qualify even a name-only expected action. * 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions Three closeout-round findings, each converted into an invariant. Every legacy-path invalidation now advances a durable epoch — including for headless and already cold-marked actors, where the marker alone leaves no CAS-visible trace — and the actor-head CAS requires the epoch observed at preparation. A concurrently prepared fork whose history predates an intervening legacy turn can no longer commit past it; the commit reports an ordinary conflict and journals. Execution identity is now emitted before post-execution output policy: when a side-effecting tool succeeds but its returned content is withheld by the output filter, the callback delivers an outputFiltered receipt with blank content — the recorder accepts it as proof (rejecting model-detached calls it cannot distinguish through the blank shape), the artifact path never sees it, and an applied action is no longer reclassified actionless and re-executed on retry. Background-capable expected actions stay off the fork path: dispatch returns a launch handle every evidence fence correctly rejects, and the completion is provenance-marked as another turn's work, so a fork would settle actionless before the external effect lands with nothing to stop a retry from dispatching it again. The gate mirrors the MCP-suffix name matching of the evidence path. * 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance The epoch fenced only the legacy turn's start: a fork preparing after the begin invalidation but before the turn's message persistence observed the new epoch and cold marker, rebuilt from history that did not yet contain the turn, and committed cleanly because nothing advanced the epoch again — making the incomplete rebuild authoritative and clearing the marker. Every legacy event turn now seals its invalidation at terminal persistence with a second epoch advance, on the success, replaced-claim, and error exits alike. Sealing deliberately carries no quiescence requirement — it must succeed while a fork fence is active, because that is exactly the mid-turn race it defeats — and a fork that already committed against the begin epoch is healed the same way: the seal re-marks the head cold, so the next event rebuilds with complete history. Seal failure never diverts the turn's own exit; the begin bump still fences everything prepared before the turn. * 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence The second epoch advance could not make a legacy turn atomic, and three findings shared that root cause: two conditional updates left a headless gap a fork could create the head inside; the error exit sealed before saveErrorTurn made the error history durable; and any crash or failure between persistence and sealing left an incomplete fork authoritative, because the seal was best-effort and its failure was swallowed. A legacy turn now carries one durable fence. A token is written before execution by a single update-pipeline write — no two-write gap, and the cold marker is applied only where a head exists via $cond/$$REMOVE. While the token is present no fork may prepare (the adapter refuses) or commit (the CAS requires its absence), because the turn's messages are not yet durable. One atomic write clears the exact token and advances the epoch once history is persisted — after saveErrorTurn on the error exit — and success, replacement, and error exits all route through it. Failure is now fail-closed rather than silent: a failed seal leaves the token set, which keeps blocking forks and is logged as such, and a fence abandoned by a crashed turn is reclaimed only once stale, advancing the epoch and marking any head cold so the next event rebuilds from whatever history actually survived. * fix: serialize legacy event actor turns * fix: close legacy actor fence ownership gaps * fix: preserve legacy actor persistence fences |
||
Marco Beretta
|
0383030817
|
⚡ perf: Avoid Parallel Full and Paged Prompt Loading on Startup (#15031)
Co-authored-by: Danny Avila <danny@librechat.ai> |
||
James Todaro
|
018775de07
|
🧾 fix: Honor Disabled Transactions on the Assistants Usage Path (#15100)
* 🧾 fix: Honor Disabled Transactions on the Assistants Usage Path Thread the resolved transactions config through `recordUsage` from each of its callers, so `transactions.enabled: false` is honored on the assistants token spend path. * 🧾 fix: Thread the transactions config through the vision-request caller Address review: `ToolService.processVisionRequest` also records usage without the resolved config, and `recordUsage`'s documented return type did not match the function. * 🧾 fix: Set the resolved transactions config after the usage spread - provider usage could carry a `transactions` key that overwrote the trusted value - matches the ordering the other `recordUsage` callers already use |
||
|
Danny Avila
|
4b113697b5
|
🔌 feat: Background Execution Toggles for Actions & Plugin Tools (#14407)
* 🧵 feat: Background Execution Toggles for Actions & Plugin Tools * 🩹 fix: Resolve action background opt-in across encoded-domain forms and scope it per action * 🧹 refactor: Resolve action domain in a single pass * 🧩 fix: Merge Normalized Action Background Options * 🪢 fix: Reconcile Action Background Aliases * 🧭 fix: Harden Action Background Compatibility * 🕰️ test: Allow Settled Task TTL Expiry * 🧬 fix: Merge Refreshed Action Tool Registrations |
||
|
Danny Avila
|
d641c398d5
|
🧳 fix: Port Subagent Control Receipt Writes to DocumentDB-Safe Operators (#15171)
* fix: harden subagent control receipt persistence * fix: harden durable subagent control replay * fix: await terminal control receipts on shutdown * fix: close subagent control replay races * test: type stale-owner transport fixture * fix: quiesce durable subagent controls * test: await subagent shutdown durability boundary * fix: serialize durable subagent controls * fix: fail shutdown on cleanup errors * fix: report cancellable result availability accurately * fix: fence subagent control receipt ownership * fix: close distributed control receipt races * test: type control receipt race fixture * chore: require authoritative control receipts * fix: close subagent control lifecycle races * style: separate control reservation member * test: harden subagent settlement wait * fix: preserve authoritative control replay state |
||
|
Danny Avila
|
a997275902
|
🧾 feat: Persist Authoritative Subagent Control Receipts (#15168)
* feat: persist subagent control receipts * fix: require control receipt persistence * fix: preserve authoritative control history |
||
Paco Cartones
|
bf1e13b806
|
🥁 fix: Compare TOTP Codes in Constant Time (#15157)
Co-authored-by: pacocartones <pacocartones@users.noreply.github.com> |
||
|
Danny Avila
|
fc2b8584c4
|
📇 feat: Surface Event Child Activity Through a Bounded Parent Index (#15142)
* feat: surface event-driven child activity * fix: keep child task aggregation documentdb-compatible * fix: address event activity review findings * test: provide markdown message context defaults * fix: report bounded child history truncation * fix: preserve current child activity state * fix: preserve durable event child activity * fix: handle missing task timestamps * fix: keep active event snapshots live * fix: preserve event activity across valid anchors * fix: close event child activity gaps * fix: preserve event activity across resume |
||
|
Danny Avila
|
8969ee4b18
|
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) | ||
|
Danny Avila
|
c2aa688d73
|
🖥️ feat: Stream PTC Inner Tool Calls as a CLI-Style Trace (#15115)
* 🖥️ feat: Stream PTC Inner Tool Calls as a CLI-Style Trace Programmatic tool calling runs a whole program inside the sandbox, and the tool calls that program makes open no run step of their own. The card showed one running spinner for the entire execution, with no sign of what the code was doing. Emit a new `on_ptc_tool_call` step event for each inner invocation — once on dispatch, once on settle — and render them under the code as a terminal-style trace: status glyph, tool identity, argument preview, duration, with a failure message printed under the call that produced it. The seam is the tool map the sandbox bridge resolves inner calls against. `instrumentPtcToolMap` proxies `invoke` on each entry and leaves every other property (name, schema, mcp) passing straight through, so nothing about execution changes and emission failures can never fail a tool call. Client state is a per-tool-call Recoil atom keyed like the sandbox-starting and subagent atoms — live for the session, cleared on conversation switch so a finished program's trace stays readable. * 🩹 fix: Address Codex Review on the PTC Tool Trace Five findings, all confirmed against the source before fixing. Scope the trace atoms to a message occurrence. The hook already documents that providers repeat a tool_call_id across turns and even within one message, and `call_id` restarts at :0 for every outer call — so two programs sharing `call_0` merged into one card. Key by (response message id, tool call id) via `ptcTraceKey`, mirroring `subagentProgressKey`; the event's `runId` already carries the message id and the card reads its own from MessageContext. Prune unsettled rows on resume. Inner calls are not content parts, so the resume snapshot cannot rebuild them, and `trackReplayEvent` only persists OAuth events — a call that settled during a disconnect left a spinner that never resolved. Settled rows are real history and stay. Make the argument preview budget-aware. Iterate keys rather than entries so the budget check can actually skip work, and clip against a bounded window so a multi-megabyte value is never collapsed in full to build a 40-character preview. Catch the resumable emission promise. The synchronous try/catch around the emitter cannot observe a rejected `emitChunk`, so a failing transport raised an unhandled rejection per event instead of dropping telemetry. Announce completion to assistive technology. The check glyph is decorative and a fast call renders no duration, so a settled row previously announced no outcome; each row now carries an sr-only status and the visible cell that duplicated it is hidden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🧹 fix: Repair CI Failures on the PTC Tool Trace Two failures on the previous head, both mine. `Tests: api (shard 2/3)` — 46 failures in `initialize.spec.js`, all `TypeError: createPtcProgressEmitter is not a function`. The suite mocks the callbacks module with an object literal, and wiring the new emitter into `initialize.js` without adding it there left the factory undefined at call time. Added it alongside `createAttachmentEmitter`, plus an assertion that it receives the same generation fence as every other resumable emitter — a stale epoch would leak one run's inner calls into the next. `Static checks` — import-order drift in `PtcToolTrace.tsx` and `handlers.ts`, repaired with `scripts/sort-imports.mts`. ESLint and Prettier both passed, so only the dedicated check caught it. `openai.js` and `responses.js` never take the emitter, so their specs were unaffected; verified the initialize mock now covers every name the module destructures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🔐 fix: Address Second Codex Review on the PTC Tool Trace Three of five findings actioned; two answered on the thread. Respect tool-argument PII filtering (P1). Inner calls never reach `filteredToolArgumentsResult` — the sandbox bridge invokes them directly — so the trace was the one path putting their values on the wire in a deployment that had configured `filters.toolArguments.pii`. When any of the name / arguments / output fields are filtered, the emitter now omits both the argument preview and the failure message, which routinely quotes the argument that caused it. Name, status and duration still report. Drop the light/dark-specific background (P1). `dark:bg-transparent` stepped outside the semantic roles and would lose the intended separation under a custom theme. The pane now sets no background at all and inherits the card's surface, which resolves to the same color the override produced in both default themes and stays correct when a theme reassigns its roles. Bound the live trace (P2). A program looping over a large collection made every event copy an ever-growing array and rendered a row per call. The trace now keeps a rolling tail of 100 rows and counts what it evicted, surfaced as "+N earlier calls" so the cap is never silent. A settle whose row is gone — evicted, or pruned across a resume gap — no longer reappears out of order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ✅ test: Keep PTC Trace Tests Aligned With Caller-Capability Filtering Left out of the merge commit by a staging slip; without them `handlers.spec.ts` fails on the merged tree. `#15105` restricts the PTC tool map to tools whose `allowed_callers` admit code execution, so the existing trace test's registry entry — which declared none, defaulting to `direct` — was filtered out before the instrumentation could see it. Declare the fixture `code_execution`. Add a guard for the resolution itself: a `direct`-only tool must never appear in the instrumented map. Tracing wraps the eligible map, and this fails if a later change reorders that and lets the trace widen what the sandbox reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🛡️ fix: Close Name Disclosure and Follow the PTC Trace Tail Two findings from the third Codex pass on ` |
||
|
Danny Avila
|
1de88e7e91
|
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress * style: sort agent ingress imports * fix: harden agent event ingress * fix: bind event provenance to API keys * fix: inspect event input with legacy PII filters * fix: scope event status reads to source keys * fix: bind event status reads to remote sources * feat: add bound event-driven child turns * fix: harden event-bound child continuations * fix: satisfy event binding type contracts * fix: close event actor lifecycle races * fix: harden event actor dispatch continuity * fix: fence event actor resume lifecycle * fix: bind event actor state to lifecycle * fix: preserve cascade write outcomes * test: type cascade failure injection * style: sort cascade test imports * fix: harden event child lifecycle boundaries * fix: make event cleanup retryable * fix: annotate event retention clock * fix: reconcile partial cascade metadata * fix: recheck event binding expiry on resume * fix: fence event actors by retention deadline * fix: close event actor lifecycle races * fix: harden event child lease acquisition * fix: lazy-load event child lease adapter |
||
|
Danny Avila
|
89494d45fd
|
🚦 fix: Restrict Programmatic Tool Execution Maps (#15105)
* fix: restrict programmatic tool execution maps * chore: bump `@librechat/agents` to v3.6.10 * fix: honor live programmatic caller projections * style: sort caller capability imports * test: expect caller projection loader argument * chore: bump agents sdk to v3.6.11 * refactor: use SDK caller projection type * style: sort agent handler imports |
||
|
Danny Avila
|
d3e70159ca
|
📡 feat: Stream Detached Subagent Activity (#15111)
* feat: stream detached subagent activity * fix: annotate activity stream limits * fix: isolate subagent activity imports * fix: harden detached subagent activity lifecycle * test: cover synchronous activity transport failure * test: include required subagent activity identity * fix: identify and reconnect subagent activity events * fix: bound subagent activity lifecycles * fix: close subagent activity handoff races * fix: bind and synchronize activity subscriptions * fix: detect fresh activity attachment * fix: complete activity synchronization handoff * fix: bind activity sync and failure circuits * fix: expose subscription-bound synchronization * fix: fence activity reconnect publications * test: make detached timeout settlement deterministic * fix: fence Redis activity attachments * fix: close failed activity streams * perf: reuse fenced activity frontier * style: sort subagent thread imports * fix: preserve queued subagent activity * test: type activity publication counter * fix: disconnect subagent activity subscriber * fix: close background activity lifecycle gaps * fix: preserve streamed activity spacing * fix: preserve bounded live subagent activity * fix: merge durable subagent activity safely * fix: model detached activity coverage * fix: type detached activity inputs * fix: order overlapping subagent activity * chore: sort activity test imports * fix: buffer subagent activity handoff gaps * fix: flush activity after parent close * fix: advance closed activity suffixes * fix: preserve detached activity ordering * fix: close detached activity delivery races * fix: bound shared Redis subscriber readiness * fix: expire shared Redis subscription readiness * fix: clean up late Redis subscriptions * fix: preserve late Redis subscription fallback |
||
|
Danny Avila
|
3ebef4c84e
|
📨 feat: Add Authenticated Agent Event Ingress (#15110)
* feat: add authenticated agent event ingress * style: sort agent ingress imports * fix: harden agent event ingress * fix: bind event provenance to API keys * fix: inspect event input with legacy PII filters * fix: scope event status reads to source keys * fix: bind event status reads to remote sources |
||
|
Danny Avila
|
67b7b441b2
|
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam * feat: enforce source-aware content filters * feat: complete source-aware content enforcement * test: activate skill file-text fail-close fixtures * fix: harden source-aware content filters * fix: harden model-bound content filtering * fix: preserve legacy filters and generated files * fix: inspect shared scalar metadata * test: align mocks with current dev dependencies * feat: add persisted content filter safeguards * feat: complete source-aware content filter enforcement * fix: move resume content preflight into TypeScript * fix: close content inspection edge cases * fix: harden content protection boundaries * fix: complete content protection safeguards * test: align persisted memory filter coverage * fix: reconcile content protection with current dev * fix: reconcile content protection with latest dev * fix: close content protection review gaps * fix: enforce source-aware provider boundaries * fix: preserve legacy PII preflight semantics * test: stabilize stored branch preflight fixture * fix: defer agent writes until protected model admission * perf: harden source-aware model-bound filtering * fix: canonicalize provider lineage before validation * fix: satisfy model-bound callback type checks * perf: Bound content protection filtering work * fix: Bound submission array traversal * fix: Stabilize bounded content snapshots * fix: Scope model-bound traversal overflows * fix: Preserve scoped content inspection * fix: Accumulate aggregate traversal scopes * fix: centralize content policy boundaries * test: align deferred tool policy context * test: align controller policy mocks * style: normalize content protection imports * fix: close content policy review gaps * fix: narrow active skill policy config * fix: address content protection review boundaries * fix: retain exact provenance overflow sentinel * fix: preserve literal and scoped provenance updates * fix: narrow persisted edit provenance * fix: isolate exact overflow attribution * fix: centralize stored prompt protection * fix: fail closed on incomplete transcript evidence * fix: align canonical transcript routing * refactor: centralize content policy preflights * fix: isolate upload policy error typing * style: sort policy preflight imports * refactor: centralize content policy boundaries |
||
|
Danny Avila
|
8ae94afa91
|
🪡 fix: Thread Parent Message ID Through MCP Request-Scoped Bodies (#15095)
* fix: Unify MCP request-scoped headers * fix: address request-scoped MCP review findings * test: preserve request scope on status errors * fix: treat authorized on-demand MCP servers as ready * refactor: separate MCP readiness from connection state * fix: preserve on-demand MCP readiness labels * test: satisfy OpenAI conversation ownership guard * fix: keep MCP action predicates boolean * fix: close deferred MCP request context gaps * fix: preserve on-demand MCP configuration actions * fix: fail closed on unavailable MCP parent context * test: complete MCP connecting-state mocks * fix: preserve missing MCP parent on continuations * fix: align native MCP request identities * fix: preserve edited MCP parent identity * test: use scoped Agent initializer fixture * test: expose MCP request body helper * fix: preserve MCP turn identity across resume * style: sort stream metadata imports * fix: carry normalized MCP identity to execution |
||
|
Dustin Healy
|
f02ce63d57
|
✂️ fix: Strip Redundant Server-Name Prefixes from MCP Tool Keys (#14732)
* ✂️ fix: Strip Redundant Server-Name Prefixes from MCP Tool Keys MCP servers that prefix every tool with their own name produce model-facing keys that embed the server twice once the _mcp_<server> suffix is appended, pushing long tool names past provider 64-character function-name limits. Tool keys now drop a leading <normalizedServerName>_ prefix (case-insensitive, skipped when a sibling tool already owns the stripped name). The original upstream name is recorded as serverToolName on the cached definition and is always what tool calls send to the server, and runtime lookups also try the stripped spelling of persisted pre-strip keys so existing agents keep resolving. * 🩹 fix: Keep Stripped MCP Tool Keys Provider-Safe and Collision-Free Assistant writers submit catalog entries verbatim, so the internal serverToolName mapping is now removed from provider-facing definitions before they reach create/update payloads. Prefix stripping is collision-guarded over the resulting name set rather than raw siblings only, which also covers case-variant prefixed pairs under the case-insensitive match. Assistant payload healing now rewrites a pre-strip persisted key to the stripped catalog key when that key actually exists in the loaded definitions, and legacy agent references keep their persisted spelling as the runtime instance name so per-tool options stay applied while the upstream call still uses the matched entry's raw name. * 🧷 fix: Harden Stripped MCP Tool Keys Against Heal, Collision, and Cache Edges The pre-strip heal now resolves the key boundary against both raw and normalized server spellings, mapping back to the raw name for the shadow and membership guards, so keys persisted after server-name normalization heal too. Collision detection iterates to a fixpoint so a fallback to a raw name cannot silently collide with another sibling's stripped result, and a stripped remainder equal to a synthetic marker (wildcard or server pin) is never produced. MCP catalog cache slices are versioned so replicas that predate serverToolName never read stripped entries during a rolling deploy; stale slices expire on their own. * 🔎 fix: Resolve Pre-Strip Keys in Event-Driven Definitions and Reinspect Persisted Catalogs The event-driven definitions loader now tries the stripped spelling of a persisted key when the exact lookup misses, keeping the persisted name so it matches the runtime instance, which stops legacy agents from failing initialization with expected tools unavailable. The registry storage schema version is bumped so followers rebuild persisted toolFunctions instead of republishing pre-strip definitions into the versioned catalog namespace. The assistants heal also fails closed when a normalized-suffix reference lands on a contested server-name slot, since rewriting persisted data must not bind an ambiguous reference to the tie-break winner. * 🛰️ fix: Reserve the Synthetic OAuth Name and Heal User-Owned Server Keys A stripped remainder equal to oauth would make the client stream handlers treat a real tool call as a synthetic authentication prompt, so it joins the reserved remainders alongside the wildcard and pin markers. The assistants heal now audits the FULL accessible server set on every run instead of operator config names only, since assistants reference user-owned servers whose catalogs the definitions loader already resolves; an unavailable audit still skips healing entirely. * 🧬 fix: Verify Upstream Identity for Legacy Keys and Reserve Sibling Raw Names Stripped results now reserve every sibling's raw name even when that sibling itself strips, so a stripped key can never shadow another tool's pre-rollout persisted references within the same snapshot. Every legacy fallback (runtime lookup, event-driven definitions, assistants heal) accepts a stripped-spelling match only when the entry's recorded serverToolName proves the same upstream tool, so a stale key for a removed tool degrades to unavailable instead of calling a different sibling. To keep that identity visible to the heal, assistant tool definitions retain serverToolName and the controllers sanitize entries through toProviderToolDefinition at the provider submission boundary instead. The agent editor migrates pre-strip persisted ids the same identity-verified way, with the upstream name exposed on the MCP tools payload. * 🧭 fix: Heal Wildcard Tool Options and Reserve the OAuth Namespace Wildcard-expanded catalogs rename stripped tools without any agent.tools entry to preserve the spelling, so buildToolClassification now aliases persisted pre-strip tool_options keys onto the current instance names in place, identity-gated on the definition's recorded upstream name and never overriding an explicit entry. Both loading modes flow through it: instances carry mcpServerToolName from createToolInstance and event-driven definitions thread serverToolName from the catalog. stripServerNamePrefix also reserves the entire oauth namespace rather than the exact name, since the client stream handlers classify every oauth-prefixed key as a synthetic authentication call. * 🛡️ fix: Derive the Full Reserved Namespace and Heal Approval Policies The reservation guard now covers every namespace consumers classify by prefix: the wildcard and server-pin markers alongside oauth, plus the server-scoped mcp_ pluginKey namespace that pre-strip keys could never enter. Stripping also never produces a key whose isActionTool classification differs from the raw key's, since a server whose normalized name contains _action_ would otherwise see a real MCP tool routed down the OpenAPI action path past MCP authorization. Admin toolApproval globs written against upstream tool naming keep applying: pattern lists are healed at run wiring with the current names of tools whose pre-strip spelling matches, list-level so deny, ask, and allow precedence is unchanged and a non-matching deny can no longer fail open. The MCP tools wire type also declares serverToolName end to end. * 🪪 fix: Alias Both Key Spellings for Approval Policies and Hook Matchers Identity aliases are now collected once at tool classification, in both directions: a stripped instance aliases its pre-strip spelling and a legacy-named instance aliases its current catalog spelling, with the current name recorded on legacy matches by the runtime lookup and the event-driven definitions loader alike. The aliases ride the agent config through both loading modes, so approval pattern healing applies to deny rules written against either spelling, closing the bypass where a rule targeting the current name missed an unedited agent's legacy instance. Programmatic approval hook matchers get the same treatment: each hook is additionally registered under an anchored exact-name pattern for tools whose other spelling its regex matches, keeping the admin's matcher semantics intact while argument, user, and tenant specific deny or ask decisions keep executing for renamed tools. * 🔁 fix: Alias Tool Options in Both Spelling Directions Options aliasing now consumes the same bidirectional alias pairs as policy healing and hook matchers, so options the editor migrated to the current catalog spelling still reach a legacy-named instance retained by an unedited agent.tools entry. The previous serverToolName-only derivation skipped exactly that case since the legacy key equals the instance name there. * 🤝 fix: Reserve the Agent Handoff Namespace Before Stripping The client renders any lc_transfer_to_ prefixed call as an agent handoff and the background and intent passes exclude such names, so a stripped remainder inside that namespace would misclassify a real upstream tool. It joins the mcp_ pluginKey namespace as a bare-prefix reservation, which pre-strip keys could never enter. * ⚡ fix: Reuse the Loader's Server Snapshot and Index the Editor Catalog getAssistantToolDefinitions now returns the accessible-server snapshot from the same merged registry read that resolved the catalogs, and the heal consumes it instead of repeating the app-config and registry round trips on the assistant write path; without a snapshot the heal still fetches and fails closed as before. The agent editor's id migration uses a memoized tool_id map, so the per-key form heal does constant-time lookups instead of scanning the catalog per option. --------- Co-authored-by: Danny Avila <danny@librechat.ai> |
||
|
Danny Avila
|
8c14f03432
|
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects Adds an optional chat-project destination to a schedule, plus the operator config to require one — or to pin every scheduled run to a specific project. Feature: - `chatProjectId` on the schedule row, accepted on create/update, projected on the wire, and carried into the run's conversation through the durable trigger envelope's `run` context. - `interface.schedules.requireProject` refuses schedules that are not filed under a project; `interface.schedules.projectId` pins every run to one project and implies the requirement. - Dialog gains a project picker (required when configured, a read-only row when pinned); the card shows the destination and the new disabled reasons. Invariants: - ONE resolver (`resolveScheduleProjectId`) decides the destination for the write handler, the fire path, and the wire projection alike, and an operator pin OUTRANKS the stored id in all three. Tightening the config therefore redirects — or stops — existing schedules instead of grandfathering where their runs land. A pin implies `requireProject` for the same reason: without it, a row created before the pin would keep firing with no project at all. - Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write this handler accepts is one the next fire also accepts. Any edit leaving a schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips the requirement, or a schedule auto-disabled for `project_required` could never be turned off. - Fire-time enforcement auto-disables rather than filing runs loose, matching agent_deleted: new `project_required` (requirement raised after creation) and `project_deleted` (gone, or pinned to a project this owner does not have) reasons, both refused BEFORE a billed generation is dispatched, and both advancing so a schedule can never wedge on the occurrence. - `computeCreateDigest` appends the field only when present, so a payload without a project digests byte-identically to one from before this change — a create retried across the upgrade still matches its own row instead of reading as key reuse. - Project reads are scoped to the owner, so ownership and existence are the same lookup; a read ERROR propagates instead of failing closed, so a Mongo blip retries the fire rather than auto-disabling the schedule. The trigger idempotency key hashes principal/event/target and never `envelope.run`, so the added run field cannot destabilize delivery identity. * 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget The project picker landed as a new ROW in the schedule dialog, which broke the e2e edit spec: `md:overflow-visible` turns off the template's scrolling from `md` up, so the dialog's content must fit the viewport. The extra row pushed the footer's Save button below a 720x1280 window, where Playwright reported a visible, enabled button it could never click — 226 scroll-into-view retries and a 2-minute timeout, on all three attempts. Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720): dev 688 (32px slack) project row ~790 (off-screen, CI failure) 3-column row 704 (16px slack — half the budget spent) this commit 686 (34px slack, 2px better than dev) The identity row is now three columns — name, agent, project — and its caption moved out of the agent cell to sit full width beneath the row: at a third of the dialog that sentence wraps an extra line, and the row is the tallest thing competing for the budget. The caption is grouped with the row rather than left to the form's own 4-unit rhythm, which spent more height on the gap than the caption occupies. The e2e spec now asserts the button is in the viewport before clicking it, so the next field that overflows this dialog says so in one line instead of a two-minute timeout on a visible element. FOLLOWUPS.md records what the planned dialog controls (multi-day weekly, timezone, attachments) need first: give `ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the popovers into the dialog content, and let the form scroll again. * 🧹 fix: Address Codex Review on Scheduled Chat Project Scope Four P2 findings, all real. Unreachable clearing path (ScheduleDialog). The picker only held live projects, so `com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a schedule had a project the owner could never take it away, leaving the server's `chatProjectId: null` path reachable only by API. The picker now carries a real "No project" option whenever a project is optional, and omits it when one is required, where there is nothing valid to select. Placeholder shown for a real project (ScheduleDialog). A stored or pinned project outside the first loaded page had no name in the paged map, and the combobox renders its placeholder for an empty display value — telling the owner a scoped schedule had no project. That one project is now read by id, with the raw id as a last resort: a poor label, but an honest one. Project policy skipped at the resume boundary (service.ts). `claimScheduleResume` re-applied the schedules gate, the revision fence, the kill switch and SCHEDULES:USE, but not the project policy this PR added. Approving a paused run whose project was deleted — or whose owner now sits under a requirement or a pin it no longer satisfies — billed a continuation the very next scheduled fire would refuse and auto-disable the schedule for. The effective-project resolution now runs there too, refused before the lease and the capacity slot so a policy refusal costs nothing and leaves no state to unwind. NOTE: agent access and balance are still not rechecked on resume; that gap predates this PR and is left alone. Per-card project derivation (ScheduleCard). Every card ran the projects hook and rebuilt the full option array, name map, and one icon element per project, to use a single name — O(schedules x projects) per render and per project-list refresh. The hook is split: `useChatProjectNames` (map only, for the panel, which resolves every card's name once and passes it down) and `useChatProjectPicker` (options and pagination, for the dialog's one combobox). The panel skips the query entirely until some schedule actually has a scope. Tests: four at the resume boundary (verified failing without the gate) and four in the dialog spec. The picker selection in one existing test now goes through the search field — the popover's VIRTUALIZED renderer sizes its window from a scroll height jsdom always reports as 0, so with three options it materialized only two. Full schedules e2e re-run green against the rebuilt client. * 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent Second Codex round, four P2s. Three were consequences of the resume gate added in the previous commit, which was half-built: it admitted where it should not and stranded the run where it refused. Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy` branch. Both entry points consult that branch FIRST, and both already route its refusal through abort-and-settle — so a policy stop now settles the occurrence instead of answering a bare 409 while the job stays `requires_action`, the card keeps reading "Needs approval", and every retry repeats the same 409 until expiry. No change to resume.js: the existing branch does the work. The rule is deliberately NARROW. It refuses only where no valid destination is left — the requirement is on with nothing satisfying it, or the schedule's own project is gone (which also unset it on the conversation). It does NOT refuse because an operator's pin moved: the paused conversation cannot be rebound (`chatProjectId` is excluded from the resume context and the continuation reuses the same conversationId), so refusing would strand a pending approval over a destination it can never reach, for a pin that governs only where the NEXT run lands — which the fire path already redirects. Create retries are idempotent again. Project policy had been applied BEFORE the `clientRequestId` replay lookup, so a raised requirement, a deleted project, or a moved pin could answer 400 for a create that already committed — pushing the client to rotate its key and create a DUPLICATE schedule, the exact failure the key exists to prevent. Policy now applies only to a genuinely new insert, and the digest is computed from the CLIENT's payload rather than the resolved destination, so today's policy can no longer re-digest a genuine retry into a mismatch. An explicit `chatProjectId: null` under a pin is refused rather than silently resolved to the pin. The payload contract defines `null` as clearing the scope; answering 201 while filing under the pin reported success for the opposite of what was asked. Only an OMITTED field takes the pin silently. Tests: five on the policy branch (both refusals verified failing without it, plus guards that a moved pin and a live project still admit) and two on the handlers (the pinned explicit clear, and a committed create recovered by retry after the policy tightened — verified answering 400 without the reordering). * 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved Third Codex round, four P2s. The root confusion behind the resume findings: an operator pin outranks the stored id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row keeps its old value. The row therefore LIED about where that occurrence's conversation went, and every later re-validation — the resume boundary above all — checked a project the conversation was never filed under. A schedule storing A, pinned to B, with B later deleted and A still live, was admitted for resume into a conversation that had just been unscoped. Fixed at the source rather than at each reader: a fire that resolves a destination different from the stored one writes it back, claim-token fenced like every other worker-side write and deliberately WITHOUT a configRevision bump — this is the server reconciling itself to policy, not an owner edit, and a bump would fence an in-flight occurrence off its own run. Written only AFTER the destination validates, so an unusable pin never lands in the row, and best-effort: the envelope already carries the right destination, so a failed write costs accuracy on a later recheck, never the run itself. The wire projection already reported the pin, so this also stops the row and the UI disagreeing. An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit path too. The pin check and the requirement are independent rules, and folding them together let `{enabled: false, chatProjectId: null}` skip the pin check entirely, unset the row, and answer with a wire projection still naming the pin. Only the requirement is waived for a disabling edit. The dialog no longer requires a project for an edit that leaves a schedule DISABLED. The server waives the requirement there precisely so a row auto-disabled for `project_required` can still be renamed or tidied up; requiring it in the form made that unreachable, and an owner with no projects could not edit the stopped schedule at all. FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second deletion race inside the resume claim window (which needs the effective project persisted per OCCURRENCE plus a distinct policy conflict routed through abort-and-settle), and the fact that convergence happens only when a schedule fires. Tests: four on convergence (pin written, no write when unchanged, no write for a destination that failed validation, fire survives a failed write), two on the disabling-edit pin rules, two in the dialog. Schedules e2e re-run green. * 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest `computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and an explicit `null` produced the same digest. Because the replay lookup and `matchesCreateIntent` deliberately run before project policy, a request could reuse a pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and receive 201 describing the pinned row — success reported for the opposite of what it asked, and the pinned-clear refusal the normal create path applies never reached. Now `!== undefined`: an omitted field still digests byte-identically to a payload from before project scope existed, so a create in flight across the upgrade still matches its own row, while an explicit clear is a distinct intent and digests differently. A pre-scope client never sent the field at all, so nothing legacy can carry an explicit null. * 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used The schedule-wide convergence from the previous commit was not enough, and the reason is the single-active run index: it covers `status: 'started'` only, so a PAUSED run does not block the next occurrence. While run 1 sat paused in project A, a pin move plus one later fire rewrote the schedule row to B — and the resume policy then validated B while run 1's conversation was still filed under A. Delete A and the continuation was admitted into a conversation that had just been unscoped. That window lasts as long as the pause, not the sub-second race the previous commit documented. The reservation now records the destination THIS occurrence used, and `isScheduleLive` validates that record when given the occurrence's `scheduledFor` — which `resume.js` already reads two lines above the call. No new conflict type and no settlement-path surgery: the refusal rides the abort-and-settle branch that check already has. An ABSENT record falls back to the schedule-level resolution rather than reading as unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as they do today. Schedule-wide convergence stays: it keeps the row honest for the UI and for every check that has no occurrence in hand. FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the claim window, which needs a distinct policy conflict routed through abort-and-settle rather than the bare 409 an `inactive` conflict produces. Tests: three on occurrence-vs-row precedence (the decisive one verified failing without the lookup), two on what the reservation records, and the resume controller spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green. * 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One The occurrence fallback added in the previous commit conflated two different absences. A post-upgrade run that deliberately went unscoped omitted the field exactly like a row written before the field existed, so both took the fallback — and a paused unscoped run was then validated against the schedule's CURRENT project. Under a requirement or a pin added while it sat paused, that admitted a billed continuation into a conversation satisfying no present policy. The reservation now ALWAYS records its decision, `null` for unscoped, and the read reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record — a pre-scope row, or no row at all — falls back to the schedule-level resolution; a recorded null is the genuinely unscoped occurrence it says it is, and is refused once a project becomes required. The distinction rests entirely on a stored `null` surviving as a present key while a never-written field stays absent, so that is asserted against real Mongo rather than assumed: if it ever stopped holding, unscoped runs would silently start being validated against the schedule's current project again. Tests: two against mongodb-memory-server (recorded null vs never-written vs missing row), plus refusal of a recorded-unscoped occurrence under a new requirement and the preserved fallback for a pre-scope one. Schedules e2e green. * 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence The initial-start policy check in `request.js` called `isScheduleLive(..., { policy: true })` without `scheduledFor`, so it fell back to the schedule-level resolution even though the run row already carries the occurrence's recorded scope. An occurrence reserved unscoped, with a pin introduced while its loopback request sat queued, was therefore admitted against the new pin — producing a billed unscoped conversation under a requirement it does not satisfy, from an envelope already built without a project. `scheduledFor` was already in scope there. Passing it makes the initial start and the resume validate the same way: against the destination the occurrence itself recorded. |
||
|
Danny Avila
|
a5cb041f47
|
🕊️ feat: Yield to Subagent Completion Wakeups (#15066)
* 🕊️ feat: yield to subagent completion wakeups * ⚡ fix: bound wakeup status guidance |
||
|
Danny Avila
|
c5276fc63d
|
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations
Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.
Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
and capacity slots, timezone-aware cadence with DST-conservative floors and
misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
(signed schedule-fire JWT claims, per-occurrence idempotency key inside the
route's clientRequestId charset), overlap/balance/capacity/duplicate skip
policies with auto-disable streaks (too_many_failures, insufficient_balance),
reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
path: HITL pauses surface on the card (requires_action), resumes re-apply the
fire boundary's admission policy (revision fence, enabled, global kill switch,
SCHEDULES:USE, availability) before continuing a billed generation.
Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
run's terminal outcome write; Stop/complete/pause race through single-winner
terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
completedAt-less evidence for scheduled fires plus an owner-intended outcome
stamp (scheduleOutcome) the reconciler prefers over re-derived success —
round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
error/skipped_balance instead of success on both initial and resumed paths;
stale stamps are refreshed evidence-first when persistence plus the Mongo
outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
from-status; republication escalates to the transport's acknowledged variant;
the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
of scheduled + interactive work with durable per-stream abort fences
(positive-evidence acknowledgement only), owner-side finalization markers for
post-terminal billed writes, deferred-deletion sweep (explicitly ensured
partial index) that completes cascades autonomously. Remote OpenAI-compatible/
Responses requests are documented as outside the quiesce and tracked in issue
14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
contract with coherent degradation (registration fails -> synchronous title
fallback; count reads 0).
* fix: fence terminal response persistence from deletion; retain stale-pause evidence
Two blockers from the third external review round.
Terminal persistence visible to account deletion:
- The finalization marker was registered only when post-terminal TITLE work was
possible, but every persistence-owning terminal CAS opens the same window: the
claim drops the job out of the active set BEFORE the response save (and the
background user-message/convo saves), so a deletion quiesce landing there saw
neither an active job nor a marker and could cascade while the admitted request
could still recreate messages. Both controllers now register the marker before
every persistence-owning claim — the fresh-turn path and the HITL resume — and
release it only after their pending saves (and any post-terminal title) have
landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
`terminalPersistencePending` is set: terminal at the CAS is not settled while
the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
refuses a store without it at configure time, keeping the failure loud and
deterministic) while remaining optional on the legacy public IJobStore type for
source compatibility. The silent degrade path from the previous round is gone —
it was not deletion-safe.
Stale-pause recovery retains scheduled evidence:
- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
and the InMemory/Redis stale-pause cleanups — unconditionally stamped
`completedAt`, putting a scheduled fire's failed-pause error terminal on the
short completed TTL. A Mongo outage longer than that TTL erased the evidence and
the reconciler recovered the run as `interrupted` instead of `error`. All three
now follow the controller-observed path from the previous round: scheduled jobs
omit `completedAt` (retained-evidence TTL) and stamp the error outcome.
The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).
* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths
One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.
- Generation-scoped markers. Entries were keyed (userId, streamId), so a
Stop-superseded generation finishing late could clear the marker its
replacement registered on the same conversation. Marker fields are now
qualified by the generation's createdAt; clears must present the same
identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
user's existing hash expires between the two commands (or the process dies
there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
the terminal CAS instead of proceeding uncovered: the completion claim throws
into the error path, the error path skips completeJob and leaves the job
ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
and abortJob returns a new retryable `fence_unavailable` failure the Stop
route answers with 503 and the deletion quiesce treats as an unacknowledged
stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
the abort CAS, clear in its finally — the Stop route's checkpoint prune and
partial save run inside beforePublish, between CAS and publication); the
interactive and background generation-error paths register before their
completeJob; the resume controller's error finalization registers before its
completeJob. A lost or thrown claim releases the marker after pending saves
flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
terminalPersistencePending for ALL statuses — including `aborted`, whose
route-side persistence the previous guard missed.
Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.
* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment
Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.
- Lease tokens. Marker fields were (streamId, createdAt), shared by every
contender on the same generation — completion and Stop, or two racing Stops —
so a losing contender's cleanup erased the winner's still-live marker. Every
registrant now carries a unique lease token in the field and may only ever
clear its own lease; unqualified legacy clears cannot touch qualified entries.
- Admission fence. Authentication can pass before the deletion barrier goes up,
and the durable createJob is several async steps later — a deletion quiesce in
that window saw neither an active job nor a marker and could cascade before
the admitted request created its job. The controller now registers an
admission lease and THEN rereads the deletion barrier: the ordering guarantees
either this request observes the barrier (403, lease released, slot/claim
cleanup) or the quiesce observes the lease and defers. Held until createJob is
durable; released on every refusal and initialization-error path. Fail closed
when the lease itself cannot be registered (503 retryable).
- Undelivered-Stop retention. abortJob released its lease in a finally even when
delivery AND publication had provably failed — the job reads terminal
(invisible to active-set scans), a user Stop writes no durable abort fence,
and the remote owner keeps generating and will persist its abort-catch writes
whenever the signal finally lands. The lease is now retained in exactly that
case, and each resignal attempt heartbeats a fresh lease so the fence outlives
the TTL for as long as delivery is still being driven. The abort-winning
turn's own loser-side pending saves are additionally fenced in the controller
catch (best-effort — those writes are already in flight).
- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
non-assistants endpoints) awaited abortJob and then spent usage and saved the
stopped response AFTER the abort's lease was released. Both writes now run
inside `beforePublish`, between the abort CAS and publication, covered by the
same lease as every other abort.
Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.
* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races
Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.
- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
-job handoff is only atomic against a reader in the OPPOSITE order of the
writer: writers hold the lease strictly until the job is active-set visible,
so leases-first shows every interleaving either the lease or the job.
Jobs-first allowed a request to create its job after the scan and release its
lease before the count — hiding both, cascading, and letting the new
generation persist into a deleted account.
- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
the moment the owner's AbortController tripped; the stopping side released its
lease on that ACK while the owner's asynchronous abort-catch persistence was
still ahead. The transport now awaits a manager-installed pre-ACK hook that
registers a DETERMINISTIC owner lease (exactly one owner exists per
generation, and determinism is what lets the signal-time registrant and the
owner's catch-side release agree across processes) before the acknowledgement
is persisted or published; an owned same-replica abort bridges to the same
lease before tripping its local controller. Both generation-owner catches
(fresh turn, resume) release it once their writes land.
- A failed replacement handoff no longer orphans the predecessor. The atomic
replacement removes it from active storage, and an unconfirmed handoff
terminalizes the replacement too — leaving nothing a quiesce could discover
while the predecessor's provider may still be generating. Its owner lease is
now retained at the point the receipt fails delivery; the owner replica renews
it through the pre-ACK fence when the signal finally lands.
- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
holder; live persistence — a stalled save, a long deferred title — must never
outlive its own fence. holdUserFinalization registers and renews every minute
until released; the controllers' completion/error/admission leases all hold.
The undelivered-Stop retention moved to a deterministic `stop` lease that
every resignal attempt renews and the first successful one clears (no more
opaque leases accumulating to TTL), and a THROWN abort transition releases the
contender lease instead of leaking it.
- The user-document abort-fence mutations now invalidate the auth user-doc
cache, matching every other user-doc write.
Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.
* fix: one manager-owned owner-lease span across every abort delivery path
Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.
- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
the transport ACKed even when it failed; a same-replica owned abort likewise
swallowed registration failure. The hook now acquires a HELD owner lease
(heartbeat until the owner's catch releases it via releaseOwnerLease) and a
rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
behind its retention lease, and every resignal re-drives the handler. The hook
also stopped gating on `job.createdAt === generationId`: during a replacement
handoff the store holds the replacement while the abort targets the
predecessor, and that gate silently skipped exactly the generation being
acknowledged (the store job is owner identity, never a generation gate). A
local owned abort acquires the same held lease before tripping its provider;
post-CAS the trip cannot be withheld, so acquisition failure downgrades
delivery and the retention handoff keeps the user fenced.
- Committed-but-lost-reply disambiguation. A thrown abort transition released
the contender lease as if nothing had happened, but a Lua CAS can commit and
lose its reply — an aborted job invisible to active-set scans whose provider
was never signalled, with no fence left. The throw path now re-reads the exact
generation: only a job still live under the caller's identity proves no
commit; committed or ambiguous outcomes hand the fence to the deterministic
stop lease (kept on the contender lease if even that fails) before rethrowing.
- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
predecessor's held owner lease before the trip (failure reports the receipt
undelivered, engaging retention). Failed-handoff retention is no longer a
swallowed one-shot: it heartbeats with the durable acknowledgement proof as
its renewal predicate — acquisition failures keep retrying for as long as the
fence is needed, and the retainer stands down (without clearing the shared
field) once the remote owner ACKs and thereby holds its own lease.
- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
retained stop lease, and keeps the stop lease when that handoff fails.
Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.
* fix: finish scheduled chat lifecycle hardening
* fix: close scheduled chat review follow-ups
* fix: generation-fence abort recovery evidence
* test: wait for settled approval tool output
* test: preserve scheduled init reconciliation option
* refactor: rebuild scheduled chats on durable agent triggers
* test: reset MCP cache mock between cases
* test: isolate scheduler startup in server specs
* fix: harden scheduled run lifecycle
* fix: normalize schedule capacity conflicts
* test: type schedule collision fixture
* fix: re-fence scheduled resume and expiry
* fix: fence scheduled resume handoffs
* fix: release superseded manual schedule leases
* fix: release failed run-now claims
* fix: release superseded engine claims
* fix: repair schedule dialog interaction and rework its form
The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.
Alongside that:
- Extract the agent builder's instructions editor (special-variable menu plus
expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
schedule prompt. Insertions now route through `onChange`, so react-hook-form
sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
the selection with `bg-surface-hover` on an outline button whose hover is the
same token, so the selected option was indistinguishable from a hovered one;
`Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.
The new dialog spec fails when the portal fix is reverted.
* test: cover scheduled and subagent deletion drains
* refactor: own the form-control appearance in the client primitives
Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.
Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.
This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.
The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.
* fix: keep an explicit schedules disable from becoming an opt-in
`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.
Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.
Both regressions fail without the normalizer.
* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep
Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.
Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.
Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension
Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.
Replace the destructive marking with a REVERSIBLE, token-fenced suspension:
- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
cannot copy field values under DocumentDB), fenced per row so an already-
suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
re-arming only rows still carrying the exact attempt token and not independently
deleted — so an owner-deleted or newer-attempt-suspended schedule is never
resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
any failure that cancels the user-deletion fence restores the suspended rows. A
successful deletion hard-deletes them (and their snapshots) via the existing
cascade and never restores.
Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol
Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.
Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):
- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
acknowledges only after beforePublish persistence succeeds, and on a
persistence failure leaves the barrier unresolved so the run stays preserved
(client retries; stale-owner timeout is the bounded recovery). A failed abort
releases the stamp it placed so a replacement/retry is never blocked through
its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
proceeds immediately. The paused Stop settles only after its own ack.
Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation
Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.
Translate durable delivery state into the schedule outcome:
- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
(computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
admission is live, never orphan; dead → record `error` from the durable
lastError and release capacity promptly (no 30-minute wait), through the
ordinary outcome/auto-disable path; succeeded or no record → the existing
legacy orphan policy (interrupted only past the cutoff); a delivery lookup
failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
moderation middleware writes no schedule state.
Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions
Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.
Make the host lifecycle work durable rather than schedule-specific:
- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
fenced by generation identity (clearTerminalHostAction), so a replacement
generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
across restarts and replicas and retries the idempotent hook; the relay only
re-invokes while the marker is unacknowledged, so a successful ack prevents
duplicate work. Store-won expiry marks it too, so a loser-replica relay still
crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.
Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts
CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): repair the deletion CLI and order restore before the fence release
Addresses three findings from the fresh Codex review.
P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.
P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.
Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.
Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge dead deliveries in topology-safe maintenance
Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.
Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.
Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.
Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): refresh host-action retention on each retry attempt
Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.
Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.
Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): defer settlement when the Stop barrier times out
Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.
The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.
Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops
Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.
P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.
P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.
P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.
P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.
Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge terminal runs in clustered workers, retry suspension restore
Fourth Codex round on
|