mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 15:02:02 +00:00
27 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7e7f559ea6
|
🏎️ perf: Parallelize Independent Agent and Schedule Lookups (#17465)
* perf: Parallelize Serial Lookups on Agent Load, Discovery, Skills, and Schedules * fix: Address Review Findings on Parallel Lookups Stage same-level discovery results before collecting edges so a shared sibling is initialized once, bound resume snapshot fetches to small chunks so over-budget graphs stop early, and surface the first ordered image failure without waiting on later reads. Update the skill name test for bounded concurrency. * test: Tag Existing Mock Specs as Acceptance Scenarios for Parallel Lookups * feat: Make Parallel Lookup Bounds Configurable Add a concurrency section to librechat.yaml for MCP catalog, skill name, skill file and resume agent lookups. Defaults equal the previous hard-coded values. * style: Sort Imports for Configurable Lookup Bounds * test: Match Model Spec Skill Resolution Args With App Config * test: Close the Builder Drawer Before Chatting on Narrow Viewports * fix: Surface Fatal Discovery Errors Without Awaiting Later Siblings * fix: Reject Discovery on the First Fatal Load Regardless of Queue Order * fix: Keep Image Encoding Fail-Fast With Parallel Reads * fix: Reject Resume Snapshot Lookups Without Waiting on the Whole Chunk * feat: Make Connected Agent Discovery Concurrency Configurable |
||
|
|
253112ff8d
|
🧷 fix: Close Only Scheduled MCP Authority at Request Cleanup (#17354)
* 🧷 fix: Close Only Scheduled MCP Authority at Request Cleanup resolveScheduledMCPBearerConfig runs on every MCPManager.callTool, and it threw ScheduledMCPBearerError('binding_mismatch') whenever the request context had started cleanup, before checking whether the context was scheduled at all. A detached subagent keeps its parent turn's MCP request context; once the parent turn ended, every pooled MCP call from the child threw that error, and with no scheduled identity bound the receipt recorder fenced it as "Scheduled generation is no longer current." Unscheduled contexts now return the declared config before the cleanup check, as pooled calls did before #16699. A scheduled context still denies after cleanup (scopes are never cleared), quiesce stays unconditional, and request-scoped servers still refuse a closed context at connection checkout. * 🧪 test: Cover Pooled Connection Creation After Request Cleanup The pooled-call integration test made its first call before cleanup, so it only reached the resolver on connection reuse (MCPManager.callTool). A detached child's first call to a server its parent never used, or a call after idle eviction, creates the pooled connection instead (UserConnectionManager -> MCPConnectionFactory). The new case cleans the context first and then makes the first pooled call, so a cleanup check added at either creation site now fails CI. The resolver comment now says why the early return is safe for every unscheduled context: request-scoped checkout refuses a closed context. |
||
|
|
db225c0919
|
🔐 feat: Admin Config Secrets v2 (MCP and Azure Credentials, Re-entry, Env Allowlist, Rebinding) (#17353)
* 🔐 feat: Admin Config Secrets v2 (MCP and Azure Credentials, Re-entry, Env Allowlist, Rebinding) The admin config API stored MCP server credentials and Azure group keys in plaintext, returned them to config readers, let an admin who cannot read a stored credential send it to a host of their choosing, and let override values reference any environment variable. MCP credentials and Azure keys, in `groups` and in the `groupMap` the Azure clients read, are now encrypted with masked previews, hidden from every read and decrypted once after the merge. A write that keeps a stored MCP credential or web search key while changing where it is sent must enter it again, judged by the value the written document holds. `${VAR}` references in overrides are limited to an allowlist, named where a change batch leaves them, and the merge drops a credential a layer moves to a new destination without entering it, listing it in write responses only for sections the caller can read. A custom endpoint's key is bound to its base URL, its provider and the destination options in addParams and dropParams (such as the anthropicApiUrl the native Anthropic client sends to), matched across layers under the name the merge folds (ollama is Ollama). A restated `${VAR}` counts as entered, so allowlisted names are not bound to a destination. Azure's `${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` stay templates in a group's `baseURL`, and a scheduled job's MCP consent no longer depends on the masked previews that change when an admin rotates the key in place. * ♻️ refactor: Move the MCP Key Re-entry Guard Out of the Binding Helpers to Break an Import Cycle With #17351's OpenID provisioning on dev, the app config service reached itself through the rebinding guard: app/service -> app/overrides -> app/rebinding -> mcp/registry/binding -> mcp/errors (and mcp/types) -> mcp/oauth/obo -> utils/identity -> auth/openid -> auth/provision -> app/resolve -> app/service. The binding helpers are pure diffs; only requireApiKeyReentryForRebinding needs the MCP error class, so it moves to mcp/registry/reentry.ts, and the helpers take MCPOptions from librechat-data-provider as mcp/headers.ts does. No behaviour changes. * 🧪 test: Drop a Base-Held Endpoint Key That a Lower-Priority Profile Moves Base merges first since #17352, so a profile below Base's priority still lands over it. The rebinding guard drops the key Base stores for a custom endpoint whose baseURL the profile changes, and the PATCH response lists it in `ignored`. * 🔐 fix: Serve Admin Azure Group Keys and Keep Deployment Keys at Their Own Destinations Azure groups a profile writes now take effect: the merge resolves a layer's `endpoints.azureOpenAI.groups` as AppService resolves librechat.yaml's (`resolveAzureGroups`, shared with `azureConfigSetup`) into the `groupMap`, `modelGroupMap`, model list and assistant models the Azure clients read. A layer's groups replace the groups below whole, as its array does; groups that do not resolve stay unread with a warning. The rebinding guard judges the resolved groups like any `groupMap` entry, and an Azure group's destination now also covers a deployment name its base URL takes its host from and the destination options of its `addParams`/`dropParams`. A write that keeps a stored Azure group key (in `groups` or `groupMap`) while changing where the group sends it answers 400 `CONFIG_SECRET_REENTRY_REQUIRED` on PUT, PATCH, field deletes, tombstones and change batches, judged as the document resolves over librechat.yaml and on each `groups` item as written. OCR fills an empty key from `OCR_API_KEY` only for its default destination (no base URL, or `${OCR_BASEURL}`), so a key the guard dropped for an override's `${VAR}` base URL is not replaced by the deployment's. The env reference policy includes the `${OCR_API_KEY}` loading fills in for an `ocr` section without a key, so a client can write back what GET `/base` returned. The custom endpoint guard no longer needs the layer's overrides to name an entry: an entry changed some other way is judged as inherited (positional tombstones inside the list are not applied by the merge, so none reached one). A web search URL a layer points at a `${VAR}` other than librechat.yaml's without entering its key is set back: web search sends the deployment's key, or its client's environment default, to any `${VAR}` URL. * 🔐 fix: Keep Azure Assistants Served, Opaque Keys as Entered and Batch Env Checks on the End State Azure assistants: a profile's `groups` replace the groups below whole, so groups with no model that supports assistants under librechat.yaml's `assistants: true` left `assistants` on with an empty `assistantModels`, and the Azure Assistants initializer picked `assistantModels[0]` (undefined). `azureConfigSetup` refuses that combination at startup; the merge now turns `assistants` off when none of the models it leaves supports them, with a warning, judged on the merged result so a later layer whose groups serve assistants keeps them on. The check is `lacksAzureAssistantModels`, shared with `azureConfigSetup`. Opaque credentials: every runtime reader of the OCR, speech, web search and Langfuse keys uses the decrypted value as is (`resolveConfigSecret`, `resolveStoredConfigSecret`, `decryptConfigSecret`), so a `${...}` inside a key the write encrypts is part of the key. Those fields are marked `opaque`: the write-time check skips a value the write encrypts there, and the runtime pass keeps their ciphertext instead of blanking it. A whole `${VAR}` reference there, which the field passes through and its reader resolves, is still checked, and so is stored plaintext. Custom endpoint (`initializeCustom`), Azure group (`mapModelToAzureConfig`) and MCP (`processMCPEnv`) credentials are expanded by their clients once decrypted and stay checked; the Assistants keys have no reader yet and stay checked too. Change batches: env references are judged on the document the batch leaves, as its other issues are. An issue counts only when the end state still holds a disallowed reference (ciphertext by its plaintext) where the batch sent one, at a location the batch changed, so a value a later op removes or replaces (`set ocr.baseURL`, then `unset ocr`) is no longer refused, and a stored reference the batch puts back where it was is left to merge time. * 🔐 fix: Proxy-Bound OAuth Secrets, Consistent Azure Group State, Batch Issues by Entry Key and Loaded Web Search References - The MCP proxy is part of the OAuth client secret's binding (user tier, admin re-entry and the merge's rebinding guard share `getChangedOAuthSecretBindingFields`): the access token the secret obtains travels through it. - A tombstone of `endpoints.azureOpenAI.groups` resolves the groups to none in the merge (empty `groupMap`, `modelGroupMap` and model lists, so the model list does not fall back to the environment's defaults), and removes what they resolve to in `applyConfigTombstones`. - The merged Azure model lists name only models a merged group serves (`narrowAzureServedModels`), so an override's `assistantModels` no group serves no longer keeps Azure assistants on. - `/changes` env issues are held by entry identity (`toConfigIdentityPath`), followed through renames and resolved on the batch's end state, so a removal or rename no longer shifts them. - References loading librechat.yaml introduces (OCR's `${OCR_API_KEY}`, the web search defaults) are allowed at the location they are loaded at, so GET /base's sections are accepted written back, and nowhere else. * 🔐 fix: Re-entry for Endpoint, OCR and Speech Keys, Own Keys Bound Over Base and User Key Destinations Kept - The document that stores a custom endpoint, OCR or speech key can no longer move its destination without the key: PUT, PATCH, field deletes, tombstones and change batches answer CONFIG_SECRET_REENTRY_REQUIRED at the changed field (getBoundKeyReentryIssues), judged by value as the document alone sends it. - A profile's own key for an endpoint, MCP server, OCR or speech service, or Azure groupMap group that Base defines is bound to that destination when it is entered: the rebinding guard reports a fingerprint of where it goes, the write stores it beside the key (<secret>Binding), and the guard keeps the key there until a lower layer moves it. Reads and the merged config leave the fingerprint out; kept keys carry it; client values are replaced. - An inherited custom endpoint or MCP server that sends a credential each user holds (user_provided, customUserVars, apiKey.source user, OpenID or Graph token placeholders) keeps its destination: a layer's move is set back and listed in ignored. |
||
|
|
8c51a3f9f8
|
🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields (#17327)
* 🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields Config profiles are the `__base__`, role, group and user documents written through `/api/admin/config`. A profile can tombstone `mcpServers.<name>`, and the merge then removes that server from the principal's resolved `mcpConfig`. * 🛡️ fix: Apply MCP Hiding to Server Instructions, Failed Removals and the Startup Audit - Server instructions bound for the model now leave out librechat.yaml servers hidden from the acting user, including one a model spec pins. `applyContextToAgent` passes the request user through `formatInstructionsForContext`, which drops the user's hidden set from the user-less catalog it reads. - A config-tier candidate that removes a librechat.yaml value (a field, or a key of a record such as `headers`) and fails inspection now stands as a failed config-source entry instead of falling back to the librechat.yaml base that still carries the removed value. A failed change that removes nothing still falls back, and an entry with no transport (fields written over a hidden server) keeps the documented librechat.yaml fallback. - The startup tombstone report reads every tenant's profiles in the system tenant scope, so it also runs from entry points without an outer system scope under strict tenant isolation, and names each profile's tenant. * 🗂️ fix: Project the Startup MCP Tombstone Report Query and Run It Alongside Connections * 🔑 fix: Keep Removed OAuth Headers Out of the MCP OAuth Callback * 📝 docs: Note That Removed OAuth Headers Stay Out of MCP Sign-In * 🔐 fix: Recheck the MCP OAuth Flow's Server Before Initiation Redirects OAuth initiation redirected to a flow's stored authorization URL, or started a new authorization from the flow's stored server URL, without resolving the server again, so a principal whose profile hid the server after the flow was created could still be sent to its authorization endpoint. The callback's active-server check and its OAuth header fallback lived in the CJS route. resolveMCPOAuthTarget (packages/api/src/mcp/oauth/target.ts) now owns both: it resolves the server in the principal's current config with a strict read, refuses a flow admitted for another server, a server the principal can no longer use, or a redefined one, and selects the headers the flow sends (the initiation snapshot, else the principal's config). Initiation calls it before either redirect (404 or 400); the callback calls it before the exchange and from the token-store fence. * 📝 docs: Note That a Hidden MCP Server Ends OAuth Sign-Ins Already Started * 🚩 feat: Publish the yamlMcpHide Admin Config Meta Flag Profiles can hide librechat.yaml MCP servers and a hidden server is an access boundary, so GET /api/admin/config/meta now reports features.yamlMcpHide for the admin panel to gate its hide UI on. The meta.spec probe checks the behaviour itself: the admin API stores an mcpServers.<name> tombstone, the merged config drops the server, the registry withholds it from principal lookups and checkout but not from app-level lookups, process-backed servers stay, and an unreadable principal config withholds every hideable server. * 🧪 test: Align MCP Tombstone Specs With the Portable Config Store The admin config handlers now write whole documents through writeConfigDocument, so the end-to-end tombstone spec wires it in place of the removed upsertConfig/patchConfigFields deps. A reset of a whole MCP record now also prunes the parents it leaves empty. |
||
|
|
07299822e5
|
🧪 refactor: Move Test-Only Helpers Out of Production Modules (#17139)
* refactor: Drop Queue Reset Helper From The Client Queue Module * refactor: Drop Unused Shortcut Getters And Subagent Batch Wrappers * refactor: Move HITL Approval Payload Builder To Test Fixtures And Drop Dead Helpers |
||
|
|
ecd6943194
|
✂️ refactor: Remove Dead Helpers, Classes and Dependencies From packages/api (#17118)
* refactor: Remove Unused Agent Helpers and Superseded Wrappers * refactor: Remove Dead Auth, Middleware, Protection and Prompt Helpers * refactor: Remove Dead MCP, File, Skill and Endpoint Code * refactor: Remove Dead Stream, Telemetry, Utility and Type Exports * chore: Drop Unused packages/api Dependencies From bun.lock * refactor: Keep Published Compatibility and Deprecated packages/api Exports |
||
|
|
e2b2839375
|
🗃️ refactor: Remove Dead Data Schemas Methods and Data Provider Exports (#17113) | ||
|
|
0fea0240d6
|
⏳ feat: Reset Balance Allowances Without Rollover (#16781)
* feat: add periodic balance reset without rollover * fix: synchronize balance policy before periodic reset * style: preserve context balance class order * fix: Synchronize Reset Policy Before Granting Allowances * fix: Apply Current Balance Policy During Admission |
||
|
|
1418d44ad9
|
🕊️ feat: Supply Authorized Scheduled MCP Resource Bearers (#16699)
* 🕊️ feat: Supply Authorized Scheduled MCP Resource Bearers * 🥟 fix: Preserve Scheduled Bearer Denials and Request Headers * 🪼 fix: Retain Bearer Denials Across Scheduled Recovery * 🐡 fix: Retain Unsettled Scheduled Generation Evidence * 🧭 fix: Recover Retained Scheduled Generation Obligations * 🪢 fix: Drive Retained Recovery in Every Scheduler Mode * 🦔 fix: Fence Complete Scheduled Denial Receipts * 🪵 fix: Require Durable Scheduled Denial Acknowledgement * 🕯️ fix: Preserve Fresh Terminal Writers During Cleanup * 🪷 fix: Preserve Durable Scheduled Receipt Acknowledgements * 🧿 fix: Acknowledge Scheduled Evidence Before Erasure * 🧪 test: Verify Merged Scheduled Receipt Union in Redis * 🪡 fix: Deduplicate Merged MCP Agent Context * 🪷 fix: Reauthorize Scheduled MCP Transport Requests * 🛟 fix: Preserve Scheduled Catalog Authority Denials * 🛟 fix: Fence Scheduled Dispatch and Stale Owner Settlement * 🛟 fix: Gate Scheduled Cleanup on Owner Evidence * 🛟 fix: Preserve Scheduled Drain and Transport Rejection Evidence * 🛟 fix: Quiesce Scheduled MCP Before Occurrence Settlement * 🛟 fix: Fence Scheduled Owner Recovery and Back Off Receipt Admission * 🛟 fix: Index Initially Retained Scheduled Generations * 🛟 fix: Require Durable Admission for Scheduled Policy Denials * 🛟 fix: Abort Occurrence Bearer Work Before Quiescence * 🛟 fix: Cancel Connection Credential Waiters During Failed Initialization * 🛟 fix: Fence Scheduled HTTP Dispatch at the Cancellation Cutoff * 🛟 fix: Bind SDK Request Deadlines to Scheduled Denial Admission * 🛟 fix: Recover Retained Creation and Isolate SDK Caller Cutoffs * 🛟 fix: Fence Scheduled App Reads and Share Cold Connection Ownership * 🛟 fix: Retain Observed Scheduled Denials Across Caller Cancellation * 🛟 fix: Invalidate Retained Discovery on Each Re-Arm * 🛂 fix: Classify Scheduled Resource 403 as Permission Denial --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
38a9dab3f5
|
🦫 feat: Enforce Scheduled MCP Read-Only Authority (#16681)
* feat: enforce scheduled MCP read-only authority * fix: preserve startup admission and enforce consent deadline * fix: fail closed on malformed scheduled consent * fix: preserve scheduled MCP authority failure receipts * fix: preserve scheduled MCP handoffs and manual runs * fix: persist mandatory scheduled tool policy denials * fix: fence late scheduled consent and retain denial recovery * fix: retain scheduled transport denials and prevent replay * fix: merge retained scheduled denials during settlement * fix: prevent enrolled schedule completion wakeup escapes * fix: close scheduled consent admission and receipt races * fix: retain scheduled completion authority across approval * fix: retain safe scheduled authorization dependency denials * fix: sanitize scheduled authorization preparation failures * fix: bind schedule completion authority to task origin --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
3f3de65612
|
🪔 feat: Add Bound Scheduled MCP Consent (#16652)
* test: reuse reviewed scheduled MCP authorization fixtures * feat: add bound scheduled MCP consent lifecycle * fix: preserve scheduled root agent capability access * refactor: share scheduled MCP requirement resolution * fix: bind scheduled MCP consent to connection routing * test: type scheduled MCP connection identity fixtures * fix: reject unbound scheduled MCP routing templates --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
76fac09531
|
🦭 test: Add Scheduled MCP Authorization Contract Fixtures (#16645)
* docs: define scheduled MCP authorization contracts * test: keep scheduled MCP contracts beside executable fixtures --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
6658e3b314
|
📴 fix: Surface Missing Scheduled OBO Authorization (#16416)
* 📴 fix: Surface Missing Scheduled OBO Authorization * 🛟 fix: Preserve Scheduled OBO Failures Across Clients and Resumes * 🧊 fix: Skip Scheduling Setup for Unrelated MCP Errors * 🧭 fix: Keep Scheduled MCP Auth Evidence Through Recovery * 🎨 style: Format Scheduled MCP Receipt Checks --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
f13b0eaef4
|
🚚 chore: Point Repository and Image References at LibreChat-AI (#16005)
Some checks failed
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
Update package repository and bugs URLs, GitHub links in docs, templates and comments, and image references (compose files, deployed-update, Helm values and renovate hints) from danny-avila to LibreChat-AI / librechat-ai. Covers the rag-api repository and its images, which moved to the organization. Sponsor links and the sealed star-history chart are unchanged. |
||
|
|
a26a6709ff
|
📪 feat: Send Per-Request MCP Headers Without Hiding the Tool Catalog (#15988)
* 📪 feat: Send Per-Request MCP Headers Without Hiding the Tool Catalog
A `{{LIBRECHAT_BODY_*}}` placeholder in an MCP server's `headers` marks the
server request-scoped, and catalog discovery has no conversation to resolve it
against, so LibreChat skipped tool discovery and the server appeared with no
individual tools.
Operators can now declare those headers separately in `requestHeaders`, which
discovery omits and chat turns resolve and merge over `headers`.
* fix: normalize requestHeaders at each resolution entry point
Registering the new field in every consumer that enumerates header-bearing
config fields left five gaps: startup inspection, case-insensitive override,
the admin-configurable comparison surface, Graph token preprocessing, and
direct-bearer recovery.
Fold `requestHeaders` into `headers` at the entry of each resolution pipeline
instead, so Graph preprocessing, direct-bearer detection, `processMCPEnv` and
the transports keep reading one header map and never learn a second exists.
Catalog paths strip it at the same boundary, startup inspection included.
* fix: keep servers with chat-only headers off the shared app connection
A static `requestHeaders` map left a server eligible for app-level sharing, and
`ConnectionsRepository.loadConnection` handed the raw config to the factory, so
the shared session's own `initialize` and `tools/list` carried headers
documented as chat-only — and every later catalog read reused that session.
One session cannot serve both sides of the field. Exclude such configs from
`canUseAppConnection`, the single predicate both `connectAppServers` and the
repository's own app-level gate already consult, so the chat turn keeps its
headers on a user-scoped connection while catalog work never sees them.
* fix: normalize requestHeaders before the manager's bearer decisions
`createUserConnectionInternal` reads `usesDirectOpenIDBearerRecovery` and
`resolveDirectOpenIDBearerConfig` on the config it receives, while the merge sat
in `resolveRuntimeConfig`, a leaf helper below those calls. An `Authorization`
template declared in `requestHeaders` therefore never entered direct-bearer
mode: a `tools/list` 401 skipped the force-refresh and the connection was
returned with a stale bearer.
Normalize at the birth of that pipeline's config instead, ahead of all four
decisions, and make `getMCPAppToolsPublicationGeneration` normalize before
hashing. Connection paths hold a merged config while cache paths hold the
declared one, so without a canonical hash the two would address different
catalog slices.
* fix: Preserve MCP request header ownership across lifecycle boundaries
* fix: Narrow remote header identity before rebuilding config
* fix: Preserve request header precedence over catalog API keys
* docs: Synchronize generated agent API schemas
* fix: Ignore shadowed generated MCP user key requirements
---------
Co-authored-by: Lia <lia@librechat.ai>
Co-authored-by: Danny Avila <danny@librechat.ai>
|
||
|
|
21a9edbe74
|
🪪 feat: Pass Schedule Identity to MCP Token Providers (#15962)
* feat: carry schedule context into MCP token providers * fix: restore scheduled token context after approval pauses * fix: make scheduled identity boundaries explicit * test: keep owner-drain leases live through CI delays |
||
|
|
345cf8877f
|
🍵 fix: Support Renewable MCP Credentials for Schedules (#15908)
* fix(schedules): forward renewable MCP credentials * fix(schedules): require unattended MCP credentials * fix(schedules): resolve MCP credentials during execution * refactor(schedules): inject unattended credentials * fix(schedules): scope renewable credential resolution * fix: isolate lazy scheduled OBO credentials across MCP lifecycle * fix: preserve MCP cancellation and retryable credential lookup * fix: bind MCP cancellation to the owning run signal * test: supply flow manager in OBO cancellation regression * fix: preserve MCP cancellation across exchange and API ingress * style: sort MCP manager imports * fix: propagate cancellation through agent discovery * test: preserve agent discovery edge literals * fix: propagate agent initialization cancellation * fix: expose initialization signal in OpenAI adapter |
||
|
|
696bcb6c1e
|
🏦 fix: Reserve Credits for In-Flight Requests (#15879)
* 💳 fix: Reserve Balance for In-Flight Requests Admit each balance-checked request against the credits that other in-flight requests have not already reserved, and hold its prompt cost on the balance record until the turn settles. Apply a due auto-refill in the same fenced write so one refill window credits exactly once. - data-schemas: reserveBalance / releaseBalanceReservation replace createAutoRefillTransaction; the write is a compare-and-swap on tokenCredits, reservations and lastRefill (no pipeline updates) - balance.reservations is select:false so existing reads never see it - checkBalance returns a reservation handle; BaseClient and the assistants controllers release it once the turn settles - balance.reservationTtlMs bounds how long an unreleased reservation counts (default 30 minutes) * 🧮 refactor: Admit Reservations Without Contention and Record Refills Durably - admission is a conditional $push/$inc against a reservedCredits total, so concurrent admissions of a funded balance commit on their first write - expired reservations are pruned per element with their credits - auto-refill is fenced on every setting its decision reads and leaves a pendingRefill ledger marker in the same write; any later read records the transaction under the fixed id and clears the marker - a missing record is created with $setOnInsert inside reserveBalance - reservation lifecycle moves to packages/api (createBalanceReservations, withBalanceReservations); BaseClient releases once usage is recorded * 🪪 fix: Resolve Each User to One Balance Record - balance records a creator inserts are keyed by the user id, so creators racing on a missing record converge on one document; every balance read and write resolves a user to their oldest record, so existing duplicates stay inert - upsertBalanceFields takes insert-only fields; the balance-config middleware, schedules and lazy initialization pass the starting credit through them so it never overwrites a record another writer created - the auto-refill write is also fenced on reservedCredits - the scheduled-chat balance pre-skip counts reserved credits as unavailable * ⏳ fix: Count Only Unexpired Reservations in Scheduled Balance Checks findBalanceByUser with includeReservedCredits now totals the reservations that have not expired, so a reservation left by a crashed request stops pre-skipping scheduled chats at its expiry instead of when a later reservation write prunes it. * 🔁 fix: Renew Balance Reservations While Their Request Runs A reservation is renewed every half TTL until it is released, so only a reservation whose process stopped expires. Pruning removes a reservation only while it is still expired at the write, so one renewed after the pruning read survives. * 🛟 fix: Keep Reservations Alive Through Renewal Failures and Lost Acks - renewal reschedules itself: every half TTL, capped to the timer range, and a failed renewal retries within a tenth of that window - an admission write that errors removes its possibly committed reservation before rethrowing * 🛑 fix: Hold a Stopped Turn's Reservation Until Its Charge Lands - an aborted turn lets its reservations lapse after a minute (or the TTL) instead of releasing them, so the Stop request's charge lands before the credits are admitted again - balance.reservationTtlMs has a 10 second minimum, enforced in the schema and at runtime, so renewal stays bounded * ↩️ revert: Release a Stopped Turn's Reservation When the Turn Settles An aborted turn releases its reservations when it settles, the same as every other exit, instead of holding them for a fixed lapse window that refused funded requests after every Stop. The minimum reservation TTL stays. * ⏸️ fix: Hold Assistants Reservations Through Background Run Retries An in-progress Assistants run continues in the background after the handler responds; the turn's reservations now stay held until that run settles (createBalanceReservations().holdUntil). * 🔁 fix: Apply an Auto-Refill Once per Admission A refill interval that is due again the moment it is applied, such as a zero interval, let one admission refill repeatedly until the retry bound. Each admission now applies at most one refill, as the previous check did. * 🧹 fix: Prune Expired Reservations in One Write An admission that found expired reservations issued one write per hold, so a backlog left by a stopped process fanned out into as many concurrent writes. The prune is now a single write fenced on every hold it removes; a hold renewed after the read makes the write miss and the admission re-read. * ⏱️ fix: Renew Reservations From Their Stored Expiry The next renewal was scheduled half a TTL after the previous write settled, and the first one half a TTL after the admission returned, so a slow write could let a live hold expire before its renewal. Renewals are now scheduled from the expiry the last write stored. Expired holds are pruned as one bulk write of per-hold fenced updates, so a hold that changed after the read is skipped while the rest are removed, and the admission no longer spends a retry on the whole prune. |
||
|
|
442d477acf
|
🛎️ fix: Preflight MCP Readiness for Scheduled Chats (#15782)
* fix: Validate Scheduled MCP Readiness 🕰️
* fix: Keep Schedule Permission Store Failures Retryable
* perf: Parallelize Scheduled MCP Preflight
* fix: Preserve Scheduled MCP Invariants
* fix: Close scheduled MCP recovery gaps
* fix: harden scheduled MCP recovery
* fix: bound scheduled MCP validation
* fix: harden scheduled MCP admission bounds
* fix: align scheduled MCP graph admission
* fix: bound scheduled MCP admission
* fix: align scheduled MCP preflight with runtime
* fix: persist scheduled MCP failure outcomes
* fix: match scheduled MCP runtime semantics
* fix: parallelize scheduled MCP admissions
* fix: mirror scheduled subagent traversal
* fix: enforce scheduled subagent config limits
* fix: bound scheduled MCP initialization
* fix: mirror lazy subagent metadata budgets
* fix: complete scheduled MCP recovery
* fix: attribute missing MCP tools
* fix: harden scheduled MCP admission
* fix: preserve scheduled admission invariants
* fix: retain inaccessible MCP requirements
* fix: make MCP admission recovery durable
|
||
|
|
cdc3339a78
|
🪬 fix: Fence Agent Checkpoint Ownership and Cleanup Recovery (#15680)
* 🔒 fix: Isolate Checkpoint Cleanup by Generation Ownership * 🔒 fix: Harden Checkpoint Cleanup Ownership * 🔒 fix: Retain Checkpoint Cleanup Evidence * 🔒 fix: Persist checkpoint cleanup receipts * 🔒 fix: Bound checkpoint cleanup receipt lifecycle * fix: Bind checkpoint cleanup to durable ownership and deletion intent * fix: Bind event actor checkpoint storage to authenticated owners * fix: Resolve exact actor checkpoint ownership before lifecycle cleanup * fix: Preserve actor storage compatibility with durable ownership and prune work * fix: Defer checkpoint erasure until conversation deletion commits * fix: Reclaim orphan actor ownership and batch deletion lookups * fix: bind actor checkpoint ownership to payload rows * fix: isolate actor readers and await account persistence * fix: gate owned suspensions and drain complete account state * fix: unify checkpoint deletion lifecycle and storage authority * fix: Replay checkpoint deletion from durable owner state * fix: retain checkpoint cleanup identity before topology loss * fix: complete checkpoint catalog capture and retirement lifecycle |
||
|
|
deda142d58
|
🗓️ fix: Refresh the Sidebar When a Scheduled Run Fires or Settles (#15631)
* 🗓️ fix: Refresh the Sidebar When a Scheduled Run Fires or Settles An automatic occurrence is the one generation nothing in the client starts, so no submission, stream or event handler tells the sidebar its chat exists, and the conversation list's five-minute staleness leaves it out of a tab that stays focused. The schedules panel is already polling for its cards, and `lastRun` moves at exactly the two moments the list should be re-read — a fire adds a chat, a settlement retitles and reorders it — so read the refresh out of that poll rather than adding one. Only transitions refresh; a schedule first seen is recorded in silence, so a panel opened long after a run does not refetch a list that has held its chat all along. This also settles the row Run Now seeds: the schedule's own completion is what replaces the schedule name with the generated title. * 🗓️ fix: Name the Running Occurrence's Chat So the Sidebar Can Fetch It Four review rounds each found a new way the schedule projection diverges from the run it stands in for: `lastRun` is not written at fire, `nextRunAt` moves before the conversation exists, and an edit mid-flight fences the terminal write so `lastRun` never changes at settlement. Nothing on a schedule document means "this run's chat is listable now", so every signal read from one was too early or too late. Stop inferring. The list handler already reads the user's schedules and limits in parallel; read their active run rows alongside — one user-scoped query, joined by schedule id — and project the running occurrence's conversation id as `activeRun`. It comes from the run's own row, so the configRevision fence on the schedule projection cannot hide it, and it is absent until dispatch has actually reserved a conversation. The panel hook then hands every id the list names to the same admission watch Run Now uses, which probes until the server has the chat and adds the server's own row: one mechanism for both paths, idempotent per id. A run leaving the list has settled, and the list is re-read once for the order its settlement changed. * 🗓️ fix: Fetch Every Chat the Schedule List Names, and Scope the Run Read Three from codex on the rebuilt design. A run short enough to start and settle between two polls is never seen running, and most scheduled chats are short. The list also names the chat the last occurrence produced, so the panel now fetches every chat the list names that it has not seen — active or settled — through the same admission watch. The first observation still fetches only runs in flight: a panel opened long after a run must not go and fetch history. Project every running occurrence rather than the latest. A `requires_action` pause does not block the next occurrence, so a schedule can hold two, and reducing to one hid the pause's settlement. This also deletes the picker. The single-schedule route's run read started before ownership was known and was scoped only by schedule id. The response was gated, but a speculative read must stay scoped to the caller: use the user-bound query the list uses and narrow it, rather than touch rows the caller may not own. * 🗓️ fix: Let One Layer Decide What to Watch, and Say Which Runs Are Generating Two from codex on the previous round. The admission watch forgets a run whose delivery outlasted its budget so a later announcement can retry, but the panel kept its own permanent record of every id it had handed over and never announced one twice — so that retry could never happen. Drop the panel's record: every chat the list names is handed to the watch on every observation, and the watch is the one place that decides whether an id is landed, in flight, or worth trying again. The one thing the panel still must not do is fetch history. Opening it used to start a watch per active occurrence, and a run parked on an approval can sit there indefinitely and accumulate, its chat listed long ago. Carry each occurrence's status on the wire, and on the first observation treat only a generating occurrence as in flight; a settled one and a paused one are history, and a pause resolving later still re-reads the list. * 🗓️ fix: Read Only Generating Runs for the List, and Hand Over Only Those Three from codex, and this round removes more than it adds. The list's run read returned every paused row, unbounded, on each poll. `ScheduleRun` is indexed by status rather than by user, `requires_action` rows accumulate for as long as their approvals wait, and `started` rows are bounded globally by the capacity slots — so the read now asks for `started` alone, through an optional status narrowing on the existing method. A pause resolving is still seen through `lastRun`; what this gives up is only the corner where that projection is fenced by a mid-flight edit. A run that failed during initialization settles with its reserved id but never writes a conversation, so its route answers 404 for good — and handing settled ids to the watch meant re-probing it every poll, forever. The watch only ever needed generating runs: a settled one is already covered by the list re-read its settlement triggers, which brings the server's own row. A deleted schedule leaves the list, and comparing only the schedules still present missed the abort-and-settle that deletion performs on its run. A schedule vanishing now counts as its state moving. * 🗓️ fix: Compare a New Schedule Against Idle, and Bound the Landed-Run Record Two from codex on the head GitHub replayed onto dev. A schedule created in another tab can arrive in this client's list with its first run already settled, and a comparison that walks only the schedules seen before never looked at it. Compare a newcomer against the idle state: one that arrives with a run behind it is news the list should hear, one that arrives with no run yet is not. The watch remembers every run it landed so a later announcement is a no-op, and a panel that stays mounted sees a new id per occurrence for as long as schedules keep firing. Keep the most recent 256: a run is announced only while in flight, and none is still in flight by the time that many have started after it, so the oldest can go without a watch ever running twice. * 🗓️ fix: Re-read on First Sight Only When a Run Outran the Sidebar, and Never Evict a Live Watch Two from codex. "The first observation is silent" assumed the sidebar's lists had been read after any run they should hold, and that is not guaranteed: open the panel after a run fired and finished with it closed, and the card says the run succeeded while the sidebar was read before it ever fired. The first observation has no earlier one to compare against, so compare against the sidebar instead — re-read only if some run fired after the lists were last read, which is precisely the case where they lack its chat, and stay silent otherwise. The landed-run cap assumed fewer than 256 generations could be in flight at once, and the config schema bounds neither maxPerUser nor fireConcurrency. Keep in-flight watches in their own set, bounded by real concurrency and never evicted — a second watch for a run still being watched is the one thing that set exists to prevent — and cap only the landed record, where a forgotten id costs at most one extra probe. * 🗓️ fix: Re-read on First Sight When Anything Has Run, Release Runs the List Drops, Observe Every Poll Three from codex, each a defect in an earlier fix, and each answered by removing something. Comparing a run's fire time against the list cache's `dataUpdatedAt` was measuring cache writes, not server reads: a foreground chat inserted after the run advanced the stamp without adding the scheduled chat. There is no timestamp for "last read from the server", so stop guessing — the first observation re-reads if any schedule has run at all, and one refetch on opening the panel is the honest price. A run can be landed and still generating for minutes, announced on every poll; a count cap evicted it and restarted its probes. The hook is the one place that sees an occurrence leave the list, so it releases the id then, and the watch remembers a run exactly as long as something can announce it. No cap. React Query hands back the same array for a deep-equal poll, so an effect keyed on the data never ran again while a run stayed in flight — and a watch that gave up was never asked to try again. Key on the poll's timestamp. * fix: Release Scheduled Run Retention Across Admission and Unmount 🧹 |
||
|
|
6360821470
|
🪢 feat: Coalesce Bound Agent Event Bursts (#15208)
* 🪢 feat: Coalesce Bound Agent Event Bursts
* fix: Restore API Package Build
* fix: Keep Trigger Batches Within One Window
* fix: intersect trigger batch windows
* fix: fence trigger batch lifecycle
* style: sort trigger service imports
* fix: make batch requeue crash-recoverable
* fix: fence concurrent batch requeues
|
||
|
|
44d97f859d
|
⏰ feat: Custom Cron Cadence for Scheduled Chats (#15084)
* feat: custom cron cadence for scheduled chats Scheduled chats could only be built from four fixed presets, each pinned to a single hour and minute, so anything outside that shape (twice a day, every 15 minutes, the 1st of the month) was not expressible. This adds a Custom cadence that takes a raw five-field cron expression. The cadence schema becomes a discriminated union on `frequency`. A cron row carries `expression` instead of the hour and minute it cannot represent, since there is no single hour for `0 9,17 * * 1-5`, and the Mongo schema requires each field only for the shape that has it: a blanket `required` would reject every cron write, and dropping it entirely would let a structured cadence silently fire at 00:00 with a missing hour. Five fields only. croner also reads a six-field form carrying seconds and a seven-field form that pins a year, and both are refused. Seconds would promise a precision the engine does not keep, since it polls on a thirty-second tick and offsets each schedule by up to two minutes of jitter. A pinned year makes a cadence that runs out, and every place that computes a next run reads "no next occurrence" as a cadence it cannot read. Compilation, validation, next-run previews and interval measurement live in packages/data-provider so the dialog and the engine share one parser and cannot drift. The dialog previews the next occurrences, enforces the admin interval floor and disables its own submit from the same functions the server validates with, so it cannot offer a Create the API answers 400 to. The interval floor now covers cron, and measures it twice, taking the smaller. The nominal gap is probed in UTC and discounted by the same worst-case DST allowance the structured branches carry, which keeps `0 9 * * *` reporting exactly what the Daily preset reports. Real elapsed time is then measured in the schedule's own zone across each of that zone's transitions, because spring-forward compresses a gap that straddles one: `0 0,12 * * *` in America/New_York is 11 hours that day, not 12, and a floor between the two would otherwise be bypassed. The floor ships with the schedules list so the dialog can mirror it rather than surfacing it as a 400 after submit. Radio gains a wrap variant, since five frequency segments no longer fit one row in a phone-width dialog and a translated label can push even a desktop one over. Its indicator follows the selection across rows; the single-row default is unchanged. * fix: mark the cron input invalid when the interval floor rejects it A floor-violating expression disabled Create and rendered the cadence message, but the input itself still said aria-invalid=false and its aria-describedby never reached that message, leaving a screen reader user with a disabled Create and no stated reason. |
||
|
|
8c14f03432
|
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects Adds an optional chat-project destination to a schedule, plus the operator config to require one — or to pin every scheduled run to a specific project. Feature: - `chatProjectId` on the schedule row, accepted on create/update, projected on the wire, and carried into the run's conversation through the durable trigger envelope's `run` context. - `interface.schedules.requireProject` refuses schedules that are not filed under a project; `interface.schedules.projectId` pins every run to one project and implies the requirement. - Dialog gains a project picker (required when configured, a read-only row when pinned); the card shows the destination and the new disabled reasons. Invariants: - ONE resolver (`resolveScheduleProjectId`) decides the destination for the write handler, the fire path, and the wire projection alike, and an operator pin OUTRANKS the stored id in all three. Tightening the config therefore redirects — or stops — existing schedules instead of grandfathering where their runs land. A pin implies `requireProject` for the same reason: without it, a row created before the pin would keep firing with no project at all. - Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write this handler accepts is one the next fire also accepts. Any edit leaving a schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips the requirement, or a schedule auto-disabled for `project_required` could never be turned off. - Fire-time enforcement auto-disables rather than filing runs loose, matching agent_deleted: new `project_required` (requirement raised after creation) and `project_deleted` (gone, or pinned to a project this owner does not have) reasons, both refused BEFORE a billed generation is dispatched, and both advancing so a schedule can never wedge on the occurrence. - `computeCreateDigest` appends the field only when present, so a payload without a project digests byte-identically to one from before this change — a create retried across the upgrade still matches its own row instead of reading as key reuse. - Project reads are scoped to the owner, so ownership and existence are the same lookup; a read ERROR propagates instead of failing closed, so a Mongo blip retries the fire rather than auto-disabling the schedule. The trigger idempotency key hashes principal/event/target and never `envelope.run`, so the added run field cannot destabilize delivery identity. * 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget The project picker landed as a new ROW in the schedule dialog, which broke the e2e edit spec: `md:overflow-visible` turns off the template's scrolling from `md` up, so the dialog's content must fit the viewport. The extra row pushed the footer's Save button below a 720x1280 window, where Playwright reported a visible, enabled button it could never click — 226 scroll-into-view retries and a 2-minute timeout, on all three attempts. Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720): dev 688 (32px slack) project row ~790 (off-screen, CI failure) 3-column row 704 (16px slack — half the budget spent) this commit 686 (34px slack, 2px better than dev) The identity row is now three columns — name, agent, project — and its caption moved out of the agent cell to sit full width beneath the row: at a third of the dialog that sentence wraps an extra line, and the row is the tallest thing competing for the budget. The caption is grouped with the row rather than left to the form's own 4-unit rhythm, which spent more height on the gap than the caption occupies. The e2e spec now asserts the button is in the viewport before clicking it, so the next field that overflows this dialog says so in one line instead of a two-minute timeout on a visible element. FOLLOWUPS.md records what the planned dialog controls (multi-day weekly, timezone, attachments) need first: give `ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the popovers into the dialog content, and let the form scroll again. * 🧹 fix: Address Codex Review on Scheduled Chat Project Scope Four P2 findings, all real. Unreachable clearing path (ScheduleDialog). The picker only held live projects, so `com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a schedule had a project the owner could never take it away, leaving the server's `chatProjectId: null` path reachable only by API. The picker now carries a real "No project" option whenever a project is optional, and omits it when one is required, where there is nothing valid to select. Placeholder shown for a real project (ScheduleDialog). A stored or pinned project outside the first loaded page had no name in the paged map, and the combobox renders its placeholder for an empty display value — telling the owner a scoped schedule had no project. That one project is now read by id, with the raw id as a last resort: a poor label, but an honest one. Project policy skipped at the resume boundary (service.ts). `claimScheduleResume` re-applied the schedules gate, the revision fence, the kill switch and SCHEDULES:USE, but not the project policy this PR added. Approving a paused run whose project was deleted — or whose owner now sits under a requirement or a pin it no longer satisfies — billed a continuation the very next scheduled fire would refuse and auto-disable the schedule for. The effective-project resolution now runs there too, refused before the lease and the capacity slot so a policy refusal costs nothing and leaves no state to unwind. NOTE: agent access and balance are still not rechecked on resume; that gap predates this PR and is left alone. Per-card project derivation (ScheduleCard). Every card ran the projects hook and rebuilt the full option array, name map, and one icon element per project, to use a single name — O(schedules x projects) per render and per project-list refresh. The hook is split: `useChatProjectNames` (map only, for the panel, which resolves every card's name once and passes it down) and `useChatProjectPicker` (options and pagination, for the dialog's one combobox). The panel skips the query entirely until some schedule actually has a scope. Tests: four at the resume boundary (verified failing without the gate) and four in the dialog spec. The picker selection in one existing test now goes through the search field — the popover's VIRTUALIZED renderer sizes its window from a scroll height jsdom always reports as 0, so with three options it materialized only two. Full schedules e2e re-run green against the rebuilt client. * 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent Second Codex round, four P2s. Three were consequences of the resume gate added in the previous commit, which was half-built: it admitted where it should not and stranded the run where it refused. Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy` branch. Both entry points consult that branch FIRST, and both already route its refusal through abort-and-settle — so a policy stop now settles the occurrence instead of answering a bare 409 while the job stays `requires_action`, the card keeps reading "Needs approval", and every retry repeats the same 409 until expiry. No change to resume.js: the existing branch does the work. The rule is deliberately NARROW. It refuses only where no valid destination is left — the requirement is on with nothing satisfying it, or the schedule's own project is gone (which also unset it on the conversation). It does NOT refuse because an operator's pin moved: the paused conversation cannot be rebound (`chatProjectId` is excluded from the resume context and the continuation reuses the same conversationId), so refusing would strand a pending approval over a destination it can never reach, for a pin that governs only where the NEXT run lands — which the fire path already redirects. Create retries are idempotent again. Project policy had been applied BEFORE the `clientRequestId` replay lookup, so a raised requirement, a deleted project, or a moved pin could answer 400 for a create that already committed — pushing the client to rotate its key and create a DUPLICATE schedule, the exact failure the key exists to prevent. Policy now applies only to a genuinely new insert, and the digest is computed from the CLIENT's payload rather than the resolved destination, so today's policy can no longer re-digest a genuine retry into a mismatch. An explicit `chatProjectId: null` under a pin is refused rather than silently resolved to the pin. The payload contract defines `null` as clearing the scope; answering 201 while filing under the pin reported success for the opposite of what was asked. Only an OMITTED field takes the pin silently. Tests: five on the policy branch (both refusals verified failing without it, plus guards that a moved pin and a live project still admit) and two on the handlers (the pinned explicit clear, and a committed create recovered by retry after the policy tightened — verified answering 400 without the reordering). * 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved Third Codex round, four P2s. The root confusion behind the resume findings: an operator pin outranks the stored id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row keeps its old value. The row therefore LIED about where that occurrence's conversation went, and every later re-validation — the resume boundary above all — checked a project the conversation was never filed under. A schedule storing A, pinned to B, with B later deleted and A still live, was admitted for resume into a conversation that had just been unscoped. Fixed at the source rather than at each reader: a fire that resolves a destination different from the stored one writes it back, claim-token fenced like every other worker-side write and deliberately WITHOUT a configRevision bump — this is the server reconciling itself to policy, not an owner edit, and a bump would fence an in-flight occurrence off its own run. Written only AFTER the destination validates, so an unusable pin never lands in the row, and best-effort: the envelope already carries the right destination, so a failed write costs accuracy on a later recheck, never the run itself. The wire projection already reported the pin, so this also stops the row and the UI disagreeing. An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit path too. The pin check and the requirement are independent rules, and folding them together let `{enabled: false, chatProjectId: null}` skip the pin check entirely, unset the row, and answer with a wire projection still naming the pin. Only the requirement is waived for a disabling edit. The dialog no longer requires a project for an edit that leaves a schedule DISABLED. The server waives the requirement there precisely so a row auto-disabled for `project_required` can still be renamed or tidied up; requiring it in the form made that unreachable, and an owner with no projects could not edit the stopped schedule at all. FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second deletion race inside the resume claim window (which needs the effective project persisted per OCCURRENCE plus a distinct policy conflict routed through abort-and-settle), and the fact that convergence happens only when a schedule fires. Tests: four on convergence (pin written, no write when unchanged, no write for a destination that failed validation, fire survives a failed write), two on the disabling-edit pin rules, two in the dialog. Schedules e2e re-run green. * 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest `computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and an explicit `null` produced the same digest. Because the replay lookup and `matchesCreateIntent` deliberately run before project policy, a request could reuse a pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and receive 201 describing the pinned row — success reported for the opposite of what it asked, and the pinned-clear refusal the normal create path applies never reached. Now `!== undefined`: an omitted field still digests byte-identically to a payload from before project scope existed, so a create in flight across the upgrade still matches its own row, while an explicit clear is a distinct intent and digests differently. A pre-scope client never sent the field at all, so nothing legacy can carry an explicit null. * 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used The schedule-wide convergence from the previous commit was not enough, and the reason is the single-active run index: it covers `status: 'started'` only, so a PAUSED run does not block the next occurrence. While run 1 sat paused in project A, a pin move plus one later fire rewrote the schedule row to B — and the resume policy then validated B while run 1's conversation was still filed under A. Delete A and the continuation was admitted into a conversation that had just been unscoped. That window lasts as long as the pause, not the sub-second race the previous commit documented. The reservation now records the destination THIS occurrence used, and `isScheduleLive` validates that record when given the occurrence's `scheduledFor` — which `resume.js` already reads two lines above the call. No new conflict type and no settlement-path surgery: the refusal rides the abort-and-settle branch that check already has. An ABSENT record falls back to the schedule-level resolution rather than reading as unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as they do today. Schedule-wide convergence stays: it keeps the row honest for the UI and for every check that has no occurrence in hand. FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the claim window, which needs a distinct policy conflict routed through abort-and-settle rather than the bare 409 an `inactive` conflict produces. Tests: three on occurrence-vs-row precedence (the decisive one verified failing without the lookup), two on what the reservation records, and the resume controller spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green. * 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One The occurrence fallback added in the previous commit conflated two different absences. A post-upgrade run that deliberately went unscoped omitted the field exactly like a row written before the field existed, so both took the fallback — and a paused unscoped run was then validated against the schedule's CURRENT project. Under a requirement or a pin added while it sat paused, that admitted a billed continuation into a conversation satisfying no present policy. The reservation now ALWAYS records its decision, `null` for unscoped, and the read reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record — a pre-scope row, or no row at all — falls back to the schedule-level resolution; a recorded null is the genuinely unscoped occurrence it says it is, and is refused once a project becomes required. The distinction rests entirely on a stored `null` surviving as a present key while a never-written field stays absent, so that is asserted against real Mongo rather than assumed: if it ever stopped holding, unscoped runs would silently start being validated against the schedule's current project again. Tests: two against mongodb-memory-server (recorded null vs never-written vs missing row), plus refusal of a recorded-unscoped occurrence under a new requirement and the preserved fallback for a pre-scope one. Schedules e2e green. * 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence The initial-start policy check in `request.js` called `isScheduleLive(..., { policy: true })` without `scheduledFor`, so it fell back to the schedule-level resolution even though the run row already carries the occurrence's recorded scope. An occurrence reserved unscoped, with a pin introduced while its loopback request sat queued, was therefore admitted against the new pin — producing a billed unscoped conversation under a requirement it does not satisfy, from an envelope already built without a project. `scheduledFor` was already in scope there. Passing it makes the initial start and the resume validate the same way: against the destination the occurrence itself recorded. |
||
|
|
c7e355b219
|
🛑 fix: Confirm Scheduled Stops and Separate Terminal Scheduler Failure from Startup (#15051)
* 🛑 fix: Confirm Scheduled Stops and Separate Terminal Scheduler Failure from Startup Fixes #15042, fixes #15043. `resume.js` inferred a confirmed stop from the ABSENCE of `failureReason`, but `abortJob` had four `success: false` paths that returned no reason at all. Those settled the occurrence as `interrupted` and pruned the checkpoint on aborts that never landed — including one where a REPLACEMENT generation owned the conversation, which pruned the successor's checkpoint. Every `success: false` return now names itself (`job_not_found`, `already_settled` added alongside the existing `generation_replaced` / `job_still_active`), and a single canonical `isStopConfirmed` predicate decides whether durable state may be settled. `already_settled` confirms a stop — `awaitProviderDrain` has proven the provider segment can no longer persist — so a permanently terminal generation is not answered with a retry loop. Separately, a schedule engine that failed to arm advertised its permanent outage as a transient 503 with `Retry-After`, so a client obeying it would poll forever. Readiness is now tri-state (`starting` / `armed` / `unavailable`): the retry contract applies only while arming is genuinely pending, and a failed arm returns a terminal `SCHEDULES_UNAVAILABLE` with no `Retry-After` and an error-level log. * 🏷️ fix: Declare Schedule Write Gate Return Types `--isolatedDeclarations` requires an explicit return type on the exported factory and on the middleware it returns (TS9007). Adds a named `ScheduleWriteGate` type matching the existing `ShareMiddleware` shape. |
||
|
|
868cfbb343
|
🎡 fix: Scheduled Chat Slot Accounting and Reconciliation Rotation (#15040)
* fix(schedules): harden limits and reconciliation * docs(schedules): align reconciliation invariants * fix(schedules): preserve prompt editor contract |
||
|
|
c5276fc63d
|
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations
Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.
Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
and capacity slots, timezone-aware cadence with DST-conservative floors and
misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
(signed schedule-fire JWT claims, per-occurrence idempotency key inside the
route's clientRequestId charset), overlap/balance/capacity/duplicate skip
policies with auto-disable streaks (too_many_failures, insufficient_balance),
reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
path: HITL pauses surface on the card (requires_action), resumes re-apply the
fire boundary's admission policy (revision fence, enabled, global kill switch,
SCHEDULES:USE, availability) before continuing a billed generation.
Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
run's terminal outcome write; Stop/complete/pause race through single-winner
terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
completedAt-less evidence for scheduled fires plus an owner-intended outcome
stamp (scheduleOutcome) the reconciler prefers over re-derived success —
round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
error/skipped_balance instead of success on both initial and resumed paths;
stale stamps are refreshed evidence-first when persistence plus the Mongo
outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
from-status; republication escalates to the transport's acknowledged variant;
the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
of scheduled + interactive work with durable per-stream abort fences
(positive-evidence acknowledgement only), owner-side finalization markers for
post-terminal billed writes, deferred-deletion sweep (explicitly ensured
partial index) that completes cascades autonomously. Remote OpenAI-compatible/
Responses requests are documented as outside the quiesce and tracked in issue
14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
contract with coherent degradation (registration fails -> synchronous title
fallback; count reads 0).
* fix: fence terminal response persistence from deletion; retain stale-pause evidence
Two blockers from the third external review round.
Terminal persistence visible to account deletion:
- The finalization marker was registered only when post-terminal TITLE work was
possible, but every persistence-owning terminal CAS opens the same window: the
claim drops the job out of the active set BEFORE the response save (and the
background user-message/convo saves), so a deletion quiesce landing there saw
neither an active job nor a marker and could cascade while the admitted request
could still recreate messages. Both controllers now register the marker before
every persistence-owning claim — the fresh-turn path and the HITL resume — and
release it only after their pending saves (and any post-terminal title) have
landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
`terminalPersistencePending` is set: terminal at the CAS is not settled while
the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
refuses a store without it at configure time, keeping the failure loud and
deterministic) while remaining optional on the legacy public IJobStore type for
source compatibility. The silent degrade path from the previous round is gone —
it was not deletion-safe.
Stale-pause recovery retains scheduled evidence:
- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
and the InMemory/Redis stale-pause cleanups — unconditionally stamped
`completedAt`, putting a scheduled fire's failed-pause error terminal on the
short completed TTL. A Mongo outage longer than that TTL erased the evidence and
the reconciler recovered the run as `interrupted` instead of `error`. All three
now follow the controller-observed path from the previous round: scheduled jobs
omit `completedAt` (retained-evidence TTL) and stamp the error outcome.
The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).
* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths
One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.
- Generation-scoped markers. Entries were keyed (userId, streamId), so a
Stop-superseded generation finishing late could clear the marker its
replacement registered on the same conversation. Marker fields are now
qualified by the generation's createdAt; clears must present the same
identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
user's existing hash expires between the two commands (or the process dies
there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
the terminal CAS instead of proceeding uncovered: the completion claim throws
into the error path, the error path skips completeJob and leaves the job
ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
and abortJob returns a new retryable `fence_unavailable` failure the Stop
route answers with 503 and the deletion quiesce treats as an unacknowledged
stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
the abort CAS, clear in its finally — the Stop route's checkpoint prune and
partial save run inside beforePublish, between CAS and publication); the
interactive and background generation-error paths register before their
completeJob; the resume controller's error finalization registers before its
completeJob. A lost or thrown claim releases the marker after pending saves
flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
terminalPersistencePending for ALL statuses — including `aborted`, whose
route-side persistence the previous guard missed.
Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.
* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment
Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.
- Lease tokens. Marker fields were (streamId, createdAt), shared by every
contender on the same generation — completion and Stop, or two racing Stops —
so a losing contender's cleanup erased the winner's still-live marker. Every
registrant now carries a unique lease token in the field and may only ever
clear its own lease; unqualified legacy clears cannot touch qualified entries.
- Admission fence. Authentication can pass before the deletion barrier goes up,
and the durable createJob is several async steps later — a deletion quiesce in
that window saw neither an active job nor a marker and could cascade before
the admitted request created its job. The controller now registers an
admission lease and THEN rereads the deletion barrier: the ordering guarantees
either this request observes the barrier (403, lease released, slot/claim
cleanup) or the quiesce observes the lease and defers. Held until createJob is
durable; released on every refusal and initialization-error path. Fail closed
when the lease itself cannot be registered (503 retryable).
- Undelivered-Stop retention. abortJob released its lease in a finally even when
delivery AND publication had provably failed — the job reads terminal
(invisible to active-set scans), a user Stop writes no durable abort fence,
and the remote owner keeps generating and will persist its abort-catch writes
whenever the signal finally lands. The lease is now retained in exactly that
case, and each resignal attempt heartbeats a fresh lease so the fence outlives
the TTL for as long as delivery is still being driven. The abort-winning
turn's own loser-side pending saves are additionally fenced in the controller
catch (best-effort — those writes are already in flight).
- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
non-assistants endpoints) awaited abortJob and then spent usage and saved the
stopped response AFTER the abort's lease was released. Both writes now run
inside `beforePublish`, between the abort CAS and publication, covered by the
same lease as every other abort.
Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.
* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races
Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.
- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
-job handoff is only atomic against a reader in the OPPOSITE order of the
writer: writers hold the lease strictly until the job is active-set visible,
so leases-first shows every interleaving either the lease or the job.
Jobs-first allowed a request to create its job after the scan and release its
lease before the count — hiding both, cascading, and letting the new
generation persist into a deleted account.
- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
the moment the owner's AbortController tripped; the stopping side released its
lease on that ACK while the owner's asynchronous abort-catch persistence was
still ahead. The transport now awaits a manager-installed pre-ACK hook that
registers a DETERMINISTIC owner lease (exactly one owner exists per
generation, and determinism is what lets the signal-time registrant and the
owner's catch-side release agree across processes) before the acknowledgement
is persisted or published; an owned same-replica abort bridges to the same
lease before tripping its local controller. Both generation-owner catches
(fresh turn, resume) release it once their writes land.
- A failed replacement handoff no longer orphans the predecessor. The atomic
replacement removes it from active storage, and an unconfirmed handoff
terminalizes the replacement too — leaving nothing a quiesce could discover
while the predecessor's provider may still be generating. Its owner lease is
now retained at the point the receipt fails delivery; the owner replica renews
it through the pre-ACK fence when the signal finally lands.
- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
holder; live persistence — a stalled save, a long deferred title — must never
outlive its own fence. holdUserFinalization registers and renews every minute
until released; the controllers' completion/error/admission leases all hold.
The undelivered-Stop retention moved to a deterministic `stop` lease that
every resignal attempt renews and the first successful one clears (no more
opaque leases accumulating to TTL), and a THROWN abort transition releases the
contender lease instead of leaking it.
- The user-document abort-fence mutations now invalidate the auth user-doc
cache, matching every other user-doc write.
Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.
* fix: one manager-owned owner-lease span across every abort delivery path
Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.
- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
the transport ACKed even when it failed; a same-replica owned abort likewise
swallowed registration failure. The hook now acquires a HELD owner lease
(heartbeat until the owner's catch releases it via releaseOwnerLease) and a
rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
behind its retention lease, and every resignal re-drives the handler. The hook
also stopped gating on `job.createdAt === generationId`: during a replacement
handoff the store holds the replacement while the abort targets the
predecessor, and that gate silently skipped exactly the generation being
acknowledged (the store job is owner identity, never a generation gate). A
local owned abort acquires the same held lease before tripping its provider;
post-CAS the trip cannot be withheld, so acquisition failure downgrades
delivery and the retention handoff keeps the user fenced.
- Committed-but-lost-reply disambiguation. A thrown abort transition released
the contender lease as if nothing had happened, but a Lua CAS can commit and
lose its reply — an aborted job invisible to active-set scans whose provider
was never signalled, with no fence left. The throw path now re-reads the exact
generation: only a job still live under the caller's identity proves no
commit; committed or ambiguous outcomes hand the fence to the deterministic
stop lease (kept on the contender lease if even that fails) before rethrowing.
- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
predecessor's held owner lease before the trip (failure reports the receipt
undelivered, engaging retention). Failed-handoff retention is no longer a
swallowed one-shot: it heartbeats with the durable acknowledgement proof as
its renewal predicate — acquisition failures keep retrying for as long as the
fence is needed, and the retainer stands down (without clearing the shared
field) once the remote owner ACKs and thereby holds its own lease.
- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
retained stop lease, and keeps the stop lease when that handoff fails.
Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.
* fix: finish scheduled chat lifecycle hardening
* fix: close scheduled chat review follow-ups
* fix: generation-fence abort recovery evidence
* test: wait for settled approval tool output
* test: preserve scheduled init reconciliation option
* refactor: rebuild scheduled chats on durable agent triggers
* test: reset MCP cache mock between cases
* test: isolate scheduler startup in server specs
* fix: harden scheduled run lifecycle
* fix: normalize schedule capacity conflicts
* test: type schedule collision fixture
* fix: re-fence scheduled resume and expiry
* fix: fence scheduled resume handoffs
* fix: release superseded manual schedule leases
* fix: release failed run-now claims
* fix: release superseded engine claims
* fix: repair schedule dialog interaction and rework its form
The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.
Alongside that:
- Extract the agent builder's instructions editor (special-variable menu plus
expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
schedule prompt. Insertions now route through `onChange`, so react-hook-form
sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
the selection with `bg-surface-hover` on an outline button whose hover is the
same token, so the selected option was indistinguishable from a hovered one;
`Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.
The new dialog spec fails when the portal fix is reverted.
* test: cover scheduled and subagent deletion drains
* refactor: own the form-control appearance in the client primitives
Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.
Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.
This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.
The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.
* fix: keep an explicit schedules disable from becoming an opt-in
`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.
Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.
Both regressions fail without the normalizer.
* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep
Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.
Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.
Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension
Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.
Replace the destructive marking with a REVERSIBLE, token-fenced suspension:
- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
cannot copy field values under DocumentDB), fenced per row so an already-
suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
re-arming only rows still carrying the exact attempt token and not independently
deleted — so an owner-deleted or newer-attempt-suspended schedule is never
resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
any failure that cancels the user-deletion fence restores the suspended rows. A
successful deletion hard-deletes them (and their snapshots) via the existing
cascade and never restores.
Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol
Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.
Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):
- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
acknowledges only after beforePublish persistence succeeds, and on a
persistence failure leaves the barrier unresolved so the run stays preserved
(client retries; stale-owner timeout is the bounded recovery). A failed abort
releases the stamp it placed so a replacement/retry is never blocked through
its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
proceeds immediately. The paused Stop settles only after its own ack.
Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation
Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.
Translate durable delivery state into the schedule outcome:
- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
(computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
admission is live, never orphan; dead → record `error` from the durable
lastError and release capacity promptly (no 30-minute wait), through the
ordinary outcome/auto-disable path; succeeded or no record → the existing
legacy orphan policy (interrupted only past the cutoff); a delivery lookup
failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
moderation middleware writes no schedule state.
Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions
Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.
Make the host lifecycle work durable rather than schedule-specific:
- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
fenced by generation identity (clearTerminalHostAction), so a replacement
generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
across restarts and replicas and retries the idempotent hook; the relay only
re-invokes while the marker is unacknowledged, so a successful ack prevents
duplicate work. Store-won expiry marks it too, so a loser-replica relay still
crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.
Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts
CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): repair the deletion CLI and order restore before the fence release
Addresses three findings from the fresh Codex review.
P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.
P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.
Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.
Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge dead deliveries in topology-safe maintenance
Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.
Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.
Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.
Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): refresh host-action retention on each retry attempt
Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.
Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.
Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): defer settlement when the Stop barrier times out
Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.
The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.
Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops
Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.
P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.
P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.
P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.
P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.
Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge terminal runs in clustered workers, retry suspension restore
Fourth Codex round on
|