mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-16 21:08:17 +00:00
67 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
42219ec8e9
|
🚜 feat: Support Native PostgreSQL Tool Favorites (#18032)
* 🚜 feat: Support Native PostgreSQL Tool Favorites
* fix: Guard PostgreSQL Reconciliation and Connection Leases
* fix: Scope Favorites Erasure Fences to Deletion Attempts
* fix: Preserve Account Erasure Ownership Until Cancellation Completes
* fix: Reject Unsupported PostgreSQL Command Flags Before Connecting
* fix: Keep Command Validation Compatible With the TypeScript Target
* fix: Keep Migration Previews Read-Only and Preserve Erasure Recovery Identity
* fix: Fence Erasure Recovery Across Tenant and Ownership Changes
* fix: Isolate Migration Dependencies and Keep Mongo Scans Outside Transactions
* fix: Inject PostgreSQL Backend Factories at Composition Roots
* ci: Shard Complete Mutation Work Without Relaxing the Aggregate Gate
---------
Co-authored-by: Lia <lia@librechat.ai>
|
||
|
|
fdf75e0174
|
🔧 fix: Fence Channel Startup and Transaction Capability (#17831)
* 🔧 fix: Fence Channel Startup and Transaction Capability * 🔧 fix: Gate Clustered Workers Until Startup Completes |
||
|
|
8596615d1d
|
🌐 feat: Wire Responses and Channel Lifecycle APIs (#17611)
* 🌐 feat: Wire Responses and Channel Lifecycle APIs
* fix: Align Channel Application Policy and Source Authority
* fix: Preserve Configured App Budgets in Canonical Response Commits
* Expose Fenced Channel Receipt Reconciliation Routes
* fix: Keep Ordinary Chat Capture In Its Existing Owner
* fix(channels): scope worker authority and defer browser activation
* fix(responses): gate admission on canonical queue readiness
* fix: Recover Durable Response Dispatch Across Queue Shutdown
* fix: Preserve Upstream Endpoint Config Insertions
* test: Align Channel Runtime Configuration Contract
|
||
|
|
a3e8426588
|
🛡️ fix: Reject Unknown Admin Config Sections and Scope Write Responses to Read Access (#17315)
* 🛡️ fix: Reject Unknown Admin Config Sections and Scope Write Responses to Read Access - Admin config writes (PUT, PATCH, tombstone) now accept only top-level sections that configSchema defines. PUT reports `unknown_section` issues under CONFIG_OVERRIDE_INVALID; PATCH and tombstone return 400 UNKNOWN_SECTION without echoing the path. This closes override writes to AppService runtime state (`paths`, the raw-YAML `config`, `availableTools`, and the `interfaceConfig`/`turnstileConfig`/`mcpConfig` aliases). - mergeConfigOverrides ignores stored override keys and tombstones outside configSchema, with one warning per document, so data an older server stored is neutralized without a migration. DELETE /fields still removes such stored entries, which is the only per-field way to clear a legacy tombstone. - Write responses (PUT, PATCH, tombstone, field delete, toggle) are redacted and then filtered to the sections the caller can read plus the sections the write managed; broad managers see the whole document as before. - Admin reads of the resolved config redact literal Azure group keys (merged `groups`, `groupMap`, raw YAML copy) and the Vertex `serviceKeyFile` (`vertex` and `vertexConfig`), keeping `${ENV}` and `user_provided` references readable. Write behaviour for these paths is unchanged. - The experimental cluster entry boots from librechat.yaml only (`baseOnly`), like server/index.js, so `__base__` overrides no longer reach file storage, skill sync, skill deletion or the expired-file sweep at boot. * 🛡️ fix: Resolve Admin Config Write Read Access Before Committing - Section-scoped writes (PATCH, tombstone, field delete) resolve the caller's readable sections before the mutation, alongside the pre-write reads they already make, instead of after the write and audit have committed. The batched lookup covers every `configSchema` section, so it no longer waits on the stored document. - A failed grant lookup now fails the request before anything is stored, so a 500 never follows a committed write and invites a duplicate retry. Response filtering after the write is pure. - Assign-only scope creation and toggles match only empty documents (`expectEmpty`), so their responses skip the lookup entirely. * 🚩 feat: Report the Section Allowlist and Read-Filtered Write Responses in Config Meta - `GET /api/admin/config/meta` now reports `sectionAllowlist: true` and `readFilteredWriteResponses: true`, which this branch implements (BE-17, BE-18). - The meta.spec probes check each flag against the handlers rather than the validator alone: PUT answers `unknown_section` with `segments`, PATCH and tombstone answer `UNKNOWN_SECTION`, a stored `paths` override and tombstone are ignored at merge, and PATCH, tombstone and field delete responses omit a section the caller cannot read. - A dotted top-level key is reported once, as `dotted_key`, instead of also as `unknown_section`. - The unknown-section handler and integration tests expect the `segments` that issues carry since #17322. * 🧪 test: Type the Tombstoned Meta Probe Document Through storedConfig * 🧪 test: Align Allowlist and Filtered-Response Specs with Version-Checked Web Search Writes dev's version-checked webSearch writes pass a session and precondition to patchConfigFields/unsetConfigField, read the stored document before removing a binding path, and have getBaseConfig load the librechat.yaml config beside a merged read. The field delete spec now stores the document it removes from and asserts the precondition alongside the filtered response. * 🧪 test: Pin Section-Manager Write Responses to What a Read Shows The grant layer resolves read:configs:<section> from the same section's manage:configs:<section> (getParentCapabilities, #14472), so a section manager's GET already returns the whole managed section. The comment on resolveWriteReadAccess said the opposite; it now states the model. A new integration spec runs the handlers over real SystemGrant documents through generateCapabilityCheck: a caller holding only manage:configs:endpoints patches one endpoints field, and its write response equals what GET returns it, other endpoint overrides and tombstones included, with memory and webSearch left out of both. * 🧪 test: Mark Config-Service Generation Tests With a Schema Section The base-config generation tests that came with the portable config store marked each base config and override layer with an `x` key, which no section in `configSchema` defines. This branch dropped `x` from the spec's `TestConfig` and ignores such keys when overrides are merged, so those tests no longer typechecked, and the override-layer ones would stop seeing the layer they read. They now mark layers with the `version` section, as this branch's other service tests do. * 🧪 test: Expect the yamlMcpHide Flag Beside the Section Allowlist Flags #17327 publishes features.yamlMcpHide on dev, so the exact feature list this PR asserts in meta.spec now includes it. |
||
|
|
2591a15c10
|
🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract (#17322)
* 🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract Keep dotted record keys in admin config write bodies behind a path-scoped sanitizer, reject $-prefixed and prototype keys after auth, report dotted_key issues with segments, and publish GET /api/admin/config/meta (features, rules, limits, versions) so clients can gate on backend capabilities. * 🧪 test: Expect Issue Segments in the Config Override e2e Scenario Override validation issues now carry `segments`; assert the full issue shape. |
||
|
|
73847f2db5
|
🔌 feat: Integrate Durable Skill Lifecycle Across Writers and Routes (#17043)
* feat: integrate durable Skill lifecycle across writers and routes * 🧪 test: Cover Skill Management Deletion End to End * fix: Queue superseded Skill blobs for durable cleanup * fix: Reject Skill file storage moves without new identity * 🧪 test: Cover Skill UI Delete and Cleanup Restart * 🧪 test: Assert Durable Cleanup for Replaced Skill Files * fix: release Skill names during account deletion marking * fix: use reserved Skill blob plans across writers * fix: hold superseded Skill blobs until sync commits * test: preserve account deletion retry coverage after restack * test: verify durable Skill cleanup after account erasure * fix: complete Skill cleanup leases and avoid historical metadata scans * fix: bound Skill cleanup manifest processing |
||
|
|
ccae5b988d
|
📜 feat: Audit Admin Role, Group, Config and IdP Role Mutations (#17052)
* feat: Extend the Audit Action Vocabulary to Admin, Sync and Credential Mutations * feat: Audit Admin Role, Group and Config Mutations Role deletion now removes the role's grants, ACL entries and config overrides before the role itself and returns ROLE_CLEANUP_FAILED instead of success when any of them fails, so a recreated role cannot inherit leftover privileges. * feat: Audit IdP and Startup Role Changes * fix: Keep the Startup Role Sync Running When Its Audit Verification Read Fails * test: Cover Admin Role Audit Trail End to End * fix: Audit a Remote Agent Role Sync Only After It Persists * fix: Restore Role Dependents When Role Deletion Fails Midway * fix: Restore Role Dependents Only When the Role Still Exists * style: Sort Role Admin Imports * fix: Keep the ACL Snapshot When Role Cleanup Fails Ambiguously * test: Read Role Audit Entries From the Audit Log Route * fix: Audit Only Real Membership Changes and Restore Deleted Role Config in Full * fix: Restore Deleted Role Grants in Full and Sanitize Audit Write Logging * fix: Audit and Report a Role Deletion Confirmed After a Lost Response * fix: Never Restore Dependents Onto a Recreated Role * fix: Snapshot Role Dependents Before Deleting and Detect Membership Changes Atomically * fix: Restore From the Data a Cascade Step Actually Removed * fix: Restore Only Config Overrides When a Role Delete Fails * fix: Log Only Safe Metadata When the Startup Audit Read Fails * fix: Audit Grants Removed by a Failed Role Delete * fix: Audit Removed Grants on Unverifiable Role Delete Exits * fix: Honor Fail-Closed Auditing in the Admin Config, Group and Role Routes * fix: Bind Role Deletion to the Captured Id and Audit Only Atomic Role and Config Changes * fix: Skip the Config Restore When the Targeted Role Is Gone * fix: Audit Initial Group Members and Skip No-Op Role Description Updates * fix: Record Role Deletion as Pending Before Cleanup and Append Its Outcome * fix: Audit Reassigned Role Members and Return the Persisted Config After a Tracked Toggle * fix: Keep a Committed Role Deletion Successful When Cascade Audit Writes Fail * fix: Decide Group Update Audits From the Atomic Write and Sanitize Permission Update Logs * fix: Audit Grants From a Snapshot, Fail on Member Lookup Errors and Chunk Member Audit Entries * fix: Audit Group Deletion Members and Previous Role Removal, Abort on Grant Snapshot Failure * fix: Invalidate Config Caches After Partial Secret Writes * fix: Retry Failed Cascade Restores, Report Changed Group Fields and Audit Every Reassigned Member * fix: Log Safe Metadata When Cache Invalidation Fails After a Partial Write * fix: Keep Error Messages Out of the Audit Write Failure Log * style: Sort Audit Log Spec Imports * fix: Sanitize Admin Handler Error Logs and Dispatch System Audit Writes Without Waiting |
||
|
|
20d2c0d43d
|
🧹 fix: Run the Full Erasure Cascade When an Admin Deletes a User (#17050)
* fix: Run the Erasure Cascade in Admin User Deletion * refactor: Share Generation Stop Logic Between Self-Service and Admin Deletion * refactor: Move Admin Erasure Dependency Assembly Into packages/api * fix: Bind getCleanupJob When Waiting for Generation Persistence * test: Pin the Deletion Fence Before Admin Generation Stop * fix: Resolve the Erasure Target Config Fail Closed * fix: Carry the Target Profile Into Admin Erasure Storage Cleanup * fix: Keep the Erasure Target Contract Storage Neutral and Reuse the Loaded Identity * fix: Refuse New Generations Once Account Deletion Has Begun * fix: Enforce Account Deletion Admission Once in createJob and Narrow the Erasure File Type * fix: Install the Admission Guard in Every Entrypoint and Purge Refused Jobs * fix: Guarantee Refused Job Purge and Sanitize Admission Failures * fix: Route Remote Admission Through createJob and Keep Refusal Cleanup Sanitized * fix: Log Admission Cleanup Failures With Safe Error Metadata * refactor: Check Account Deletion Before Creating a Generation and Drain Twice * refactor: Order Admission Against the Deletion Marker Without a Grace Period * fix: Sanitize Post-Commit Admission Retirement Failures * fix: Wire Shared-Server MCP OAuth Cleanup Into Admin Erasure * fix: Scope Admin Erasure Lookups to the Target Tenant and Await Shared-Link Cleanup * fix: Resolve the Erasure Target Config Under the Target Tenant * fix: Run Admin Erasure OAuth Fencing in the Target Tenant * fix: Match the Split OAuth Invalidate and Disconnect Hooks in Admin Erasure * refactor: Inject the Tenant Executor Into the Erasure Resolver |
||
|
|
219a48b55d
|
⏳ fix: Expire Dead-Lettered Trigger Deliveries and Clean Up Orphaned Image Uploads (#17049)
* fix: Expire Dead-Lettered Trigger Deliveries After a Review Window * fix: Delete Stored Image Uploads When Record Creation Fails * fix: Clean Up Stored Objects on Every Storage-Before-Record Upload Path * refactor: Resolve the Delete Strategy Lazily in Stored Upload Cleanup * test: Cover Dead Letter Expiry After the Review Window * fix: Carry Owner to Stored Upload Cleanup and Keep Batch Dead Letter Deadlines * refactor: Select the Stored Upload Cleanup Target in the API Package * fix: Check for a Committed Record Before Deleting an Upload and Keep Dead Letters Unless Configured * fix: Check the Persisted Record Id for Converted Assistants Images * fix: Arm a Dead Batch Root Deadline After Its Members Settle * fix: Settle Expired Dead Batches and Skip the Record Guard Before the File Write * fix: Skip the Record Guard for Pre-Write Failures and Build the Lookup in packages/api * fix: Build the Record Lookup Lazily in the Upload Service * fix: Pass the Dead-Letter Retention Window Through Experimental Startup * fix: Carry the Vector Copy Into Failed-Upload Cleanup * fix: Cover the Code Environment and Indeterminate Vector Copies in Upload Cleanup * fix: Give URL-Upload Cleanup a Request When the Caller Omits One * fix: Route Failed-Upload Cleanup Through Secondary-Storage Deletion * fix: Attempt Every Copy Independently When Cleaning Up a Failed Upload * fix: Track the Record Write in Every Upload Path Whose Retention Lookup Can Fail |
||
|
|
37ec4b56c9
|
🎨 feat: Paint the Operator Theme on a First-Ever Visit (#16962)
* feat: Paint the Operator Theme on a First-Ever Visit The server embeds the deployment's base interface.theme in the HTML shell, with both modes the build resolved for a bundled name, so the boot script and the first commit of DeploymentTheme paint it before /api/config answers. A tab's owned cache still outranks it, shared links skip it, and a request scoped by a trusted X-Tenant-Id header gets the shell without it. * test: Type the Inline Shell Theme Fixture as Raw JSON * test: Tag the First-Visit Scenarios Literally * fix: Ignore a Theme Boot File That Is Not an Object * fix: Keep the Base Theme for a Tenant Header the Config Rejects * fix: Report a Theme Boot File That Exists but Cannot Be Read * fix: Embed the Base Config Theme in the Experimental Shell * test: Stand In the Shell Theme for an Owned Copy Whose Session Ended |
||
|
|
5cb396e85c
|
👋 feat: Optional Greeting Heading and Footer-Free Welcome Screen (#17055) | ||
|
|
e216f08289
|
🌗 feat: Hand the Colour Mode Over on an Entry Link (#17056) | ||
|
|
0bc344fcb9
|
🔑 feat: Let Users Connect Their Own Account for Pull Request Lookups (#16989)
* feat: Let Users Connect Their Own Account for Pull Request Lookups * fix: Refuse Cross-Tenant Pull Request Callbacks and Surface Key Failures * fix: Keep Pull Request Grants Through Outages and Remove Them on Disconnect A token endpoint server error is an outage even with an OAuth error body, so it no longer deletes the grant, and a refresh outage reaches the lookup as unavailable instead of falling back to no token. Disconnect removes every pr_oauth row of the user, whatever app or host is configured now. A grant that never expires extends its retention when used, and the flow, request and retention limits are configurable. The connection routes are built in packages/api, mounted behind the pre-auth tenant middleware in both servers, and answer failures with stable codes. * fix: Keep the Pull Request OAuth App Out of Overrides A role, group, user or tenant override can no longer change or remove pullRequests.oauth, so the connect capability and the callback see the operator's app. * fix: Read a User's Pull Request Token Only With the Feature On * fix: Report a Malformed Stored Pull Request Grant as a Failure * fix: Remove a User's Stored Tokens When an Administrator Deletes the User * fix: Answer No Pull Request for a Conversation Without a Lane When the User Token Fails * fix: Share One Grant Read Between Concurrent Callers and Report an Expired Grant Disconnected * fix: Remove a Pull Request Grant Only on a Definite Refusal and Past Its Retention * fix: Version Pull Request Grants and Coordinate Refresh, Refusal and Retention Through Them * fix: Invalidate a Refused User Token and Read It Only for a Lane That Needs It * fix: Finish a Pull Request Connection Under the User's Effective Config Behind the Login Limiter * fix: Type the Pull Request User Token for Isolated Declarations and Settle Retention Before Answering * fix: Bind a Pull Request Connection Flow to the Generation a Disconnect Replaces * fix: Start a Batch's User Token Read Only Within Its Deadline * fix: Fence Pull Request Token Reads Against a Disconnect and Report a Failed Retention Cleanup * fix: Fence Connection Status Like Token Reads and Keep a Newer Read Joinable * fix: Report a Pull Request Grant Without Its App and Name the Sign-In That Made It * fix: Read a User's Unexpired Pull Request Grants Newest First * fix: Mark the Connection Callback as an OAuth Navigation and Report a Failed Extension * fix: Name the Pull Request Sign-In Flow on Its Landing Page * fix: Remove a Pull Request Grant of a Replaced App When It Is Read * fix: Keep a Replaced App's Grant Untouched and Extend Retention From Each Day's Use * fix: Accept a Pull Request Grant Only Under the User's Current Connection Generation * fix: Renew the Connection Generation With a Grant Still in Use * fix: Restore the Previous Connection When a Sign-In Cannot Store Its Grant * fix: Decide a Sign-In by What Was Stored When a Write Does Not Answer * fix: Roll Back a Sign-In's Generation Along With a Grant It Must Remove * fix: Refresh an Access Token a Lookup Was Refused With Before Ending the Connection * fix: Undo a Sign-In Only for a Confirmed Fence, Not a Failed Check * fix: Read a User Token Again When the Connection Changes Under the Read * fix: Apply the Current Retention Again When Another Replica Changed the Grant First * test: Answer Each Concurrent Sign-In by Its Own Code * fix: Skip Retention Rewrites for Grants That Expire Without a Refresh Token * fix: Retry a Token Rejection That Loses to a Concurrent Grant Write * fix: Skip User Token Reads for Lanes Without a Commit and Reread Status After a Reconnect |
||
|
|
f88f3fb2fe
|
🪇 perf: Coalesce Ready Background Completions (#16650)
* 🪇 perf: Coalesce Ready Background Completions
* fix: Recover Receipt Batches and Gate Rolling Deploys
* fix: Fence Background Batch Handoffs and Cleanup Epochs
* fix: Bind Resumed Cleanup to Its Observed Batch
* fix: Preserve Background Results Through Recovery Handoffs
* fix: Recover Unconfirmed Manual Background Claims
* fix: Fence Automatic Recovery of Manual Handoffs
* fix: Preserve Background Reconciliation and Recovery Retention
* fix: Preserve Replayed Claims and Finish Follower Proofs
* fix: Recover Stale Background Projection Epochs
---------
Co-authored-by: Lia <lia@librechat.ai>
|
||
|
|
bb0543fd15
|
📼 fix: Replay Running Subagent Activity (#16541)
* 📼 fix: Replay Running Subagent Activity * 📼 fix: Preserve Activity Across Rollouts and Reconnects * 📼 fix: Backfill Missing Foreground Activity * 📼 fix: Fence Replay and Drain Admitted Activity * 📼 fix: Close Subagent Replay Review Gaps * 📼 fix: Resume Activity After Replay Recovery Expires * 📼 fix: Use Singular Subagent Omission Copy * 📼 fix: Preserve Public Replay and Displayed Activity * 📼 fix: Write Expanded Live Activity Before Queuing * 📼 fix: Track Disjoint Subagent Replay Coverage * 📼 fix: Preserve Tool Details and Omission Totals * 📼 fix: Recover Tool Timing During Replay Backfill * 📼 fix: Reconcile Covered Replay Timing Metadata * 📼 fix: Restore Message Phases During Replay Backfill --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
4519fe09ae |
⚖️ refactor: Move Policy Links to the Signup Consent (#16138)
(cherry picked from commit
|
||
|
|
50ce28957f
|
🎚️ fix: Apply librechat.yaml Rate Limits to Route Limiters (#16432)
Route modules build their rate limiters while they load, and both server entries required them before startup checks copied rateLimits from librechat.yaml into the environment. The conversationsImport, tts, stt and route-scoped fileUploads budgets therefore ignored the yaml unless the matching environment variable was also set. Require the routes only after performStartupChecks in index.js and experimental.js. |
||
|
|
c606e5065a
|
🛬 fix: Settle Background Tool Results Before the Server Shuts Down (#16365)
* 🛬 fix: Settle Background Tool Results Before the Server Shuts Down * 🛬 fix: Preserve Background Shutdown Results Through Fallback * 🧪 test: Match Formatted Cluster Shutdown Wiring --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
ffd3d1e139
|
🔕 perf: Back Off Idle Durable Agent Recovery Polls (#16303)
* 🔕 perf: Back Off Idle Durable Agent Recovery Polls * 🎛️ style: Format Queued-Turn Recovery Test * fix: Preserve Durable Recovery Wakeups and Discovery Invariants --------- Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
36efc7b4e2
|
🧺 fix: Bound Background Completion Batches (#16102)
* 🧺 fix: Coalesce Durable Background Wakeups
* fix: Bound replayed background result claims
* fix: Harden background result coalescing controls
* fix: Reconcile background result claims
* fix: close background batch ownership races
* fix: Keep completion batching atomic and bound serialized metadata
* refactor: Reuse canonical background claim contracts
* fix: Expose serialized claim budget in the persistence contract
* test: Preserve claimed V1 results across later background batches
|
||
|
|
69f0dd2a12
|
🔌 feat: OpenAPI spec and docs for the public Agents API (#15928)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
* 🔌 feat: OpenAPI spec and docs for the public Agents API * 🔧 fix: address review on Agents API OpenAPI spec and docs: - adapter: mark JSON request bodies as required - openapi.js: move gate, config read and spec serving into packages/api as an injected router factory; leave /api as wiring only - docs: compute URLs in-browser and use a relative servers url so docs work under a base path (e.g. /chat) - build: copy the generated spec into dist so the multi-stage image ships it - agents: reuse the enforced list envelope for the docs so the AgentList schema cannot drift from the runtime cursor and strictness * 🔧 fix: align Agents API OpenAPI contract with enforced runtime Second review round on the public Agents API spec. Three cases where the published contract described something the server does not do, so generated clients would fail against the live API: - Upload form: document the `purpose` field (enum: file_search, execute_code, context) that POST /agents/{id}/files actually requires, instead of the internal-only `tool_resource`; requests following the old form were rejected with 400 because `purpose` was absent. - Auth: drop the `apiKeyBearer` scheme from the management and skill endpoints. They gate on OIDC/M2M via requireAgentManagementAuth and deliberately reject the API-key fallback; the API key belongs to the out-of-scope OpenAI-compatible endpoints and returns when those are documented. - Skill update: document the 409 conflict PATCH /skills/{id} returns on a stale expectedVersion. The agent update path collapses its internal 409 to 400, so it correctly keeps no 409. * 🔧 fix: serve docs on experimental server and document the 401 Third review round on the public Agents API spec. - Experimental server: mount routes.openapi in api/server/experimental.js before its /api 404 handler. Without it, npm run backend:experimental returns 404 for /api/docs and /api/openapi.json even when openapi.enabled is true, because that server never mounted the router. - Auth failure: document the 401 that requireAgentManagementAuth returns on a missing, expired, malformed, or unbound token. Its body is { "error": "Unauthorized" }, a flatter shape than the management error schema, so add a dedicated UnauthorizedError schema and a shared 401 response to the agent and skill contracts rather than reusing the standard error schema. * 🔧 fix: document skill-update, auth-conflict, upload-limit, and watch-copy gaps * 🔧 fix: added error shapes for permission denied and internal server errors in API responses docs * fix: cover the auth-layer 500 shape and pin the account-deletion code * fix: update OpenApiRouterDeps documentation and simplify request handling in isEnabled function * 🔧 fix: gate the Swagger asset mount behind the OpenAPI toggle * 🔧 fix: accept the malformed-JSON 400 envelope in the contract * fix: accept the tenant-isolation 403 envelope in the contract * fix: add description for content property in SkillFileUpdateRequest to clarify byte limit |
||
|
|
7e05c4848a
|
👑 fix: Elect a Single Expired-File Sweeper (#15909)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: elect one expired-file sweeper * fix: unify clustered sweep ownership * fix: retain clustered sweep fallback * fix: use normalized Redis sweep ownership * fix: reconcile sweep startup and concurrent leader ownership |
||
|
|
8e6e6f062d
|
🪧 fix: Emit the Configured-Footer Answer With the HTML Shell (#15886)
* 🪧 fix: Emit the Configured-Footer Answer With the HTML Shell The chat surface lays out against whether the deployment configured a footer (customFooter, privacy policy, terms of service). The client answered that from localStorage — whatever the last resolved `/api/config` recorded — so the answer was a guess on a first-ever visit and could be stale for a deployment whose configuration changed. The backend already knows when it serves client/dist/index.html, so it says so there: `injectConfiguredFooterBootstrap` writes `hasConfiguredFooter` into `window.__LIBRECHAT_CONFIG__` ahead of the app's own scripts, once at boot from the base app config, and the client reads it on its first render instead of remembering the last answer. The `configured-footer` storage atom is gone. Client and server share one predicate (`hasConfiguredFooter` in librechat-data-provider) so the two cannot disagree, and the shell injection is generalized (`injectBootstrapConfig`), with the query-devtools flag now going through it. * 🧷 fix: Keep Bootstrap Values Literal and State the Shell's Contract `injectBootstrapConfig` passed its script as a replacement string, so a `$&`, `$1` or `$'` inside an injected value was expanded by `String.replace` instead of reaching the client. Both insertion points take a replacement function now, and the spec reads its values back off the global through an explicit result type rather than an unchecked dictionary. The shell answers from the deployment's own configuration, which a per-tenant, role or user override of `interface.privacyPolicy` or `termsOfService` can disagree with — `/api/config` resolves the caller's and the client prefers it. That contract was overstated in three comments ("a deployment-lifetime fact", "the same one the config will give"); they now say what the answer is and what resolves the rest. Two scenarios cover the disagreement end to end: a document served without the answer, and a policy link only this caller's config carries. |
||
|
|
9cf33cc8be
|
🧿 fix: Bind Login Flows to the Browser That Started Them (#15880)
* 🧿 fix: Bind Login Flows to the Browser That Started Them Social OAuth logins (Google, Facebook, GitHub, Discord, Apple) now store a random state in an HttpOnly cookie scoped to the callback path and only exchange the returned code when the callback carries that state. Apple's form_post callback uses a SameSite=None cookie, and passport-apple's own fixed state no longer bypasses the store. Local login, temp-token 2FA verification and admin local login reject requests a browser sent from another origin (Sec-Fetch-Site, falling back to Origin vs Host), trusting DOMAIN_CLIENT and DOMAIN_SERVER. Requests without browser fetch metadata, such as server-side callers, pass unchanged. The sessionless OAuth callbacks drop failureMessage, which wrote to req.session and turned any rejected callback into a 500. * 🧿 fix: Configure Login State Cookies and Localize Origin Rejections - OAuth state cookies take secureCookie and maxAgeMs from the caller; registration.oauthStateTtlMs in librechat.yaml sets the lifetime (default 10 minutes). - Secure deployments use a __Host- cookie at Path=/ so a sibling subdomain cannot plant a state. - The cookie keeps up to three pending states, so logins started in separate tabs all complete, and an unmatched callback no longer discards them. - passport-apple's preset state is dropped by deferStateToStore in packages/api. - The Origin fallback compares scheme and host against the request itself. - The same-origin rejection carries ErrorTypes.AUTH_CROSS_ORIGIN, which the login and 2FA screens show as a localized message. * 🧿 refactor: Sign Login State Against a Per-Browser Binding - The OAuth state store keeps a random per-browser binding in a __Host- cookie (reused across flows) and signs each state as issuedAt.nonce.HMAC(JWT_SECRET, provider, binding, issuedAt, nonce). Nothing is stored per flow, so any number of tabs complete, each state expires on its own clock, and concurrent starts no longer overwrite each other. - The experimental server entrypoint passes its app config to configureSocialLogins. - The same-origin guard also trusts an explicitly configured ADMIN_PANEL_URL. * 🧿 fix: Keep Separate Bindings for Racing First Logins A browser without a login-state binding now gets one under a cookie name with a random suffix, so two tabs that start their first login at once each keep their own binding. Later starts reuse an existing binding, and a callback is accepted when its state was signed for any binding the browser presents. * 🧿 fix: Sanitize Request-Derived Values in the Cross-Site Rejection Log The same-origin guard logged the request Origin and path verbatim. A non-browser client fully controls the Origin header, and the console transport uses a printf format that does not escape control characters, so a crafted value could forge log lines. Strip ASCII control characters (CR, LF, NUL and the rest) before logging, matching the pattern apiNotFound already uses. |
||
|
|
65bb64bc68
|
🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784)
* fix: recover CodeAPI uploads after throttling * fix: cover eager CodeAPI uploads * fix: scope CodeAPI upload recovery * test: clarify upload recovery deadlines * fix: isolate CodeAPI recovery state * fix: cancel CodeAPI recovery waits * test: provide CodeAPI registry in handler fixture * fix: propagate CodeAPI recovery controls * fix: bound CodeAPI recovery by run * fix: preserve skill upload cancellation * test: update deferred loader signal contract * fix: preserve recovery cancellation * fix: cancel recovery source reads * fix: classify Axios upload cancellation * fix: preserve lazy provisioning cancellation * fix: close resource recovery boundaries * fix: preserve provisioning failures compatibly |
||
|
|
000349681e
|
🩻 feat(insights): Scope Insights to Authorized Agents (#15549)
* feat(insights): add agent-scoped access * fix(insights): exclude unattributed assistant messages * fix(sharing): key role insight toggles by principal * fix(sharing): preserve insight grant snapshot * fix(permissions): preserve insight bits atomically * fix(permissions): guard insight audit rollback * fix(insights): keep admin grant controls accessible * fix(permissions): reconcile duplicate principal updates * fix(insights): show automatic admin access * fix(permissions): harden insights access updates * fix(insights): simplify authorization and preserve client state |
||
|
|
57c52de33c
|
🛎️ fix: Start Code Environment Reconciliation After Mongo Connects (#15620) | ||
|
|
d346e5ab5e
|
🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short (#15532)
* 🚰 fix: Record the Provider Drain When Shutdown Cuts a Generation Short
* fix: Await generation settlement on shutdown instead of publishing the drain
* fix: Spend the real shutdown budget and cover the initialization window
* fix: Keep the shutdown registration literal the index spec asserts on
* fix: Keep draining settlements registered while shutdown is waiting
* fix: Track provider executions at the begin/drain chokepoints instead of per caller
* fix: Open the provider tracker before the begin CAS and cap cluster workers to the primary deadline
* fix: Propagate the cluster deadline and release trackers on reconfigure and abandonment
* fix: Hold SIGTERM until the worker acknowledges the cluster deadline
* fix: Refuse provider starts after shutdown begins and harden the cluster deadline handoff
* test: Assert the multi-execution shutdown wait with both executions begun before shutdown
|
||
|
|
fcae1025c0
|
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments * fix: address code environment CI coverage * fix: expose principal code environments in endpoint config * fix: reject missing code environment bodies * fix: harden code environment lifecycle * fix: revalidate principal code environments * fix: synchronize code environment authorization * fix: bind code environments to current principals * fix: fail closed on code ACL cache errors * fix: prevent code environment override shadowing * fix: preserve principal environment defaults * fix: suppress revoked code environment aliases * fix: fail closed on environment augmentation * fix: narrow code environment defaults * fix: isolate code environment fallback * style: sort code config imports |
||
|
|
a9ccac8656
|
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing * fix: satisfy code environment type checks * fix: secure code environment administration * fix: isolate code pairing control plane * fix: validate code pairing control responses * fix: secure code pairing transport * fix: validate code pairing wire format * fix: harden pairing secret lookup |
||
|
|
487984193a
|
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable * fix: align detached actor checks with merged base * fix: close detached actor lifecycle gaps * fix: break detached action outcome cycle * fix: deepen detached action ownership * fix: fence detached launch handoffs * fix: preserve detached rollout fencing * test: type detached recovery failure * fix: close detached ownership handoffs * fix: stage detached action activation * fix: persist detached terminal retries * fix: fence detached durability ownership * fix: fence trigger lane publication * fix: fence detached transition generations * fix: retire failed detached predecessors * fix: close detached rollout consumers * test: type detached retry turns * fix: close detached ownership gaps * fix: fence detached action resumes * fix: fence prior-head event actor resumes * fix: preserve legacy event actor resumes * fix: align detached resume boundaries |
||
|
|
de59da9636
|
🎟️ refactor: Require Credentials for Local Image Access by Default (#15252)
* 🔐 fix: Protect Local Image Access by Default * 🔐 fix: Scope Image Authorization to Active Sessions * 🧹 style: Format Image Authorization Checks * 🛡️ fix: Harden Image Avatar Authorization * 🧭 style: Sort Image Authorization Imports * 🔐 fix: Close Image Authorization Review Gaps * 🧭 fix: Normalize Stored Avatar Base Paths * 🏢 fix: Resolve Tenant Assistant Image Policy * 🛂 fix: Enforce Effective Image Access Policy * 🧹 style: Flatten Assistant Config Selection * 🧷 fix: Preserve Image Access Compatibility * 🪪 fix: Make Image Sessions Revocable * 🏗️ fix: Move Image Session Policy Into API |
||
|
|
44edcbe014
|
🧾 feat: Store Durable Event Actor Receipts (#15265)
* 🧾 feat: Store durable event actor receipts * 📊 fix: Scope actor delivery metrics * 🧱 fix: Close durable receipt recovery gaps * fix: serialize actor action admission * fix: retire terminal batch members * fix: close actor terminal recovery races * fix: reclaim lanes after terminal receipts * fix: close actor receipt recovery gaps * fix: preserve ambiguous legacy handling * fix: recover legacy actor outcomes * fix: admit leased actor actions * fix: serialize actor admission recovery * fix: close actor recovery races * fix: token-fence actor admissions * fix: preserve mixed-version actor fences * fix: harden actor rollout and metrics * fix: gate durable actor receipt rollout * fix: enforce base actor receipt rollout * fix: align experimental actor rollout |
||
|
|
f34a49007d
|
🔣 fix: Escape SPA Language Attribute (#15248) | ||
|
|
290b8664d9
|
🧾 feat: Track Authoritative Agent Event Outcomes (#15213)
* feat: track authoritative agent event outcomes * fix: isolate agent event outcome types * fix: declare agent event handler result * fix: simplify agent event status selection * fix: preserve authoritative event outcomes * fix: preserve terminal event evidence * test: use completed run-step envelope * test: scope deferred HITL question locator * fix: settle every agent event terminal path * style: sort terminal host action imports * fix: fence agent event terminal evidence * fix: recover agent event terminal settlement * fix: scope terminal retry hints by generation * fix: settle terminal host actions exactly |
||
|
|
877b9b2f1a
|
🐚 feat: Nonce-Based Content Security Policy for the SPA Shell (#14446)
* 🛡️ feat: Configurable Baseline HTTP Security Headers Adds helmet's CSP-independent headers (HSTS, X-Frame-Options, X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response, with contentSecurityPolicy explicitly disabled. Every header that can break a deployment is configurable, so there is no allow-list to go stale the way #7377's hardcoded CSP directives did. HSTS includeSubDomains defaults off rather than matching helmet's on-by-default: it would otherwise pin every sibling subdomain to HTTPS for a year in every visitor's browser, and undoing that requires serving max-age=0 from each affected host. * 🛡️ feat: Nonce-Based Content Security Policy for the SPA Shell Adds an opt-in, per-response nonce CSP on the HTML response, resolved once at startup so each request only mints a nonce and concatenates the header. Report-only by default, since that is the rollout step #7377 skipped. Rebase and correctness pass over #13226: - Styles carry no nonce. A nonce in style-src makes browsers ignore 'unsafe-inline', which would have blocked the <style> element the theme script injects at runtime, plus every style third-party components inject. - frame-ancestors 'self' is now a default rather than opt-in, so enabling CSP actually covers the clickjacking half of #7110. - CSP_SCRIPT_SRC_EXTRA now drops 'strict-dynamic', which would otherwise make browsers ignore the very hosts the operator configured. - Nonce stamping runs after the query-devtools bootstrap injection so that injected script is covered too. * fix: replace frame-ancestors instead of merging it Merging the configured value into the default turned a deliberate CSP_FRAME_ANCESTORS='none' into `frame-ancestors 'self' 'none'`, which browsers resolve back to 'self'. Also bail out if the serialized policy somehow lacks the nonce slot rather than emitting a header the shell cannot match. * fix: address Codex review findings on the CSP defaults All five were real against LibreChat's actual runtime: - CSP_REPORT_ONLY now only enforces on an explicit false/off/0/no. A typo or `1` previously fell through isEnabled() to enforcing, turning a config slip into a blocked SPA. Shares the parse helper with headers.ts via a new security/env.ts. - Module preloads are stamped. A production client/dist/index.html carries 32 parser-inserted `<link rel="modulepreload">` tags, which 'strict-dynamic' does not cover and 'self' cannot rescue. - Stale nonce attributes are replaced rather than preserved; only the current response's nonce is authorized. - worker-src allows data:, which Monaco's default CDN loader needs to bootstrap its workers (there is no loader.config() in the client). - script-src allows 'wasm-unsafe-eval' for the HEIC upload path, which compiles WebAssembly through heic-to. Narrower than 'unsafe-eval'. Verified against the real built shell: 4 scripts and all 32 preloads nonced, stylesheets/icons/manifest and <style> untouched. * fix: address second Codex round on CSP rollout controls - SECURITY_HEADERS=false now disables CSP too. It is documented as the global kill switch, and an operator reaching for it to recover a shell broken by an enforcing policy must not be left with that policy on. - The SPA shell is forced to `no-store` while CSP is enabled, ignoring INDEX_CACHE_CONTROL/INDEX_PRAGMA/INDEX_EXPIRES and warning when they are set. A cacheable shell pins one nonce across page loads and users, which is the whole thing a nonce policy defends against. - Added CSP_ALLOW_WASM and CSP_ALLOW_DATA_WORKERS. The previous commit's .env.example claimed CSP_ADDITIONAL_DIRECTIVES could drop 'wasm-unsafe-eval' and data:, but merging only ever appends sources, so the documented hardening step was impossible. These toggles make it real. |
||
|
|
2ef12b1e1d
|
🦺 feat: Configurable Baseline HTTP Security Headers (#14445)
Adds helmet's CSP-independent headers (HSTS, X-Frame-Options, X-Content-Type-Options, COOP, CORP, Referrer-Policy) on every response, with contentSecurityPolicy explicitly disabled. Every header that can break a deployment is configurable, so there is no allow-list to go stale the way #7377's hardcoded CSP directives did. HSTS includeSubDomains defaults off rather than matching helmet's on-by-default: it would otherwise pin every sibling subdomain to HTTPS for a year in every visitor's browser, and undoing that requires serving max-age=0 from each affected host. |
||
|
|
c5276fc63d
|
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations
Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.
Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
and capacity slots, timezone-aware cadence with DST-conservative floors and
misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
(signed schedule-fire JWT claims, per-occurrence idempotency key inside the
route's clientRequestId charset), overlap/balance/capacity/duplicate skip
policies with auto-disable streaks (too_many_failures, insufficient_balance),
reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
path: HITL pauses surface on the card (requires_action), resumes re-apply the
fire boundary's admission policy (revision fence, enabled, global kill switch,
SCHEDULES:USE, availability) before continuing a billed generation.
Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
run's terminal outcome write; Stop/complete/pause race through single-winner
terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
completedAt-less evidence for scheduled fires plus an owner-intended outcome
stamp (scheduleOutcome) the reconciler prefers over re-derived success —
round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
error/skipped_balance instead of success on both initial and resumed paths;
stale stamps are refreshed evidence-first when persistence plus the Mongo
outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
from-status; republication escalates to the transport's acknowledged variant;
the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
of scheduled + interactive work with durable per-stream abort fences
(positive-evidence acknowledgement only), owner-side finalization markers for
post-terminal billed writes, deferred-deletion sweep (explicitly ensured
partial index) that completes cascades autonomously. Remote OpenAI-compatible/
Responses requests are documented as outside the quiesce and tracked in issue
14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
contract with coherent degradation (registration fails -> synchronous title
fallback; count reads 0).
* fix: fence terminal response persistence from deletion; retain stale-pause evidence
Two blockers from the third external review round.
Terminal persistence visible to account deletion:
- The finalization marker was registered only when post-terminal TITLE work was
possible, but every persistence-owning terminal CAS opens the same window: the
claim drops the job out of the active set BEFORE the response save (and the
background user-message/convo saves), so a deletion quiesce landing there saw
neither an active job nor a marker and could cascade while the admitted request
could still recreate messages. Both controllers now register the marker before
every persistence-owning claim — the fresh-turn path and the HITL resume — and
release it only after their pending saves (and any post-terminal title) have
landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
`terminalPersistencePending` is set: terminal at the CAS is not settled while
the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
refuses a store without it at configure time, keeping the failure loud and
deterministic) while remaining optional on the legacy public IJobStore type for
source compatibility. The silent degrade path from the previous round is gone —
it was not deletion-safe.
Stale-pause recovery retains scheduled evidence:
- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
and the InMemory/Redis stale-pause cleanups — unconditionally stamped
`completedAt`, putting a scheduled fire's failed-pause error terminal on the
short completed TTL. A Mongo outage longer than that TTL erased the evidence and
the reconciler recovered the run as `interrupted` instead of `error`. All three
now follow the controller-observed path from the previous round: scheduled jobs
omit `completedAt` (retained-evidence TTL) and stamp the error outcome.
The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).
* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths
One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.
- Generation-scoped markers. Entries were keyed (userId, streamId), so a
Stop-superseded generation finishing late could clear the marker its
replacement registered on the same conversation. Marker fields are now
qualified by the generation's createdAt; clears must present the same
identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
user's existing hash expires between the two commands (or the process dies
there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
the terminal CAS instead of proceeding uncovered: the completion claim throws
into the error path, the error path skips completeJob and leaves the job
ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
and abortJob returns a new retryable `fence_unavailable` failure the Stop
route answers with 503 and the deletion quiesce treats as an unacknowledged
stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
the abort CAS, clear in its finally — the Stop route's checkpoint prune and
partial save run inside beforePublish, between CAS and publication); the
interactive and background generation-error paths register before their
completeJob; the resume controller's error finalization registers before its
completeJob. A lost or thrown claim releases the marker after pending saves
flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
terminalPersistencePending for ALL statuses — including `aborted`, whose
route-side persistence the previous guard missed.
Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.
* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment
Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.
- Lease tokens. Marker fields were (streamId, createdAt), shared by every
contender on the same generation — completion and Stop, or two racing Stops —
so a losing contender's cleanup erased the winner's still-live marker. Every
registrant now carries a unique lease token in the field and may only ever
clear its own lease; unqualified legacy clears cannot touch qualified entries.
- Admission fence. Authentication can pass before the deletion barrier goes up,
and the durable createJob is several async steps later — a deletion quiesce in
that window saw neither an active job nor a marker and could cascade before
the admitted request created its job. The controller now registers an
admission lease and THEN rereads the deletion barrier: the ordering guarantees
either this request observes the barrier (403, lease released, slot/claim
cleanup) or the quiesce observes the lease and defers. Held until createJob is
durable; released on every refusal and initialization-error path. Fail closed
when the lease itself cannot be registered (503 retryable).
- Undelivered-Stop retention. abortJob released its lease in a finally even when
delivery AND publication had provably failed — the job reads terminal
(invisible to active-set scans), a user Stop writes no durable abort fence,
and the remote owner keeps generating and will persist its abort-catch writes
whenever the signal finally lands. The lease is now retained in exactly that
case, and each resignal attempt heartbeats a fresh lease so the fence outlives
the TTL for as long as delivery is still being driven. The abort-winning
turn's own loser-side pending saves are additionally fenced in the controller
catch (best-effort — those writes are already in flight).
- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
non-assistants endpoints) awaited abortJob and then spent usage and saved the
stopped response AFTER the abort's lease was released. Both writes now run
inside `beforePublish`, between the abort CAS and publication, covered by the
same lease as every other abort.
Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.
* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races
Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.
- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
-job handoff is only atomic against a reader in the OPPOSITE order of the
writer: writers hold the lease strictly until the job is active-set visible,
so leases-first shows every interleaving either the lease or the job.
Jobs-first allowed a request to create its job after the scan and release its
lease before the count — hiding both, cascading, and letting the new
generation persist into a deleted account.
- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
the moment the owner's AbortController tripped; the stopping side released its
lease on that ACK while the owner's asynchronous abort-catch persistence was
still ahead. The transport now awaits a manager-installed pre-ACK hook that
registers a DETERMINISTIC owner lease (exactly one owner exists per
generation, and determinism is what lets the signal-time registrant and the
owner's catch-side release agree across processes) before the acknowledgement
is persisted or published; an owned same-replica abort bridges to the same
lease before tripping its local controller. Both generation-owner catches
(fresh turn, resume) release it once their writes land.
- A failed replacement handoff no longer orphans the predecessor. The atomic
replacement removes it from active storage, and an unconfirmed handoff
terminalizes the replacement too — leaving nothing a quiesce could discover
while the predecessor's provider may still be generating. Its owner lease is
now retained at the point the receipt fails delivery; the owner replica renews
it through the pre-ACK fence when the signal finally lands.
- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
holder; live persistence — a stalled save, a long deferred title — must never
outlive its own fence. holdUserFinalization registers and renews every minute
until released; the controllers' completion/error/admission leases all hold.
The undelivered-Stop retention moved to a deterministic `stop` lease that
every resignal attempt renews and the first successful one clears (no more
opaque leases accumulating to TTL), and a THROWN abort transition releases the
contender lease instead of leaking it.
- The user-document abort-fence mutations now invalidate the auth user-doc
cache, matching every other user-doc write.
Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.
* fix: one manager-owned owner-lease span across every abort delivery path
Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.
- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
the transport ACKed even when it failed; a same-replica owned abort likewise
swallowed registration failure. The hook now acquires a HELD owner lease
(heartbeat until the owner's catch releases it via releaseOwnerLease) and a
rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
behind its retention lease, and every resignal re-drives the handler. The hook
also stopped gating on `job.createdAt === generationId`: during a replacement
handoff the store holds the replacement while the abort targets the
predecessor, and that gate silently skipped exactly the generation being
acknowledged (the store job is owner identity, never a generation gate). A
local owned abort acquires the same held lease before tripping its provider;
post-CAS the trip cannot be withheld, so acquisition failure downgrades
delivery and the retention handoff keeps the user fenced.
- Committed-but-lost-reply disambiguation. A thrown abort transition released
the contender lease as if nothing had happened, but a Lua CAS can commit and
lose its reply — an aborted job invisible to active-set scans whose provider
was never signalled, with no fence left. The throw path now re-reads the exact
generation: only a job still live under the caller's identity proves no
commit; committed or ambiguous outcomes hand the fence to the deterministic
stop lease (kept on the contender lease if even that fails) before rethrowing.
- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
predecessor's held owner lease before the trip (failure reports the receipt
undelivered, engaging retention). Failed-handoff retention is no longer a
swallowed one-shot: it heartbeats with the durable acknowledgement proof as
its renewal predicate — acquisition failures keep retrying for as long as the
fence is needed, and the retainer stands down (without clearing the shared
field) once the remote owner ACKs and thereby holds its own lease.
- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
retained stop lease, and keeps the stop lease when that handoff fails.
Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.
* fix: finish scheduled chat lifecycle hardening
* fix: close scheduled chat review follow-ups
* fix: generation-fence abort recovery evidence
* test: wait for settled approval tool output
* test: preserve scheduled init reconciliation option
* refactor: rebuild scheduled chats on durable agent triggers
* test: reset MCP cache mock between cases
* test: isolate scheduler startup in server specs
* fix: harden scheduled run lifecycle
* fix: normalize schedule capacity conflicts
* test: type schedule collision fixture
* fix: re-fence scheduled resume and expiry
* fix: fence scheduled resume handoffs
* fix: release superseded manual schedule leases
* fix: release failed run-now claims
* fix: release superseded engine claims
* fix: repair schedule dialog interaction and rework its form
The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.
Alongside that:
- Extract the agent builder's instructions editor (special-variable menu plus
expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
schedule prompt. Insertions now route through `onChange`, so react-hook-form
sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
the selection with `bg-surface-hover` on an outline button whose hover is the
same token, so the selected option was indistinguishable from a hovered one;
`Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.
The new dialog spec fails when the portal fix is reverted.
* test: cover scheduled and subagent deletion drains
* refactor: own the form-control appearance in the client primitives
Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.
Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.
This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.
The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.
* fix: keep an explicit schedules disable from becoming an opt-in
`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.
Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.
Both regressions fail without the normalizer.
* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep
Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.
Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.
Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension
Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.
Replace the destructive marking with a REVERSIBLE, token-fenced suspension:
- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
cannot copy field values under DocumentDB), fenced per row so an already-
suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
re-arming only rows still carrying the exact attempt token and not independently
deleted — so an owner-deleted or newer-attempt-suspended schedule is never
resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
any failure that cancels the user-deletion fence restores the suspended rows. A
successful deletion hard-deletes them (and their snapshots) via the existing
cascade and never restores.
Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol
Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.
Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):
- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
acknowledges only after beforePublish persistence succeeds, and on a
persistence failure leaves the barrier unresolved so the run stays preserved
(client retries; stale-owner timeout is the bounded recovery). A failed abort
releases the stamp it placed so a replacement/retry is never blocked through
its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
proceeds immediately. The paused Stop settles only after its own ack.
Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation
Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.
Translate durable delivery state into the schedule outcome:
- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
(computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
admission is live, never orphan; dead → record `error` from the durable
lastError and release capacity promptly (no 30-minute wait), through the
ordinary outcome/auto-disable path; succeeded or no record → the existing
legacy orphan policy (interrupted only past the cutoff); a delivery lookup
failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
moderation middleware writes no schedule state.
Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions
Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.
Make the host lifecycle work durable rather than schedule-specific:
- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
fenced by generation identity (clearTerminalHostAction), so a replacement
generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
across restarts and replicas and retries the idempotent hook; the relay only
re-invokes while the marker is unacknowledged, so a successful ack prevents
duplicate work. Store-won expiry marks it too, so a loser-replica relay still
crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.
Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts
CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): repair the deletion CLI and order restore before the fence release
Addresses three findings from the fresh Codex review.
P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.
P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.
Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.
Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge dead deliveries in topology-safe maintenance
Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.
Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.
Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.
Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): refresh host-action retention on each retry attempt
Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.
Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.
Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): defer settlement when the Stop barrier times out
Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.
The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.
Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops
Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.
P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.
P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.
P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.
P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.
Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge terminal runs in clustered workers, retry suspension restore
Fourth Codex round on
|
||
|
|
259f1e0c32
|
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas * fix: initialize task routing in cluster workers * fix: harden cross-replica task routing * fix: expire routed task owners independently * fix: close cross-replica routing edge cases * fix: bound owner refresh and close routed cancellation gaps Refresh owned task registrations in bounded parallel batches so a full heartbeat pass stays well inside the 30-second directory lease instead of serializing one Redis EVAL per registration. Route conversation-deletion cancellation through a dedicated owner-side scope operation. The owner applies the deletion predicate to its complete local task set, so a scope holding more children than the model-facing list cap no longer leaves live executors running after their parent is removed. Key a consumed claim's retained response by its operation rather than by one caller's correlation id, so a later poll recovers a terminal result whose responses were all lost. Live claim statuses stay uncached so a poll always observes the task's current state. Type the model-facing `maxLength` bounds with a narrow local string schema; the SDK's JsonSchemaType does not declare the keyword, and the runtime checks continue to enforce the same limits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: retain claimed results apart from control replays A consumed claim is the only routed response whose loss destroys data, so it no longer shares one bounded cache with control replays that unrelated command traffic can evict. Claims are retained under their own budget, and the requester acknowledges a result it received so the owner releases the copy immediately instead of holding it for the full replay window. Resolve the post-delete cancellation pass from durable leases. The deleted conversations cannot be read back, so re-reading each one only scaled the cascade while probing the owner directory once per removed id; one lease read now resolves every live child address instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: never consume a result the owner cannot replay Retention for consumed claims is bounded, so a burst of undelivered results could evict an earlier one and lose it for good. The owner now admits a claim only while it can retain a worst-case result, and refuses the routed claim otherwise instead of consuming it, leaving the result on the task for a later poll. Retained claims are never displaced; control replays keep evicting. Key a control replay by the command itself rather than by one caller's correlation id. The transport's own retry reuses a single envelope, but a caller that saw the owner as unavailable reissues the command under a new id, which steered, queued, or interrupted the child a second time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: own a claimed result until it is acknowledged A consumed terminal result is task-owned state, not a cache entry. It now carries no expiry at all: the owner holds it until a caller acknowledges receipt, and only then is it released. Retention stays bounded by the existing admission gate, which refuses a claim the owner could not keep rather than consuming a result it might drop. Identify a control by the caller's invocation instead of by its content. The tool mints one id per invocation and routing carries it, so a routed retransmission of that invocation replays the owner's result while two deliberate identical commands arrive under distinct ids and both apply. Content-derived identity could not tell those apart and would have answered the second from a stale snapshot. Wait for the dpkg frontend lock in the best-effort Playwright font step. Its timeout kills npx while the apt-get it spawned keeps the lock, which then failed the fatal Redis install and ended the MCP replica jobs before any test ran (#14983). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: treat acknowledgement as part of delivering a result Publishing an acknowledgement once and ignoring the outcome meant a result could be reported as delivered while the owner never learned it could let go, and since that retention neither expires nor evicts, enough lost acknowledgements would fill it and refuse every later remote claim. An acknowledgement is now confirmed: publishing to zero subscribers is not success, it retries inside the ordinary request window, and a claim whose acknowledgement cannot be confirmed reports the retryable unavailable path instead of handing back a result the owner still holds. A later poll recovers that result and acknowledges it, and releasing is idempotent. Owner registration also outlives the task while a result is unacknowledged, so the retained result cannot become unreachable. Take the control invocation identity from the provider's tool-call id rather than minting one per execution, so replaying the same tool call stays idempotent while two distinct calls with identical payloads both apply. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * style: sort the widened node:crypto import Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: own control invocations and cancellation plans at the task seam Applies one logical control exactly once for its owning task rather than in the transport, so a local caller and a routed caller of the same invocation agree, and reusing an invocation id for different content is refused instead of silently applied. Invocation identity now comes from the run, agent, and provider tool-call id hashed to a bounded 32 characters, so a repeated `call_0` never bleeds across tasks and no id can overrun the routed bound. Cancellation for conversation deletion is now resolved into a plan while those rows are still readable, then replayed against the owner directory after the cascade is deleted. Owner registration is awaited before any provider work, so a child that cannot be addressed never starts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * style: separate the control invocation map from the next member Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: close subagent deletion, claim, and control invocation gaps Bulk conversation deletion now runs behind a durable owner admission fence. Draining alone could not close the race: a child admitted on another replica after the drain read its leases would start provider work against a parent about to disappear. The fence is written before any lease is read and each child revalidates it after its own lease is written, so one of the two always observes the other. It expires on its own, so a process lost mid-deletion cannot leave an account unable to run subagents. A terminal child result is no longer kept alive in the owning replica's memory until someone acknowledges it. Collection is recorded durably on the child's own message against the polling invocation, so the poll whose response was lost recovers its own result while a different invocation is told the result was already collected. Owner-side retention returns to an ordinary bounded cache that expires, which is what abandoned polls needed: they can no longer occupy claim capacity until the process restarts. The deletion drain now cancels each task under one invocation held for the whole drain, stops re-sending once the owner answers, and retries only deliveries it could not confirm. A routed control replay also validates the command fingerprint, so one invocation id carrying different content reaches the owner to be refused instead of collecting the earlier command's success. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: assert the drain's calls before restoring its spies Restoring a spy also clears its recorded calls, so the drain assertions ran against an emptied mock. Formats the durable claim method tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: close the follow-on gaps in the deletion fence and result claim The admission fence now carries an ownership token, so an overlapping deletion's fence is never lifted by the one that finishes first, and both fence writes invalidate the cached auth user document. It also covers the other bulk-delete path: `DELETE /` with no conversation filter removes every conversation, so it runs behind the same fence rather than a bare drain. The durable record now decides who holds a one-shot result. An owner replaying a retained response could hand the same terminal claim to a second invocation; that invocation is told the result was already collected, while the one that consumed it still recovers its own. A task with no durable record to arbitrate keeps whatever the owner answered. Drain cancellation treats `not_found` as unconfirmed: a missing registration while the durable lease is still live means the child may be running, so the command is retried under its invocation once the owner republishes itself. Control fingerprints are hashed, so retaining one per invocation costs a fixed few bytes instead of a bounded message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: hold every deletion fence and keep live idempotency records An owner now holds one admission fence per concurrent bulk deletion instead of one at a time, so admission reopens only when the last deletion finishes regardless of completion order. Expired fences are pruned as new ones arrive and the set is bounded, so an abandoned fence cannot accumulate or lock an account out. A failed durable claim write is no longer read as an absent record. Handing a terminal result over without recording its claimant would let another invocation collect the same one-shot output once the database recovered, so the collection reports the retryable unavailable path and leaves the result for a later poll. Control invocation records now evict tasks the store no longer holds before live ones, over a bounded scan. Dropping a live task's record would let a caller retry apply its queue, steer, or interrupt a second time once the transport replay had also expired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: keep the deletion fence portable and never drop a live record The admission fence is written with plain update operators again. DocumentDB rejects pipeline-form updates, and this runs before any deletion, so the pipeline form would have failed both bulk-delete endpoints outright on a supported database target. An excess deletion is now refused rather than silently displacing the oldest active fence, which would have reopened admission for a deletion still running. Expired fences are pruned before the cap is tested, so only genuinely concurrent deletions count against it. Control invocation records now sweep every settled task's entry when the window fills, and a window of entirely live records refuses the new control before touching the child instead of evicting one. Applying a command with no room to record it would let the caller's own retry apply it twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: hold the fence, bound recovered results, and expire stale commands The admission fence is renewed for as long as its deletion runs, so a very large account or a stalled database cannot let it lapse while conversations are still being removed. Only the deletion's own fence is renewed, and the renewal stops with the operation. Cancellation now covers every conversation the cascade removed, not only the ones a plan named: a grandchild lives in its own parent's scope, which a plan naming the deleted root never reaches. A routed request carries the deadline its caller waits for, and an owner drops one that arrives past it. A publisher disconnected mid-request queues the envelope offline and delivers it after the caller was told the owner was unavailable, which would otherwise steer a child the caller believes untouched. A result recovered from its durable child message is bounded like a routed one. The message keeps the child's untruncated output, so recovery could otherwise return far more than the routed result limit allows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: size the fence window so a renewal can be observed The renewal test set a 30ms drain timeout but the five-minute grace window dominates it, so the interval was 100 seconds and no renewal could fire inside the test's deletion. The grace window is an option now, matching the store's other timings, and the test sizes the window to 90ms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: wire the durable claim method and close the fence follow-ons The production store never received `claimSubagentTaskResult`, so every terminal result would have surfaced as unavailable once a task settled. The host wires that object from JavaScript, where the factory's parameter type checks nothing, so the factory now refuses a store missing any method it calls rather than failing at the first claim. The routing transport takes a dedicated publisher with the offline queue disabled. The shared client held commands issued during a disconnect and delivered them after the caller had given up, which the request deadline narrowed but could not close inside the clock-skew allowance. Fence renewal invalidates the cached auth document like the fence and release paths, and a renewal reporting its entry gone re-takes the fence instead of letting the deletion run on unfenced. The post-delete cancellation retries a transiently unreachable owner: the conversations are already gone, so it is the only pass that can still stop a late-admitted child. A replaced replay entry no longer leaves its bytes counted, which would have inflated the cache's total until unrelated responses were evicted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: wait on observed lease renewal instead of a fixed delay The shared-lease renewal test held a 60ms lease and slept 100ms before asserting an overlapping worker was refused, so a loaded runner that starved the 10ms heartbeat past the TTL let the lease lapse and the second worker run. Spy on acquisition and renewal, then wait until a renewal succeeds past the acquired lease's own deadline — direct evidence the heartbeat carried it past expiry, with no timing assumption — and give the lease enough headroom that a stalled timer no longer decides the outcome. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): close the routing, fence, and cache gaps found in review Five separate seams, each with its own failure: `Cluster.duplicate` reads its first argument as a startup-node list and its second as the overrides, unlike `Redis.duplicate`, so the publisher's `enableOfflineQueue: false` was silently dropped under `USE_REDIS_CLUSTER` and a command issued mid-disconnect could still reach a child after its caller was told `unavailable`. Route both through `duplicateIoRedisClient`. The control window's capacity refusal ran before the store knew whether it owned the task, so unrelated local load could veto a cancellation bound for another replica. Establish that the task is local first and leave a remote one to its owner's window. `clearInterval` stops only future fence renewals. One already waiting on the database could resolve after the release, read its own lifted fence as expiry, and write a replacement that nothing remained to lift — closing subagent admission for the account until it aged out. Track the in-flight renewal, refuse overlapping passes, and await it before releasing. Every owner bounds its own task list, but the aggregation appended each batch whole, so the model-facing list grew with the number of replicas holding the scope. Cap the merged list while still reading every reply for the stale-registration sweep. The admission-fence prune commits independently of the fence that follows it, so a refused or failed push left the cached auth document describing entries the collection no longer held. Invalidate whichever way the second write goes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): cap the merged task list the poll tool actually reads Each owner bounds its own reply and the remote aggregation bounds their sum, but `listTasks` merged that bounded remote list with however many children this replica owns and returned it whole. `check_background_task` could therefore still receive roughly twice the advertised cap. Bound the deduplicated, sorted result and export the cap so both seams share one number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: admit every task the merged-list cap test starts The base store admits ten concurrent runs per scope by default, so starting 150 at once left most refused for capacity and the assertion never reached the merge it was written to check. Raise the cap for this store only; admission is a different invariant with its own tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): let a deletion notice its admission fence lapsing Renewal failures were logged and swallowed, so a run of rejected writes let the last confirmed `fencedUntil` pass while the deletion carried on believing admission was still closed — long enough for another replica to admit a child against conversations about to be removed. Track the deadline only a confirmed write advances, and check it after the drain, before anything is deleted: nothing has been removed at that point, so the operation fails closed and the caller retries once the fence can be held. A lapse detected after the rows are gone is logged instead, since reporting failure there would invite a retry against conversations that no longer exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: raise both concurrency caps the merged-list test trips Raising the per-scope limit left the store-wide `maxRunningTotal` at its default hundred, so fifty of the hundred and fifty starts were still refused. Verified against the base store directly this time: with only the per-scope cap raised it admits a hundred, and with both raised it admits all hundred and fifty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): close the fence renewal gap and keep running tasks listed A renewal that started before its deadline but landed after it was still credited with extending the fence from its own start time, so a window in which admission stood open was papered over: a child could take a lease the drain had already read past and the deletion would proceed without cancelling it. The deadline now only advances when the write lands while the previous one still holds; anything later records a lapse the fence cannot be restored backwards over. The model-facing cap sorted oldest-first and sliced, which dropped the newest tasks — including children that had only just started running, and which the poll tool offers no other way to discover. Bound by status instead: running children first, then the most recent settled results. Both caps share one helper, and the routed aggregation now bounds after its loop so the choice is made across every owner's reply rather than by whichever answered first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): finish the cap and the fence at the seams they still missed The status-aware cap only reached the requester: an owner's own reply still sliced positionally, so a replica holding more than the cap dropped its running children before the requester could bound anything. Both sides now share `boundedTaskList`. A fence that lapsed during the deletion itself was only logged. The rows are gone by then, so failing is still wrong, but the child another replica admitted while the fence was down is not: the fence is retaken and the drain repeated to cancel it. A child's lease renewal had the same retroactive hole the admission fence had — Mongo filters on the `now` captured before the call, so a write landing after the lease expired still moves the row forward, while an owner drain reading active leases in that gap saw the thread as free. The lease now carries its own deadline and a late renewal stops the executor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: cover the lease lapse and the post-deletion re-drain The owner-side cap shipped with a regression test; these two did not. One drives a lease renewal that succeeds only after the lease it was extending had expired and asserts the executor stops; the other lets the fence lapse during the deletion itself and asserts a second drain runs while the request still reports success. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix(agents): close live-task lifecycle gaps * test(redis): exercise cluster node discovery * fix(test): type cluster discovery seam * fix(ci): wait for orphaned apt processes * fix(ci): reserve time for apt drain * fix(ci): skip optional fonts in MCP jobs * fix(agents): recover tasks after owner loss * fix(agents): preserve local task discovery * fix(agents): initialize fail-fast cluster publisher * style(agents): sort routing test imports --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
006e421cd2
|
💡 feat: add DB-backed admin insights (#14898)
* feat: add Mongo-backed admin insights * feat: gate insights with environment variable * fix: tighten insights access and activity metrics * fix: preserve insights date selections * perf: parallelize insights search aggregation * test: wait for MCP conflict recovery * test: satisfy strict MCP recovery typing * fix: disable insights pagination while loading * fix: localize insights range shortcuts * fix: bound insights search input |
||
|
|
aa35cd42b1
|
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch * feat: add durable agent trigger delivery * fix: annotate trigger envelope byte limit * test: isolate trigger startup in server specs * fix: fence trigger delivery during account deletion * test: isolate trigger service in user controller specs * fix: close trigger deletion admission race * fix: harden account deletion fences * fix: close durable trigger review gaps * fix: require offline stale-fence recovery * fix: type trigger lane sequence ids * fix: fence admin user deletion triggers * fix: make trigger deletion recovery durable * fix: harden offline user deletion * fix: serialize trigger lane publication * style: sort trigger delivery imports * fix: recover orphaned trigger publications * fix: preserve trigger recovery ordering * fix: fence trigger publication during purge * fix: defer remote trigger deletion fences * fix: close durable delivery cleanup races * fix: drain CLI generation owners before deletion |
||
|
|
f829aca9fb
|
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution * fix: Preserve Operator-Owned MCP Entries * fix: Preserve Configuration Source Ownership * style: Normalize Middleware Import Order * fix: Preserve Process Server Precedence * test: Align Tenant-Aware E2E Setup |
||
|
|
1596df724a
|
chore: Remove Published Credential Defaults (#14680) | ||
|
|
0e14d91ed9
|
⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine (ReDoS) (#14555)
* ⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine convertStringsToRegex compiled admin-configured supportedMimeTypes with the native RegExp engine, and checkType runs those patterns against an uploaded file's Content-Type on the server event loop, so a catastrophic-backtracking pattern in fileConfig could ReDoS the whole process on upload. The MIME-pattern compiler is now swappable. It defaults to native RegExp, which browser builds keep so no engine is added to the client bundle, and the server injects a linear-time engine (RE2JS) at startup. Only test is ever called on these matchers, so the shared type widens to a structural RegexLike with no behavior change for valid patterns. The browser stays on native because a client-side stall would only affect that one tab. * ⏱️ fix: Wire the linear MIME compiler in the experimental entry point api/server/experimental.js mounts the same upload routes and calls mergeFileConfig but never set the linear-time compiler, so admin MIME patterns still compiled with native RegExp there. Mirror the setup, and widen the client-side supportedMimeTypes type to the shared RegexLike so the browser typechecks against the same structural matcher. * 🧹 refactor: Configure the file-config linear engine from a shared helper Move the RE2 wiring out of both JS server entry points into a single configureFileConfigRegexEngine helper exported from @librechat/api, so /api stays a thin caller and the setup no longer has to be kept in sync across index.js and experimental.js. Also warn loudly when compiling an endpoint's supportedMimeTypes drops every pattern (an empty allowlist would reject all uploads), and correct the isMimeTypeSupported docstring to say RegexLike rather than RegExp. * fix: fail closed when every MIME pattern fails to compile convertStringsToRegex returned [] when all configured patterns failed to compile, and filter.ts reads an empty allowlist as no restriction, so a restrictive config whose patterns all fail allowed every attachment. Return a single reject-all matcher instead so every consumer fails closed. |
||
|
|
dd159c4566
|
🔐 fix: Preserve Structured JWT Auth Context (#14652)
* 🔐 fix: Preserve structured JWT auth context
* fix: Omit identity from auth correlation logs
* fix: Isolate pre-auth request context
* style: Sort auth context imports
* fix: Preserve structured auth metadata
* fix: Narrow structured log formatter types
* fix: Annotate structured log context keys
* test: Fix request fixture typing
* fix: Namespace request path log context
* fix: Namespace request method log context
* fix: Preserve captured tenant error paths
* style: Sort tenant error imports
* fix: Classify bulk tenant isolation failures
* fix: Enforce safe request correlation invariants
|
||
|
|
3f0a1ec8d9
|
🛡️ fix: Run message-filter PII patterns on a linear-time regex engine (ReDoS) (#14554)
* 🛡️ fix: Run message-filter PII patterns on a linear-time regex engine The messageFilter.pii middleware compiled admin-configured customPatterns with the native RegExp engine and ran them synchronously against every message on the shared event loop, so a catastrophic-backtracking pattern such as (a+)+$ could stall the entire process (native RegExp takes tens of seconds at roughly 32 characters) and take the instance down for every user. Compile these patterns with RE2JS, a linear-time RE2 port with no native addon, so catastrophic backtracking is impossible regardless of the pattern rather than something the code tries to detect. Patterns using features RE2 does not support, such as backreferences, fail to compile and are dropped and logged exactly as an invalid pattern already is. The filter only tests for a match, so this is a drop-in engine swap with no behavior change for valid patterns. * 🛡️ fix: Reject RE2-incompatible messageFilter patterns at config load The customPatterns regex was validated with native RegExp at config load, but the runtime now compiles it with a linear-time engine (RE2) that does not support backreferences or lookaround. Such a pattern passed validation, then failed to compile and was silently dropped at request time, quietly removing PII protection after upgrade. Reject backreferences and lookaround during config validation with an explicit message, and document RE2 syntax in the example config instead of "JavaScript-flavor". The runtime engine remains the authoritative boundary and still drops-and-logs anything this load-time check misses. * 🧹 test: Use direct MessageFilterPiiConfig annotations in the PII specs The added ReDoS cases satisfy the exported MessageFilterPiiConfig type directly, so the `as unknown as` assertions were unnecessary. Annotate the config objects directly, matching the repo's type-safety guidance. * 🧹 fix: Reject named backreferences in messageFilter patterns at config load Extend the config-load check to also reject named backreferences (\k<name>), which are valid JavaScript regex but unsupported by the linear-time runtime engine, so they surface at load rather than being dropped at request time. Together with the existing numeric-backreference and lookaround checks this covers the RE2-incompatible construct set; the runtime engine remains authoritative. * 🛡️ fix: Preserve Unicode whitespace matching in messageFilter starter patterns RE2's \s is ASCII-only, so after the engine swap the built-in api-key and Bearer starters no longer matched a secret separated by non-ASCII whitespace (e.g. a non-breaking space), which native RegExp did match. Broaden the whitespace classes to [\s\p{Zs}] so those patterns keep their original coverage, and add a regression test for a non-breaking-space separator. * 🛡️ fix: Validate messageFilter patterns with the RE2 engine at config load Replace the syntax blacklist (numeric/named backreferences, lookaround) with authoritative validation: config load now compiles each custom pattern with the same linear-time engine the runtime uses, so any RE2-incompatible construct (including control escapes like \cA) is rejected at load with a clear error instead of being silently dropped at request time. The validator is swappable and defaults to native RegExp so browser builds add no engine; the server wires the RE2-backed check at startup via configureMessageFilterRegexValidator in both entry points. * 🛡️ fix: Match the full whitespace set in messageFilter starter patterns RE2's `\s` omits the vertical tab and `\p{Zs}` omits U+2028, U+2029, and U+FEFF, so a separator built from one of those characters slipped past the `api-key` and `Bearer` starter patterns and reached the model. Broaden the starter whitespace class to the full JavaScript whitespace set so those separators are covered again. * fix: fail closed when messageFilter.pii compiles to zero patterns DB and admin config overrides bypass the RE2 schema validation (it only runs at YAML load), so an override whose only pattern is RE2-incompatible was dropped at compile time, left zero patterns, and let the request through. compile() now returns a failClosed flag when a config declared patterns but every one failed to compile; the middleware returns 400 and findPiiMatchInMessages returns a distinct misconfigured match that the OpenAI and Responses controllers surface with an admin-facing message. * 🛡️ fix: Fail closed when any messageFilter.pii custom pattern drops compile() previously set failClosed only when every pattern dropped (patterns.length === 0 && dropped > 0). With the default starters present, a single RE2-incompatible custom override incremented dropped but left patterns.length > 0, so the filter silently enforced only the surviving subset and text matching only the dropped rule passed. failClosed now keys off dropped > 0, so any dropped custom pattern blocks with the misconfigured 400. YAML patterns are RE2-validated at load, so dropped stays 0 for valid configs and only unvalidated DB or admin overrides can trip it. Reframed the two keeps-others-active specs to assert fail-closed and added a default-starters partial-drop regression. * 🧹 fix: Correct the misconfigured JSDoc and drop redundant casts in the PII specs The misconfigured flag now means any configured custom pattern failed to compile, not that every pattern failed, so its JSDoc on PiiMatch is updated to match. The partial-drop regressions now use direct MessageFilterPiiConfig annotations instead of as-unknown-as casts, keeping the specs type-checked, consistent with the rest of the suite. |
||
|
|
324584552c
|
⏱️ feat: Configurable HTTP Server Timeouts (#14481)
* http server config added
* Fix TypeScript compatibility by accepting NodeJS.ProcessEnv directly when applying optional HTTP server timeout configuration.
* fix(api): configure HTTP server timeouts for clustered workers
* 🕰️ fix: Warn When HTTP Timeouts Are Not Enforced
Codex review of the rebased contributor work surfaced two ways these settings
silently do nothing. Both reproduce, and neither was reported to the operator.
Bun accepts the four property assignments and reflects them back, but does not
enforce them: with keepAliveTimeout=100 and buffer=1000, Bun 1.3.13 held a
keep-alive connection past 3s where Node 24 closed it at 1101ms. Since `b:api`
runs the server under Bun, the existing info log confirmed a configuration that
was not in effect. Warn instead.
Node sweeps header/request timeouts on `connectionsCheckingInterval`, a
createServer option that `app.listen()` leaves at 30s, so sub-30s values round
up to it: headersTimeout=2000 returned 408 at 30004ms by default versus 2010ms
with a 250ms interval. Warn on values below the sweep interval rather than
restructure server construction, since every documented value and both Node
defaults already sit well above it. keepAliveTimeout is socket-driven and stays
exact, so it is excluded.
Both caveats documented in .env.example.
* 🩹 fix: Inject Runtime Versions Instead of Mutating `process.versions`
The spec deleted `process.versions.bun` to reset between cases, which failed
typecheck with TS2790: `@types/bun` is a packages/api dependency and augments
NodeJS.ProcessVersions with a required `bun: string`, so the property is not
optional and cannot be deleted. Assigning undefined would fail for the same
reason.
That augmentation also made the production check dishonest: TypeScript saw
`process.versions.bun` as always a string, so `!= null` read as a no-op branch
even though it is correct at runtime under Node.
Both resolved by taking runtime versions as a third injectable parameter,
matching the existing `environment` parameter. Callers in api/server are
unchanged, the narrow `{ bun?: string }` type restores honest narrowing, and
the tests no longer mutate global state, so they assert the same behavior
whether the suite runs under Node or `bun jest`.
* 📏 fix: Stop Claiming a Ceiling on Sweep-Delayed Timeouts
The warning added in
|
||
|
|
73699b5c25
|
⚡ perf: Reduce Agent Chat Startup Latency (#14423)
* perf: reduce agent chat startup latency * test: align Redis stream readiness assertions * perf: overlap remaining agent startup work * perf: persist initial agent job metadata atomically * test: add agent startup latency benchmark * fix: harden resumable agent stream lifecycle * fix: isolate replacement stream lifecycles * fix: preserve terminal stream epochs |
||
|
|
84fa6aa820
|
🧹 feat: Eager HITL Checkpoint Cleanup (Expiry + Deletion) & Full-Wiring E2E (#14123)
* feat: eager HITL checkpoint cleanup on expiry + deletion, full-wiring e2e Follow-up to the lazy checkpointer (#14024): two paths still left a paused run's durable checkpoint to the 24h Mongo TTL, and no test exercised the whole HITL seam with real components. 1. Approval expiry: GenerationJobManager.setApprovalExpiredHandler(fn) — a non-destructive host hook fired after expireApproval's CAS succeeds (periodic sweeper AND stale-submit path), safe on startups that run constructor defaults. Both startups (index.js configureGenerationStreams, experimental.js) register a handler that prunes the checkpoint, resolving config lazily per expiry (streamId === conversationId === thread_id). 2. Conversation deletion: deleteConvos now returns the deleted conversationIds; the three deletion paths (DELETE /convos, DELETE /convos/all, account deletion) prune them via the new bulk deleteAgentCheckpoints (one $in deleteMany per collection). The delete routes gain configMiddleware for the checkpointer config. 3. Full-wiring e2e (hitlCheckpoint.e2e.spec.js): real SDK Run driven by FakeChatModel calling a gated tool, real PreToolUse/humanInTheLoop wiring, real LazyMongoSaver over mongodb-memory-server, real GenerationJobManager, real /resume controller via supertest. Asserts: clean turn persists nothing; error turn persists nothing; pause -> HTTP approve -> gated tool executes exactly once -> finalize prunes the checkpoint; expiry prunes the abandoned pause eagerly. Tests: 3 handler unit tests (pendingAction.spec), 2 bulk-prune integration tests (checkpointer.integration.spec), convos route + deleteUser specs updated, 4 e2e scenarios. 207 tests green across changed areas. * fix: tenant-scoped expiry prune, store-won expiry relay, resilient deleteConvos ids Codex round 1 on #14123 — all three valid: 1. The approval-expired handler now receives the expired JOB so both startups resolve config in the paused job's tenant/user scope (getAppConfig({userId, tenantId})) — a tenant checkpointer override no longer sends the prune to the base config's collections. expireApproval fetches the job best-effort. 2. Multi-replica: when RedisJobStore.cleanupRequiresActionIndex wins the expiry CAS on another replica, this replica's sweeper relay branch now runs the approval-expired cleanup too (prune is idempotent) — store-driven expiry no longer bypasses the hook. 3. deleteConvos: post-delete cleanup (deleteMessages, project stats refresh) is now best-effort — the conversations are already gone, so throwing hid the deletion and dropped the conversationIds the checkpoint prune needs, unrecoverable on retry. Updated the existing tag-decrement-on-failure test to the new contract (ids still returned). Tests: handler-receives-job, store-won relay path, ids-survive-cleanup-failure. 134 tests green across changed suites. * fix: relay cleanup independent of cached errorEvent; enter tenant ALS context Codex round 2 on #14123: 1. The sweeper's relay branch gated BOTH the terminal-error emit and the new checkpoint cleanup on !runtime.errorEvent — but a reconnect seeds errorEvent from the aborted job (runtime-state creation), which then suppressed the cleanup entirely. The emit stays gated; the idempotent cleanup now runs independent of the cached error, once per runtime lifetime (approvalCleanupRan flag — the aborted job is swept repeatedly). 2. Passing userId/tenantId to getAppConfig only keys the config cache; the Config query is ALS-scoped by the tenant-isolation plugin. Both startup handlers now ENTER the paused job's tenant context via tenantStorage.run before resolving config + pruning, so a tenant checkpointer override is honored in strict and non-strict modes. Tests: relay-cleanup-with-cached-error (reconnect simulation), repeated sweeps run the cleanup once. 27+4 green. * fix: dedup expiry cleanup across winner and relay paths Codex round 3 (P3): expireApproval ran the handler without marking the runtime's approvalCleanupRan flag, so the next sweep's relay branch (the aborted job outlives expiry for the completed-job TTL) ran the cleanup a second time. The dedup now lives inside runApprovalExpiredHandler — the single choke point both paths call — set-before-run, once per runtime lifetime. Test: local expiry followed by a sweep fires the handler once. |
||
|
|
424ccffd83
|
🪝 feat: Configurable Tool-Approval Policy via Programmatic Hooks (#14025)
Some checks failed
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Has been cancelled
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
* 🪝 feat: Programmatic tool-approval hook seam (configurable beyond on/off) Adds a process-wide registry so host code can plug context-aware PreToolUse decision hooks into the tool-approval policy, composing with the static `endpoints.agents.toolApproval` config instead of replacing it. - `registerToolApprovalHook(factory, { matcher? })` — register a factory that builds a PreToolUse hook per run from a ToolApprovalHookContext (userId, conversationId, tenantId, appConfig); return undefined to opt the run out. Returns an unregister fn. - `buildHITLRunWiring(policy, context)` now registers the static-config policy hook as the baseline, then layers each resolved host hook after it. Decisions fold in the SDK as deny > ask > allow, so a host hook can only TIGHTEN a configured ask/deny — it can never silently auto-approve past policy (to loosen, change the static policy). updatedInput / allowedDecisions follow the SDK's last-writer-wins, so host hooks win over the baseline. - `createRun` threads the per-run context (user / conversation / tenant / appConfig) into the wiring; non-HITL and HITL-disabled runs never invoke any factory. This unlocks dynamic policy the static name-lists can't express — per-args (e.g. ask before write_file outside a workspace, the SDK's createWorkspacePolicyHook shape), per-agent, per-user. Inert until tool approval is enabled and the caller is hitlCapable. Tests: registry register/unregister/opt-out/order (hooks.spec.ts) + wiring composition, context passthrough, and disabled-path inertness (runtime.spec.ts). Full HITL suite green. * 🪝 feat: Config-driven tool-approval hook loader (librechat.yaml → hook modules) Lets operators declare programmatic tool-approval hooks in config instead of code, so the registerToolApprovalHook seam is usable without a custom build. - Config (data-provider): `endpoints.agents.toolApproval.hooks[]`, each entry `{ module, matcher?, options? }`. `module` is a bare package name or a path (resolved against the app root); its default export is a builder `(options?) => ToolApprovalHookFactory`. - Loader (@librechat/api `loadToolApprovalHooks`): imports each module, builds the factory with the entry's options, and registers it (with its optional tool-name matcher). Reload- safe (each call first unregisters its previous batch, leaving code-registered hooks alone) and robust — an unimportable module / non-function export / throwing builder is logged and skipped, never crashing startup or blocking the other hooks. Importer is injectable for tests. - Startup (api/server/index.js): loads the configured hooks once after appConfig resolves. SECURITY: modules are dynamically imported + executed in-process; this is admin-level config, documented as trusted-code-only. Tests: 9 loader cases (default/no-default export, options passthrough, bad-export skip, builder-returns-non-function skip, import-failure resilience, continue-past-bad-entry, reload de-dup). Full HITL suite green (80). * 💄 style: Sort imports in HITL hook spec files (CI sort-imports:check) * 🛡️ fix: Harden tool-approval hook loader (Codex review) Six P2 findings on the hook loader / startup wiring: - CJS/transpiled interop: unwrap a nested `default` (TS/Babel `exports.default = fn` surfaces through import() as `{ default: { default: fn } }`) before rejecting a module, so documented default-export hook modules actually load. - Validate the matcher regex at load time and skip invalid ones — the SDK compiles it with `new RegExp` at run-build time, where a bad pattern would throw out of buildHITLRunWiring and break EVERY HITL run instead of just skipping the one bad hook. - Honor the `enabled` kill switch: startup now passes hooks to the loader only when toolApproval is enabled, so a disabled endpoint imports/runs nothing (and unregisters any prior batch). - Resolve app-root-relative paths without a leading dot: a bare specifier that is a real file under basePath (e.g. `config/hooks/workspace.js`) resolves as a path; scoped/other bare names still import as packages. - Base-config-only: documented that hooks register once process-wide at startup and are NOT reloaded from per-role/user/tenant overrides — encode per-tenant logic inside the hook. - Wire the loader into the clustered startup path (api/server/experimental.js) too, not just the standard server. Tests: CJS-interop unwrap, invalid-matcher skip (+ sibling still loads), and specifier resolution (app-root file / bare package / ./relative). Full HITL suite green. * 🛡️ fix: Read tool-approval hooks from base config in clustered startup (Codex) The clustered experimental.js path read toolApproval from getAppConfig() (which merges DB __base__ overrides with no principal), so a DB override could enable/disable/replace toolApproval.hooks and import those modules in every worker — violating the base-config-only contract and diverging from the standard server. Fetch getAppConfig({ baseOnly: true }) specifically for the hook loader, matching api/server/index.js. |