mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-21 15:33:45 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
784582850a
|
🧯 fix: Guard SSE Writes and Interval Cleanups Against Crashes (#17462)
* fix: Harden Timer Sweeps, SSE Writes, and Violation Counters Guard SSE writes against ended or destroyed responses through one helper, catch and single-flight the job store and TTL cache cleanup timers, serialize in-process counter updates, cap the violation log, prune stale OAuth failure records, batch the active-job lookup, and count Redis keys with DBSIZE. * refactor: Drop the redundant violation log entry copy * refactor: Log single-flight sweep failures from the API package * test: Cover active generation listing and client disconnect mid-stream * refactor: Configure violation log retention and move title timeout into the API package * feat: Configure OAuth failure retention through mcpSettings * fix: Prune OAuth failures promptly under short retention |
||
|
|
56301c1b5a
|
🪚 fix: Count Large Prompts in Bounded Tokenizer Chunks (#16305)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Tokenize budgeting inputs in bounded chunks instead of charging a token per UTF-8 byte above 4 KiB. Subdivide long whitespace/non-whitespace runs to bound BPE merge work and preserve surrogate pairs. Keep strict provider-accounting counts and unavailable-tokenizer fallbacks separate. Cover both encodings, SDK slice composition, programmatic tool prompts, summarization, real overflow, cache reuse, and failure recovery. Related to #16278. Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
e95e696d70
|
🧾 fix: Count the Tool Results a Tool-Limit Stop Retains (#15893)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 🧾 fix: Count the Tool Results a Tool-Limit Stop Retains Context snapshots reach the client only through the SDK's pre-invoke `ON_CONTEXT_USAGE`, so the results of the tools a call requests are never in that call's snapshot — the next call's snapshot carries them as kept-message context. A run that stops at the tool-call limit makes no next call, so the tool result it retains lives in the response and in no snapshot: the gauge reported `(budget − remaining) + completedOutputTokens` and left the retained result out of used tokens and out of the tool-call share until the following turn. The save path now counts those results with the run's own tokenizer and persists them as `retainedToolTokens`, a second post-snapshot delta alongside `completedOutputTokens` rather than a number folded into the provider-reconciled `messageTokens`. `resolveRetainedToolTokens` owns the rule that only a tool-limit stop retains anything, and the snapshot handler records where its content ended so the count starts at the right boundary. Counting had to avoid `Tokenizer.getTokenCount`, whose fallbacks would have put a guess inside exact accounting: above 4 KiB it returns byte length, several times the real count on ordinary text, and it estimates from character length while an encoding loads. `countExactTokens` tokenizes in bounded slices cut on code-point boundaries and returns nothing at all when the encoding is cold, so an uncountable result withdraws the figure instead of inflating it. The client adds the field to used tokens, subtracts it from the runway headroom and widens the tool-call share, in the live snapshot after finalization and in the persisted blob after a reload. * 🧹 style: Wrap the Retained-Counter Assertion as Prettier Requires * 🧮 fix: Address the Review of the Retained-Tool Count Three findings from the first round, each a real defect in how the figure was produced rather than a style point. The boundary was a content index recorded mid-run, but completion reshapes the array — skill cards are unshifted onto the front and `hide_sequential_outputs` replaces it with a filtered one — so a saved index no longer means the same position. The snapshot now records the tool-call ids it already accounts for, and the save path counts the results of the calls missing from that set: ids survive every reshape, and a filtered-away call is correctly left out. Counting in 4 KiB slices was not exact either: a BPE merge spanning a seam is charged twice, measured at ~1 token per slice, and the field exists precisely to be an exact addend. `countExactTokens` now tokenizes the whole input — ~60 ms/MB, paid once at the end of a stopped turn — and refuses content past 8 MiB rather than estimating it. The counter takes its exact-count function instead of reaching for the tokenizer singleton, so `resolveRetainedToolTokens` owns the default (the run's own encoding) and a caller or test can supply another. That also removes the mock of global state from the specs. `compactionReclaim` now includes the retained result in the total it subtracts the kept exchange from. `latestExchangeTokens` already counts that result on the other side, so leaving it out subtracted content the total never carried and understated the savings — to zero on a large final result. * 🧯 fix: Bound One Turn's Retained-Result Tokenization The tokenizer refuses a single result past 8 MiB, but a final call that requested several tools in parallel would pay that bound once per result. The counter now holds a budget for the whole turn and withdraws its figure past it, so the save path cannot be made to tokenize an unbounded pile of output. * 🎚️ feat: Configure the Retained-Result Tokenization Budget The exact count the gauge adds costs ~60 ms/MB of retained tool output, and the ceiling on that work was hard-coded in two places. It is now one lever: `endpoints.agents.maxRetainedToolCountChars`, defaulting to the 8 MiB that reproduces today's behavior, shared by the schema and the save path through `DEFAULT_MAX_RETAINED_TOOL_COUNT_CHARS`. Deployments whose tools legitimately return more can raise it; slower hardware can lower it, or set `0` to withhold the figure entirely. `Tokenizer.countExactTokens` no longer carries a bound of its own — the caller owns the budget — and `resolveRetainedToolTokens` passes the configured value to the counter, which spends it across all of a final call's parallel results. --------- Co-authored-by: Danny Avila <danny@librechat.ai> |
||
|
|
6d8b1cb013
|
🧢 fix: Bound Tokenizer Work Per Input (#15246) | ||
|
|
77cb72e50c
|
🧮 perf: Enable Agent Context Count Reuse (#15130)
* perf: enable agent context count reuse * fix: declare token counter return type * fix: initialize cached token counters * style: sort token counter imports * fix: Keep cached token counts exact |
||
|
|
6bc75d24c8
|
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the @librechat/api package build, mirroring the merged data-schemas migration. - Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps, bundle first-party `~/` + relative imports) - Annotate exports for isolatedDeclarations (codefix-driven). Collapse the tokens.ts model->token maps to Record<string, Record<string, number>> and switch validation.ts's runtime `files` field from z.any() to z.unknown() so no explicit `any` is introduced - Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output - Add src/telemetry.ts entry shim so the two index.ts entries don't collide in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts}) - Delete rollup.config.js Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass, both entries load via require(), legacy /api consumes them unchanged. * 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys The build-api cache keys hashed `packages/api/server-rollup.config.js`, which never existed (api used `rollup.config.js`, now removed) — a copy-paste artifact from the data-provider key that matched no file. Replace it with the new `packages/api/tsdown.config.mjs` so edits to the build config (entry, format, externals) bust the api build cache, matching the data-schemas key. |
||
|
|
9a5d7eaa4e
|
⚡ refactor: Replace tiktoken with ai-tokenizer (#12175)
* chore: Update dependencies by adding ai-tokenizer and removing tiktoken - Added ai-tokenizer version 1.0.6 to package.json and package-lock.json across multiple packages. - Removed tiktoken version 1.0.15 from package.json and package-lock.json in the same locations, streamlining dependency management. * refactor: replace js-tiktoken with ai-tokenizer - Added support for 'claude' encoding in the AgentClient class to improve model compatibility. - Updated Tokenizer class to utilize 'ai-tokenizer' for both 'o200k_base' and 'claude' encodings, replacing the previous 'tiktoken' dependency. - Refactored tests to reflect changes in tokenizer behavior and ensure accurate token counting for both encoding types. - Removed deprecated references to 'tiktoken' and adjusted related tests for improved clarity and functionality. * chore: remove tiktoken mocks from DALLE3 tests - Eliminated mock implementations of 'tiktoken' from DALLE3-related test files to streamline test setup and align with recent dependency updates. - Adjusted related test structures to ensure compatibility with the new tokenizer implementation. * chore: Add distinct encoding support for Anthropic Claude models - Introduced a new method `getEncoding` in the AgentClient class to handle the specific BPE tokenizer for Claude models, ensuring compatibility with the distinct encoding requirements. - Updated documentation to clarify the encoding logic for Claude and other models. * docs: Update return type documentation for getEncoding method in AgentClient - Clarified the return type of the getEncoding method to specify that it can return an EncodingName or undefined, enhancing code readability and type safety. * refactor: Tokenizer class and error handling - Exported the EncodingName type for broader usage. - Renamed encodingMap to encodingData for clarity. - Improved error handling in getTokenCount method to ensure recovery attempts are logged and return 0 on failure. - Updated countTokens function documentation to specify the use of 'o200k_base' encoding. * refactor: Simplify encoding documentation and export type - Updated the getEncoding method documentation to clarify the default behavior for non-Anthropic Claude models. - Exported the EncodingName type separately from the Tokenizer module for improved clarity and usage. * test: Update text processing tests for token limits - Adjusted test cases to handle smaller text sizes, changing scenarios from ~120k tokens to ~20k tokens for both the real tokenizer and countTokens functions. - Updated token limits in tests to reflect new constraints, ensuring tests accurately assess performance and call reduction. - Enhanced console log messages for clarity regarding token counts and reductions in the updated scenarios. * refactor: Update Tokenizer imports and exports - Moved Tokenizer and countTokens exports to the tokenizer module for better organization. - Adjusted imports in memory.ts to reflect the new structure, ensuring consistent usage across the codebase. - Updated memory.test.ts to mock the Tokenizer from the correct module path, enhancing test accuracy. * refactor: Tokenizer initialization and error handling - Introduced an async `initEncoding` method to preload tokenizers, improving performance and accuracy in token counting. - Updated `getTokenCount` to handle uninitialized tokenizers more gracefully, ensuring proper recovery and logging on errors. - Removed deprecated synchronous tokenizer retrieval, streamlining the overall tokenizer management process. * test: Enhance tokenizer tests with initialization and encoding checks - Added `beforeAll` hooks to initialize tokenizers for 'o200k_base' and 'claude' encodings before running tests, ensuring proper setup. - Updated tests to validate the loading of encodings and the correctness of token counts for both 'o200k_base' and 'claude'. - Improved test structure to deduplicate concurrent initialization calls, enhancing performance and reliability. |
||
|
|
8bdc808074
|
⚡ refactor: Optimize & Standardize Tokenizer Usage (#10777)
* refactor: Token Limit Processing with Enhanced Efficiency - Added a new test suite for `processTextWithTokenLimit`, ensuring comprehensive coverage of various scenarios including under, at, and exceeding token limits. - Refactored the `processTextWithTokenLimit` function to utilize a ratio-based estimation method, significantly reducing the number of token counting function calls compared to the previous binary search approach. - Improved handling of edge cases and variable token density, ensuring accurate truncation and performance across diverse text inputs. - Included direct comparisons with the old implementation to validate correctness and efficiency improvements. * refactor: Remove Tokenizer Route and Related References - Deleted the tokenizer route from the server and removed its references from the routes index and server files, streamlining the API structure. - This change simplifies the routing configuration by eliminating unused endpoints. * refactor: Migrate countTokens Utility to API Module - Removed the local countTokens utility and integrated it into the @librechat/api module for centralized access. - Updated various files to reference the new countTokens import from the API module, ensuring consistent usage across the application. - Cleaned up unused references and imports related to the previous countTokens implementation. * refactor: Centralize escapeRegExp Utility in API Module - Moved the escapeRegExp function from local utility files to the @librechat/api module for consistent usage across the application. - Updated imports in various files to reference the new centralized escapeRegExp function, ensuring cleaner code and reducing redundancy. - Removed duplicate implementations of escapeRegExp from multiple files, streamlining the codebase. * refactor: Enhance Token Counting Flexibility in Text Processing - Updated the `processTextWithTokenLimit` function to accept both synchronous and asynchronous token counting functions, improving its versatility. - Introduced a new `TokenCountFn` type to define the token counting function signature. - Added comprehensive tests to validate the behavior of `processTextWithTokenLimit` with both sync and async token counting functions, ensuring consistent results. - Implemented a wrapper to track call counts for the `countTokens` function, optimizing performance and reducing unnecessary calls. - Enhanced existing tests to compare the performance of the new implementation against the old one, demonstrating significant improvements in efficiency. * chore: documentation for Truncation Safety Buffer in Token Processing - Added a safety buffer multiplier to the character position estimates during text truncation to prevent overshooting token limits. - Updated the `processTextWithTokenLimit` function to utilize the new `TRUNCATION_SAFETY_BUFFER` constant, enhancing the accuracy of token limit processing. - Improved documentation to clarify the rationale behind the buffer and its impact on performance and efficiency in token counting. |
||
|
|
29ef91b4dd
|
🧠 feat: User Memories for Conversational Context (#7760)
* 🧠 feat: User Memories for Conversational Context
chore: mcp typing, use `t`
WIP: first pass, Memories UI
- Added MemoryViewer component for displaying, editing, and deleting user memories.
- Integrated data provider hooks for fetching, updating, and deleting memories.
- Implemented pagination and loading states for better user experience.
- Created unit tests for MemoryViewer to ensure functionality and interaction with data provider.
- Updated translation files to include new UI strings related to memories.
chore: move mcp-related files to own directory
chore: rename librechat-mcp to librechat-api
WIP: first pass, memory processing and data schemas
chore: linting in fileSearch.js query description
chore: rename librechat-api to @librechat/api across the project
WIP: first pass, functional memory agent
feat: add MemoryEditDialog and MemoryViewer components for managing user memories
- Introduced MemoryEditDialog for editing memory entries with validation and toast notifications.
- Updated MemoryViewer to support editing and deleting memories, including pagination and loading states.
- Enhanced data provider to handle memory updates with optional original key for better management.
- Added new localization strings for memory-related UI elements.
feat: add memory permissions management
- Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories.
- Added new API endpoints for updating memory permissions associated with roles.
- Created a new AdminSettings component for managing memory permissions in the frontend.
- Integrated memory permissions into the existing roles and permissions schemas.
- Updated the interface to include memory settings and permissions.
- Enhanced the MemoryViewer component to conditionally render admin settings based on user roles.
- Added localization support for memory permissions in the translation files.
feat: move AdminSettings component to a new position in MemoryViewer for better visibility
refactor: clean up commented code in MemoryViewer component
feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration
- Added a search input to filter memories in the MemoryViewer component.
- Refactored MemoryEditDialog to accept children for better customization.
- Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories.
- Improved localization support by adding new strings for memory filtering and deletion confirmation.
refactor: optimize memory filtering in MemoryViewer using match-sorter
- Replaced manual filtering logic with match-sorter for improved search functionality.
- Enhanced performance and readability of the filteredMemories computation.
feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling
feat: implement access control for MemoryEditDialog and MemoryViewer components
refactor: remove commented out code and create runMemory method
refactor: rename role based files
feat: implement access control for memory usage in AgentClient
refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code
refactor: make `agents` dir in api package
refactor: migrate Azure utilities to TypeScript and consolidate imports
refactor: move sanitizeFilename function to a new file and update imports, add related tests
refactor: update LLM configuration types and consolidate Azure options in the API package
chore: linting
chore: import order
refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file
chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json
refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports
refactor: move createRun function to a new run.ts file and update related imports
fix: ensure safeAttachments is correctly typed as an array of TFile
chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file
refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options
feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling
fix: update types due to new TEndpointOption typing
fix: ensure safe access to group parameters in initializeOpenAIOptions function
fix: remove redundant API key validation comment in initializeOpenAIOptions function
refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation
refactor: decouple req.body fields and tool loading from initializeAgentOptions
chore: linting
refactor: adjust column widths in MemoryViewer for improved layout
refactor: simplify agent initialization by creating loadAgent function and removing unused code
feat: add memory configuration loading and validation functions
WIP: first pass, memory processing with config
feat: implement memory callback and artifact handling
feat: implement memory artifacts display and processing updates
feat: add memory configuration options and schema validation for validKeys
fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements
refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling
WIP: initial tokenLimit config and move Tokenizer to @librechat/api
refactor: update mongoMeili plugin methods to use callback for better error handling
feat: enhance memory management with token tracking and usage metrics
- Added token counting for memory entries to enforce limits and provide usage statistics.
- Updated memory retrieval and update routes to include total token usage and limit.
- Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information.
- Refactored memory processing functions to handle token limits and provide feedback on memory capacity.
feat: implement memory artifact handling in attachment handler
- Enhanced useAttachmentHandler to process memory artifacts when receiving updates.
- Introduced handleMemoryArtifact utility to manage memory updates and deletions.
- Updated query client to reflect changes in memory state based on incoming data.
refactor: restructure web search key extraction logic
- Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys.
- Updated webSearchKeys to utilize the new function for improved clarity and maintainability.
- Prevents build time errors
feat: add personalization settings and memory preferences management
- Introduced a new Personalization tab in settings to manage user memory preferences.
- Implemented API endpoints and client-side logic for updating memory preferences.
- Enhanced user interface components to reflect personalization options and memory usage.
- Updated permissions to allow users to opt out of memory features.
- Added localization support for new settings and messages related to personalization.
style: personalization switch class
feat: add PersonalizationIcon and align Side Panel UI
feat: implement memory creation functionality
- Added a new API endpoint for creating memory entries, including validation for key and value.
- Introduced MemoryCreateDialog component for user interface to facilitate memory creation.
- Integrated token limit checks to prevent exceeding user memory capacity.
- Updated MemoryViewer to include a button for opening the memory creation dialog.
- Enhanced localization support for new messages related to memory creation.
feat: enhance message processing with configurable window size
- Updated AgentClient to use a configurable message window size for processing messages.
- Introduced messageWindowSize option in memory configuration schema with a default value of 5.
- Improved logic for selecting messages to process based on the configured window size.
chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json
chore: remove OpenAPIPlugin and its associated tests
chore: remove MIGRATION_README.md as migration tasks are completed
ci: fix backend tests
chore: remove unused translation keys from localization file
chore: remove problematic test file and unused var in AgentClient
chore: remove unused import and import directly for JSDoc
* feat: add api package build stage in Dockerfile for improved modularity
* docs: reorder build steps in contributing guide for clarity
|
Renamed from api/server/services/Tokenizer.js (Browse further)