Commit graph

51 commits

Author SHA1 Message Date
Marco Beretta
5b8bb3c31f
⏱️ fix: Bound Remaining Outbound Calls With Timeout Budgets (#17468)
* fix: Bound outbound HTTP calls that had no timeout

Add timeouts.toolsMs, speechMs and imageGenerationMs to the timeouts block and apply the
existing budgets to the RAG delete paths, built-in tool APIs, TTS/STT providers (TTS also
aborts when the client closes), Stable Diffusion, Firebase reads and Mailgun. Calls made
before app config exists (remote config, RAG health, service key URL, Google refresh) get a
fixed 10 second budget.

* fix: Honor configured budgets for RAG deletes and per-chunk TTS deadlines

* refactor: Share one budget helper between speech and image CJS callers

* test: Cover a stalled RAG delete with a mock scenario

* test: Wait for the RAG embed before stalling the delete

* test: Embed at upload so the stalled delete has a vector to remove

* fix: Close a started TTS response on a late chunk timeout and share the service budget helper

* fix: Fail startup when the remote config fetch times out
2026-10-29 09:55:18 +01:00
Marco Beretta
698ee25fea
🧯 fix: Close Abandoned Download and Speech Streams and Cap Speech Input (#17270)
* fix: Contain Download and Speech Stream Failures and Cap Speech Input

The code output download piped the Code API stream with .pipe, which adds no
error listener, so a body that failed mid-stream (a bad gzip body, a reset)
became an uncaught exception and the process exited. pipeToResponse uses
stream.pipeline, ending the response and destroying the upstream when the
client leaves.

Text-to-speech piped provider streams the same way and waited only for
'end', so a provider error was uncaught and a client disconnect left the
chunk loop waiting. pipeSegment resolves on end and rejects on an error or
a disconnect, which ends the loop.

A speech request had no length cap, so one request could fan out into
thousands of serial paid provider calls. speech.tts.maxInputChars bounds it
(default 20,000 characters).

* docs: Document the Speech Input Cap in the Example Config

* fix: Reject a Speech Segment When the Client Already Left

* fix: Tear Down Both Ends of a Failed Speech Segment and Cap Streamed Speech

A failed provider segment now destroys the response as well as the source,
so a half-written answer cannot hang the client, and a response-side error
rejects the segment instead of going unhandled. Automatic playback reads a
message only up to speech.tts.maxInputChars, so long messages cannot fan
out past the cap either.

The download scenario now checks the behavior .pipe lacked: leaving a code
download closes the upstream connection right away.

* fix: Keep a Late Response Write Error From Going Unhandled

* test: Key Slow Code Downloads by File and Send a Browser User Agent

* fix: Read Speech Up to the Cap Instead of Refusing It

Manual speech now synthesizes the text up to speech.tts.maxInputChars, the
same as automatic playback, through capSpeechInput, so the CJS service only
passes the capped text along and the request keeps its existing error
contract. A streamed message that reaches the cap exactly now finishes, and
a segment that fails before any audio went out leaves the response open for
the caller's HTTP error.

* fix: Count Only Text Against the Speech Cap
2026-10-19 22:20:40 +02:00
Marco Beretta
5fc4371c6a
🪦 refactor: Remove Dead Legacy API Modules and Root Tooling (#17114) 2026-10-15 12:25:24 +02:00
Marco Beretta
9068f0e922
🔇 fix: Keep Reasoning Out of Read Aloud (#16790)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: Keep Reasoning Out of Every Read Aloud Path

Read aloud spoke the Thoughts section along with the answer. The hover
button flattened think parts into the string it handed to TTS, browser
autoplay and the hooks parsed content without skipping reasoning, and
server streaming spoke the stored text field, which an aborted turn
persists with its reasoning included.

Add getSpeechText to data-provider as the single source for spoken text:
text parts only for structured content, and the legacy :::thinking block
stripped from plain text, treating a still-open block as reasoning so
streamed speech only ever grows. Route the hover button, browser
autoplay and server streaming through it, and skip reasoning in the TTS
hooks' own content parsing.

* fix: Speak String Message Content Without Its Reasoning

Imported or client-built messages can carry their answer as string
content rather than parts; read it like plain text instead of falling
back to an empty text field.

* fix: Keep Answers Between Repeated Legacy Thinking Blocks

Strip every closed :::thinking block before treating an unterminated one
as reasoning, so a second closed block no longer swallows the answer
that follows it.

* fix: Skip Read Aloud and Autoplay When Only Reasoning Remains

A turn stopped mid-reasoning has no answer to speak once reasoning is
filtered. Gate autoplay on the same speech text it will play, so the
external engine never opens a stream that can only time out, and hide
the Read aloud action when there is nothing to read. Every speech path
reads a finalized message, so speak an unmatched :::thinking marker as
the answer text the UI shows instead of dropping the rest.

* refactor: Move Speech Chunk Polling Into packages/api

The decision of what text to stream as speech lived in the legacy CJS
service. Move the chunk processor to packages/api as
createSpeechChunkProcessor, taking the message cache and lookup from the
caller, and leave streamAudio.js wiring them in. Its tests move with it
onto an in-memory cache instead of module mocks.

* fix: Speak Exactly the Answer the Message View Shows

Share the legacy :::thinking parser between the message view and speech,
so Read Aloud strips the same block the view collapses instead of a
separate pattern drifting from it. Trim structured speech text so a
whitespace-only answer is not speakable, and key autoplay's audio cache
by the spoken text, as manual playback already is, so audio cached with
reasoning before this change is never replayed.

* fix: Fall Back to Text When String Content Is Empty
2026-10-05 23:44:39 +02:00
Danny Avila
ec15075dc8
🚚 feat: Route Attachments and Provision Tools Lazily (#15763) 2026-09-08 23:28:52 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Marco Beretta
5ff282f900
🎙️ fix: Align Speech Engine Configuration With Runtime (#14736)
* fix: align speech engine configuration with runtime

* fix: guard speech recording shortcuts

* fix: reconcile speech engine availability

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-12 13:22:28 -04:00
Dustin Healy
96404bfc73
🛡️ fix: SSRF-Guard Speech (STT/TTS) and OCR Outbound Requests at Connect Time (#14560)
* fix: SSRF-guard speech (STT/TTS) and OCR outbound requests at connect time

Speech (STT/TTS) and OCR issued outbound HTTP to operator-provided target URLs
with only proxy config attached, so a target that resolves to a private, loopback,
link-local, or cloud-metadata address was reachable by the server. This is the same
class already guarded for the custom models fetch, Actions, avatar, MCP, and the
OpenAI/Anthropic endpoint clients.

Add one helper applySSRFSafeAgentIfDirect(config, url, allowedAddresses) that rejects
non-http(s) or unparseable target URLs, sets maxRedirects to 0 so a redirect cannot
bypass the connect-time check, and attaches createSSRFSafeAgents when no proxy or agent
is already set (proxy precedence preserved). Wire it into the six speech/OCR call sites
and thread each section's allowedAddresses exemption from pr-01.

Scope is private/internal SSRF only. It does not restrict forwarding a credential to a
public host, which is a separate egress-allowlist concern. Absent an allowedAddresses
entry, private targets now fail closed, matching endpoints, actions, and MCP. Document
the new fields in librechat.example.yaml with the operator warning that allowedAddresses
hostnames are trusted before the private-IP check.

* fix: block literal private-IP hosts and thread STT exemptions through generic uploads

applySSRFSafeAgentIfDirect only attached the DNS-lookup agents, but Node
skips the custom lookup for IP-literal hosts, so a literal private IP such
as http://127.0.0.1 connected unchecked. Reject literal private IPs
synchronously in the helper, reusing the same allowedAddresses exemption
logic as the lookup path.

The generic audio-upload path did not forward the section-level
allowedAddresses to sttRequest, so a private STT endpoint permitted via
speech.stt.allowedAddresses failed with ESSRF outside the speech route.
Thread the exemption through files/audio.ts and widen the STTService type.

* fix: derive effective SSRF port for literal IPs and validate OCR target before opening its stream

The literal-IP precheck normalized an empty URL.port to '', so
allowedAddresses exemptions on a default port (127.0.0.1:80, [::1]:443)
never matched. Derive 80/443 from the scheme when the port is omitted.

uploadDocumentToMistral opened the upload file stream before the SSRF
check, so a blocked or malformed target threw with the descriptor still
open. Run the proxy/SSRF setup before fs.createReadStream.

Reword the librechat.example.yaml allowedAddresses guidance to prefer a
private IP literal over a hostname, and reconcile the speech/OCR SSRF
specs to assert the synchronous literal-IP block instead of driving the
connect-time lookup that Node skips for IP literals.

* fix: block literal private IPs before the proxy return, destroy OCR stream on failure, canonicalize IPv6 exemptions

Move the literal-IP check above the proxy/agent early return so a literal
private IP is rejected even when a proxy is configured; document that a
forward proxy must be SSRF-enforcing.

Wrap the Mistral upload post in try/finally and destroy the file stream,
so an async connect-time block does not leak the descriptor.

Canonicalize IP literals in normalizeAddressCandidate through the same URL
serialization targets use, so IPv4-mapped and expanded IPv6 exemptions match.
2026-08-06 09:06:04 -04:00
Dustin Healy
4cec1a675f
🔒 feat: Add allowedAddresses Exemption to Speech (STT/TTS) and OCR Config Schemas (#14559)
* feat: add allowedAddresses exemption to speech (STT/TTS) and OCR config schemas

Add the existing allowedAddressesSchema as an optional field on sttSchema,
ttsSchema, and ocrSchema, reusing the schema already attached to endpoints,
mcpSettings, and actions so port scoping and normalization stay identical.

STT and TTS resolve a single provider by counting non-empty section keys, so
exclude the allowedAddresses key from that scan. Without the exclusion a
configured exemption list would be counted as a second provider and trip the
"Multiple providers are set" guard. The field is inert on its own: nothing
reads it for SSRF yet, and provider detection now ignores it.

* fix: preserve allowedAddresses through the OCR config loaders

loadOCRConfig rebuilt the ocr config with only apiKey, baseURL,
mistralModel, and strategy, dropping allowedAddresses before it reached
req.config.ocr. Pass it through in both the AppService loader
(packages/data-schemas/src/app/ocr.ts) and the duplicate at
packages/api/src/files/ocr.ts so the exemption survives config load.
2026-08-05 13:42:43 -04:00
Dustin Healy
f4e0888f14
🕶 feat: Generalize Admin Config Secret Redaction (#14509)
* feat: generalize admin config secret redaction to a field registry

Replace the single hardcoded langfuse.secretKey handling in the admin
config secrets module with a registry (CONFIG_SECRET_FIELDS) so every
credential-shaped config field is encrypted at rest, redacted on read,
and preserved when omitted on a subsequent write.

Registry covers langfuse.secretKey plus speech tts/stt provider apiKeys,
ocr.apiKey, the webSearch provider apiKeys, and the assistants /
azureAssistants endpoint apiKeys. Fields that conventionally hold
${ENV_VAR} references keep those references as plain, visible values;
literal secrets are always encrypted. langfuse.secretKey behavior
(display companion, always-encrypt, array-section handling) is unchanged.

Wire runtime decryption for the consumers that read these values from
the merged app config: resolveConfigSecret in the speech STT/TTS
services and decryptConfigSecret in the Mistral OCR auth loader. Legacy
plaintext literals and ${ENV_VAR} references continue to resolve.

* feat: add masked display companions for every registered config secret

Every non-langfuse field added to the secret registry was missing the
displayPath that langfuse.secretKey already had, so redacted admin reads
returned nothing for those fields instead of a masked value like
sk-mis...Z789. The registry-driven encrypt/redact/preserve/mutation-path
logic in secrets.ts was already field-agnostic; the only backend fix is
setting displayPath on the other 15 registry entries.

Add the matching optional display<Field> companion to each zod schema
in librechat-data-provider (ocr, speech tts/stt providers, webSearch
providers, the shared assistants/azureAssistants endpoint schema) so the
field is typed for consumers, mirroring langfuse.displaySecretKey. DB
overrides are Mixed-typed, so nothing breaks without this at the storage
layer, but the type is needed for any typed consumer of TCustomConfig.

A display path can never be written as a secret: direct writes to it are
rejected, and an ancestor-object write that includes a spoofed display
value alongside or instead of the real secret is overwritten or dropped,
never encrypted or persisted.

* fix: harden the config secret write path against masking and mixing bugs

getDisplaySecretKey disclosed the entire value for any secret of 10
characters or fewer, since the first-6/last-4 mask overlaps or covers
the whole string at that length (e.g. self-hosted LocalAI tokens).
Short secrets are now fully masked instead.

writeSecretIntoSection/writeDottedSecret encrypted a literal secret's
raw string verbatim, including leading/trailing whitespace, so a
padded paste round-tripped with the whitespace intact and a
whitespace-only value was not treated as empty. Literals are now
trimmed before encrypting and masking.

Both functions also returned early on an env-placeholder value without
clearing the display companion, so replacing a literal secret with
${ENV_VAR} left the previous masked value stale in the stored config,
and a client-supplied display value submitted alongside a placeholder
secret was never overwritten. The placeholder branch now clears the
display companion in both the dotted-patch and object-valued write
paths.

* fix: fail closed in Mistral OCR auth when a stored ciphertext can't decrypt

loadAuthConfig fell back to the raw v3: ciphertext string whenever
decryptConfigSecret returned undefined, so a corrupted or otherwise
undecryptable stored secret was sent to the Mistral API verbatim as
the apiKey instead of triggering the existing env-var fallback.

isEncryptedConfigSecret is now exported so the OCR auth loader can
distinguish "this looks like ciphertext and failed to decrypt" from
"this was never encrypted" and treat only the former as empty,
preserving literal and ${ENV_VAR} values exactly as before.

* fix: omit undecryptable TTS provider headers instead of sending "undefined"

openAIProvider, elevenLabsProvider, and localAIProvider built their
Authorization/xi-api-key headers directly from resolveConfigSecret's
return value, which is undefined on a decrypt failure. That produced a
literal "Bearer undefined" header (or an undefined-valued xi-api-key
header) sent to the provider instead of failing gracefully.

Each provider now resolves the key once and only includes the header
when it's non-empty, matching the pattern already used by
azureOpenAIProvider and STTService's providers.

* fix: strip secret-ancestor arrays at any depth, not just the top level

encryptConfigSecrets/redactConfigSecrets only stripped an array-valued
registered-secret ancestor when it appeared as a literal top-level key
(e.g. a dotted "speech.tts.openai" key). A true nested array at any
depth, e.g. { speech: { tts: { openai: [{ apiKey: "sk-secret" }] } } },
made walkToParent return null and silently skip that field entirely,
so the literal secret was stored unencrypted and returned verbatim to
any reader with section-level read access.

pruneSecretAncestorArrays now walks every registered field's ancestor
chain and deletes any array found at any depth before encryption or
redaction runs, closing the gap for both write and read paths.

* fix: migrate legacy plaintext secrets instead of dropping them on preserve

preserveConfigSecrets only restored an omitted secret when the existing
stored value was already v3-encrypted. Every field this PR newly
registers was previously stored as plaintext with no protection at
all, so any deployment upgrading into this registry has real
credentials sitting in Mongo as plaintext today. The first time an
admin edited an unrelated field in the same section (e.g. mistralModel
next to ocr.apiKey), the omitted plaintext secret failed the
"already encrypted" check and was silently dropped instead of
preserved, breaking the integration.

The existing value is now encrypted in place when it isn't already
ciphertext or an allowed env placeholder, so the credential survives
the edit and gets a computed display companion instead of being lost.

* refactor: derive masked-preview companions as <field>Preview

Replace the display*-prefixed companion names (displaySecretKey,
displayApiKey, displaySerperApiKey, ...) with a uniform <field>Preview
suffix (secretKeyPreview, apiKeyPreview, serperApiKeyPreview, ...) derived
automatically from the registered secret path — registry entries no longer
declare a displayPath, and the name never collides with display-label
config fields like modelDisplayLabel.

Legacy langfuse.displaySecretKey companions (the only shipped instance,
with no released reader) are stripped from writes and reads and migrated
to secretKeyPreview on preserve, so stored documents self-clean.

Also reject MongoDB operator segments ($, $[], $[id]) in admin config
field paths: isValidFieldPath previously accepted them, letting a patch
like webSearch.$[].serperApiKey reach patchConfigFields as a positional
update that bypassed secret-path validation.

* fix: translate legacy displaySecretKey to secretKeyPreview on reads

Redaction previously deleted the legacy companion outright, so the first
admin read of a not-yet-migrated document showed no configured-secret
indication until a later write migrated it. Reads now surface the legacy
value under secretKeyPreview (when no new-name preview exists) while still
stripping the legacy key from the response; stored documents migrate for
real on their next write.

* fix: detect runtime ciphertext by full encryptV3 payload shape

Runtime resolution (resolveConfigSecret, mistral OCR auth) now identifies
decryptable values by the exact v3:<32-hex-iv>:<hex> shape encryptV3
produces instead of the bare v3: prefix, so a legitimate literal credential
that merely starts with v3: (e.g. from YAML, which the admin write path
never encrypts) resolves as a literal instead of failing decryption.
Write-side prefix rejection stays broad as spoof/echo defense.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 13:55:45 -04:00
Danny Avila
98704f28c1
🌐 fix: Centralize Outbound Proxy Handling (#13726)
* fix: centralize outbound proxy handling

* chore: sort proxy imports

* test: update proxy helper mocks

* fix: honor proxy bypasses consistently

* fix: support http axios proxy targets
2026-06-14 10:47:49 -04:00
Danny Avila
c342e2345b
🪪 fix: Resolve Group-Scoped Config Overrides (#13176)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* fix: resolve group-scoped config overrides

* test: fix endpoint config request mock typing

* fix: keep remote agent preauth config tenant-scoped

* test: align config scoping expectations

* test: reproduce group endpoint override resolution
2026-05-18 10:16:20 -04:00
Danny Avila
6183303653
🔉 fix: Normalize audio MIME types in STT format validation (#12674)
* fix: normalize audio MIME types in STT format validation

Use getFileExtensionFromMime() to normalize non-standard MIME types
(e.g. audio/x-m4a, audio/x-wav, audio/x-flac) before checking against
the accepted formats list in azureOpenAIProvider. This is the same class
of bug as #12608 (text/x-markdown), but for STT audio validation.

Only audio/ and video/ MIME prefixes are normalized to prevent
non-audio types from matching via the webm default fallback.

Export getFileExtensionFromMime for testability.

Fixes #12632

* fix: reject unknown audio subtypes in STT format validation

Use MIME_TO_EXTENSION_MAP for normalization instead of
getFileExtensionFromMime() which falls back to 'webm' for unrecognized
types. Gate raw subtype matching on audio/video prefix to prevent
non-audio types (e.g. text/webm) from passing validation.

Resolves Codex review comment about unknown subtypes silently passing.

---------

Co-authored-by: Tobias Jonas <t.jonas@innfactory.de>
2026-04-15 09:58:07 -04:00
Danny Avila
877c2efc85
🏗️ feat: bulkWrite isolation, pre-auth context, strict-mode fixes (#12445)
* fix: wrap seedDatabase() in runAsSystem() for strict tenant mode

seedDatabase() was called without tenant context at startup, causing
every Mongoose operation inside it to throw when
TENANT_ISOLATION_STRICT=true. Wrapping in runAsSystem() gives it the
SYSTEM_TENANT_ID sentinel so the isolation plugin skips filtering,
matching the pattern already used for performStartupChecks and
updateInterfacePermissions.

* fix: chain tenantContextMiddleware in optionalJwtAuth

optionalJwtAuth populated req.user but never established ALS tenant
context, unlike requireJwtAuth which chains tenantContextMiddleware
after successful auth. Authenticated users hitting routes with
optionalJwtAuth (e.g. /api/banner) had no tenant isolation.

* feat: tenant-safe bulkWrite wrapper and call-site migration

Mongoose's bulkWrite() does not trigger schema-level middleware hooks,
so the applyTenantIsolation plugin cannot intercept it. This adds a
tenantSafeBulkWrite() utility that injects the current ALS tenant
context into every operation's filter/document before delegating to
native bulkWrite.

Migrates all 8 runtime bulkWrite call sites:
- agentCategory (seedCategories, ensureDefaultCategories)
- conversation (bulkSaveConvos)
- message (bulkSaveMessages)
- file (batchUpdateFiles)
- conversationTag (updateTagsForConversation, bulkIncrementTagCounts)
- aclEntry (bulkWriteAclEntries)

systemGrant.seedSystemGrants is intentionally not migrated — it uses
explicit tenantId: { $exists: false } filters and is exempt from the
isolation plugin.

* feat: pre-auth tenant middleware and tenant-scoped config cache

Adds preAuthTenantMiddleware that reads X-Tenant-Id from the request
header and wraps downstream in tenantStorage ALS context. Wired onto
/oauth, /api/auth, /api/config, and /api/share — unauthenticated
routes that need tenant scoping before JWT auth runs.

The /api/config cache key is now tenant-scoped
(STARTUP_CONFIG:${tenantId}) so multi-tenant deployments serve the
correct login page config per tenant.

The middleware is intentionally minimal — no subdomain parsing, no
OIDC claim extraction. The private fork's reverse proxy or auth
gateway sets the header.

* feat: accept optional tenantId in updateInterfacePermissions

When tenantId is provided, the function re-enters inside
tenantStorage.run({ tenantId }) so all downstream Mongoose queries
target that tenant's roles instead of the system context. This lets
the private fork's tenant provisioning flow call
updateInterfacePermissions per-tenant after creating tenant-scoped
ADMIN/USER roles.

* fix: tenant-filter $lookup in getPromptGroup aggregation

The $lookup stage in getPromptGroup() queried the prompts collection
without tenant filtering. While the outer PromptGroup aggregate is
protected by the tenantIsolation plugin's pre('aggregate') hook,
$lookup runs as an internal MongoDB operation that bypasses Mongoose
hooks entirely.

Converts from simple field-based $lookup to pipeline-based $lookup
with an explicit tenantId match when tenant context is active.

* fix: replace field-level unique indexes with tenant-scoped compounds

Field-level unique:true creates a globally-unique single-field index in
MongoDB, which would cause insert failures across tenants sharing the
same ID values.

- agent.id: removed field-level unique, added { id, tenantId } compound
- convo.conversationId: removed field-level unique (compound at line 50
  already exists: { conversationId, user, tenantId })
- message.messageId: removed field-level unique (compound at line 165
  already exists: { messageId, user, tenantId })
- preset.presetId: removed field-level unique, added { presetId, tenantId }
  compound

* fix: scope MODELS_CONFIG, ENDPOINT_CONFIG, PLUGINS, TOOLS caches by tenant

These caches store per-tenant configuration (available models, endpoint
settings, plugin availability, tool definitions) but were using global
cache keys. In multi-tenant mode, one tenant's cached config would be
served to all tenants.

Appends :${tenantId} to cache keys when tenant context is active.
Falls back to the unscoped key when no tenant context exists (backward
compatible for single-tenant OSS deployments).

Covers all read, write, and delete sites:
- ModelController.js: get/set MODELS_CONFIG
- PluginController.js: get/set PLUGINS, get/set TOOLS
- getEndpointsConfig.js: get/set/delete ENDPOINT_CONFIG
- app.js: delete ENDPOINT_CONFIG (clearEndpointConfigCache)
- mcp.js: delete TOOLS (updateMCPTools, mergeAppTools)
- importers.js: get ENDPOINT_CONFIG

* fix: add getTenantId to PluginController spec mock

The data-schemas mock was missing getTenantId, causing all
PluginController tests to throw when the controller calls
getTenantId() for tenant-scoped cache keys.

* fix: address review findings — migration, strict-mode, DRY, types

Addresses all CRITICAL, MAJOR, and MINOR review findings:

F1 (CRITICAL): Add agents, conversations, messages, presets to
SUPERSEDED_INDEXES in tenantIndexes.ts so dropSupersededTenantIndexes()
drops the old single-field unique indexes that block multi-tenant inserts.

F2 (CRITICAL): Unknown bulkWrite op types now throw in strict mode
instead of silently passing through without tenant injection.

F3 (MAJOR): Replace wildcard export with named export for
tenantSafeBulkWrite, hiding _resetBulkWriteStrictCache from the
public package API.

F5 (MAJOR): Restore AnyBulkWriteOperation<IAclEntry>[] typing on
bulkWriteAclEntries — the unparameterized wrapper accepts parameterized
ops as a subtype.

F7 (MAJOR): Fix config.js tenant precedence — JWT-derived
req.user.tenantId now takes priority over the X-Tenant-Id header for
authenticated requests.

F8 (MINOR): Extract scopedCacheKey() helper into tenantContext.ts and
replace all 11 inline occurrences across 7 files.

F9 (MINOR): Use simple localField/foreignField $lookup for the
non-tenant getPromptGroup path (more efficient index seeks).

F12 (NIT): Remove redundant BulkOp type alias.
F13 (NIT): Remove debug log that leaked raw tenantId.

* fix: add new superseded indexes to tenantIndexes test fixture

The test creates old indexes to verify the migration drops them.
Missing fixture entries for agents.id_1, conversations.conversationId_1,
messages.messageId_1, and presets.presetId_1 caused the count assertion
to fail (expected 22, got 18).

* fix: restore logger.warn for unknown bulk op types in non-strict mode

* fix: block SYSTEM_TENANT_ID sentinel from external header input

CRITICAL: preAuthTenantMiddleware accepted any string as X-Tenant-Id,
including '__SYSTEM__'. The tenantIsolation plugin treats SYSTEM_TENANT_ID
as an explicit bypass — skipping ALL query filters. A client sending
X-Tenant-Id: __SYSTEM__ to pre-auth routes (/api/share, /api/config,
/api/auth, /oauth) would execute Mongoose operations without tenant
isolation.

Fixes:
- preAuthTenantMiddleware rejects SYSTEM_TENANT_ID in header
- scopedCacheKey returns the base key (not key:__SYSTEM__) in system
  context, preventing stale cache entries during runAsSystem()
- updateInterfacePermissions guards tenantId against SYSTEM_TENANT_ID
- $lookup pipeline separates $expr join from constant tenantId match
  for better index utilization
- Regression test for sentinel rejection in preAuthTenant.spec.ts
- Remove redundant getTenantId() call in config.js

* test: add missing deleteMany/replaceOne coverage, fix vacuous ALS assertions

bulkWrite spec:
- deleteMany: verifies tenant-scoped deletion leaves other tenants untouched
- replaceOne: verifies tenantId injected into both filter and replacement
- replaceOne overwrite: verifies a conflicting tenantId in the replacement
  document is overwritten by the ALS tenant (defense-in-depth)
- empty ops array: verifies graceful handling

preAuthTenant spec:
- All negative-case tests now use the capturedNext pattern to verify
  getTenantId() inside the middleware's execution context, not the
  test runner's outer frame (which was always undefined regardless)

* feat: tenant-isolate MESSAGES cache, FLOWS cache, and GenerationJobManager

MESSAGES cache (streamAudio.js):
- Cache key now uses scopedCacheKey(messageId) to prefix with tenantId,
  preventing cross-tenant message content reads during TTS streaming.

FLOWS cache (FlowStateManager):
- getFlowKey() now generates ${type}:${tenantId}:${flowId} when tenant
  context is active, isolating OAuth flow state per tenant.

GenerationJobManager:
- tenantId added to SerializableJobData and GenerationJobMetadata
- createJob() captures the current ALS tenant context (excluding
  SYSTEM_TENANT_ID) and stores it in job metadata
- SSE subscription endpoint validates job.metadata.tenantId matches
  req.user.tenantId, blocking cross-tenant stream access
- Both InMemoryJobStore and RedisJobStore updated to accept tenantId

* fix: add getTenantId and SYSTEM_TENANT_ID to MCP OAuth test mocks

FlowStateManager.getFlowKey() now calls getTenantId() for tenant-scoped
flow keys. The 4 MCP OAuth test files mock @librechat/data-schemas
without these exports, causing TypeError at runtime.

* fix: correct import ordering per AGENTS.md conventions

Package imports sorted shortest to longest line length, local imports
sorted longest to shortest — fixes ordering violations introduced by
our new imports across 8 files.

* fix: deserialize tenantId in RedisJobStore — cross-tenant SSE guard was no-op in Redis mode

serializeJob() writes tenantId to the Redis hash via Object.entries,
but deserializeJob() manually enumerates fields and omitted tenantId.
Every getJob() from Redis returned tenantId: undefined, causing the
SSE route's cross-tenant guard to short-circuit (undefined && ... → false).

* test: SSE tenant guard, FlowStateManager key consistency, ALS scope docs

SSE stream tenant tests (streamTenant.spec.js):
- Cross-tenant user accessing another tenant's stream → 403
- Same-tenant user accessing own stream → allowed
- OSS mode (no tenantId on job) → tenant check skipped

FlowStateManager tenant tests (manager.tenant.spec.ts):
- completeFlow finds flow created under same tenant context
- completeFlow does NOT find flow under different tenant context
- Unscoped flows are separate from tenant-scoped flows

Documentation:
- JSDoc on getFlowKey documenting ALS context consistency requirement
- Comment on streamAudio.js scopedCacheKey capture site

* fix: SSE stream tests hang on success path, remove internal fork references

The success-path tests entered the SSE streaming code which never
closes, causing timeout. Mock subscribe() to end the response
immediately. Restructured assertions to verify non-403/non-404.

Removed "private fork" and "OSS" references from code and test
descriptions — replaced with "deployment layer", "multi-tenant
deployments", and "single-tenant mode".

* fix: address review findings — test rigor, tenant ID validation, docs

F1: SSE stream tests now mock subscribe() with correct signature
(streamId, writeEvent, onDone, onError) and assert 200 status,
verifying the tenant guard actually allows through same-tenant users.

F2: completeFlow logs the attempted key and ALS tenantId when flow
is not found, so reverse proxy misconfiguration (missing X-Tenant-Id
on OAuth callback) produces an actionable warning.

F3/F10: preAuthTenantMiddleware validates tenant ID format — rejects
colons, special characters, and values exceeding 128 chars. Trims
whitespace. Prevents cache key collisions via crafted headers.

F4: Documented cache invalidation scope limitation in
clearEndpointConfigCache — only the calling tenant's key is cleared;
other tenants expire via TTL.

F7: getFlowKey JSDoc now lists all 8 methods requiring consistent
ALS context.

F8: Added dedicated scopedCacheKey unit tests — base key without
context, base key in system context, scoped key with tenant, no
ALS leakage across scope boundaries.

* fix: revert flow key tenant scoping, fix SSE test timing

FlowStateManager: Reverts tenant-scoped flow keys. OAuth callbacks
arrive without tenant ALS context (provider redirects don't carry
X-Tenant-Id), so completeFlow/failFlow would never find flows
created under tenant context. Flow IDs are random UUIDs with no
collision risk, and flow data is ephemeral (TTL-bounded).

SSE tests: Use process.nextTick for onDone callback so Express
response headers are flushed before res.write/res.end are called.

* fix: restore getTenantId import for completeFlow diagnostic log

* fix: correct completeFlow warning message, add missing flow test

The warning referenced X-Tenant-Id header consistency which was only
relevant when flow keys were tenant-scoped (since reverted). Updated
to list actual causes: TTL expiry, missing flow, or routing to a
different instance without shared Keyv storage.

Removed the getTenantId() call and import — no longer needed since
flow keys are unscoped.

Added test for the !flowState branch in completeFlow — verifies
return false and logger.warn on nonexistent flow ID.

* fix: add explicit return type to recursive updateInterfacePermissions

The recursive call (tenantId branch calls itself without tenantId)
causes TypeScript to infer circular return type 'any'. Adding
explicit Promise<void> satisfies the rollup typescript plugin.

* fix: update MCPOAuthRaceCondition test to match new completeFlow warning

* fix: clearEndpointConfigCache deletes both scoped and unscoped keys

Unauthenticated /api/endpoints requests populate the unscoped
ENDPOINT_CONFIG key. Admin config mutations clear only the
tenant-scoped key, leaving the unscoped entry stale indefinitely.
Now deletes both when in tenant context.

* fix: tenant guard on abort/status endpoints, warn logs, test coverage

F1: Add tenant guard to /chat/status/:conversationId and /chat/abort
matching the existing guard on /chat/stream/:streamId. The status
endpoint exposes aggregatedContent (AI response text) which requires
tenant-level access control.

F2: preAuthTenantMiddleware now logs warn for rejected __SYSTEM__
sentinel and malformed tenant IDs, providing observability for
bypass probing attempts.

F3: Abort fallback path (getActiveJobIdsForUser) now has tenant
check after resolving the job.

F4: Test for strict mode + SYSTEM_TENANT_ID — verifies runAsSystem
bypasses tenantSafeBulkWrite without throwing in strict mode.

F5: Test for job with tenantId + user without tenantId → 403.

F10: Regex uses idiomatic hyphen-at-start form.

F11: Test descriptions changed from "rejects" to "ignores" since
middleware calls next() (not 4xx).

Also fixes MCPOAuthRaceCondition test assertion to match updated
completeFlow warning message.

* fix: test coverage for logger.warn, status/abort guards, consistency

A: preAuthTenant spec now mocks logger and asserts warn calls for
__SYSTEM__ sentinel, malformed characters, and oversized headers.

B: streamTenant spec expanded with status and abort endpoint tests —
cross-tenant status returns 403, same-tenant returns 200 with body,
cross-tenant abort returns 403.

C: Abort endpoint uses req.user.tenantId (not req.user?.tenantId)
matching stream/status pattern — requireJwtAuth guarantees req.user.

D: Malformed header warning now includes ip in log metadata,
matching the sentinel warning for consistent SOC correlation.

* fix: assert ip field in malformed header warn tests

* fix: parallelize cache deletes, document tenant guard, fix import order

- clearEndpointConfigCache uses Promise.all for independent cache
  deletes instead of sequential awaits
- SSE stream tenant guard has inline comment explaining backward-compat
  behavior for untenanted legacy jobs
- conversation.ts local imports reordered longest-to-shortest per
  AGENTS.md

* fix: tenant-qualify userJobs keys, document tenant guard backward-compat

Job store userJobs keys now include tenantId when available:
- Redis: stream:user:{tenantId:userId}:jobs (falls back to
  stream:user:{userId}:jobs when no tenant)
- InMemory: composite key tenantId:userId in userJobMap

getActiveJobIdsByUser/getActiveJobIdsForUser accept optional tenantId
parameter, threaded through from req.user.tenantId at all call sites
(/chat/active and /chat/abort fallback).

Added inline comments on all three SSE tenant guards explaining the
backward-compat design: untenanted legacy jobs remain accessible
when the userId check passes.

* fix: parallelize cache deletes, document tenant guard, fix import order

Fix InMemoryJobStore.getActiveJobIdsByUser empty-set cleanup to use
the tenant-qualified userKey instead of bare userId — prevents
orphaned empty Sets accumulating in userJobMap for multi-tenant users.

Document cross-tenant staleness in clearEndpointConfigCache JSDoc —
other tenants' scoped keys expire via TTL, not active invalidation.

* fix: cleanup userJobMap leak, startup warning, DRY tenant guard, docs

F1: InMemoryJobStore.cleanup() now removes entries from userJobMap
before calling deleteJob, preventing orphaned empty Sets from
accumulating with tenant-qualified composite keys.

F2: Startup warning when TENANT_ISOLATION_STRICT is active — reminds
operators to configure reverse proxy to control X-Tenant-Id header.

F3: mergeAppTools JSDoc documents that tenant-scoped TOOLS keys are
not actively invalidated (matching clearEndpointConfigCache pattern).

F5: Abort handler getActiveJobIdsForUser call uses req.user.tenantId
(not req.user?.tenantId) — consistent with stream/status handlers.

F6: updateInterfacePermissions JSDoc clarifies SYSTEM_TENANT_ID
behavior — falls through to caller's ALS context.

F7: Extracted hasTenantMismatch() helper, replacing three identical
inline tenant guard blocks across stream/status/abort endpoints.

F9: scopedCacheKey JSDoc documents both passthrough cases (no context
and SYSTEM_TENANT_ID context).

* fix: clean userJobMap in evictOldest — same leak as cleanup()
2026-03-28 16:43:50 -04:00
Danny Avila
9f6d8c6e93
🧵 feat: ALS Context Middleware, Tenant Threading, and Config Cache Invalidation (#12407)
* feat: add tenant context middleware for ALS-based isolation

Introduces tenantContextMiddleware that propagates req.user.tenantId
into AsyncLocalStorage, activating the Mongoose applyTenantIsolation
plugin for all downstream DB queries within a request.

- Strict mode (TENANT_ISOLATION_STRICT=true) returns 403 if no tenantId
- Non-strict mode passes through for backward compatibility
- No-op for unauthenticated requests
- Includes 6 unit tests covering all paths

* feat: register tenant middleware and wrap startup/auth in runAsSystem()

- Register tenantContextMiddleware in Express app after capability middleware
- Wrap server startup initialization in runAsSystem() for strict mode compat
- Wrap auth strategy getAppConfig() calls in runAsSystem() since they run
  before user context is established (LDAP, SAML, OpenID, social login, AuthService)

* feat: thread tenantId through all getAppConfig callers

Pass tenantId from req.user to getAppConfig() across all callers that
have request context, ensuring correct per-tenant cache key resolution.

Also fixes getBaseConfig admin endpoint to scope to requesting admin's
tenant instead of returning the unscoped base config.

Files updated:
- Controllers: UserController, PluginController
- Middleware: checkDomainAllowed, balance
- Routes: config
- Services: loadConfigModels, loadDefaultModels, getEndpointsConfig, MCP
- Audio services: TTSService, STTService, getVoices, getCustomConfigSpeech
- Admin: getBaseConfig endpoint

* feat: add config cache invalidation on admin mutations

- Add clearOverrideCache(tenantId?) to flush per-principal override caches
  by enumerating Keyv store keys matching _OVERRIDE_: prefix
- Add invalidateConfigCaches() helper that clears base config, override
  caches, tool caches, and endpoint config cache in one call
- Wire invalidation into all 5 admin config mutation handlers
  (upsert, patch, delete field, delete overrides, toggle active)
- Add strict mode warning when __default__ tenant fallback is used
- Add 3 new tests for clearOverrideCache (all/scoped/base-preserving)

* chore: update getUserPrincipals comment to reflect ALS-based tenant filtering

The TODO(#12091) about missing tenantId filtering is resolved by the
tenant context middleware + applyTenantIsolation Mongoose plugin.
Group queries are now automatically scoped by tenantId via ALS.

* fix: replace runAsSystem with baseOnly for pre-tenant code paths

App configs are tenant-owned — runAsSystem() would bypass tenant
isolation and return cross-tenant DB overrides. Instead, add
baseOnly option to getAppConfig() that returns YAML-derived config
only, with zero DB queries.

All startup code, auth strategies, and MCP initialization now use
getAppConfig({ baseOnly: true }) to get the YAML config without
touching the Config collection.

* fix: address PR review findings — middleware ordering, types, cache safety

- Chain tenantContextMiddleware inside requireJwtAuth after passport auth
  instead of global app.use() where req.user is always undefined (Finding 1)
- Remove global tenantContextMiddleware registration from index.js
- Update BalanceMiddlewareOptions to include tenantId, remove redundant cast (Finding 4)
- Add warning log when clearOverrideCache cannot enumerate keys on Redis (Finding 3)
- Use startsWith instead of includes for cache key filtering (Finding 12)
- Use generator loop instead of Array.from for key enumeration (Finding 3)
- Selective barrel export — exclude _resetTenantMiddlewareStrictCache (Finding 5)
- Move isMainThread check to module level, remove per-request check (Finding 9)
- Move mid-file require to top of app.js (Finding 8)
- Parallelize invalidateConfigCaches with Promise.all (Finding 10)
- Remove clearOverrideCache from public app.js exports (internal only)
- Strengthen getUserPrincipals comment re: ALS dependency (Finding 2)

* fix: restore runAsSystem for startup DB ops, consolidate require, clarify baseOnly

- Restore runAsSystem() around performStartupChecks, updateInterfacePermissions,
  initializeMCPs, and initializeOAuthReconnectManager — these make Mongoose
  queries that need system context in strict tenant mode (NEW-3)
- Consolidate duplicate require('@librechat/api') in requireJwtAuth.js (NEW-1)
- Document that baseOnly ignores role/userId/tenantId in JSDoc (NEW-2)

* test: add requireJwtAuth tenant chaining + invalidateConfigCaches tests

- requireJwtAuth: 5 tests verifying ALS tenant context is set after
  passport auth, isolated between concurrent requests, and not set
  when user has no tenantId (Finding 6)
- invalidateConfigCaches: 4 tests verifying all four caches are cleared,
  tenantId is threaded through, partial failure is handled gracefully,
  and operations run in parallel via Promise.all (Finding 11)

* fix: address Copilot review — passport errors, namespaced cache keys, /base scoping

- Forward passport errors in requireJwtAuth before entering tenant
  middleware — prevents silent auth failures from reaching handlers (P1)
- Account for Keyv namespace prefix in clearOverrideCache — stored keys
  are namespaced as "APP_CONFIG:_OVERRIDE_:..." not "_OVERRIDE_:...",
  so override caches were never actually matched/cleared (P2)
- Remove role from getBaseConfig — /base should return tenant-scoped
  base config, not role-merged config that drifts per admin role (P2)
- Return tenantStorage.run() for cleaner async semantics
- Update mock cache in service.spec.ts to simulate Keyv namespacing

* fix: address second review — cache safety, code quality, test reliability

- Decouple cache invalidation from mutation response: fire-and-forget
  with logging so DB mutation success is not masked by cache failures
- Extract clearEndpointConfigCache helper from inline IIFE
- Move isMainThread check to lazy once-per-process guard (no import
  side effect)
- Memoize process.env read in overrideCacheKey to avoid per-request
  env lookups and log flooding in strict mode
- Remove flaky timer-based parallelism assertion, use structural check
- Merge orphaned double JSDoc block on getUserPrincipals
- Fix stale [getAppConfig] log prefix → [ensureBaseConfig]
- Fix import order in tenant.spec.ts (package types before local values)
- Replace "Finding 1" reference with self-contained description
- Use real tenantStorage primitives in requireJwtAuth spec mock

* fix: move JSDoc to correct function after clearEndpointConfigCache extraction

* refactor: remove Redis SCAN from clearOverrideCache, rely on TTL expiry

Redis SCAN causes 60s+ stalls under concurrent load (see #12410).
APP_CONFIG defaults to FORCED_IN_MEMORY_CACHE_NAMESPACES, so the
in-memory store.keys() path handles the standard case. When APP_CONFIG
is Redis-backed, overrides expire naturally via overrideCacheTtl (60s
default) — an acceptable window for admin config mutations.

* fix: remove return from tenantStorage.run to satisfy void middleware signature

* fix: address second review — cache safety, code quality, test reliability

- Switch invalidateConfigCaches from Promise.all to Promise.allSettled
  so partial failures are logged individually instead of producing one
  undifferentiated error (Finding 3)
- Gate overrideCacheKey strict-mode warning behind a once-per-process
  flag to prevent log flooding under load (Finding 4)
- Add test for passport error forwarding in requireJwtAuth — the
  if (err) { return next(err) } branch now has coverage (Finding 5)
- Add test for real partial failure in invalidateConfigCaches where
  clearAppConfigCache rejects (not just the swallowed endpoint error)

* chore: reorder imports in index.js and app.js for consistency

- Moved logger and runAsSystem imports to maintain a consistent import order across files.
- Improved code readability by ensuring related imports are grouped together.
2026-03-26 17:35:00 -04:00
Danny Avila
8ba2bde5c1
📦 refactor: Consolidate DB models, encapsulating Mongoose usage in data-schemas (#11830)
* chore: move database model methods to /packages/data-schemas

* chore: add TypeScript ESLint rule to warn on unused variables

* refactor: model imports to streamline access

- Consolidated model imports across various files to improve code organization and reduce redundancy.
- Updated imports for models such as Assistant, Message, Conversation, and others to a unified import path.
- Adjusted middleware and service files to reflect the new import structure, ensuring functionality remains intact.
- Enhanced test files to align with the new import paths, maintaining test coverage and integrity.

* chore: migrate database models to packages/data-schemas and refactor all direct Mongoose Model usage outside of data-schemas

* test: update agent model mocks in unit tests

- Added `getAgent` mock to `client.test.js` to enhance test coverage for agent-related functionality.
- Removed redundant `getAgent` and `getAgents` mocks from `openai.spec.js` and `responses.unit.spec.js` to streamline test setup and reduce duplication.
- Ensured consistency in agent mock implementations across test files.

* fix: update types in data-schemas

* refactor: enhance type definitions in transaction and spending methods

- Updated type definitions in `checkBalance.ts` to use specific request and response types.
- Refined `spendTokens.ts` to utilize a new `SpendTxData` interface for better clarity and type safety.
- Improved transaction handling in `transaction.ts` by introducing `TransactionResult` and `TxData` interfaces, ensuring consistent data structures across methods.
- Adjusted unit tests in `transaction.spec.ts` to accommodate new type definitions and enhance robustness.

* refactor: streamline model imports and enhance code organization

- Consolidated model imports across various controllers and services to a unified import path, improving code clarity and reducing redundancy.
- Updated multiple files to reflect the new import structure, ensuring all functionalities remain intact.
- Enhanced overall code organization by removing duplicate import statements and optimizing the usage of model methods.

* feat: implement loadAddedAgent and refactor agent loading logic

- Introduced `loadAddedAgent` function to handle loading agents from added conversations, supporting multi-convo parallel execution.
- Created a new `load.ts` file to encapsulate agent loading functionalities, including `loadEphemeralAgent` and `loadAgent`.
- Updated the `index.ts` file to export the new `load` module instead of the deprecated `loadAgent`.
- Enhanced type definitions and improved error handling in the agent loading process.
- Adjusted unit tests to reflect changes in the agent loading structure and ensure comprehensive coverage.

* refactor: enhance balance handling with new update interface

- Introduced `IBalanceUpdate` interface to streamline balance update operations across the codebase.
- Updated `upsertBalanceFields` method signatures in `balance.ts`, `transaction.ts`, and related tests to utilize the new interface for improved type safety.
- Adjusted type imports in `balance.spec.ts` to include `IBalanceUpdate`, ensuring consistency in balance management functionalities.
- Enhanced overall code clarity and maintainability by refining type definitions related to balance operations.

* feat: add unit tests for loadAgent functionality and enhance agent loading logic

- Introduced comprehensive unit tests for the `loadAgent` function, covering various scenarios including null and empty agent IDs, loading of ephemeral agents, and permission checks.
- Enhanced the `initializeClient` function by moving `getConvoFiles` to the correct position in the database method exports, ensuring proper functionality.
- Improved test coverage for agent loading, including handling of non-existent agents and user permissions.

* chore: reorder memory method exports for consistency

- Moved `deleteAllUserMemories` to the correct position in the exported memory methods, ensuring a consistent and logical order of method exports in `memory.ts`.
2026-03-21 14:28:53 -04:00
Danny Avila
5879b3f518
🔊 fix: Validate language format for OpenAI STT model (#10875)
* 🔊 fix: Validate language format for OpenAI STT model

* fix: Normalize input language model assignment in STTService

* refactor: Enhance error logging and language validation in STT and TTS services

* fix: Improve language validation in getValidatedLanguageCode function
2025-12-09 22:25:45 -05:00
alfo-dev
b4892d81d3
🔊 fix: Missing Proxy config in TTS and STT Services (#10852)
* Fix TTS STT proxy

* Add STT proxy env var

* Add TTS proxy env var

* chore: import order

* chore: import order

---------

Co-authored-by: Danny Avila <danacordially@gmail.com>
2025-12-09 20:23:03 -05:00
Rakshit
772b706e20
🎙️ fix: Azure OpenAI Speech-to-Text 400 Bad Request Error (#10355) 2025-11-05 10:27:34 -05:00
Danny Avila
0e05ff484f
🔄 refactor: OAI Image Edit Proxy, Speech Settings Handling, Import Query Data Usage (#10281)
* chore: correct startupConfig usage in ImportConversations component

* refactor: properly process configured speechToText and textToSpeech settings in getCustomConfigSpeech

* refactor: proxy configuration by utilizing HttpsProxyAgent for OpenAI Image Edits
2025-10-28 09:36:03 -04:00
Danny Avila
65c83317aa
🗣️ feat: Language Support for OpenAI Speech-to-Text (#9470) 2025-09-05 12:01:00 -04:00
Danny Avila
1764de53a5
➿ fix: use appConfig correctly in getVoices 2025-08-28 00:51:22 -04:00
Danny Avila
15d7a3d221
🎵 feat: Cumulative Transcription Support for External STT (#9318)
* 🔧 fix: TTS and STT Services to use AppConfig

- Updated `getProviderSchema` and `getProvider` methods to accept an optional `appConfig` parameter, allowing for more flexible configuration retrieval.
- Improved error handling by ensuring that the app configuration is checked before accessing TTS and STT schemas.
- Refactored `processTextToSpeech` and `streamAudio` methods to utilize the new `appConfig` parameter for better clarity and maintainability.

* feat: Cumulative Transcription Support for STT External

* style: fix medium-sized styling for admin settings dialogs
2025-08-27 18:56:04 -04:00
Danny Avila
48f6f8f2f8
📎 feat: Upload as Text Support for Plaintext, STT, RAG, and Token Limits (#8868)
* 🪶 feat: Add Support for Uploading Plaintext Files

feat: delineate between OCR and text handling in fileConfig field of config file

- also adds support for passing in mimetypes as just plain file extensions

feat: add showLabel bool to support future synthetic component DynamicDropdownInput

feat: add new combination dropdown-input component in params panel to support file type token limits

refactor: move hovercard to side to align with other hovercards

chore: clean up autogenerated comments

feat: add delineation to file upload path between text and ocr configured filetypes

feat: add token limit checks during file upload

refactor: move textParsing out of ocrEnabled logic

refactor: clean up types for filetype config

refactor: finish decoupling DynamicDropdownInput from fileTokenLimits

fix: move image token cost function into file to fix circular dependency causing unittest to fail and remove unused var for linter

chore: remove out of scope code following review

refactor: make fileTokenLimit conform to existing styles

chore: remove unused localization string

chore: undo changes to DynamicInput and other strays

feat: add fileTokenLimit to all provider config panels

fix: move textParsing back into ocr tool_resource block for now so that it doesn't interfere with other upload types

* 📤 feat: Add RAG API Endpoint Support for Text Parsing (#8849)

* feat: implement RAG API integration for text parsing with fallback to native parsing

* chore: remove TODO now that placeholder and fllback are implemented

* ✈️ refactor: Migrate Text Parsing to TS (#8892)

* refactor: move generateShortLivedToken to packages/api

* refactor: move textParsing logic into packages/api

* refactor: reduce nesting and dry code with createTextFile

* fix: add proper source handling

* fix: mock new parseText and parseTextNative functions in jest file

* ci: add test coverage for textParser

* 💬 feat: Add Audio File Support to Upload as Text (#8893)

* feat: add STT support for Upload as Text

* refactor: move processAudioFile to packages/api

* refactor: move textParsing from utils to files

* fix: remove audio/mp3 from unsupported mimetypes test since it is now supported

* ✂️ feat: Configurable File Token Limits and Truncation (#8911)

* feat: add configurable fileTokenLimit default value

* fix: add stt to fileConfig merge logic

* fix: add fileTokenLimit to mergeFileConfig logic so configurable value is actually respected from yaml

* feat: add token limiting to parsed text files

* fix: add extraction logic and update tests so fileTokenLimit isnt sent to LLM providers

* fix: address comments

* refactor: rename textTokenLimiter.ts to text.ts

* chore: update form-data package to address CVE-2025-7783 and update package-lock

* feat: use default supported mime types for ocr on frontend file validation

* fix: should be using logger.debug not console.debug

* fix: mock existsSync in text.spec.ts

* fix: mock logger rather than every one of its function calls

* fix: reorganize imports and streamline file upload processing logic

* refactor: update createTextFile function to use destructured parameters and improve readability

* chore: update file validation to use EToolResources for improved type safety

* chore: update import path for types in audio processing module

* fix: update file configuration access and replace console.debug with logger.debug for improved logging

---------

Co-authored-by: Dustin Healy <dustinhealy1@gmail.com>
Co-authored-by: Dustin Healy <54083382+dustinhealy@users.noreply.github.com>
2025-08-27 03:44:39 -04:00
Danny Avila
9a210971f5
🛜 refactor: Streamline App Config Usage (#9234)
* WIP: app.locals refactoring

WIP: appConfig

fix: update memory configuration retrieval to use getAppConfig based on user role

fix: update comment for AppConfig interface to clarify purpose

🏷️ refactor: Update tests to use getAppConfig for endpoint configurations

ci: Update AppService tests to initialize app config instead of app.locals

ci: Integrate getAppConfig into remaining tests

refactor: Update multer storage destination to use promise-based getAppConfig and improve error handling in tests

refactor: Rename initializeAppConfig to setAppConfig and update related tests

ci: Mock getAppConfig in various tests to provide default configurations

refactor: Update convertMCPToolsToPlugins to use mcpManager for server configuration and adjust related tests

chore: rename `Config/getAppConfig` -> `Config/app`

fix: streamline OpenAI image tools configuration by removing direct appConfig dependency and using function parameters

chore: correct parameter documentation for imageOutputType in ToolService.js

refactor: remove `getCustomConfig` dependency in config route

refactor: update domain validation to use appConfig for allowed domains

refactor: use appConfig registration property

chore: remove app parameter from AppService invocation

refactor: update AppConfig interface to correct registration and turnstile configurations

refactor: remove getCustomConfig dependency and use getAppConfig in PluginController, multer, and MCP services

refactor: replace getCustomConfig with getAppConfig in STTService, TTSService, and related files

refactor: replace getCustomConfig with getAppConfig in Conversation and Message models, update tempChatRetention functions to use AppConfig type

refactor: update getAppConfig calls in Conversation and Message models to include user role for temporary chat expiration

ci: update related tests

refactor: update getAppConfig call in getCustomConfigSpeech to include user role

fix: update appConfig usage to access allowedDomains from actions instead of registration

refactor: enhance AppConfig to include fileStrategies and update related file strategy logic

refactor: update imports to use normalizeEndpointName from @librechat/api and remove redundant definitions

chore: remove deprecated unused RunManager

refactor: get balance config primarily from appConfig

refactor: remove customConfig dependency for appConfig and streamline loadConfigModels logic

refactor: remove getCustomConfig usage and use app config in file citations

refactor: consolidate endpoint loading logic into loadEndpoints function

refactor: update appConfig access to use endpoints structure across various services

refactor: implement custom endpoints configuration and streamline endpoint loading logic

refactor: update getAppConfig call to include user role parameter

refactor: streamline endpoint configuration and enhance appConfig usage across services

refactor: replace getMCPAuthMap with getUserMCPAuthMap and remove unused getCustomConfig file

refactor: add type annotation for loadedEndpoints in loadEndpoints function

refactor: move /services/Files/images/parse to TS API

chore: add missing FILE_CITATIONS permission to IRole interface

refactor: restructure toolkits to TS API

refactor: separate manifest logic into its own module

refactor: consolidate tool loading logic into a new tools module for startup logic

refactor: move interface config logic to TS API

refactor: migrate checkEmailConfig to TypeScript and update imports

refactor: add FunctionTool interface and availableTools to AppConfig

refactor: decouple caching and DB operations from AppService, make part of consolidated `getAppConfig`

WIP: fix tests

* fix: rebase conflicts

* refactor: remove app.locals references

* refactor: replace getBalanceConfig with getAppConfig in various strategies and middleware

* refactor: replace appConfig?.balance with getBalanceConfig in various controllers and clients

* test: add balance configuration to titleConvo method in AgentClient tests

* chore: remove unused `openai-chat-tokens` package

* chore: remove unused imports in initializeMCPs.js

* refactor: update balance configuration to use getAppConfig instead of getBalanceConfig

* refactor: integrate configMiddleware for centralized configuration handling

* refactor: optimize email domain validation by removing unnecessary async calls

* refactor: simplify multer storage configuration by removing async calls

* refactor: reorder imports for better readability in user.js

* refactor: replace getAppConfig calls with req.config for improved performance

* chore: replace getAppConfig calls with req.config in tests for centralized configuration handling

* chore: remove unused override config

* refactor: add configMiddleware to endpoint route and replace getAppConfig with req.config

* chore: remove customConfig parameter from TTSService constructor

* refactor: pass appConfig from request to processFileCitations for improved configuration handling

* refactor: remove configMiddleware from endpoint route and retrieve appConfig directly in getEndpointsConfig if not in `req.config`

* test: add mockAppConfig to processFileCitations tests for improved configuration handling

* fix: pass req.config to hasCustomUserVars and call without await after synchronous refactor

* fix: type safety in useExportConversation

* refactor: retrieve appConfig using getAppConfig in PluginController and remove configMiddleware from plugins route, to avoid always retrieving when plugins are cached

* chore: change `MongoUser` typedef to `IUser`

* fix: Add `user` and `config` fields to ServerRequest and update JSDoc type annotations from Express.Request to ServerRequest

* fix: remove unused setAppConfig mock from Server configuration tests
2025-08-26 12:10:18 -04:00
Danny Avila
29ef91b4dd
🧠 feat: User Memories for Conversational Context (#7760)
* 🧠 feat: User Memories for Conversational Context

chore: mcp typing, use `t`

WIP: first pass, Memories UI

- Added MemoryViewer component for displaying, editing, and deleting user memories.
- Integrated data provider hooks for fetching, updating, and deleting memories.
- Implemented pagination and loading states for better user experience.
- Created unit tests for MemoryViewer to ensure functionality and interaction with data provider.
- Updated translation files to include new UI strings related to memories.

chore: move mcp-related files to own directory

chore: rename librechat-mcp to librechat-api

WIP: first pass, memory processing and data schemas

chore: linting in fileSearch.js query description

chore: rename librechat-api to @librechat/api across the project

WIP: first pass, functional memory agent

feat: add MemoryEditDialog and MemoryViewer components for managing user memories

- Introduced MemoryEditDialog for editing memory entries with validation and toast notifications.
- Updated MemoryViewer to support editing and deleting memories, including pagination and loading states.
- Enhanced data provider to handle memory updates with optional original key for better management.
- Added new localization strings for memory-related UI elements.

feat: add memory permissions management

- Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories.
- Added new API endpoints for updating memory permissions associated with roles.
- Created a new AdminSettings component for managing memory permissions in the frontend.
- Integrated memory permissions into the existing roles and permissions schemas.
- Updated the interface to include memory settings and permissions.
- Enhanced the MemoryViewer component to conditionally render admin settings based on user roles.
- Added localization support for memory permissions in the translation files.

feat: move AdminSettings component to a new position in MemoryViewer for better visibility

refactor: clean up commented code in MemoryViewer component

feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration

- Added a search input to filter memories in the MemoryViewer component.
- Refactored MemoryEditDialog to accept children for better customization.
- Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories.
- Improved localization support by adding new strings for memory filtering and deletion confirmation.

refactor: optimize memory filtering in MemoryViewer using match-sorter

- Replaced manual filtering logic with match-sorter for improved search functionality.
- Enhanced performance and readability of the filteredMemories computation.

feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling

feat: implement access control for MemoryEditDialog and MemoryViewer components

refactor: remove commented out code and create runMemory method

refactor: rename role based files

feat: implement access control for memory usage in AgentClient

refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code

refactor: make `agents` dir in api package

refactor: migrate Azure utilities to TypeScript and consolidate imports

refactor: move sanitizeFilename function to a new file and update imports, add related tests

refactor: update LLM configuration types and consolidate Azure options in the API package

chore: linting

chore: import order

refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file

chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json

refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports

refactor: move createRun function to a new run.ts file and update related imports

fix: ensure safeAttachments is correctly typed as an array of TFile

chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file

refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options

feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling

fix: update types due to new TEndpointOption typing

fix: ensure safe access to group parameters in initializeOpenAIOptions function

fix: remove redundant API key validation comment in initializeOpenAIOptions function

refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation

refactor: decouple req.body fields and tool loading from initializeAgentOptions

chore: linting

refactor: adjust column widths in MemoryViewer for improved layout

refactor: simplify agent initialization by creating loadAgent function and removing unused code

feat: add memory configuration loading and validation functions

WIP: first pass, memory processing with config

feat: implement memory callback and artifact handling

feat: implement memory artifacts display and processing updates

feat: add memory configuration options and schema validation for validKeys

fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements

refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling

WIP: initial tokenLimit config and move Tokenizer to @librechat/api

refactor: update mongoMeili plugin methods to use callback for better error handling

feat: enhance memory management with token tracking and usage metrics

- Added token counting for memory entries to enforce limits and provide usage statistics.
- Updated memory retrieval and update routes to include total token usage and limit.
- Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information.
- Refactored memory processing functions to handle token limits and provide feedback on memory capacity.

feat: implement memory artifact handling in attachment handler

- Enhanced useAttachmentHandler to process memory artifacts when receiving updates.
- Introduced handleMemoryArtifact utility to manage memory updates and deletions.
- Updated query client to reflect changes in memory state based on incoming data.

refactor: restructure web search key extraction logic

- Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys.
- Updated webSearchKeys to utilize the new function for improved clarity and maintainability.
- Prevents build time errors

feat: add personalization settings and memory preferences management

- Introduced a new Personalization tab in settings to manage user memory preferences.
- Implemented API endpoints and client-side logic for updating memory preferences.
- Enhanced user interface components to reflect personalization options and memory usage.
- Updated permissions to allow users to opt out of memory features.
- Added localization support for new settings and messages related to personalization.

style: personalization switch class

feat: add PersonalizationIcon and align Side Panel UI

feat: implement memory creation functionality

- Added a new API endpoint for creating memory entries, including validation for key and value.
- Introduced MemoryCreateDialog component for user interface to facilitate memory creation.
- Integrated token limit checks to prevent exceeding user memory capacity.
- Updated MemoryViewer to include a button for opening the memory creation dialog.
- Enhanced localization support for new messages related to memory creation.

feat: enhance message processing with configurable window size

- Updated AgentClient to use a configurable message window size for processing messages.
- Introduced messageWindowSize option in memory configuration schema with a default value of 5.
- Improved logic for selecting messages to process based on the configured window size.

chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json

chore: remove OpenAPIPlugin and its associated tests

chore: remove MIGRATION_README.md as migration tasks are completed

ci: fix backend tests

chore: remove unused translation keys from localization file

chore: remove problematic test file and unused var in AgentClient

chore: remove unused import and import directly for JSDoc

* feat: add api package build stage in Dockerfile for improved modularity

* docs: reorder build steps in contributing guide for clarity
2025-06-07 18:52:22 -04:00
Danny Avila
3c91f7b0b7
🚀 feat: Enhance S3 URL Expiry with Refresh; fix: S3 File Deletion (#6647)
* refactor: Improve error logging in image fetching to base64 conversion

* fix: Add error handling for custom endpoint configuration retrieval

* fix: Update audio stream processing to parse text parts from complex message content

* chore: import order in streamAudio

* fix: S3 file deletion and optimize file upload

* feat: Implement S3 URL refresh mechanism and add cache for expiry check intervals

* feat: Add S3 URL refresh functionality for agent avatars

* chore: remove unnecessary console.log in MultiMessage component

* chore: update version of librechat-data-provider to 0.7.77
2025-03-31 18:40:06 -04:00
Marco Beretta
20f353630e
🗣️ feat: add support for gpt-4o-transcribe models (#6483) 2025-03-23 11:26:06 -04:00
Danny Avila
591a019766
🏄‍♂️ refactor: Optimize Reasoning UI & Token Streaming (#5546)
* ✨ feat: Implement Show Thinking feature; refactor: testing thinking render optimizations

* ✨ feat: Refactor Thinking component styles and enhance Markdown rendering

* chore: add back removed code, revert type changes

* chore: Add back resetCounter effect to Markdown component for improved code block indexing

* chore: bump @librechat/agents and google langchain packages

* WIP: reasoning type updates

* WIP: first pass, reasoning content blocks

* chore: revert code

* chore: bump @librechat/agents

* refactor: optimize reasoning tag handling

* style: ul indent padding

* feat: add Reasoning component to handle reasoning display

* feat: first pass, content reasoning part styling

* refactor: add content placeholder for endpoints using new stream handler

* refactor: only cache messages when requesting stream audio

* fix: circular dep.

* fix: add default param

* refactor: tts, only request after message stream, fix chrome autoplay

* style: update label for submitting state and add localization for 'Thinking...'

* fix: improve global audio pause logic and reset active run ID

* fix: handle artifact edge cases

* fix: remove unnecessary console log from artifact update test

* feat: add support for continued message handling with new streaming method

---------

Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com>
2025-01-29 19:46:58 -05:00
Marco Beretta
affcebd48c
⚙️ fix: update Azure OpenAI STT/TTS env handling (#4859) 2024-12-04 11:44:00 -05:00
Danny Avila
d012da0065
🛡️ fix: Enhance File Upload Security & Error Handling (#4705)
* fix: sanitize filename in multer storage callback

* fix: ensure temporary image upload file is deleted after processing

* fix: prevent cleanup flag from being set to false before actually deleted

* refactor: user avatar, typing, use 'file' for formData instead of 'input', add disk storage, use localization

* fix: update Avatar component to include image dimensions in formData and refactor editor reference type

* fix: refactor avatar upload handling to use fs for file reading and enhance file validation

* fix: ensure temporary image upload file is deleted after processing

* fix: refactor avatar upload routes and handlers for agents and assistants, improve file handling and validation

* fix: improve audio file validation and cleanup

* fix: add filename sanitization utility and integrate it into multer storage configuration

* fix: update group project ID check for null and refactor delete prompt group response type

* fix: invalid access control for deleting prompt groups

* fix: add error handling and logging to checkBan middleware

* fix: catch conversation parsing errors

* chore: revert unnecessary height and width parameters from avatar upload

* chore: update librechat-data-provider version to 0.7.55

* style: ensure KaTeX can spread across visible space
2024-11-12 16:41:04 -05:00
Danny Avila
2e519f9b57
🤖 feat: Custom Endpoint Agents (experimental) (#4627)
* wip: first pass, custom endpoint agents

* chore: imports

* chore: consolidate exports

* fix: imports

* feat: convert message.content array to strings for legacy format handling (deepseek/groq)

* refactor: normalize ollama endpoint name

* refactor: update mocking in isDomainAllowed.spec.js

* refactor: update deepseekModels in tokens.js and tokens.spec.js
2024-11-04 12:59:04 -05:00
Danny Avila
94d1afee84
🛡️ chore: address several npm vulnerabilities (#4151)
* chore: bump express to 4.21.0 to address CVE-2024-45590 and CVE-2024-43796

* chore: npm audit fix

* chore: uninstall unused `ws` dependency

* chore: bump nodemailer to 6.9.15

* chore: bump mongoose to v7.3.3

* chore: bump lint-staged for micromatch upgrade

* chore: bump axios to 1.7.7

* chore: npm audit fix for mongodb/mongoose vulns
2024-09-19 20:28:32 -04:00
Max Sanna
63b80c3067
🗣️ fix: Azure OpenAI STT (#3731)
* Fix for Azure OpenAI STT

* chore(STTService): imports order

---------

Co-authored-by: Danny Avila <danacordially@gmail.com>
2024-08-30 15:11:15 -04:00
Marco Beretta
62881fee54
🔧 fix: handle missing custom config speech (#3790)
* feat: Update speech settings retrieval logic to handle missing custom configuration

This commit updates the logic in the Speech component and the getCustomConfigSpeech function to handle the case where the custom configuration is missing. Previously, if no custom configuration was found, an error would occur. Now, the code checks for the presence of the custom configuration and returns a message indicating that no custom configuration was found. This improves the robustness of the application and provides a better user experience.

* refactor: changed response message when no custom config is found
2024-08-27 06:09:04 -04:00
Danny Avila
6655304753
🎙️ a11y: Screen Reader Support for Dynamic Content Updates (#3625)
* WIP: first pass, hooks

* wip: isStream arg

* feat: first pass, dynamic content updates, screen reader announcements

* chore: unrelated, styling redundancy
2024-08-13 03:04:27 -04:00
Marco Beretta
27ab00c6a9
🔉 fix: Elevenlabs TTS and STT (#3482)
* fix: add `await` on getVoices

* fix: STT function

* fix: elevenlabs

* chore(lint): remove unnecessary whitespace in ldap.js

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2024-07-30 12:46:46 -04:00
Marco Beretta
51cd847606
🗣️ refactor: speech services; fix: OpenAI STT (#3431)
* fix: OpenAI STT

* refactor: STT and TTS service, slightly imporve of performance

* fix(DecibelSelector): update default value
2024-07-30 09:18:52 -04:00
Danny Avila
f6125ccd59
🌿 fix: Message Route Improvements pt. 2 (#3379)
* fix: edge case with debugTraverse function in parsers.js

* chore: Update error message in audio stream processing

* refactor: Add ONE_MINUTE and THIRTY_SECONDS options to Time enum

* fix: logging
2024-07-18 13:56:19 -04:00
Danny Avila
5d40d0a37a
⚙️ feat: Adjust Rate of Stream Progress (#3244)
* chore: bump data-provider and add MESSAGES CacheKey

* refactor: avoid saving messages while streaming, save partial text to cache instead

* fix(ci): processChunks

* chore: logging aborted request to debug

* feat: set stream rate for token processing

* chore: specify default stream rate

* fix(ci): Update AppService.js to use optional chaining for endpointLocals assignment

* refactor: abstract the error handler

* feat: streamRate for assistants; refactor: update default rate for token

* refactor: update error handling in assistants/errors.js

* refactor: update error handling in assistants/errors.js
2024-07-17 10:47:17 -04:00
Marco Beretta
73dbf3eb20
🌐 feat: disable external engine if not configured (#3313)
* feat: disable external engine if not configured

* remove comment
2024-07-17 10:08:43 -04:00
Marco Beretta
f959ee302c
🗣️ fix: get speechTab config; feature: not overriding variables selected by user (#3282)
* fix(Speech): speechTab settings update

* fix: get speech config; refactor: moved everything to types and removed file types; feature: not overriding variables selected by user
2024-07-10 16:38:36 -04:00
Marco Beretta
a05e2c1dcc
🗣️ feat: Azure OpenAI speech (#2985)
* feat: Azure STT

* feat: Azure TTS

* refactor: use enums

* fix: frontend tests

* fix(config): wrong key provider
2024-07-10 16:33:06 -04:00
Marco Beretta
55f8d9910e
⚒️ fix(speechToText): OpenAI Provider (#3283) 2024-07-07 00:32:19 +02:00
Marco Beretta
1aad315de6
🎤 feat: add custom speech config, browser TTS/STT features, and dynamic speech tab settings (#2921)
* feat: update useTextToSpeech and useSpeechToText hooks to support external audio endpoints

This commit updates the useTextToSpeech and useSpeechToText hooks in the Input directory to support external audio endpoints. It introduces the useGetExternalTextToSpeech and useGetExternalSpeechToText hooks, which determine whether the audio endpoints should be set to 'browser' or 'external' based on the value of the endpointTTS and endpointSTT Recoil states. The useTextToSpeech and useSpeechToText hooks now use these new hooks to determine whether to use external audio endpoints

* feat: add userSelect style to ConversationModeSwitch label

* fix: remove unused updateTokenWebsocket function and import

The updateTokenWebsocket function and its import are no longer used in the OpenAIClient module. This commit removes the function and import to clean up the codebase

* feat: support external audio endpoints in useTextToSpeech and useSpeechToText hooks

This commit updates the useTextToSpeech and useSpeechToText hooks in the Input directory to support external audio endpoints. It introduces the useGetExternalTextToSpeech and useGetExternalSpeechToText hooks, which determine whether the audio endpoints should be set to 'browser' or 'external' based on the value of the endpointTTS and endpointSTT Recoil states. The useTextToSpeech and useSpeechToText hooks now use these new hooks to determine whether to use external audio endpoints

* feat: update AutomaticPlayback component to AutomaticPlaybackSwitch; tests: added AutomaticPlaybackSwitch.spec
>
> This commit renames the AutomaticPlayback component to AutomaticPlaybackSwitch in the Speech directory. The new name better reflects the purpose of the component and aligns with the naming convention used in the codebase.

* feat: update useSpeechToText hook to include interimTranscript

This commit updates the useSpeechToText hook in the client/src/components/Chat/Input/AudioRecorder.tsx file to include the interimTranscript state. This allows for real-time display of the speech-to-text transcription while the user is still speaking. The interimTranscript is now used to update the text area value during recording.

* feat: Add customConfigSpeech API endpoint for retrieving custom speech configuration

This commit adds a new API endpoint  in the  file under the  directory. This endpoint is responsible for retrieving the custom speech configuration using the  function from the  module

* feat: update store var  and ; fix: getCustomConfigSpeech

* fix: client tests, removed unused import

* feat: Update useCustomConfigSpeechQuery to return an array of custom speech configurations

This commit modifies the useCustomConfigSpeechQuery function in the client/src/data-provider/queries.ts file to return an array of custom speech configurations instead of a single object. This change allows for better handling and manipulation of the data in the application

* feat: Update useCustomConfigSpeechQuery to return an array of custom speech configurations

* refactor: Update variable name in speechTab schema

* refactor: removed unused and nested code

* fix: using recoilState

* refactor: Update Speech component to use useCallback for setting settings

* fix: test

* fix: tests

* feature: ensure that the settings don't change after modifying then through the UI

* remove comment

* fix: Handle error gracefully in getCustomConfigSpeech and getVoices endpoints

* fix: Handle error

* fix: backend tests

* fix: invalid custom config logging

* chore: add back custom config info logging

* chore: revert loadCustomConfig spec

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2024-07-05 10:13:34 -04:00
Danny Avila
5dc5d875ba
🤖 feat: Private Assistants (#2881)
* feat: add configuration for user private assistants

* filter private assistant message requests

* add test for privateAssistants

* add privateAssistants configuration to tests

* fix: destructuring error when assistants config is not added

* chore: revert chat controller changes

* chore: add payload type, add metadata types

* feat: validateAssistant

* refactor(fetchAssistants): allow for flexibility

* feat: validateAuthor

* refactor: return all assistants to ADMIN role

* feat: add assistant doc on assistant creation

* refactor(listAssistants): use `listAllAssistants` to exhaustively fetch all assistants

* chore: add suggestion to tts error

* refactor(validateAuthor): attempt database check first

* refactor: author validation when patching/deleting assistant

---------

Co-authored-by: Leon Juenemann <leon.juenemann@maibornwolff.de>
2024-05-28 08:27:45 -04:00
Danny Avila
35ba4ba1a4
🔊 fix(tts): NotAllowedError (mobile/safari), Unsupported MediaSource type (firefox), Hide Audio Element (#2854)
* fix: hide audio element on mobile

* chore: add tts docs link

* fix: select voice option on first render

* fix: NotAllowedError, prevent async playback for mobile triggers, consolidate MessageAudio code, user user-triggered unmutes

* fix: Firefox/unsupported type for MediaSource hack

* refactor(STT): make icon red when recording. consolidate logic to AudioRecorder component

* fix: revert Redis changes to use separate client for sessions
2024-05-24 12:18:11 -04:00
Danny Avila
514a502b9c
⏯️ fix(tts): Resolve Voice Selection and Manual Playback Issues (#2845)
* fix: voice setting for autoplayback TTS

* fix(useTextToSpeechExternal): resolve stateful playback issues and consolidate state logic

* refactor: initialize tts voice and provider schema once per request

* fix(tts): edge case, longer text inputs. TODO: use continuous stream for longer text inputs

* fix(tts): pause global audio on conversation change

* refactor: keyvMongo ban cache to allow db updates for unbanning, to prevent server restart

* chore: eslint fix

* refactor: make ban cache exclusively keyvMongo
2024-05-23 16:27:36 -04:00
Danny Avila
8e66683577
🗣️ fix(tts): Add Text Parser for Message Content Parts (#2840)
* fix: manual TTS trigger for message content parts

* ci(streamAudio): processChunks test
2024-05-22 23:27:37 -04:00
bsu3338
6cf1c85363
🐞 fix: add LocalAI to GET /voices (#2837) 2024-05-22 22:42:27 -04:00