LibreChat/packages/api/src/utils
Danny Avila 2f8bc3dfcd
🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937)
* feat: send video/audio to custom endpoints that opt in via supportedMimeTypes

The video and audio encoders only emitted OpenAI-format parts (video_url,
input_audio) for OpenRouter, and Google's media part for Google/Vertex. For
every other provider the file was validated and then silently dropped, so a
custom OpenAI-compatible endpoint (vLLM, LiteLLM, ...) never received the
attachment even though the wire format is identical to OpenRouter's.

Emit the OpenAI-format part for OpenAI-like providers when the endpoint's
fileConfig supportedMimeTypes explicitly allows the file's type. The inherited
default list does not count as opting in (isExplicitMimeConfig, keyed on
referential identity like the client's picker), so endpoints whose gateway
cannot handle media are unaffected.

* feat: let a configured custom endpoint pick and attach video/audio

The attach menu's Upload to Provider filter and the drag-drop viability check
only opened video/audio for Google and OpenRouter, or for a custom endpoint
with a fully permissive (.*) supportedMimeTypes. Honor any admin-configured
allowlist on a custom endpoint instead, so a finite list that includes
video/.* or audio/.* is enough, matching the server-side encoders.

* fix: canonicalize input_audio format from MIME type

The `input_audio.format` value was derived from the filename extension via
`filename.split('.').pop()`, which produces values providers reject:

- `clip.wave` (audio/wave) emitted `wave`, but the accepted format is `wav`
- a file with no extension (`recording`) emitted the whole filename, since
  splitting a dotless string yields a single-element array, so the existing
  non-empty guard never fired
- `clip.mpeg` (audio/mpeg) emitted `mpeg` rather than `mp3`

All of these pass MIME validation and then fail at the provider. Derive the
format from the MIME type instead, mirroring the canonicalization already in
STTService, and fall back to the filename extension only when it is itself a
supported format. Throws when neither source yields one rather than sending an
unsupported value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: route opted-in custom endpoint media to the provider path

In unified upload mode an upload carries no tool resource, so its delivery path is
inferred by resolveDefaultLLMDeliveryPath. That resolver judged audio and video for any
named endpoint with isMediaSupportedProvider, which lists only Google, Vertex and
OpenRouter, so a custom endpoint that opted into media through supportedMimeTypes still
recorded video as none and audio as text, and the encoder branch was never reached.

The upload resolver already holds the endpoint's file config, so it now passes the
endpoint's supportedMimeTypes through, and an explicit match on an OpenAI-compatible or
custom endpoint counts as provider-capable, mirroring isConfiguredProviderMediaType on
the encoder side. The inherited default list is still not an opt-in, and known providers
whose encoders emit nothing for media are unchanged. BaseClient re-resolves per turn with
the same function, so the turn path follows.

* fix: scope the media opt-in to custom endpoints that run as OpenAI

The opt-in applied to the built-in openAI and azureOpenAI endpoints too, but the picker
and drag-drop only open media for custom endpoints, so that was a route the client could
not send to. It is now limited to endpoints that are not a known provider.

A custom endpoint may also declare provider: anthropic, in which case initialization runs
it as Anthropic and the encoders emit no media part for it. getCustomEndpointProvider
reads that declaration off the app config, the upload callers pass it as endpointProvider,
the agent encoder and BaseClient pass the agent's resolved provider, and an endpoint known
to run as something other than OpenAI keeps its previous route: video stays off the model
path and audio still reaches transcription.

* fix: reject unsupported provider audio during upload

---------

Co-authored-by: Renato Garita Figueiredo <renato.garitafigueiredo@uni-osnabrueck.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 16:38:51 -04:00
..
__tests__
axios.spec.ts
axios.ts 🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784) 2026-09-09 15:48:11 -04:00
azure.spec.ts
azure.ts 📫 fix: Route Azure Astra Through Responses API (#15845) 2026-09-14 15:06:35 -04:00
code.spec.ts 🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784) 2026-09-09 15:48:11 -04:00
code.ts 🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784) 2026-09-09 15:48:11 -04:00
common.spec.ts 🈳 fix: Treat Blank Mongo Index Settings as Unset (#15809) 2026-09-09 23:24:18 -04:00
common.ts 🈳 fix: Treat Blank Mongo Index Settings as Unset (#15809) 2026-09-09 23:24:18 -04:00
content.spec.ts 📭 fix: Drop Empty Summary Parts Before Persistence (#15338) 2026-08-29 17:24:35 -04:00
content.ts 📭 fix: Drop Empty Summary Parts Before Persistence (#15338) 2026-08-29 17:24:35 -04:00
email.ts
env.spec.ts 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
env.ts 🪪 fix: Reject ID Tokens as the OpenID Bearer Reuse Access Token (#15317) 2026-08-28 10:11:32 -04:00
errors.spec.ts 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
errors.ts 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
events.ts
files.spec.ts 🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937) 2026-09-14 16:38:51 -04:00
files.ts 🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937) 2026-09-14 16:38:51 -04:00
generators.ts
graph.spec.ts
graph.ts
headers.spec.ts 🪪 feat: Propagate Tenant to Agent Model Headers (#15521) 2026-09-02 13:25:38 -04:00
headers.ts 🪪 feat: Propagate Tenant to Agent Model Headers (#15521) 2026-09-02 13:25:38 -04:00
http.ts
identity.spec.ts 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
identity.ts 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
import.ts
index.ts 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
key.test.ts 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
key.ts 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
latex.spec.ts
latex.ts
limiter.spec.ts 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
limiter.ts 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
llm.test.ts 🖍️ fix: Honor Agent Image Detail Settings (#15407) 2026-08-31 08:34:05 -04:00
llm.ts 🖍️ fix: Honor Agent Image Detail Settings (#15407) 2026-08-31 08:34:05 -04:00
math.integration.spec.ts
math.spec.ts
math.ts
memory.ts
message.spec.ts
message.ts
oidc.spec.ts 🪪 fix: Reject ID Tokens as the OpenID Bearer Reuse Access Token (#15317) 2026-08-28 10:11:32 -04:00
oidc.ts 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
openid.spec.ts 🔐 fix: Keep OpenID Metadata Out of Logs and Traces (#15470) 2026-09-01 13:22:12 -04:00
openid.ts 🔐 fix: Keep OpenID Metadata Out of Logs and Traces (#15470) 2026-09-01 13:22:12 -04:00
parity.spec.ts
path.spec.ts
path.ts
pluginEnv.spec.ts
ports.spec.ts
ports.ts
promise.spec.ts
promise.ts
promises.spec.ts 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
promises.ts 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
proxy.spec.ts
proxy.ts
query.spec.ts 🔎 feat: Filter and Sort the Sidebar Chats List (#15842) 2026-09-12 15:56:10 -04:00
query.ts 🔎 feat: Filter and Sort the Sidebar Chats List (#15842) 2026-09-12 15:56:10 -04:00
quotes.spec.ts
quotes.ts 🛂 feat: Filter Model-Bound Content by Source (#14425) 2026-08-21 22:43:32 -04:00
sanitizeTitle.spec.ts
sanitizeTitle.ts
text.spec.ts
text.ts
tokenizer.spec.ts 🧾 fix: Count the Tool Results a Tool-Limit Stop Retains (#15893) 2026-09-13 22:20:25 -04:00
tokenizer.ts 🧾 fix: Count the Tool Results a Tool-Limit Stop Retains (#15893) 2026-09-13 22:20:25 -04:00
tokenMap.ts
tokens.spec.ts 🌌 feat: Add GPT-6 Astra Support (#15567) 2026-09-04 10:23:29 -04:00
tokens.ts 🌌 feat: Add GPT-6 Astra Support (#15567) 2026-09-04 10:23:29 -04:00
tracing.ts
url.spec.ts
url.ts
yaml.ts