LibreChat/api/server/services
Danny Avila 2f8bc3dfcd
🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937)
* feat: send video/audio to custom endpoints that opt in via supportedMimeTypes

The video and audio encoders only emitted OpenAI-format parts (video_url,
input_audio) for OpenRouter, and Google's media part for Google/Vertex. For
every other provider the file was validated and then silently dropped, so a
custom OpenAI-compatible endpoint (vLLM, LiteLLM, ...) never received the
attachment even though the wire format is identical to OpenRouter's.

Emit the OpenAI-format part for OpenAI-like providers when the endpoint's
fileConfig supportedMimeTypes explicitly allows the file's type. The inherited
default list does not count as opting in (isExplicitMimeConfig, keyed on
referential identity like the client's picker), so endpoints whose gateway
cannot handle media are unaffected.

* feat: let a configured custom endpoint pick and attach video/audio

The attach menu's Upload to Provider filter and the drag-drop viability check
only opened video/audio for Google and OpenRouter, or for a custom endpoint
with a fully permissive (.*) supportedMimeTypes. Honor any admin-configured
allowlist on a custom endpoint instead, so a finite list that includes
video/.* or audio/.* is enough, matching the server-side encoders.

* fix: canonicalize input_audio format from MIME type

The `input_audio.format` value was derived from the filename extension via
`filename.split('.').pop()`, which produces values providers reject:

- `clip.wave` (audio/wave) emitted `wave`, but the accepted format is `wav`
- a file with no extension (`recording`) emitted the whole filename, since
  splitting a dotless string yields a single-element array, so the existing
  non-empty guard never fired
- `clip.mpeg` (audio/mpeg) emitted `mpeg` rather than `mp3`

All of these pass MIME validation and then fail at the provider. Derive the
format from the MIME type instead, mirroring the canonicalization already in
STTService, and fall back to the filename extension only when it is itself a
supported format. Throws when neither source yields one rather than sending an
unsupported value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: route opted-in custom endpoint media to the provider path

In unified upload mode an upload carries no tool resource, so its delivery path is
inferred by resolveDefaultLLMDeliveryPath. That resolver judged audio and video for any
named endpoint with isMediaSupportedProvider, which lists only Google, Vertex and
OpenRouter, so a custom endpoint that opted into media through supportedMimeTypes still
recorded video as none and audio as text, and the encoder branch was never reached.

The upload resolver already holds the endpoint's file config, so it now passes the
endpoint's supportedMimeTypes through, and an explicit match on an OpenAI-compatible or
custom endpoint counts as provider-capable, mirroring isConfiguredProviderMediaType on
the encoder side. The inherited default list is still not an opt-in, and known providers
whose encoders emit nothing for media are unchanged. BaseClient re-resolves per turn with
the same function, so the turn path follows.

* fix: scope the media opt-in to custom endpoints that run as OpenAI

The opt-in applied to the built-in openAI and azureOpenAI endpoints too, but the picker
and drag-drop only open media for custom endpoints, so that was a route the client could
not send to. It is now limited to endpoints that are not a known provider.

A custom endpoint may also declare provider: anthropic, in which case initialization runs
it as Anthropic and the encoders emit no media part for it. getCustomEndpointProvider
reads that declaration off the app config, the upload callers pass it as endpointProvider,
the agent encoder and BaseClient pass the agent's resolved provider, and an endpoint known
to run as something other than OpenAI keeps its previous route: video stays off the model
path and audio still reaches transcription.

* fix: reject unsupported provider audio during upload

---------

Co-authored-by: Renato Garita Figueiredo <renato.garitafigueiredo@uni-osnabrueck.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 16:38:51 -04:00
..
__tests__ 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
Agents 🪬 fix: Fence Agent Checkpoint Ownership and Cleanup Recovery (#15680) 2026-09-07 22:04:59 -04:00
Artifacts
Config 🔐 fix: Stop an MCP Credential Refresh From Fencing Its Own Connection (#15863) 2026-09-12 12:39:47 -04:00
Endpoints 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
Files 🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937) 2026-09-14 16:38:51 -04:00
MCP 🫧 fix: Retire Cancelled MCP OAuth State (#15702) 2026-09-07 15:43:02 -04:00
Runs
Schedules 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
Skills 📜 feat: Add Skills Discovery and Authoring to the Management API (#15642) 2026-09-06 17:26:26 -04:00
start 🧫 chore: Keep Jest Specs Out of Tool Discovery (#15766) 2026-09-08 23:58:04 -04:00
Threads ⏳ fix: Separate Temporary and General Chat Retention (#15734) 2026-09-08 11:22:54 -04:00
Tools 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
ActionService.js 🔣 refactor: Replace Mongo Filters With Typed Criteria (#15580) 2026-09-04 11:54:21 -04:00
ActionService.spec.js 🛑 fix: Forward Run Abort Signal to Foreground Tool Calls (#15517) 2026-09-02 13:32:33 -04:00
AssistantService.js
AuthService.js 🧲 fix: Match Directory Users to Existing Principals (#15855) 2026-09-12 07:36:22 -04:00
AuthService.spec.js 🧲 fix: Match Directory Users to Existing Principals (#15855) 2026-09-12 07:36:22 -04:00
cleanup.js
createRunBody.js
createRunBody.spec.js
GraphApiService.js 🕰️ fix: Guard expires_in So a Token Response Cannot Outlive Its Credential (#15321) 2026-08-28 10:47:09 -04:00
GraphApiService.spec.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
GraphTokenService.js
initializeMCPs.js 🪂 fix: Release Stalled MCP Catalog Recovery Without Dropping Refreshed Tokens (#15871) 2026-09-12 22:23:49 -04:00
initializeMCPs.plugins.spec.js 🛂 fix: Back Off MCP OAuth Discovery (#15783) 2026-09-09 22:21:34 -04:00
initializeMCPs.spec.js 🪂 fix: Release Stalled MCP Catalog Recovery Without Dropping Refreshed Tokens (#15871) 2026-09-12 22:23:49 -04:00
initializeOAuthReconnectManager.js 🛂 fix: Back Off MCP OAuth Discovery (#15783) 2026-09-09 22:21:34 -04:00
initializeOAuthReconnectManager.spec.js 🛂 fix: Back Off MCP OAuth Discovery (#15783) 2026-09-09 22:21:34 -04:00
MCP.js 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
MCP.spec.js 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
MCPAuthorizationFenceRetry.js 🛂 fix: Back Off MCP OAuth Discovery (#15783) 2026-09-09 22:21:34 -04:00
MCPAuthorizationFenceRetry.spec.js 🛂 fix: Back Off MCP OAuth Discovery (#15783) 2026-09-09 22:21:34 -04:00
MCPRequestContext.js
OboPolicyService.js
OboTokenService.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
OboTokenService.spec.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
OpenIDRefreshFlight.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
OpenIDRefreshFlight.spec.js 🪫 fix: Recharge Direct OpenID MCP Bearer Sessions (#15704) 2026-09-07 21:05:26 -04:00
OpenIDRefreshRecovery.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
OpenIDSessionRefresh.js 🪫 fix: Recharge Direct OpenID MCP Bearer Sessions (#15704) 2026-09-07 21:05:26 -04:00
OpenIDSessionRefresh.spec.js 🪫 fix: Recharge Direct OpenID MCP Bearer Sessions (#15704) 2026-09-07 21:05:26 -04:00
PermissionService.js 🧲 fix: Match Directory Users to Existing Principals (#15855) 2026-09-12 07:36:22 -04:00
PermissionService.spec.js 🧲 fix: Match Directory Users to Existing Principals (#15855) 2026-09-12 07:36:22 -04:00
PluginService.js 🕸️ feat: Complete Keenable Web Search Provider (#15288) 2026-08-27 14:29:20 -04:00
PluginService.spec.js 🕸️ feat: Complete Keenable Web Search Provider (#15288) 2026-08-27 14:29:20 -04:00
RefreshTokenBridge.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
RefreshTokenBridge.spec.js 🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334) 2026-08-29 23:55:58 -04:00
systemGrant.spec.js
ToolService.js 🍵 fix: Support Renewable MCP Credentials for Schedules (#15908) 2026-09-14 10:27:47 -04:00
twoFactorService.js 🥁 fix: Compare TOTP Codes in Constant Time (#15157) 2026-08-24 09:10:03 -04:00
twoFactorService.spec.js 🥁 fix: Compare TOTP Codes in Constant Time (#15157) 2026-08-24 09:10:03 -04:00