LibreChat/api/server
Danny Avila 2f8bc3dfcd
🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937)
* feat: send video/audio to custom endpoints that opt in via supportedMimeTypes

The video and audio encoders only emitted OpenAI-format parts (video_url,
input_audio) for OpenRouter, and Google's media part for Google/Vertex. For
every other provider the file was validated and then silently dropped, so a
custom OpenAI-compatible endpoint (vLLM, LiteLLM, ...) never received the
attachment even though the wire format is identical to OpenRouter's.

Emit the OpenAI-format part for OpenAI-like providers when the endpoint's
fileConfig supportedMimeTypes explicitly allows the file's type. The inherited
default list does not count as opting in (isExplicitMimeConfig, keyed on
referential identity like the client's picker), so endpoints whose gateway
cannot handle media are unaffected.

* feat: let a configured custom endpoint pick and attach video/audio

The attach menu's Upload to Provider filter and the drag-drop viability check
only opened video/audio for Google and OpenRouter, or for a custom endpoint
with a fully permissive (.*) supportedMimeTypes. Honor any admin-configured
allowlist on a custom endpoint instead, so a finite list that includes
video/.* or audio/.* is enough, matching the server-side encoders.

* fix: canonicalize input_audio format from MIME type

The `input_audio.format` value was derived from the filename extension via
`filename.split('.').pop()`, which produces values providers reject:

- `clip.wave` (audio/wave) emitted `wave`, but the accepted format is `wav`
- a file with no extension (`recording`) emitted the whole filename, since
  splitting a dotless string yields a single-element array, so the existing
  non-empty guard never fired
- `clip.mpeg` (audio/mpeg) emitted `mpeg` rather than `mp3`

All of these pass MIME validation and then fail at the provider. Derive the
format from the MIME type instead, mirroring the canonicalization already in
STTService, and fall back to the filename extension only when it is itself a
supported format. Throws when neither source yields one rather than sending an
unsupported value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: route opted-in custom endpoint media to the provider path

In unified upload mode an upload carries no tool resource, so its delivery path is
inferred by resolveDefaultLLMDeliveryPath. That resolver judged audio and video for any
named endpoint with isMediaSupportedProvider, which lists only Google, Vertex and
OpenRouter, so a custom endpoint that opted into media through supportedMimeTypes still
recorded video as none and audio as text, and the encoder branch was never reached.

The upload resolver already holds the endpoint's file config, so it now passes the
endpoint's supportedMimeTypes through, and an explicit match on an OpenAI-compatible or
custom endpoint counts as provider-capable, mirroring isConfiguredProviderMediaType on
the encoder side. The inherited default list is still not an opt-in, and known providers
whose encoders emit nothing for media are unchanged. BaseClient re-resolves per turn with
the same function, so the turn path follows.

* fix: scope the media opt-in to custom endpoints that run as OpenAI

The opt-in applied to the built-in openAI and azureOpenAI endpoints too, but the picker
and drag-drop only open media for custom endpoints, so that was a route the client could
not send to. It is now limited to endpoints that are not a known provider.

A custom endpoint may also declare provider: anthropic, in which case initialization runs
it as Anthropic and the encoders emit no media part for it. getCustomEndpointProvider
reads that declaration off the app config, the upload callers pass it as endpointProvider,
the agent encoder and BaseClient pass the agent's resolved provider, and an endpoint known
to run as something other than OpenAI keeps its previous route: video stays off the model
path and audio still reaches transcription.

* fix: reject unsupported provider audio during upload

---------

Co-authored-by: Renato Garita Figueiredo <renato.garitafigueiredo@uni-osnabrueck.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 16:38:51 -04:00
..
controllers 👯 fix: Reject Duplicate Tool Approval Identities (#15933) 2026-09-14 15:08:26 -04:00
middleware 🗯️ fix: Give Failed Turns Actionable Guidance (#15896) 2026-09-14 10:37:22 -04:00
routes 🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937) 2026-09-14 16:38:51 -04:00
services 🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely (#15937) 2026-09-14 16:38:51 -04:00
utils 🚄 perf: Linearize Conversation Import Lineage (#15881) 2026-09-13 03:48:22 -04:00
cleanup.js
csp.spec.js 🔌 fix: Close Leaked HTTP Server in Server Specs (#15821) 2026-09-10 15:17:06 -04:00
experimental.js 👑 fix: Elect a Single Expired-File Sweeper (#15909) 2026-09-14 02:05:52 -04:00
experimental.spec.js 🛎️ fix: Start Code Environment Reconciliation After Mongo Connects (#15620) 2026-09-04 19:08:44 -04:00
index.js 🪧 fix: Emit the Configured-Footer Answer With the HTML Shell (#15886) 2026-09-13 21:53:54 -04:00
index.metrics.spec.js 🔌 fix: Close Leaked HTTP Server in Server Specs (#15821) 2026-09-10 15:17:06 -04:00
index.spec.js 🪧 fix: Emit the Configured-Footer Answer With the HTML Shell (#15886) 2026-09-13 21:53:54 -04:00
socialLogins.js 🔁 feat: Retry OpenID Discovery After Startup Failures (#15899) 2026-09-13 23:16:26 -04:00
socialLogins.spec.js 🔁 feat: Retry OpenID Discovery After Startup Failures (#15899) 2026-09-13 23:16:26 -04:00
telemetry.js 🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565) 2026-09-03 21:16:09 -04:00
telemetry.spec.js 🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565) 2026-09-03 21:16:09 -04:00