📨 feat: Custom Headers on Built-in Provider Endpoints (#13742)

* 📨 feat: Custom Headers on Built-in Provider Endpoints

Add a `headers` config option to the built-in `openAI`, `anthropic`, and
`google` endpoints (incl. Anthropic/Google Vertex), mirroring the custom
endpoint header mechanism. Values support the same placeholder resolution
(env vars, `{{LIBRECHAT_USER_*}}`, `{{LIBRECHAT_BODY_CONVERSATIONID}}`) and
are resolved at request time so dynamic values like conversationId resolve
against the live request — without losing provider-native request shaping.

Closes #13082. Covers #13713: forwarding conversationId to a reverse proxy
is now `X-Conversation-Id: '{{LIBRECHAT_BODY_CONVERSATIONID}}'` — an unknown
header is ignored by the native Anthropic API, so no 400 and no metadata
gating needed.

- Schema: `headers` on `baseEndpointSchema` (openAI/google/anthropic/all).
- New `mergeHeaders`/`resolveConfigHeaders` utils centralize the per-provider
  header locations (`configuration.defaultHeaders`, Anthropic
  `clientOptions.defaultHeaders`, Google `customHeaders`); provider-managed
  headers (auth, `anthropic-beta`) always win on collision.
- Each initializer threads configured headers (endpoint over `all`) into the
  right place; request-time resolution runs across all locations in the main
  and title flows.

* 🩹 fix: Cast endpoints.all to TEndpoint for headers DeepPartial widening

Adding `headers` (a Record) to `baseEndpointSchema` makes `DeepPartial<TCustomConfig>`
widen its value type to `string | undefined`, which is not assignable to the
concrete `TEndpoint['headers']: Record<string, string>` at the `loadedEndpoints.all`
assignment. Cast at the assignment site, mirroring the existing
`anthropicConfig as TAnthropicEndpoint` cast in the same function.

* 🛡️ fix: Harden built-in endpoint custom headers (Codex review)

Address Codex P2 findings on the custom-headers feature:

- Anthropic title requests: `omitTitleOptions` strips the `clientOptions`
  carrier, which dropped its `defaultHeaders`. Preserve just the header carrier
  so gateway/reverse-proxy metadata still reaches title generation.
- mergeHeaders: match header names case-insensitively so an override (e.g. a
  provider-managed `Authorization`/`anthropic-beta`) replaces/uniones a
  case-variant from the base instead of emitting two names a client may collapse.
- OpenAI: withhold admin-configured headers when the user supplies the base URL
  (`user_provided`), since values may carry `${SECRET}`/token placeholders that
  must not reach a user-controlled endpoint — mirrors the custom-endpoint guard.
- Azure: honor global `endpoints.all` headers (same OpenAI carrier) while keeping
  Azure-managed `api-key`/version headers authoritative.

Adds tests for each.

* 🔐 fix: Resolve-once + provider-managed header safety (Codex review round 2)

Address Codex P2 findings:

- Azure: keep global `endpoints.all` headers unresolved at init and let
  request-time `resolveConfigHeaders` resolve them once, avoiding a
  second-order env expansion of already-substituted user values.
- Google: `resolveConfigHeaders` no longer template-resolves the
  provider-managed `Authorization` header (built from a possibly user-provided
  key), so a user key like `${ENV}` can't leak server environment values.
- Model fetches: thread configured headers (endpoint over `all`) + user object
  through `getOpenAIModels`/`getAnthropicModels` → `fetchModels`, so a
  gateway-fronted built-in provider receives the header on `/models` too. Fixed
  `fetchModels` to merge custom headers for Anthropic instead of overwriting
  them (managed `x-api-key`/version still win).

Adds/updates tests for each.

* 🧯 fix: Header provenance, memory/title coverage, idempotency (Codex round 3)

Address Codex P2 findings, including two regressions from the prior round:

- Google auth (findings 6 & 8): move native Google header resolution to init
  (`initializeGoogle`), resolving admin templates BEFORE the key-derived auth
  header is built. resolveConfigHeaders no longer touches Google `customHeaders`,
  so admin `Authorization` templates resolve again (fixes the round-2 regression)
  while the SDK auth header (possibly a user-provided key) is never env-expanded.
- Memory runs: memory extraction now calls `resolveConfigHeaders`, so native
  Anthropic (and OpenAI) headers resolve for memory requests too.
- Vertex titles: restore the ORIGINAL `clientOptions` object reference (not a
  copy) when preserving headers across `omitTitleOptions`, so the Vertex
  `createClient` closure and the resolved headers stay on the same object.
- Reuse: `resolveConfigHeaders` is now idempotent (resolve-once per header map),
  preventing a second pass from env-expanding values already substituted with
  user/body data when an agent object flows through buildAgentInput twice.

Adds/updates tests for each.

api/server/controllers/agents/client.js

@@ -9,9 +9,9 @@ const {
   logToolError,
   sanitizeTitle,
   payloadParser,
-  resolveHeaders,
   createSafeUser,
   initializeAgent,
+  resolveConfigHeaders,
   countTokens,
   getBalanceConfig,
   omitTitleOptions,
@@ -1624,12 +1624,25 @@ class AgentClient extends BaseClient {
       delete clientOptions.modelKwargs.max_output_tokens;
     }
 
+    /** `omitTitleOptions` drops the Anthropic `clientOptions` carrier (thinking,
+     *  streaming, etc.), which would also drop its `defaultHeaders` — preserve the
+     *  original `clientOptions` object so gateway/reverse-proxy metadata still
+     *  reaches title requests (the proxy may require it for auth/routing). Restore
+     *  the SAME object reference, not a copy: the Vertex `createClient` closure from
+     *  `getLLMConfig` closes over this object, so `resolveConfigHeaders` must mutate
+     *  the very object the client is built from. */
+    const anthropicClientOptions = clientOptions?.clientOptions;
+
     clientOptions = Object.assign(
       Object.fromEntries(
         Object.entries(clientOptions).filter(([key]) => !omitTitleOptions.has(key)),
       ),
     );
 
+    if (anthropicClientOptions?.defaultHeaders != null && clientOptions.clientOptions == null) {
+      clientOptions.clientOptions = anthropicClientOptions;
+    }
+
     if (
       provider === Providers.GOOGLE &&
       (endpointConfig?.titleMethod === TitleMethod.FUNCTIONS ||
@@ -1638,20 +1651,19 @@ class AgentClient extends BaseClient {
       clientOptions.json = true;
     }
 
-    /** Resolve request-based headers for Custom Endpoints. Note: if this is added to
-     *  non-custom endpoints, needs consideration of varying provider header configs.
+    /** Resolve request-based headers across provider-specific header locations:
+     *  OpenAI `configuration.defaultHeaders`, Anthropic `clientOptions.defaultHeaders`
+     *  (preserved above), and Google `customHeaders`.
      */
-    if (clientOptions?.configuration?.defaultHeaders != null) {
-      clientOptions.configuration.defaultHeaders = resolveHeaders({
-        headers: clientOptions.configuration.defaultHeaders,
-        user: createSafeUser(this.options.req?.user),
-        body: {
-          messageId: this.responseMessageId,
-          conversationId: this.conversationId,
-          parentMessageId: this.parentMessageId,
-        },
-      });
-    }
+    resolveConfigHeaders({
+      llmConfig: clientOptions,
+      user: createSafeUser(this.options.req?.user),
+      body: {
+        messageId: this.responseMessageId,
+        conversationId: this.conversationId,
+        parentMessageId: this.parentMessageId,
+      },
+    });
 
     try {
       const titleResult = await this.run.generateTitle({

api/server/controllers/agents/client.test.js

@@ -225,6 +225,51 @@ describe('AgentClient - titleConvo', () => {
       expect(generateTitleCall.clientOptions.model).toBe('gpt-3.5-turbo');
     });
 
+    it('preserves Anthropic custom headers on title requests despite omitTitleOptions', async () => {
+      const prevKey = process.env.ANTHROPIC_API_KEY;
+      process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
+      try {
+        const req = {
+          user: { id: 'user-123' },
+          body: { model: 'claude-sonnet-4-5', endpoint: EModelEndpoint.anthropic, key: null },
+          config: {
+            endpoints: {
+              [EModelEndpoint.anthropic]: {
+                headers: { 'X-Conversation-Id': '{{LIBRECHAT_BODY_CONVERSATIONID}}' },
+              },
+            },
+          },
+        };
+        const agent = {
+          id: 'agent-anthropic',
+          endpoint: EModelEndpoint.anthropic,
+          provider: EModelEndpoint.anthropic,
+          model_parameters: { model: 'claude-sonnet-4-5' },
+        };
+        const anthropicClient = new AgentClient({ req, res: {}, agent, endpointTokenConfig: {} });
+        anthropicClient.run = mockRun;
+        anthropicClient.responseMessageId = 'response-123';
+        anthropicClient.conversationId = 'convo-123';
+        anthropicClient.contentParts = [{ type: 'text', text: 'Test content' }];
+        anthropicClient.recordCollectedUsage = jest.fn().mockResolvedValue();
+
+        await anthropicClient.titleConvo({ text: 'Hello', abortController: new AbortController() });
+
+        const defaultHeaders =
+          mockRun.generateTitle.mock.calls[0][0].clientOptions?.clientOptions?.defaultHeaders;
+        // Custom header survives the `omitTitleOptions` strip and resolves the conversationId
+        expect(defaultHeaders?.['X-Conversation-Id']).toBe('convo-123');
+        // Provider-managed beta header is preserved alongside it
+        expect(defaultHeaders?.['anthropic-beta']).toBeDefined();
+      } finally {
+        if (prevKey === undefined) {
+          delete process.env.ANTHROPIC_API_KEY;
+        } else {
+          process.env.ANTHROPIC_API_KEY = prevKey;
+        }
+      }
+    });
+
     it('should handle missing endpoint config gracefully', async () => {
       // Remove endpoint config
       mockReq.config = { endpoints: {} };