mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-10-21 09:32:24 +00:00
🗺️ fix: Resolve Custom-Endpoint Providers for Summarization (#12739)
* 🔧 fix: resolve custom-endpoint providers for summarization When `summarization.provider` in `librechat.yaml` is set to a custom-endpoint name (e.g. `Ollama`), the string was passed verbatim to the agents SDK, which only knows a fixed set of provider names and threw `Unsupported LLM provider: Ollama`. Before shaping the summarization config for the SDK, resolve the provider through `getProviderConfig`: custom-endpoint labels are remapped to the underlying SDK provider (e.g. `openAI`) and the endpoint's baseURL/apiKey are injected into `parameters` so the summarization model reaches the right backend, even when summarization targets a different custom endpoint than the main agent. Unknown names and names that appear with no matching endpoint flow through unchanged so the SDK can surface a clear error. User-provided credentials and unresolved env-var references are skipped rather than forwarded, letting the SDK's self-summarize path reuse the agent's own clientOptions. Ref: LibreChat Discussion #12614 * address: widen unresolved-env-var guard, fix test naming - Reject summarization overrides when the extracted baseURL/apiKey still contains any `${...}` placeholder, including prefix/suffix patterns like `https://${UNSET}.example.com` that `envVarRegex` (exact-match) missed. - Rename the "case-insensitive" test to reflect that only `Ollama` is normalized via `normalizeEndpointName`; add coverage proving other custom-endpoint names match case-sensitively. * address: use req.config in responses.js; forward full endpoint options - `responses.js` relied on a module-level `appConfig` set via `setAppConfig`, which is never called anywhere. Use `req.config` directly so the summarization provider resolver actually runs on the responses route. - Route the custom endpoint config through `getOpenAIConfig` so summarization inherits the same `headers`, `defaultQuery`, `addParams`/`dropParams`, and `customParams` transforms (Anthropic/Google/etc.) that `initializeCustom` applies for the main agent flow. Strip the stale `model`/`modelName` defaults so `summarization.model` still wins. * address: skip overrides when summarization matches agent endpoint When `summarization.provider` resolves to the same custom endpoint as the main agent, rely on the SDK's self-summarize path (which reuses `agentContext.clientOptions` unchanged) rather than injecting overrides. Otherwise the shallow spread of `clientOverrides.configuration` would replace the agent's request-resolved state (dynamic headers, proxy/fetch options) with yaml-only config. Only applies when summarization targets a *different* endpoint from the agent; the yaml config is all we have in that case, so overrides still flow through. * address: preserve raw provider when overrides cannot be built When summarization points at a different custom endpoint than the agent and we can't resolve the endpoint's credentials (user_provided, or a still-unresolved `${VAR}` after env extraction), remapping to `openAI` without overrides would silently route summaries to the default OpenAI client. Preserve the raw provider name so the SDK raises a clear "Unsupported LLM provider" error (now also logged, via the agents SDK defense-in-depth fix) instead of sending traffic to the wrong backend. * address: resolve endpoint headers and forward PROXY to summarization - Custom-endpoint `headers` now flow through `resolveHeaders` before reaching `getOpenAIConfig`, matching the main agent path. This ensures templated values like `\${PORTKEY_API_KEY}` or `{{LIBRECHAT_BODY_...}}` are substituted for summarization requests instead of being forwarded literally. - `PROXY` env var is now passed into `getOpenAIConfig` so cross-endpoint summarization honors outbound proxy dispatchers configured for the rest of the deployment. * address: user summarization parameters win over endpoint defaults Flip the merge order so `summarization.parameters` from yaml override `clientOverrides` defaults (which come from `getOpenAIConfig` and always include `streaming: true` etc.). A user who sets `parameters.streaming: false` in their config should still see non-streaming summarization for providers that require it. * address: review feedback (logging, dead code, DRY, types, deep-merge) - Log error in the resolveSummarizationProvider catch-all so programming bugs in getProviderConfig/getOpenAIConfig/resolveHeaders surface in operator logs instead of falling through silently. - Drop dead `setAppConfig`/`appConfig` infrastructure in responses.js and fix adjacent `allowedProviders` reference that also relied on the never-initialized module-level appConfig. Uses `req.config` directly. - Import canonical `normalizeEndpointName` from librechat-data-provider instead of duplicating it locally. - Replace `SummarizationClientOverrides = Record<string, unknown>` with an explicit interface covering the known fields. - Deep-merge `configuration` when user-supplied `summarization.parameters. configuration` overlaps the resolved endpoint configuration, so user additions (e.g. `defaultQuery`) don't wipe out `baseURL`/`defaultHeaders`. - Wrap `process.env` mutations in test in `try/finally` so a failed assertion doesn't leak env state into subsequent tests. - Drop `as unknown as AppConfig` in test helper; fixture now matches the `AppConfig` shape directly using a `Partial<TEndpoint>` union. - Trim JSDoc that restated the name it was attached to. * address: review nits — import order, local test type, conflict test - Move `import { logger }` up into the package value-imports section so it no longer sits between `import type` blocks. - Replace `as unknown as SummarizationConfig['parameters']` in the deep-merge test with a named `TestSummarizationParameters` type and a single narrowing cast at the call site, making intent explicit. - Add a test proving that user-supplied `configuration.baseURL` wins over the resolved endpoint baseURL, locking in the deep-merge's user-wins-on-conflict semantics that the previous suite only exercised additively.
This commit is contained in:
Danny Avila
GitHub
parent
d2cbd551b7
commit
8e866e6010
5 changed files with 708 additions and 18 deletions
|
|
@ -836,6 +836,7 @@ class AgentClient extends BaseClient {
|
|||
requestBody: config.configurable.requestBody,
|
||||
user: createSafeUser(this.options.req?.user),
|
||||
summarizationConfig: appConfig?.summarization,
|
||||
appConfig,
|
||||
tokenCounter,
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -584,6 +584,7 @@ const OpenAIChatCompletionController = async (req, res) => {
|
|||
initialSummary,
|
||||
runId: responseId,
|
||||
summarizationConfig,
|
||||
appConfig,
|
||||
signal: abortController.signal,
|
||||
customHandlers: handlers,
|
||||
requestBody: {
|
||||
|
|
|
|||
|
|
@ -53,17 +53,6 @@ const { getModelsConfig } = require('~/server/controllers/ModelController');
|
|||
const { logViolation } = require('~/cache');
|
||||
const db = require('~/models');
|
||||
|
||||
/** @type {import('@librechat/api').AppConfig | null} */
|
||||
let appConfig = null;
|
||||
|
||||
/**
|
||||
* Set the app config for the controller
|
||||
* @param {import('@librechat/api').AppConfig} config
|
||||
*/
|
||||
function setAppConfig(config) {
|
||||
appConfig = config;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a tool loader function for the agent.
|
||||
* @param {AbortSignal} signal - The abort signal
|
||||
|
|
@ -345,7 +334,7 @@ const createResponse = async (req, res) => {
|
|||
|
||||
// Build allowed providers set
|
||||
const allowedProviders = new Set(
|
||||
appConfig?.endpoints?.[EModelEndpoint.agents]?.allowedProviders,
|
||||
req.config?.endpoints?.[EModelEndpoint.agents]?.allowedProviders,
|
||||
);
|
||||
|
||||
// Create tool loader
|
||||
|
|
@ -599,6 +588,7 @@ const createResponse = async (req, res) => {
|
|||
initialSummary,
|
||||
runId: responseId,
|
||||
summarizationConfig,
|
||||
appConfig: req.config,
|
||||
signal: abortController.signal,
|
||||
customHandlers: handlers,
|
||||
requestBody: {
|
||||
|
|
@ -764,6 +754,7 @@ const createResponse = async (req, res) => {
|
|||
initialSummary,
|
||||
runId: responseId,
|
||||
summarizationConfig,
|
||||
appConfig: req.config,
|
||||
signal: abortController.signal,
|
||||
customHandlers: handlers,
|
||||
requestBody: {
|
||||
|
|
@ -1048,5 +1039,4 @@ module.exports = {
|
|||
createResponse,
|
||||
getResponse,
|
||||
listModels,
|
||||
setAppConfig,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,4 +1,6 @@
|
|||
import type { SummarizationConfig } from 'librechat-data-provider';
|
||||
import type { AppConfig } from '@librechat/data-schemas';
|
||||
import type { SummarizationConfig, TEndpoint } from 'librechat-data-provider';
|
||||
import { EModelEndpoint, FileSources } from 'librechat-data-provider';
|
||||
import { createRun } from '~/agents/run';
|
||||
|
||||
// Mock winston logger
|
||||
|
|
@ -57,6 +59,7 @@ async function callAndCapture(
|
|||
agents?: ReturnType<typeof makeAgent>[];
|
||||
summarizationConfig?: SummarizationConfig;
|
||||
initialSummary?: { text: string; tokenCount: number };
|
||||
appConfig?: AppConfig;
|
||||
} = {},
|
||||
) {
|
||||
const agents = opts.agents ?? [makeAgent()];
|
||||
|
|
@ -67,6 +70,7 @@ async function callAndCapture(
|
|||
signal,
|
||||
summarizationConfig: opts.summarizationConfig,
|
||||
initialSummary: opts.initialSummary,
|
||||
appConfig: opts.appConfig,
|
||||
streaming: true,
|
||||
streamUsage: true,
|
||||
});
|
||||
|
|
@ -77,6 +81,40 @@ async function callAndCapture(
|
|||
return callArgs.graphConfig.agents as Array<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
/** Minimal AppConfig with a single custom endpoint for testing provider resolution. */
|
||||
type TestCustomEndpoint = Partial<TEndpoint> & {
|
||||
name: string;
|
||||
baseURL: string;
|
||||
apiKey: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Shape of summarization parameters used in tests. The LibreChat config
|
||||
* schema restricts yaml `parameters` to primitive values, but the SDK
|
||||
* passes any record through as-is — tests need the wider shape to exercise
|
||||
* cross-endpoint `configuration` merging.
|
||||
*/
|
||||
type TestSummarizationParameters = Record<string, unknown> & {
|
||||
configuration?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
/**
|
||||
* Minimal AppConfig fixture for testing. Only `endpoints` is read by
|
||||
* `resolveSummarizationProvider`; other required AppConfig fields are
|
||||
* filled with empty/default values so the shape matches without needing
|
||||
* `as unknown as AppConfig`.
|
||||
*/
|
||||
function makeAppConfig(customEndpoints: TestCustomEndpoint[]): AppConfig {
|
||||
return {
|
||||
config: {},
|
||||
fileStrategy: FileSources.local,
|
||||
imageOutputType: 'png',
|
||||
endpoints: {
|
||||
[EModelEndpoint.custom]: customEndpoints,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
|
@ -322,3 +360,448 @@ describe('initialSummary passthrough', () => {
|
|||
expect(agents[0].initialSummary).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Suite 7: custom-endpoint provider resolution
|
||||
// ---------------------------------------------------------------------------
|
||||
describe('custom-endpoint provider resolution', () => {
|
||||
it('remaps a custom endpoint name to openAI and injects baseURL/apiKey', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('openAI');
|
||||
expect(config.model).toBe('llama3');
|
||||
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
expect(parameters).toMatchObject({
|
||||
configuration: { baseURL: 'http://localhost:11434/v1' },
|
||||
apiKey: 'ollama-key',
|
||||
});
|
||||
});
|
||||
|
||||
it('matches Ollama case-insensitively (via normalizeEndpointName)', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('openAI');
|
||||
expect((config.parameters as Record<string, unknown>).apiKey).toBe('ollama-key');
|
||||
});
|
||||
|
||||
it('resolves non-Ollama endpoints on exact-case match', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Together', baseURL: 'https://api.together.ai/v1', apiKey: 'together-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Together', model: 'mixtral' },
|
||||
appConfig,
|
||||
});
|
||||
expect((agents[0].summarizationConfig as Record<string, unknown>).provider).toBe('openAI');
|
||||
});
|
||||
|
||||
it('does not match non-Ollama endpoints with different casing', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Together', baseURL: 'https://api.together.ai/v1', apiKey: 'together-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'together', model: 'mixtral' },
|
||||
appConfig,
|
||||
});
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('together');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('leaves known SDK providers untouched', async () => {
|
||||
const appConfig = makeAppConfig([]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'anthropic', model: 'claude' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('anthropic');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('preserves unknown provider names when appConfig is missing', async () => {
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('Ollama');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('leaves unrecognized names untouched when no matching custom endpoint exists', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'nonexistent', model: 'foo' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('nonexistent');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('extracts ${ENV_VAR} references in custom endpoint credentials', async () => {
|
||||
process.env.TEST_OLLAMA_KEY = 'resolved-key-value';
|
||||
try {
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'http://localhost:11434/v1',
|
||||
apiKey: '${TEST_OLLAMA_KEY}',
|
||||
},
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
expect(parameters.apiKey).toBe('resolved-key-value');
|
||||
} finally {
|
||||
delete process.env.TEST_OLLAMA_KEY;
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps raw provider when apiKey is marked user_provided', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'user_provided' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
/**
|
||||
* Keep the raw name so the SDK raises "Unsupported LLM provider: Ollama"
|
||||
* rather than silently remapping to `openAI` and routing summaries to the
|
||||
* default backend. (User-provided creds cannot be resolved here — the
|
||||
* async DB lookup is out of scope for this synchronous code path.)
|
||||
*/
|
||||
expect(config.provider).toBe('Ollama');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('keeps raw provider when env var reference cannot be resolved', async () => {
|
||||
delete process.env.UNSET_TEST_KEY;
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'http://localhost:11434/v1',
|
||||
apiKey: '${UNSET_TEST_KEY}',
|
||||
},
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('Ollama');
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('keeps raw provider when partial env var reference (prefix/suffix) stays unresolved', async () => {
|
||||
delete process.env.UNSET_TEST_SEGMENT;
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'https://${UNSET_TEST_SEGMENT}.example.com/v1',
|
||||
apiKey: 'ollama-key',
|
||||
},
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('Ollama');
|
||||
/** Even though the baseURL is a partial-match pattern, it must not be forwarded. */
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('merges overrides alongside user-supplied parameters', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: {
|
||||
provider: 'Ollama',
|
||||
model: 'llama3',
|
||||
parameters: { temperature: 0.2 },
|
||||
},
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
expect(parameters).toMatchObject({
|
||||
temperature: 0.2,
|
||||
apiKey: 'ollama-key',
|
||||
});
|
||||
const configuration = parameters.configuration as Record<string, unknown>;
|
||||
expect(configuration.baseURL).toBe('http://localhost:11434/v1');
|
||||
});
|
||||
|
||||
it('forwards custom-endpoint headers as configuration.defaultHeaders', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'http://localhost:11434/v1',
|
||||
apiKey: 'ollama-key',
|
||||
headers: { 'X-Custom-Header': 'value-123' },
|
||||
},
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
const configuration = parameters.configuration as Record<string, unknown>;
|
||||
const defaultHeaders = configuration.defaultHeaders as Record<string, string>;
|
||||
expect(defaultHeaders['X-Custom-Header']).toBe('value-123');
|
||||
});
|
||||
|
||||
it('runs custom-endpoint headers through resolveHeaders (not forwarded raw)', async () => {
|
||||
const { resolveHeaders } = jest.requireMock('~/utils/env') as {
|
||||
resolveHeaders: jest.Mock;
|
||||
};
|
||||
resolveHeaders.mockClear();
|
||||
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'http://localhost:11434/v1',
|
||||
apiKey: 'ollama-key',
|
||||
headers: { Authorization: 'Bearer ${TEST_PORTKEY_KEY}' },
|
||||
},
|
||||
]);
|
||||
await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
/**
|
||||
* Templated header values must go through the same `resolveHeaders`
|
||||
* pipeline the main agent flow uses, so `${VAR}`/`{{BODY_FIELD}}`
|
||||
* references don't get forwarded verbatim to the summarization backend.
|
||||
*/
|
||||
const call = resolveHeaders.mock.calls.find(
|
||||
(args: unknown[]) =>
|
||||
(args[0] as { headers?: Record<string, string> }).headers?.Authorization ===
|
||||
'Bearer ${TEST_PORTKEY_KEY}',
|
||||
);
|
||||
expect(call).toBeDefined();
|
||||
});
|
||||
|
||||
it('forwards PROXY env var into summarization client configuration', async () => {
|
||||
const originalProxy = process.env.PROXY;
|
||||
process.env.PROXY = 'http://proxy.internal:3128';
|
||||
try {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
const configuration = parameters.configuration as Record<string, unknown>;
|
||||
/** getOpenAIConfig wires proxy through to fetchOptions.dispatcher (undici ProxyAgent). */
|
||||
expect(configuration.fetchOptions).toBeDefined();
|
||||
} finally {
|
||||
if (originalProxy === undefined) {
|
||||
delete process.env.PROXY;
|
||||
} else {
|
||||
process.env.PROXY = originalProxy;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('skips overrides when summarization targets the same endpoint as the agent', async () => {
|
||||
/**
|
||||
* When summarization provider matches the agent's endpoint, we rely on
|
||||
* the SDK's self-summarize path (which reuses agentContext.clientOptions).
|
||||
* Overriding here would shallow-replace the agent's resolved configuration
|
||||
* (dynamic headers, proxy/fetch options) with yaml-only config.
|
||||
*/
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
agents: [makeAgent({ provider: 'openAI', endpoint: 'Ollama' })],
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.provider).toBe('openAI');
|
||||
/** No overrides injected — SDK will pull from agentContext.clientOptions. */
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('skips overrides when endpoints differ only by case for Ollama', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
agents: [makeAgent({ provider: 'openAI', endpoint: 'Ollama' })],
|
||||
summarizationConfig: { provider: 'ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.parameters).toBeUndefined();
|
||||
});
|
||||
|
||||
it('applies overrides when summarization targets a different endpoint than the agent', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
{ name: 'Together', baseURL: 'https://api.together.ai/v1', apiKey: 'together-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
agents: [makeAgent({ provider: 'openAI', endpoint: 'Ollama' })],
|
||||
summarizationConfig: { provider: 'Together', model: 'mixtral' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
expect(parameters.apiKey).toBe('together-key');
|
||||
expect((parameters.configuration as Record<string, unknown>).baseURL).toBe(
|
||||
'https://api.together.ai/v1',
|
||||
);
|
||||
});
|
||||
|
||||
it('deep-merges user configuration with endpoint-resolved configuration', async () => {
|
||||
/**
|
||||
* User-supplied `parameters.configuration.defaultQuery` must merge with —
|
||||
* not replace — the resolved `configuration` (baseURL, defaultHeaders).
|
||||
*/
|
||||
const appConfig = makeAppConfig([
|
||||
{
|
||||
name: 'Ollama',
|
||||
baseURL: 'http://localhost:11434/v1',
|
||||
apiKey: 'ollama-key',
|
||||
headers: { 'X-Required-Header': 'keep-me' },
|
||||
},
|
||||
]);
|
||||
const parameters: TestSummarizationParameters = {
|
||||
configuration: { defaultQuery: { 'api-version': '2024-06-01' } },
|
||||
};
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: {
|
||||
provider: 'Ollama',
|
||||
model: 'llama3',
|
||||
parameters: parameters as SummarizationConfig['parameters'],
|
||||
},
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const resolvedParameters = config.parameters as Record<string, unknown>;
|
||||
const configuration = resolvedParameters.configuration as Record<string, unknown>;
|
||||
/** Endpoint defaults preserved... */
|
||||
expect(configuration.baseURL).toBe('http://localhost:11434/v1');
|
||||
expect((configuration.defaultHeaders as Record<string, string>)['X-Required-Header']).toBe(
|
||||
'keep-me',
|
||||
);
|
||||
/** ...alongside the user's additions. */
|
||||
expect(configuration.defaultQuery).toEqual({ 'api-version': '2024-06-01' });
|
||||
});
|
||||
|
||||
it('user-supplied configuration.baseURL overrides resolved baseURL', async () => {
|
||||
/**
|
||||
* Deep-merge still lets user keys win on conflict — if a user explicitly
|
||||
* sets `configuration.baseURL` in their summarization parameters, it
|
||||
* must override the baseURL resolved from the endpoint config.
|
||||
*/
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const parameters: TestSummarizationParameters = {
|
||||
configuration: { baseURL: 'https://user-override.example.com/v1' },
|
||||
};
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: {
|
||||
provider: 'Ollama',
|
||||
model: 'llama3',
|
||||
parameters: parameters as SummarizationConfig['parameters'],
|
||||
},
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const resolvedParameters = config.parameters as Record<string, unknown>;
|
||||
const configuration = resolvedParameters.configuration as Record<string, unknown>;
|
||||
expect(configuration.baseURL).toBe('https://user-override.example.com/v1');
|
||||
});
|
||||
|
||||
it('user-supplied summarization.parameters override endpoint defaults', async () => {
|
||||
/**
|
||||
* `getOpenAIConfig` defaults `streaming: true`, but a user who sets
|
||||
* `summarization.parameters.streaming: false` in their config has
|
||||
* explicitly opted out; the user's setting must win over endpoint
|
||||
* defaults injected from the custom endpoint config.
|
||||
*/
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: {
|
||||
provider: 'Ollama',
|
||||
model: 'llama3',
|
||||
parameters: { streaming: false },
|
||||
},
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
expect(parameters.streaming).toBe(false);
|
||||
/** Endpoint defaults still injected for the rest. */
|
||||
expect(parameters.apiKey).toBe('ollama-key');
|
||||
});
|
||||
|
||||
it('does not leak model/modelName from getOpenAIConfig defaults', async () => {
|
||||
const appConfig = makeAppConfig([
|
||||
{ name: 'Ollama', baseURL: 'http://localhost:11434/v1', apiKey: 'ollama-key' },
|
||||
]);
|
||||
const agents = await callAndCapture({
|
||||
summarizationConfig: { provider: 'Ollama', model: 'llama3' },
|
||||
appConfig,
|
||||
});
|
||||
|
||||
const config = agents[0].summarizationConfig as Record<string, unknown>;
|
||||
expect(config.model).toBe('llama3');
|
||||
const parameters = config.parameters as Record<string, unknown>;
|
||||
/** Summarization.model must win — parameters must not carry a stale model/modelName. */
|
||||
expect(parameters.model).toBeUndefined();
|
||||
expect(parameters.modelName).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,5 +1,11 @@
|
|||
import { logger } from '@librechat/data-schemas';
|
||||
import { Run, Providers, Constants } from '@librechat/agents';
|
||||
import { providerEndpointMap, KnownEndpoints } from 'librechat-data-provider';
|
||||
import {
|
||||
KnownEndpoints,
|
||||
extractEnvVariable,
|
||||
providerEndpointMap,
|
||||
normalizeEndpointName,
|
||||
} from 'librechat-data-provider';
|
||||
import type {
|
||||
SummarizationConfig as AgentSummarizationConfig,
|
||||
MultiAgentGraphConfig,
|
||||
|
|
@ -15,9 +21,12 @@ import type {
|
|||
} from '@librechat/agents';
|
||||
import type { Agent, SummarizationConfig } from 'librechat-data-provider';
|
||||
import type { BaseMessage } from '@langchain/core/messages';
|
||||
import type { IUser } from '@librechat/data-schemas';
|
||||
import type { AppConfig, IUser } from '@librechat/data-schemas';
|
||||
import type * as t from '~/types';
|
||||
import { getProviderConfig } from '~/endpoints/config/providers';
|
||||
import { getOpenAIConfig } from '~/endpoints/openai/config';
|
||||
import { resolveHeaders, createSafeUser } from '~/utils/env';
|
||||
import { isUserProvided } from '~/utils/common';
|
||||
|
||||
/** Expected shape of JSON tool search results */
|
||||
interface ToolSearchJsonResult {
|
||||
|
|
@ -186,26 +195,223 @@ function isNonEmptyString(value: unknown): value is string {
|
|||
return typeof value === 'string' && value.trim().length > 0;
|
||||
}
|
||||
|
||||
const UNRESOLVED_ENV_VAR_PLACEHOLDER = /\$\{[^}]+\}/;
|
||||
|
||||
function hasUnresolvedPlaceholder(value: string): boolean {
|
||||
return UNRESOLVED_ENV_VAR_PLACEHOLDER.test(value);
|
||||
}
|
||||
|
||||
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
||||
return value != null && typeof value === 'object' && !Array.isArray(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Merges user-supplied summarization parameters on top of endpoint-resolved
|
||||
* overrides. User params win for top-level keys; `configuration` is
|
||||
* deep-merged so user additions (e.g. `defaultQuery`) don't wipe out the
|
||||
* resolved `baseURL`/`defaultHeaders`/`fetchOptions`.
|
||||
*/
|
||||
function mergeParameters(
|
||||
overrides: SummarizationClientOverrides,
|
||||
userParams: SummarizationConfig['parameters'],
|
||||
): Record<string, unknown> {
|
||||
const merged: Record<string, unknown> = { ...overrides, ...(userParams ?? {}) };
|
||||
const userConfiguration = (userParams as Record<string, unknown> | undefined)?.configuration;
|
||||
if (isPlainObject(overrides.configuration) && isPlainObject(userConfiguration)) {
|
||||
merged.configuration = { ...overrides.configuration, ...userConfiguration };
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirrors `getOpenAIConfig`'s `llmConfig` shape (plus its `configOptions`
|
||||
* assigned to `configuration`). Index signature covers fields that the
|
||||
* helper emits dynamically per provider variant.
|
||||
*/
|
||||
interface SummarizationClientOverrides {
|
||||
apiKey?: string;
|
||||
streaming?: boolean;
|
||||
configuration?: t.OpenAIConfiguration;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves a summarization provider string (which may be a custom-endpoint name
|
||||
* like "Ollama") into the SDK-recognized provider and any client-option
|
||||
* overrides required to talk to that endpoint.
|
||||
*
|
||||
* Without this step, a `summarization.provider: "Ollama"` entry in
|
||||
* `librechat.yaml` flows verbatim to the agents SDK, which only knows a fixed
|
||||
* set of provider names and throws "Unsupported LLM provider: Ollama".
|
||||
*/
|
||||
function resolveSummarizationProvider(
|
||||
rawProvider: string,
|
||||
appConfig: AppConfig | undefined,
|
||||
headerContext: { user?: IUser; requestBody?: t.RequestBody },
|
||||
): {
|
||||
provider: string;
|
||||
clientOverrides?: SummarizationClientOverrides;
|
||||
} {
|
||||
if (!appConfig || !isNonEmptyString(rawProvider)) {
|
||||
return { provider: rawProvider };
|
||||
}
|
||||
try {
|
||||
const { overrideProvider, customEndpointConfig } = getProviderConfig({
|
||||
provider: rawProvider,
|
||||
appConfig,
|
||||
});
|
||||
if (!customEndpointConfig) {
|
||||
return { provider: overrideProvider };
|
||||
}
|
||||
const rawApiKey = customEndpointConfig.apiKey ?? '';
|
||||
const rawBaseURL = customEndpointConfig.baseURL ?? '';
|
||||
/**
|
||||
* User-provided credentials require an async DB lookup and expiry checks
|
||||
* that are out of scope here. Keep the raw provider so the SDK surfaces
|
||||
* a clear "Unsupported LLM provider" error rather than silently
|
||||
* remapping to `openAI` and routing summaries to the default backend.
|
||||
* Callers wanting user-provided summarization against a non-agent
|
||||
* endpoint must hit the same endpoint as the agent (handled upstream).
|
||||
*/
|
||||
if (isUserProvided(rawApiKey) || isUserProvided(rawBaseURL)) {
|
||||
return { provider: rawProvider };
|
||||
}
|
||||
const apiKey = extractEnvVariable(rawApiKey);
|
||||
const baseURL = extractEnvVariable(rawBaseURL);
|
||||
/**
|
||||
* `extractEnvVariable` leaves any unresolved `${VAR}` placeholder in place
|
||||
* — including in the middle of a prefix/suffix string — when the env var
|
||||
* is missing. If the value is still broken, keep the raw provider so the
|
||||
* SDK errors out loudly instead of forwarding a malformed URL/key.
|
||||
*/
|
||||
if (
|
||||
!apiKey ||
|
||||
!baseURL ||
|
||||
hasUnresolvedPlaceholder(apiKey) ||
|
||||
hasUnresolvedPlaceholder(baseURL)
|
||||
) {
|
||||
return { provider: rawProvider };
|
||||
}
|
||||
/**
|
||||
* Resolve templated header values (e.g. `${PORTKEY_API_KEY}`,
|
||||
* `{{LIBRECHAT_BODY_PARENTMESSAGEID}}`) before handing them to
|
||||
* `getOpenAIConfig`, matching the agent main flow where `resolveHeaders`
|
||||
* runs on `llmConfig.configuration.defaultHeaders`.
|
||||
*/
|
||||
const resolvedHeaders =
|
||||
customEndpointConfig.headers != null
|
||||
? resolveHeaders({
|
||||
headers: customEndpointConfig.headers as Record<string, string>,
|
||||
user: createSafeUser(headerContext.user),
|
||||
body: headerContext.requestBody,
|
||||
})
|
||||
: undefined;
|
||||
/**
|
||||
* Run the endpoint config through `getOpenAIConfig` so summarization
|
||||
* inherits the same `headers`, `defaultQuery`, `addParams`/`dropParams`,
|
||||
* and `customParams` transforms that `initializeCustom` applies for the
|
||||
* main agent flow. Without this, summarization drops endpoint-specific
|
||||
* behavior (e.g. Anthropic/Google param transforms, required headers)
|
||||
* that the main agent relied on. `proxy` is forwarded so outbound proxy
|
||||
* dispatchers (`PROXY` env var) apply to cross-endpoint summarization.
|
||||
*/
|
||||
const { llmConfig, configOptions } = getOpenAIConfig(
|
||||
apiKey,
|
||||
{
|
||||
reverseProxyUrl: baseURL,
|
||||
proxy: process.env.PROXY ?? null,
|
||||
headers: resolvedHeaders,
|
||||
addParams: customEndpointConfig.addParams,
|
||||
dropParams: customEndpointConfig.dropParams,
|
||||
customParams: customEndpointConfig.customParams,
|
||||
directEndpoint: customEndpointConfig.directEndpoint,
|
||||
},
|
||||
rawProvider,
|
||||
);
|
||||
const clientOverrides: SummarizationClientOverrides = {
|
||||
...llmConfig,
|
||||
};
|
||||
if (configOptions) {
|
||||
clientOverrides.configuration = configOptions;
|
||||
}
|
||||
/**
|
||||
* `model`/`modelName` on `llmConfig` default to whatever `getOpenAIConfig`
|
||||
* produces from empty modelOptions. Strip them so the user-supplied
|
||||
* `summarization.model` wins.
|
||||
*/
|
||||
delete clientOverrides.model;
|
||||
delete clientOverrides.modelName;
|
||||
return {
|
||||
provider: overrideProvider,
|
||||
clientOverrides,
|
||||
};
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
`[resolveSummarizationProvider] failed to resolve "${rawProvider}"; falling back to raw provider`,
|
||||
error,
|
||||
);
|
||||
return { provider: rawProvider };
|
||||
}
|
||||
}
|
||||
|
||||
/** Shapes a SummarizationConfig into the format expected by AgentInputs. */
|
||||
function shapeSummarizationConfig(
|
||||
config: SummarizationConfig | undefined,
|
||||
fallbackProvider: string,
|
||||
fallbackModel: string | undefined,
|
||||
appConfig: AppConfig | undefined,
|
||||
agentEndpoint: string | undefined,
|
||||
headerContext: { user?: IUser; requestBody?: t.RequestBody },
|
||||
) {
|
||||
const provider = config?.provider ?? fallbackProvider;
|
||||
const rawProvider = config?.provider ?? fallbackProvider;
|
||||
/**
|
||||
* When the summarization provider resolves to the same custom endpoint as
|
||||
* the main agent, skip client-option overrides. The SDK's self-summarize
|
||||
* path will reuse `agentContext.clientOptions` as-is, preserving any
|
||||
* request-resolved dynamic headers, fetch/proxy options, and other state
|
||||
* that `getOpenAIConfig` produced from raw yaml config does not capture.
|
||||
*/
|
||||
const isSameEndpointAsAgent =
|
||||
agentEndpoint != null &&
|
||||
isNonEmptyString(rawProvider) &&
|
||||
normalizeEndpointName(rawProvider) === normalizeEndpointName(agentEndpoint);
|
||||
|
||||
const { provider, clientOverrides } = isSameEndpointAsAgent
|
||||
? { provider: fallbackProvider, clientOverrides: undefined }
|
||||
: resolveSummarizationProvider(rawProvider, appConfig, headerContext);
|
||||
|
||||
const model = config?.model ?? fallbackModel;
|
||||
const trigger =
|
||||
config?.trigger?.type && typeof config?.trigger?.value === 'number'
|
||||
? { type: config.trigger.type, value: config.trigger.value }
|
||||
: undefined;
|
||||
|
||||
/**
|
||||
* Custom-endpoint overrides are merged into `parameters` so the SDK's
|
||||
* `buildSummarizationClientConfig` spreads them onto the summarization
|
||||
* client options. Only applied when summarization targets a *different*
|
||||
* custom endpoint than the main agent; the same-endpoint case leaves
|
||||
* `parameters` untouched so `agentContext.clientOptions` wins.
|
||||
*
|
||||
* Order matters: `clientOverrides` supplies endpoint defaults (baseURL,
|
||||
* apiKey, headers, transforms), then explicit user `summarization.parameters`
|
||||
* are spread on top so settings like `streaming: false` still win over
|
||||
* `getOpenAIConfig`'s defaults. `configuration` is deep-merged so a user
|
||||
* adding e.g. `configuration.defaultQuery` keeps the resolved `baseURL`
|
||||
* and `defaultHeaders` rather than replacing the whole object.
|
||||
*/
|
||||
const parameters =
|
||||
clientOverrides != null
|
||||
? mergeParameters(clientOverrides, config?.parameters)
|
||||
: config?.parameters;
|
||||
|
||||
return {
|
||||
enabled: config?.enabled !== false && isNonEmptyString(provider) && isNonEmptyString(model),
|
||||
config: {
|
||||
trigger,
|
||||
provider,
|
||||
model,
|
||||
parameters: config?.parameters,
|
||||
parameters,
|
||||
prompt: config?.prompt,
|
||||
updatePrompt: config?.updatePrompt,
|
||||
reserveRatio: config?.reserveRatio,
|
||||
|
|
@ -260,6 +466,7 @@ export async function createRun({
|
|||
summarizationConfig,
|
||||
initialSummary,
|
||||
calibrationRatio,
|
||||
appConfig,
|
||||
streaming = true,
|
||||
streamUsage = true,
|
||||
}: {
|
||||
|
|
@ -277,6 +484,11 @@ export async function createRun({
|
|||
initialSummary?: { text: string; tokenCount: number };
|
||||
/** Calibration ratio from previous run's contextMeta, seeds the pruner EMA */
|
||||
calibrationRatio?: number;
|
||||
/**
|
||||
* Resolved app config. Used to translate custom-endpoint provider names
|
||||
* (e.g. "Ollama") in the summarization config to SDK-recognized providers.
|
||||
*/
|
||||
appConfig?: AppConfig;
|
||||
} & Pick<RunConfig, 'tokenCounter' | 'customHandlers' | 'indexTokenCountMap'>): Promise<
|
||||
Run<IState>
|
||||
> {
|
||||
|
|
@ -307,6 +519,9 @@ export async function createRun({
|
|||
agent.summarization ?? summarizationConfig,
|
||||
provider as string,
|
||||
selfModel,
|
||||
appConfig,
|
||||
agent.endpoint ?? undefined,
|
||||
{ user, requestBody },
|
||||
);
|
||||
|
||||
const llmConfig: t.RunLLMConfig = Object.assign(
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue