mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 12:13:30 +00:00
📡 feat: Route Web Search and Scrape Egress through the SSRF-safe Agent (#14606)
* feat(web-search): route outbound search and scrape requests through the SSRF-safe agent Build the SSRF-safe agents at the web-search tool-assembly site and pass them into the search tool config so outbound search and scrape connections are validated at connect time against their resolved IP, on every hop including redirects, consistent with the other outbound clients. Add allowedAddresses to webSearchSchema, reusing allowedAddressesSchema, so self-hosters can permit a deliberately-private search or scrape endpoint (for example a private SearXNG instance). The field is resolved directly from the webSearch config at the createSearchTool call site, not through loadWebSearchAuth, because it is config and not an auth credential. webSearchSchema is flat (providers are chosen by enums, not by counting keys), so the field is inert with respect to provider selection. Document the field and its operator warning in librechat.example.yaml, and assert the wiring in handleTools.test.js: the SSRF-safe agents are threaded into the search tool config, allowedAddresses is passed through when set, and omitting it still threads the agents with no exemptions. TODO awaits @librechat/agents release with the httpAgent hook: this consumes optional httpAgent/httpsAgent fields on the search-tool config that are not yet in a published @librechat/agents. package.json is intentionally left at the current version; bump it to the release that ships the hook before this lands. Validated locally against a revendored @librechat/agents build, not a published release. * 🛡️ fix: Apply allowedAddresses to the Web Search SSRF Preflight The connect-time SSRF agent already honors webSearch.allowedAddresses, but loadWebSearchAuth ran the isSSRFUrl preflight without it, so an admin-permitted private search or scrape URL was stripped before the agent could ever use it. Thread allowedAddresses and the URL's effective port through isSSRFTarget and resolveHostnameSSRF so the exemption is consistent across both SSRF layers. * 🛡️ fix: Validate Web Search Destinations and Defer to Configured Proxies Handing agents to createSearchTool covered only the connect-time DNS lookup, which Node skips for IP-literal hosts, and a configured proxy connects on our behalf without running that check. A literal private target such as http://169.254.169.254 could therefore reach the network. Route every resolved web-search destination through the existing applySSRFSafeAgentIfDirect contract so a blocked literal target throws before any request is made, and withhold the agents when a proxy owns egress, since one agent pair is shared by every provider and a direct-connect agent on a proxied connection would break the request while asserting protection the proxy's network context cannot provide. * 🛡️ fix: Keep Web Search SSRF Agents Under a Proxy and Restore Pooling Withholding the agents whenever a proxy was configured removed protection from every direct and NO_PROXY destination in exchange for preventing a failure that cannot occur: for an https target Axios substitutes its own CONNECT tunnel, so the injected agent is never used for the proxy connection. Only a plaintext http target keeps our agent and repoints it at the proxy, and only a proxy whose hostname resolves private then trips the connect-time check. Always pass the agents and exempt the proxy endpoint instead, deriving host:port from the same PROXY, HTTP_PROXY, and HTTPS_PROXY resolution the rest of LibreChat uses so the proxy hop stays reachable while destinations remain guarded. Axios already applies NO_PROXY per request, so bypassed routes keep enforcement with no extra logic. Drop the load-time destination validation. It duplicated the isSSRFTarget preflight for user-provided URLs, rejected admin values that were previously legal, and threw from inside loadTools, where both loader wrappers swallow the error and drop every tool for the turn rather than degrading web search alone. Build the agents with keepAlive and cache them per exemption list. A bare http.Agent does not pool, so the previous code replaced the pooled global agents for every search, scrape, and rerank call and allocated a fresh pair per turn. * 🛡️ fix: Reject IP-Literal Private Targets on Web Search Connections Node resolves nothing for a literal host, so the connect-time lookup never saw one: a destination or a redirect target given as http://169.254.169.254 reached the network. Redirect hops pass through the same createConnection, so checking the literal there covers both cases and removes the need for a maxRedirects control that createSearchTool cannot accept. Gate it behind blockLiteralHosts so only web search opts in. A caller that reaches a proxy or a deliberate private service by literal address must exempt it first, and the merged consumers of createSSRFSafeAgents have no such exemption, so enabling this everywhere would break configurations that work today. * 🛡️ fix: Keep IPv6 Brackets on Derived Proxy Exemptions The exemption parser accepts an IPv6 entry only as [ipv6]:port, so stripping the brackets produced fd00::1:3128, which carries three colons and is dropped as malformed. An IPv6 proxy therefore stayed unexempted and the connect-time check rejected it, failing every web-search request routed through it. Use the URL hostname as parsed, which already carries the brackets. * 🛡️ fix: Exempt Proxies Configured Through ALL_PROXY Axios resolves a proxy through proxy-from-env, which falls back to all_proxy in either case after <protocol>_proxy, so ALL_PROXY on its own is enough to route a request through a proxy. Exemptions were derived from PROXY, HTTP_PROXY, and HTTPS_PROXY only, leaving such a proxy unexempted and rejected with ESSRF. Derive the exemptions from the full set of variables that can put a proxy in front of these requests instead. The installed proxy-from-env 2.1.0 reads no npm_config variables, so those are deliberately not included. * 🛡️ fix: Drop the Unearned PROXY Exemption and Harden the Web Search Guard Nothing on this path consumes PROXY: Axios resolves proxies through proxy-from-env, which reads only <protocol>_proxy and all_proxy, and web search never calls applyAxiosProxyConfig. Exempting it therefore granted a bypass rather than preserving a working route, and a user-settable search URL that redirects to that address reached it and returned the body. Remove PROXY and proxy, and skip a socks endpoint for the same reason, since Axios cannot proxy through one. Tolerate a non-array allowedAddresses instead of spreading it, which threw out of loadTools and dropped every tool for the turn. The YAML path is schema-validated but the admin override path merges without parsing, so the value is reachable. Separate cache keys with NUL rather than a newline, so an entry containing a newline cannot collide with two separate entries, and bound the cache. Give the agents the idle timeout the global agents carry, which keepAlive alone did not restore. Reject a unix socket, which carries no host to validate. Also treat fec0::/10 site-local as private, matching the fe80::/10 handling beside it. Exercise the real resolver in handleTools.test.js rather than mocking it, so the wiring test now fails if the agents it threads do not actually block a private target. * 🛡️ fix: Derive Proxy Exemptions Through Axios's Own Resolver Unioning every populated proxy variable exempted addresses that never carry a request. proxy-from-env picks a protocol-specific variable before all_proxy and lowercase before uppercase, so an ignored value became a trusted host:port that a redirect onto a direct route could reach. It also normalizes a scheme-less value such as proxy.internal:3128 to an http URL, where parsing the raw string yielded an empty hostname and no exemption at all, breaking the proxy hop. Resolve through getProxyForUrl, the entry point Axios itself calls, so precedence, scheme normalization, and NO_PROXY match exactly and cannot drift. NO_PROXY covering everything now yields no exemption, since nothing is proxied. Declared locally rather than adding a types package, alongside the existing declaration in the same directory. Also revert the fec0::/10 site-local change. domain.spec asserts that boundary deliberately to prove the fe80::/10 mask does not over-reach, and the shared address schema still classifies fec0 as public, so a runtime block there would leave operators unable to configure the exemption. It belongs with those two together, not in this PR. * 🛡️ fix: Resolve Proxy Exemptions Against the Real Destinations Resolving against placeholder probe hosts applied destination-specific NO_PROXY rules to a host nobody dials. With NO_PROXY matching the probe domain but not a real provider, no exemption was derived even though Axios still proxied the actual request, so the agent rejected the private proxy hop with ESSRF. Resolve per configured destination instead, passing the values loadWebSearchAuth already resolved. Only plaintext http destinations are considered, since for an https destination Axios substitutes its own CONNECT tunnel and never uses the injected agent for the proxy connection, which is also why provider defaults need no exemption: every one of them is https. * 🛡️ fix: Accept Embedded-IPv4 IPv6 Forms in the Address Exemption Schema The runtime guard blocks 6to4, NAT64, and Teredo addresses whose embedded IPv4 is private, but the schema's local copy recognized only ULA, link-local, and the dotted IPv4-mapped form, so an entry such as [64:ff9b::a00:1]:8080 was dropped as a public literal. An operator reaching a private endpoint that way could not configure the exemption at all. Mirror hasPrivateEmbeddedIPv4 in the schema helper, which the surrounding comment already asks to keep in sync. Public embedded addresses stay rejected, since an exemption there has no defensive purpose.
This commit is contained in:
Dustin Healy
GitHub
parent
7cf4c3f73f
commit
87a8b9aa12
15 changed files with 730 additions and 26 deletions
|
|
@ -18,6 +18,7 @@ const {
|
|||
DELETE_MEMORY_TOOL_NAME,
|
||||
createAskUserQuestionTool,
|
||||
ASK_USER_QUESTION_TOOL_NAME,
|
||||
resolveWebSearchSSRFAgents,
|
||||
buildWebSearchDynamicContext,
|
||||
} = require('@librechat/api');
|
||||
const {
|
||||
|
|
@ -396,6 +397,10 @@ const loadTools = async ({
|
|||
webSearchConfig: webSearch,
|
||||
});
|
||||
const { onSearchResults, onGetHighlights } = options?.[Tools.web_search] ?? {};
|
||||
const { httpAgent, httpsAgent } = resolveWebSearchSSRFAgents(
|
||||
result.authResult,
|
||||
webSearch?.allowedAddresses,
|
||||
);
|
||||
requestedTools[tool] = async () => {
|
||||
toolContextMap[tool] = buildWebSearchContext();
|
||||
dynamicToolContextMap[tool] = buildWebSearchDynamicContext(
|
||||
|
|
@ -403,6 +408,8 @@ const loadTools = async ({
|
|||
);
|
||||
return createSearchTool({
|
||||
...result.authResult,
|
||||
httpAgent,
|
||||
httpsAgent,
|
||||
onSearchResults,
|
||||
onGetHighlights,
|
||||
logger,
|
||||
|
|
|
|||
|
|
@ -12,6 +12,21 @@ const mockCreateMCPTools = jest.fn();
|
|||
const mockGetServerConfig = jest.fn();
|
||||
const mockGetAccessibleMcpServerNames = jest.fn(async () => []);
|
||||
|
||||
const mockCreateSearchTool = jest.fn(() => ({ name: 'web_search' }));
|
||||
const mockLoadWebSearchAuth = jest.fn(async () => ({
|
||||
authResult: { searchProvider: 'serper', searxngInstanceUrl: 'http://searxng.internal:8080' },
|
||||
}));
|
||||
|
||||
jest.mock('@librechat/agents', () => ({
|
||||
...jest.requireActual('@librechat/agents'),
|
||||
createSearchTool: (...args) => mockCreateSearchTool(...args),
|
||||
}));
|
||||
|
||||
jest.mock('@librechat/api', () => ({
|
||||
...jest.requireActual('@librechat/api'),
|
||||
loadWebSearchAuth: (...args) => mockLoadWebSearchAuth(...args),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/PluginService', () => mockPluginService);
|
||||
|
||||
jest.mock('~/server/services/Config', () => ({
|
||||
|
|
@ -70,7 +85,7 @@ jest.mock('~/config', () => ({
|
|||
}));
|
||||
|
||||
const { Calculator } = require('@librechat/agents');
|
||||
const { Constants } = require('librechat-data-provider');
|
||||
const { Tools, Constants } = require('librechat-data-provider');
|
||||
const { ASK_USER_QUESTION_TOOL_NAME } = require('@librechat/api');
|
||||
|
||||
const { User } = require('~/db/models');
|
||||
|
|
@ -814,4 +829,56 @@ describe('Tool Handlers', () => {
|
|||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('web_search SSRF-safe agent wiring', () => {
|
||||
const buildReq = () => ({
|
||||
user: { id: fakeUser._id.toString(), role: 'USER' },
|
||||
body: {},
|
||||
});
|
||||
|
||||
/** Uses the real resolver, so this fails if the wiring delivers agents that do not guard. */
|
||||
async function loadWebSearchConfig(webSearch) {
|
||||
const toolMap = await loadTools({
|
||||
user: fakeUser._id.toString(),
|
||||
tools: [Tools.web_search],
|
||||
returnMap: true,
|
||||
webSearch,
|
||||
options: { req: buildReq() },
|
||||
});
|
||||
await toolMap[Tools.web_search]();
|
||||
return mockCreateSearchTool.mock.calls.at(-1)[0];
|
||||
}
|
||||
|
||||
it('threads pooled SSRF-safe agents into the search tool config', async () => {
|
||||
const config = await loadWebSearchConfig({ allowedAddresses: ['localhost:8888'] });
|
||||
|
||||
expect(typeof config.httpAgent.createConnection).toBe('function');
|
||||
expect(typeof config.httpsAgent.createConnection).toBe('function');
|
||||
expect(config.httpAgent.options.keepAlive).toBe(true);
|
||||
});
|
||||
|
||||
it('threads agents that actually reject a private target', async () => {
|
||||
const config = await loadWebSearchConfig({});
|
||||
|
||||
expect(() =>
|
||||
config.httpAgent.createConnection({ host: '169.254.169.254', port: 80 }),
|
||||
).toThrow(expect.objectContaining({ code: 'ESSRF' }));
|
||||
});
|
||||
|
||||
it('honors allowedAddresses end to end, exempting the configured host:port only', async () => {
|
||||
const config = await loadWebSearchConfig({ allowedAddresses: ['127.0.0.1:8080'] });
|
||||
|
||||
const socket = config.httpAgent.createConnection({ host: '127.0.0.1', port: 8080 });
|
||||
socket?.destroy?.();
|
||||
expect(() => config.httpAgent.createConnection({ host: '127.0.0.1', port: 9 })).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('does not throw out of loadTools when allowedAddresses is not an array', async () => {
|
||||
await expect(
|
||||
loadWebSearchConfig({ allowedAddresses: { '10.0.0.5:11434': true } }),
|
||||
).resolves.toBeDefined();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -870,6 +870,33 @@ endpoints:
|
|||
# # Content scrapers
|
||||
# firecrawlApiKey: '${FIRECRAWL_API_KEY}'
|
||||
# firecrawlApiUrl: '${FIRECRAWL_API_URL}'
|
||||
# # Outbound search and scrape requests are validated at connect time against
|
||||
# # their resolved IP and blocked from reaching private, loopback, link-local,
|
||||
# # or cloud-metadata space. `allowedAddresses` is an SSRF exemption list, NOT a
|
||||
# # strict whitelist: hostname/IP + port pairs listed here bypass that block for
|
||||
# # one deliberately-private endpoint (for example a self-hosted SearXNG
|
||||
# # instance); public destinations continue to work normally.
|
||||
# #
|
||||
# # Entries must include a port: `host:port`, `private.ip:port`, or `[ipv6]:port`.
|
||||
# # Do not use URLs, paths, CIDR ranges, bare hosts/IPs, or public IP literals.
|
||||
# # A hostname entry trusts whatever IP that name resolves to on the listed port,
|
||||
# # so only list hosts you fully control and whose DNS cannot be repointed by an
|
||||
# # attacker. Listing an attacker-controllable or DNS-rebindable host re-opens the
|
||||
# # private-address path this guard closes. Prefer literal IPs where you can.
|
||||
# #
|
||||
# # Self-hosted endpoints need an entry. A private destination such as
|
||||
# # `http://searxng:8080`, `http://firecrawl:3002`, or `http://127.0.0.1:8080` is
|
||||
# # blocked once this guard is active, so list it here or those requests will fail.
|
||||
# #
|
||||
# # A proxy from `HTTP_PROXY`, `HTTPS_PROXY`, or `ALL_PROXY` (either case) is
|
||||
# # exempted automatically and needs no entry. That exemption is applied to the
|
||||
# # whole tool rather than per destination, so a host `NO_PROXY` sends direct also
|
||||
# # carries it. Note that when a proxy carries the request the proxy resolves the
|
||||
# # destination, so destination egress policy is the proxy's to enforce, and for
|
||||
# # https targets the proxy's own tunnel replaces this guard entirely.
|
||||
# # allowedAddresses:
|
||||
# # - 'searxng:8080'
|
||||
# # - '127.0.0.1:8080'
|
||||
#
|
||||
# Tavily as both search and scraper provider example:
|
||||
# webSearch:
|
||||
|
|
|
|||
3
package-lock.json
generated
3
package-lock.json
generated
|
|
@ -42667,7 +42667,8 @@
|
|||
"dependencies": {
|
||||
"@langchain/langgraph-checkpoint": "^1.1.2",
|
||||
"@langchain/langgraph-checkpoint-mongodb": "^1.4.0",
|
||||
"cluster-key-slot": "^1.1.2"
|
||||
"cluster-key-slot": "^1.1.2",
|
||||
"proxy-from-env": "^2.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/preset-env": "^7.29.5",
|
||||
|
|
|
|||
|
|
@ -168,6 +168,7 @@
|
|||
"dependencies": {
|
||||
"@langchain/langgraph-checkpoint": "^1.1.2",
|
||||
"@langchain/langgraph-checkpoint-mongodb": "^1.4.0",
|
||||
"cluster-key-slot": "^1.1.2"
|
||||
"cluster-key-slot": "^1.1.2",
|
||||
"proxy-from-env": "^2.1.0"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -87,24 +87,77 @@ function buildSSRFSafeLookup(
|
|||
/** Default lookup with no exemptions. Kept for callers that don't need allowedAddresses. */
|
||||
const ssrfSafeLookup: LookupFunction = buildSSRFSafeLookup();
|
||||
|
||||
/** Connect options Node hands to `createConnection`; typed here because the seam is untyped. */
|
||||
interface ConnectOptions {
|
||||
host?: unknown;
|
||||
port?: unknown;
|
||||
defaultPort?: unknown;
|
||||
socketPath?: unknown;
|
||||
lookup?: LookupFunction;
|
||||
}
|
||||
|
||||
/** Internal agent shape exposing createConnection (exists at runtime but not in TS types) */
|
||||
type AgentInternal = {
|
||||
createConnection: (options: Record<string, unknown>, oncreate?: unknown) => unknown;
|
||||
createConnection: (options: ConnectOptions, oncreate?: unknown) => unknown;
|
||||
};
|
||||
|
||||
function getConnectionPort(options: Record<string, unknown>): string {
|
||||
function getConnectionPort(options: ConnectOptions): string {
|
||||
return normalizePort(options.port ?? options.defaultPort);
|
||||
}
|
||||
|
||||
/**
|
||||
* Rejects a connection whose host is already an IP literal in blocked space.
|
||||
*
|
||||
* Node resolves nothing for a literal host, so the SSRF lookup below never runs for one.
|
||||
* Redirect hops reach this same `createConnection`, so checking here is what covers a
|
||||
* redirect whose target is a literal private address. Opt-in: a caller that reaches a
|
||||
* proxy or a deliberate private service by literal address must exempt it first, so
|
||||
* enabling this by default would break existing configurations.
|
||||
*/
|
||||
function assertLiteralHostAllowed(
|
||||
options: ConnectOptions,
|
||||
allowedAddresses?: string[] | null,
|
||||
): void {
|
||||
/** A unix socket carries no host to validate, and these agents exist for http(s) URLs only. */
|
||||
if (options.socketPath != null) {
|
||||
throw createSSRFLookupError('socketPath', String(options.socketPath));
|
||||
}
|
||||
|
||||
const host = typeof options.host === 'string' ? options.host.replace(/^\[|\]$/g, '') : '';
|
||||
if (host.length === 0 || !isIP(host)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const port = getConnectionPort(options);
|
||||
if (isAddressInAllowedSet(host, normalizeAllowedAddressesSet(allowedAddresses), port)) {
|
||||
return;
|
||||
}
|
||||
if (isPrivateIP(host)) {
|
||||
throw createSSRFLookupError(host, host);
|
||||
}
|
||||
}
|
||||
|
||||
export interface SSRFProtectionOptions {
|
||||
/** Also reject IP-literal hosts, covering literal destinations and literal redirect targets. */
|
||||
blockLiteralHosts?: boolean;
|
||||
}
|
||||
|
||||
/** Patches an agent instance to inject SSRF-safe DNS lookup at connect time */
|
||||
function withSSRFProtection<T extends http.Agent>(agent: T, allowedAddresses?: string[] | null): T {
|
||||
function withSSRFProtection<T extends http.Agent>(
|
||||
agent: T,
|
||||
allowedAddresses?: string[] | null,
|
||||
options?: SSRFProtectionOptions,
|
||||
): T {
|
||||
const internal = agent as unknown as AgentInternal;
|
||||
const origCreate = internal.createConnection.bind(agent);
|
||||
internal.createConnection = (options: Record<string, unknown>, oncreate?: unknown) => {
|
||||
options.lookup = allowedAddresses?.length
|
||||
? buildSSRFSafeLookup(allowedAddresses, getConnectionPort(options))
|
||||
internal.createConnection = (connectOptions: ConnectOptions, oncreate?: unknown) => {
|
||||
if (options?.blockLiteralHosts) {
|
||||
assertLiteralHostAllowed(connectOptions, allowedAddresses);
|
||||
}
|
||||
connectOptions.lookup = allowedAddresses?.length
|
||||
? buildSSRFSafeLookup(allowedAddresses, getConnectionPort(connectOptions))
|
||||
: ssrfSafeLookup;
|
||||
return origCreate(options, oncreate);
|
||||
return origCreate(connectOptions, oncreate);
|
||||
};
|
||||
return agent;
|
||||
}
|
||||
|
|
@ -116,14 +169,21 @@ function withSSRFProtection<T extends http.Agent>(agent: T, allowedAddresses?: s
|
|||
* pre-validation but to a private IP when the actual connection is made.
|
||||
*
|
||||
* @param allowedAddresses - Optional admin exemption list of host:port pairs that bypass the block.
|
||||
* @param agentOptions - Agent options, e.g. `{ keepAlive: true }` to retain pooling that the
|
||||
* default global agents provide and a bare `new http.Agent()` does not.
|
||||
*/
|
||||
export function createSSRFSafeAgents(allowedAddresses?: string[] | null): {
|
||||
export function createSSRFSafeAgents(
|
||||
allowedAddresses?: string[] | null,
|
||||
agentOptions?: (http.AgentOptions & https.AgentOptions) | null,
|
||||
protection?: SSRFProtectionOptions,
|
||||
): {
|
||||
httpAgent: http.Agent;
|
||||
httpsAgent: https.Agent;
|
||||
} {
|
||||
const options = agentOptions ?? undefined;
|
||||
return {
|
||||
httpAgent: withSSRFProtection(new http.Agent(), allowedAddresses),
|
||||
httpsAgent: withSSRFProtection(new https.Agent(), allowedAddresses),
|
||||
httpAgent: withSSRFProtection(new http.Agent(options), allowedAddresses, protection),
|
||||
httpsAgent: withSSRFProtection(new https.Agent(options), allowedAddresses, protection),
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -276,7 +276,7 @@ function defaultPortForProtocol(protocol: SupportedProtocol | string | null): st
|
|||
return '';
|
||||
}
|
||||
|
||||
function getEffectivePort(
|
||||
export function getEffectivePort(
|
||||
protocol: SupportedProtocol | string | null,
|
||||
port?: string | null,
|
||||
): string {
|
||||
|
|
|
|||
9
packages/api/src/types/proxy-from-env.d.ts
vendored
Normal file
9
packages/api/src/types/proxy-from-env.d.ts
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
/**
|
||||
* `proxy-from-env` ships no types. Declared narrowly rather than pulling in a types package,
|
||||
* since only `getProxyForUrl` is used: it returns the proxy URL for a target, applying
|
||||
* `<protocol>_proxy` before `all_proxy`, lowercase before uppercase, and `NO_PROXY`, or an
|
||||
* empty string when the target should be reached directly.
|
||||
*/
|
||||
declare module 'proxy-from-env' {
|
||||
export function getProxyForUrl(target: string): string;
|
||||
}
|
||||
312
packages/api/src/web/agent.spec.ts
Normal file
312
packages/api/src/web/agent.spec.ts
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
import { resolveWebSearchSSRFAgents } from './agent';
|
||||
import { isAddressAllowed } from '../auth';
|
||||
|
||||
/** `options` exists on a Node agent at runtime but is absent from the bundled type. */
|
||||
interface AgentOptionsProbe {
|
||||
options: { keepAlive?: boolean; timeout?: number };
|
||||
}
|
||||
|
||||
function agentOptions(agent: object): AgentOptionsProbe['options'] {
|
||||
return (agent as AgentOptionsProbe).options;
|
||||
}
|
||||
|
||||
/** Drives the seam every request and every redirect hop passes through. */
|
||||
interface ConnectProbe {
|
||||
createConnection: (options: Record<string, unknown>) => unknown;
|
||||
}
|
||||
|
||||
function connectRaw(agent: object, options: Record<string, unknown>): void {
|
||||
const socket = (agent as ConnectProbe).createConnection(options);
|
||||
(socket as { destroy?: () => void })?.destroy?.();
|
||||
}
|
||||
|
||||
function connect(agent: object, host: string, port: number): void {
|
||||
connectRaw(agent, { host, port });
|
||||
}
|
||||
|
||||
const HTTP_DEST = { searxngInstanceUrl: 'http://searxng.internal:8080' };
|
||||
|
||||
describe('resolveWebSearchSSRFAgents', () => {
|
||||
const originalEnv = process.env;
|
||||
|
||||
beforeEach(() => {
|
||||
process.env = { ...originalEnv };
|
||||
for (const key of [
|
||||
'PROXY',
|
||||
'proxy',
|
||||
'HTTP_PROXY',
|
||||
'http_proxy',
|
||||
'HTTPS_PROXY',
|
||||
'https_proxy',
|
||||
'ALL_PROXY',
|
||||
'all_proxy',
|
||||
'NO_PROXY',
|
||||
'no_proxy',
|
||||
]) {
|
||||
delete process.env[key];
|
||||
}
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.env = originalEnv;
|
||||
});
|
||||
|
||||
it('always returns a pooled agent pair', () => {
|
||||
const { httpAgent, httpsAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(httpAgent).toBeDefined();
|
||||
expect(httpsAgent).toBeDefined();
|
||||
expect(agentOptions(httpAgent).keepAlive).toBe(true);
|
||||
expect(agentOptions(httpAgent).timeout).toBe(5000);
|
||||
expect(agentOptions(httpsAgent).keepAlive).toBe(true);
|
||||
});
|
||||
|
||||
it('still returns agents when a proxy is configured, so direct routes stay guarded', () => {
|
||||
process.env.HTTPS_PROXY = 'http://proxy.internal:3128';
|
||||
|
||||
const { httpAgent, httpsAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(httpAgent).toBeDefined();
|
||||
expect(httpsAgent).toBeDefined();
|
||||
});
|
||||
|
||||
it('exempts the proxy endpoint so the proxy hop stays reachable', () => {
|
||||
process.env.HTTP_PROXY = 'http://proxy.internal:3128';
|
||||
|
||||
resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(isAddressAllowed('proxy.internal', ['proxy.internal:3128'], '3128')).toBe(true);
|
||||
});
|
||||
|
||||
it('scopes the proxy exemption to its port, so another private port stays blocked', () => {
|
||||
expect(isAddressAllowed('proxy.internal', ['proxy.internal:3128'], '9')).toBe(false);
|
||||
});
|
||||
|
||||
it('derives the default proxy port from the proxy scheme', () => {
|
||||
process.env.HTTP_PROXY = 'http://proxy.internal';
|
||||
process.env.HTTPS_PROXY = 'https://secure-proxy.internal';
|
||||
|
||||
const first = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
process.env.HTTP_PROXY = 'http://proxy.internal:80';
|
||||
process.env.HTTPS_PROXY = 'https://secure-proxy.internal:443';
|
||||
|
||||
expect(resolveWebSearchSSRFAgents(HTTP_DEST)).toBe(first);
|
||||
});
|
||||
|
||||
it('grants no exemption for PROXY, which Axios never reads on this path', () => {
|
||||
process.env.PROXY = 'http://10.4.4.4:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.4.4.4', 3128)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('exempts a scheme-less proxy value, which the resolver normalizes to http', () => {
|
||||
process.env.HTTP_PROXY = 'proxy.internal:3128';
|
||||
|
||||
resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(isAddressAllowed('proxy.internal', ['proxy.internal:3128'], '3128')).toBe(true);
|
||||
});
|
||||
|
||||
it('exempts a scheme-less literal proxy so the hop stays reachable', () => {
|
||||
process.env.HTTP_PROXY = '10.1.2.3:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.1.2.3', 3128)).not.toThrow();
|
||||
});
|
||||
|
||||
it('exempts only the variable that wins precedence, not every populated one', () => {
|
||||
process.env.HTTP_PROXY = 'http://10.1.1.1:3128';
|
||||
process.env.HTTPS_PROXY = 'http://10.1.1.1:3128';
|
||||
process.env.ALL_PROXY = 'http://10.2.2.2:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.1.1.1', 3128)).not.toThrow();
|
||||
expect(() => connect(httpAgent, '10.2.2.2', 3128)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('prefers the lowercase variable, matching the resolver', () => {
|
||||
process.env.http_proxy = 'http://10.3.3.3:3128';
|
||||
process.env.HTTP_PROXY = 'http://10.4.4.4:3128';
|
||||
process.env.https_proxy = 'http://10.3.3.3:3128';
|
||||
process.env.HTTPS_PROXY = 'http://10.3.3.3:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.3.3.3', 3128)).not.toThrow();
|
||||
expect(() => connect(httpAgent, '10.4.4.4', 3128)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('grants no exemption when NO_PROXY covers the destination, since nothing is proxied', () => {
|
||||
process.env.HTTP_PROXY = 'http://10.6.6.6:3128';
|
||||
process.env.NO_PROXY = 'searxng.internal';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.6.6.6', 3128)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('exempts the proxy when NO_PROXY does not cover the destination', () => {
|
||||
process.env.HTTP_PROXY = 'http://10.7.7.7:3128';
|
||||
process.env.NO_PROXY = 'other.internal';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.7.7.7', 3128)).not.toThrow();
|
||||
});
|
||||
|
||||
it('owes no exemption for an https destination, where Axios tunnels instead', () => {
|
||||
process.env.HTTPS_PROXY = 'http://10.8.8.8:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents({
|
||||
searxngInstanceUrl: 'https://searx.example.com',
|
||||
});
|
||||
|
||||
expect(() => connect(httpAgent, '10.8.8.8', 3128)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('grants no exemption for a socks proxy, which Axios cannot use', () => {
|
||||
process.env.ALL_PROXY = 'socks5://10.5.5.5:1080';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.5.5.5', 1080)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('tolerates a non-array allowedAddresses instead of throwing during tool load', () => {
|
||||
const asUnknown = { '10.0.0.5:11434': true } as unknown;
|
||||
|
||||
expect(() => resolveWebSearchSSRFAgents(HTTP_DEST, asUnknown as string[])).not.toThrow();
|
||||
});
|
||||
|
||||
it('does not collide cache keys when an entry contains a newline', () => {
|
||||
const joined = resolveWebSearchSSRFAgents(HTTP_DEST, ['searxng:8080\nfirecrawl:3002']);
|
||||
const separate = resolveWebSearchSSRFAgents(HTTP_DEST, ['searxng:8080', 'firecrawl:3002']);
|
||||
|
||||
expect(joined).not.toBe(separate);
|
||||
});
|
||||
|
||||
it('rejects a unix socket, which carries no host to validate', () => {
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connectRaw(httpAgent, { socketPath: '/tmp/x.sock' })).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('treats an empty proxy variable as unset, matching the value compose forwards', () => {
|
||||
const withoutProxy = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
process.env.HTTP_PROXY = '';
|
||||
process.env.HTTPS_PROXY = ' ';
|
||||
|
||||
expect(resolveWebSearchSSRFAgents(HTTP_DEST)).toBe(withoutProxy);
|
||||
});
|
||||
|
||||
it('reuses one pair per exemption list and separates distinct lists', () => {
|
||||
const first = resolveWebSearchSSRFAgents(HTTP_DEST, ['searxng:8080']);
|
||||
const again = resolveWebSearchSSRFAgents(HTTP_DEST, ['searxng:8080']);
|
||||
const other = resolveWebSearchSSRFAgents(HTTP_DEST, ['firecrawl:3002']);
|
||||
|
||||
expect(again).toBe(first);
|
||||
expect(other).not.toBe(first);
|
||||
});
|
||||
|
||||
it('keeps admin allowedAddresses entries alongside the derived proxy exemption', () => {
|
||||
process.env.HTTP_PROXY = 'http://proxy.internal:3128';
|
||||
|
||||
expect(resolveWebSearchSSRFAgents(HTTP_DEST, ['searxng:8080'])).not.toBe(
|
||||
resolveWebSearchSSRFAgents(HTTP_DEST, []),
|
||||
);
|
||||
});
|
||||
|
||||
it('does not throw while building agents, leaving enforcement at connect time', () => {
|
||||
expect(() => resolveWebSearchSSRFAgents(HTTP_DEST, ['127.0.0.1:8080'])).not.toThrow();
|
||||
});
|
||||
|
||||
it('rejects an IP-literal private host at connect time, covering literal redirect targets', () => {
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '169.254.169.254', 80)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
expect(() => connect(httpAgent, '127.0.0.1', 8080)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('exempts an IP-literal host listed in allowedAddresses, scoped to its port', () => {
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST, ['127.0.0.1:8080']);
|
||||
|
||||
expect(() => connect(httpAgent, '127.0.0.1', 8080)).not.toThrow();
|
||||
expect(() => connect(httpAgent, '127.0.0.1', 9)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps a literal-address proxy reachable, since the proxy hop is exempted', () => {
|
||||
process.env.HTTP_PROXY = 'http://10.1.2.3:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.1.2.3', 3128)).not.toThrow();
|
||||
});
|
||||
|
||||
it('exempts a proxy configured only through ALL_PROXY, which Axios still honors', () => {
|
||||
process.env.ALL_PROXY = 'http://10.1.2.3:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.1.2.3', 3128)).not.toThrow();
|
||||
expect(() => connect(httpAgent, '10.1.2.3', 9)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('exempts a lowercase all_proxy as well, since the resolver is case-insensitive', () => {
|
||||
process.env.all_proxy = 'http://10.9.9.9:8080';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '10.9.9.9', 8080)).not.toThrow();
|
||||
});
|
||||
|
||||
it('keeps an IPv6-literal proxy reachable, which needs the bracketed exemption form', () => {
|
||||
process.env.HTTP_PROXY = 'http://[fd00::1]:3128';
|
||||
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, 'fd00::1', 3128)).not.toThrow();
|
||||
expect(() => connect(httpAgent, 'fd00::1', 9)).toThrow(
|
||||
expect.objectContaining({ code: 'ESSRF' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('allows a public IP literal', () => {
|
||||
const { httpAgent } = resolveWebSearchSSRFAgents(HTTP_DEST);
|
||||
|
||||
expect(() => connect(httpAgent, '93.184.216.34', 80)).not.toThrow();
|
||||
});
|
||||
|
||||
it('ignores an unparseable proxy value rather than throwing during tool load', () => {
|
||||
process.env.HTTP_PROXY = 'not a url';
|
||||
|
||||
expect(() => resolveWebSearchSSRFAgents(HTTP_DEST)).not.toThrow();
|
||||
});
|
||||
});
|
||||
109
packages/api/src/web/agent.ts
Normal file
109
packages/api/src/web/agent.ts
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
import { getProxyForUrl } from 'proxy-from-env';
|
||||
import type { TWebSearchConfig } from 'librechat-data-provider';
|
||||
import type https from 'node:https';
|
||||
import type http from 'node:http';
|
||||
import { createSSRFSafeAgents } from '../auth';
|
||||
|
||||
export interface WebSearchSSRFAgents {
|
||||
httpAgent: http.Agent;
|
||||
httpsAgent: https.Agent;
|
||||
}
|
||||
|
||||
/** Resolved web-search fields that carry an outbound destination. */
|
||||
const WEB_SEARCH_URL_KEYS = [
|
||||
'searxngInstanceUrl',
|
||||
'firecrawlApiUrl',
|
||||
'jinaApiUrl',
|
||||
'tavilySearchUrl',
|
||||
'tavilyExtractUrl',
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* For a plaintext http destination Axios repoints the caller's agent at the proxy host, so the
|
||||
* connect-time check would resolve the proxy itself and reject a private one. Exempting that
|
||||
* endpoint keeps the hop reachable while every direct destination stays guarded.
|
||||
*
|
||||
* Resolution runs per destination through `getProxyForUrl`, the same `proxy-from-env` entry point
|
||||
* Axios uses, so variable precedence, scheme-less normalization, and `NO_PROXY` all match for the
|
||||
* host actually being dialed. Only http destinations are considered: for an https destination
|
||||
* Axios substitutes its own CONNECT tunnel and never uses the injected agent for the proxy
|
||||
* connection, so no exemption is owed. Provider defaults are all https for the same reason.
|
||||
*/
|
||||
function getProxyExemptions(authResult: Partial<TWebSearchConfig>): string[] {
|
||||
const entries = new Set<string>();
|
||||
for (const key of WEB_SEARCH_URL_KEYS) {
|
||||
const destination = authResult[key];
|
||||
if (typeof destination !== 'string' || destination.length === 0) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
if (new URL(destination).protocol !== 'http:') {
|
||||
continue;
|
||||
}
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
const proxyUrl = getProxyForUrl(destination);
|
||||
if (!proxyUrl) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
/** `hostname` keeps IPv6 brackets, which the exemption parser requires as `[ipv6]:port`. */
|
||||
const { hostname, port, protocol } = new URL(proxyUrl);
|
||||
/** Axios proxies over http(s) only, so a socks endpoint never carries these requests. */
|
||||
if (hostname.length === 0 || (protocol !== 'http:' && protocol !== 'https:')) {
|
||||
continue;
|
||||
}
|
||||
entries.add(`${hostname}:${port || (protocol === 'https:' ? '443' : '80')}`);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return [...entries];
|
||||
}
|
||||
|
||||
/** Keyed by exemption list so repeated tool loads reuse one pooled pair. */
|
||||
const agentsByExemptions = new Map<string, WebSearchSSRFAgents>();
|
||||
|
||||
/** Distinct exemption lists are bounded by admin configuration; the cap only guards a leak. */
|
||||
const MAX_CACHED_AGENT_PAIRS = 32;
|
||||
|
||||
/**
|
||||
* Connect-time SSRF agents for the web-search tool, which issues its own requests and accepts
|
||||
* only a shared agent pair.
|
||||
*
|
||||
* Redirect hops traverse these agents, so `blockLiteralHosts` covers a redirect to a private
|
||||
* address whether it is named or a literal, which is what `maxRedirects` would otherwise be
|
||||
* needed for. A blocked redirect surfaces as `ERR_FR_REDIRECTION_FAILURE` with the `ESSRF`
|
||||
* message attached, because `follow-redirects` wraps the agent's error.
|
||||
*
|
||||
* When a proxy carries the request the proxy resolves the destination, so enforcement there
|
||||
* belongs to the proxy's egress policy. The proxy endpoint is exempted for the whole pair rather
|
||||
* than per destination, since one shared pair cannot discriminate: a destination that `NO_PROXY`
|
||||
* sends direct therefore also carries that exemption.
|
||||
*/
|
||||
export function resolveWebSearchSSRFAgents(
|
||||
authResult: Partial<TWebSearchConfig>,
|
||||
allowedAddresses?: string[] | null,
|
||||
): WebSearchSSRFAgents {
|
||||
const configured = Array.isArray(allowedAddresses) ? allowedAddresses : [];
|
||||
const exemptions = [...configured, ...getProxyExemptions(authResult ?? {})];
|
||||
const cacheKey = exemptions.join('\0');
|
||||
|
||||
const cached = agentsByExemptions.get(cacheKey);
|
||||
if (cached) {
|
||||
return cached;
|
||||
}
|
||||
|
||||
const agents = createSSRFSafeAgents(
|
||||
exemptions,
|
||||
{ keepAlive: true, timeout: 5000 },
|
||||
{ blockLiteralHosts: true },
|
||||
);
|
||||
if (agentsByExemptions.size >= MAX_CACHED_AGENT_PAIRS) {
|
||||
agentsByExemptions.clear();
|
||||
}
|
||||
agentsByExemptions.set(cacheKey, agents);
|
||||
return agents;
|
||||
}
|
||||
|
|
@ -1 +1,2 @@
|
|||
export * from './web';
|
||||
export * from './agent';
|
||||
|
|
|
|||
|
|
@ -24,6 +24,12 @@ const mockResolveHostnameSSRF = jest.fn().mockResolvedValue(false);
|
|||
jest.mock('../auth', () => ({
|
||||
isSSRFTarget: (...args: unknown[]) => mockIsSSRFTarget(...args),
|
||||
resolveHostnameSSRF: (...args: unknown[]) => mockResolveHostnameSSRF(...args),
|
||||
getEffectivePort: (protocol: string, port?: string) => {
|
||||
if (port) {
|
||||
return port;
|
||||
}
|
||||
return protocol === 'https:' ? '443' : '80';
|
||||
},
|
||||
}));
|
||||
|
||||
describe('web.ts', () => {
|
||||
|
|
@ -353,8 +359,16 @@ describe('web.ts', () => {
|
|||
expect(result.authenticated).toBe(true);
|
||||
expect(result.authResult.tavilySearchUrl).toBe('https://tenant-search.example/search');
|
||||
expect(result.authResult.tavilyExtractUrl).toBe('https://tenant-extract.example/extract');
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith('tenant-search.example');
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith('tenant-extract.example');
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith(
|
||||
'tenant-search.example',
|
||||
undefined,
|
||||
'443',
|
||||
);
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith(
|
||||
'tenant-extract.example',
|
||||
undefined,
|
||||
'443',
|
||||
);
|
||||
expect(result.authTypes).toEqual([
|
||||
['providers', AuthType.USER_PROVIDED],
|
||||
['scrapers', AuthType.USER_PROVIDED],
|
||||
|
|
@ -365,6 +379,52 @@ describe('web.ts', () => {
|
|||
}
|
||||
});
|
||||
|
||||
it('threads allowedAddresses and the effective port into the SSRF preflight for user-provided URLs', async () => {
|
||||
mockIsSSRFTarget.mockClear();
|
||||
mockResolveHostnameSSRF.mockClear();
|
||||
mockIsSSRFTarget.mockReturnValue(false);
|
||||
mockResolveHostnameSSRF.mockResolvedValue(false);
|
||||
|
||||
const originalEnv = process.env;
|
||||
try {
|
||||
process.env = { ...originalEnv };
|
||||
delete process.env.SEARXNG_INSTANCE_URL;
|
||||
|
||||
const searxngConfig = {
|
||||
searxngInstanceUrl: '${SEARXNG_INSTANCE_URL}',
|
||||
searchProvider: 'searxng' as SearchProviders,
|
||||
rerankerType: 'none' as RerankerTypes,
|
||||
allowedAddresses: ['localhost:8888'],
|
||||
} as TWebSearchConfig;
|
||||
|
||||
mockLoadAuthValues.mockImplementation(({ authFields }) => {
|
||||
const result: Record<string, string> = {};
|
||||
authFields.forEach((field: string) => {
|
||||
if (field === 'SEARXNG_INSTANCE_URL') {
|
||||
result[field] = 'http://localhost:8888';
|
||||
}
|
||||
});
|
||||
return Promise.resolve(result);
|
||||
});
|
||||
|
||||
const result = await loadWebSearchAuth({
|
||||
userId,
|
||||
webSearchConfig: searxngConfig,
|
||||
loadAuthValues: mockLoadAuthValues,
|
||||
});
|
||||
|
||||
expect(mockIsSSRFTarget).toHaveBeenCalledWith('localhost', ['localhost:8888'], '8888');
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith(
|
||||
'localhost',
|
||||
['localhost:8888'],
|
||||
'8888',
|
||||
);
|
||||
expect(result.authResult.searxngInstanceUrl).toBe('http://localhost:8888');
|
||||
} finally {
|
||||
process.env = originalEnv;
|
||||
}
|
||||
});
|
||||
|
||||
it('should preserve safeSearch setting from webSearchConfig', async () => {
|
||||
// Mock successful authentication
|
||||
mockLoadAuthValues.mockImplementation(({ authFields }) => {
|
||||
|
|
@ -1675,7 +1735,7 @@ describe('web.ts', () => {
|
|||
});
|
||||
|
||||
expect(result.authResult.jinaApiUrl).toBeUndefined();
|
||||
expect(mockIsSSRFTarget).toHaveBeenCalledWith('localhost');
|
||||
expect(mockIsSSRFTarget).toHaveBeenCalledWith('localhost', undefined, '8080');
|
||||
});
|
||||
|
||||
it('should block user-provided firecrawlApiUrl resolving to private IP', async () => {
|
||||
|
|
@ -1832,7 +1892,7 @@ describe('web.ts', () => {
|
|||
expect(result.authResult.tavilySearchUrl).toBeUndefined();
|
||||
expect(result.authResult.searchProvider).toBe('tavily');
|
||||
expect(result.authenticated).toBe(true);
|
||||
expect(mockIsSSRFTarget).toHaveBeenCalledWith('localhost');
|
||||
expect(mockIsSSRFTarget).toHaveBeenCalledWith('localhost', undefined, '8080');
|
||||
} finally {
|
||||
process.env = originalEnv;
|
||||
}
|
||||
|
|
@ -1880,7 +1940,11 @@ describe('web.ts', () => {
|
|||
expect(result.authResult.tavilyExtractUrl).toBeUndefined();
|
||||
expect(result.authResult.scraperProvider).toBe('tavily');
|
||||
expect(result.authenticated).toBe(true);
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith('extract.internal-service.com');
|
||||
expect(mockResolveHostnameSSRF).toHaveBeenCalledWith(
|
||||
'extract.internal-service.com',
|
||||
undefined,
|
||||
'443',
|
||||
);
|
||||
} finally {
|
||||
process.env = originalEnv;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { webSearchAuth } from '@librechat/data-schemas';
|
||||
import {
|
||||
AuthType,
|
||||
SafeSearchTypes,
|
||||
|
|
@ -6,10 +7,9 @@ import {
|
|||
ScraperProviders,
|
||||
extractVariableName,
|
||||
} from 'librechat-data-provider';
|
||||
import { webSearchAuth } from '@librechat/data-schemas';
|
||||
import type { RerankerTypes, TCustomConfig, TWebSearchConfig } from 'librechat-data-provider';
|
||||
import type { TWebSearchKeys, TWebSearchCategories } from '@librechat/data-schemas';
|
||||
import { isSSRFTarget, resolveHostnameSSRF } from '../auth';
|
||||
import { isSSRFTarget, resolveHostnameSSRF, getEffectivePort } from '../auth';
|
||||
|
||||
/**
|
||||
* User-provided URL keys that may pass through after SSRF preflight.
|
||||
|
|
@ -35,8 +35,10 @@ function isUserProvidedEnabled(field: string): boolean {
|
|||
/**
|
||||
* Returns true if the URL should be blocked for SSRF risk.
|
||||
* Fail-closed: unparseable URLs and non-HTTP(S) schemes return true.
|
||||
* `allowedAddresses` keeps this preflight consistent with the connect-time agent
|
||||
* so an admin-permitted private endpoint is not stripped before the agent runs.
|
||||
*/
|
||||
async function isSSRFUrl(url: string): Promise<boolean> {
|
||||
async function isSSRFUrl(url: string, allowedAddresses?: string[] | null): Promise<boolean> {
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(url);
|
||||
|
|
@ -46,10 +48,11 @@ async function isSSRFUrl(url: string): Promise<boolean> {
|
|||
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
|
||||
return true;
|
||||
}
|
||||
if (isSSRFTarget(parsed.hostname)) {
|
||||
const port = getEffectivePort(parsed.protocol, parsed.port);
|
||||
if (isSSRFTarget(parsed.hostname, allowedAddresses, port)) {
|
||||
return true;
|
||||
}
|
||||
return resolveHostnameSSRF(parsed.hostname);
|
||||
return resolveHostnameSSRF(parsed.hostname, allowedAddresses, port);
|
||||
}
|
||||
|
||||
export function extractWebSearchEnvVars({
|
||||
|
|
@ -216,7 +219,11 @@ export async function loadWebSearchAuth({
|
|||
continue;
|
||||
}
|
||||
|
||||
if (isUserProvidedUrlEnabled && isFieldUserProvided && (await isSSRFUrl(value))) {
|
||||
if (
|
||||
isUserProvidedUrlEnabled &&
|
||||
isFieldUserProvided &&
|
||||
(await isSSRFUrl(value, webSearchConfig?.allowedAddresses))
|
||||
) {
|
||||
if (!optionalSet.has(field)) {
|
||||
allFieldsAuthenticated = false;
|
||||
break;
|
||||
|
|
|
|||
|
|
@ -377,6 +377,10 @@ describe('allowedAddressesSchema', () => {
|
|||
['[fc00::1]:8080', 'IPv6 unique-local with port'],
|
||||
['[fd00::1]:8080', 'IPv6 unique-local with port'],
|
||||
['[fe80::1]:8080', 'IPv6 link-local with port'],
|
||||
['[::ffff:10.0.0.5]:8080', 'IPv4-mapped IPv6 of a private address'],
|
||||
['[64:ff9b::a00:1]:8080', 'NAT64 embedding private 10.0.0.1'],
|
||||
['[2002:a00:1::]:8080', '6to4 embedding private 10.0.0.1'],
|
||||
['[2001::ffff:f5ff:fffe]:8080', 'Teredo embedding a private address'],
|
||||
])('accepts "%s" (%s)', (entry) => {
|
||||
expect(allowedAddressesSchema.parse([entry])).toEqual([entry]);
|
||||
});
|
||||
|
|
@ -398,6 +402,8 @@ describe('allowedAddressesSchema', () => {
|
|||
['https://internal.example', 'https URL'],
|
||||
['ws://10.0.0.5', 'ws URL'],
|
||||
['10.0.0.0/24', 'CIDR range'],
|
||||
['[64:ff9b::808:808]:8080', 'NAT64 embedding public 8.8.8.8'],
|
||||
['[2002:808:808::]:8080', '6to4 embedding public 8.8.8.8'],
|
||||
['/path', 'leading slash / path'],
|
||||
['10.0.0.5/api', 'embedded path'],
|
||||
['localhost', 'bare hostname'],
|
||||
|
|
|
|||
|
|
@ -113,6 +113,38 @@ function isPrivateIPv4Literal(value: string): boolean {
|
|||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirrors `hasPrivateEmbeddedIPv4` in `@librechat/api`'s ip helpers: 6to4, NAT64, and Teredo
|
||||
* carry an IPv4 address inside the IPv6 one, and the runtime guard blocks those when the
|
||||
* embedded address is private. Kept in sync so an operator can exempt what the runtime blocks.
|
||||
*/
|
||||
function hasPrivateEmbeddedIPv4Literal(value: string): boolean {
|
||||
const is6to4 = value.startsWith('2002:');
|
||||
const isNat64 = value.startsWith('64:ff9b::');
|
||||
const isTeredo = value.startsWith('2001::');
|
||||
if (!is6to4 && !isNat64 && !isTeredo) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const segments = value.split(':').filter((segment) => segment !== '');
|
||||
const pair = is6to4 ? segments.slice(1, 3) : segments.slice(-2);
|
||||
if (pair.length !== 2) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const hi = parseInt(pair[0], 16);
|
||||
const lo = parseInt(pair[1], 16);
|
||||
if (isNaN(hi) || isNaN(lo)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
/** RFC 4380: Teredo stores the external IPv4 as a bitwise complement. */
|
||||
const high = isTeredo ? ~hi : hi;
|
||||
const low = isTeredo ? ~lo : lo;
|
||||
const octets = [(high >> 8) & 0xff, high & 0xff, (low >> 8) & 0xff, low & 0xff];
|
||||
return isPrivateIPv4Literal(octets.join('.'));
|
||||
}
|
||||
|
||||
function isPrivateIPv6Literal(value: string): boolean {
|
||||
if (!value.includes(':')) return false;
|
||||
if (value === '::1' || value === '::') return true;
|
||||
|
|
@ -126,7 +158,7 @@ function isPrivateIPv6Literal(value: string): boolean {
|
|||
// 4-in-6: ::ffff:A.B.C.D
|
||||
const mappedMatch = value.match(/^::ffff:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})$/);
|
||||
if (mappedMatch) return isPrivateIPv4Literal(mappedMatch[1]);
|
||||
return false;
|
||||
return hasPrivateEmbeddedIPv4Literal(value);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1711,6 +1743,7 @@ export enum SafeSearchTypes {
|
|||
}
|
||||
|
||||
export const webSearchSchema = z.object({
|
||||
allowedAddresses: allowedAddressesSchema,
|
||||
serperApiKey: z.string().optional().default('${SERPER_API_KEY}'),
|
||||
serperApiKeyPreview: apiKeyPreviewSchema,
|
||||
searxngInstanceUrl: z.string().optional().default('${SEARXNG_INSTANCE_URL}'),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue