From 5867f1a065c0c251361bd4ccd23ec545ce9863f9 Mon Sep 17 00:00:00 2001 From: Dustin Healy <54083382+dustinhealy@users.noreply.github.com> Date: Wed, 10 Jun 2026 06:03:05 -0700 Subject: [PATCH] =?UTF-8?q?=F0=9F=9B=A1=EF=B8=8F=20feat:=20Configurable=20?= =?UTF-8?q?Message=20PII=20Filter=20(#13602)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 🛡️ feat: Reject chat messages matching configured credential patterns Adds an opt-in `messagePiiFilter` middleware mounted on the agent chat route ahead of `moderateText`. When the configured patterns match the user's input the request is refused with 400, so the credential never reaches OpenAI moderation, the model, or MongoDB. Three starter patterns ship by default and operators can subset them or add their own regex via `customPatterns` in librechat.yaml. * 🧪 test: Memoize compiled patterns + add middleware spec Memoize the compiled pattern array via a WeakMap keyed by the messagePiiFilter config object so repeat requests against the same config skip the per-request RegExp construction. Cache entries are released automatically when the config object itself rotates. Adds packages/api/src/middleware/messagePiiFilter.spec.ts covering the default-starter rejections, the starterPatterns subset and empty-array semantics, customPatterns matching layered on top of and in place of the starters, the no-config and empty-text pass-through paths, and a memoization regression check. * 🛡️ fix: Skip invalid customPattern regexes instead of crashing the request Admin DB overrides for `messagePiiFilter.customPatterns` reach `req.config` via `mergeConfigOverrides`, which deep-merges raw override values without re-running `configSchema`. A typo'd regex like `(` would slip past the YAML-load validation and throw inside `new RegExp(...)` during `compile()`, returning 500 for every chat request until the operator rolled the override back. Wrapped the per-pattern compile in a try/catch that logs the invalid pattern id + reason and skips it, so other valid patterns (starters and other custom entries) keep filtering. Added a regression test alongside the existing spec. * 🛡️ feat: Extend PII filter to OpenAI-compatible and Responses agent APIs The chat-route middleware operates on `req.body.text`, but the remote agent API endpoints (`/api/agents/v1/chat/completions`, `/api/agents/v1/responses`) accept the same prompt content as a `messages` array or an `input` field. A caller using their API key could send a credential-shaped value through either route and bypass the configured PII filter even though they share the same agent and model backbone the middleware is meant to guard. Factored out `findPiiMatchInMessages`, a tolerant walker that handles both `content: string` and `content: ContentPart[]` user-message shapes against the same compiled, cached pattern list. Wired it into the OpenAI-compat controller after agent lookup and into the Responses controller right after `convertToInternalMessages`. Each returns the endpoint's native 400 error shape (`sendErrorResponse` / `sendResponsesErrorResponse`) with the `message_pii_filter_block` code when a user message matches. * 🩹 test: Add findPiiMatchInMessages to OpenAI + Responses controller mocks The OpenAI-compat and Responses controller specs mock `@librechat/api` with a hand-listed object. The new `findPiiMatchInMessages` export wired into both controllers in 3ea35af9a was missing from those mocks, so the production lookup returned undefined and the controllers threw at request time under jest. Added the missing entries (default mock: returns null so the handlers fall through to the existing happy paths). All 278 agents-controller tests pass locally. * 🧹 refactor: Namespace messagePiiFilter under messageFilter.pii + fix import order Renames the yaml field `messagePiiFilter` to `messageFilter.pii`, the module to `messageFilterPii`, the factory to `createMessageFilterPii`, the type to `MessageFilterPiiConfig`, and the error code to `message_filter_pii_block`. The wrapper `messageFilter` namespace gives future safety filters (e.g. `messageFilter.toxicity`) a place to plug in without restructuring the config later. The `findPiiMatchInMessages` helper kept its name because it already describes what it does at the value level. Also fixes import order Danny flagged on the OpenAI-compatible and Responses controllers: `findPiiMatchInMessages` was appended at the bottom of two `require('@librechat/api')` destructures rather than placed in the length-sorted slot the house style expects. * 🧹 chore: Length-sort the general require destructure in responses.js Reorders the general sub-group inside the `require('@librechat/api')` destructure shortest to longest so the whole block conforms to the length-sort rule the file's `// Responses API` sub-group already follows. Pure reorder, no other changes. * 🧹 chore: Length-sort the defaultConfig block in AppService Reorders the `defaultConfig` keys in `packages/data-schemas/src/app/service.ts` shortest-line to longest-line, with the explicit-value entries (`mcpConfig`, `fileStrategies`, `cloudfront`) trailing the shorthand ones. Pure reorder, no behavior change. --- .../agents/__tests__/openai.spec.js | 1 + .../agents/__tests__/responses.unit.spec.js | 1 + api/server/controllers/agents/openai.js | 12 + api/server/controllers/agents/responses.js | 22 +- api/server/routes/agents/chat.js | 3 +- librechat.example.yaml | 18 ++ packages/api/src/middleware/index.ts | 1 + .../src/middleware/messageFilterPii.spec.ts | 222 ++++++++++++++++++ .../api/src/middleware/messageFilterPii.ts | 145 ++++++++++++ packages/data-provider/src/config.ts | 33 +++ packages/data-schemas/src/app/service.ts | 6 +- packages/data-schemas/src/types/app.ts | 2 + 12 files changed, 458 insertions(+), 8 deletions(-) create mode 100644 packages/api/src/middleware/messageFilterPii.spec.ts create mode 100644 packages/api/src/middleware/messageFilterPii.ts diff --git a/api/server/controllers/agents/__tests__/openai.spec.js b/api/server/controllers/agents/__tests__/openai.spec.js index 8e65d6b10c..5b7576335a 100644 --- a/api/server/controllers/agents/__tests__/openai.spec.js +++ b/api/server/controllers/agents/__tests__/openai.spec.js @@ -132,6 +132,7 @@ jest.mock('@librechat/api', () => ({ resolveRecursionLimit: jest.fn().mockReturnValue(50), createToolExecuteHandler: jest.fn().mockReturnValue({ handle: jest.fn() }), isChatCompletionValidationFailure: jest.fn().mockReturnValue(false), + findPiiMatchInMessages: jest.fn().mockReturnValue(null), discoverConnectedAgents: jest.fn().mockResolvedValue({ agentConfigs: new Map(), edges: [], diff --git a/api/server/controllers/agents/__tests__/responses.unit.spec.js b/api/server/controllers/agents/__tests__/responses.unit.spec.js index f89de34d4d..1e55659106 100644 --- a/api/server/controllers/agents/__tests__/responses.unit.spec.js +++ b/api/server/controllers/agents/__tests__/responses.unit.spec.js @@ -122,6 +122,7 @@ jest.mock('@librechat/api', () => ({ buildResponse: jest.fn().mockReturnValue({ id: 'resp_123', output: [] }), generateResponseId: jest.fn().mockReturnValue('resp_mock-123'), isValidationFailure: jest.fn().mockReturnValue(false), + findPiiMatchInMessages: jest.fn().mockReturnValue(null), emitResponseCreated: jest.fn(), createResponseContext: jest.fn().mockReturnValue({ responseId: 'resp_123' }), createResponseTracker: jest.fn().mockReturnValue({ diff --git a/api/server/controllers/agents/openai.js b/api/server/controllers/agents/openai.js index 89422b2eaa..aedc700204 100644 --- a/api/server/controllers/agents/openai.js +++ b/api/server/controllers/agents/openai.js @@ -25,6 +25,7 @@ const { recordCollectedUsage, getTransactionsConfig, resolveRecursionLimit, + findPiiMatchInMessages, discoverConnectedAgents, getRemoteAgentPermissions, createToolExecuteHandler, @@ -177,6 +178,17 @@ const OpenAIChatCompletionController = async (req, res) => { ); } + const piiHit = findPiiMatchInMessages(request.messages, appConfig?.messageFilter?.pii); + if (piiHit != null) { + return sendErrorResponse( + res, + 400, + `Message contains a ${piiHit.label}. Remove it and try again.`, + 'invalid_request_error', + 'message_filter_pii_block', + ); + } + const responseId = `chatcmpl-${nanoid()}`; const created = Math.floor(Date.now() / 1000); diff --git a/api/server/controllers/agents/responses.js b/api/server/controllers/agents/responses.js index 9022c1e6f0..f2d8992a7f 100644 --- a/api/server/controllers/agents/responses.js +++ b/api/server/controllers/agents/responses.js @@ -12,18 +12,19 @@ const { const { createRun, buildToolSet, - loadSkillStates, - resolveAgentScopedSkillIds, createSafeUser, initializeAgent, + loadSkillStates, getBalanceConfig, + injectSkillPrimes, + extractManualSkills, recordCollectedUsage, getTransactionsConfig, - extractManualSkills, - injectSkillPrimes, - createToolExecuteHandler, + findPiiMatchInMessages, discoverConnectedAgents, + createToolExecuteHandler, getRemoteAgentPermissions, + resolveAgentScopedSkillIds, // Responses API writeDone, buildResponse, @@ -575,6 +576,17 @@ const createResponse = async (req, res) => { typeof request.input === 'string' ? request.input : request.input, ); + const piiHit = findPiiMatchInMessages(inputMessages, appConfig?.messageFilter?.pii); + if (piiHit != null) { + return sendResponsesErrorResponse( + res, + 400, + `Message contains a ${piiHit.label}. Remove it and try again.`, + 'invalid_request', + 'message_filter_pii_block', + ); + } + // Merge previous messages with new input const allMessages = [...previousMessages, ...inputMessages]; diff --git a/api/server/routes/agents/chat.js b/api/server/routes/agents/chat.js index 0543b0b1aa..8ffbde6552 100644 --- a/api/server/routes/agents/chat.js +++ b/api/server/routes/agents/chat.js @@ -1,5 +1,5 @@ const express = require('express'); -const { generateCheckAccess, skipAgentCheck } = require('@librechat/api'); +const { createMessageFilterPii, generateCheckAccess, skipAgentCheck } = require('@librechat/api'); const { PermissionTypes, Permissions, PermissionBits } = require('librechat-data-provider'); const { moderateText, @@ -25,6 +25,7 @@ const checkAgentResourceAccess = canAccessAgentFromBody({ requiredPermission: PermissionBits.VIEW, }); +router.use(createMessageFilterPii({ getConfig: (req) => req.config?.messageFilter?.pii })); router.use(moderateText); router.use(checkAgentAccess); router.use(checkAgentResourceAccess); diff --git a/librechat.example.yaml b/librechat.example.yaml index bd0385bd16..4ab2240ea1 100644 --- a/librechat.example.yaml +++ b/librechat.example.yaml @@ -781,3 +781,21 @@ endpoints: # # instructions: "You are a memory management assistant. Store and manage user information accurately." # # model_parameters: # # temperature: 0.1 + +# Reject chat messages whose text matches credential-shaped patterns +# before they reach moderation, the model, or persistence. Filter +# types live under `messageFilter.`; today only `pii` ships, but +# the namespace is structured so future filter types can plug in. +# Omit the whole section to disable. +# messageFilter: +# pii: +# # (optional) Pick a subset of the starter catalog by id; omit to +# # enable all starters (sk_prefix, bearer_header, api_key_header). +# starterPatterns: [sk_prefix, bearer_header, api_key_header] +# # (optional) Operator-defined patterns. Each entry needs id, +# # label, and a JavaScript-flavor regex; the regex is validated +# # at config load time. +# customPatterns: +# - id: anthropic_api_key +# label: Anthropic API key +# regex: "sk-ant-[A-Za-z0-9_-]{20,}" diff --git a/packages/api/src/middleware/index.ts b/packages/api/src/middleware/index.ts index 6ac95683fe..fbefaac54b 100644 --- a/packages/api/src/middleware/index.ts +++ b/packages/api/src/middleware/index.ts @@ -16,3 +16,4 @@ export * from './concurrency'; export * from './checkBalance'; export * from './remoteAgentAuth'; export * from './share'; +export * from './messageFilterPii'; diff --git a/packages/api/src/middleware/messageFilterPii.spec.ts b/packages/api/src/middleware/messageFilterPii.spec.ts new file mode 100644 index 0000000000..7f9f8e049e --- /dev/null +++ b/packages/api/src/middleware/messageFilterPii.spec.ts @@ -0,0 +1,222 @@ +import type { MessageFilterPiiConfig } from 'librechat-data-provider'; +import type { Request, Response, NextFunction } from 'express'; + +jest.mock('@librechat/data-schemas', () => ({ + logger: { warn: jest.fn(), error: jest.fn(), info: jest.fn(), debug: jest.fn() }, +})); +import { createMessageFilterPii, findPiiMatchInMessages } from './messageFilterPii'; + +type CapturedResponse = { status?: number; body?: unknown }; + +function runMiddleware( + config: MessageFilterPiiConfig | undefined, + body: unknown, +): { capturedRes: CapturedResponse; nextCalls: number } { + const captured: CapturedResponse = {}; + let nextCalls = 0; + const mw = createMessageFilterPii({ getConfig: () => config }); + const req = { body } as unknown as Request; + const res = { + status(code: number) { + captured.status = code; + return this; + }, + json(payload: unknown) { + captured.body = payload; + return this; + }, + } as unknown as Response; + const next: NextFunction = () => { + nextCalls++; + }; + mw(req, res, next); + return { capturedRes: captured, nextCalls }; +} + +describe('messageFilterPii middleware', () => { + it('passes through when no config is provided', () => { + const { capturedRes, nextCalls } = runMiddleware(undefined, { + text: 'my key is sk-proj-FAKE1234567890ABCDEF', + }); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('passes through when req.body.text is missing', () => { + const { capturedRes, nextCalls } = runMiddleware({}, {}); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('passes through when req.body.text is the empty string', () => { + const { capturedRes, nextCalls } = runMiddleware({}, { text: '' }); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('passes through plain text that matches no pattern', () => { + const { capturedRes, nextCalls } = runMiddleware({}, { text: 'hello world' }); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('rejects with 400 when an sk- token is present (default starters)', () => { + const { capturedRes, nextCalls } = runMiddleware( + {}, + { text: 'my key is sk-proj-FAKE1234567890ABCDEF please' }, + ); + expect(nextCalls).toBe(0); + expect(capturedRes.status).toBe(400); + expect(capturedRes.body).toEqual({ + error: 'message_filter_pii_block', + message: 'Message contains a sk- prefix token. Remove it and try again.', + }); + }); + + it('rejects with 400 when a Bearer header is present', () => { + const { capturedRes, nextCalls } = runMiddleware( + {}, + { text: 'Authorization: Bearer eyJabc.def-ghi' }, + ); + expect(nextCalls).toBe(0); + expect(capturedRes.status).toBe(400); + expect(capturedRes.body).toMatchObject({ error: 'message_filter_pii_block' }); + }); + + it('rejects with 400 when an api-key header is present', () => { + const { capturedRes, nextCalls } = runMiddleware({}, { text: 'api-key: foo123bar' }); + expect(nextCalls).toBe(0); + expect(capturedRes.status).toBe(400); + }); + + it('honors a starterPatterns subset (sk passes when only bearer is enabled)', () => { + const { capturedRes, nextCalls } = runMiddleware( + { starterPatterns: ['bearer_header'] }, + { text: 'my key is sk-proj-FAKE1234567890ABCDEF please' }, + ); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('treats starterPatterns: [] as disabling all starters', () => { + const { capturedRes, nextCalls } = runMiddleware( + { starterPatterns: [] }, + { text: 'my key is sk-proj-FAKE1234567890ABCDEF please' }, + ); + expect(nextCalls).toBe(1); + expect(capturedRes.status).toBeUndefined(); + }); + + it('rejects on a customPatterns match with the operator-supplied label', () => { + const { capturedRes, nextCalls } = runMiddleware( + { + starterPatterns: [], + customPatterns: [{ id: 'org', label: 'Org token', regex: '\\bORG-[A-Z0-9]{6,}' }], + }, + { text: 'token ORG-DEADBEEF here' }, + ); + expect(nextCalls).toBe(0); + expect(capturedRes.status).toBe(400); + expect(capturedRes.body).toEqual({ + error: 'message_filter_pii_block', + message: 'Message contains a Org token. Remove it and try again.', + }); + }); + + it('layers customPatterns on top of starters', () => { + const { capturedRes, nextCalls } = runMiddleware( + { + customPatterns: [{ id: 'org', label: 'Org token', regex: '\\bORG-[A-Z0-9]{6,}' }], + }, + { text: 'token ORG-DEADBEEF here' }, + ); + expect(nextCalls).toBe(0); + expect(capturedRes.status).toBe(400); + }); + + it('returns the same compiled pattern array for repeat calls with the same config (memoization)', () => { + const config: MessageFilterPiiConfig = { + customPatterns: [{ id: 'org', label: 'Org token', regex: '\\bORG-[A-Z0-9]{6,}' }], + }; + const a = runMiddleware(config, { text: 'plain' }); + const b = runMiddleware(config, { text: 'plain' }); + expect(a.nextCalls).toBe(1); + expect(b.nextCalls).toBe(1); + }); + + it('drops an invalid customPattern regex without throwing and keeps other patterns active', () => { + const config = { + starterPatterns: [], + customPatterns: [ + { id: 'broken', label: 'Broken', regex: '(' }, + { id: 'org', label: 'Org token', regex: '\\bORG-[A-Z0-9]{6,}' }, + ], + } as unknown as MessageFilterPiiConfig; + const benign = runMiddleware(config, { text: 'plain text' }); + expect(benign.nextCalls).toBe(1); + expect(benign.capturedRes.status).toBeUndefined(); + const matching = runMiddleware(config, { text: 'token ORG-DEADBEEF here' }); + expect(matching.nextCalls).toBe(0); + expect(matching.capturedRes.status).toBe(400); + }); +}); + +describe('findPiiMatchInMessages', () => { + it('returns null for missing or empty messages', () => { + expect(findPiiMatchInMessages(undefined, {})).toBeNull(); + expect(findPiiMatchInMessages([], {})).toBeNull(); + }); + + it('returns null when no config is provided', () => { + expect( + findPiiMatchInMessages([{ role: 'user', content: 'sk-proj-FAKE123' }], undefined), + ).toBeNull(); + }); + + it('skips non-user messages', () => { + const hit = findPiiMatchInMessages( + [ + { role: 'system', content: 'sk-proj-FAKE1234567890ABCDEF' }, + { role: 'assistant', content: 'sk-proj-FAKE1234567890ABCDEF' }, + ], + {}, + ); + expect(hit).toBeNull(); + }); + + it('matches a string-content user message', () => { + const hit = findPiiMatchInMessages( + [{ role: 'user', content: 'my key is sk-proj-FAKE1234567890ABCDEF' }], + {}, + ); + expect(hit).toEqual({ id: 'sk_prefix', label: 'sk- prefix token' }); + }); + + it('matches a content-parts user message (text part)', () => { + const hit = findPiiMatchInMessages( + [ + { + role: 'user', + content: [ + { type: 'image_url', image_url: { url: 'data:...' } }, + { type: 'text', text: 'Authorization: Bearer abc.def.ghi' }, + ], + }, + ], + {}, + ); + expect(hit).toEqual({ id: 'bearer_header', label: 'Bearer token' }); + }); + + it('returns null when no user message matches', () => { + expect(findPiiMatchInMessages([{ role: 'user', content: 'hello world' }], {})).toBeNull(); + }); + + it('honors customPatterns from config', () => { + const hit = findPiiMatchInMessages([{ role: 'user', content: 'token ORG-DEADBEEF here' }], { + starterPatterns: [], + customPatterns: [{ id: 'org', label: 'Org token', regex: '\\bORG-[A-Z0-9]{6,}' }], + }); + expect(hit).toEqual({ id: 'org', label: 'Org token' }); + }); +}); diff --git a/packages/api/src/middleware/messageFilterPii.ts b/packages/api/src/middleware/messageFilterPii.ts new file mode 100644 index 0000000000..aa09caac5b --- /dev/null +++ b/packages/api/src/middleware/messageFilterPii.ts @@ -0,0 +1,145 @@ +import { logger } from '@librechat/data-schemas'; +import type { + NextFunction, + RequestHandler, + Request as ServerRequest, + Response as ServerResponse, +} from 'express'; +import type { MessageFilterPiiConfig } from 'librechat-data-provider'; + +type CompiledPattern = { id: string; label: string; pattern: RegExp }; + +const STARTER_PATTERNS: CompiledPattern[] = [ + { id: 'sk_prefix', label: 'sk- prefix token', pattern: /\b(sk-)[a-zA-Z0-9_-]+/g }, + { id: 'bearer_header', label: 'Bearer token', pattern: /\b(Bearer )[^\s"']+/gi }, + { id: 'api_key_header', label: 'api-key header', pattern: /\b(api-key:?\s+)[^\s"']+/gi }, +]; + +const STARTER_BY_ID = new Map(STARTER_PATTERNS.map((p) => [p.id, p])); + +function selectStarter(ids?: string[]): CompiledPattern[] { + if (ids == null) { + return STARTER_PATTERNS; + } + const out: CompiledPattern[] = []; + for (const id of ids) { + const entry = STARTER_BY_ID.get(id); + if (entry != null) { + out.push(entry); + } + } + return out; +} + +const COMPILE_CACHE = new WeakMap(); + +function compile(config: MessageFilterPiiConfig): CompiledPattern[] { + const cached = COMPILE_CACHE.get(config); + if (cached != null) { + return cached; + } + const starter = selectStarter(config.starterPatterns); + const custom: CompiledPattern[] = []; + for (const p of config.customPatterns ?? []) { + try { + custom.push({ id: p.id, label: p.label, pattern: new RegExp(p.regex, 'g') }); + } catch (err) { + logger.warn( + `[messageFilter.pii] dropping invalid customPattern ${JSON.stringify(p.id)}: ${(err as Error).message}`, + ); + } + } + const result = [...starter, ...custom]; + COMPILE_CACHE.set(config, result); + return result; +} + +function findMatch(text: string, patterns: CompiledPattern[]): CompiledPattern | null { + for (const p of patterns) { + p.pattern.lastIndex = 0; + if (p.pattern.test(text)) { + return p; + } + } + return null; +} + +export interface PiiMatch { + id: string; + label: string; +} + +type ContentPart = { type?: string; text?: string; [key: string]: unknown }; +type ChatLikeMessage = { + role?: string; + content?: string | ContentPart[]; +}; + +export function findPiiMatchInMessages( + messages: ChatLikeMessage[] | undefined, + config: MessageFilterPiiConfig | undefined, +): PiiMatch | null { + if (config == null || !Array.isArray(messages) || messages.length === 0) { + return null; + } + const patterns = compile(config); + if (patterns.length === 0) { + return null; + } + for (const msg of messages) { + if (msg == null || msg.role !== 'user') { + continue; + } + if (typeof msg.content === 'string') { + const hit = findMatch(msg.content, patterns); + if (hit != null) { + return { id: hit.id, label: hit.label }; + } + continue; + } + if (Array.isArray(msg.content)) { + for (const part of msg.content) { + if (part != null && typeof part.text === 'string') { + const hit = findMatch(part.text, patterns); + if (hit != null) { + return { id: hit.id, label: hit.label }; + } + } + } + } + } + return null; +} + +export interface CreateMessageFilterPiiOptions { + getConfig: (req: ServerRequest) => MessageFilterPiiConfig | undefined; +} + +export function createMessageFilterPii(options: CreateMessageFilterPiiOptions): RequestHandler { + return function messageFilterPii(req: ServerRequest, res: ServerResponse, next: NextFunction) { + const config = options.getConfig(req); + if (config == null) { + next(); + return; + } + const text = req.body?.text; + if (typeof text !== 'string' || text.length === 0) { + next(); + return; + } + const patterns = compile(config); + if (patterns.length === 0) { + next(); + return; + } + const match = findMatch(text, patterns); + if (match == null) { + next(); + return; + } + res.status(400).json({ + error: 'message_filter_pii_block', + message: `Message contains a ${match.label}. Remove it and try again.`, + }); + }; +} diff --git a/packages/data-provider/src/config.ts b/packages/data-provider/src/config.ts index 23d12c14d9..a431031050 100644 --- a/packages/data-provider/src/config.ts +++ b/packages/data-provider/src/config.ts @@ -1423,6 +1423,38 @@ export type SummarizationConfig = z.infer; const customEndpointsSchema = z.array(endpointSchema.partial()).optional(); +const messageFilterPiiCustomPatternSchema = z.object({ + id: z.string().min(1), + label: z.string().min(1), + regex: z + .string() + .min(1) + .refine( + (value) => { + try { + new RegExp(value, 'g'); + return true; + } catch { + return false; + } + }, + { message: 'Invalid regex' }, + ), +}); + +export const messageFilterPiiSchema = z.object({ + starterPatterns: z.array(z.string()).optional(), + customPatterns: z.array(messageFilterPiiCustomPatternSchema).optional(), +}); + +export type MessageFilterPiiConfig = z.infer; + +export const messageFilterSchema = z.object({ + pii: messageFilterPiiSchema.optional(), +}); + +export type MessageFilterConfig = z.infer; + export const configSchema = z.object({ version: z.string(), cache: z.boolean().default(true), @@ -1470,6 +1502,7 @@ export const configSchema = z.object({ rateLimits: rateLimitSchema.optional(), fileConfig: fileConfigSchema.optional(), modelSpecs: specsConfigSchema.optional(), + messageFilter: messageFilterSchema.optional(), endpoints: z .object({ allowedAddresses: allowedAddressesSchema, diff --git a/packages/data-schemas/src/app/service.ts b/packages/data-schemas/src/app/service.ts index 19fec9f5a1..4d41a824f4 100644 --- a/packages/data-schemas/src/app/service.ts +++ b/packages/data-schemas/src/app/service.ts @@ -110,6 +110,7 @@ export const AppService = async (params?: { const interfaceConfig = await loadDefaultInterface({ config, configDefaults }); const turnstileConfig = loadTurnstileConfig(config, configDefaults); const speech = config.speech; + const messageFilter = config.messageFilter; const defaultConfig = { ocr, @@ -117,15 +118,16 @@ export const AppService = async (params?: { config, memory, speech, - balance, actions, + balance, webSearch, mcpSettings, - transactions, fileStrategy, registration, + transactions, filteredTools, includedTools, + messageFilter, summarization, availableTools, imageOutputType, diff --git a/packages/data-schemas/src/types/app.ts b/packages/data-schemas/src/types/app.ts index 4562e588ee..2b47e6f011 100644 --- a/packages/data-schemas/src/types/app.ts +++ b/packages/data-schemas/src/types/app.ts @@ -62,6 +62,8 @@ export interface AppConfig { summarization?: SummarizationConfig; /** Web search configuration */ webSearch?: TCustomConfig['webSearch']; + /** Message filter configuration (PII and future filter types) */ + messageFilter?: TCustomConfig['messageFilter']; /** File storage strategy ('local', 's3', 'firebase', 'azure_blob', 'cloudfront') */ fileStrategy: FileStorage; /** File strategies configuration */