🧑‍⚖️ feat: Preserve Input Policy and Persistence Admission for Remote Models (#18271)

This commit is contained in:
Danny Avila 2026-11-18 06:16:26 -05:00 • committed by GitHub
parent dc07a93343
commit 4cd70c5af9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
20 changed files with 1270 additions and 143 deletions

View file

@ -109,8 +109,7 @@ factory must return an actual `@librechat/agents` Run with the supplied run and
and preserve supplied handlers, hooks, checkpointer, and cancellation signal. Workers own their
installed tools and custom graph. Model configuration and credentials remain worker-owned unless the operator enables saved-agent host models or the caller supplies the explicit host model binding below. Host model callbacks,
provider clients, native tool registries, and arbitrary RunConfig closures do not cross the wire.
Remote execution therefore fails closed on deployments with active model-bound content/PII
protection until its saved-agent adapter is wired. Explicit programmatic provider-text and model bindings are described below.
Protected chat requires the opt-in host model and the host-native model-bound callback. Callers without that callback still fail closed. Explicit programmatic provider-text and model bindings are described below.
Stream circuit-breaker limits are forwarded to the worker.
Host policy hooks still apply, and policy transport failures stop execution. A trusted runtime is
executable operator code, not a sandbox for untrusted agent authors.
@ -154,27 +153,27 @@ acceptance target remains the main chat's functionality and experience, with mea
Using native events and the native renderer proves only the behaviors tested below. Missing
adapters and unverified behaviors remain release gates for products that depend on them.
| Main-chat capability | This bridge | Remaining work / verification |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
| Model/PII/provider-text protection | Explicit SDK host-service adapter supports protected provider text; saved-agent model/PII guard remains | Builder models, input/model callbacks and product policy wiring remain required before protected product rollout. |
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
| Main-chat capability | This bridge | Remaining work / verification |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
| Model/PII/provider-text protection | Host-model chat supports input/PII callbacks and initial persistence admission; explicit provider-text transport remains available | Combining initial admission with provider-text output protection requires SDK invocation-context support. Protected Responses/OpenAI callers without native callbacks remain gated. |
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
Do not enable a workflow on the assumption that a native chat control is honored remotely. Its
required rows must be implemented and verified first. Keep this matrix current as adapters land;
@ -263,13 +262,11 @@ Both sides require SDK 4.2.7 and the new bridge's `providerText: 1` capability.
Protected runs fail before factory admission on older workers. The worker receives
`config.providerTextProtection`; its factory must retain that exact policy on its
returned SDK graph. Omitting it fails before model execution. Custom graphs remain
trusted code and must install it on every applicable SDK Run. Main-chat's model/PII
rejection in saved-agent dispatch stays in place: this output-text adapter does not
trusted code and must install it on every applicable SDK Run. This output-text adapter alone does not
supply input/model callbacks, authoritative builder model selection, host model
credentials, or policy for model calls made outside the SDK. SDK 4.2.7 also provides
`HostChatModel` and `createHostModelService` for host-owned model execution; those
contracts now have the explicit authenticated transport below; saved-agent integration and
product policy verification are still required before this guard can be removed.
contracts have the authenticated transport below. The input-admission adapter and this output-protection adapter currently cannot be combined, as explained below.
The worker buffers a complete model attempt through the same SDK release boundary
as local execution, then sends one `policy.providerText` RPC. Only canonical approved
@ -364,6 +361,7 @@ endpoints:
agents:
sdkHostModels:
maxConcurrentCalls: 32
maxAdmissionEntries: 4096
maxBufferedRequestBytes: 67108864
timeoutMs: 60000
maxRequestBytes: 4194304
@ -393,10 +391,38 @@ is required to change the process-wide budgets. Existing transport frame and buf
apply; the model byte budgets do not enlarge them. Embedders using `createRun` or
`createAgentChatCompletion` inject the same long-lived `sdkModelHost` dependency into every caller.
This selects the primary model; it does not establish full main-chat parity. Active model/PII
protection still fails closed because root admission callbacks and deferred persistence need an
explicit remote lifecycle adapter. Local executable/graph tools and configured provider fallbacks
also fail closed in this mode rather than being silently discarded. Context pruning, summary and
compaction policy, child/multi-model routing, files, accounting and tracing remain separate parity
work. Intrinsic callback execution is covered; production billing and trace ownership are not yet
accepted end to end.
### Initial model admission and protected chat
Host-model chat now retains the native model-bound input/PII callback and the existing initial
persistence barrier. Ordinary deferred user messages become eligible for persistence only after a
successful host-native attempt and completion of its starting graph node. Captured private text
still commits before the provider begins; a rejected input or failed private-text commit prevents
the provider call. Policy errors stay on the host and reach the existing local error handler; the
worker sees only a sanitized model error. A policy rejection cannot be cleared by a worker retry.
The worker correlates SDK model callbacks from the starting root with encrypted model requests.
Node completion carries IDs and a boolean, not messages or policy details. The host verifies that
it observed a successful native attempt before accepting completion. After the host acknowledges
admission, both sides discard correlation state and stop these lifecycle RPCs for the remainder
of the turn. `maxAdmissionEntries` bounds the initial callback/attempt tracking per execution;
this is distinct from the process-wide provider concurrency budget. Cancellation and connection
failure close admission. Live connection recovery retains the same barrier and does not repeat
provider calls or persistence.
Workers must advertise `modelAdmission: 1` when an initial barrier is present; older workers fail
before graph execution. No additional browser credentials or policy callbacks cross the wire.
The existing guard stays active unless the caller supplies the native model-bound callback and
the registration uses the injected host model. Thus protected chat can opt in, while protected
Responses/OpenAI-compatible callers without that callback remain gated. Local graph tools and
provider fallbacks retain their earlier explicit rejection.
**Remaining limitation:** the SDK provider-text output-protection wrapper deliberately withholds
raw callback managers from its underlying model. Until the SDK preserves a separate invocation
identity, combining that wrapper with initial remote admission is rejected on both sides before
graph execution. Existing programmatic output protection without an initial barrier continues to
work. Do not disable output protection to enable this combination.
Before initial admission, child/summary model calls without root lineage and concurrent root
attempts fail closed; parallel/composed admission needs its own ownership contract. This does not
establish full main-chat parity. Context pruning, summary/compaction configuration, child/multi-model
routing, files, production billing/tracing and Console UI acceptance remain separate work.

View file

@ -169,6 +169,8 @@ Enable `endpoints.agents.sdkHostModels: {}` and `remote.hostModel: true` on the
runtime registration to select the saved agent's primary model and parameters. Both examples
consume `input.agent.model` and retain `input.hostModel`; model credentials stay in LibreChat.
The custom graph keeps its plan/review/refine nodes and uses the host-authorized tool schemas.
This mode requires updated workers with `builderModel: 1`. It currently rejects active model/PII
protection, local graph tools and provider fallbacks; see [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
This mode requires updated workers with `builderModel: 1` and, for initial persistence admission,
`modelAdmission: 1`. Protected chat uses native host input callbacks. Protected API callers without
those callbacks, local graph tools, provider fallbacks, and the combination of initial admission
with provider-text output protection remain gated; see [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
and [configuration](../../docs/sdk/agents.md#saved-agent-builder-models) before enabling it.

View file

@ -820,9 +820,11 @@ endpoints:
# agents:
# # Optional primary-model execution for remote SDK agents. Budgets are per process.
# # Restart to change budgets; workers must support builderModel: 1.
# # Model/PII protection, local graph tools and provider fallbacks remain gated.
# # Initial protected-chat admission also requires modelAdmission: 1 on workers.
# # Local graph tools, provider fallbacks and combined output protection remain gated.
# sdkHostModels:
# maxConcurrentCalls: 32
# maxAdmissionEntries: 4096
# maxBufferedRequestBytes: 67108864
# timeoutMs: 60000
# maxRequestBytes: 4194304

View file

@ -0,0 +1,214 @@
import { randomUUID } from 'node:crypto';
import { CallbackManager, ensureHandler } from '@langchain/core/callbacks/manager';
import {
createInitialModelBoundAdmissionCallback,
createModelBoundChatModelCallback,
} from '../../middleware/modelBoundContent';
import { createHostModelLifecycle, createWorkerModelLifecycle } from './admission';
function fixture() {
const persist = jest.fn();
const initial = createInitialModelBoundAdmissionCallback({
agentIds: ['root'],
isActive: () => true,
onAllowed: persist,
});
const host = createHostModelLifecycle('root', 16);
host.attach([initial]);
const callback = host.wrapCallbacks([]);
if (!Array.isArray(callback)) throw new Error('Expected callbacks');
return { persist, host, observer: callback[0] };
}
const attempt = () => ({ nodeId: randomUUID(), modelId: randomUUID() });
it('requires both an observed native success and graph-node completion before persistence', async () => {
const { host, persist, observer } = fixture();
const scope = attempt();
await host.invoke(scope, async () => {
await observer.handleChatModelStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
[],
randomUUID(),
);
expect(persist).not.toHaveBeenCalled();
});
expect(persist).not.toHaveBeenCalled();
host.end({ nodeId: scope.nodeId, completed: true });
expect(persist).toHaveBeenCalledTimes(1);
await expect(host.invoke(undefined, async () => 'subsequent call')).resolves.toBe(
'subsequent call',
);
expect(() => host.end({ nodeId: scope.nodeId, completed: true })).toThrow();
});
it.each(['unobserved', 'failed', 'cancelled', 'incomplete'] as const)(
'never admits an %s attempt',
async (outcome) => {
const { host, persist, observer } = fixture();
const scope = attempt();
try {
await host.invoke(scope, async () => {
if (outcome !== 'unobserved')
await observer.handleChatModelStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
[],
randomUUID(),
);
if (outcome === 'failed') throw new Error('provider failure');
if (outcome === 'cancelled') host.dispose();
});
} catch {
/* expected provider failure */
}
if (outcome !== 'cancelled')
host.end({ nodeId: scope.nodeId, completed: outcome !== 'incomplete' });
expect(persist).not.toHaveBeenCalled();
},
);
it('rejects completion while provider work is pending, missing lineage and replayed attempts', async () => {
const { host, persist } = fixture();
const scope = attempt();
await host.invoke(scope, async () => {
expect(() => host.end({ nodeId: scope.nodeId, completed: true })).toThrow();
});
await expect(host.invoke(scope, async () => {})).rejects.toThrow();
await expect(host.invoke(undefined, async () => {})).rejects.toThrow();
expect(persist).not.toHaveBeenCalled();
});
it('preserves exact messages while supplying only host-derived admission metadata to policy hooks', async () => {
const { host } = fixture();
const allowed = jest.fn();
const filter = createModelBoundChatModelCallback({}, { onContentAllowed: allowed });
const manager = new CallbackManager();
const original = ensureHandler({
name: filter.name,
raiseError: true,
awaitHandlers: true,
handleChatModelStart: (...args) =>
filter.handleChatModelStart(undefined, [], args[2], args[3], args[4], args[5], args[6]),
});
manager.addHandler(original, true);
const wrapped = host.wrapCallbacks(manager);
if (Array.isArray(wrapped)) throw new Error('Expected callback manager');
const policy = wrapped.handlers.find((callback) => callback.name === filter.name)!;
const scope = attempt();
const modelId = randomUUID();
await host.invoke(scope, async () => {
await policy.handleChatModelStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
[],
modelId,
'untrusted-parent',
undefined,
undefined,
{ agentId: 'untrusted' },
);
});
expect(allowed).toHaveBeenCalledWith({
runId: modelId,
parentRunId: scope.nodeId,
metadata: { agentId: 'root', langgraph_node: 'agent=root' },
});
expect(manager.handlers[0]).toBe(original);
});
it('bounds admission state across many completed graph nodes', async () => {
const { host } = fixture();
for (let index = 0; index < 16; index++) {
const scope = attempt();
await host.invoke(scope, async () => {});
host.end({ nodeId: scope.nodeId, completed: false });
}
await expect(host.invoke(attempt(), async () => {})).rejects.toThrow();
});
it('latches policy rejection even if the worker retries with safe input', async () => {
const { host, persist } = fixture();
const filter = createModelBoundChatModelCallback({
filters: {
messages: {
pii: {
fields: ['text'],
starterPatterns: [],
customPatterns: [{ id: 'private', label: 'Private', regex: 'PRIVATE-VALUE' }],
},
},
},
});
const policy = ensureHandler({
name: filter.name,
handleChatModelStart: (_model, _messages, runId, parent, extra, tags, metadata) =>
filter.handleChatModelStart(
undefined,
[[{ role: 'human', content: 'PRIVATE-VALUE' }]],
runId,
parent,
extra,
tags,
metadata,
),
});
const callbacks = host.wrapCallbacks([policy]);
if (!Array.isArray(callbacks)) throw new Error('Expected callbacks');
const first = attempt();
await expect(
host.invoke(first, async () =>
callbacks[0].handleChatModelStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
[],
randomUUID(),
),
),
).rejects.toMatchObject({ code: 'content_filter_block' });
const invoke = jest.fn(async () => {});
await expect(host.invoke(attempt(), invoke)).rejects.toMatchObject({
code: 'content_filter_block',
});
expect(invoke).not.toHaveBeenCalled();
expect(() => host.end({ nodeId: first.nodeId, completed: true })).toThrow();
expect(host.failure()).toMatchObject({ code: 'content_filter_block' });
expect(persist).not.toHaveBeenCalled();
});
it('rejects concurrent root attempts instead of treating a sibling as the starting root', async () => {
const { host } = fixture();
await host.invoke(attempt(), async () => {
await expect(host.invoke(attempt(), async () => {})).rejects.toThrow();
});
});
it('does not enroll summary or unrelated models into the starting graph admission', async () => {
const call = jest.fn();
const worker = createWorkerModelLifecycle('root', 16, { call });
const nodeId = randomUUID();
const callback = worker.callback;
await callback.handleChainStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
{},
nodeId,
undefined,
undefined,
{ langgraph_node: 'agent=root' },
undefined,
'agent=root',
);
for (const metadata of [
{ agentId: 'other', langgraph_node: 'agent=root' },
{ agentId: 'root', langgraph_node: 'agent=root', summarization: true },
]) {
await callback.handleChatModelStart?.(
{ id: [], lc: 1, type: 'not_implemented' },
[],
randomUUID(),
nodeId,
undefined,
undefined,
metadata,
);
}
await callback.handleChainEnd?.({ messages: [{}] }, nodeId);
expect(call).not.toHaveBeenCalled();
expect(() => worker.current()).toThrow();
});

View file

@ -0,0 +1,367 @@
import { z } from 'zod';
import { AsyncLocalStorage } from 'node:async_hooks';
import { ensureHandler } from '@langchain/core/callbacks/manager';
import { HostChatModel, HostModelServiceError } from '@librechat/agents';
import type { BaseCallbackHandler, CallbackHandlerMethods } from '@langchain/core/callbacks/base';
import type { CallbackManagerForLLMRun, Callbacks } from '@langchain/core/callbacks/manager';
import type { ChatGenerationChunk } from '@langchain/core/outputs';
import type { BaseMessage } from '@langchain/core/messages';
import type { InitialModelBoundAdmissionCallback } from '../../middleware/modelBoundContent';
import type { SdkRpc } from './peer';
import { isContentFilterError } from '../../middleware/contentFilter';
export interface RemoteModelAttempt {
nodeId: string;
modelId: string;
}
export const modelAttemptSchema: z.ZodType<RemoteModelAttempt> = z
.object({ nodeId: z.string().uuid(), modelId: z.string().uuid() })
.strict();
const nodeEndSchema = z.object({ nodeId: z.string().uuid(), completed: z.boolean() }).strict();
const blocked = () => new HostModelServiceError('blocked');
/** Run each iterator operation inside its scope; constructing an async generator does not enter it. */
async function* scopedStream<T, C>(
storage: AsyncLocalStorage<C>,
context: C,
source: () => AsyncIterable<T>,
): AsyncGenerator<T> {
const iterator = storage.run(context, () => source()[Symbol.asyncIterator]());
try {
for (;;) {
const next = await storage.run(context, () => iterator.next());
if (next.done) return;
yield next.value;
}
} finally {
await storage.run(context, () => iterator.return?.());
}
}
type Attempt = { nodeId: string; nativeIds: Set<string> };
export interface HostModelLifecycle {
readonly maxEntries: number;
attach(callbacks: Callbacks | readonly InitialModelBoundAdmissionCallback[] | undefined): boolean;
failure(): Error | undefined;
wrapCallbacks(callbacks: Callbacks | undefined): Callbacks;
invoke<T>(context: RemoteModelAttempt | undefined, operation: () => Promise<T>): Promise<T>;
stream<T>(
context: RemoteModelAttempt | undefined,
operation: () => AsyncIterable<T>,
): AsyncIterable<T>;
end(data: unknown): boolean;
dispose(): void;
}
/** Reuse the host's admission callbacks; worker completion cannot substitute for a successful native call. */
export function createHostModelLifecycle(agentId: string, maxEntries: number): HostModelLifecycle {
if (!Number.isSafeInteger(maxEntries) || maxEntries < 16 || maxEntries > 65536) throw blocked();
const storage = new AsyncLocalStorage<Attempt>();
const nodes = new Map<string, { pending: number; successful: boolean }>();
const usedNodes = new Set<string>();
const usedModels = new Set<string>();
let initial: InitialModelBoundAdmissionCallback | undefined;
let attached = false;
let disposed = false;
let policyFailure: Error | undefined;
let admitted = false;
const metadata = { agentId, langgraph_node: `agent=${agentId}` };
const begin = (context: RemoteModelAttempt | undefined): Attempt | undefined => {
if (disposed || policyFailure) throw policyFailure ?? blocked();
if (!initial || admitted) return undefined;
if (!context) throw blocked();
const { nodeId, modelId } = modelAttemptSchema.parse(context);
if (usedModels.has(modelId) || usedModels.size >= maxEntries) throw blocked();
let node = nodes.get(nodeId);
// This bridge admits one starting root. Parallel/child composition needs its own ownership contract.
if ((nodes.size > 0 && !node) || (node && node.pending > 0)) throw blocked();
if (!node) {
if (usedNodes.has(nodeId) || usedNodes.size >= maxEntries) throw blocked();
usedNodes.add(nodeId);
node = { pending: 0, successful: false };
nodes.set(nodeId, node);
initial.handleChainStart(
undefined,
{},
nodeId,
undefined,
undefined,
metadata,
undefined,
metadata.langgraph_node,
);
}
usedModels.add(modelId);
node.pending++;
return { nodeId, nativeIds: new Set() };
};
const finish = (attempt: Attempt | undefined, successful: boolean) => {
if (!attempt || disposed) return;
const node = nodes.get(attempt.nodeId);
if (!node) throw blocked();
node.pending--;
node.successful ||= successful && attempt.nativeIds.size > 0;
for (const id of attempt.nativeIds) {
if (successful) initial?.handleLLMEnd({}, id);
else initial?.handleLLMError(blocked(), id);
}
};
const translate: NonNullable<CallbackHandlerMethods['handleChatModelStart']> = (...args) => {
const context = storage.getStore();
if (!initial || admitted) return;
if (!context || disposed) throw blocked();
const runId = args[2];
if (!context.nativeIds.has(runId) && context.nativeIds.size >= maxEntries) throw blocked();
context.nativeIds.add(runId);
initial.handleChatModelStart(
undefined,
[],
runId,
context.nodeId,
undefined,
undefined,
metadata,
);
};
const wrap = (handler: BaseCallbackHandler | CallbackHandlerMethods) => {
if (!('name' in handler) || handler.name !== 'librechat-model-bound-content-filter')
return handler;
return ensureHandler({
name: handler.name,
raiseError: true,
awaitHandlers: true,
handleChatModelStart: async (...args) => {
try {
if (disposed || policyFailure) throw policyFailure ?? blocked();
if (!initial || admitted) return await handler.handleChatModelStart?.(...args);
translate(...args);
const context = storage.getStore();
if (!context) throw blocked();
return await handler.handleChatModelStart?.(
args[0],
args[1],
args[2],
context.nodeId,
args[4],
args[5],
metadata,
);
} catch (error) {
if (isContentFilterError(error)) policyFailure ??= error;
throw error;
}
},
});
};
return {
maxEntries,
failure: () => policyFailure,
attach(callbacks) {
if (attached || disposed) throw blocked();
attached = true;
const matches = (
callbacks && 'handlers' in callbacks ? callbacks.handlers : (callbacks ?? [])
).filter(
(handler) =>
'name' in handler && handler.name === 'librechat-initial-model-bound-admission',
);
if (matches.length > 1) throw blocked();
const candidate = matches[0];
if (candidate) {
for (const name of [
'handleChainStart',
'handleChatModelStart',
'handleLLMEnd',
'handleLLMError',
'handleChainEnd',
'handleChainError',
] as const)
if (typeof candidate[name] !== 'function') throw blocked();
initial = candidate as InitialModelBoundAdmissionCallback;
}
return initial != null;
},
wrapCallbacks(callbacks) {
const observer = ensureHandler({
name: 'librechat-host-model-admission',
raiseError: true,
awaitHandlers: true,
handleChatModelStart: translate,
});
if (!callbacks || Array.isArray(callbacks)) return [...(callbacks ?? []).map(wrap), observer];
const manager = callbacks.copy();
manager.handlers = manager.handlers.map((handler) => ensureHandler(wrap(handler)));
manager.inheritableHandlers = manager.inheritableHandlers.map((handler) =>
ensureHandler(wrap(handler)),
);
manager.addHandler(observer, true);
return manager;
},
async invoke(context, operation) {
const attempt = begin(context);
let successful = false;
try {
const value = attempt ? await storage.run(attempt, operation) : await operation();
successful = true;
return value;
} finally {
finish(attempt, successful);
}
},
async *stream(context, operation) {
const attempt = begin(context);
let successful = false;
try {
yield* attempt ? scopedStream(storage, attempt, operation) : operation();
successful = true;
} finally {
finish(attempt, successful);
}
},
end(data) {
if (!initial || disposed || policyFailure) throw policyFailure ?? blocked();
const { nodeId, completed } = nodeEndSchema.parse(data);
const node = nodes.get(nodeId);
if (!node || node.pending !== 0) throw blocked();
nodes.delete(nodeId);
if (completed && node.successful) initial.handleChainEnd({ messages: [{}] }, nodeId);
else initial.handleChainError(blocked(), nodeId);
admitted = initial.isAdmitted?.() === true;
if (admitted) {
usedModels.clear();
usedNodes.clear();
}
return admitted;
},
dispose() {
if (disposed) return;
disposed = true;
for (const nodeId of nodes.keys()) initial?.handleChainError(blocked(), nodeId);
// Also rejects private admission when failure preceded the first native call.
initial?.handleChainError(blocked(), 'remote-execution');
nodes.clear();
usedModels.clear();
usedNodes.clear();
},
};
}
export interface WorkerModelLifecycle {
callback: BaseCallbackHandler;
current(): RemoteModelAttempt | undefined;
model(options: ConstructorParameters<typeof HostChatModel>[0]): HostChatModel;
}
/** Only callbacks installed on this root execution can enroll a model attempt. No message text is relayed. */
export function createWorkerModelLifecycle(
agentId: string,
maxEntries: number,
peer: { call: SdkRpc },
): WorkerModelLifecycle {
const parents = new Map<string, string>();
const nodes = new Set<string>();
const models = new Map<string, RemoteModelAttempt>();
const attemptedNodes = new Set<string>();
const storage = new AsyncLocalStorage<RemoteModelAttempt | undefined>();
let admitted = false;
const budget = () => {
if (parents.size + nodes.size + models.size >= maxEntries) throw blocked();
};
const context = (manager?: CallbackManagerForLLMRun) => {
if (admitted) return undefined;
const value = manager && models.get(manager.runId);
if (!value) throw blocked();
attemptedNodes.add(value.nodeId);
return value;
};
const end = async (nodeId: string, completed: boolean) => {
parents.delete(nodeId);
nodes.delete(nodeId);
for (const [modelId, attempt] of models) if (attempt.nodeId === nodeId) models.delete(modelId);
if (!attemptedNodes.delete(nodeId)) return;
const reply = z
.object({ version: z.literal(1), ok: z.literal(true), admitted: z.boolean() })
.parse(await peer.call('model.node.end', { nodeId, completed }));
if (reply.admitted) {
admitted = true;
parents.clear();
nodes.clear();
models.clear();
attemptedNodes.clear();
}
};
class CorrelatedHostModel extends HostChatModel {
override _generate(
messages: BaseMessage[],
options: this['ParsedCallOptions'],
manager?: CallbackManagerForLLMRun,
) {
return storage.run(context(manager), () => super._generate(messages, options));
}
override async *_streamResponseChunks(
messages: BaseMessage[],
options: this['ParsedCallOptions'],
manager?: CallbackManagerForLLMRun,
): AsyncGenerator<ChatGenerationChunk> {
yield* scopedStream(storage, context(manager), () =>
super._streamResponseChunks(messages, options, manager),
);
}
}
return {
current() {
if (admitted) return undefined;
const value = storage.getStore();
if (!value) throw blocked();
return value;
},
model: (options) => new CorrelatedHostModel(options),
callback: ensureHandler({
name: 'librechat-remote-model-admission',
raiseError: true,
awaitHandlers: true,
handleChainStart: (_chain, _inputs, runId, parentRunId, _tags, metadata, _type, name) => {
if (admitted) return;
budget();
if (parentRunId) parents.set(runId, parentRunId);
if (name === `agent=${agentId}` && metadata?.langgraph_node === name) nodes.add(runId);
},
handleChatModelStart: (_model, _messages, runId, parentRunId, _extra, _tags, metadata) => {
if (admitted) return;
if (
metadata?.summarization === true ||
metadata?.agentId !== agentId ||
metadata.langgraph_node !== `agent=${agentId}`
)
return;
const seen = new Set<string>();
let parent = parentRunId;
while (parent && !seen.has(parent)) {
if (nodes.has(parent)) {
budget();
models.set(runId, { nodeId: parent, modelId: runId });
return;
}
seen.add(parent);
parent = parents.get(parent);
}
},
handleChainEnd: async (output, runId) => {
if (admitted) return;
if (!nodes.has(runId)) {
parents.delete(runId);
return;
}
const result = output as { messages?: unknown; summarizationRequest?: unknown } | null;
await end(
runId,
Array.isArray(result?.messages) &&
result.messages.length > 0 &&
result.summarizationRequest == null,
);
},
handleChainError: (_error, runId) => (admitted ? undefined : end(runId, false)),
}),
};
}

View file

@ -4,8 +4,9 @@ import { randomUUID } from 'node:crypto';
import WebSocket, { WebSocketServer } from 'ws';
import { tool } from '@librechat/agents/langchain/tools';
import { ChatGenerationChunk } from '@langchain/core/outputs';
import { agentsEndpointSchema } from 'librechat-data-provider';
import { ensureHandler } from '@langchain/core/callbacks/manager';
import { FakeChatModel as TestModel } from '@langchain/core/utils/testing';
import { agentsEndpointSchema, FileSources } from 'librechat-data-provider';
import { HumanMessage, AIMessageChunk } from '@librechat/agents/langchain/messages';
import {
createHostProviderTextService,
@ -19,12 +20,29 @@ import {
MemorySaver,
HookRegistry,
} from '@librechat/agents';
import type { AppConfig, IUser } from '@librechat/data-schemas';
import type { IUser } from '@librechat/data-schemas';
import type { RunConfig } from '@librechat/agents';
import type { AddressInfo } from 'node:net';
import type { Request } from 'express';
import type { InitialModelBoundAdmissionCallback } from '../../middleware/modelBoundContent';
import type { RemoteSdkServerOptions } from './server';
import type { RemoteSdkFactoryInput } from './types';
import {
createPrivateTextInitialAdmissionCallback,
getPrivateTextModelHooks,
withPrivateTextAdmissionConfig,
} from '../../protection/private/admission';
import {
createPrivateTextIngress,
stampPrivateTextMessage,
getPrivateTextInspectionTokens,
} from '../../protection/private/submission';
import {
createInitialModelBoundAdmissionCallback,
createModelBoundChatModelCallback,
} from '../../middleware/modelBoundContent';
import { createSdkAgentRun, resolveSdkAgentBinding } from '../sdk';
import { createHostModelLifecycle } from './admission';
import { createRun as createHostRun } from '../run';
import { createRemoteSdkServer } from './server';
import { createSdkModelHost } from './builder';
@ -1205,7 +1223,7 @@ describe('remote host provider-text protection', () => {
});
});
it.each(['providerText', 'model', 'builderModel'] as const)(
it.each(['providerText', 'model', 'builderModel', 'modelAdmission'] as const)(
'rejects a worker without %s capability before factory admission',
async (capability) => {
const server = new WebSocketServer({ port: 0, host: '127.0.0.1' });
@ -1223,7 +1241,9 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
return {
transportVersion: 2,
hostTools: 2,
...(capability === 'builderModel' ? { hostModel: 1 } : {}),
...(['builderModel', 'modelAdmission'].includes(capability)
? { hostModel: 1, ...(capability === 'modelAdmission' ? { builderModel: 1 } : {}) }
: {}),
};
},
}),
@ -1249,6 +1269,9 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
? { providerText }
: {
model: nativeModelBinding(),
...(capability === 'modelAdmission'
? { modelLifecycle: createHostModelLifecycle(identity.agentId, 16) }
: {}),
...(capability === 'builderModel'
? {
modelProfile: {
@ -1260,9 +1283,23 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
: {}),
},
});
await expect(run.processStream({ messages: [] }, invocation())).rejects.toThrow(
'SDK runtime execution failed',
);
await expect(
run.processStream(
{ messages: [] },
{
...invocation(),
callbacks: [
admissionCallback(
createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: () => {},
}),
),
],
},
),
).rejects.toThrow('SDK runtime execution failed');
expect(methods).toEqual(['capabilities']);
} finally {
for (const socket of server.clients) socket.terminate();
@ -1366,6 +1403,23 @@ describe('saved-agent builder host model', () => {
message: new AIMessageChunk({ content: 'builder reply' }),
});
};
const filters = {
messages: {
pii: {
fields: ['text' as const],
starterPatterns: [],
customPatterns: [{ id: 'private', label: 'Private', regex: 'PRIVATE-CALLBACK' }],
},
},
};
const policy = createModelBoundChatModelCallback({ filters });
Object.assign(native, { callbacks: [policy] });
const persisted = jest.fn();
const initial = createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: persisted,
});
const initialize = jest.fn(() => native);
const delta = jest.fn();
const run = await createHostRun({
@ -1374,7 +1428,14 @@ describe('saved-agent builder host model', () => {
signal: new AbortController().signal,
tokenCounter: () => 1,
user: { id: identity.userId, role: 'ADMIN', tenantId: identity.tenantId } as IUser,
appConfig: { endpoints: { agents: endpoint } } as AppConfig,
appConfig: {
config: {},
fileStrategy: FileSources.local,
imageOutputType: 'png',
filters,
endpoints: { agents: endpoint },
},
modelCallbacks: [policy],
sdkModelHost: createSdkModelHost(endpoint.sdkHostModels, initialize),
agents: [
{
@ -1400,7 +1461,11 @@ describe('saved-agent builder host model', () => {
],
customHandlers: { [GraphEvents.ON_MESSAGE_DELTA]: { handle: delta } },
});
await run.processStream({ messages: [new HumanMessage('hello')] }, invocation());
await run.processStream(
{ messages: [new HumanMessage('hello')] },
{ ...invocation(), callbacks: [admissionCallback(initial)] },
);
expect(persisted).toHaveBeenCalledTimes(1);
expect(initialize).toHaveBeenCalledWith(
expect.objectContaining({
provider: Providers.OPENAI,
@ -1510,3 +1575,290 @@ describe('saved-agent builder host model', () => {
).rejects.toMatchObject({ code: 'sdk_runtime_unavailable' });
});
});
function admissionCallback(initial: InitialModelBoundAdmissionCallback) {
// The host callback inspects lineage only. LangChain's native-message type is wider than its policy projection.
return ensureHandler({
...initial,
handleChatModelStart: (_model, _messages, runId, parent, extra, tags, metadata) =>
initial.handleChatModelStart(undefined, [], runId, parent, extra, tags, metadata),
});
}
describe('remote native policy admission', () => {
it('rejects combined output protection and initial admission before worker execution', async () => {
const createRun = jest.fn(factory);
const runtime = await fixture(createRun);
try {
const initial = createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: jest.fn(),
});
const run = await createRemoteSdkRun({
config: hostConfig(),
identity,
remote: runtime.remote,
hostServices: {
model: nativeModelBinding(),
modelLifecycle: createHostModelLifecycle(identity.agentId, 16),
providerText: createHostProviderTextService({
identity,
policy: {
version: 1,
timeoutMs: 1000,
maxAttemptBytes: 4096,
maxBufferedBytes: 8192,
classify: () => 'prose',
inspect: () => ({ version: 1, ok: false, error: { code: 'blocked' } }),
},
}),
},
});
await expect(
run.processStream(
{ messages: [] },
{ ...invocation(), callbacks: [admissionCallback(initial)] },
),
).rejects.toThrow('SDK invocation context support');
expect(createRun).not.toHaveBeenCalled();
} finally {
await runtime.close();
}
});
it('does not persist a deferred message when a partially streamed model is cancelled', async () => {
const persisted = jest.fn();
const initial = createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: persisted,
});
const abort = new AbortController();
let entered!: () => void;
let release!: () => void;
const started = new Promise<void>((resolve) => {
entered = resolve;
});
const pending = new Promise<void>((resolve) => {
release = resolve;
});
const runtime = await fixture(async (input) => {
const run = await factory(input);
run.Graph!.overrideModel = input.hostModel;
return run;
});
try {
const model = new TestModel({});
model._streamResponseChunks = async function* () {
yield new ChatGenerationChunk({
text: 'partial',
message: new AIMessageChunk({ content: 'partial' }),
});
entered();
await pending;
};
const run = await createSdkAgentRun({
config: hostConfig(),
identity,
signal: abort.signal,
registration: {
binding: { id: 'custom', protocolVersion: 1 },
remote: { ...runtime.remote, hostModel: true },
},
modelHost: createSdkModelHost(
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
() => model,
),
});
const execution = run.processStream(
{ messages: [new HumanMessage('safe')] },
{ ...invocation(), signal: abort.signal, callbacks: [admissionCallback(initial)] },
);
void execution.catch(() => {});
await started;
abort.abort();
release();
await expect(execution).rejects.toThrow('cancelled');
expect(persisted).not.toHaveBeenCalled();
} finally {
release();
await runtime.close();
}
});
it.each([false, true])(
'admits deferred persistence only after the safe model node (blocked=%s)',
async (blocked) => {
const persisted = jest.fn();
const nativeCall = jest.fn();
const log = jest.spyOn(console, 'error').mockImplementation(() => {});
const callback = createModelBoundChatModelCallback({
filters: {
messages: {
pii: {
fields: ['text'],
starterPatterns: [],
customPatterns: [{ id: 'private', label: 'private', regex: 'PRIVATE-CALLBACK' }],
},
},
},
});
const initial = createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: persisted,
});
const runtime = await fixture(async (input) => {
const run = await factory(input);
run.Graph!.overrideModel = input.hostModel;
return run;
});
try {
const model = new TestModel({});
Object.assign(model, { callbacks: [callback] });
model._streamResponseChunks = async function* () {
nativeCall();
expect(persisted).not.toHaveBeenCalled();
yield new ChatGenerationChunk({
text: 'safe reply',
message: new AIMessageChunk({ content: 'safe reply' }),
});
};
const endpoint = agentsEndpointSchema.parse({ sdkHostModels: {} });
const run = await createSdkAgentRun({
config: hostConfig(),
identity,
registration: {
binding: { id: 'custom', protocolVersion: 1 },
remote: { ...runtime.remote, hostModel: true },
},
modelHost: createSdkModelHost(endpoint.sdkHostModels, () => model),
});
const promise = run.processStream(
{ messages: [new HumanMessage(blocked ? 'PRIVATE-CALLBACK' : 'safe input')] },
{ ...invocation(), callbacks: [admissionCallback(initial)] },
);
if (blocked) {
await expect(promise).rejects.toMatchObject({ code: 'content_filter_block' });
expect(nativeCall).not.toHaveBeenCalled();
expect(persisted).not.toHaveBeenCalled();
} else {
await expect(promise).resolves.toBeDefined();
expect(nativeCall).toHaveBeenCalledTimes(1);
expect(persisted).toHaveBeenCalledTimes(1);
}
} finally {
log.mockRestore();
await runtime.close();
}
},
);
it.each([false, true])(
'commits captured private text before provider execution (persistence failure=%s)',
async (failPersistence) => {
const filters = {
messages: {
pii: {
action: 'redact' as const,
fields: ['text' as const],
starterPatterns: [],
customPatterns: [
{
id: 'email',
label: 'Email',
regex: 'alice@example\\.com',
category: 'email' as const,
},
],
},
},
};
const req = {
path: '/',
user: { id: identity.userId },
body: { text: 'alice@example.com', clientRequestId: 'private-remote' },
} as unknown as Request;
createPrivateTextIngress({
getFilters: () => filters,
getLegacyPii: () => undefined,
getKey: () => 'ab'.repeat(32),
})(req, {} as never, jest.fn());
const message = stampPrivateTextMessage(req, {
messageId: 'user',
conversationId: identity.conversationId,
text: req.body.text,
isCreatedByUser: true,
});
const persisted = jest.fn(async () => {
if (failPersistence) throw new Error('persistence unavailable');
return { message };
});
const cancelled = jest.fn();
const created = jest.fn();
const nativeCall = jest.fn();
const initial = createPrivateTextInitialAdmissionCallback(req, {
agentIds: [identity.agentId],
start: persisted,
cancel: cancelled,
onPersisted: created,
})!;
const callback = createModelBoundChatModelCallback(
{
filters,
storedMessages: [message],
privateTextTokens: getPrivateTextInspectionTokens([message]),
},
getPrivateTextModelHooks(req, initial, persisted, cancelled, created),
);
const log = jest.spyOn(console, 'error').mockImplementation(() => {});
const runtime = await fixture(async (input) => {
const run = await factory(input);
run.Graph!.overrideModel = input.hostModel;
return run;
});
try {
const model = new TestModel({});
Object.assign(model, { callbacks: [callback] });
model._streamResponseChunks = async function* () {
expect(created).toHaveBeenCalledTimes(1);
nativeCall();
yield new ChatGenerationChunk({
text: 'safe reply',
message: new AIMessageChunk({ content: 'safe reply' }),
});
};
const run = await createSdkAgentRun({
config: hostConfig(),
identity,
registration: {
binding: { id: 'custom', protocolVersion: 1 },
remote: { ...runtime.remote, hostModel: true },
},
modelHost: createSdkModelHost(
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
() => model,
),
});
const promise = run.processStream(
{ messages: [new HumanMessage(req.body.text)] },
{
...withPrivateTextAdmissionConfig(invocation(), initial),
callbacks: [admissionCallback(initial)],
},
);
if (failPersistence) {
await expect(promise).rejects.toThrow('SDK runtime execution failed');
expect(nativeCall).not.toHaveBeenCalled();
expect(cancelled).toHaveBeenCalled();
} else {
await expect(promise).resolves.toBeDefined();
expect(nativeCall).toHaveBeenCalledTimes(1);
}
expect(persisted).toHaveBeenCalledTimes(1);
} finally {
log.mockRestore();
await runtime.close();
}
},
);
});

View file

@ -4,6 +4,7 @@ import type { TAgentsEndpoint } from 'librechat-data-provider';
import type { RunConfig } from '@librechat/agents';
import type { RemoteSdkHostServices } from './types';
import type { SdkAgentIdentity } from './identity';
import { createHostModelLifecycle } from './admission';
type Limits = NonNullable<NonNullable<TAgentsEndpoint>['sdkHostModels']>;
@ -62,12 +63,15 @@ export function createSdkModelHost(
signal?.throwIfAborted();
const model = createModel({ provider: agent.provider, clientOptions: agent.clientOptions });
if (!(model instanceof BaseChatModel)) throw new SdkHostModelError();
const modelLifecycle = createHostModelLifecycle(identity.agentId, limits.maxAdmissionEntries);
model.callbacks = modelLifecycle.wrapCallbacks(model.callbacks);
const modelProfile = {
provider: agent.provider,
name: modelName,
streaming: agent.clientOptions?.streaming !== false,
};
return {
modelLifecycle,
modelProfile,
model: createHostModelService({
identity,

View file

@ -200,8 +200,15 @@ export async function createRemoteSdkRun({
return socket;
};
const socket = remote.recovery
? await connectRecoveringSdkSocket({
const modelAdmission = hostServices?.modelLifecycle?.attach(options.callbacks) === true;
if (modelAdmission && hostServices?.providerText) {
hostServices.modelLifecycle?.dispose();
throw new Error(
'Remote model admission with provider-text protection requires SDK invocation context support',
);
}
const connection = remote.recovery
? connectRecoveringSdkSocket({
url: remote.url,
token: makeToken,
generation:
@ -213,7 +220,11 @@ export async function createRemoteSdkRun({
signal: options.signal,
onError: remote.onError,
})
: await legacy();
: legacy();
const socket = await connection.catch((error) => {
hostServices?.modelLifecycle?.dispose();
throw error;
});
const execution = new AbortController();
const providerText = hostServices?.providerText
? createHostPolicyChannel(hostServices.providerText, limits)
@ -227,17 +238,19 @@ export async function createRemoteSdkRun({
const finish = (value?: RemoteSdkSnapshot) => {
if (settled) return;
settled = true;
if (value == null)
if (value == null || hostServices?.modelLifecycle?.failure())
reject(
options.signal?.aborted
? Object.assign(new Error('SDK runtime execution cancelled'), { name: 'AbortError' })
: new Error('SDK runtime execution failed'),
: (hostServices?.modelLifecycle?.failure() ??
new Error('SDK runtime execution failed')),
);
else resolve(value);
};
hostModelChannel = hostServices?.model
? createHostModelChannel({
model: hostServices.model,
lifecycle: modelAdmission ? hostServices.modelLifecycle : undefined,
peer: { call: (...args) => peer.call(...args) },
limits,
signal: execution.signal,
@ -280,6 +293,12 @@ export async function createRemoteSdkRun({
onCall: async (method, data, _timeoutMs, signal) => {
if (execution.signal.aborted || options.signal?.aborted)
throw new Error('SDK runtime execution stopped');
if (method === 'model.node.end') {
if (!modelAdmission || !hostServices?.modelLifecycle)
throw new Error('Model admission unavailable');
const admitted = hostServices.modelLifecycle.end(data);
return { version: 1, ok: true, admitted };
}
if (method === 'model.invoke' || method === 'model.stream') {
if (!hostModelChannel) throw new Error('Host model is unavailable');
const invocation = hostModelChannel.call(method, data, signal);
@ -423,6 +442,9 @@ export async function createRemoteSdkRun({
'agents' in config.graphConfig ? config.graphConfig.agents[0] : config.graphConfig;
const start: RemoteSdkStart = {
transportVersion: 2,
modelAdmission: modelAdmission
? { version: 1, maxEntries: hostServices!.modelLifecycle!.maxEntries }
: undefined,
model: hostModelChannel?.descriptor,
providerText: providerText?.descriptor,
hostTools: descriptors.length ? descriptors : undefined,
@ -471,9 +493,12 @@ export async function createRemoteSdkRun({
providerText?: number;
hostModel?: number;
builderModel?: number;
modelAdmission?: number;
};
if (capabilities?.transportVersion !== 2)
throw new Error('Worker transport capability unavailable');
if (modelAdmission && capabilities?.modelAdmission !== 1)
throw new Error('Worker model admission capability unavailable');
if (hostServices?.modelProfile && capabilities?.builderModel !== 1)
throw new Error('Worker builder model capability unavailable');
if (hostServices?.model && capabilities?.hostModel !== 1)
@ -493,6 +518,7 @@ export async function createRemoteSdkRun({
return snapshot.content?.filter((part) => part != null);
} finally {
execution.abort();
hostServices?.modelLifecycle?.dispose();
providerText?.dispose();
hostModelChannel?.dispose();
// A failed/cancelled remote run may leave host cleanup running. Keep the host

View file

@ -1,3 +1,4 @@
import { z } from 'zod';
import { once } from 'node:events';
import { randomUUID } from 'node:crypto';
import WebSocket, { WebSocketServer } from 'ws';
@ -9,8 +10,8 @@ import type { HostModelDescriptor, HostModelRequest } from '@librechat/agents';
import type { AddressInfo } from 'node:net';
import { createHostModelChannel, createWorkerModelChannel } from './model';
import { createPolicyReceiver, createPolicySender } from './sealed';
import { encodeSdkValue, decodeSdkValue } from './wire';
import { remoteSdkLimits } from './types';
import { encodeSdkValue } from './wire';
import { createSdkPeer } from './peer';
const identity = {
@ -106,6 +107,42 @@ async function fixture(model = new FakeChatModel({}), tools: HostModelDescriptor
}
const tick = () => new Promise<void>((resolve) => setTimeout(resolve, 10));
it('preserves the strict older-host request shape without negotiated admission', async () => {
const receiver = createPolicyReceiver(limits.maxFrameBytes, 'model');
const controller = new AbortController();
const call = jest.fn(async (_method: string, data: unknown) => {
const operation = z
.object({ id: z.string().uuid(), publicKey: z.string().length(60), request: z.unknown() })
.strict()
.parse(decodeSdkValue(JSON.parse(receiver.open(data))));
const seal = createPolicySender(operation.publicKey, limits.maxFrameBytes, 'model');
return {
version: 1,
ok: true,
message: seal(
JSON.stringify(
encodeSdkValue({ id: operation.id, message: new AIMessage('legacy reply') }),
),
),
};
});
const worker = createWorkerModelChannel({ peer: { call }, limits, signal: controller.signal });
try {
const model = worker.bind(identity, {
version: 1,
publicKey: receiver.publicKey,
descriptor,
});
await expect(model.invoke([...request.messages])).resolves.toMatchObject({
content: 'legacy reply',
});
expect(call).toHaveBeenCalledTimes(1);
} finally {
controller.abort();
receiver.dispose();
}
});
it('streams native tool fragments, rich content and typed metadata over a real socket', async () => {
const native = new FakeChatModel({});
native._streamResponseChunks = async function* () {

View file

@ -9,13 +9,20 @@ import type {
RemoteSdkLimits,
RemoteSdkStart,
} from './types';
import type { HostModelLifecycle, WorkerModelLifecycle } from './admission';
import type { SdkAgentIdentity } from './identity';
import type { SdkRpc } from './peer';
import { createPolicyReceiver, createPolicySender } from './sealed';
import { encodeSdkValue, decodeSdkValue } from './wire';
import { modelAttemptSchema } from './admission';
const operationSchema = z
.object({ id: z.string().uuid(), publicKey: z.string().length(60), request: z.unknown() })
.object({
id: z.string().uuid(),
publicKey: z.string().length(60),
request: z.unknown(),
context: modelAttemptSchema.optional(),
})
.strict();
const chunkSchema = z
.object({ id: z.string().uuid(), index: z.number().int().nonnegative(), message: z.unknown() })
@ -61,6 +68,7 @@ function sealValue(seal: ReturnType<typeof createPolicySender>, value: unknown,
/** One byte window shared by every model stream on this execution's connection. */
export function createHostModelChannel(options: {
model: NonNullable<RemoteSdkHostServices['model']>;
lifecycle?: HostModelLifecycle;
peer: { call: SdkRpc };
limits: RemoteSdkLimits;
signal: AbortSignal;
@ -91,7 +99,10 @@ export function createHostModelChannel(options: {
// The SDK binding validates identity, reference, messages and allowed tools before admission.
const request = parsed.data.request as HostModelRequest;
if (method === 'model.invoke') {
const message = await options.model.service.invoke(request, boundSignal);
const invoke = () => options.model.service.invoke(request, boundSignal);
const message = await (options.lifecycle
? options.lifecycle.invoke(parsed.data.context, invoke)
: invoke());
return {
version: 1,
ok: true,
@ -100,7 +111,11 @@ export function createHostModelChannel(options: {
}
if (method !== 'model.stream') throw new HostModelServiceError('incompatible');
let index = 0;
for await (const chunk of options.model.service.stream(request, boundSignal)) {
const stream = () => options.model.service.stream(request, boundSignal);
const source = options.lifecycle
? options.lifecycle.stream(parsed.data.context, stream)
: stream();
for await (const chunk of source) {
boundSignal.throwIfAborted();
if (deliveryError) throw deliveryError;
const payload = sealValue(
@ -195,11 +210,17 @@ export function createWorkerModelChannel(options: {
const streams = new Map<string, ModelStream>();
let receiver: ReturnType<typeof createPolicyReceiver> | undefined;
let seal: ReturnType<typeof createPolicySender> | undefined;
let lifecycle: WorkerModelLifecycle | undefined;
const encodeRequest = (id: string, request: HostModelRequest) => {
if (!receiver || !seal) throw new HostModelServiceError('incompatible');
return sealValue(
seal,
{ id, publicKey: receiver.publicKey, request },
{
id,
publicKey: receiver.publicKey,
request,
...(lifecycle && { context: lifecycle.current() }),
},
options.limits.maxFrameBytes,
);
};
@ -300,12 +321,15 @@ export function createWorkerModelChannel(options: {
bind: (
identity: Readonly<SdkAgentIdentity>,
descriptor: NonNullable<RemoteSdkStart['model']>,
admission?: WorkerModelLifecycle,
): NonNullable<RemoteSdkFactoryInput['hostModel']> => {
if (receiver || options.signal.aborted || descriptor.version !== 1)
throw new HostModelServiceError('incompatible');
lifecycle = admission;
receiver = createPolicyReceiver(options.limits.maxFrameBytes, 'model');
seal = createPolicySender(descriptor.publicKey, options.limits.maxFrameBytes, 'model');
return new HostChatModel({ identity, descriptor: descriptor.descriptor, service }).bindTools([
const config = { identity, descriptor: descriptor.descriptor, service };
return (lifecycle ? lifecycle.model(config) : new HostChatModel(config)).bindTools([
...descriptor.descriptor.tools,
]);
},

View file

@ -30,6 +30,12 @@ export type SdkFrame =
| { type: 'done'; data: unknown }
| { type: 'error' };
export interface SdkPeer {
call: SdkRpc;
send(frame: SdkFrame): void;
close(): void;
}
const frameSchema = z.discriminatedUnion('type', [
z.object({ type: z.literal('cancel'), id: z.string().uuid() }),
z.object({
@ -67,7 +73,7 @@ export function createSdkPeer(
onClose: (error: Error) => void;
onError?: (error: unknown) => void;
},
) {
): SdkPeer {
const pending = new Map<
string,
{ resolve: (data: unknown) => void; reject: (error: Error) => void }

View file

@ -6,6 +6,7 @@ import { request as httpRequest } from 'node:http';
import { tool } from '@librechat/agents/langchain/tools';
import { setTimeout as delay } from 'node:timers/promises';
import { ChatGenerationChunk } from '@langchain/core/outputs';
import { ensureHandler } from '@langchain/core/callbacks/manager';
import { FakeChatModel as TestModel } from '@langchain/core/utils/testing';
import { HumanMessage, AIMessageChunk } from '@librechat/agents/langchain/messages';
import {
@ -28,6 +29,8 @@ import {
createRedisTestClient,
clearRedisTestPrefix,
} from '../../../stream/__tests__/helpers/redis';
import { createInitialModelBoundAdmissionCallback } from '../../../middleware/modelBoundContent';
import { createHostModelLifecycle } from '../admission';
import { createRedisRuntimeJournal } from './redis';
import { createRemoteSdkServer } from '../server';
import { createRuntimeRecovery } from './server';
@ -1020,89 +1023,113 @@ it('recovers a policy reply after a wire outage without repeating inspection or
}
});
it('resumes host model chunks after disconnect without another provider invocation', async () => {
const runtime = await fixture(async (input) => {
const run = await factory(input);
run.Graph!.overrideModel = input.hostModel;
return run;
});
const native = new TestModel({});
let invocations = 0;
let hostStarts = 0;
native._streamResponseChunks = async function* () {
invocations++;
yield new ChatGenerationChunk({ text: 'first ', message: new AIMessageChunk('first ') });
runtime.drop();
await delay(50);
yield new ChatGenerationChunk({
text: 'PRIVATE-CANDIDATE ',
message: new AIMessageChunk('PRIVATE-CANDIDATE '),
it.each([false, true])(
'resumes host model chunks without repeating the provider (input admission=%s)',
async (inputAdmission) => {
const runtime = await fixture(async (input) => {
const run = await factory(input);
run.Graph!.overrideModel = input.hostModel;
return run;
});
yield new ChatGenerationChunk({ text: 'third', message: new AIMessageChunk('third') });
};
const admission = new HostModelAdmission({ maxConcurrentCalls: 1, maxRequestBytes: 65536 });
const model = createHostModelService({
identity,
model: native,
admission,
descriptor: {
version: 1,
reference: 'primary',
provider: Providers.OPENAI,
model: 'native',
streaming: true,
tools: [],
},
limits: { timeoutMs: 5000, maxRequestBytes: 65536, maxResponseBytes: 65536 },
config: {
callbacks: [
{
handleChatModelStart: () => {
hostStarts++;
},
},
],
},
});
const providerText = createHostProviderTextService({
identity,
policy: {
version: 1,
timeoutMs: 5000,
maxAttemptBytes: 65536,
maxBufferedBytes: 131072,
classify: () => 'prose',
inspect: ({ content }) => ({
version: 1,
ok: true,
value: {
content: content.replaceAll('PRIVATE-CANDIDATE', 'approved'),
replacements: 1,
categories: [],
},
}),
},
});
try {
const run = await createRemoteSdkRun({
config: hostConfig(async () => {}),
const native = new TestModel({});
let invocations = 0;
let hostStarts = 0;
const persisted = jest.fn();
const initial = createInitialModelBoundAdmissionCallback({
agentIds: [identity.agentId],
isActive: () => true,
onAllowed: persisted,
});
const modelLifecycle = createHostModelLifecycle(identity.agentId, 64);
if (inputAdmission) native.callbacks = modelLifecycle.wrapCallbacks(native.callbacks);
native._streamResponseChunks = async function* () {
invocations++;
yield new ChatGenerationChunk({ text: 'first ', message: new AIMessageChunk('first ') });
runtime.drop();
await delay(50);
yield new ChatGenerationChunk({
text: inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE ',
message: new AIMessageChunk(inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE '),
});
yield new ChatGenerationChunk({ text: 'third', message: new AIMessageChunk('third') });
};
const admission = new HostModelAdmission({ maxConcurrentCalls: 1, maxRequestBytes: 65536 });
const model = createHostModelService({
identity,
remote: runtime.remote,
hostServices: { model, providerText },
model: native,
admission,
descriptor: {
version: 1,
reference: 'primary',
provider: Providers.OPENAI,
model: 'native',
streaming: true,
tools: [],
},
limits: { timeoutMs: 5000, maxRequestBytes: 65536, maxResponseBytes: 65536 },
config: {
callbacks: [
{
handleChatModelStart: () => {
hostStarts++;
},
},
],
},
});
const content = await run.processStream(
{ messages: [new HumanMessage('go')] },
{ version: 'v2', configurable: { thread_id: identity.conversationId } },
);
expect(JSON.stringify(content)).toContain('first approved third');
expect(runtime.journalEntries.join('')).not.toContain('PRIVATE-CANDIDATE');
expect(invocations).toBe(1);
expect(hostStarts).toBe(1);
expect(runtime.connections()).toBeGreaterThan(1);
expect(runtime.createRun).toHaveBeenCalledTimes(1);
expect(runtime.errors).toEqual([]);
expect(admission.snapshot().calls).toBe(0);
} finally {
await runtime.close();
}
});
const providerText = createHostProviderTextService({
identity,
policy: {
version: 1,
timeoutMs: 5000,
maxAttemptBytes: 65536,
maxBufferedBytes: 131072,
classify: () => 'prose',
inspect: ({ content }) => ({
version: 1,
ok: true,
value: {
content: content.replaceAll('PRIVATE-CANDIDATE', 'approved'),
replacements: 1,
categories: [],
},
}),
},
});
try {
const run = await createRemoteSdkRun({
config: hostConfig(async () => {}),
identity,
remote: runtime.remote,
hostServices: { model, ...(inputAdmission ? { modelLifecycle } : { providerText }) },
});
const content = await run.processStream(
{ messages: [new HumanMessage('go')] },
{
version: 'v2',
configurable: { thread_id: identity.conversationId },
callbacks: inputAdmission
? [
ensureHandler({
...initial,
handleChatModelStart: (_llm, _messages, id, parent, extra, tags, metadata) =>
initial.handleChatModelStart(undefined, [], id, parent, extra, tags, metadata),
}),
]
: undefined,
},
);
expect(JSON.stringify(content)).toContain('first approved third');
expect(runtime.journalEntries.join('')).not.toContain('PRIVATE-CANDIDATE');
expect(invocations).toBe(1);
expect(persisted).toHaveBeenCalledTimes(inputAdmission ? 1 : 0);
expect(hostStarts).toBe(1);
expect(runtime.connections()).toBeGreaterThan(1);
expect(runtime.createRun).toHaveBeenCalledTimes(1);
expect(runtime.errors).toEqual([]);
expect(admission.snapshot().calls).toBe(0);
} finally {
await runtime.close();
}
},
);

View file

@ -24,6 +24,7 @@ import { policyDescriptorSchema, createWorkerProviderTextPolicy } from './policy
import { createRuntimeRecovery, recoveryClaimSchema } from './recovery/server';
import { remoteSdkLimits, isRemoteSdkSecret } from './types';
import { RECOVERY_HEADER_BYTES } from './recovery/socket';
import { createWorkerModelLifecycle } from './admission';
import { createWorkerToolService } from './hostTools';
import { RemoteSdkCheckpointer } from './checkpoint';
import { createWorkerModelChannel } from './model';
@ -51,6 +52,10 @@ const claimSchema = z.object({
});
const startSchema = z.object({
transportVersion: z.literal(2),
modelAdmission: z
.object({ version: z.literal(1), maxEntries: z.number().int().min(16).max(65536) })
.strict()
.optional(),
// The SDK HostChatModel constructor owns descriptor validation.
model: z
.object({
@ -292,6 +297,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
providerText: 1,
hostModel: 1,
builderModel: 1,
modelAdmission: 1,
};
if (method === 'model.chunk') return workerModel.accept(data, signal);
if (method === 'tool.result') return workerTools.accept(data);
@ -299,6 +305,10 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
if (method !== 'run' || started) throw new Error('SDK execution already admitted');
started = true;
const request = startSchema.parse(data) as RemoteSdkStart;
if (request.modelAdmission && request.providerText)
throw new Error(
'Combined model admission and provider-text protection is unsupported',
);
const identity = Object.freeze(claim.identity);
if (
request.agent.id !== identity.agentId ||
@ -374,7 +384,18 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
limits,
})
: undefined;
const hostModel = request.model ? workerModel.bind(identity, request.model) : undefined;
if (request.modelAdmission && !request.model)
throw new Error('Model admission requires a host model');
const modelLifecycle = request.modelAdmission
? createWorkerModelLifecycle(
identity.agentId,
request.modelAdmission.maxEntries,
peer,
)
: undefined;
const hostModel = request.model
? workerModel.bind(identity, request.model, modelLifecycle)
: undefined;
const profile = request.agent.model;
if (
profile &&
@ -439,6 +460,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
}
const callerOptions = {
...request.options,
...(modelLifecycle && { callbacks: [modelLifecycle.callback] }),
version: 'v2' as const,
signal: controller.signal,
};

View file

@ -13,6 +13,7 @@ import type {
IState,
HookEvent,
} from '@librechat/agents';
import type { HostModelLifecycle } from './admission';
import type { SdkAgentIdentity } from './identity';
export interface RemoteSdkRecoveryOptions {
@ -68,6 +69,7 @@ export interface RemoteSdkModelProfile {
/** Explicit host-owned capabilities; never reconstructed from worker claims or YAML. */
export interface RemoteSdkHostServices {
modelLifecycle?: HostModelLifecycle;
modelProfile?: RemoteSdkModelProfile;
model?: ReturnType<typeof createHostModelService>;
providerText?: ReturnType<typeof createHostProviderTextService>;
@ -75,6 +77,7 @@ export interface RemoteSdkHostServices {
export interface RemoteSdkStart {
transportVersion: 2;
modelAdmission?: { version: 1; maxEntries: number };
model?: { version: 1; publicKey: string; descriptor: HostModelDescriptor };
providerText?: { version: 1; publicKey: string; limits: HostProviderTextDescriptor };
hostTools?: HostToolDescriptor[];

View file

@ -2495,6 +2495,10 @@ export async function createRun({
user?.role,
hasModelBoundContentProtection(appConfig?.filters) ||
hasActivePiiPatterns(appConfig?.messageFilter?.pii),
sdkModelHost != null &&
modelCallbacks?.some(
(callback) => callback.name === 'librechat-model-bound-content-filter',
) === true,
);
const runFilesActive =
runFiles?.activate(

View file

@ -85,6 +85,7 @@ export function resolveSdkAgentBinding(
endpoint: RuntimeConfig | undefined,
role: string | undefined,
modelProtectionActive = false,
hostPolicySupported = false,
): SdkRegistration | undefined {
if (!agents.some((agent) => agent.sdk_runtime != null)) return undefined;
const binding = agents[0]?.sdk_runtime;
@ -100,7 +101,11 @@ export function resolveSdkAgentBinding(
if (role == null || !configured.roles.includes(role)) {
throw new SdkAgentRuntimeError('sdk_runtime_forbidden');
}
if ('remote' in configured && modelProtectionActive) {
if (
'remote' in configured &&
modelProtectionActive &&
!(configured.remote.hostModel && hostPolicySupported)
) {
throw new SdkAgentRuntimeError('sdk_runtime_policy_unsupported');
}
return 'module' in configured

View file

@ -691,6 +691,8 @@ export interface ModelBoundChatModelCallback {
}
export interface InitialModelBoundAdmissionCallback {
/** True only after the starting-root admission barrier has completed. */
readonly isAdmitted?: () => boolean;
readonly name: 'librechat-initial-model-bound-admission';
readonly raiseError: true;
readonly awaitHandlers: true;
@ -3401,6 +3403,7 @@ export function createInitialModelBoundAdmissionCallback(
};
const callback: InitialModelBoundAdmissionCallback = {
isAdmitted: () => allowed,
name: 'librechat-initial-model-bound-admission',
raiseError: true,
awaitHandlers: true,

View file

@ -120,6 +120,7 @@ export function createPrivateTextInitialAdmissionCallback(
};
return Object.freeze({
name: 'librechat-initial-model-bound-admission',
isAdmitted: () => state === 'admitted',
raiseError: true,
awaitHandlers: true,
requiredConcurrency: pending.size,

View file

@ -3534,6 +3534,7 @@ describe('remote SDK runtime configuration', () => {
});
expect(result.sdkHostModels).toEqual({
maxConcurrentCalls: 32,
maxAdmissionEntries: 4096,
maxBufferedRequestBytes: 67108864,
timeoutMs: 60000,
maxRequestBytes: 4194304,

View file

@ -1900,6 +1900,7 @@ export const agentsEndpointSchema = baseEndpointSchema
sdkHostModels: z
.object({
maxConcurrentCalls: z.number().int().min(1).max(10000).default(32),
maxAdmissionEntries: z.number().int().min(16).max(65536).default(4096),
maxBufferedRequestBytes: z.number().int().min(1024).max(1073741824).default(67108864),
timeoutMs: z.number().int().min(100).max(86400000).default(60000),
maxRequestBytes: z.number().int().min(1024).max(67108864).default(4194304),