mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 23:13:35 +00:00
🧑⚖️ feat: Preserve Input Policy and Persistence Admission for Remote Models (#18271)
This commit is contained in:
parent
dc07a93343
commit
4cd70c5af9
20 changed files with 1270 additions and 143 deletions
|
|
@ -109,8 +109,7 @@ factory must return an actual `@librechat/agents` Run with the supplied run and
|
|||
and preserve supplied handlers, hooks, checkpointer, and cancellation signal. Workers own their
|
||||
installed tools and custom graph. Model configuration and credentials remain worker-owned unless the operator enables saved-agent host models or the caller supplies the explicit host model binding below. Host model callbacks,
|
||||
provider clients, native tool registries, and arbitrary RunConfig closures do not cross the wire.
|
||||
Remote execution therefore fails closed on deployments with active model-bound content/PII
|
||||
protection until its saved-agent adapter is wired. Explicit programmatic provider-text and model bindings are described below.
|
||||
Protected chat requires the opt-in host model and the host-native model-bound callback. Callers without that callback still fail closed. Explicit programmatic provider-text and model bindings are described below.
|
||||
Stream circuit-breaker limits are forwarded to the worker.
|
||||
Host policy hooks still apply, and policy transport failures stop execution. A trusted runtime is
|
||||
executable operator code, not a sandbox for untrusted agent authors.
|
||||
|
|
@ -154,27 +153,27 @@ acceptance target remains the main chat's functionality and experience, with mea
|
|||
Using native events and the native renderer proves only the behaviors tested below. Missing
|
||||
adapters and unverified behaviors remain release gates for products that depend on them.
|
||||
|
||||
| Main-chat capability | This bridge | Remaining work / verification |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
|
||||
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
|
||||
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
|
||||
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
|
||||
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
|
||||
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
|
||||
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
|
||||
| Model/PII/provider-text protection | Explicit SDK host-service adapter supports protected provider text; saved-agent model/PII guard remains | Builder models, input/model callbacks and product policy wiring remain required before protected product rollout. |
|
||||
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
|
||||
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
|
||||
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
|
||||
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
|
||||
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
|
||||
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
|
||||
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
|
||||
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
|
||||
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
|
||||
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
|
||||
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
|
||||
| Main-chat capability | This bridge | Remaining work / verification |
|
||||
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
|
||||
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
|
||||
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
|
||||
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
|
||||
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
|
||||
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
|
||||
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
|
||||
| Model/PII/provider-text protection | Host-model chat supports input/PII callbacks and initial persistence admission; explicit provider-text transport remains available | Combining initial admission with provider-text output protection requires SDK invocation-context support. Protected Responses/OpenAI callers without native callbacks remain gated. |
|
||||
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
|
||||
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
|
||||
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
|
||||
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
|
||||
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
|
||||
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
|
||||
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
|
||||
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
|
||||
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
|
||||
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
|
||||
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
|
||||
|
||||
Do not enable a workflow on the assumption that a native chat control is honored remotely. Its
|
||||
required rows must be implemented and verified first. Keep this matrix current as adapters land;
|
||||
|
|
@ -263,13 +262,11 @@ Both sides require SDK 4.2.7 and the new bridge's `providerText: 1` capability.
|
|||
Protected runs fail before factory admission on older workers. The worker receives
|
||||
`config.providerTextProtection`; its factory must retain that exact policy on its
|
||||
returned SDK graph. Omitting it fails before model execution. Custom graphs remain
|
||||
trusted code and must install it on every applicable SDK Run. Main-chat's model/PII
|
||||
rejection in saved-agent dispatch stays in place: this output-text adapter does not
|
||||
trusted code and must install it on every applicable SDK Run. This output-text adapter alone does not
|
||||
supply input/model callbacks, authoritative builder model selection, host model
|
||||
credentials, or policy for model calls made outside the SDK. SDK 4.2.7 also provides
|
||||
`HostChatModel` and `createHostModelService` for host-owned model execution; those
|
||||
contracts now have the explicit authenticated transport below; saved-agent integration and
|
||||
product policy verification are still required before this guard can be removed.
|
||||
contracts have the authenticated transport below. The input-admission adapter and this output-protection adapter currently cannot be combined, as explained below.
|
||||
|
||||
The worker buffers a complete model attempt through the same SDK release boundary
|
||||
as local execution, then sends one `policy.providerText` RPC. Only canonical approved
|
||||
|
|
@ -364,6 +361,7 @@ endpoints:
|
|||
agents:
|
||||
sdkHostModels:
|
||||
maxConcurrentCalls: 32
|
||||
maxAdmissionEntries: 4096
|
||||
maxBufferedRequestBytes: 67108864
|
||||
timeoutMs: 60000
|
||||
maxRequestBytes: 4194304
|
||||
|
|
@ -393,10 +391,38 @@ is required to change the process-wide budgets. Existing transport frame and buf
|
|||
apply; the model byte budgets do not enlarge them. Embedders using `createRun` or
|
||||
`createAgentChatCompletion` inject the same long-lived `sdkModelHost` dependency into every caller.
|
||||
|
||||
This selects the primary model; it does not establish full main-chat parity. Active model/PII
|
||||
protection still fails closed because root admission callbacks and deferred persistence need an
|
||||
explicit remote lifecycle adapter. Local executable/graph tools and configured provider fallbacks
|
||||
also fail closed in this mode rather than being silently discarded. Context pruning, summary and
|
||||
compaction policy, child/multi-model routing, files, accounting and tracing remain separate parity
|
||||
work. Intrinsic callback execution is covered; production billing and trace ownership are not yet
|
||||
accepted end to end.
|
||||
### Initial model admission and protected chat
|
||||
|
||||
Host-model chat now retains the native model-bound input/PII callback and the existing initial
|
||||
persistence barrier. Ordinary deferred user messages become eligible for persistence only after a
|
||||
successful host-native attempt and completion of its starting graph node. Captured private text
|
||||
still commits before the provider begins; a rejected input or failed private-text commit prevents
|
||||
the provider call. Policy errors stay on the host and reach the existing local error handler; the
|
||||
worker sees only a sanitized model error. A policy rejection cannot be cleared by a worker retry.
|
||||
|
||||
The worker correlates SDK model callbacks from the starting root with encrypted model requests.
|
||||
Node completion carries IDs and a boolean, not messages or policy details. The host verifies that
|
||||
it observed a successful native attempt before accepting completion. After the host acknowledges
|
||||
admission, both sides discard correlation state and stop these lifecycle RPCs for the remainder
|
||||
of the turn. `maxAdmissionEntries` bounds the initial callback/attempt tracking per execution;
|
||||
this is distinct from the process-wide provider concurrency budget. Cancellation and connection
|
||||
failure close admission. Live connection recovery retains the same barrier and does not repeat
|
||||
provider calls or persistence.
|
||||
|
||||
Workers must advertise `modelAdmission: 1` when an initial barrier is present; older workers fail
|
||||
before graph execution. No additional browser credentials or policy callbacks cross the wire.
|
||||
The existing guard stays active unless the caller supplies the native model-bound callback and
|
||||
the registration uses the injected host model. Thus protected chat can opt in, while protected
|
||||
Responses/OpenAI-compatible callers without that callback remain gated. Local graph tools and
|
||||
provider fallbacks retain their earlier explicit rejection.
|
||||
|
||||
**Remaining limitation:** the SDK provider-text output-protection wrapper deliberately withholds
|
||||
raw callback managers from its underlying model. Until the SDK preserves a separate invocation
|
||||
identity, combining that wrapper with initial remote admission is rejected on both sides before
|
||||
graph execution. Existing programmatic output protection without an initial barrier continues to
|
||||
work. Do not disable output protection to enable this combination.
|
||||
|
||||
Before initial admission, child/summary model calls without root lineage and concurrent root
|
||||
attempts fail closed; parallel/composed admission needs its own ownership contract. This does not
|
||||
establish full main-chat parity. Context pruning, summary/compaction configuration, child/multi-model
|
||||
routing, files, production billing/tracing and Console UI acceptance remain separate work.
|
||||
|
|
|
|||
|
|
@ -169,6 +169,8 @@ Enable `endpoints.agents.sdkHostModels: {}` and `remote.hostModel: true` on the
|
|||
runtime registration to select the saved agent's primary model and parameters. Both examples
|
||||
consume `input.agent.model` and retain `input.hostModel`; model credentials stay in LibreChat.
|
||||
The custom graph keeps its plan/review/refine nodes and uses the host-authorized tool schemas.
|
||||
This mode requires updated workers with `builderModel: 1`. It currently rejects active model/PII
|
||||
protection, local graph tools and provider fallbacks; see [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
|
||||
This mode requires updated workers with `builderModel: 1` and, for initial persistence admission,
|
||||
`modelAdmission: 1`. Protected chat uses native host input callbacks. Protected API callers without
|
||||
those callbacks, local graph tools, provider fallbacks, and the combination of initial admission
|
||||
with provider-text output protection remain gated; see [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
|
||||
and [configuration](../../docs/sdk/agents.md#saved-agent-builder-models) before enabling it.
|
||||
|
|
|
|||
|
|
@ -820,9 +820,11 @@ endpoints:
|
|||
# agents:
|
||||
# # Optional primary-model execution for remote SDK agents. Budgets are per process.
|
||||
# # Restart to change budgets; workers must support builderModel: 1.
|
||||
# # Model/PII protection, local graph tools and provider fallbacks remain gated.
|
||||
# # Initial protected-chat admission also requires modelAdmission: 1 on workers.
|
||||
# # Local graph tools, provider fallbacks and combined output protection remain gated.
|
||||
# sdkHostModels:
|
||||
# maxConcurrentCalls: 32
|
||||
# maxAdmissionEntries: 4096
|
||||
# maxBufferedRequestBytes: 67108864
|
||||
# timeoutMs: 60000
|
||||
# maxRequestBytes: 4194304
|
||||
|
|
|
|||
214
packages/api/src/agents/remoteSdk/admission.spec.ts
Normal file
214
packages/api/src/agents/remoteSdk/admission.spec.ts
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
import { randomUUID } from 'node:crypto';
|
||||
import { CallbackManager, ensureHandler } from '@langchain/core/callbacks/manager';
|
||||
import {
|
||||
createInitialModelBoundAdmissionCallback,
|
||||
createModelBoundChatModelCallback,
|
||||
} from '../../middleware/modelBoundContent';
|
||||
import { createHostModelLifecycle, createWorkerModelLifecycle } from './admission';
|
||||
|
||||
function fixture() {
|
||||
const persist = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: ['root'],
|
||||
isActive: () => true,
|
||||
onAllowed: persist,
|
||||
});
|
||||
const host = createHostModelLifecycle('root', 16);
|
||||
host.attach([initial]);
|
||||
const callback = host.wrapCallbacks([]);
|
||||
if (!Array.isArray(callback)) throw new Error('Expected callbacks');
|
||||
return { persist, host, observer: callback[0] };
|
||||
}
|
||||
const attempt = () => ({ nodeId: randomUUID(), modelId: randomUUID() });
|
||||
|
||||
it('requires both an observed native success and graph-node completion before persistence', async () => {
|
||||
const { host, persist, observer } = fixture();
|
||||
const scope = attempt();
|
||||
await host.invoke(scope, async () => {
|
||||
await observer.handleChatModelStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
[],
|
||||
randomUUID(),
|
||||
);
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
});
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
host.end({ nodeId: scope.nodeId, completed: true });
|
||||
expect(persist).toHaveBeenCalledTimes(1);
|
||||
await expect(host.invoke(undefined, async () => 'subsequent call')).resolves.toBe(
|
||||
'subsequent call',
|
||||
);
|
||||
expect(() => host.end({ nodeId: scope.nodeId, completed: true })).toThrow();
|
||||
});
|
||||
|
||||
it.each(['unobserved', 'failed', 'cancelled', 'incomplete'] as const)(
|
||||
'never admits an %s attempt',
|
||||
async (outcome) => {
|
||||
const { host, persist, observer } = fixture();
|
||||
const scope = attempt();
|
||||
try {
|
||||
await host.invoke(scope, async () => {
|
||||
if (outcome !== 'unobserved')
|
||||
await observer.handleChatModelStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
[],
|
||||
randomUUID(),
|
||||
);
|
||||
if (outcome === 'failed') throw new Error('provider failure');
|
||||
if (outcome === 'cancelled') host.dispose();
|
||||
});
|
||||
} catch {
|
||||
/* expected provider failure */
|
||||
}
|
||||
if (outcome !== 'cancelled')
|
||||
host.end({ nodeId: scope.nodeId, completed: outcome !== 'incomplete' });
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it('rejects completion while provider work is pending, missing lineage and replayed attempts', async () => {
|
||||
const { host, persist } = fixture();
|
||||
const scope = attempt();
|
||||
await host.invoke(scope, async () => {
|
||||
expect(() => host.end({ nodeId: scope.nodeId, completed: true })).toThrow();
|
||||
});
|
||||
await expect(host.invoke(scope, async () => {})).rejects.toThrow();
|
||||
await expect(host.invoke(undefined, async () => {})).rejects.toThrow();
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('preserves exact messages while supplying only host-derived admission metadata to policy hooks', async () => {
|
||||
const { host } = fixture();
|
||||
const allowed = jest.fn();
|
||||
const filter = createModelBoundChatModelCallback({}, { onContentAllowed: allowed });
|
||||
const manager = new CallbackManager();
|
||||
const original = ensureHandler({
|
||||
name: filter.name,
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
handleChatModelStart: (...args) =>
|
||||
filter.handleChatModelStart(undefined, [], args[2], args[3], args[4], args[5], args[6]),
|
||||
});
|
||||
manager.addHandler(original, true);
|
||||
const wrapped = host.wrapCallbacks(manager);
|
||||
if (Array.isArray(wrapped)) throw new Error('Expected callback manager');
|
||||
const policy = wrapped.handlers.find((callback) => callback.name === filter.name)!;
|
||||
const scope = attempt();
|
||||
const modelId = randomUUID();
|
||||
await host.invoke(scope, async () => {
|
||||
await policy.handleChatModelStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
[],
|
||||
modelId,
|
||||
'untrusted-parent',
|
||||
undefined,
|
||||
undefined,
|
||||
{ agentId: 'untrusted' },
|
||||
);
|
||||
});
|
||||
expect(allowed).toHaveBeenCalledWith({
|
||||
runId: modelId,
|
||||
parentRunId: scope.nodeId,
|
||||
metadata: { agentId: 'root', langgraph_node: 'agent=root' },
|
||||
});
|
||||
expect(manager.handlers[0]).toBe(original);
|
||||
});
|
||||
|
||||
it('bounds admission state across many completed graph nodes', async () => {
|
||||
const { host } = fixture();
|
||||
for (let index = 0; index < 16; index++) {
|
||||
const scope = attempt();
|
||||
await host.invoke(scope, async () => {});
|
||||
host.end({ nodeId: scope.nodeId, completed: false });
|
||||
}
|
||||
await expect(host.invoke(attempt(), async () => {})).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('latches policy rejection even if the worker retries with safe input', async () => {
|
||||
const { host, persist } = fixture();
|
||||
const filter = createModelBoundChatModelCallback({
|
||||
filters: {
|
||||
messages: {
|
||||
pii: {
|
||||
fields: ['text'],
|
||||
starterPatterns: [],
|
||||
customPatterns: [{ id: 'private', label: 'Private', regex: 'PRIVATE-VALUE' }],
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
const policy = ensureHandler({
|
||||
name: filter.name,
|
||||
handleChatModelStart: (_model, _messages, runId, parent, extra, tags, metadata) =>
|
||||
filter.handleChatModelStart(
|
||||
undefined,
|
||||
[[{ role: 'human', content: 'PRIVATE-VALUE' }]],
|
||||
runId,
|
||||
parent,
|
||||
extra,
|
||||
tags,
|
||||
metadata,
|
||||
),
|
||||
});
|
||||
const callbacks = host.wrapCallbacks([policy]);
|
||||
if (!Array.isArray(callbacks)) throw new Error('Expected callbacks');
|
||||
const first = attempt();
|
||||
await expect(
|
||||
host.invoke(first, async () =>
|
||||
callbacks[0].handleChatModelStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
[],
|
||||
randomUUID(),
|
||||
),
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'content_filter_block' });
|
||||
const invoke = jest.fn(async () => {});
|
||||
await expect(host.invoke(attempt(), invoke)).rejects.toMatchObject({
|
||||
code: 'content_filter_block',
|
||||
});
|
||||
expect(invoke).not.toHaveBeenCalled();
|
||||
expect(() => host.end({ nodeId: first.nodeId, completed: true })).toThrow();
|
||||
expect(host.failure()).toMatchObject({ code: 'content_filter_block' });
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects concurrent root attempts instead of treating a sibling as the starting root', async () => {
|
||||
const { host } = fixture();
|
||||
await host.invoke(attempt(), async () => {
|
||||
await expect(host.invoke(attempt(), async () => {})).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
it('does not enroll summary or unrelated models into the starting graph admission', async () => {
|
||||
const call = jest.fn();
|
||||
const worker = createWorkerModelLifecycle('root', 16, { call });
|
||||
const nodeId = randomUUID();
|
||||
const callback = worker.callback;
|
||||
await callback.handleChainStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
{},
|
||||
nodeId,
|
||||
undefined,
|
||||
undefined,
|
||||
{ langgraph_node: 'agent=root' },
|
||||
undefined,
|
||||
'agent=root',
|
||||
);
|
||||
for (const metadata of [
|
||||
{ agentId: 'other', langgraph_node: 'agent=root' },
|
||||
{ agentId: 'root', langgraph_node: 'agent=root', summarization: true },
|
||||
]) {
|
||||
await callback.handleChatModelStart?.(
|
||||
{ id: [], lc: 1, type: 'not_implemented' },
|
||||
[],
|
||||
randomUUID(),
|
||||
nodeId,
|
||||
undefined,
|
||||
undefined,
|
||||
metadata,
|
||||
);
|
||||
}
|
||||
await callback.handleChainEnd?.({ messages: [{}] }, nodeId);
|
||||
expect(call).not.toHaveBeenCalled();
|
||||
expect(() => worker.current()).toThrow();
|
||||
});
|
||||
367
packages/api/src/agents/remoteSdk/admission.ts
Normal file
367
packages/api/src/agents/remoteSdk/admission.ts
Normal file
|
|
@ -0,0 +1,367 @@
|
|||
import { z } from 'zod';
|
||||
import { AsyncLocalStorage } from 'node:async_hooks';
|
||||
import { ensureHandler } from '@langchain/core/callbacks/manager';
|
||||
import { HostChatModel, HostModelServiceError } from '@librechat/agents';
|
||||
import type { BaseCallbackHandler, CallbackHandlerMethods } from '@langchain/core/callbacks/base';
|
||||
import type { CallbackManagerForLLMRun, Callbacks } from '@langchain/core/callbacks/manager';
|
||||
import type { ChatGenerationChunk } from '@langchain/core/outputs';
|
||||
import type { BaseMessage } from '@langchain/core/messages';
|
||||
import type { InitialModelBoundAdmissionCallback } from '../../middleware/modelBoundContent';
|
||||
import type { SdkRpc } from './peer';
|
||||
import { isContentFilterError } from '../../middleware/contentFilter';
|
||||
|
||||
export interface RemoteModelAttempt {
|
||||
nodeId: string;
|
||||
modelId: string;
|
||||
}
|
||||
export const modelAttemptSchema: z.ZodType<RemoteModelAttempt> = z
|
||||
.object({ nodeId: z.string().uuid(), modelId: z.string().uuid() })
|
||||
.strict();
|
||||
const nodeEndSchema = z.object({ nodeId: z.string().uuid(), completed: z.boolean() }).strict();
|
||||
const blocked = () => new HostModelServiceError('blocked');
|
||||
|
||||
/** Run each iterator operation inside its scope; constructing an async generator does not enter it. */
|
||||
async function* scopedStream<T, C>(
|
||||
storage: AsyncLocalStorage<C>,
|
||||
context: C,
|
||||
source: () => AsyncIterable<T>,
|
||||
): AsyncGenerator<T> {
|
||||
const iterator = storage.run(context, () => source()[Symbol.asyncIterator]());
|
||||
try {
|
||||
for (;;) {
|
||||
const next = await storage.run(context, () => iterator.next());
|
||||
if (next.done) return;
|
||||
yield next.value;
|
||||
}
|
||||
} finally {
|
||||
await storage.run(context, () => iterator.return?.());
|
||||
}
|
||||
}
|
||||
|
||||
type Attempt = { nodeId: string; nativeIds: Set<string> };
|
||||
export interface HostModelLifecycle {
|
||||
readonly maxEntries: number;
|
||||
attach(callbacks: Callbacks | readonly InitialModelBoundAdmissionCallback[] | undefined): boolean;
|
||||
failure(): Error | undefined;
|
||||
wrapCallbacks(callbacks: Callbacks | undefined): Callbacks;
|
||||
invoke<T>(context: RemoteModelAttempt | undefined, operation: () => Promise<T>): Promise<T>;
|
||||
stream<T>(
|
||||
context: RemoteModelAttempt | undefined,
|
||||
operation: () => AsyncIterable<T>,
|
||||
): AsyncIterable<T>;
|
||||
end(data: unknown): boolean;
|
||||
dispose(): void;
|
||||
}
|
||||
|
||||
/** Reuse the host's admission callbacks; worker completion cannot substitute for a successful native call. */
|
||||
export function createHostModelLifecycle(agentId: string, maxEntries: number): HostModelLifecycle {
|
||||
if (!Number.isSafeInteger(maxEntries) || maxEntries < 16 || maxEntries > 65536) throw blocked();
|
||||
const storage = new AsyncLocalStorage<Attempt>();
|
||||
const nodes = new Map<string, { pending: number; successful: boolean }>();
|
||||
const usedNodes = new Set<string>();
|
||||
const usedModels = new Set<string>();
|
||||
let initial: InitialModelBoundAdmissionCallback | undefined;
|
||||
let attached = false;
|
||||
let disposed = false;
|
||||
let policyFailure: Error | undefined;
|
||||
let admitted = false;
|
||||
const metadata = { agentId, langgraph_node: `agent=${agentId}` };
|
||||
const begin = (context: RemoteModelAttempt | undefined): Attempt | undefined => {
|
||||
if (disposed || policyFailure) throw policyFailure ?? blocked();
|
||||
if (!initial || admitted) return undefined;
|
||||
if (!context) throw blocked();
|
||||
const { nodeId, modelId } = modelAttemptSchema.parse(context);
|
||||
if (usedModels.has(modelId) || usedModels.size >= maxEntries) throw blocked();
|
||||
let node = nodes.get(nodeId);
|
||||
// This bridge admits one starting root. Parallel/child composition needs its own ownership contract.
|
||||
if ((nodes.size > 0 && !node) || (node && node.pending > 0)) throw blocked();
|
||||
if (!node) {
|
||||
if (usedNodes.has(nodeId) || usedNodes.size >= maxEntries) throw blocked();
|
||||
usedNodes.add(nodeId);
|
||||
node = { pending: 0, successful: false };
|
||||
nodes.set(nodeId, node);
|
||||
initial.handleChainStart(
|
||||
undefined,
|
||||
{},
|
||||
nodeId,
|
||||
undefined,
|
||||
undefined,
|
||||
metadata,
|
||||
undefined,
|
||||
metadata.langgraph_node,
|
||||
);
|
||||
}
|
||||
usedModels.add(modelId);
|
||||
node.pending++;
|
||||
return { nodeId, nativeIds: new Set() };
|
||||
};
|
||||
const finish = (attempt: Attempt | undefined, successful: boolean) => {
|
||||
if (!attempt || disposed) return;
|
||||
const node = nodes.get(attempt.nodeId);
|
||||
if (!node) throw blocked();
|
||||
node.pending--;
|
||||
node.successful ||= successful && attempt.nativeIds.size > 0;
|
||||
for (const id of attempt.nativeIds) {
|
||||
if (successful) initial?.handleLLMEnd({}, id);
|
||||
else initial?.handleLLMError(blocked(), id);
|
||||
}
|
||||
};
|
||||
const translate: NonNullable<CallbackHandlerMethods['handleChatModelStart']> = (...args) => {
|
||||
const context = storage.getStore();
|
||||
if (!initial || admitted) return;
|
||||
if (!context || disposed) throw blocked();
|
||||
const runId = args[2];
|
||||
if (!context.nativeIds.has(runId) && context.nativeIds.size >= maxEntries) throw blocked();
|
||||
context.nativeIds.add(runId);
|
||||
initial.handleChatModelStart(
|
||||
undefined,
|
||||
[],
|
||||
runId,
|
||||
context.nodeId,
|
||||
undefined,
|
||||
undefined,
|
||||
metadata,
|
||||
);
|
||||
};
|
||||
const wrap = (handler: BaseCallbackHandler | CallbackHandlerMethods) => {
|
||||
if (!('name' in handler) || handler.name !== 'librechat-model-bound-content-filter')
|
||||
return handler;
|
||||
return ensureHandler({
|
||||
name: handler.name,
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
handleChatModelStart: async (...args) => {
|
||||
try {
|
||||
if (disposed || policyFailure) throw policyFailure ?? blocked();
|
||||
if (!initial || admitted) return await handler.handleChatModelStart?.(...args);
|
||||
translate(...args);
|
||||
const context = storage.getStore();
|
||||
if (!context) throw blocked();
|
||||
return await handler.handleChatModelStart?.(
|
||||
args[0],
|
||||
args[1],
|
||||
args[2],
|
||||
context.nodeId,
|
||||
args[4],
|
||||
args[5],
|
||||
metadata,
|
||||
);
|
||||
} catch (error) {
|
||||
if (isContentFilterError(error)) policyFailure ??= error;
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
});
|
||||
};
|
||||
return {
|
||||
maxEntries,
|
||||
failure: () => policyFailure,
|
||||
attach(callbacks) {
|
||||
if (attached || disposed) throw blocked();
|
||||
attached = true;
|
||||
const matches = (
|
||||
callbacks && 'handlers' in callbacks ? callbacks.handlers : (callbacks ?? [])
|
||||
).filter(
|
||||
(handler) =>
|
||||
'name' in handler && handler.name === 'librechat-initial-model-bound-admission',
|
||||
);
|
||||
if (matches.length > 1) throw blocked();
|
||||
const candidate = matches[0];
|
||||
if (candidate) {
|
||||
for (const name of [
|
||||
'handleChainStart',
|
||||
'handleChatModelStart',
|
||||
'handleLLMEnd',
|
||||
'handleLLMError',
|
||||
'handleChainEnd',
|
||||
'handleChainError',
|
||||
] as const)
|
||||
if (typeof candidate[name] !== 'function') throw blocked();
|
||||
initial = candidate as InitialModelBoundAdmissionCallback;
|
||||
}
|
||||
return initial != null;
|
||||
},
|
||||
wrapCallbacks(callbacks) {
|
||||
const observer = ensureHandler({
|
||||
name: 'librechat-host-model-admission',
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
handleChatModelStart: translate,
|
||||
});
|
||||
if (!callbacks || Array.isArray(callbacks)) return [...(callbacks ?? []).map(wrap), observer];
|
||||
const manager = callbacks.copy();
|
||||
manager.handlers = manager.handlers.map((handler) => ensureHandler(wrap(handler)));
|
||||
manager.inheritableHandlers = manager.inheritableHandlers.map((handler) =>
|
||||
ensureHandler(wrap(handler)),
|
||||
);
|
||||
manager.addHandler(observer, true);
|
||||
return manager;
|
||||
},
|
||||
async invoke(context, operation) {
|
||||
const attempt = begin(context);
|
||||
let successful = false;
|
||||
try {
|
||||
const value = attempt ? await storage.run(attempt, operation) : await operation();
|
||||
successful = true;
|
||||
return value;
|
||||
} finally {
|
||||
finish(attempt, successful);
|
||||
}
|
||||
},
|
||||
async *stream(context, operation) {
|
||||
const attempt = begin(context);
|
||||
let successful = false;
|
||||
try {
|
||||
yield* attempt ? scopedStream(storage, attempt, operation) : operation();
|
||||
successful = true;
|
||||
} finally {
|
||||
finish(attempt, successful);
|
||||
}
|
||||
},
|
||||
end(data) {
|
||||
if (!initial || disposed || policyFailure) throw policyFailure ?? blocked();
|
||||
const { nodeId, completed } = nodeEndSchema.parse(data);
|
||||
const node = nodes.get(nodeId);
|
||||
if (!node || node.pending !== 0) throw blocked();
|
||||
nodes.delete(nodeId);
|
||||
if (completed && node.successful) initial.handleChainEnd({ messages: [{}] }, nodeId);
|
||||
else initial.handleChainError(blocked(), nodeId);
|
||||
admitted = initial.isAdmitted?.() === true;
|
||||
if (admitted) {
|
||||
usedModels.clear();
|
||||
usedNodes.clear();
|
||||
}
|
||||
return admitted;
|
||||
},
|
||||
dispose() {
|
||||
if (disposed) return;
|
||||
disposed = true;
|
||||
for (const nodeId of nodes.keys()) initial?.handleChainError(blocked(), nodeId);
|
||||
// Also rejects private admission when failure preceded the first native call.
|
||||
initial?.handleChainError(blocked(), 'remote-execution');
|
||||
nodes.clear();
|
||||
usedModels.clear();
|
||||
usedNodes.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export interface WorkerModelLifecycle {
|
||||
callback: BaseCallbackHandler;
|
||||
current(): RemoteModelAttempt | undefined;
|
||||
model(options: ConstructorParameters<typeof HostChatModel>[0]): HostChatModel;
|
||||
}
|
||||
|
||||
/** Only callbacks installed on this root execution can enroll a model attempt. No message text is relayed. */
|
||||
export function createWorkerModelLifecycle(
|
||||
agentId: string,
|
||||
maxEntries: number,
|
||||
peer: { call: SdkRpc },
|
||||
): WorkerModelLifecycle {
|
||||
const parents = new Map<string, string>();
|
||||
const nodes = new Set<string>();
|
||||
const models = new Map<string, RemoteModelAttempt>();
|
||||
const attemptedNodes = new Set<string>();
|
||||
const storage = new AsyncLocalStorage<RemoteModelAttempt | undefined>();
|
||||
let admitted = false;
|
||||
const budget = () => {
|
||||
if (parents.size + nodes.size + models.size >= maxEntries) throw blocked();
|
||||
};
|
||||
const context = (manager?: CallbackManagerForLLMRun) => {
|
||||
if (admitted) return undefined;
|
||||
const value = manager && models.get(manager.runId);
|
||||
if (!value) throw blocked();
|
||||
attemptedNodes.add(value.nodeId);
|
||||
return value;
|
||||
};
|
||||
const end = async (nodeId: string, completed: boolean) => {
|
||||
parents.delete(nodeId);
|
||||
nodes.delete(nodeId);
|
||||
for (const [modelId, attempt] of models) if (attempt.nodeId === nodeId) models.delete(modelId);
|
||||
if (!attemptedNodes.delete(nodeId)) return;
|
||||
const reply = z
|
||||
.object({ version: z.literal(1), ok: z.literal(true), admitted: z.boolean() })
|
||||
.parse(await peer.call('model.node.end', { nodeId, completed }));
|
||||
if (reply.admitted) {
|
||||
admitted = true;
|
||||
parents.clear();
|
||||
nodes.clear();
|
||||
models.clear();
|
||||
attemptedNodes.clear();
|
||||
}
|
||||
};
|
||||
class CorrelatedHostModel extends HostChatModel {
|
||||
override _generate(
|
||||
messages: BaseMessage[],
|
||||
options: this['ParsedCallOptions'],
|
||||
manager?: CallbackManagerForLLMRun,
|
||||
) {
|
||||
return storage.run(context(manager), () => super._generate(messages, options));
|
||||
}
|
||||
|
||||
override async *_streamResponseChunks(
|
||||
messages: BaseMessage[],
|
||||
options: this['ParsedCallOptions'],
|
||||
manager?: CallbackManagerForLLMRun,
|
||||
): AsyncGenerator<ChatGenerationChunk> {
|
||||
yield* scopedStream(storage, context(manager), () =>
|
||||
super._streamResponseChunks(messages, options, manager),
|
||||
);
|
||||
}
|
||||
}
|
||||
return {
|
||||
current() {
|
||||
if (admitted) return undefined;
|
||||
const value = storage.getStore();
|
||||
if (!value) throw blocked();
|
||||
return value;
|
||||
},
|
||||
model: (options) => new CorrelatedHostModel(options),
|
||||
callback: ensureHandler({
|
||||
name: 'librechat-remote-model-admission',
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
handleChainStart: (_chain, _inputs, runId, parentRunId, _tags, metadata, _type, name) => {
|
||||
if (admitted) return;
|
||||
budget();
|
||||
if (parentRunId) parents.set(runId, parentRunId);
|
||||
if (name === `agent=${agentId}` && metadata?.langgraph_node === name) nodes.add(runId);
|
||||
},
|
||||
handleChatModelStart: (_model, _messages, runId, parentRunId, _extra, _tags, metadata) => {
|
||||
if (admitted) return;
|
||||
if (
|
||||
metadata?.summarization === true ||
|
||||
metadata?.agentId !== agentId ||
|
||||
metadata.langgraph_node !== `agent=${agentId}`
|
||||
)
|
||||
return;
|
||||
const seen = new Set<string>();
|
||||
let parent = parentRunId;
|
||||
while (parent && !seen.has(parent)) {
|
||||
if (nodes.has(parent)) {
|
||||
budget();
|
||||
models.set(runId, { nodeId: parent, modelId: runId });
|
||||
return;
|
||||
}
|
||||
seen.add(parent);
|
||||
parent = parents.get(parent);
|
||||
}
|
||||
},
|
||||
handleChainEnd: async (output, runId) => {
|
||||
if (admitted) return;
|
||||
if (!nodes.has(runId)) {
|
||||
parents.delete(runId);
|
||||
return;
|
||||
}
|
||||
const result = output as { messages?: unknown; summarizationRequest?: unknown } | null;
|
||||
await end(
|
||||
runId,
|
||||
Array.isArray(result?.messages) &&
|
||||
result.messages.length > 0 &&
|
||||
result.summarizationRequest == null,
|
||||
);
|
||||
},
|
||||
handleChainError: (_error, runId) => (admitted ? undefined : end(runId, false)),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
|
@ -4,8 +4,9 @@ import { randomUUID } from 'node:crypto';
|
|||
import WebSocket, { WebSocketServer } from 'ws';
|
||||
import { tool } from '@librechat/agents/langchain/tools';
|
||||
import { ChatGenerationChunk } from '@langchain/core/outputs';
|
||||
import { agentsEndpointSchema } from 'librechat-data-provider';
|
||||
import { ensureHandler } from '@langchain/core/callbacks/manager';
|
||||
import { FakeChatModel as TestModel } from '@langchain/core/utils/testing';
|
||||
import { agentsEndpointSchema, FileSources } from 'librechat-data-provider';
|
||||
import { HumanMessage, AIMessageChunk } from '@librechat/agents/langchain/messages';
|
||||
import {
|
||||
createHostProviderTextService,
|
||||
|
|
@ -19,12 +20,29 @@ import {
|
|||
MemorySaver,
|
||||
HookRegistry,
|
||||
} from '@librechat/agents';
|
||||
import type { AppConfig, IUser } from '@librechat/data-schemas';
|
||||
import type { IUser } from '@librechat/data-schemas';
|
||||
import type { RunConfig } from '@librechat/agents';
|
||||
import type { AddressInfo } from 'node:net';
|
||||
import type { Request } from 'express';
|
||||
import type { InitialModelBoundAdmissionCallback } from '../../middleware/modelBoundContent';
|
||||
import type { RemoteSdkServerOptions } from './server';
|
||||
import type { RemoteSdkFactoryInput } from './types';
|
||||
import {
|
||||
createPrivateTextInitialAdmissionCallback,
|
||||
getPrivateTextModelHooks,
|
||||
withPrivateTextAdmissionConfig,
|
||||
} from '../../protection/private/admission';
|
||||
import {
|
||||
createPrivateTextIngress,
|
||||
stampPrivateTextMessage,
|
||||
getPrivateTextInspectionTokens,
|
||||
} from '../../protection/private/submission';
|
||||
import {
|
||||
createInitialModelBoundAdmissionCallback,
|
||||
createModelBoundChatModelCallback,
|
||||
} from '../../middleware/modelBoundContent';
|
||||
import { createSdkAgentRun, resolveSdkAgentBinding } from '../sdk';
|
||||
import { createHostModelLifecycle } from './admission';
|
||||
import { createRun as createHostRun } from '../run';
|
||||
import { createRemoteSdkServer } from './server';
|
||||
import { createSdkModelHost } from './builder';
|
||||
|
|
@ -1205,7 +1223,7 @@ describe('remote host provider-text protection', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it.each(['providerText', 'model', 'builderModel'] as const)(
|
||||
it.each(['providerText', 'model', 'builderModel', 'modelAdmission'] as const)(
|
||||
'rejects a worker without %s capability before factory admission',
|
||||
async (capability) => {
|
||||
const server = new WebSocketServer({ port: 0, host: '127.0.0.1' });
|
||||
|
|
@ -1223,7 +1241,9 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
|
|||
return {
|
||||
transportVersion: 2,
|
||||
hostTools: 2,
|
||||
...(capability === 'builderModel' ? { hostModel: 1 } : {}),
|
||||
...(['builderModel', 'modelAdmission'].includes(capability)
|
||||
? { hostModel: 1, ...(capability === 'modelAdmission' ? { builderModel: 1 } : {}) }
|
||||
: {}),
|
||||
};
|
||||
},
|
||||
}),
|
||||
|
|
@ -1249,6 +1269,9 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
|
|||
? { providerText }
|
||||
: {
|
||||
model: nativeModelBinding(),
|
||||
...(capability === 'modelAdmission'
|
||||
? { modelLifecycle: createHostModelLifecycle(identity.agentId, 16) }
|
||||
: {}),
|
||||
...(capability === 'builderModel'
|
||||
? {
|
||||
modelProfile: {
|
||||
|
|
@ -1260,9 +1283,23 @@ it.each(['providerText', 'model', 'builderModel'] as const)(
|
|||
: {}),
|
||||
},
|
||||
});
|
||||
await expect(run.processStream({ messages: [] }, invocation())).rejects.toThrow(
|
||||
'SDK runtime execution failed',
|
||||
);
|
||||
await expect(
|
||||
run.processStream(
|
||||
{ messages: [] },
|
||||
{
|
||||
...invocation(),
|
||||
callbacks: [
|
||||
admissionCallback(
|
||||
createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: () => {},
|
||||
}),
|
||||
),
|
||||
],
|
||||
},
|
||||
),
|
||||
).rejects.toThrow('SDK runtime execution failed');
|
||||
expect(methods).toEqual(['capabilities']);
|
||||
} finally {
|
||||
for (const socket of server.clients) socket.terminate();
|
||||
|
|
@ -1366,6 +1403,23 @@ describe('saved-agent builder host model', () => {
|
|||
message: new AIMessageChunk({ content: 'builder reply' }),
|
||||
});
|
||||
};
|
||||
const filters = {
|
||||
messages: {
|
||||
pii: {
|
||||
fields: ['text' as const],
|
||||
starterPatterns: [],
|
||||
customPatterns: [{ id: 'private', label: 'Private', regex: 'PRIVATE-CALLBACK' }],
|
||||
},
|
||||
},
|
||||
};
|
||||
const policy = createModelBoundChatModelCallback({ filters });
|
||||
Object.assign(native, { callbacks: [policy] });
|
||||
const persisted = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const initialize = jest.fn(() => native);
|
||||
const delta = jest.fn();
|
||||
const run = await createHostRun({
|
||||
|
|
@ -1374,7 +1428,14 @@ describe('saved-agent builder host model', () => {
|
|||
signal: new AbortController().signal,
|
||||
tokenCounter: () => 1,
|
||||
user: { id: identity.userId, role: 'ADMIN', tenantId: identity.tenantId } as IUser,
|
||||
appConfig: { endpoints: { agents: endpoint } } as AppConfig,
|
||||
appConfig: {
|
||||
config: {},
|
||||
fileStrategy: FileSources.local,
|
||||
imageOutputType: 'png',
|
||||
filters,
|
||||
endpoints: { agents: endpoint },
|
||||
},
|
||||
modelCallbacks: [policy],
|
||||
sdkModelHost: createSdkModelHost(endpoint.sdkHostModels, initialize),
|
||||
agents: [
|
||||
{
|
||||
|
|
@ -1400,7 +1461,11 @@ describe('saved-agent builder host model', () => {
|
|||
],
|
||||
customHandlers: { [GraphEvents.ON_MESSAGE_DELTA]: { handle: delta } },
|
||||
});
|
||||
await run.processStream({ messages: [new HumanMessage('hello')] }, invocation());
|
||||
await run.processStream(
|
||||
{ messages: [new HumanMessage('hello')] },
|
||||
{ ...invocation(), callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
expect(persisted).toHaveBeenCalledTimes(1);
|
||||
expect(initialize).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
provider: Providers.OPENAI,
|
||||
|
|
@ -1510,3 +1575,290 @@ describe('saved-agent builder host model', () => {
|
|||
).rejects.toMatchObject({ code: 'sdk_runtime_unavailable' });
|
||||
});
|
||||
});
|
||||
|
||||
function admissionCallback(initial: InitialModelBoundAdmissionCallback) {
|
||||
// The host callback inspects lineage only. LangChain's native-message type is wider than its policy projection.
|
||||
return ensureHandler({
|
||||
...initial,
|
||||
handleChatModelStart: (_model, _messages, runId, parent, extra, tags, metadata) =>
|
||||
initial.handleChatModelStart(undefined, [], runId, parent, extra, tags, metadata),
|
||||
});
|
||||
}
|
||||
|
||||
describe('remote native policy admission', () => {
|
||||
it('rejects combined output protection and initial admission before worker execution', async () => {
|
||||
const createRun = jest.fn(factory);
|
||||
const runtime = await fixture(createRun);
|
||||
try {
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: jest.fn(),
|
||||
});
|
||||
const run = await createRemoteSdkRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: {
|
||||
model: nativeModelBinding(),
|
||||
modelLifecycle: createHostModelLifecycle(identity.agentId, 16),
|
||||
providerText: createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 1000,
|
||||
maxAttemptBytes: 4096,
|
||||
maxBufferedBytes: 8192,
|
||||
classify: () => 'prose',
|
||||
inspect: () => ({ version: 1, ok: false, error: { code: 'blocked' } }),
|
||||
},
|
||||
}),
|
||||
},
|
||||
});
|
||||
await expect(
|
||||
run.processStream(
|
||||
{ messages: [] },
|
||||
{ ...invocation(), callbacks: [admissionCallback(initial)] },
|
||||
),
|
||||
).rejects.toThrow('SDK invocation context support');
|
||||
expect(createRun).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await runtime.close();
|
||||
}
|
||||
});
|
||||
it('does not persist a deferred message when a partially streamed model is cancelled', async () => {
|
||||
const persisted = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const abort = new AbortController();
|
||||
let entered!: () => void;
|
||||
let release!: () => void;
|
||||
const started = new Promise<void>((resolve) => {
|
||||
entered = resolve;
|
||||
});
|
||||
const pending = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
try {
|
||||
const model = new TestModel({});
|
||||
model._streamResponseChunks = async function* () {
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'partial',
|
||||
message: new AIMessageChunk({ content: 'partial' }),
|
||||
});
|
||||
entered();
|
||||
await pending;
|
||||
};
|
||||
const run = await createSdkAgentRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
signal: abort.signal,
|
||||
registration: {
|
||||
binding: { id: 'custom', protocolVersion: 1 },
|
||||
remote: { ...runtime.remote, hostModel: true },
|
||||
},
|
||||
modelHost: createSdkModelHost(
|
||||
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
|
||||
() => model,
|
||||
),
|
||||
});
|
||||
const execution = run.processStream(
|
||||
{ messages: [new HumanMessage('safe')] },
|
||||
{ ...invocation(), signal: abort.signal, callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
void execution.catch(() => {});
|
||||
await started;
|
||||
abort.abort();
|
||||
release();
|
||||
await expect(execution).rejects.toThrow('cancelled');
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
release();
|
||||
await runtime.close();
|
||||
}
|
||||
});
|
||||
it.each([false, true])(
|
||||
'admits deferred persistence only after the safe model node (blocked=%s)',
|
||||
async (blocked) => {
|
||||
const persisted = jest.fn();
|
||||
const nativeCall = jest.fn();
|
||||
const log = jest.spyOn(console, 'error').mockImplementation(() => {});
|
||||
const callback = createModelBoundChatModelCallback({
|
||||
filters: {
|
||||
messages: {
|
||||
pii: {
|
||||
fields: ['text'],
|
||||
starterPatterns: [],
|
||||
customPatterns: [{ id: 'private', label: 'private', regex: 'PRIVATE-CALLBACK' }],
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
try {
|
||||
const model = new TestModel({});
|
||||
Object.assign(model, { callbacks: [callback] });
|
||||
model._streamResponseChunks = async function* () {
|
||||
nativeCall();
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'safe reply',
|
||||
message: new AIMessageChunk({ content: 'safe reply' }),
|
||||
});
|
||||
};
|
||||
const endpoint = agentsEndpointSchema.parse({ sdkHostModels: {} });
|
||||
const run = await createSdkAgentRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
registration: {
|
||||
binding: { id: 'custom', protocolVersion: 1 },
|
||||
remote: { ...runtime.remote, hostModel: true },
|
||||
},
|
||||
modelHost: createSdkModelHost(endpoint.sdkHostModels, () => model),
|
||||
});
|
||||
const promise = run.processStream(
|
||||
{ messages: [new HumanMessage(blocked ? 'PRIVATE-CALLBACK' : 'safe input')] },
|
||||
{ ...invocation(), callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
if (blocked) {
|
||||
await expect(promise).rejects.toMatchObject({ code: 'content_filter_block' });
|
||||
expect(nativeCall).not.toHaveBeenCalled();
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
} else {
|
||||
await expect(promise).resolves.toBeDefined();
|
||||
expect(nativeCall).toHaveBeenCalledTimes(1);
|
||||
expect(persisted).toHaveBeenCalledTimes(1);
|
||||
}
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
await runtime.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([false, true])(
|
||||
'commits captured private text before provider execution (persistence failure=%s)',
|
||||
async (failPersistence) => {
|
||||
const filters = {
|
||||
messages: {
|
||||
pii: {
|
||||
action: 'redact' as const,
|
||||
fields: ['text' as const],
|
||||
starterPatterns: [],
|
||||
customPatterns: [
|
||||
{
|
||||
id: 'email',
|
||||
label: 'Email',
|
||||
regex: 'alice@example\\.com',
|
||||
category: 'email' as const,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
const req = {
|
||||
path: '/',
|
||||
user: { id: identity.userId },
|
||||
body: { text: 'alice@example.com', clientRequestId: 'private-remote' },
|
||||
} as unknown as Request;
|
||||
createPrivateTextIngress({
|
||||
getFilters: () => filters,
|
||||
getLegacyPii: () => undefined,
|
||||
getKey: () => 'ab'.repeat(32),
|
||||
})(req, {} as never, jest.fn());
|
||||
const message = stampPrivateTextMessage(req, {
|
||||
messageId: 'user',
|
||||
conversationId: identity.conversationId,
|
||||
text: req.body.text,
|
||||
isCreatedByUser: true,
|
||||
});
|
||||
const persisted = jest.fn(async () => {
|
||||
if (failPersistence) throw new Error('persistence unavailable');
|
||||
return { message };
|
||||
});
|
||||
const cancelled = jest.fn();
|
||||
const created = jest.fn();
|
||||
const nativeCall = jest.fn();
|
||||
const initial = createPrivateTextInitialAdmissionCallback(req, {
|
||||
agentIds: [identity.agentId],
|
||||
start: persisted,
|
||||
cancel: cancelled,
|
||||
onPersisted: created,
|
||||
})!;
|
||||
const callback = createModelBoundChatModelCallback(
|
||||
{
|
||||
filters,
|
||||
storedMessages: [message],
|
||||
privateTextTokens: getPrivateTextInspectionTokens([message]),
|
||||
},
|
||||
getPrivateTextModelHooks(req, initial, persisted, cancelled, created),
|
||||
);
|
||||
const log = jest.spyOn(console, 'error').mockImplementation(() => {});
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
try {
|
||||
const model = new TestModel({});
|
||||
Object.assign(model, { callbacks: [callback] });
|
||||
model._streamResponseChunks = async function* () {
|
||||
expect(created).toHaveBeenCalledTimes(1);
|
||||
nativeCall();
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'safe reply',
|
||||
message: new AIMessageChunk({ content: 'safe reply' }),
|
||||
});
|
||||
};
|
||||
const run = await createSdkAgentRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
registration: {
|
||||
binding: { id: 'custom', protocolVersion: 1 },
|
||||
remote: { ...runtime.remote, hostModel: true },
|
||||
},
|
||||
modelHost: createSdkModelHost(
|
||||
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
|
||||
() => model,
|
||||
),
|
||||
});
|
||||
const promise = run.processStream(
|
||||
{ messages: [new HumanMessage(req.body.text)] },
|
||||
{
|
||||
...withPrivateTextAdmissionConfig(invocation(), initial),
|
||||
callbacks: [admissionCallback(initial)],
|
||||
},
|
||||
);
|
||||
if (failPersistence) {
|
||||
await expect(promise).rejects.toThrow('SDK runtime execution failed');
|
||||
expect(nativeCall).not.toHaveBeenCalled();
|
||||
expect(cancelled).toHaveBeenCalled();
|
||||
} else {
|
||||
await expect(promise).resolves.toBeDefined();
|
||||
expect(nativeCall).toHaveBeenCalledTimes(1);
|
||||
}
|
||||
expect(persisted).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
await runtime.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import type { TAgentsEndpoint } from 'librechat-data-provider';
|
|||
import type { RunConfig } from '@librechat/agents';
|
||||
import type { RemoteSdkHostServices } from './types';
|
||||
import type { SdkAgentIdentity } from './identity';
|
||||
import { createHostModelLifecycle } from './admission';
|
||||
|
||||
type Limits = NonNullable<NonNullable<TAgentsEndpoint>['sdkHostModels']>;
|
||||
|
||||
|
|
@ -62,12 +63,15 @@ export function createSdkModelHost(
|
|||
signal?.throwIfAborted();
|
||||
const model = createModel({ provider: agent.provider, clientOptions: agent.clientOptions });
|
||||
if (!(model instanceof BaseChatModel)) throw new SdkHostModelError();
|
||||
const modelLifecycle = createHostModelLifecycle(identity.agentId, limits.maxAdmissionEntries);
|
||||
model.callbacks = modelLifecycle.wrapCallbacks(model.callbacks);
|
||||
const modelProfile = {
|
||||
provider: agent.provider,
|
||||
name: modelName,
|
||||
streaming: agent.clientOptions?.streaming !== false,
|
||||
};
|
||||
return {
|
||||
modelLifecycle,
|
||||
modelProfile,
|
||||
model: createHostModelService({
|
||||
identity,
|
||||
|
|
|
|||
|
|
@ -200,8 +200,15 @@ export async function createRemoteSdkRun({
|
|||
|
||||
return socket;
|
||||
};
|
||||
const socket = remote.recovery
|
||||
? await connectRecoveringSdkSocket({
|
||||
const modelAdmission = hostServices?.modelLifecycle?.attach(options.callbacks) === true;
|
||||
if (modelAdmission && hostServices?.providerText) {
|
||||
hostServices.modelLifecycle?.dispose();
|
||||
throw new Error(
|
||||
'Remote model admission with provider-text protection requires SDK invocation context support',
|
||||
);
|
||||
}
|
||||
const connection = remote.recovery
|
||||
? connectRecoveringSdkSocket({
|
||||
url: remote.url,
|
||||
token: makeToken,
|
||||
generation:
|
||||
|
|
@ -213,7 +220,11 @@ export async function createRemoteSdkRun({
|
|||
signal: options.signal,
|
||||
onError: remote.onError,
|
||||
})
|
||||
: await legacy();
|
||||
: legacy();
|
||||
const socket = await connection.catch((error) => {
|
||||
hostServices?.modelLifecycle?.dispose();
|
||||
throw error;
|
||||
});
|
||||
const execution = new AbortController();
|
||||
const providerText = hostServices?.providerText
|
||||
? createHostPolicyChannel(hostServices.providerText, limits)
|
||||
|
|
@ -227,17 +238,19 @@ export async function createRemoteSdkRun({
|
|||
const finish = (value?: RemoteSdkSnapshot) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
if (value == null)
|
||||
if (value == null || hostServices?.modelLifecycle?.failure())
|
||||
reject(
|
||||
options.signal?.aborted
|
||||
? Object.assign(new Error('SDK runtime execution cancelled'), { name: 'AbortError' })
|
||||
: new Error('SDK runtime execution failed'),
|
||||
: (hostServices?.modelLifecycle?.failure() ??
|
||||
new Error('SDK runtime execution failed')),
|
||||
);
|
||||
else resolve(value);
|
||||
};
|
||||
hostModelChannel = hostServices?.model
|
||||
? createHostModelChannel({
|
||||
model: hostServices.model,
|
||||
lifecycle: modelAdmission ? hostServices.modelLifecycle : undefined,
|
||||
peer: { call: (...args) => peer.call(...args) },
|
||||
limits,
|
||||
signal: execution.signal,
|
||||
|
|
@ -280,6 +293,12 @@ export async function createRemoteSdkRun({
|
|||
onCall: async (method, data, _timeoutMs, signal) => {
|
||||
if (execution.signal.aborted || options.signal?.aborted)
|
||||
throw new Error('SDK runtime execution stopped');
|
||||
if (method === 'model.node.end') {
|
||||
if (!modelAdmission || !hostServices?.modelLifecycle)
|
||||
throw new Error('Model admission unavailable');
|
||||
const admitted = hostServices.modelLifecycle.end(data);
|
||||
return { version: 1, ok: true, admitted };
|
||||
}
|
||||
if (method === 'model.invoke' || method === 'model.stream') {
|
||||
if (!hostModelChannel) throw new Error('Host model is unavailable');
|
||||
const invocation = hostModelChannel.call(method, data, signal);
|
||||
|
|
@ -423,6 +442,9 @@ export async function createRemoteSdkRun({
|
|||
'agents' in config.graphConfig ? config.graphConfig.agents[0] : config.graphConfig;
|
||||
const start: RemoteSdkStart = {
|
||||
transportVersion: 2,
|
||||
modelAdmission: modelAdmission
|
||||
? { version: 1, maxEntries: hostServices!.modelLifecycle!.maxEntries }
|
||||
: undefined,
|
||||
model: hostModelChannel?.descriptor,
|
||||
providerText: providerText?.descriptor,
|
||||
hostTools: descriptors.length ? descriptors : undefined,
|
||||
|
|
@ -471,9 +493,12 @@ export async function createRemoteSdkRun({
|
|||
providerText?: number;
|
||||
hostModel?: number;
|
||||
builderModel?: number;
|
||||
modelAdmission?: number;
|
||||
};
|
||||
if (capabilities?.transportVersion !== 2)
|
||||
throw new Error('Worker transport capability unavailable');
|
||||
if (modelAdmission && capabilities?.modelAdmission !== 1)
|
||||
throw new Error('Worker model admission capability unavailable');
|
||||
if (hostServices?.modelProfile && capabilities?.builderModel !== 1)
|
||||
throw new Error('Worker builder model capability unavailable');
|
||||
if (hostServices?.model && capabilities?.hostModel !== 1)
|
||||
|
|
@ -493,6 +518,7 @@ export async function createRemoteSdkRun({
|
|||
return snapshot.content?.filter((part) => part != null);
|
||||
} finally {
|
||||
execution.abort();
|
||||
hostServices?.modelLifecycle?.dispose();
|
||||
providerText?.dispose();
|
||||
hostModelChannel?.dispose();
|
||||
// A failed/cancelled remote run may leave host cleanup running. Keep the host
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { z } from 'zod';
|
||||
import { once } from 'node:events';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import WebSocket, { WebSocketServer } from 'ws';
|
||||
|
|
@ -9,8 +10,8 @@ import type { HostModelDescriptor, HostModelRequest } from '@librechat/agents';
|
|||
import type { AddressInfo } from 'node:net';
|
||||
import { createHostModelChannel, createWorkerModelChannel } from './model';
|
||||
import { createPolicyReceiver, createPolicySender } from './sealed';
|
||||
import { encodeSdkValue, decodeSdkValue } from './wire';
|
||||
import { remoteSdkLimits } from './types';
|
||||
import { encodeSdkValue } from './wire';
|
||||
import { createSdkPeer } from './peer';
|
||||
|
||||
const identity = {
|
||||
|
|
@ -106,6 +107,42 @@ async function fixture(model = new FakeChatModel({}), tools: HostModelDescriptor
|
|||
}
|
||||
const tick = () => new Promise<void>((resolve) => setTimeout(resolve, 10));
|
||||
|
||||
it('preserves the strict older-host request shape without negotiated admission', async () => {
|
||||
const receiver = createPolicyReceiver(limits.maxFrameBytes, 'model');
|
||||
const controller = new AbortController();
|
||||
const call = jest.fn(async (_method: string, data: unknown) => {
|
||||
const operation = z
|
||||
.object({ id: z.string().uuid(), publicKey: z.string().length(60), request: z.unknown() })
|
||||
.strict()
|
||||
.parse(decodeSdkValue(JSON.parse(receiver.open(data))));
|
||||
const seal = createPolicySender(operation.publicKey, limits.maxFrameBytes, 'model');
|
||||
return {
|
||||
version: 1,
|
||||
ok: true,
|
||||
message: seal(
|
||||
JSON.stringify(
|
||||
encodeSdkValue({ id: operation.id, message: new AIMessage('legacy reply') }),
|
||||
),
|
||||
),
|
||||
};
|
||||
});
|
||||
const worker = createWorkerModelChannel({ peer: { call }, limits, signal: controller.signal });
|
||||
try {
|
||||
const model = worker.bind(identity, {
|
||||
version: 1,
|
||||
publicKey: receiver.publicKey,
|
||||
descriptor,
|
||||
});
|
||||
await expect(model.invoke([...request.messages])).resolves.toMatchObject({
|
||||
content: 'legacy reply',
|
||||
});
|
||||
expect(call).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
controller.abort();
|
||||
receiver.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
it('streams native tool fragments, rich content and typed metadata over a real socket', async () => {
|
||||
const native = new FakeChatModel({});
|
||||
native._streamResponseChunks = async function* () {
|
||||
|
|
|
|||
|
|
@ -9,13 +9,20 @@ import type {
|
|||
RemoteSdkLimits,
|
||||
RemoteSdkStart,
|
||||
} from './types';
|
||||
import type { HostModelLifecycle, WorkerModelLifecycle } from './admission';
|
||||
import type { SdkAgentIdentity } from './identity';
|
||||
import type { SdkRpc } from './peer';
|
||||
import { createPolicyReceiver, createPolicySender } from './sealed';
|
||||
import { encodeSdkValue, decodeSdkValue } from './wire';
|
||||
import { modelAttemptSchema } from './admission';
|
||||
|
||||
const operationSchema = z
|
||||
.object({ id: z.string().uuid(), publicKey: z.string().length(60), request: z.unknown() })
|
||||
.object({
|
||||
id: z.string().uuid(),
|
||||
publicKey: z.string().length(60),
|
||||
request: z.unknown(),
|
||||
context: modelAttemptSchema.optional(),
|
||||
})
|
||||
.strict();
|
||||
const chunkSchema = z
|
||||
.object({ id: z.string().uuid(), index: z.number().int().nonnegative(), message: z.unknown() })
|
||||
|
|
@ -61,6 +68,7 @@ function sealValue(seal: ReturnType<typeof createPolicySender>, value: unknown,
|
|||
/** One byte window shared by every model stream on this execution's connection. */
|
||||
export function createHostModelChannel(options: {
|
||||
model: NonNullable<RemoteSdkHostServices['model']>;
|
||||
lifecycle?: HostModelLifecycle;
|
||||
peer: { call: SdkRpc };
|
||||
limits: RemoteSdkLimits;
|
||||
signal: AbortSignal;
|
||||
|
|
@ -91,7 +99,10 @@ export function createHostModelChannel(options: {
|
|||
// The SDK binding validates identity, reference, messages and allowed tools before admission.
|
||||
const request = parsed.data.request as HostModelRequest;
|
||||
if (method === 'model.invoke') {
|
||||
const message = await options.model.service.invoke(request, boundSignal);
|
||||
const invoke = () => options.model.service.invoke(request, boundSignal);
|
||||
const message = await (options.lifecycle
|
||||
? options.lifecycle.invoke(parsed.data.context, invoke)
|
||||
: invoke());
|
||||
return {
|
||||
version: 1,
|
||||
ok: true,
|
||||
|
|
@ -100,7 +111,11 @@ export function createHostModelChannel(options: {
|
|||
}
|
||||
if (method !== 'model.stream') throw new HostModelServiceError('incompatible');
|
||||
let index = 0;
|
||||
for await (const chunk of options.model.service.stream(request, boundSignal)) {
|
||||
const stream = () => options.model.service.stream(request, boundSignal);
|
||||
const source = options.lifecycle
|
||||
? options.lifecycle.stream(parsed.data.context, stream)
|
||||
: stream();
|
||||
for await (const chunk of source) {
|
||||
boundSignal.throwIfAborted();
|
||||
if (deliveryError) throw deliveryError;
|
||||
const payload = sealValue(
|
||||
|
|
@ -195,11 +210,17 @@ export function createWorkerModelChannel(options: {
|
|||
const streams = new Map<string, ModelStream>();
|
||||
let receiver: ReturnType<typeof createPolicyReceiver> | undefined;
|
||||
let seal: ReturnType<typeof createPolicySender> | undefined;
|
||||
let lifecycle: WorkerModelLifecycle | undefined;
|
||||
const encodeRequest = (id: string, request: HostModelRequest) => {
|
||||
if (!receiver || !seal) throw new HostModelServiceError('incompatible');
|
||||
return sealValue(
|
||||
seal,
|
||||
{ id, publicKey: receiver.publicKey, request },
|
||||
{
|
||||
id,
|
||||
publicKey: receiver.publicKey,
|
||||
request,
|
||||
...(lifecycle && { context: lifecycle.current() }),
|
||||
},
|
||||
options.limits.maxFrameBytes,
|
||||
);
|
||||
};
|
||||
|
|
@ -300,12 +321,15 @@ export function createWorkerModelChannel(options: {
|
|||
bind: (
|
||||
identity: Readonly<SdkAgentIdentity>,
|
||||
descriptor: NonNullable<RemoteSdkStart['model']>,
|
||||
admission?: WorkerModelLifecycle,
|
||||
): NonNullable<RemoteSdkFactoryInput['hostModel']> => {
|
||||
if (receiver || options.signal.aborted || descriptor.version !== 1)
|
||||
throw new HostModelServiceError('incompatible');
|
||||
lifecycle = admission;
|
||||
receiver = createPolicyReceiver(options.limits.maxFrameBytes, 'model');
|
||||
seal = createPolicySender(descriptor.publicKey, options.limits.maxFrameBytes, 'model');
|
||||
return new HostChatModel({ identity, descriptor: descriptor.descriptor, service }).bindTools([
|
||||
const config = { identity, descriptor: descriptor.descriptor, service };
|
||||
return (lifecycle ? lifecycle.model(config) : new HostChatModel(config)).bindTools([
|
||||
...descriptor.descriptor.tools,
|
||||
]);
|
||||
},
|
||||
|
|
|
|||
|
|
@ -30,6 +30,12 @@ export type SdkFrame =
|
|||
| { type: 'done'; data: unknown }
|
||||
| { type: 'error' };
|
||||
|
||||
export interface SdkPeer {
|
||||
call: SdkRpc;
|
||||
send(frame: SdkFrame): void;
|
||||
close(): void;
|
||||
}
|
||||
|
||||
const frameSchema = z.discriminatedUnion('type', [
|
||||
z.object({ type: z.literal('cancel'), id: z.string().uuid() }),
|
||||
z.object({
|
||||
|
|
@ -67,7 +73,7 @@ export function createSdkPeer(
|
|||
onClose: (error: Error) => void;
|
||||
onError?: (error: unknown) => void;
|
||||
},
|
||||
) {
|
||||
): SdkPeer {
|
||||
const pending = new Map<
|
||||
string,
|
||||
{ resolve: (data: unknown) => void; reject: (error: Error) => void }
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import { request as httpRequest } from 'node:http';
|
|||
import { tool } from '@librechat/agents/langchain/tools';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { ChatGenerationChunk } from '@langchain/core/outputs';
|
||||
import { ensureHandler } from '@langchain/core/callbacks/manager';
|
||||
import { FakeChatModel as TestModel } from '@langchain/core/utils/testing';
|
||||
import { HumanMessage, AIMessageChunk } from '@librechat/agents/langchain/messages';
|
||||
import {
|
||||
|
|
@ -28,6 +29,8 @@ import {
|
|||
createRedisTestClient,
|
||||
clearRedisTestPrefix,
|
||||
} from '../../../stream/__tests__/helpers/redis';
|
||||
import { createInitialModelBoundAdmissionCallback } from '../../../middleware/modelBoundContent';
|
||||
import { createHostModelLifecycle } from '../admission';
|
||||
import { createRedisRuntimeJournal } from './redis';
|
||||
import { createRemoteSdkServer } from '../server';
|
||||
import { createRuntimeRecovery } from './server';
|
||||
|
|
@ -1020,89 +1023,113 @@ it('recovers a policy reply after a wire outage without repeating inspection or
|
|||
}
|
||||
});
|
||||
|
||||
it('resumes host model chunks after disconnect without another provider invocation', async () => {
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
const native = new TestModel({});
|
||||
let invocations = 0;
|
||||
let hostStarts = 0;
|
||||
native._streamResponseChunks = async function* () {
|
||||
invocations++;
|
||||
yield new ChatGenerationChunk({ text: 'first ', message: new AIMessageChunk('first ') });
|
||||
runtime.drop();
|
||||
await delay(50);
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'PRIVATE-CANDIDATE ',
|
||||
message: new AIMessageChunk('PRIVATE-CANDIDATE '),
|
||||
it.each([false, true])(
|
||||
'resumes host model chunks without repeating the provider (input admission=%s)',
|
||||
async (inputAdmission) => {
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
yield new ChatGenerationChunk({ text: 'third', message: new AIMessageChunk('third') });
|
||||
};
|
||||
const admission = new HostModelAdmission({ maxConcurrentCalls: 1, maxRequestBytes: 65536 });
|
||||
const model = createHostModelService({
|
||||
identity,
|
||||
model: native,
|
||||
admission,
|
||||
descriptor: {
|
||||
version: 1,
|
||||
reference: 'primary',
|
||||
provider: Providers.OPENAI,
|
||||
model: 'native',
|
||||
streaming: true,
|
||||
tools: [],
|
||||
},
|
||||
limits: { timeoutMs: 5000, maxRequestBytes: 65536, maxResponseBytes: 65536 },
|
||||
config: {
|
||||
callbacks: [
|
||||
{
|
||||
handleChatModelStart: () => {
|
||||
hostStarts++;
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const providerText = createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 5000,
|
||||
maxAttemptBytes: 65536,
|
||||
maxBufferedBytes: 131072,
|
||||
classify: () => 'prose',
|
||||
inspect: ({ content }) => ({
|
||||
version: 1,
|
||||
ok: true,
|
||||
value: {
|
||||
content: content.replaceAll('PRIVATE-CANDIDATE', 'approved'),
|
||||
replacements: 1,
|
||||
categories: [],
|
||||
},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const run = await createRemoteSdkRun({
|
||||
config: hostConfig(async () => {}),
|
||||
const native = new TestModel({});
|
||||
let invocations = 0;
|
||||
let hostStarts = 0;
|
||||
const persisted = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const modelLifecycle = createHostModelLifecycle(identity.agentId, 64);
|
||||
if (inputAdmission) native.callbacks = modelLifecycle.wrapCallbacks(native.callbacks);
|
||||
native._streamResponseChunks = async function* () {
|
||||
invocations++;
|
||||
yield new ChatGenerationChunk({ text: 'first ', message: new AIMessageChunk('first ') });
|
||||
runtime.drop();
|
||||
await delay(50);
|
||||
yield new ChatGenerationChunk({
|
||||
text: inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE ',
|
||||
message: new AIMessageChunk(inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE '),
|
||||
});
|
||||
yield new ChatGenerationChunk({ text: 'third', message: new AIMessageChunk('third') });
|
||||
};
|
||||
const admission = new HostModelAdmission({ maxConcurrentCalls: 1, maxRequestBytes: 65536 });
|
||||
const model = createHostModelService({
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: { model, providerText },
|
||||
model: native,
|
||||
admission,
|
||||
descriptor: {
|
||||
version: 1,
|
||||
reference: 'primary',
|
||||
provider: Providers.OPENAI,
|
||||
model: 'native',
|
||||
streaming: true,
|
||||
tools: [],
|
||||
},
|
||||
limits: { timeoutMs: 5000, maxRequestBytes: 65536, maxResponseBytes: 65536 },
|
||||
config: {
|
||||
callbacks: [
|
||||
{
|
||||
handleChatModelStart: () => {
|
||||
hostStarts++;
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const content = await run.processStream(
|
||||
{ messages: [new HumanMessage('go')] },
|
||||
{ version: 'v2', configurable: { thread_id: identity.conversationId } },
|
||||
);
|
||||
expect(JSON.stringify(content)).toContain('first approved third');
|
||||
expect(runtime.journalEntries.join('')).not.toContain('PRIVATE-CANDIDATE');
|
||||
expect(invocations).toBe(1);
|
||||
expect(hostStarts).toBe(1);
|
||||
expect(runtime.connections()).toBeGreaterThan(1);
|
||||
expect(runtime.createRun).toHaveBeenCalledTimes(1);
|
||||
expect(runtime.errors).toEqual([]);
|
||||
expect(admission.snapshot().calls).toBe(0);
|
||||
} finally {
|
||||
await runtime.close();
|
||||
}
|
||||
});
|
||||
const providerText = createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 5000,
|
||||
maxAttemptBytes: 65536,
|
||||
maxBufferedBytes: 131072,
|
||||
classify: () => 'prose',
|
||||
inspect: ({ content }) => ({
|
||||
version: 1,
|
||||
ok: true,
|
||||
value: {
|
||||
content: content.replaceAll('PRIVATE-CANDIDATE', 'approved'),
|
||||
replacements: 1,
|
||||
categories: [],
|
||||
},
|
||||
}),
|
||||
},
|
||||
});
|
||||
try {
|
||||
const run = await createRemoteSdkRun({
|
||||
config: hostConfig(async () => {}),
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: { model, ...(inputAdmission ? { modelLifecycle } : { providerText }) },
|
||||
});
|
||||
const content = await run.processStream(
|
||||
{ messages: [new HumanMessage('go')] },
|
||||
{
|
||||
version: 'v2',
|
||||
configurable: { thread_id: identity.conversationId },
|
||||
callbacks: inputAdmission
|
||||
? [
|
||||
ensureHandler({
|
||||
...initial,
|
||||
handleChatModelStart: (_llm, _messages, id, parent, extra, tags, metadata) =>
|
||||
initial.handleChatModelStart(undefined, [], id, parent, extra, tags, metadata),
|
||||
}),
|
||||
]
|
||||
: undefined,
|
||||
},
|
||||
);
|
||||
expect(JSON.stringify(content)).toContain('first approved third');
|
||||
expect(runtime.journalEntries.join('')).not.toContain('PRIVATE-CANDIDATE');
|
||||
expect(invocations).toBe(1);
|
||||
expect(persisted).toHaveBeenCalledTimes(inputAdmission ? 1 : 0);
|
||||
expect(hostStarts).toBe(1);
|
||||
expect(runtime.connections()).toBeGreaterThan(1);
|
||||
expect(runtime.createRun).toHaveBeenCalledTimes(1);
|
||||
expect(runtime.errors).toEqual([]);
|
||||
expect(admission.snapshot().calls).toBe(0);
|
||||
} finally {
|
||||
await runtime.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ import { policyDescriptorSchema, createWorkerProviderTextPolicy } from './policy
|
|||
import { createRuntimeRecovery, recoveryClaimSchema } from './recovery/server';
|
||||
import { remoteSdkLimits, isRemoteSdkSecret } from './types';
|
||||
import { RECOVERY_HEADER_BYTES } from './recovery/socket';
|
||||
import { createWorkerModelLifecycle } from './admission';
|
||||
import { createWorkerToolService } from './hostTools';
|
||||
import { RemoteSdkCheckpointer } from './checkpoint';
|
||||
import { createWorkerModelChannel } from './model';
|
||||
|
|
@ -51,6 +52,10 @@ const claimSchema = z.object({
|
|||
});
|
||||
const startSchema = z.object({
|
||||
transportVersion: z.literal(2),
|
||||
modelAdmission: z
|
||||
.object({ version: z.literal(1), maxEntries: z.number().int().min(16).max(65536) })
|
||||
.strict()
|
||||
.optional(),
|
||||
// The SDK HostChatModel constructor owns descriptor validation.
|
||||
model: z
|
||||
.object({
|
||||
|
|
@ -292,6 +297,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
providerText: 1,
|
||||
hostModel: 1,
|
||||
builderModel: 1,
|
||||
modelAdmission: 1,
|
||||
};
|
||||
if (method === 'model.chunk') return workerModel.accept(data, signal);
|
||||
if (method === 'tool.result') return workerTools.accept(data);
|
||||
|
|
@ -299,6 +305,10 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
if (method !== 'run' || started) throw new Error('SDK execution already admitted');
|
||||
started = true;
|
||||
const request = startSchema.parse(data) as RemoteSdkStart;
|
||||
if (request.modelAdmission && request.providerText)
|
||||
throw new Error(
|
||||
'Combined model admission and provider-text protection is unsupported',
|
||||
);
|
||||
const identity = Object.freeze(claim.identity);
|
||||
if (
|
||||
request.agent.id !== identity.agentId ||
|
||||
|
|
@ -374,7 +384,18 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
limits,
|
||||
})
|
||||
: undefined;
|
||||
const hostModel = request.model ? workerModel.bind(identity, request.model) : undefined;
|
||||
if (request.modelAdmission && !request.model)
|
||||
throw new Error('Model admission requires a host model');
|
||||
const modelLifecycle = request.modelAdmission
|
||||
? createWorkerModelLifecycle(
|
||||
identity.agentId,
|
||||
request.modelAdmission.maxEntries,
|
||||
peer,
|
||||
)
|
||||
: undefined;
|
||||
const hostModel = request.model
|
||||
? workerModel.bind(identity, request.model, modelLifecycle)
|
||||
: undefined;
|
||||
const profile = request.agent.model;
|
||||
if (
|
||||
profile &&
|
||||
|
|
@ -439,6 +460,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
}
|
||||
const callerOptions = {
|
||||
...request.options,
|
||||
...(modelLifecycle && { callbacks: [modelLifecycle.callback] }),
|
||||
version: 'v2' as const,
|
||||
signal: controller.signal,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ import type {
|
|||
IState,
|
||||
HookEvent,
|
||||
} from '@librechat/agents';
|
||||
import type { HostModelLifecycle } from './admission';
|
||||
import type { SdkAgentIdentity } from './identity';
|
||||
|
||||
export interface RemoteSdkRecoveryOptions {
|
||||
|
|
@ -68,6 +69,7 @@ export interface RemoteSdkModelProfile {
|
|||
|
||||
/** Explicit host-owned capabilities; never reconstructed from worker claims or YAML. */
|
||||
export interface RemoteSdkHostServices {
|
||||
modelLifecycle?: HostModelLifecycle;
|
||||
modelProfile?: RemoteSdkModelProfile;
|
||||
model?: ReturnType<typeof createHostModelService>;
|
||||
providerText?: ReturnType<typeof createHostProviderTextService>;
|
||||
|
|
@ -75,6 +77,7 @@ export interface RemoteSdkHostServices {
|
|||
|
||||
export interface RemoteSdkStart {
|
||||
transportVersion: 2;
|
||||
modelAdmission?: { version: 1; maxEntries: number };
|
||||
model?: { version: 1; publicKey: string; descriptor: HostModelDescriptor };
|
||||
providerText?: { version: 1; publicKey: string; limits: HostProviderTextDescriptor };
|
||||
hostTools?: HostToolDescriptor[];
|
||||
|
|
|
|||
|
|
@ -2495,6 +2495,10 @@ export async function createRun({
|
|||
user?.role,
|
||||
hasModelBoundContentProtection(appConfig?.filters) ||
|
||||
hasActivePiiPatterns(appConfig?.messageFilter?.pii),
|
||||
sdkModelHost != null &&
|
||||
modelCallbacks?.some(
|
||||
(callback) => callback.name === 'librechat-model-bound-content-filter',
|
||||
) === true,
|
||||
);
|
||||
const runFilesActive =
|
||||
runFiles?.activate(
|
||||
|
|
|
|||
|
|
@ -85,6 +85,7 @@ export function resolveSdkAgentBinding(
|
|||
endpoint: RuntimeConfig | undefined,
|
||||
role: string | undefined,
|
||||
modelProtectionActive = false,
|
||||
hostPolicySupported = false,
|
||||
): SdkRegistration | undefined {
|
||||
if (!agents.some((agent) => agent.sdk_runtime != null)) return undefined;
|
||||
const binding = agents[0]?.sdk_runtime;
|
||||
|
|
@ -100,7 +101,11 @@ export function resolveSdkAgentBinding(
|
|||
if (role == null || !configured.roles.includes(role)) {
|
||||
throw new SdkAgentRuntimeError('sdk_runtime_forbidden');
|
||||
}
|
||||
if ('remote' in configured && modelProtectionActive) {
|
||||
if (
|
||||
'remote' in configured &&
|
||||
modelProtectionActive &&
|
||||
!(configured.remote.hostModel && hostPolicySupported)
|
||||
) {
|
||||
throw new SdkAgentRuntimeError('sdk_runtime_policy_unsupported');
|
||||
}
|
||||
return 'module' in configured
|
||||
|
|
|
|||
|
|
@ -691,6 +691,8 @@ export interface ModelBoundChatModelCallback {
|
|||
}
|
||||
|
||||
export interface InitialModelBoundAdmissionCallback {
|
||||
/** True only after the starting-root admission barrier has completed. */
|
||||
readonly isAdmitted?: () => boolean;
|
||||
readonly name: 'librechat-initial-model-bound-admission';
|
||||
readonly raiseError: true;
|
||||
readonly awaitHandlers: true;
|
||||
|
|
@ -3401,6 +3403,7 @@ export function createInitialModelBoundAdmissionCallback(
|
|||
};
|
||||
|
||||
const callback: InitialModelBoundAdmissionCallback = {
|
||||
isAdmitted: () => allowed,
|
||||
name: 'librechat-initial-model-bound-admission',
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
|
|
|
|||
|
|
@ -120,6 +120,7 @@ export function createPrivateTextInitialAdmissionCallback(
|
|||
};
|
||||
return Object.freeze({
|
||||
name: 'librechat-initial-model-bound-admission',
|
||||
isAdmitted: () => state === 'admitted',
|
||||
raiseError: true,
|
||||
awaitHandlers: true,
|
||||
requiredConcurrency: pending.size,
|
||||
|
|
|
|||
|
|
@ -3534,6 +3534,7 @@ describe('remote SDK runtime configuration', () => {
|
|||
});
|
||||
expect(result.sdkHostModels).toEqual({
|
||||
maxConcurrentCalls: 32,
|
||||
maxAdmissionEntries: 4096,
|
||||
maxBufferedRequestBytes: 67108864,
|
||||
timeoutMs: 60000,
|
||||
maxRequestBytes: 4194304,
|
||||
|
|
|
|||
|
|
@ -1900,6 +1900,7 @@ export const agentsEndpointSchema = baseEndpointSchema
|
|||
sdkHostModels: z
|
||||
.object({
|
||||
maxConcurrentCalls: z.number().int().min(1).max(10000).default(32),
|
||||
maxAdmissionEntries: z.number().int().min(16).max(65536).default(4096),
|
||||
maxBufferedRequestBytes: z.number().int().min(1024).max(1073741824).default(67108864),
|
||||
timeoutMs: z.number().int().min(100).max(86400000).default(60000),
|
||||
maxRequestBytes: z.number().int().min(1024).max(67108864).default(4194304),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue