mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 06:51:59 +00:00
🥪 feat: Compose Remote Model Admission With Output Protection (#18276)
* feat: compose remote model admission with output protection * chore: adopt agents 4.2.9 for remote policy composition
This commit is contained in:
parent
4cd70c5af9
commit
7d1f618b24
12 changed files with 328 additions and 156 deletions
|
|
@ -153,27 +153,27 @@ acceptance target remains the main chat's functionality and experience, with mea
|
|||
Using native events and the native renderer proves only the behaviors tested below. Missing
|
||||
adapters and unverified behaviors remain release gates for products that depend on them.
|
||||
|
||||
| Main-chat capability | This bridge | Remaining work / verification |
|
||||
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
|
||||
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
|
||||
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
|
||||
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
|
||||
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
|
||||
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
|
||||
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
|
||||
| Model/PII/provider-text protection | Host-model chat supports input/PII callbacks and initial persistence admission; explicit provider-text transport remains available | Combining initial admission with provider-text output protection requires SDK invocation-context support. Protected Responses/OpenAI callers without native callbacks remain gated. |
|
||||
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
|
||||
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
|
||||
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
|
||||
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
|
||||
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
|
||||
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
|
||||
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
|
||||
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
|
||||
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
|
||||
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
|
||||
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
|
||||
| Main-chat capability | This bridge | Remaining work / verification |
|
||||
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Saved agent identity, role authorization, picker | Reuses the existing registration and picker | Remote enrollment/discovery is not implemented; operators configure deployments. |
|
||||
| Text and tool-step rendering, persistence | Verified with a minimal Run and a custom graph in real LibreChat | Deterministic models were used; repeat with production providers and representative workloads. |
|
||||
| Reasoning, citations, rich/custom tool events | Native event transport is available | Provider-specific and rich-content end-to-end parity has not been verified. |
|
||||
| Custom graph, nested SDK node, conditional refinement | Verified in another process | Broader graphs, parallel branches, and large checkpoints need workload acceptance tests. |
|
||||
| Approval after browser reload and worker restart | Approve verified with host Mongo checkpoints | Edit/reject, ask-user questions, and repeated/batched interruptions need remote browser tests. |
|
||||
| Stop and browser reconnection to an active job | Verified; stop reaches the worker abort signal and persists no error | Cancellation is cooperative and cannot undo completed tool effects. |
|
||||
| Host-worker connection recovery | Opt-in same-worker live reattachment; durable terminal replay after restart | No host-process adoption, live cross-worker migration, or arbitrary in-flight graph restart. Started executions never automatically rerun. |
|
||||
| Model/PII/provider-text protection | Host-model chat supports input/PII callbacks and initial persistence admission; explicit provider-text transport remains available | Combined admission/output protection uses SDK 4.2.9 invocation context and capability negotiation. Older workers remain gated; protected API callers without native callbacks still need integration. |
|
||||
| Stream circuit breakers and tool-policy hooks | Forwarded to the worker; hook failure stops execution | Domain tools must also authorize signed caller identity; trusted factories must retain the supplied adapters. |
|
||||
| Agent-builder model, parameters, tools, capabilities | Opt-in saved-agent primary model settings execute on the host; host schema tools are available | Worker must retain the host model/profile. Fallbacks, local graph tools, and context/summary policy still require adapters. |
|
||||
| Host MCP, web/code/browser tools, user tool credentials | Authorized root-agent schema tools execute through host RPC | Host credentials remain local; child-agent authorization and broader host-service parity remain separate integrations. |
|
||||
| Uploaded files, generated artifacts, download authorization | Message data can cross the wire | Host file services, sandbox storage, and artifact authorization do not have remote adapters; attachment/file parity is not established. |
|
||||
| Live steering and mid-stream preemption | Not integrated | A remote execution-control adapter and acceptance tests are required. |
|
||||
| Foreground/nested/detached subagents | Worker may construct its own SDK graph | Native configured compositions are rejected; detached child checkpoint threads/context/usage sinks are not integrated. |
|
||||
| Context pruning, compaction, summaries, cache/fading policy | Worker can use SDK features; terminal context metadata is returned | Host configuration and continuation policy are not automatically installed on the worker. Main-chat behavioral/performance parity is not established. |
|
||||
| Token usage, cost/balance accounting | Native model events can be transported | Arbitrary worker model attribution and nested usage/balance enforcement need an explicit contract and tests; do not assume billing parity. |
|
||||
| Tracing and tenant trace/export/redaction policy | Worker can configure SDK tracing itself | Host tracing callbacks and tenant export policy are not automatically inherited. Correlated, policy-safe tracing remains an integration gap. |
|
||||
| Schedules/background/Responses callers | The bridge returns a Run through the common execution seam | Full lifecycle acceptance for these callers has not been performed. |
|
||||
| Console Assistant and Schema Advisor | No product cutover in this change | Wire Schema Advisor, close its required gaps, then adopt shared chat components and compare performance. |
|
||||
|
||||
Do not enable a workflow on the assumption that a native chat control is honored remotely. Its
|
||||
required rows must be implemented and verified first. Keep this matrix current as adapters land;
|
||||
|
|
@ -266,7 +266,7 @@ trusted code and must install it on every applicable SDK Run. This output-text a
|
|||
supply input/model callbacks, authoritative builder model selection, host model
|
||||
credentials, or policy for model calls made outside the SDK. SDK 4.2.7 also provides
|
||||
`HostChatModel` and `createHostModelService` for host-owned model execution; those
|
||||
contracts have the authenticated transport below. The input-admission adapter and this output-protection adapter currently cannot be combined, as explained below.
|
||||
contracts have the authenticated transport below. Combining the input-admission and output-protection adapters requires SDK invocation context and the additional worker capability described below.
|
||||
|
||||
The worker buffers a complete model attempt through the same SDK release boundary
|
||||
as local execution, then sends one `policy.providerText` RPC. Only canonical approved
|
||||
|
|
@ -416,11 +416,29 @@ the registration uses the injected host model. Thus protected chat can opt in, w
|
|||
Responses/OpenAI-compatible callers without that callback remain gated. Local graph tools and
|
||||
provider fallbacks retain their earlier explicit rejection.
|
||||
|
||||
**Remaining limitation:** the SDK provider-text output-protection wrapper deliberately withholds
|
||||
raw callback managers from its underlying model. Until the SDK preserves a separate invocation
|
||||
identity, combining that wrapper with initial remote admission is rejected on both sides before
|
||||
graph execution. Existing programmatic output protection without an initial barrier continues to
|
||||
work. Do not disable output protection to enable this combination.
|
||||
### Combining input admission and output protection
|
||||
|
||||
With [SDK 4.2.9 invocation-context support](https://github.com/LibreChat-AI/agents/pull/704),
|
||||
the worker uses the native `HostChatModel` service context to correlate each model
|
||||
attempt with its root graph callback. The context contains only a version and run ID;
|
||||
raw callback managers remain suppressed by the output-protection boundary. Correlation
|
||||
still requires an enrolled root callback and the authenticated host's observed native
|
||||
success. Input rejection prevents provider execution; output rejection prevents ordinary
|
||||
deferred persistence. Approved output reaches native events and persistence once.
|
||||
|
||||
This combination requires the additive `modelAdmissionWithProviderText: 1` worker
|
||||
capability, advertised only when its installed SDK supports invocation context. The
|
||||
host checks it before factory execution, and the worker independently rejects unsupported
|
||||
combined requests. `modelAdmission: 1` is unchanged, preserving older host behavior.
|
||||
LibreChat depends on SDK 4.2.9 or later. Upgrade separately deployed workers to SDK 4.2.9
|
||||
and this bridge version to enable combined mode. SDK 4.2.8 retains the existing input-only
|
||||
adapter and does **not** enable the combination; old workers fail closed rather than
|
||||
weakening protection.
|
||||
|
||||
Callers still explicitly supply the host provider-text service; this negotiation does
|
||||
not automatically install product output policies or enable protected API callers lacking
|
||||
native input callbacks. Separate-process and live-recovery tests cover the combined service
|
||||
path. Provider-specific UI acceptance, billing and tracing remain separate work.
|
||||
|
||||
Before initial admission, child/summary model calls without root lineage and concurrent root
|
||||
attempts fail closed; parallel/composed admission needs its own ownership contract. This does not
|
||||
|
|
|
|||
|
|
@ -171,6 +171,8 @@ consume `input.agent.model` and retain `input.hostModel`; model credentials stay
|
|||
The custom graph keeps its plan/review/refine nodes and uses the host-authorized tool schemas.
|
||||
This mode requires updated workers with `builderModel: 1` and, for initial persistence admission,
|
||||
`modelAdmission: 1`. Protected chat uses native host input callbacks. Protected API callers without
|
||||
those callbacks, local graph tools, provider fallbacks, and the combination of initial admission
|
||||
with provider-text output protection remain gated; see [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
|
||||
those callbacks, local graph tools and provider fallbacks remain gated. Combining initial admission
|
||||
with explicit provider-text output protection requires `modelAdmissionWithProviderText: 1`, advertised
|
||||
only with SDK 4.2.9 invocation-context support. Upgrade separately deployed workers to SDK 4.2.9
|
||||
and this bridge version; older workers stay gated. See [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
|
||||
and [configuration](../../docs/sdk/agents.md#saved-agent-builder-models) before enabling it.
|
||||
|
|
|
|||
|
|
@ -821,7 +821,8 @@ endpoints:
|
|||
# # Optional primary-model execution for remote SDK agents. Budgets are per process.
|
||||
# # Restart to change budgets; workers must support builderModel: 1.
|
||||
# # Initial protected-chat admission also requires modelAdmission: 1 on workers.
|
||||
# # Local graph tools, provider fallbacks and combined output protection remain gated.
|
||||
# # Combined output protection requires modelAdmissionWithProviderText: 1 and an explicit policy service.
|
||||
# # Local graph tools and provider fallbacks remain gated.
|
||||
# sdkHostModels:
|
||||
# maxConcurrentCalls: 32
|
||||
# maxAdmissionEntries: 4096
|
||||
|
|
|
|||
8
package-lock.json
generated
8
package-lock.json
generated
|
|
@ -9479,9 +9479,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@librechat/agents": {
|
||||
"version": "4.2.8",
|
||||
"resolved": "https://registry.npmjs.org/@librechat/agents/-/agents-4.2.8.tgz",
|
||||
"integrity": "sha512-I26rtj6ODcNqN9hzqFIQN4Ma/v1z5ShI6ew8jmRqZMev1nzIjLLYQeWn4p8gUi+NXE+Zm3n4bdsUouQ7LU3MGQ==",
|
||||
"version": "4.2.9",
|
||||
"resolved": "https://registry.npmjs.org/@librechat/agents/-/agents-4.2.9.tgz",
|
||||
"integrity": "sha512-0i896o9zFNg3I/yxMfrCFRCGspA+HGlt0e2Ng+2YA1MHC3rA/TQ5LqSGqheuBaixfXs8CbcMVKJF7MnZFVdQTQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.115.0",
|
||||
|
|
@ -42445,7 +42445,7 @@
|
|||
"@azure/identity": "^4.14.0",
|
||||
"@azure/storage-blob": "^12.30.0",
|
||||
"@keyv/redis": "5.1.6",
|
||||
"@librechat/agents": "^4.2.8",
|
||||
"@librechat/agents": "^4.2.9",
|
||||
"@librechat/data-schemas": "*",
|
||||
"@modelcontextprotocol/sdk": "^1.32.1",
|
||||
"@opentelemetry/api": "^1.9.0",
|
||||
|
|
|
|||
|
|
@ -120,7 +120,7 @@
|
|||
"@azure/identity": "^4.14.0",
|
||||
"@azure/storage-blob": "^12.30.0",
|
||||
"@keyv/redis": "5.1.6",
|
||||
"@librechat/agents": "^4.2.8",
|
||||
"@librechat/agents": "^4.2.9",
|
||||
"@librechat/data-schemas": "*",
|
||||
"@modelcontextprotocol/sdk": "^1.32.1",
|
||||
"@opentelemetry/api": "^1.9.0",
|
||||
|
|
|
|||
|
|
@ -4,7 +4,15 @@ import {
|
|||
createInitialModelBoundAdmissionCallback,
|
||||
createModelBoundChatModelCallback,
|
||||
} from '../../middleware/modelBoundContent';
|
||||
import { createHostModelLifecycle, createWorkerModelLifecycle } from './admission';
|
||||
import {
|
||||
createHostModelLifecycle,
|
||||
createWorkerModelLifecycle,
|
||||
supportsModelInvocationContext,
|
||||
} from './admission';
|
||||
|
||||
it('installs an SDK that exercises combined admission and output protection', () => {
|
||||
expect(supportsModelInvocationContext).toBe(true);
|
||||
});
|
||||
|
||||
function fixture() {
|
||||
const persist = jest.fn();
|
||||
|
|
|
|||
|
|
@ -18,6 +18,12 @@ export const modelAttemptSchema: z.ZodType<RemoteModelAttempt> = z
|
|||
.object({ nodeId: z.string().uuid(), modelId: z.string().uuid() })
|
||||
.strict();
|
||||
const nodeEndSchema = z.object({ nodeId: z.string().uuid(), completed: z.boolean() }).strict();
|
||||
const invocationContextSchema = z
|
||||
.object({ version: z.literal(1), runId: z.string().uuid() })
|
||||
.strict();
|
||||
/** Additive SDK capability: older SDKs keep the input-only model adapter during rollout. */
|
||||
export const supportsModelInvocationContext: boolean =
|
||||
'invocationContextVersion' in HostChatModel && HostChatModel.invocationContextVersion === 1;
|
||||
const blocked = () => new HostModelServiceError('blocked');
|
||||
|
||||
/** Run each iterator operation inside its scope; constructing an async generator does not enter it. */
|
||||
|
|
@ -248,7 +254,7 @@ export function createHostModelLifecycle(agentId: string, maxEntries: number): H
|
|||
|
||||
export interface WorkerModelLifecycle {
|
||||
callback: BaseCallbackHandler;
|
||||
current(): RemoteModelAttempt | undefined;
|
||||
current(context?: unknown): RemoteModelAttempt | undefined;
|
||||
model(options: ConstructorParameters<typeof HostChatModel>[0]): HostChatModel;
|
||||
}
|
||||
|
||||
|
|
@ -310,13 +316,23 @@ export function createWorkerModelLifecycle(
|
|||
}
|
||||
}
|
||||
return {
|
||||
current() {
|
||||
current(context) {
|
||||
if (admitted) return undefined;
|
||||
if (supportsModelInvocationContext) {
|
||||
const { runId } = invocationContextSchema.parse(context);
|
||||
const attempt = models.get(runId);
|
||||
if (!attempt) throw blocked();
|
||||
attemptedNodes.add(attempt.nodeId);
|
||||
return attempt;
|
||||
}
|
||||
const value = storage.getStore();
|
||||
if (!value) throw blocked();
|
||||
return value;
|
||||
},
|
||||
model: (options) => new CorrelatedHostModel(options),
|
||||
model: (options) =>
|
||||
supportsModelInvocationContext
|
||||
? new HostChatModel(options)
|
||||
: new CorrelatedHostModel(options),
|
||||
callback: ensureHandler({
|
||||
name: 'librechat-remote-model-admission',
|
||||
raiseError: true,
|
||||
|
|
|
|||
|
|
@ -41,8 +41,8 @@ import {
|
|||
createInitialModelBoundAdmissionCallback,
|
||||
createModelBoundChatModelCallback,
|
||||
} from '../../middleware/modelBoundContent';
|
||||
import { createHostModelLifecycle, supportsModelInvocationContext } from './admission';
|
||||
import { createSdkAgentRun, resolveSdkAgentBinding } from '../sdk';
|
||||
import { createHostModelLifecycle } from './admission';
|
||||
import { createRun as createHostRun } from '../run';
|
||||
import { createRemoteSdkServer } from './server';
|
||||
import { createSdkModelHost } from './builder';
|
||||
|
|
@ -1223,7 +1223,13 @@ describe('remote host provider-text protection', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it.each(['providerText', 'model', 'builderModel', 'modelAdmission'] as const)(
|
||||
it.each([
|
||||
'providerText',
|
||||
'model',
|
||||
'builderModel',
|
||||
'modelAdmission',
|
||||
'modelAdmissionWithProviderText',
|
||||
] as const)(
|
||||
'rejects a worker without %s capability before factory admission',
|
||||
async (capability) => {
|
||||
const server = new WebSocketServer({ port: 0, host: '127.0.0.1' });
|
||||
|
|
@ -1241,6 +1247,9 @@ it.each(['providerText', 'model', 'builderModel', 'modelAdmission'] as const)(
|
|||
return {
|
||||
transportVersion: 2,
|
||||
hostTools: 2,
|
||||
...(capability === 'modelAdmissionWithProviderText'
|
||||
? { providerText: 1, modelAdmission: 1, hostModel: 1, builderModel: 1 }
|
||||
: {}),
|
||||
...(['builderModel', 'modelAdmission'].includes(capability)
|
||||
? { hostModel: 1, ...(capability === 'modelAdmission' ? { builderModel: 1 } : {}) }
|
||||
: {}),
|
||||
|
|
@ -1269,7 +1278,8 @@ it.each(['providerText', 'model', 'builderModel', 'modelAdmission'] as const)(
|
|||
? { providerText }
|
||||
: {
|
||||
model: nativeModelBinding(),
|
||||
...(capability === 'modelAdmission'
|
||||
...(capability === 'modelAdmissionWithProviderText' ? { providerText } : {}),
|
||||
...(['modelAdmission', 'modelAdmissionWithProviderText'].includes(capability)
|
||||
? { modelLifecycle: createHostModelLifecycle(identity.agentId, 16) }
|
||||
: {}),
|
||||
...(capability === 'builderModel'
|
||||
|
|
@ -1586,105 +1596,206 @@ function admissionCallback(initial: InitialModelBoundAdmissionCallback) {
|
|||
}
|
||||
|
||||
describe('remote native policy admission', () => {
|
||||
it('rejects combined output protection and initial admission before worker execution', async () => {
|
||||
const createRun = jest.fn(factory);
|
||||
const runtime = await fixture(createRun);
|
||||
try {
|
||||
it.each(['allowed', 'input-blocked', 'output-blocked'] as const)(
|
||||
'negotiates combined policy and admits only approved output (%s)',
|
||||
async (outcome) => {
|
||||
const persisted = jest.fn();
|
||||
const nativeCall = jest.fn();
|
||||
const visible: string[] = [];
|
||||
const inspect = jest.fn(() => {
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
return outcome === 'output-blocked'
|
||||
? { version: 1 as const, ok: false as const, error: { code: 'blocked' as const } }
|
||||
: {
|
||||
version: 1 as const,
|
||||
ok: true as const,
|
||||
value: { content: 'approved', replacements: 1, categories: [] },
|
||||
};
|
||||
});
|
||||
const createRun = jest.fn(async (input: RemoteSdkFactoryInput) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
const runtime = await fixture(createRun);
|
||||
const log = jest.spyOn(console, 'error').mockImplementation(() => {});
|
||||
try {
|
||||
const native = new TestModel({});
|
||||
Object.assign(native, {
|
||||
callbacks: [
|
||||
createModelBoundChatModelCallback({
|
||||
filters: {
|
||||
messages: {
|
||||
pii: {
|
||||
fields: ['text'],
|
||||
starterPatterns: [],
|
||||
customPatterns: [{ id: 'private', label: 'Private', regex: 'PRIVATE-INPUT' }],
|
||||
},
|
||||
},
|
||||
},
|
||||
}),
|
||||
],
|
||||
});
|
||||
native._streamResponseChunks = async function* () {
|
||||
nativeCall();
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'PRIVATE-OUTPUT',
|
||||
message: new AIMessageChunk('PRIVATE-OUTPUT'),
|
||||
});
|
||||
};
|
||||
const config = hostConfig();
|
||||
config.customHandlers = {
|
||||
[GraphEvents.ON_MESSAGE_DELTA]: {
|
||||
handle: async (_event, data) => {
|
||||
visible.push(JSON.stringify(data));
|
||||
},
|
||||
},
|
||||
};
|
||||
const host = createSdkModelHost(
|
||||
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
|
||||
() => native,
|
||||
)!;
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const run = await createRemoteSdkRun({
|
||||
config,
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: {
|
||||
...host.bind(config, identity),
|
||||
providerText: createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 1000,
|
||||
maxAttemptBytes: 4096,
|
||||
maxBufferedBytes: 8192,
|
||||
classify: () => 'prose',
|
||||
inspect,
|
||||
},
|
||||
}),
|
||||
},
|
||||
});
|
||||
const result = run.processStream(
|
||||
{ messages: [new HumanMessage(outcome === 'input-blocked' ? 'PRIVATE-INPUT' : 'hello')] },
|
||||
{ ...invocation(), callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
if (!supportsModelInvocationContext) {
|
||||
await expect(result).rejects.toThrow('SDK runtime execution failed');
|
||||
expect(createRun).not.toHaveBeenCalled();
|
||||
expect(nativeCall).not.toHaveBeenCalled();
|
||||
expect(inspect).not.toHaveBeenCalled();
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
} else if (outcome === 'allowed') {
|
||||
const content = await result;
|
||||
expect(JSON.stringify(content)).toContain('approved');
|
||||
expect(persisted).toHaveBeenCalledTimes(1);
|
||||
expect(nativeCall).toHaveBeenCalledTimes(1);
|
||||
expect(inspect).toHaveBeenCalledTimes(1);
|
||||
expect(visible.join('')).toContain('approved');
|
||||
expect(JSON.stringify(run.getRunMessages())).not.toContain('PRIVATE-OUTPUT');
|
||||
} else {
|
||||
await expect(result).rejects.toThrow();
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
expect(nativeCall).toHaveBeenCalledTimes(outcome === 'input-blocked' ? 0 : 1);
|
||||
expect(inspect).toHaveBeenCalledTimes(outcome === 'input-blocked' ? 0 : 1);
|
||||
}
|
||||
expect(visible.join('')).not.toContain('PRIVATE-OUTPUT');
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
await runtime.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
it.each([false, true])(
|
||||
'does not persist cancelled partial output (combined=%s)',
|
||||
async (combined) => {
|
||||
const persisted = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: jest.fn(),
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const run = await createRemoteSdkRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: {
|
||||
model: nativeModelBinding(),
|
||||
modelLifecycle: createHostModelLifecycle(identity.agentId, 16),
|
||||
providerText: createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 1000,
|
||||
maxAttemptBytes: 4096,
|
||||
maxBufferedBytes: 8192,
|
||||
classify: () => 'prose',
|
||||
inspect: () => ({ version: 1, ok: false, error: { code: 'blocked' } }),
|
||||
},
|
||||
}),
|
||||
},
|
||||
const abort = new AbortController();
|
||||
let entered!: () => void;
|
||||
let release!: () => void;
|
||||
const started = new Promise<void>((resolve) => {
|
||||
entered = resolve;
|
||||
});
|
||||
await expect(
|
||||
run.processStream(
|
||||
{ messages: [] },
|
||||
{ ...invocation(), callbacks: [admissionCallback(initial)] },
|
||||
),
|
||||
).rejects.toThrow('SDK invocation context support');
|
||||
expect(createRun).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await runtime.close();
|
||||
}
|
||||
});
|
||||
it('does not persist a deferred message when a partially streamed model is cancelled', async () => {
|
||||
const persisted = jest.fn();
|
||||
const initial = createInitialModelBoundAdmissionCallback({
|
||||
agentIds: [identity.agentId],
|
||||
isActive: () => true,
|
||||
onAllowed: persisted,
|
||||
});
|
||||
const abort = new AbortController();
|
||||
let entered!: () => void;
|
||||
let release!: () => void;
|
||||
const started = new Promise<void>((resolve) => {
|
||||
entered = resolve;
|
||||
});
|
||||
const pending = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
try {
|
||||
const model = new TestModel({});
|
||||
model._streamResponseChunks = async function* () {
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'partial',
|
||||
message: new AIMessageChunk({ content: 'partial' }),
|
||||
});
|
||||
entered();
|
||||
await pending;
|
||||
};
|
||||
const run = await createSdkAgentRun({
|
||||
config: hostConfig(),
|
||||
identity,
|
||||
signal: abort.signal,
|
||||
registration: {
|
||||
binding: { id: 'custom', protocolVersion: 1 },
|
||||
remote: { ...runtime.remote, hostModel: true },
|
||||
},
|
||||
modelHost: createSdkModelHost(
|
||||
const pending = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
return run;
|
||||
});
|
||||
try {
|
||||
const model = new TestModel({});
|
||||
model._streamResponseChunks = async function* () {
|
||||
yield new ChatGenerationChunk({
|
||||
text: 'partial',
|
||||
message: new AIMessageChunk({ content: 'partial' }),
|
||||
});
|
||||
entered();
|
||||
await pending;
|
||||
};
|
||||
const config = hostConfig();
|
||||
const modelHost = createSdkModelHost(
|
||||
agentsEndpointSchema.parse({ sdkHostModels: {} }).sdkHostModels,
|
||||
() => model,
|
||||
),
|
||||
});
|
||||
const execution = run.processStream(
|
||||
{ messages: [new HumanMessage('safe')] },
|
||||
{ ...invocation(), signal: abort.signal, callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
void execution.catch(() => {});
|
||||
await started;
|
||||
abort.abort();
|
||||
release();
|
||||
await expect(execution).rejects.toThrow('cancelled');
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
release();
|
||||
await runtime.close();
|
||||
}
|
||||
});
|
||||
)!;
|
||||
const inspect = jest.fn(() => ({
|
||||
version: 1 as const,
|
||||
ok: true as const,
|
||||
value: { content: 'approved', replacements: 0, categories: [] },
|
||||
}));
|
||||
const run = await createRemoteSdkRun({
|
||||
config,
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: {
|
||||
...modelHost.bind(config, identity, abort.signal),
|
||||
...(combined && {
|
||||
providerText: createHostProviderTextService({
|
||||
identity,
|
||||
policy: {
|
||||
version: 1,
|
||||
timeoutMs: 1000,
|
||||
maxAttemptBytes: 4096,
|
||||
maxBufferedBytes: 8192,
|
||||
classify: () => 'prose',
|
||||
inspect,
|
||||
},
|
||||
}),
|
||||
}),
|
||||
},
|
||||
});
|
||||
const execution = run.processStream(
|
||||
{ messages: [new HumanMessage('safe')] },
|
||||
{ ...invocation(), signal: abort.signal, callbacks: [admissionCallback(initial)] },
|
||||
);
|
||||
void execution.catch(() => {});
|
||||
if (combined && !supportsModelInvocationContext) {
|
||||
await expect(execution).rejects.toThrow('SDK runtime execution failed');
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
await started;
|
||||
abort.abort();
|
||||
release();
|
||||
await expect(execution).rejects.toThrow('cancelled');
|
||||
expect(inspect).not.toHaveBeenCalled();
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
release();
|
||||
await runtime.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
it.each([false, true])(
|
||||
'admits deferred persistence only after the safe model node (blocked=%s)',
|
||||
async (blocked) => {
|
||||
|
|
|
|||
|
|
@ -201,12 +201,6 @@ export async function createRemoteSdkRun({
|
|||
return socket;
|
||||
};
|
||||
const modelAdmission = hostServices?.modelLifecycle?.attach(options.callbacks) === true;
|
||||
if (modelAdmission && hostServices?.providerText) {
|
||||
hostServices.modelLifecycle?.dispose();
|
||||
throw new Error(
|
||||
'Remote model admission with provider-text protection requires SDK invocation context support',
|
||||
);
|
||||
}
|
||||
const connection = remote.recovery
|
||||
? connectRecoveringSdkSocket({
|
||||
url: remote.url,
|
||||
|
|
@ -494,11 +488,18 @@ export async function createRemoteSdkRun({
|
|||
hostModel?: number;
|
||||
builderModel?: number;
|
||||
modelAdmission?: number;
|
||||
modelAdmissionWithProviderText?: number;
|
||||
};
|
||||
if (capabilities?.transportVersion !== 2)
|
||||
throw new Error('Worker transport capability unavailable');
|
||||
if (modelAdmission && capabilities?.modelAdmission !== 1)
|
||||
throw new Error('Worker model admission capability unavailable');
|
||||
if (
|
||||
modelAdmission &&
|
||||
hostServices?.providerText &&
|
||||
capabilities?.modelAdmissionWithProviderText !== 1
|
||||
)
|
||||
throw new Error('Worker protected model admission capability unavailable');
|
||||
if (hostServices?.modelProfile && capabilities?.builderModel !== 1)
|
||||
throw new Error('Worker builder model capability unavailable');
|
||||
if (hostServices?.model && capabilities?.hostModel !== 1)
|
||||
|
|
|
|||
|
|
@ -211,7 +211,7 @@ export function createWorkerModelChannel(options: {
|
|||
let receiver: ReturnType<typeof createPolicyReceiver> | undefined;
|
||||
let seal: ReturnType<typeof createPolicySender> | undefined;
|
||||
let lifecycle: WorkerModelLifecycle | undefined;
|
||||
const encodeRequest = (id: string, request: HostModelRequest) => {
|
||||
const encodeRequest = (id: string, request: HostModelRequest, context?: unknown) => {
|
||||
if (!receiver || !seal) throw new HostModelServiceError('incompatible');
|
||||
return sealValue(
|
||||
seal,
|
||||
|
|
@ -219,7 +219,7 @@ export function createWorkerModelChannel(options: {
|
|||
id,
|
||||
publicKey: receiver.publicKey,
|
||||
request,
|
||||
...(lifecycle && { context: lifecycle.current() }),
|
||||
...(lifecycle && { context: lifecycle.current(context) }),
|
||||
},
|
||||
options.limits.maxFrameBytes,
|
||||
);
|
||||
|
|
@ -242,12 +242,12 @@ export function createWorkerModelChannel(options: {
|
|||
options.signal.addEventListener('abort', dispose, { once: true });
|
||||
const service: HostModelService = {
|
||||
version: 1,
|
||||
invoke: async (request, signal) => {
|
||||
invoke: async (request, signal, context?: unknown) => {
|
||||
const id = randomUUID();
|
||||
const result = reply(
|
||||
await options.peer.call(
|
||||
'model.invoke',
|
||||
encodeRequest(id, request),
|
||||
encodeRequest(id, request, context),
|
||||
options.limits.maxRunMs,
|
||||
signal ? AbortSignal.any([options.signal, signal]) : options.signal,
|
||||
),
|
||||
|
|
@ -259,9 +259,9 @@ export function createWorkerModelChannel(options: {
|
|||
throw new HostModelServiceError('incompatible');
|
||||
return value.message;
|
||||
},
|
||||
stream: async function* (request, signal) {
|
||||
stream: async function* (request, signal, context?: unknown) {
|
||||
const id = randomUUID();
|
||||
const payload = encodeRequest(id, request);
|
||||
const payload = encodeRequest(id, request, context);
|
||||
const controller = new AbortController();
|
||||
const boundSignal = AbortSignal.any([
|
||||
options.signal,
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ import {
|
|||
clearRedisTestPrefix,
|
||||
} from '../../../stream/__tests__/helpers/redis';
|
||||
import { createInitialModelBoundAdmissionCallback } from '../../../middleware/modelBoundContent';
|
||||
import { createHostModelLifecycle } from '../admission';
|
||||
import { createHostModelLifecycle, supportsModelInvocationContext } from '../admission';
|
||||
import { createRedisRuntimeJournal } from './redis';
|
||||
import { createRemoteSdkServer } from '../server';
|
||||
import { createRuntimeRecovery } from './server';
|
||||
|
|
@ -1023,9 +1023,11 @@ it('recovers a policy reply after a wire outage without repeating inspection or
|
|||
}
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
'resumes host model chunks without repeating the provider (input admission=%s)',
|
||||
async (inputAdmission) => {
|
||||
it.each(['input', 'output', 'combined'] as const)(
|
||||
'resumes host model chunks without repeating the provider (protection=%s)',
|
||||
async (mode) => {
|
||||
const inputAdmission = mode !== 'output';
|
||||
const outputProtection = mode !== 'input';
|
||||
const runtime = await fixture(async (input) => {
|
||||
const run = await factory(input);
|
||||
run.Graph!.overrideModel = input.hostModel;
|
||||
|
|
@ -1048,8 +1050,8 @@ it.each([false, true])(
|
|||
runtime.drop();
|
||||
await delay(50);
|
||||
yield new ChatGenerationChunk({
|
||||
text: inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE ',
|
||||
message: new AIMessageChunk(inputAdmission ? 'approved ' : 'PRIVATE-CANDIDATE '),
|
||||
text: outputProtection ? 'PRIVATE-CANDIDATE ' : 'approved ',
|
||||
message: new AIMessageChunk(outputProtection ? 'PRIVATE-CANDIDATE ' : 'approved '),
|
||||
});
|
||||
yield new ChatGenerationChunk({ text: 'third', message: new AIMessageChunk('third') });
|
||||
};
|
||||
|
|
@ -1101,9 +1103,13 @@ it.each([false, true])(
|
|||
config: hostConfig(async () => {}),
|
||||
identity,
|
||||
remote: runtime.remote,
|
||||
hostServices: { model, ...(inputAdmission ? { modelLifecycle } : { providerText }) },
|
||||
hostServices: {
|
||||
model,
|
||||
...(inputAdmission && { modelLifecycle }),
|
||||
...(outputProtection && { providerText }),
|
||||
},
|
||||
});
|
||||
const content = await run.processStream(
|
||||
const result = run.processStream(
|
||||
{ messages: [new HumanMessage('go')] },
|
||||
{
|
||||
version: 'v2',
|
||||
|
|
@ -1119,6 +1125,14 @@ it.each([false, true])(
|
|||
: undefined,
|
||||
},
|
||||
);
|
||||
if (mode === 'combined' && !supportsModelInvocationContext) {
|
||||
await expect(result).rejects.toThrow();
|
||||
expect(invocations).toBe(0);
|
||||
expect(persisted).not.toHaveBeenCalled();
|
||||
expect(runtime.createRun).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
const content = await result;
|
||||
expect(JSON.stringify(content)).toContain('first approved third');
|
||||
expect(runtime.journalEntries.join('')).not.toContain('PRIVATE-CANDIDATE');
|
||||
expect(invocations).toBe(1);
|
||||
|
|
|
|||
|
|
@ -20,11 +20,11 @@ import type { Server } from 'node:http';
|
|||
import type { RemoteSdkFactoryInput, RemoteSdkLimits, RemoteSdkStart, SdkRun } from './types';
|
||||
import type { RuntimeRecoveryOptions } from './recovery/server';
|
||||
import type { SdkAgentIdentity } from './identity';
|
||||
import { createWorkerModelLifecycle, supportsModelInvocationContext } from './admission';
|
||||
import { policyDescriptorSchema, createWorkerProviderTextPolicy } from './policy';
|
||||
import { createRuntimeRecovery, recoveryClaimSchema } from './recovery/server';
|
||||
import { remoteSdkLimits, isRemoteSdkSecret } from './types';
|
||||
import { RECOVERY_HEADER_BYTES } from './recovery/socket';
|
||||
import { createWorkerModelLifecycle } from './admission';
|
||||
import { createWorkerToolService } from './hostTools';
|
||||
import { RemoteSdkCheckpointer } from './checkpoint';
|
||||
import { createWorkerModelChannel } from './model';
|
||||
|
|
@ -298,6 +298,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
hostModel: 1,
|
||||
builderModel: 1,
|
||||
modelAdmission: 1,
|
||||
...(supportsModelInvocationContext && { modelAdmissionWithProviderText: 1 }),
|
||||
};
|
||||
if (method === 'model.chunk') return workerModel.accept(data, signal);
|
||||
if (method === 'tool.result') return workerTools.accept(data);
|
||||
|
|
@ -305,7 +306,7 @@ export function createRemoteSdkServer(options: RemoteSdkServerOptions): {
|
|||
if (method !== 'run' || started) throw new Error('SDK execution already admitted');
|
||||
started = true;
|
||||
const request = startSchema.parse(data) as RemoteSdkStart;
|
||||
if (request.modelAdmission && request.providerText)
|
||||
if (request.modelAdmission && request.providerText && !supportsModelInvocationContext)
|
||||
throw new Error(
|
||||
'Combined model admission and provider-text protection is unsupported',
|
||||
);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue