mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-09-15 13:10:02 +00:00
🖥️ feat: Stream PTC Inner Tool Calls as a CLI-Style Trace (#15115)
* 🖥️ feat: Stream PTC Inner Tool Calls as a CLI-Style Trace Programmatic tool calling runs a whole program inside the sandbox, and the tool calls that program makes open no run step of their own. The card showed one running spinner for the entire execution, with no sign of what the code was doing. Emit a new `on_ptc_tool_call` step event for each inner invocation — once on dispatch, once on settle — and render them under the code as a terminal-style trace: status glyph, tool identity, argument preview, duration, with a failure message printed under the call that produced it. The seam is the tool map the sandbox bridge resolves inner calls against. `instrumentPtcToolMap` proxies `invoke` on each entry and leaves every other property (name, schema, mcp) passing straight through, so nothing about execution changes and emission failures can never fail a tool call. Client state is a per-tool-call Recoil atom keyed like the sandbox-starting and subagent atoms — live for the session, cleared on conversation switch so a finished program's trace stays readable. * 🩹 fix: Address Codex Review on the PTC Tool Trace Five findings, all confirmed against the source before fixing. Scope the trace atoms to a message occurrence. The hook already documents that providers repeat a tool_call_id across turns and even within one message, and `call_id` restarts at :0 for every outer call — so two programs sharing `call_0` merged into one card. Key by (response message id, tool call id) via `ptcTraceKey`, mirroring `subagentProgressKey`; the event's `runId` already carries the message id and the card reads its own from MessageContext. Prune unsettled rows on resume. Inner calls are not content parts, so the resume snapshot cannot rebuild them, and `trackReplayEvent` only persists OAuth events — a call that settled during a disconnect left a spinner that never resolved. Settled rows are real history and stay. Make the argument preview budget-aware. Iterate keys rather than entries so the budget check can actually skip work, and clip against a bounded window so a multi-megabyte value is never collapsed in full to build a 40-character preview. Catch the resumable emission promise. The synchronous try/catch around the emitter cannot observe a rejected `emitChunk`, so a failing transport raised an unhandled rejection per event instead of dropping telemetry. Announce completion to assistive technology. The check glyph is decorative and a fast call renders no duration, so a settled row previously announced no outcome; each row now carries an sr-only status and the visible cell that duplicated it is hidden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🧹 fix: Repair CI Failures on the PTC Tool Trace Two failures on the previous head, both mine. `Tests: api (shard 2/3)` — 46 failures in `initialize.spec.js`, all `TypeError: createPtcProgressEmitter is not a function`. The suite mocks the callbacks module with an object literal, and wiring the new emitter into `initialize.js` without adding it there left the factory undefined at call time. Added it alongside `createAttachmentEmitter`, plus an assertion that it receives the same generation fence as every other resumable emitter — a stale epoch would leak one run's inner calls into the next. `Static checks` — import-order drift in `PtcToolTrace.tsx` and `handlers.ts`, repaired with `scripts/sort-imports.mts`. ESLint and Prettier both passed, so only the dedicated check caught it. `openai.js` and `responses.js` never take the emitter, so their specs were unaffected; verified the initialize mock now covers every name the module destructures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🔐 fix: Address Second Codex Review on the PTC Tool Trace Three of five findings actioned; two answered on the thread. Respect tool-argument PII filtering (P1). Inner calls never reach `filteredToolArgumentsResult` — the sandbox bridge invokes them directly — so the trace was the one path putting their values on the wire in a deployment that had configured `filters.toolArguments.pii`. When any of the name / arguments / output fields are filtered, the emitter now omits both the argument preview and the failure message, which routinely quotes the argument that caused it. Name, status and duration still report. Drop the light/dark-specific background (P1). `dark:bg-transparent` stepped outside the semantic roles and would lose the intended separation under a custom theme. The pane now sets no background at all and inherits the card's surface, which resolves to the same color the override produced in both default themes and stays correct when a theme reassigns its roles. Bound the live trace (P2). A program looping over a large collection made every event copy an ever-growing array and rendered a row per call. The trace now keeps a rolling tail of 100 rows and counts what it evicted, surfaced as "+N earlier calls" so the cap is never silent. A settle whose row is gone — evicted, or pruned across a resume gap — no longer reappears out of order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ✅ test: Keep PTC Trace Tests Aligned With Caller-Capability Filtering Left out of the merge commit by a staging slip; without them `handlers.spec.ts` fails on the merged tree. `#15105` restricts the PTC tool map to tools whose `allowed_callers` admit code execution, so the existing trace test's registry entry — which declared none, defaulting to `direct` — was filtered out before the instrumentation could see it. Declare the fixture `code_execution`. Add a guard for the resolution itself: a `direct`-only tool must never appear in the instrumented map. Tracing wraps the eligible map, and this fails if a later change reorders that and lets the trace widen what the sandbox reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🛡️ fix: Close Name Disclosure and Follow the PTC Trace Tail Two findings from the third Codex pass on `17a9ec9`. Redact filtered inner-tool names (P1). The previous gate suppressed argument and failure previews but the event still carried `name` verbatim, so a deployment whose `filters.toolArguments.pii.fields` includes `name` could see a blocked identifier disclosed through the trace — the one path inner calls take, since they never reach `filteredToolArgumentsResult`. Inner tool names are now inspected once per PTC call with the same `extractToolArgumentContent` + `inspectContent` pair the executor uses; any that trip the policy are left unwrapped, so they still execute and emit nothing. An un-inspectable name fails closed. Follow the trace tail (P2). The row list is a 200px scroller that never moved, so once a program exceeded the viewport the card sat on the oldest calls while live activity accumulated below the fold. Reuse `useFollowScroll` — the hook the code and command panes already use — which pins to the tail while calls are running and yields the moment the reader scrolls up. The host card threads its disclosure state so a collapsed pane is never scrolled invisibly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 📌 fix: Pin the PTC Trace Through Its Final Settle The fourth Codex pass on `4bf68e1`, one P2 finding. `useFollowScroll` returned early whenever `active` was false, so the one change it most needed to follow was the one it skipped. A failing inner call settles by appending its error line in the same commit that clears the last running row: the content grows and the stream ends together, and the pin that would have revealed that line never fired. On an expanded, bottom-pinned pane the failure — the row a reader most wants — stayed below the fold. The falling edge of `active` now pins too, but only when the content changed with it. Ending a stream on its own still leaves the pane where the reader left it, which is what the existing contract promises and what the sibling code and command panes rely on; a reader who has scrolled up is untouched either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * 🔌 fix: Keep PTC Calls That Outlive a Reconnect Fifth Codex pass on `085a83f`; one of its two findings. Pruning rows across a resume gap deleted every `running` row, but a stream gap is not proof the call ended. A call still executing across the reconnect settles normally on the restored live stream — and `applyPtcToolCall` drops a settle whose row is gone, by design, so an evicted row cannot reappear out of order. The call therefore vanished from the trace despite having run, which is worse than the spinner the pruning existed to prevent. Rows are now marked `interrupted` instead of removed. A call whose settle was genuinely lost in the gap reports that honestly rather than spinning forever, and one that survives the gap settles onto the row it opened, reporting its real outcome and duration. `interrupted` is a client-side conclusion, so it widens the row status locally and leaves the wire contract alone. Two cases added: the gap marks rather than drops, and a post-reconnect settle lands on its marked row; plus a render case for the new outcome. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Danny Avila
GitHub
parent
caa938fec6
commit
c2aa688d73
26 changed files with 1707 additions and 12 deletions
|
|
@ -10,6 +10,7 @@ import type {
|
|||
ToolExecuteResult,
|
||||
ToolCallRequest,
|
||||
} from '@librechat/agents';
|
||||
import type { PtcToolCallEvent } from 'librechat-data-provider';
|
||||
import type { CodeExecutionContext } from './execution';
|
||||
import { createToolExecuteHandler, ToolExecuteOptions } from './handlers';
|
||||
import { markSandboxReady } from './prewarm';
|
||||
|
|
@ -976,6 +977,85 @@ describe('createToolExecuteHandler', () => {
|
|||
expect(capturedConfigs[0].disallowedToolDefs).toEqual([]);
|
||||
expect(capturedConfigs[0].toolMap).toEqual(new Map());
|
||||
});
|
||||
|
||||
it('instruments the PTC tool map so inner calls report progress', async () => {
|
||||
const capturedConfigs: Record<string, unknown>[] = [];
|
||||
const ptcTool = createMockTool(Constants.PROGRAMMATIC_TOOL_CALLING, capturedConfigs);
|
||||
/** `allowed_callers` must admit code execution, or the caller-capability
|
||||
* filter drops the tool before the trace ever sees it. */
|
||||
const toolRegistry = new Map([
|
||||
['custom_tool', { name: 'custom_tool', allowed_callers: ['code_execution'] }],
|
||||
]);
|
||||
const ptcToolMap = new Map([['custom_tool', createMockTool('custom_tool', [])]]);
|
||||
const loadTools: ToolExecuteOptions['loadTools'] = jest.fn(async () => ({
|
||||
loadedTools: [ptcTool] as never[],
|
||||
configurable: { toolRegistry, ptcToolMap },
|
||||
}));
|
||||
const events: PtcToolCallEvent[] = [];
|
||||
const handler = createToolExecuteHandler({
|
||||
loadTools,
|
||||
emitPtcProgress: (event) => events.push(event),
|
||||
});
|
||||
|
||||
await invokeHandler(handler, [
|
||||
{
|
||||
id: 'call_ptc',
|
||||
name: Constants.PROGRAMMATIC_TOOL_CALLING,
|
||||
args: { code: 'custom_tool "{}"' },
|
||||
},
|
||||
]);
|
||||
|
||||
const injectedMap = capturedConfigs[0].toolMap as Map<
|
||||
string,
|
||||
{ name: string; invoke: (input: unknown, config?: unknown) => Promise<unknown> }
|
||||
>;
|
||||
expect(injectedMap).not.toBe(ptcToolMap);
|
||||
expect(injectedMap.get('custom_tool')?.name).toBe('custom_tool');
|
||||
|
||||
await injectedMap
|
||||
.get('custom_tool')
|
||||
?.invoke({ path: 'a.ts' }, { metadata: { [Constants.PROGRAMMATIC_TOOL_CALLING]: true } });
|
||||
|
||||
expect(events.map((event) => event.status)).toEqual(['running', 'success']);
|
||||
expect(events[0]).toMatchObject({
|
||||
tool_call_id: 'call_ptc',
|
||||
name: 'custom_tool',
|
||||
args: 'path=a.ts',
|
||||
});
|
||||
});
|
||||
|
||||
it('instruments only the tools the caller-capability filter admits', async () => {
|
||||
const capturedConfigs: Record<string, unknown>[] = [];
|
||||
const ptcTool = createMockTool(Constants.PROGRAMMATIC_TOOL_CALLING, capturedConfigs);
|
||||
const toolRegistry = new Map([
|
||||
['code_tool', { name: 'code_tool', allowed_callers: ['code_execution'] }],
|
||||
['direct_tool', { name: 'direct_tool', allowed_callers: ['direct'] }],
|
||||
]);
|
||||
const ptcToolMap = new Map([
|
||||
['code_tool', createMockTool('code_tool', [])],
|
||||
['direct_tool', createMockTool('direct_tool', [])],
|
||||
]);
|
||||
const loadTools: ToolExecuteOptions['loadTools'] = jest.fn(async () => ({
|
||||
loadedTools: [ptcTool] as never[],
|
||||
configurable: { toolRegistry, ptcToolMap },
|
||||
}));
|
||||
const handler = createToolExecuteHandler({
|
||||
loadTools,
|
||||
emitPtcProgress: () => {},
|
||||
});
|
||||
|
||||
await invokeHandler(handler, [
|
||||
{
|
||||
id: 'call_ptc',
|
||||
name: Constants.PROGRAMMATIC_TOOL_CALLING,
|
||||
args: { code: 'code_tool "{}"' },
|
||||
},
|
||||
]);
|
||||
|
||||
/** Tracing must not widen what the sandbox can reach. */
|
||||
const injectedMap = capturedConfigs[0].toolMap as Map<string, unknown>;
|
||||
expect([...injectedMap.keys()]).toEqual(['code_tool']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('host file authoring collisions', () => {
|
||||
|
|
|
|||
|
|
@ -15,8 +15,8 @@ import type {
|
|||
CallerCapabilityProjectionSnapshot,
|
||||
} from '@librechat/agents';
|
||||
import type { StructuredToolInterface } from '@librechat/agents/langchain/tools';
|
||||
import type { CodeEnvRef, PtcToolCallEvent } from 'librechat-data-provider';
|
||||
import type { ValidationIssue } from '@librechat/data-schemas';
|
||||
import type { CodeEnvRef } from 'librechat-data-provider';
|
||||
import type { SkillFileRecord, PrimeSkillFilesResult } from './skillFiles';
|
||||
import type { CodeExecutionContext } from './execution';
|
||||
import type { TextContentFragment } from '~/protection';
|
||||
|
|
@ -72,6 +72,7 @@ import { buildSkillPrimeMessage, SKILL_FILE_PREFIX } from './skills';
|
|||
import { parseFrontmatter } from '../skills/import';
|
||||
import { cleanCodeToolOutput } from './cleanup';
|
||||
import { primeSkillFiles } from './skillFiles';
|
||||
import { instrumentPtcToolMap } from './ptc';
|
||||
import { markSandboxReady } from './prewarm';
|
||||
|
||||
export interface ToolEndCallbackData {
|
||||
|
|
@ -136,6 +137,13 @@ export interface ToolExecuteOptions {
|
|||
}) => Promise<{ attachments?: unknown[] } | null>;
|
||||
/** Emits an `attachment` SSE event on the current request's live stream. */
|
||||
emitAttachment?: (attachment: unknown) => void;
|
||||
/**
|
||||
* Emits an `on_ptc_tool_call` SSE event for one inner tool invocation made
|
||||
* by a programmatic tool-calling program. Absent on transports that don't
|
||||
* carry the LibreChat step stream (Open Responses), which simply skips the
|
||||
* instrumentation.
|
||||
*/
|
||||
emitPtcProgress?: (event: PtcToolCallEvent) => void;
|
||||
/**
|
||||
* Loads a skill by name with ACL constraint (returns full body for injection).
|
||||
*
|
||||
|
|
@ -742,6 +750,35 @@ function filteredToolArgumentsResult(
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Inner tool names the `name` PII policy would block. `filteredToolArgumentsResult`
|
||||
* inspects `tc.name` for direct calls, but inner calls bypass it entirely — and
|
||||
* the trace event carries the name unconditionally, so without this the trace
|
||||
* becomes the disclosure path the policy exists to close. The eligible map holds
|
||||
* a handful of names, each inspected once per PTC call.
|
||||
*/
|
||||
function collectFilteredPtcToolNames(
|
||||
names: Iterable<string>,
|
||||
req: ServerRequest | undefined,
|
||||
): ReadonlySet<string> | undefined {
|
||||
const filters = req?.config?.filters;
|
||||
if (filters == null || !hasActivePiiFields(filters.toolArguments?.pii, ['name'])) {
|
||||
return undefined;
|
||||
}
|
||||
const blocked = new Set<string>();
|
||||
for (const name of names) {
|
||||
try {
|
||||
if (inspectContent(extractToolArgumentContent({ name }), { filters }) != null) {
|
||||
blocked.add(name);
|
||||
}
|
||||
} catch {
|
||||
/* An un-inspectable name is treated as blocked: fail closed. */
|
||||
blocked.add(name);
|
||||
}
|
||||
}
|
||||
return blocked.size > 0 ? blocked : undefined;
|
||||
}
|
||||
|
||||
function filteredToolOutputResult(
|
||||
tc: ToolCallRequest,
|
||||
req: ServerRequest | undefined,
|
||||
|
|
@ -4150,8 +4187,14 @@ function buildToolCallConfig(
|
|||
}
|
||||
|
||||
export function createToolExecuteHandler(options: ToolExecuteOptions): EventHandler {
|
||||
const { loadTools, toolEndCallback, persistBackgroundCodeResult, emitAttachment, subagentTasks } =
|
||||
options;
|
||||
const {
|
||||
loadTools,
|
||||
toolEndCallback,
|
||||
persistBackgroundCodeResult,
|
||||
emitAttachment,
|
||||
emitPtcProgress,
|
||||
subagentTasks,
|
||||
} = options;
|
||||
|
||||
return {
|
||||
handle: async (_event: string, data: ToolExecuteBatchRequest) => {
|
||||
|
|
@ -5007,9 +5050,42 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand
|
|||
toolCallConfig.toolDefs = toolDefs;
|
||||
toolCallConfig.disallowedToolDefs = disallowedToolDefs;
|
||||
const eligibleNames = new Set(toolDefs.map((toolDef) => toolDef.name));
|
||||
toolCallConfig.toolMap = new Map(
|
||||
/* Instrument the ELIGIBLE map, never the raw one: the
|
||||
* caller-capability restriction decides what the sandbox
|
||||
* may reach, and tracing must not widen it. */
|
||||
const eligiblePtcToolMap = new Map(
|
||||
[...(ptcToolMap ?? toolMap)].filter(([name]) => eligibleNames.has(name)),
|
||||
);
|
||||
/* Inner calls produce no run step and no card of their
|
||||
* own, so the only record of what the program did is
|
||||
* this trace. `invoke` is the single seam every inner
|
||||
* call passes through.
|
||||
*
|
||||
* They also never reach `filteredToolArgumentsResult` —
|
||||
* the sandbox bridge invokes them directly — so when the
|
||||
* deployment filters tool arguments for PII, the trace
|
||||
* must not put their values on the wire. */
|
||||
const ptcReq = mergedConfigurable?.req as ServerRequest | undefined;
|
||||
const ptcArgumentPii = ptcReq?.config?.filters?.toolArguments?.pii;
|
||||
toolCallConfig.toolMap = emitPtcProgress
|
||||
? instrumentPtcToolMap({
|
||||
toolMap: eligiblePtcToolMap,
|
||||
toolCallId: tc.id,
|
||||
runId: (metadata as Record<string, unknown>)?.run_id as
|
||||
| string
|
||||
| undefined,
|
||||
includePreviews: !hasActivePiiFields(ptcArgumentPii, [
|
||||
'name',
|
||||
'arguments',
|
||||
'output',
|
||||
]),
|
||||
traceExclusions: collectFilteredPtcToolNames(
|
||||
eligiblePtcToolMap.keys(),
|
||||
ptcReq,
|
||||
),
|
||||
emit: emitPtcProgress,
|
||||
})
|
||||
: eligiblePtcToolMap;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ export * from './orphans';
|
|||
export * from './migration';
|
||||
export * from './parameters';
|
||||
export * from './prewarm';
|
||||
export * from './ptc';
|
||||
export * from './openai';
|
||||
export * from './transactions';
|
||||
export * from './traversal';
|
||||
|
|
|
|||
247
packages/api/src/agents/ptc.spec.ts
Normal file
247
packages/api/src/agents/ptc.spec.ts
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
import type { StructuredToolInterface } from '@librechat/agents/langchain/tools';
|
||||
import type { PtcToolCallEvent } from 'librechat-data-provider';
|
||||
import { instrumentPtcToolMap, summarizePtcArgs } from './ptc';
|
||||
|
||||
/**
|
||||
* Minimal stand-in for a loaded tool: `executeTools` in `@librechat/agents`
|
||||
* only resolves a tool by name, reads `schema`/`mcp`, and calls `invoke` — so
|
||||
* the wrapper must leave all of that intact.
|
||||
*/
|
||||
function createTool(
|
||||
name: string,
|
||||
invoke: (input: unknown, config?: unknown) => Promise<unknown>,
|
||||
extra: Record<string, unknown> = {},
|
||||
): StructuredToolInterface {
|
||||
return { name, invoke, ...extra } as unknown as StructuredToolInterface;
|
||||
}
|
||||
|
||||
describe('summarizePtcArgs', () => {
|
||||
it('renders an object input as a key=value line', () => {
|
||||
expect(summarizePtcArgs({ query: 'librechat', limit: 5 })).toBe('query=librechat, limit=5');
|
||||
});
|
||||
|
||||
it('drops empty and nullish values', () => {
|
||||
expect(summarizePtcArgs({ path: 'a.ts', cursor: null, filter: '' })).toBe('path=a.ts');
|
||||
});
|
||||
|
||||
it('collapses whitespace so a multi-line value stays one line', () => {
|
||||
expect(summarizePtcArgs({ code: 'a\n b' })).toBe('code=a b');
|
||||
});
|
||||
|
||||
it('clips a long value without dropping the keys after it', () => {
|
||||
const summary = summarizePtcArgs({ body: 'x'.repeat(200), path: 'a.ts' });
|
||||
expect(summary).toContain('…');
|
||||
expect(summary).toContain('path=a.ts');
|
||||
});
|
||||
|
||||
it('bounds the whole preview', () => {
|
||||
const summary = summarizePtcArgs(
|
||||
Object.fromEntries(Array.from({ length: 20 }, (_, i) => [`key${i}`, `value${i}`])),
|
||||
);
|
||||
expect(summary.length).toBeLessThanOrEqual(97);
|
||||
});
|
||||
|
||||
it('stops once the preview budget is spent instead of visiting every key', () => {
|
||||
const seen: string[] = [];
|
||||
const probe: Record<string, unknown> = {};
|
||||
for (let i = 0; i < 40; i++) {
|
||||
Object.defineProperty(probe, `key${i}`, {
|
||||
enumerable: true,
|
||||
get() {
|
||||
seen.push(`key${i}`);
|
||||
return `value${i}`;
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
summarizePtcArgs(probe);
|
||||
|
||||
expect(seen.length).toBeLessThan(40);
|
||||
});
|
||||
|
||||
it('does not rewrite the whole of an oversized value to build a short preview', () => {
|
||||
const huge = 'a b '.repeat(500_000);
|
||||
const started = Date.now();
|
||||
const summary = summarizePtcArgs({ content: huge, path: 'a.ts' });
|
||||
|
||||
expect(summary).toContain('path=a.ts');
|
||||
expect(summary.length).toBeLessThanOrEqual(97);
|
||||
expect(Date.now() - started).toBeLessThan(150);
|
||||
});
|
||||
|
||||
it('falls back to the raw string for a non-object input', () => {
|
||||
expect(summarizePtcArgs('ls -la')).toBe('ls -la');
|
||||
});
|
||||
|
||||
it('returns an empty preview for an absent input', () => {
|
||||
expect(summarizePtcArgs(undefined)).toBe('');
|
||||
expect(summarizePtcArgs({})).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('instrumentPtcToolMap', () => {
|
||||
const collect = () => {
|
||||
const events: PtcToolCallEvent[] = [];
|
||||
return { events, emit: (event: PtcToolCallEvent) => events.push(event) };
|
||||
};
|
||||
|
||||
it('emits a running event and a success event around an inner call', async () => {
|
||||
const { events, emit } = collect();
|
||||
const toolMap = new Map([['read_file', createTool('read_file', async () => 'file contents')]]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit });
|
||||
const result = await instrumented.get('read_file')?.invoke({ path: 'a.ts' });
|
||||
|
||||
expect(result).toBe('file contents');
|
||||
expect(events).toHaveLength(2);
|
||||
expect(events[0]).toMatchObject({
|
||||
tool_call_id: 'call_1',
|
||||
name: 'read_file',
|
||||
status: 'running',
|
||||
args: 'path=a.ts',
|
||||
});
|
||||
expect(events[1]).toMatchObject({
|
||||
tool_call_id: 'call_1',
|
||||
call_id: events[0].call_id,
|
||||
status: 'success',
|
||||
});
|
||||
expect(events[1].durationMs).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
|
||||
it('reports a failed inner call and rethrows so the sandbox still sees the error', async () => {
|
||||
const { events, emit } = collect();
|
||||
const toolMap = new Map([
|
||||
[
|
||||
'write_file',
|
||||
createTool('write_file', async () => {
|
||||
throw new Error('Permission denied');
|
||||
}),
|
||||
],
|
||||
]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit });
|
||||
|
||||
await expect(instrumented.get('write_file')?.invoke({ path: '/etc/x' })).rejects.toThrow(
|
||||
'Permission denied',
|
||||
);
|
||||
expect(events[1]).toMatchObject({ status: 'error', error: 'Permission denied' });
|
||||
});
|
||||
|
||||
it('gives each inner call its own id so concurrent calls do not collide', async () => {
|
||||
const { events, emit } = collect();
|
||||
const toolMap = new Map([['search', createTool('search', async () => 'ok')]]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit });
|
||||
const search = instrumented.get('search');
|
||||
await Promise.all([search?.invoke({ q: 'a' }), search?.invoke({ q: 'b' })]);
|
||||
|
||||
const startIds = events.filter((e) => e.status === 'running').map((e) => e.call_id);
|
||||
expect(new Set(startIds).size).toBe(2);
|
||||
});
|
||||
|
||||
it('passes the invoke config through untouched', async () => {
|
||||
const { emit } = collect();
|
||||
const seen: unknown[] = [];
|
||||
const toolMap = new Map([
|
||||
[
|
||||
'read_file',
|
||||
createTool('read_file', async (_input, config) => {
|
||||
seen.push(config);
|
||||
return 'ok';
|
||||
}),
|
||||
],
|
||||
]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit });
|
||||
const config = { metadata: { run_tools_with_code: true } };
|
||||
await instrumented.get('read_file')?.invoke({ path: 'a.ts' }, config);
|
||||
|
||||
expect(seen[0]).toBe(config);
|
||||
});
|
||||
|
||||
it('leaves every other property readable on the wrapped tool', () => {
|
||||
const { emit } = collect();
|
||||
const toolMap = new Map([
|
||||
[
|
||||
'search_code_mcp_github',
|
||||
createTool('search_code_mcp_github', async () => 'ok', {
|
||||
mcp: true,
|
||||
schema: { type: 'object' },
|
||||
}),
|
||||
],
|
||||
]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit });
|
||||
const tool = instrumented.get('search_code_mcp_github') as StructuredToolInterface & {
|
||||
mcp?: boolean;
|
||||
};
|
||||
|
||||
expect(tool.name).toBe('search_code_mcp_github');
|
||||
expect(tool.mcp).toBe(true);
|
||||
expect(tool.schema).toEqual({ type: 'object' });
|
||||
});
|
||||
|
||||
it('omits argument and failure previews when tool-argument filtering is on', async () => {
|
||||
const { events, emit } = collect();
|
||||
const toolMap = new Map([
|
||||
[
|
||||
'write_file',
|
||||
createTool('write_file', async () => {
|
||||
throw new Error('rejected value 555-01-0000');
|
||||
}),
|
||||
],
|
||||
]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({
|
||||
toolMap,
|
||||
toolCallId: 'call_1',
|
||||
includePreviews: false,
|
||||
emit,
|
||||
});
|
||||
|
||||
await expect(instrumented.get('write_file')?.invoke({ ssn: '555-01-0000' })).rejects.toThrow();
|
||||
|
||||
/** Name, status and duration still report; nothing derived from the
|
||||
* arguments or the failure text reaches the stream. */
|
||||
expect(events[0].args).toBeUndefined();
|
||||
expect(events[1].error).toBeUndefined();
|
||||
expect(events.map((e) => e.status)).toEqual(['running', 'error']);
|
||||
expect(events[1].durationMs).toBeGreaterThanOrEqual(0);
|
||||
expect(JSON.stringify(events)).not.toContain('555-01-0000');
|
||||
});
|
||||
|
||||
it('emits nothing at all for a tool whose name the policy filters', async () => {
|
||||
const { events, emit } = collect();
|
||||
const toolMap = new Map([
|
||||
['ok_tool', createTool('ok_tool', async () => 'ok')],
|
||||
['blocked_name_tool', createTool('blocked_name_tool', async () => 'ok')],
|
||||
]);
|
||||
|
||||
const instrumented = instrumentPtcToolMap({
|
||||
toolMap,
|
||||
toolCallId: 'call_1',
|
||||
traceExclusions: new Set(['blocked_name_tool']),
|
||||
emit,
|
||||
});
|
||||
|
||||
/** Excluded tools still execute — only their telemetry is suppressed. */
|
||||
await expect(instrumented.get('blocked_name_tool')?.invoke({ a: 1 })).resolves.toBe('ok');
|
||||
await instrumented.get('ok_tool')?.invoke({ a: 1 });
|
||||
|
||||
expect(events.map((e) => e.name)).toEqual(['ok_tool', 'ok_tool']);
|
||||
expect(JSON.stringify(events)).not.toContain('blocked_name_tool');
|
||||
});
|
||||
|
||||
it('runs the inner call even when the emitter throws', async () => {
|
||||
const toolMap = new Map([['read_file', createTool('read_file', async () => 'ok')]]);
|
||||
const instrumented = instrumentPtcToolMap({
|
||||
toolMap,
|
||||
toolCallId: 'call_1',
|
||||
emit: () => {
|
||||
throw new Error('stream closed');
|
||||
},
|
||||
});
|
||||
|
||||
await expect(instrumented.get('read_file')?.invoke({ path: 'a.ts' })).resolves.toBe('ok');
|
||||
});
|
||||
});
|
||||
200
packages/api/src/agents/ptc.ts
Normal file
200
packages/api/src/agents/ptc.ts
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
import type { StructuredToolInterface } from '@librechat/agents/langchain/tools';
|
||||
import type { PtcToolCallEvent } from 'librechat-data-provider';
|
||||
|
||||
/** Whole-preview budget for one inner call's arguments. */
|
||||
const ARGS_PREVIEW_MAX_CHARS = 96;
|
||||
/** Per-value budget, so one long string can't crowd out the other keys. */
|
||||
const ARGS_VALUE_MAX_CHARS = 40;
|
||||
const ERROR_PREVIEW_MAX_CHARS = 160;
|
||||
/**
|
||||
* Collapsing whitespace can only shorten a string, so a window this many times
|
||||
* the visible budget is always long enough to fill it. Slicing to the window
|
||||
* before rewriting matters: this runs synchronously ahead of every inner
|
||||
* `invoke`, and without it a multi-megabyte argument would be collapsed in
|
||||
* full to produce a forty-character preview.
|
||||
*/
|
||||
const CLIP_OVERSCAN = 4;
|
||||
|
||||
/** Bounded collapse-and-clip: never rewrites more of `input` than the budget
|
||||
* can possibly need, and marks any truncation it performed. */
|
||||
const clip = (input: string, max: number): string => {
|
||||
const window = input.length > max * CLIP_OVERSCAN ? input.slice(0, max * CLIP_OVERSCAN) : input;
|
||||
const collapsed = window.replace(/\s+/g, ' ').trim();
|
||||
if (collapsed.length <= max && window.length === input.length) {
|
||||
return collapsed;
|
||||
}
|
||||
return `${collapsed.slice(0, max)}…`;
|
||||
};
|
||||
|
||||
/**
|
||||
* Collapses an inner call's input into a single `key=value, key=value` line
|
||||
* for the CLI-style trace. Values are clipped individually and iteration stops
|
||||
* as soon as the joined preview can no longer grow, so a call with a large
|
||||
* body or many keys costs the same as a small one.
|
||||
*/
|
||||
export function summarizePtcArgs(input: unknown): string {
|
||||
if (input == null) {
|
||||
return '';
|
||||
}
|
||||
if (typeof input === 'string') {
|
||||
return clip(input, ARGS_PREVIEW_MAX_CHARS);
|
||||
}
|
||||
if (typeof input !== 'object' || Array.isArray(input)) {
|
||||
return clip(String(input), ARGS_PREVIEW_MAX_CHARS);
|
||||
}
|
||||
|
||||
const record = input as Record<string, unknown>;
|
||||
const entries: string[] = [];
|
||||
let budget = ARGS_PREVIEW_MAX_CHARS;
|
||||
/* Keys, not entries: `Object.entries` would materialize every value before
|
||||
* the loop starts, so the budget check below could never skip the work it
|
||||
* exists to skip. */
|
||||
for (const key of Object.keys(record)) {
|
||||
if (budget <= 0) {
|
||||
break;
|
||||
}
|
||||
const value = record[key];
|
||||
if (value == null || value === '') {
|
||||
continue;
|
||||
}
|
||||
/* Strings are the values that get large (file bodies, request payloads),
|
||||
* and `clip` bounds them without touching the tail. Everything else is
|
||||
* small enough that serializing it first is cheaper than inspecting it. */
|
||||
const rendered = typeof value === 'string' ? value : safeStringify(value);
|
||||
if (rendered === '') {
|
||||
continue;
|
||||
}
|
||||
const entry = `${key}=${clip(rendered, ARGS_VALUE_MAX_CHARS)}`;
|
||||
entries.push(entry);
|
||||
budget -= entry.length + 2;
|
||||
}
|
||||
|
||||
return clip(entries.join(', '), ARGS_PREVIEW_MAX_CHARS);
|
||||
}
|
||||
|
||||
function safeStringify(value: unknown): string {
|
||||
try {
|
||||
return JSON.stringify(value) ?? '';
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
export interface InstrumentPtcToolMapParams {
|
||||
/** The tool map the PTC runner resolves inner calls against. */
|
||||
toolMap: Map<string, StructuredToolInterface>;
|
||||
/** The PTC run step's tool call id — the card the trace renders under. */
|
||||
toolCallId: string;
|
||||
runId?: string;
|
||||
/**
|
||||
* Whether argument and failure previews may ride the stream. False when the
|
||||
* deployment filters tool arguments for PII: inner calls never pass through
|
||||
* `filteredToolArgumentsResult` (the sandbox bridge invokes them directly),
|
||||
* so a preview would put values on the wire that the configured policy
|
||||
* exists to keep off it — and a failure message routinely quotes the very
|
||||
* argument that caused it. The trace still reports name, status and duration.
|
||||
*/
|
||||
includePreviews?: boolean;
|
||||
/**
|
||||
* Inner tools whose *name* trips the deployment's PII policy. The event
|
||||
* carries the tool name unconditionally, so a name the `name` filter would
|
||||
* have blocked on a direct call cannot be allowed to ride the trace instead.
|
||||
* These tools still execute — they are simply left unwrapped, so no event
|
||||
* about them is ever emitted.
|
||||
*/
|
||||
traceExclusions?: ReadonlySet<string>;
|
||||
emit: (event: PtcToolCallEvent) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wraps every tool the PTC sandbox can reach so each inner invocation reports
|
||||
* its lifecycle on the live stream. The runner (`executeTools` in
|
||||
* `@librechat/agents`) resolves a tool by name and calls `invoke` on it, so a
|
||||
* `Proxy` intercepting only `invoke` is enough — `name`, `schema`, `mcp` and
|
||||
* every other property the runner reads pass straight through to the real
|
||||
* tool, and nothing about execution changes.
|
||||
*/
|
||||
export function instrumentPtcToolMap({
|
||||
toolMap,
|
||||
toolCallId,
|
||||
runId,
|
||||
includePreviews = true,
|
||||
traceExclusions,
|
||||
emit,
|
||||
}: InstrumentPtcToolMapParams): Map<string, StructuredToolInterface> {
|
||||
let sequence = 0;
|
||||
/** Emission is telemetry: a dead stream must never fail the program. */
|
||||
const safeEmit = (event: PtcToolCallEvent): void => {
|
||||
try {
|
||||
emit(event);
|
||||
} catch {
|
||||
/* stream closed or transport rejected — the run continues */
|
||||
}
|
||||
};
|
||||
|
||||
const instrumented = new Map<string, StructuredToolInterface>();
|
||||
for (const [name, tool] of toolMap) {
|
||||
if (traceExclusions?.has(name)) {
|
||||
instrumented.set(name, tool);
|
||||
continue;
|
||||
}
|
||||
instrumented.set(
|
||||
name,
|
||||
new Proxy(tool, {
|
||||
get(target, property) {
|
||||
if (property !== 'invoke') {
|
||||
/** `target` as the receiver, not the proxy: LangChain tools read
|
||||
* private class state through their own getters. */
|
||||
return Reflect.get(target, property, target);
|
||||
}
|
||||
return async (input: unknown, config?: unknown): Promise<unknown> => {
|
||||
const callId = `${toolCallId}:${sequence++}`;
|
||||
const startedAt = Date.now();
|
||||
safeEmit({
|
||||
tool_call_id: toolCallId,
|
||||
call_id: callId,
|
||||
name,
|
||||
status: 'running',
|
||||
...(includePreviews ? { args: summarizePtcArgs(input) } : {}),
|
||||
...(runId != null ? { runId } : {}),
|
||||
});
|
||||
try {
|
||||
const result = await (
|
||||
target.invoke as (input: unknown, config?: unknown) => Promise<unknown>
|
||||
).call(target, input, config);
|
||||
safeEmit({
|
||||
tool_call_id: toolCallId,
|
||||
call_id: callId,
|
||||
name,
|
||||
status: 'success',
|
||||
durationMs: Date.now() - startedAt,
|
||||
...(runId != null ? { runId } : {}),
|
||||
});
|
||||
return result;
|
||||
} catch (error) {
|
||||
safeEmit({
|
||||
tool_call_id: toolCallId,
|
||||
call_id: callId,
|
||||
name,
|
||||
status: 'error',
|
||||
...(includePreviews
|
||||
? {
|
||||
error: clip(
|
||||
error instanceof Error ? error.message : String(error),
|
||||
ERROR_PREVIEW_MAX_CHARS,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
durationMs: Date.now() - startedAt,
|
||||
...(runId != null ? { runId } : {}),
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
},
|
||||
}) as StructuredToolInterface,
|
||||
);
|
||||
}
|
||||
|
||||
return instrumented;
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue