From c2aa688d73e463d8b9bca9544c23331d598a8d64 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Sun, 23 Aug 2026 01:18:14 -0400 Subject: [PATCH] =?UTF-8?q?=F0=9F=96=A5=EF=B8=8F=20feat:=20Stream=20PTC=20?= =?UTF-8?q?Inner=20Tool=20Calls=20as=20a=20CLI-Style=20Trace=20(#15115)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ๐Ÿ–ฅ๏ธ feat: Stream PTC Inner Tool Calls as a CLI-Style Trace Programmatic tool calling runs a whole program inside the sandbox, and the tool calls that program makes open no run step of their own. The card showed one running spinner for the entire execution, with no sign of what the code was doing. Emit a new `on_ptc_tool_call` step event for each inner invocation โ€” once on dispatch, once on settle โ€” and render them under the code as a terminal-style trace: status glyph, tool identity, argument preview, duration, with a failure message printed under the call that produced it. The seam is the tool map the sandbox bridge resolves inner calls against. `instrumentPtcToolMap` proxies `invoke` on each entry and leaves every other property (name, schema, mcp) passing straight through, so nothing about execution changes and emission failures can never fail a tool call. Client state is a per-tool-call Recoil atom keyed like the sandbox-starting and subagent atoms โ€” live for the session, cleared on conversation switch so a finished program's trace stays readable. * ๐Ÿฉน fix: Address Codex Review on the PTC Tool Trace Five findings, all confirmed against the source before fixing. Scope the trace atoms to a message occurrence. The hook already documents that providers repeat a tool_call_id across turns and even within one message, and `call_id` restarts at :0 for every outer call โ€” so two programs sharing `call_0` merged into one card. Key by (response message id, tool call id) via `ptcTraceKey`, mirroring `subagentProgressKey`; the event's `runId` already carries the message id and the card reads its own from MessageContext. Prune unsettled rows on resume. Inner calls are not content parts, so the resume snapshot cannot rebuild them, and `trackReplayEvent` only persists OAuth events โ€” a call that settled during a disconnect left a spinner that never resolved. Settled rows are real history and stay. Make the argument preview budget-aware. Iterate keys rather than entries so the budget check can actually skip work, and clip against a bounded window so a multi-megabyte value is never collapsed in full to build a 40-character preview. Catch the resumable emission promise. The synchronous try/catch around the emitter cannot observe a rejected `emitChunk`, so a failing transport raised an unhandled rejection per event instead of dropping telemetry. Announce completion to assistive technology. The check glyph is decorative and a fast call renders no duration, so a settled row previously announced no outcome; each row now carries an sr-only status and the visible cell that duplicated it is hidden. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ๐Ÿงน fix: Repair CI Failures on the PTC Tool Trace Two failures on the previous head, both mine. `Tests: api (shard 2/3)` โ€” 46 failures in `initialize.spec.js`, all `TypeError: createPtcProgressEmitter is not a function`. The suite mocks the callbacks module with an object literal, and wiring the new emitter into `initialize.js` without adding it there left the factory undefined at call time. Added it alongside `createAttachmentEmitter`, plus an assertion that it receives the same generation fence as every other resumable emitter โ€” a stale epoch would leak one run's inner calls into the next. `Static checks` โ€” import-order drift in `PtcToolTrace.tsx` and `handlers.ts`, repaired with `scripts/sort-imports.mts`. ESLint and Prettier both passed, so only the dedicated check caught it. `openai.js` and `responses.js` never take the emitter, so their specs were unaffected; verified the initialize mock now covers every name the module destructures. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ๐Ÿ” fix: Address Second Codex Review on the PTC Tool Trace Three of five findings actioned; two answered on the thread. Respect tool-argument PII filtering (P1). Inner calls never reach `filteredToolArgumentsResult` โ€” the sandbox bridge invokes them directly โ€” so the trace was the one path putting their values on the wire in a deployment that had configured `filters.toolArguments.pii`. When any of the name / arguments / output fields are filtered, the emitter now omits both the argument preview and the failure message, which routinely quotes the argument that caused it. Name, status and duration still report. Drop the light/dark-specific background (P1). `dark:bg-transparent` stepped outside the semantic roles and would lose the intended separation under a custom theme. The pane now sets no background at all and inherits the card's surface, which resolves to the same color the override produced in both default themes and stays correct when a theme reassigns its roles. Bound the live trace (P2). A program looping over a large collection made every event copy an ever-growing array and rendered a row per call. The trace now keeps a rolling tail of 100 rows and counts what it evicted, surfaced as "+N earlier calls" so the cap is never silent. A settle whose row is gone โ€” evicted, or pruned across a resume gap โ€” no longer reappears out of order. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * โœ… test: Keep PTC Trace Tests Aligned With Caller-Capability Filtering Left out of the merge commit by a staging slip; without them `handlers.spec.ts` fails on the merged tree. `#15105` restricts the PTC tool map to tools whose `allowed_callers` admit code execution, so the existing trace test's registry entry โ€” which declared none, defaulting to `direct` โ€” was filtered out before the instrumentation could see it. Declare the fixture `code_execution`. Add a guard for the resolution itself: a `direct`-only tool must never appear in the instrumented map. Tracing wraps the eligible map, and this fails if a later change reorders that and lets the trace widen what the sandbox reaches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ๐Ÿ›ก๏ธ fix: Close Name Disclosure and Follow the PTC Trace Tail Two findings from the third Codex pass on `17a9ec9`. Redact filtered inner-tool names (P1). The previous gate suppressed argument and failure previews but the event still carried `name` verbatim, so a deployment whose `filters.toolArguments.pii.fields` includes `name` could see a blocked identifier disclosed through the trace โ€” the one path inner calls take, since they never reach `filteredToolArgumentsResult`. Inner tool names are now inspected once per PTC call with the same `extractToolArgumentContent` + `inspectContent` pair the executor uses; any that trip the policy are left unwrapped, so they still execute and emit nothing. An un-inspectable name fails closed. Follow the trace tail (P2). The row list is a 200px scroller that never moved, so once a program exceeded the viewport the card sat on the oldest calls while live activity accumulated below the fold. Reuse `useFollowScroll` โ€” the hook the code and command panes already use โ€” which pins to the tail while calls are running and yields the moment the reader scrolls up. The host card threads its disclosure state so a collapsed pane is never scrolled invisibly. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ๐Ÿ“Œ fix: Pin the PTC Trace Through Its Final Settle The fourth Codex pass on `4bf68e1`, one P2 finding. `useFollowScroll` returned early whenever `active` was false, so the one change it most needed to follow was the one it skipped. A failing inner call settles by appending its error line in the same commit that clears the last running row: the content grows and the stream ends together, and the pin that would have revealed that line never fired. On an expanded, bottom-pinned pane the failure โ€” the row a reader most wants โ€” stayed below the fold. The falling edge of `active` now pins too, but only when the content changed with it. Ending a stream on its own still leaves the pane where the reader left it, which is what the existing contract promises and what the sibling code and command panes rely on; a reader who has scrolled up is untouched either way. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP * ๐Ÿ”Œ fix: Keep PTC Calls That Outlive a Reconnect Fifth Codex pass on `085a83f`; one of its two findings. Pruning rows across a resume gap deleted every `running` row, but a stream gap is not proof the call ended. A call still executing across the reconnect settles normally on the restored live stream โ€” and `applyPtcToolCall` drops a settle whose row is gone, by design, so an evicted row cannot reappear out of order. The call therefore vanished from the trace despite having run, which is worse than the spinner the pruning existed to prevent. Rows are now marked `interrupted` instead of removed. A call whose settle was genuinely lost in the gap reports that honestly rather than spinning forever, and one that survives the gap settles onto the row it opened, reporting its real outcome and duration. `interrupted` is a client-side conclusion, so it widens the row status locally and leaves the wire contract alone. Two cases added: the gap marks rather than drops, and a post-reconnect settle lands on its marked row; plus a render case for the new outcome. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MBRQUsPXSvDBnjrSDoXWHP --------- Co-authored-by: Claude --- .../agents/__tests__/callbacks.spec.js | 78 ++++- api/server/controllers/agents/callbacks.js | 39 +++ .../services/Endpoints/agents/initialize.js | 2 + .../Endpoints/agents/initialize.spec.js | 9 + .../Chat/Messages/Content/Parts/BashCall.tsx | 6 + .../Messages/Content/Parts/ExecuteCode.tsx | 6 + .../Messages/Content/Parts/PtcToolTrace.tsx | 170 ++++++++++ .../Parts/__tests__/ExecuteCode.test.tsx | 48 +++ .../Parts/__tests__/PtcToolTrace.test.tsx | 140 ++++++++ .../Parts/__tests__/useFollowScroll.spec.tsx | 28 ++ .../Chat/Messages/Content/Parts/index.ts | 1 + .../Messages/Content/Parts/useFollowScroll.ts | 23 +- .../SSE/__tests__/useResumableSSE.spec.ts | 1 + .../SSE/__tests__/useStepHandler.spec.ts | 314 ++++++++++++++++++ client/src/hooks/SSE/useEventHandlers.ts | 12 +- client/src/hooks/SSE/useResumableSSE.ts | 10 + client/src/hooks/SSE/useStepHandler.ts | 122 ++++++- client/src/locales/en/translation.json | 6 + client/src/store/index.ts | 1 + client/src/store/ptc.ts | 64 ++++ packages/api/src/agents/handlers.spec.ts | 80 +++++ packages/api/src/agents/handlers.ts | 84 ++++- packages/api/src/agents/index.ts | 1 + packages/api/src/agents/ptc.spec.ts | 247 ++++++++++++++ packages/api/src/agents/ptc.ts | 200 +++++++++++ packages/data-provider/src/types/runs.ts | 27 ++ 26 files changed, 1707 insertions(+), 12 deletions(-) create mode 100644 client/src/components/Chat/Messages/Content/Parts/PtcToolTrace.tsx create mode 100644 client/src/components/Chat/Messages/Content/Parts/__tests__/PtcToolTrace.test.tsx create mode 100644 client/src/store/ptc.ts create mode 100644 packages/api/src/agents/ptc.spec.ts create mode 100644 packages/api/src/agents/ptc.ts diff --git a/api/server/controllers/agents/__tests__/callbacks.spec.js b/api/server/controllers/agents/__tests__/callbacks.spec.js index 541edfdeb2..9b1f3b6eb7 100644 --- a/api/server/controllers/agents/__tests__/callbacks.spec.js +++ b/api/server/controllers/agents/__tests__/callbacks.spec.js @@ -1,4 +1,4 @@ -const { Tools } = require('librechat-data-provider'); +const { Tools, StepEvents } = require('librechat-data-provider'); // Mock all dependencies before requiring the module jest.mock('nanoid', () => ({ @@ -144,6 +144,82 @@ describe('resumable event generation fencing', () => { }); }); +describe('createPtcProgressEmitter', () => { + const ptcEvent = { + tool_call_id: 'call_ptc', + call_id: 'call_ptc:0', + name: 'read_file', + status: 'running', + args: 'path=a.ts', + }; + + beforeEach(() => jest.clearAllMocks()); + + it('emits the inner tool-call event on the resumable job stream', () => { + const { GenerationJobManager } = require('@librechat/api'); + const { createPtcProgressEmitter } = require('../callbacks'); + const emit = createPtcProgressEmitter({ + res: { write: jest.fn() }, + streamId: 'conversation-1', + jobCreatedAt: 1234, + }); + + emit(ptcEvent); + + expect(GenerationJobManager.emitChunk).toHaveBeenCalledWith( + 'conversation-1', + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent }, + { expectedCreatedAt: 1234 }, + ); + }); + + it('writes to the live response when no stream id is in play', () => { + const { sendEvent } = require('@librechat/api'); + const { createPtcProgressEmitter } = require('../callbacks'); + const res = { write: jest.fn(), headersSent: true, writableEnded: false }; + const emit = createPtcProgressEmitter({ res }); + + emit(ptcEvent); + + expect(sendEvent).toHaveBeenCalledWith(res, { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent, + }); + }); + + it('absorbs a rejected resumable emit instead of leaving an unhandled rejection', async () => { + const { GenerationJobManager } = require('@librechat/api'); + const { createPtcProgressEmitter } = require('../callbacks'); + GenerationJobManager.emitChunk.mockRejectedValueOnce(new Error('transport down')); + const unhandled = jest.fn(); + process.on('unhandledRejection', unhandled); + + const emit = createPtcProgressEmitter({ + res: { write: jest.fn() }, + streamId: 'conversation-1', + jobCreatedAt: 1234, + }); + + expect(() => emit(ptcEvent)).not.toThrow(); + await new Promise((resolve) => setImmediate(resolve)); + process.off('unhandledRejection', unhandled); + + expect(unhandled).not.toHaveBeenCalled(); + }); + + it('drops the event once the response has closed', () => { + const { sendEvent } = require('@librechat/api'); + const { createPtcProgressEmitter } = require('../callbacks'); + const emit = createPtcProgressEmitter({ + res: { write: jest.fn(), headersSent: true, writableEnded: true }, + }); + + emit(ptcEvent); + + expect(sendEvent).not.toHaveBeenCalled(); + }); +}); + describe('createToolEndCallback', () => { let req, res, artifactPromises, createToolEndCallback; let logger; diff --git a/api/server/controllers/agents/callbacks.js b/api/server/controllers/agents/callbacks.js index 1eb56cf284..7e6f5f4af7 100644 --- a/api/server/controllers/agents/callbacks.js +++ b/api/server/controllers/agents/callbacks.js @@ -1084,6 +1084,44 @@ function createAttachmentEmitter({ res, streamId = null, jobCreatedAt }) { }; } +/** + * Streams `on_ptc_tool_call` lifecycle events for the tool calls a + * programmatic tool-calling program makes from inside the sandbox. Those + * inner calls open no run step of their own, so without this the card shows + * a running spinner for the whole program with no sign of what it is doing. + * + * Fire-and-forget like the attachment emitter: a closed stream drops the + * event rather than failing the tool call that produced it. + * + * @param {Object} params + * @param {ServerResponse} params.res + * @param {string | null} [params.streamId] + * @param {number} [params.jobCreatedAt] + * @returns {(event: import('librechat-data-provider').PtcToolCallEvent) => void} + */ +function createPtcProgressEmitter({ res, streamId = null, jobCreatedAt }) { + return (event) => { + if (!event || !isStreamWritable(res, streamId)) { + return; + } + const payload = { event: StepEvents.ON_PTC_TOOL_CALL, data: event }; + if (streamId) { + /* Absorb a rejected transport here. The emitter is called from a + * synchronous try/catch inside `instrumentPtcToolMap`, which cannot + * observe a rejected promise โ€” without this catch a failed emit would + * surface as an unhandled rejection on every affected inner call + * instead of being dropped as the telemetry it is. */ + Promise.resolve( + GenerationJobManager.emitChunk(streamId, payload, { expectedCreatedAt: jobCreatedAt }), + ).catch(() => { + /* dropped: the trace is best-effort */ + }); + return; + } + sendEvent(res, payload); + }; +} + /** * Leading sub-second retries cover the common case of a fast background task * settling moments before the dispatch turn finalizes its message row โ€” an @@ -1438,6 +1476,7 @@ module.exports = { getDefaultHandlers, createToolEndCallback, createAttachmentEmitter, + createPtcProgressEmitter, createBackgroundCodeResultHandler, isStreamWritable, markSummarizationUsage, diff --git a/api/server/services/Endpoints/agents/initialize.js b/api/server/services/Endpoints/agents/initialize.js index 1e818b481a..4a3a5aef6e 100644 --- a/api/server/services/Endpoints/agents/initialize.js +++ b/api/server/services/Endpoints/agents/initialize.js @@ -39,6 +39,7 @@ const { const { createToolEndCallback, createAttachmentEmitter, + createPtcProgressEmitter, createBackgroundCodeResultHandler, getDefaultHandlers, } = require('~/server/controllers/agents/callbacks'); @@ -374,6 +375,7 @@ const initializeClient = async ({ updateToolCallResult: db.updateToolCallResult, }), emitAttachment: createAttachmentEmitter({ res, streamId, jobCreatedAt }), + emitPtcProgress: createPtcProgressEmitter({ res, streamId, jobCreatedAt }), ...getSkillToolDeps(), }; diff --git a/api/server/services/Endpoints/agents/initialize.spec.js b/api/server/services/Endpoints/agents/initialize.spec.js index 927291b6b1..66335332ef 100644 --- a/api/server/services/Endpoints/agents/initialize.spec.js +++ b/api/server/services/Endpoints/agents/initialize.spec.js @@ -49,6 +49,7 @@ const mockArtifactToolEndCallback = jest.fn(); jest.mock('~/server/controllers/agents/callbacks', () => ({ createToolEndCallback: jest.fn(() => mockArtifactToolEndCallback), createAttachmentEmitter: jest.fn(() => jest.fn()), + createPtcProgressEmitter: jest.fn(() => jest.fn()), createBackgroundCodeResultHandler: jest.fn(() => jest.fn()), getDefaultHandlers: jest.fn((opts) => { capturedDefaultHandlerOptions = opts; @@ -206,6 +207,7 @@ describe('initializeClient โ€” processAgent ACL gate', () => { it('threads the owning job epoch into resumable event handlers', async () => { const { createAttachmentEmitter, + createPtcProgressEmitter, createToolEndCallback, } = require('~/server/controllers/agents/callbacks'); mockInitializeAgent.mockResolvedValue(makePrimaryConfig([])); @@ -237,6 +239,13 @@ describe('initializeClient โ€” processAgent ACL gate', () => { streamId: 'conv_1', jobCreatedAt: 1234, }); + /** The PTC trace emitter is generation-fenced like every other resumable + * emitter; a stale epoch would leak one run's inner calls into the next. */ + expect(createPtcProgressEmitter).toHaveBeenCalledWith({ + res: {}, + streamId: 'conv_1', + jobCreatedAt: 1234, + }); mockLoadToolsForExecution.mockResolvedValue({ loadedTools: [], configurable: {} }); await capturedToolExecuteOptions.loadTools([], PRIMARY_ID); diff --git a/client/src/components/Chat/Messages/Content/Parts/BashCall.tsx b/client/src/components/Chat/Messages/Content/Parts/BashCall.tsx index 570e370370..b50dfcbaf6 100644 --- a/client/src/components/Chat/Messages/Content/Parts/BashCall.tsx +++ b/client/src/components/Chat/Messages/Content/Parts/BashCall.tsx @@ -14,6 +14,7 @@ import useFollowScroll from './useFollowScroll'; import { ERROR_PATTERNS } from './ExecuteCode'; import { AttachmentGroup } from './Attachment'; import { useToolCallIntent } from './intent'; +import PtcToolTrace from './PtcToolTrace'; import { useLocalize } from '~/hooks'; import { cn } from '~/utils'; @@ -176,6 +177,11 @@ export default function BashCall({ )} + {hasOutput && backgroundHandle == null && ( 
{highlighted}
)} + {hasOutput && backgroundHandle == null && (
= { + running: 'โ€บ', + success: 'โœ“', + error: 'โœ—', + interrupted: '?', +}; + +/** Spoken equivalent of the glyph. The glyph itself is decorative, and a + * fast call renders no duration, so without this a completed row would + * announce no outcome at all. */ +const STATUS_LABEL_KEYS: Record = { + running: 'com_ui_ptc_trace_running', + success: 'com_ui_ptc_trace_done', + error: 'com_ui_ptc_trace_failed', + interrupted: 'com_ui_ptc_trace_interrupted', +}; + +/** + * One line of the trace. Reads as a shell transcript โ€” status glyph, the tool + * being called, its arguments, and how long it took โ€” with the failure message + * printed underneath the call that produced it, the way a CLI reports errors. + */ +function PtcTraceLine({ entry }: { entry: PtcTraceEntry }) { + const localize = useLocalize(); + const { i18n } = useTranslation(); + const mcpServerNames = useMCPServerNames(); + const parsed = parseToolName(entry.name, mcpServerNames); + const running = entry.status === 'running'; + const failed = entry.status === 'error'; + const duration = isReportableRunStepDuration(entry.durationMs) + ? getRunStepDurationLabels(entry.durationMs as number, i18n.language) + : undefined; + + return ( +
  • +
    + + {localize(STATUS_LABEL_KEYS[entry.status])} + + {parsed.mcpServer && ( + <> + {parsed.mcpServer} + ยท + + )} + {parsed.friendlyKey ? localize(parsed.friendlyKey) : parsed.toolName} + + {entry.args ?? ''} + {duration ? ( + <> + + + {localize(duration.announcedKey, duration.announcedValues)} + + + ) : ( + /* The sr-only status above already speaks this cell's meaning. */ + + )} +
    + {failed && ( +
    +
    + )} +
    • + ); +} + +/** + * Live trace of the tool calls a programmatic tool-calling program makes from + * inside the sandbox, rendered under the code that is making them. Those inner + * calls open no run step and get no card of their own, so without this the + * card shows one running spinner for the whole program. + * + * Renders nothing when the tool call made no inner calls, which is every + * non-PTC code execution โ€” callers don't need to know which kind they hold. + */ +export default function PtcToolTrace({ + toolCallId, + expanded = false, + className, +}: { + toolCallId?: string; + /** The host card's disclosure state โ€” a collapsed pane must not be scrolled + * invisibly, or it opens at the tail instead of the first call. */ + expanded?: boolean; + className?: string; +}) { + const localize = useLocalize(); + /** Scope to this card's own message: providers reuse `tool_call_id` across + * turns, so the raw id alone would show a later program's calls here. */ + const { messageId } = useMessageContext(); + const trace = useRecoilValue( + ptcTraceByToolCallId(toolCallId && messageId ? ptcTraceKey(messageId, toolCallId) : ''), + ); + + /** One character per row, so the pin re-fires both when a call is appended + * and when one settles โ€” a settle can add an error line and change height. */ + const followKey = `${trace.dropped}:${trace.entries.map((entry) => entry.status[0]).join('')}`; + const running = trace.entries.some((entry) => entry.status === 'running'); + const { ref: listRef, onScroll } = useFollowScroll( + followKey, + running, + expanded, + ); + + if (trace.entries.length === 0) { + return null; + } + + /** No background of its own: the pane inherits the card's surface in both + * themes, which is what the sibling output pane resolves to anyway. A + * `dark:` override here would step outside the semantic roles and lose the + * intended separation under a custom theme. */ + return ( +
    +
    + {localize('com_ui_ptc_trace_title')} +
    +
      + {/* The retained tail is a window, not the whole program โ€” say so + rather than let the reader assume these are all the calls. */} + {trace.dropped > 0 && ( +
    1. + {localize('com_ui_ptc_trace_earlier', { count: trace.dropped })} +
      2. + )} + {trace.entries.map((entry) => ( + + ))} +
    +
    + ); +} diff --git a/client/src/components/Chat/Messages/Content/Parts/__tests__/ExecuteCode.test.tsx b/client/src/components/Chat/Messages/Content/Parts/__tests__/ExecuteCode.test.tsx index 6e37ed1bab..8b0c39b14c 100644 --- a/client/src/components/Chat/Messages/Content/Parts/__tests__/ExecuteCode.test.tsx +++ b/client/src/components/Chat/Messages/Content/Parts/__tests__/ExecuteCode.test.tsx @@ -1,6 +1,9 @@ import React from 'react'; import { RecoilRoot } from 'recoil'; import { render } from '@testing-library/react'; +import type { PtcTraceEntry } from '~/store/ptc'; +import { ptcTraceByToolCallId, ptcTraceKey } from '~/store/ptc'; +import { MessageContext } from '~/Providers/MessageContext'; import ExecuteCode from '../ExecuteCode'; import store from '~/store'; @@ -55,9 +58,14 @@ jest.mock('../useLazyHighlight', () => { }); jest.mock('~/utils', () => ({ + ...jest.requireActual('~/utils/toolLabels'), + ...jest.requireActual('~/utils/runStepDuration'), cn: (...classes: Array) => classes.filter(Boolean).join(' '), })); +jest.mock('react-i18next', () => ({ useTranslation: () => ({ i18n: { language: 'en' } }) })); +jest.mock('~/hooks/MCP', () => ({ useMCPServerNames: () => [] })); + /** * jsdom has no layout, so the capped pane's scroll geometry is stubbed: * `clientHeight` is fixed, `scrollHeight` either reads from mutable state @@ -134,3 +142,43 @@ describe('ExecuteCode streaming follow-scroll', () => { expect(state.writes).toHaveLength(0); }); }); + +describe('ExecuteCode programmatic tool trace', () => { + const TOOL_CALL_ID = 'call_ptc_1'; + const MESSAGE_ID = 'response-msg-1'; + + const renderWithTrace = (entries: PtcTraceEntry[]) => + render( + { + set(store.autoExpandTools, true); + set(ptcTraceByToolCallId(ptcTraceKey(MESSAGE_ID, TOOL_CALL_ID)), { entries, dropped: 0 }); + }} + > + + + + , + ); + + it('lists the inner tool calls the running program has made', () => { + const { getByText } = renderWithTrace([ + { callId: 'a', name: 'read_file', status: 'success', args: 'path=a.ts', durationMs: 1200 }, + ]); + + expect(getByText('read_file')).toBeInTheDocument(); + expect(getByText('path=a.ts')).toBeInTheDocument(); + }); + + it('adds nothing to a plain code execution that made no inner calls', () => { + const { queryByText } = renderWithTrace([]); + + expect(queryByText('com_ui_ptc_trace_title')).not.toBeInTheDocument(); + }); +}); diff --git a/client/src/components/Chat/Messages/Content/Parts/__tests__/PtcToolTrace.test.tsx b/client/src/components/Chat/Messages/Content/Parts/__tests__/PtcToolTrace.test.tsx new file mode 100644 index 0000000000..bab6809622 --- /dev/null +++ b/client/src/components/Chat/Messages/Content/Parts/__tests__/PtcToolTrace.test.tsx @@ -0,0 +1,140 @@ +import React from 'react'; +import { RecoilRoot, useSetRecoilState } from 'recoil'; +import { render, screen } from '@testing-library/react'; +import type { PtcTraceEntry } from '~/store/ptc'; +import { ptcTraceByToolCallId, ptcTraceKey } from '~/store/ptc'; +import { MessageContext } from '~/Providers/MessageContext'; +import PtcToolTrace from '../PtcToolTrace'; + +jest.mock('~/hooks', () => ({ + useLocalize: + () => + (key: string, values?: Record): string => { + const translations: Record = { + com_ui_ptc_trace_title: 'Tool calls', + com_ui_ptc_trace_running: 'running', + com_ui_ptc_trace_failed: 'failed', + com_ui_ptc_trace_done: 'done', + com_ui_ptc_trace_interrupted: 'interrupted', + com_ui_ptc_trace_earlier: `+${values?.count} earlier calls`, + com_ui_tool_name_code: 'Code', + }; + if (key === 'com_ui_duration_seconds') { + return `${values?.[0]}s`; + } + return translations[key] ?? key; + }, +})); + +jest.mock('react-i18next', () => ({ useTranslation: () => ({ i18n: { language: 'en' } }) })); +jest.mock('~/hooks/MCP', () => ({ useMCPServerNames: () => ['github'] })); +jest.mock('~/utils', () => ({ + ...jest.requireActual('~/utils/toolLabels'), + ...jest.requireActual('~/utils/runStepDuration'), + cn: (...classes: unknown[]) => classes.filter(Boolean).join(' '), +})); + +const TOOL_CALL_ID = 'call_ptc_1'; +const MESSAGE_ID = 'response-msg-1'; + +function renderTrace(entries: PtcTraceEntry[], messageId = MESSAGE_ID, dropped = 0) { + const Seed = () => { + const set = useSetRecoilState(ptcTraceByToolCallId(ptcTraceKey(MESSAGE_ID, TOOL_CALL_ID))); + React.useEffect(() => { + if (entries.length > 0) { + set({ entries, dropped }); + } + }, [set]); + return null; + }; + + return render( + + + + + + , + ); +} + +describe('PtcToolTrace', () => { + it('renders nothing when the call made no inner tool calls', () => { + const { container } = renderTrace([]); + expect(container).toBeEmptyDOMElement(); + }); + + it('lists each inner call with its arguments', () => { + renderTrace([ + { callId: 'a', name: 'read_file', status: 'success', args: 'path=a.ts', durationMs: 1200 }, + { callId: 'b', name: 'write_file', status: 'running', args: 'path=b.ts' }, + ]); + + expect(screen.getByText('Tool calls')).toBeInTheDocument(); + expect(screen.getByText('read_file')).toBeInTheDocument(); + expect(screen.getByText('path=a.ts')).toBeInTheDocument(); + expect(screen.getByText('write_file')).toBeInTheDocument(); + /* Twice by design: the visible cell plus the screen-reader status. */ + expect(screen.getAllByText('running')).toHaveLength(2); + }); + + it('splits an MCP tool id into its server and tool name', () => { + renderTrace([{ callId: 'a', name: 'search_code_mcp_github', status: 'running' }]); + + expect(screen.getByText('github')).toBeInTheDocument(); + expect(screen.getByText('search_code')).toBeInTheDocument(); + }); + + it('shows a settled duration', () => { + renderTrace([{ callId: 'a', name: 'read_file', status: 'success', durationMs: 1200 }]); + + expect(screen.getByText('1.2s')).toBeInTheDocument(); + }); + + it('prints the failure message under the call that produced it', () => { + renderTrace([ + { callId: 'a', name: 'write_file', status: 'error', args: 'path=/etc/x', error: 'Denied' }, + ]); + + expect(screen.getByText('path=/etc/x')).toBeInTheDocument(); + expect(screen.getByText('Denied')).toBeInTheDocument(); + }); + + it('falls back to a generic failure label when no message came through', () => { + renderTrace([{ callId: 'a', name: 'write_file', status: 'error' }]); + + expect(screen.getAllByText('failed').length).toBeGreaterThan(0); + }); + + it('speaks a completion status even when the call was too fast to time', () => { + renderTrace([{ callId: 'a', name: 'read_file', status: 'success' }]); + + expect(screen.getByText('done')).toBeInTheDocument(); + }); + + it('speaks an interrupted outcome for a call a stream gap cut off', () => { + renderTrace([{ callId: 'c1', name: 'write_file', status: 'interrupted' }]); + expect(screen.getByText('interrupted')).toBeInTheDocument(); + /** Neither a failure nor still in flight: no error styling, no spinner. */ + expect(screen.queryByText('failed')).not.toBeInTheDocument(); + expect(screen.queryByText('running')).not.toBeInTheDocument(); + }); + + it('says how many calls the cap dropped rather than truncating silently', () => { + renderTrace([{ callId: 'a', name: 'read_file', status: 'success' }], MESSAGE_ID, 42); + + expect(screen.getByText('+42 earlier calls')).toBeInTheDocument(); + }); + + it('shows no truncation notice when nothing was dropped', () => { + renderTrace([{ callId: 'a', name: 'read_file', status: 'success' }]); + + expect(screen.queryByText(/earlier calls/)).not.toBeInTheDocument(); + }); + + it('does not show another message\u2019s trace in this card', () => { + renderTrace([{ callId: 'a', name: 'read_file', status: 'success' }], 'a-different-message'); + + expect(screen.queryByText('read_file')).not.toBeInTheDocument(); + }); +}); diff --git a/client/src/components/Chat/Messages/Content/Parts/__tests__/useFollowScroll.spec.tsx b/client/src/components/Chat/Messages/Content/Parts/__tests__/useFollowScroll.spec.tsx index 8573d53b64..4a996c7ca9 100644 --- a/client/src/components/Chat/Messages/Content/Parts/__tests__/useFollowScroll.spec.tsx +++ b/client/src/components/Chat/Messages/Content/Parts/__tests__/useFollowScroll.spec.tsx @@ -112,6 +112,34 @@ describe('useFollowScroll', () => { expect(state.writes).toEqual([900]); }); + /** A failing PTC call settles by appending its error line in the same + * commit that clears the last running row, so the growth and the end of + * the stream arrive together. */ + it('pins one last time when the change that ends the stream also adds content', () => { + const { rerender, state } = setup(); + state.scrollHeight = 1200; + rerender(); + expect(state.writes).toEqual([1200]); + }); + + it('does not pin the settling change for a reader who scrolled up', () => { + const { rerender, pane, state } = setup(); + state.scrollTop = 100; + fireEvent.scroll(pane); + state.scrollHeight = 1200; + rerender(); + expect(state.writes).toHaveLength(0); + }); + + it('pins only the settling change, not the ones after it', () => { + const { rerender, state } = setup(); + rerender(); + expect(state.writes).toEqual([900]); + state.scrollHeight = 1200; + rerender(); + expect(state.writes).toEqual([900]); + }); + it('detaches when the user scrolls up beyond the follow threshold', () => { const { rerender, pane, state } = setup(); state.scrollTop = 100; diff --git a/client/src/components/Chat/Messages/Content/Parts/index.ts b/client/src/components/Chat/Messages/Content/Parts/index.ts index 5c6b6ae9a4..2deb26ef2e 100644 --- a/client/src/components/Chat/Messages/Content/Parts/index.ts +++ b/client/src/components/Chat/Messages/Content/Parts/index.ts @@ -13,6 +13,7 @@ export { default as SkillCall } from './SkillCall'; export { default as ReadFileCall } from './ReadFileCall'; export { default as FileAuthoringCall } from './FileAuthoringCall'; export { default as BashCall } from './BashCall'; +export { default as PtcToolTrace } from './PtcToolTrace'; export { default as SubagentCall } from './SubagentCall'; export { default as SteerPart } from './SteerPart'; export { default as AuthorHeader } from './AuthorHeader'; diff --git a/client/src/components/Chat/Messages/Content/Parts/useFollowScroll.ts b/client/src/components/Chat/Messages/Content/Parts/useFollowScroll.ts index 2cf0d6ca4b..37c3c820d4 100644 --- a/client/src/components/Chat/Messages/Content/Parts/useFollowScroll.ts +++ b/client/src/components/Chat/Messages/Content/Parts/useFollowScroll.ts @@ -32,8 +32,16 @@ const FOLLOW_THRESHOLD_PX = 40; * would open at the tail. Gated, it opens reading from the top, while an * expand mid-stream pins immediately. The pin's own scroll event measures * distance zero and keeps the attached state, so programmatic scrolls - * need no special-casing. Finished panes (`active === false`) are never - * scrolled. + * need no special-casing. + * + * A finished pane (`active === false`) is never scrolled, with one + * exception: the very change that ends the stream. That change often adds + * content โ€” a failing PTC call settles by appending its error line in the + * same commit that clears the last running row โ€” and gating on `active` + * alone would drop exactly the pin that content needs, leaving the final + * line below the fold. So the falling edge of `active` still pins, but + * only when the content changed with it; ending the stream on its own + * leaves the pane where the reader left it. */ export default function useFollowScroll( content: string | readonly ReactNode[], @@ -42,6 +50,8 @@ export default function useFollowScroll( ): { ref: RefObject; onScroll: UIEventHandler } { const ref = useRef(null); const followRef = useRef(true); + const wasActiveRef = useRef(active); + const previousContentRef = useRef(content); const onScroll = useCallback>((event) => { const el = event.currentTarget; @@ -49,7 +59,14 @@ export default function useFollowScroll( }, []); useLayoutEffect(() => { - if (!active || !expanded || !followRef.current) { + const settling = wasActiveRef.current && !Object.is(previousContentRef.current, content); + previousContentRef.current = content; + wasActiveRef.current = active; + + if (!expanded || !followRef.current) { + return; + } + if (!active && !settling) { return; } const el = ref.current; diff --git a/client/src/hooks/SSE/__tests__/useResumableSSE.spec.ts b/client/src/hooks/SSE/__tests__/useResumableSSE.spec.ts index 49fd41fe8e..514f6b6456 100644 --- a/client/src/hooks/SSE/__tests__/useResumableSSE.spec.ts +++ b/client/src/hooks/SSE/__tests__/useResumableSSE.spec.ts @@ -236,6 +236,7 @@ jest.mock('~/hooks/SSE/useEventHandlers', () => { contentHandler: jest.fn(), resetContentHandler: jest.fn(), syncStepMessage: jest.fn(), + prunePtcTraces: jest.fn(), clearStepMaps: mockClearStepMaps, flushPendingDeltas: jest.fn(), messageHandler: jest.fn(), diff --git a/client/src/hooks/SSE/__tests__/useStepHandler.spec.ts b/client/src/hooks/SSE/__tests__/useStepHandler.spec.ts index 81f0986dc9..310f9bd0d8 100644 --- a/client/src/hooks/SSE/__tests__/useStepHandler.spec.ts +++ b/client/src/hooks/SSE/__tests__/useStepHandler.spec.ts @@ -15,8 +15,11 @@ import type { TConversation, TMessage, SubagentUpdateEvent, + PtcToolCallEvent, Agents, } from 'librechat-data-provider'; +import type { PtcTrace, PtcTraceEntry } from '~/store/ptc'; +import { ptcTraceByToolCallId, ptcTraceKey, PTC_TRACE_MAX_ENTRIES } from '~/store/ptc'; import { subagentProgressByToolCallId, subagentProgressKey } from '~/store/subagents'; import { resolveAskUserQuestionPart } from '~/utils/approval'; import useStepHandler from '~/hooks/SSE/useStepHandler'; @@ -3913,4 +3916,315 @@ describe('useStepHandler', () => { expect(response?.content).toHaveLength(3); }); }); + + describe('PTC inner tool-call trace', () => { + const renderWithTraceReader = () => { + const hookResult = renderHook( + () => { + const stepHandler = useStepHandler(createHookParams()); + const read = useRecoilCallback( + ({ snapshot }) => + (messageId: string, toolCallId: string): PtcTrace => + snapshot + .getLoadable(ptcTraceByToolCallId(ptcTraceKey(messageId, toolCallId))) + .valueOrThrow(), + [], + ); + return { ...stepHandler, read }; + }, + { wrapper: RecoilRoot }, + ); + return { + result: hookResult.result, + getEntries: (toolCallId: string, messageId = 'response-msg-1'): PtcTraceEntry[] => + ( + hookResult.result.current as unknown as { read: (m: string, id: string) => PtcTrace } + ).read(messageId, toolCallId).entries, + getTrace: (toolCallId: string, messageId = 'response-msg-1'): PtcTrace => + ( + hookResult.result.current as unknown as { read: (m: string, id: string) => PtcTrace } + ).read(messageId, toolCallId), + }; + }; + + const ptcEvent = (overrides: Partial): PtcToolCallEvent => ({ + tool_call_id: 'call_ptc', + call_id: 'call_ptc:0', + name: 'read_file', + status: 'running', + runId: 'response-msg-1', + ...overrides, + }); + + it('appends a row for each inner call the program starts', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:0', args: 'path=a.ts' }), + }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:1', name: 'write_file', args: 'path=b.ts' }), + }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toEqual([ + { callId: 'call_ptc:0', name: 'read_file', status: 'running', args: 'path=a.ts' }, + { callId: 'call_ptc:1', name: 'write_file', status: 'running', args: 'path=b.ts' }, + ]); + }); + + it('settles a row in place instead of appending a duplicate', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ args: 'path=a.ts' }) }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ status: 'success', durationMs: 1200 }), + }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toEqual([ + { + callId: 'call_ptc:0', + name: 'read_file', + status: 'success', + args: 'path=a.ts', + durationMs: 1200, + }, + ]); + }); + + it('keeps each PTC call trace under its own tool call id', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({}) }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ tool_call_id: 'call_other', call_id: 'call_other:0' }), + }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toHaveLength(1); + expect(getEntries('call_other')).toHaveLength(1); + }); + + it('drops an envelope with no call identity rather than seeding a blank row', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ call_id: '' }) }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toEqual([]); + }); + + it('keeps a reused tool_call_id isolated across parent messages', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ name: 'read_file' }) }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ name: 'write_file', runId: 'response-msg-2' }), + }, + submission, + ); + }); + + expect(getEntries('call_ptc').map((e) => e.name)).toEqual(['read_file']); + expect(getEntries('call_ptc', 'response-msg-2').map((e) => e.name)).toEqual(['write_file']); + }); + + it('drops an envelope that cannot be scoped to a parent message', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ runId: undefined }) }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toEqual([]); + }); + + it('marks rows still running across a resume gap and keeps settled ones', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + /** Real order: each row opens as `running` before it settles. */ + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ call_id: 'call_ptc:0' }) }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:0', status: 'success', durationMs: 900 }), + }, + submission, + ); + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:1', name: 'write_file' }), + }, + submission, + ); + }); + expect(getEntries('call_ptc')).toHaveLength(2); + + act(() => { + (result.current as unknown as { prunePtcTraces: () => void }).prunePtcTraces(); + }); + + expect(getEntries('call_ptc').map((e) => [e.callId, e.status])).toEqual([ + ['call_ptc:0', 'success'], + ['call_ptc:1', 'interrupted'], + ]); + }); + + it('lets a call still running across the gap settle onto its marked row', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:0', name: 'write_file' }), + }, + submission, + ); + }); + + act(() => { + (result.current as unknown as { prunePtcTraces: () => void }).prunePtcTraces(); + }); + expect(getEntries('call_ptc').map((e) => e.status)).toEqual(['interrupted']); + + /** The call outlived the disconnect, so its terminal event arrives on + * the restored live stream and must land on the row it opened. */ + act(() => { + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ + call_id: 'call_ptc:0', + name: 'write_file', + status: 'success', + durationMs: 1400, + }), + }, + submission, + ); + }); + + expect(getEntries('call_ptc')).toEqual([ + expect.objectContaining({ callId: 'call_ptc:0', status: 'success', durationMs: 1400 }), + ]); + }); + + it('caps the retained rows and reports how many it dropped', () => { + const { result, getTrace } = renderWithTraceReader(); + const submission = createSubmission(); + const total = PTC_TRACE_MAX_ENTRIES + 25; + + act(() => { + for (let i = 0; i < total; i++) { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ call_id: `call_ptc:${i}` }) }, + submission, + ); + } + }); + + const trace = getTrace('call_ptc'); + expect(trace.entries).toHaveLength(PTC_TRACE_MAX_ENTRIES); + expect(trace.dropped).toBe(25); + /** The tail is retained, so the newest call is still visible. */ + expect(trace.entries.at(-1)?.callId).toBe(`call_ptc:${total - 1}`); + }); + + it('ignores a settle whose row the cap already evicted', () => { + const { result, getTrace } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + for (let i = 0; i < PTC_TRACE_MAX_ENTRIES + 5; i++) { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({ call_id: `call_ptc:${i}` }) }, + submission, + ); + } + result.current.stepHandler( + { + event: StepEvents.ON_PTC_TOOL_CALL, + data: ptcEvent({ call_id: 'call_ptc:0', status: 'success', durationMs: 10 }), + }, + submission, + ); + }); + + const trace = getTrace('call_ptc'); + expect(trace.entries).toHaveLength(PTC_TRACE_MAX_ENTRIES); + expect(trace.entries.some((e) => e.callId === 'call_ptc:0')).toBe(false); + }); + + it('releases the trace atoms on reset', () => { + const { result, getEntries } = renderWithTraceReader(); + const submission = createSubmission(); + + act(() => { + result.current.stepHandler( + { event: StepEvents.ON_PTC_TOOL_CALL, data: ptcEvent({}) }, + submission, + ); + }); + expect(getEntries('call_ptc')).toHaveLength(1); + + act(() => { + (result.current as unknown as { resetPtcAtoms: () => void }).resetPtcAtoms(); + }); + + expect(getEntries('call_ptc')).toEqual([]); + }); + }); }); diff --git a/client/src/hooks/SSE/useEventHandlers.ts b/client/src/hooks/SSE/useEventHandlers.ts index 570eafbcb5..d1b7d4f247 100644 --- a/client/src/hooks/SSE/useEventHandlers.ts +++ b/client/src/hooks/SSE/useEventHandlers.ts @@ -375,6 +375,8 @@ export default function useEventHandlers({ stepHandler, clearStepMaps, resetSubagentAtoms, + resetPtcAtoms, + prunePtcTraces, syncStepMessage, cancelPendingDeltaFlush, flushPendingDeltas, @@ -420,8 +422,12 @@ export default function useEventHandlers({ shouldResetSubagentAtomsOnConversationChange(previous, paramId, preserveNewConversationId) ) { resetSubagentAtoms(); + /** PTC traces are live-only for the same reason and share the boundary: + * keep them through a run so a finished program stays auditable, drop + * them when the conversation changes. */ + resetPtcAtoms(); } - }, [paramId, resetSubagentAtoms]); + }, [paramId, resetSubagentAtoms, resetPtcAtoms]); /** Final cleanup on component unmount. `useStepHandler` keeps the * set of known atom keys in a ref; when the hook unmounts (user @@ -432,8 +438,9 @@ export default function useEventHandlers({ useEffect( () => () => { resetSubagentAtoms(); + resetPtcAtoms(); }, - [resetSubagentAtoms], + [resetSubagentAtoms, resetPtcAtoms], ); const messageHandler = useCallback( @@ -1205,6 +1212,7 @@ export default function useEventHandlers({ createdHandler, titleHandler, syncStepMessage, + prunePtcTraces, cancelPendingDeltaFlush, flushPendingDeltas, attachmentHandler, diff --git a/client/src/hooks/SSE/useResumableSSE.ts b/client/src/hooks/SSE/useResumableSSE.ts index 613a86e48c..64aa5bda93 100644 --- a/client/src/hooks/SSE/useResumableSSE.ts +++ b/client/src/hooks/SSE/useResumableSSE.ts @@ -1137,6 +1137,7 @@ export default function useResumableSSE( createdHandler, titleHandler, syncStepMessage, + prunePtcTraces, attachmentHandler, resetContentHandler, flushPendingDeltas, @@ -2126,6 +2127,14 @@ export default function useResumableSSE( titleHandler(data.resumeState.titleEvent); } + /** PTC inner calls are not content parts, so the resume snapshot + * can't rebuild them and a settling event lost in this gap is + * never replayed. Mark rows still `running` as interrupted rather + * than leave a spinner that never resolves โ€” and rather than drop + * them, since a call still executing across the reconnect settles + * normally on the restored stream and updates its row in place. */ + prunePtcTraces(); + if (data.resumeState?.replayEvents?.length > 0) { logger.log( 'ResumableSSE', @@ -3474,6 +3483,7 @@ export default function useResumableSSE( contentHandler, resetContentHandler, syncStepMessage, + prunePtcTraces, clearStepMaps, messageHandler, errorHandler, diff --git a/client/src/hooks/SSE/useStepHandler.ts b/client/src/hooks/SSE/useStepHandler.ts index 890217eacb..934fad74a0 100644 --- a/client/src/hooks/SSE/useStepHandler.ts +++ b/client/src/hooks/SSE/useStepHandler.ts @@ -19,6 +19,7 @@ import type { TMessageContentParts, SubagentUpdateEvent, SandboxStartingEvent, + PtcToolCallEvent, } from 'librechat-data-provider'; import type { SetterOrUpdater } from 'recoil'; import type { AnnounceOptions } from '~/common'; @@ -29,9 +30,12 @@ import { registerSubagentProgressKey, subagentParentStreamOpenByToolCallId, subagentProgressByToolCallId, - subagentProgressKey, takeRegisteredSubagentProgressKeys, sandboxStartingByToolCallId, + ptcTraceByToolCallId, + PTC_TRACE_MAX_ENTRIES, + subagentProgressKey, + ptcTraceKey, } from '~/store'; import { isAskUserQuestionPart, isAnsweredAskUserQuestionPart } from '~/utils/approval'; import { MESSAGE_UPDATE_INTERVAL } from '~/common'; @@ -63,7 +67,8 @@ type TStepEvent = | { event: StepEvents.ON_SUMMARIZE_DELTA; data: Agents.SummarizeDeltaEvent } | { event: StepEvents.ON_SUMMARIZE_COMPLETE; data: Agents.SummarizeCompleteEvent } | { event: StepEvents.ON_SUBAGENT_UPDATE; data: SubagentUpdateEvent } - | { event: StepEvents.ON_SANDBOX_STARTING; data: SandboxStartingEvent }; + | { event: StepEvents.ON_SANDBOX_STARTING; data: SandboxStartingEvent } + | { event: StepEvents.ON_PTC_TOOL_CALL; data: PtcToolCallEvent }; type MessageDeltaUpdate = { type: ContentTypes.TEXT; @@ -346,6 +351,108 @@ export default function useStepHandler({ [], ); + /** PTC tool call ids with a live trace, so the atoms can be released. */ + const knownPtcAtomKeys = useRef(new Set()); + + /** + * Folds one `on_ptc_tool_call` envelope into its program's trace: the + * `running` event appends a row, the settling event updates that row in + * place by `call_id`. Order follows the sandbox's dispatch order, which is + * what the code reads like โ€” a round trip can settle out of order. + */ + const applyPtcToolCall = useRecoilCallback( + ({ set }) => + (event: PtcToolCallEvent, parentMessageId: string): void => { + const { tool_call_id: toolCallId, call_id: callId, name, status } = event; + /** No parent message means no occurrence to scope this to; a raw + * `tool_call_id` would leak the rows into whichever card reused it. */ + if (!toolCallId || !callId || !parentMessageId) { + return; + } + const atomKey = ptcTraceKey(parentMessageId, toolCallId); + knownPtcAtomKeys.current.add(atomKey); + set(ptcTraceByToolCallId(atomKey), (previous) => { + const index = previous.entries.findIndex((entry) => entry.callId === callId); + const entry = { + callId, + name, + status, + ...(event.args ? { args: event.args } : {}), + ...(event.error ? { error: event.error } : {}), + ...(event.durationMs != null ? { durationMs: event.durationMs } : {}), + }; + + if (index !== -1) { + const next = [...previous.entries]; + next[index] = { ...previous.entries[index], ...entry }; + return { entries: next, dropped: previous.dropped }; + } + + /** A settle whose row is gone โ€” evicted by the cap, or pruned across + * a resume gap โ€” must not reappear at the tail out of order. */ + if (status !== 'running') { + return previous; + } + + const appended = [...previous.entries, entry]; + const overflow = appended.length - PTC_TRACE_MAX_ENTRIES; + if (overflow <= 0) { + return { entries: appended, dropped: previous.dropped }; + } + return { + entries: appended.slice(overflow), + dropped: previous.dropped + overflow, + }; + }); + }, + [], + ); + + const resetPtcAtoms = useRecoilCallback( + ({ reset }) => + (): void => { + for (const atomKey of knownPtcAtomKeys.current) { + reset(ptcTraceByToolCallId(atomKey)); + } + knownPtcAtomKeys.current.clear(); + }, + [], + ); + + /** + * Settles rows still marked `running` after a stream gap as `interrupted`. + * Inner calls carry no durable state โ€” they are not content parts, so the + * resume snapshot cannot rebuild them and a settling event lost in the gap + * is never replayed โ€” which means such a row would otherwise spin forever. + * + * Marked, not removed: a call can also still be executing across the + * reconnect, and its settling event then arrives normally on the restored + * live stream. That event updates this row in place, so the call reports its + * real outcome. Deleting the row would strand it โ€” a settle whose row is + * gone is dropped rather than re-appended out of order โ€” and the call would + * vanish from the trace despite having run. Settled rows are untouched. + */ + const prunePtcTraces = useRecoilCallback( + ({ set }) => + (): void => { + for (const atomKey of knownPtcAtomKeys.current) { + set(ptcTraceByToolCallId(atomKey), (previous) => + previous.entries.some((entry) => entry.status === 'running') + ? { + ...previous, + entries: previous.entries.map((entry) => + entry.status === 'running' + ? { ...entry, status: 'interrupted' as const } + : entry, + ), + } + : previous, + ); + } + }, + [], + ); + /** * Calculate content index for a run step. * @@ -1258,6 +1365,14 @@ export default function useStepHandler({ ); } else if (stepEvent.event === StepEvents.ON_SANDBOX_STARTING) { setSandboxStarting(stepEvent.data.tool_call_id); + } else if (stepEvent.event === StepEvents.ON_PTC_TOOL_CALL) { + /** `runId` is the response message id (the run configurable's + * `run_id`), the same correlation the subagent path uses. */ + let responseMessageId = stepEvent.data.runId ?? ''; + if (responseMessageId === Constants.USE_PRELIM_RESPONSE_MESSAGE_ID) { + responseMessageId = submission?.initialResponse?.messageId ?? ''; + } + applyPtcToolCall(stepEvent.data, responseMessageId); } else if (stepEvent.event === StepEvents.ON_SUBAGENT_UPDATE) { let responseMessageId = stepEvent.data.runId; if (responseMessageId === Constants.USE_PRELIM_RESPONSE_MESSAGE_ID) { @@ -1369,6 +1484,7 @@ export default function useStepHandler({ applySubagentUpdate, setSandboxStarting, clearSandboxStarting, + applyPtcToolCall, onSkillAuthoringComplete, ], ); @@ -1459,6 +1575,8 @@ export default function useStepHandler({ stepHandler, clearStepMaps, resetSubagentAtoms, + resetPtcAtoms, + prunePtcTraces, syncStepMessage, cancelPendingDeltaFlush, flushPendingDeltas, diff --git a/client/src/locales/en/translation.json b/client/src/locales/en/translation.json index 3632505a76..4840932437 100644 --- a/client/src/locales/en/translation.json +++ b/client/src/locales/en/translation.json @@ -1742,6 +1742,12 @@ "com_ui_provider": "Provider", "com_ui_provider_api_keys_description": "Manage API keys for endpoints configured to use a user-provided key.", "com_ui_provider_api_keys_not_set": "No key set", + "com_ui_ptc_trace_done": "done", + "com_ui_ptc_trace_earlier": "+{{count}} earlier calls", + "com_ui_ptc_trace_failed": "failed", + "com_ui_ptc_trace_interrupted": "interrupted", + "com_ui_ptc_trace_running": "running", + "com_ui_ptc_trace_title": "Tool calls", "com_ui_quality": "Quality", "com_ui_question_failed": "Question wasn't shown", "com_ui_question_failed_description": "The agent couldn't show this question and may retry automatically.", diff --git a/client/src/store/index.ts b/client/src/store/index.ts index 391d0ee481..4d0eb715e0 100644 --- a/client/src/store/index.ts +++ b/client/src/store/index.ts @@ -17,6 +17,7 @@ export * from './mcp'; export * from './favorites'; export * from './subagents'; export * from './sandbox'; +export * from './ptc'; export * from './usage'; export * from './steer'; diff --git a/client/src/store/ptc.ts b/client/src/store/ptc.ts new file mode 100644 index 0000000000..32332f1860 --- /dev/null +++ b/client/src/store/ptc.ts @@ -0,0 +1,64 @@ +import { atomFamily } from 'recoil'; +import type { PtcToolCallStatus } from 'librechat-data-provider'; + +/** + * A row's outcome. Widens the wire status with `interrupted`, which the + * backend never sends: it is what the client concludes locally about a call + * that was still running when a stream gap swallowed its settling event. + */ +export type PtcTraceStatus = PtcToolCallStatus | 'interrupted'; + +/** One tool call a programmatic (PTC) program made from inside the sandbox. */ +export interface PtcTraceEntry { + /** Stable id from the backend; the settle event updates this row in place. */ + callId: string; + /** Inner tool id, e.g. `search_code_mcp_github`. */ + name: string; + status: PtcTraceStatus; + /** `key=value` preview of the call's input. */ + args?: string; + /** Truncated failure message on a failed call. */ + error?: string; + durationMs?: number; +} + +/** + * Stable identity for one PTC invocation in its parent message. Providers + * reuse `tool_call_id` across turns and agents, so the raw id is not + * sufficient identity for live progress โ€” the same rule `subagentProgressKey` + * exists for. Without the message scope, a later program's rows would land in + * an earlier card that happened to share `call_0`. + */ +export const ptcTraceKey = (parentMessageId: string, toolCallId: string) => + `${parentMessageId}\u0000${toolCallId}`; + +/** + * Rolling cap on retained rows. A program looping over a large collection can + * make thousands of inner calls; without a bound, every event would copy an + * ever-growing array and the card would render a row per call. + */ +export const PTC_TRACE_MAX_ENTRIES = 100; + +/** One PTC program's trace: the retained tail plus what the cap discarded. */ +export interface PtcTrace { + entries: PtcTraceEntry[]; + /** Rows evicted by {@link PTC_TRACE_MAX_ENTRIES}, so the cap is never silent. */ + dropped: number; +} + +/** Shared empty value โ€” one reference, so untouched atoms compare equal. */ +export const EMPTY_PTC_TRACE: PtcTrace = { entries: [], dropped: 0 }; + +/** + * Live trace of the inner tool calls made by one PTC run step, in the order + * the sandbox started them (`on_ptc_tool_call` SSE events). Keyed by + * {@link ptcTraceKey} โ€” one concrete invocation in one message. + * + * Session-scoped like the subagent ticker: inner calls open no run step, so + * nothing persists them on the message. Cleared on conversation switch rather + * than at run boundaries, so a finished program's trace stays readable. + */ +export const ptcTraceByToolCallId = atomFamily({ + key: 'ptcTraceByToolCallId', + default: EMPTY_PTC_TRACE, +}); diff --git a/packages/api/src/agents/handlers.spec.ts b/packages/api/src/agents/handlers.spec.ts index 12283aa1e1..86d6c50821 100644 --- a/packages/api/src/agents/handlers.spec.ts +++ b/packages/api/src/agents/handlers.spec.ts @@ -10,6 +10,7 @@ import type { ToolExecuteResult, ToolCallRequest, } from '@librechat/agents'; +import type { PtcToolCallEvent } from 'librechat-data-provider'; import type { CodeExecutionContext } from './execution'; import { createToolExecuteHandler, ToolExecuteOptions } from './handlers'; import { markSandboxReady } from './prewarm'; @@ -976,6 +977,85 @@ describe('createToolExecuteHandler', () => { expect(capturedConfigs[0].disallowedToolDefs).toEqual([]); expect(capturedConfigs[0].toolMap).toEqual(new Map()); }); + + it('instruments the PTC tool map so inner calls report progress', async () => { + const capturedConfigs: Record[] = []; + const ptcTool = createMockTool(Constants.PROGRAMMATIC_TOOL_CALLING, capturedConfigs); + /** `allowed_callers` must admit code execution, or the caller-capability + * filter drops the tool before the trace ever sees it. */ + const toolRegistry = new Map([ + ['custom_tool', { name: 'custom_tool', allowed_callers: ['code_execution'] }], + ]); + const ptcToolMap = new Map([['custom_tool', createMockTool('custom_tool', [])]]); + const loadTools: ToolExecuteOptions['loadTools'] = jest.fn(async () => ({ + loadedTools: [ptcTool] as never[], + configurable: { toolRegistry, ptcToolMap }, + })); + const events: PtcToolCallEvent[] = []; + const handler = createToolExecuteHandler({ + loadTools, + emitPtcProgress: (event) => events.push(event), + }); + + await invokeHandler(handler, [ + { + id: 'call_ptc', + name: Constants.PROGRAMMATIC_TOOL_CALLING, + args: { code: 'custom_tool "{}"' }, + }, + ]); + + const injectedMap = capturedConfigs[0].toolMap as Map< + string, + { name: string; invoke: (input: unknown, config?: unknown) => Promise } + >; + expect(injectedMap).not.toBe(ptcToolMap); + expect(injectedMap.get('custom_tool')?.name).toBe('custom_tool'); + + await injectedMap + .get('custom_tool') + ?.invoke({ path: 'a.ts' }, { metadata: { [Constants.PROGRAMMATIC_TOOL_CALLING]: true } }); + + expect(events.map((event) => event.status)).toEqual(['running', 'success']); + expect(events[0]).toMatchObject({ + tool_call_id: 'call_ptc', + name: 'custom_tool', + args: 'path=a.ts', + }); + }); + + it('instruments only the tools the caller-capability filter admits', async () => { + const capturedConfigs: Record[] = []; + const ptcTool = createMockTool(Constants.PROGRAMMATIC_TOOL_CALLING, capturedConfigs); + const toolRegistry = new Map([ + ['code_tool', { name: 'code_tool', allowed_callers: ['code_execution'] }], + ['direct_tool', { name: 'direct_tool', allowed_callers: ['direct'] }], + ]); + const ptcToolMap = new Map([ + ['code_tool', createMockTool('code_tool', [])], + ['direct_tool', createMockTool('direct_tool', [])], + ]); + const loadTools: ToolExecuteOptions['loadTools'] = jest.fn(async () => ({ + loadedTools: [ptcTool] as never[], + configurable: { toolRegistry, ptcToolMap }, + })); + const handler = createToolExecuteHandler({ + loadTools, + emitPtcProgress: () => {}, + }); + + await invokeHandler(handler, [ + { + id: 'call_ptc', + name: Constants.PROGRAMMATIC_TOOL_CALLING, + args: { code: 'code_tool "{}"' }, + }, + ]); + + /** Tracing must not widen what the sandbox can reach. */ + const injectedMap = capturedConfigs[0].toolMap as Map; + expect([...injectedMap.keys()]).toEqual(['code_tool']); + }); }); describe('host file authoring collisions', () => { diff --git a/packages/api/src/agents/handlers.ts b/packages/api/src/agents/handlers.ts index ce4280694f..4d5cf8428c 100644 --- a/packages/api/src/agents/handlers.ts +++ b/packages/api/src/agents/handlers.ts @@ -15,8 +15,8 @@ import type { CallerCapabilityProjectionSnapshot, } from '@librechat/agents'; import type { StructuredToolInterface } from '@librechat/agents/langchain/tools'; +import type { CodeEnvRef, PtcToolCallEvent } from 'librechat-data-provider'; import type { ValidationIssue } from '@librechat/data-schemas'; -import type { CodeEnvRef } from 'librechat-data-provider'; import type { SkillFileRecord, PrimeSkillFilesResult } from './skillFiles'; import type { CodeExecutionContext } from './execution'; import type { TextContentFragment } from '~/protection'; @@ -72,6 +72,7 @@ import { buildSkillPrimeMessage, SKILL_FILE_PREFIX } from './skills'; import { parseFrontmatter } from '../skills/import'; import { cleanCodeToolOutput } from './cleanup'; import { primeSkillFiles } from './skillFiles'; +import { instrumentPtcToolMap } from './ptc'; import { markSandboxReady } from './prewarm'; export interface ToolEndCallbackData { @@ -136,6 +137,13 @@ export interface ToolExecuteOptions { }) => Promise<{ attachments?: unknown[] } | null>; /** Emits an `attachment` SSE event on the current request's live stream. */ emitAttachment?: (attachment: unknown) => void; + /** + * Emits an `on_ptc_tool_call` SSE event for one inner tool invocation made + * by a programmatic tool-calling program. Absent on transports that don't + * carry the LibreChat step stream (Open Responses), which simply skips the + * instrumentation. + */ + emitPtcProgress?: (event: PtcToolCallEvent) => void; /** * Loads a skill by name with ACL constraint (returns full body for injection). * @@ -742,6 +750,35 @@ function filteredToolArgumentsResult( } } +/** + * Inner tool names the `name` PII policy would block. `filteredToolArgumentsResult` + * inspects `tc.name` for direct calls, but inner calls bypass it entirely โ€” and + * the trace event carries the name unconditionally, so without this the trace + * becomes the disclosure path the policy exists to close. The eligible map holds + * a handful of names, each inspected once per PTC call. + */ +function collectFilteredPtcToolNames( + names: Iterable, + req: ServerRequest | undefined, +): ReadonlySet | undefined { + const filters = req?.config?.filters; + if (filters == null || !hasActivePiiFields(filters.toolArguments?.pii, ['name'])) { + return undefined; + } + const blocked = new Set(); + for (const name of names) { + try { + if (inspectContent(extractToolArgumentContent({ name }), { filters }) != null) { + blocked.add(name); + } + } catch { + /* An un-inspectable name is treated as blocked: fail closed. */ + blocked.add(name); + } + } + return blocked.size > 0 ? blocked : undefined; +} + function filteredToolOutputResult( tc: ToolCallRequest, req: ServerRequest | undefined, @@ -4150,8 +4187,14 @@ function buildToolCallConfig( } export function createToolExecuteHandler(options: ToolExecuteOptions): EventHandler { - const { loadTools, toolEndCallback, persistBackgroundCodeResult, emitAttachment, subagentTasks } = - options; + const { + loadTools, + toolEndCallback, + persistBackgroundCodeResult, + emitAttachment, + emitPtcProgress, + subagentTasks, + } = options; return { handle: async (_event: string, data: ToolExecuteBatchRequest) => { @@ -5007,9 +5050,42 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand toolCallConfig.toolDefs = toolDefs; toolCallConfig.disallowedToolDefs = disallowedToolDefs; const eligibleNames = new Set(toolDefs.map((toolDef) => toolDef.name)); - toolCallConfig.toolMap = new Map( + /* Instrument the ELIGIBLE map, never the raw one: the + * caller-capability restriction decides what the sandbox + * may reach, and tracing must not widen it. */ + const eligiblePtcToolMap = new Map( [...(ptcToolMap ?? toolMap)].filter(([name]) => eligibleNames.has(name)), ); + /* Inner calls produce no run step and no card of their + * own, so the only record of what the program did is + * this trace. `invoke` is the single seam every inner + * call passes through. + * + * They also never reach `filteredToolArgumentsResult` โ€” + * the sandbox bridge invokes them directly โ€” so when the + * deployment filters tool arguments for PII, the trace + * must not put their values on the wire. */ + const ptcReq = mergedConfigurable?.req as ServerRequest | undefined; + const ptcArgumentPii = ptcReq?.config?.filters?.toolArguments?.pii; + toolCallConfig.toolMap = emitPtcProgress + ? instrumentPtcToolMap({ + toolMap: eligiblePtcToolMap, + toolCallId: tc.id, + runId: (metadata as Record)?.run_id as + | string + | undefined, + includePreviews: !hasActivePiiFields(ptcArgumentPii, [ + 'name', + 'arguments', + 'output', + ]), + traceExclusions: collectFilteredPtcToolNames( + eligiblePtcToolMap.keys(), + ptcReq, + ), + emit: emitPtcProgress, + }) + : eligiblePtcToolMap; } } diff --git a/packages/api/src/agents/index.ts b/packages/api/src/agents/index.ts index 4e4b04eb94..9444d41bc2 100644 --- a/packages/api/src/agents/index.ts +++ b/packages/api/src/agents/index.ts @@ -25,6 +25,7 @@ export * from './orphans'; export * from './migration'; export * from './parameters'; export * from './prewarm'; +export * from './ptc'; export * from './openai'; export * from './transactions'; export * from './traversal'; diff --git a/packages/api/src/agents/ptc.spec.ts b/packages/api/src/agents/ptc.spec.ts new file mode 100644 index 0000000000..3a3f04179a --- /dev/null +++ b/packages/api/src/agents/ptc.spec.ts @@ -0,0 +1,247 @@ +import type { StructuredToolInterface } from '@librechat/agents/langchain/tools'; +import type { PtcToolCallEvent } from 'librechat-data-provider'; +import { instrumentPtcToolMap, summarizePtcArgs } from './ptc'; + +/** + * Minimal stand-in for a loaded tool: `executeTools` in `@librechat/agents` + * only resolves a tool by name, reads `schema`/`mcp`, and calls `invoke` โ€” so + * the wrapper must leave all of that intact. + */ +function createTool( + name: string, + invoke: (input: unknown, config?: unknown) => Promise, + extra: Record = {}, +): StructuredToolInterface { + return { name, invoke, ...extra } as unknown as StructuredToolInterface; +} + +describe('summarizePtcArgs', () => { + it('renders an object input as a key=value line', () => { + expect(summarizePtcArgs({ query: 'librechat', limit: 5 })).toBe('query=librechat, limit=5'); + }); + + it('drops empty and nullish values', () => { + expect(summarizePtcArgs({ path: 'a.ts', cursor: null, filter: '' })).toBe('path=a.ts'); + }); + + it('collapses whitespace so a multi-line value stays one line', () => { + expect(summarizePtcArgs({ code: 'a\n b' })).toBe('code=a b'); + }); + + it('clips a long value without dropping the keys after it', () => { + const summary = summarizePtcArgs({ body: 'x'.repeat(200), path: 'a.ts' }); + expect(summary).toContain('โ€ฆ'); + expect(summary).toContain('path=a.ts'); + }); + + it('bounds the whole preview', () => { + const summary = summarizePtcArgs( + Object.fromEntries(Array.from({ length: 20 }, (_, i) => [`key${i}`, `value${i}`])), + ); + expect(summary.length).toBeLessThanOrEqual(97); + }); + + it('stops once the preview budget is spent instead of visiting every key', () => { + const seen: string[] = []; + const probe: Record = {}; + for (let i = 0; i < 40; i++) { + Object.defineProperty(probe, `key${i}`, { + enumerable: true, + get() { + seen.push(`key${i}`); + return `value${i}`; + }, + }); + } + + summarizePtcArgs(probe); + + expect(seen.length).toBeLessThan(40); + }); + + it('does not rewrite the whole of an oversized value to build a short preview', () => { + const huge = 'a b '.repeat(500_000); + const started = Date.now(); + const summary = summarizePtcArgs({ content: huge, path: 'a.ts' }); + + expect(summary).toContain('path=a.ts'); + expect(summary.length).toBeLessThanOrEqual(97); + expect(Date.now() - started).toBeLessThan(150); + }); + + it('falls back to the raw string for a non-object input', () => { + expect(summarizePtcArgs('ls -la')).toBe('ls -la'); + }); + + it('returns an empty preview for an absent input', () => { + expect(summarizePtcArgs(undefined)).toBe(''); + expect(summarizePtcArgs({})).toBe(''); + }); +}); + +describe('instrumentPtcToolMap', () => { + const collect = () => { + const events: PtcToolCallEvent[] = []; + return { events, emit: (event: PtcToolCallEvent) => events.push(event) }; + }; + + it('emits a running event and a success event around an inner call', async () => { + const { events, emit } = collect(); + const toolMap = new Map([['read_file', createTool('read_file', async () => 'file contents')]]); + + const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit }); + const result = await instrumented.get('read_file')?.invoke({ path: 'a.ts' }); + + expect(result).toBe('file contents'); + expect(events).toHaveLength(2); + expect(events[0]).toMatchObject({ + tool_call_id: 'call_1', + name: 'read_file', + status: 'running', + args: 'path=a.ts', + }); + expect(events[1]).toMatchObject({ + tool_call_id: 'call_1', + call_id: events[0].call_id, + status: 'success', + }); + expect(events[1].durationMs).toBeGreaterThanOrEqual(0); + }); + + it('reports a failed inner call and rethrows so the sandbox still sees the error', async () => { + const { events, emit } = collect(); + const toolMap = new Map([ + [ + 'write_file', + createTool('write_file', async () => { + throw new Error('Permission denied'); + }), + ], + ]); + + const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit }); + + await expect(instrumented.get('write_file')?.invoke({ path: '/etc/x' })).rejects.toThrow( + 'Permission denied', + ); + expect(events[1]).toMatchObject({ status: 'error', error: 'Permission denied' }); + }); + + it('gives each inner call its own id so concurrent calls do not collide', async () => { + const { events, emit } = collect(); + const toolMap = new Map([['search', createTool('search', async () => 'ok')]]); + + const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit }); + const search = instrumented.get('search'); + await Promise.all([search?.invoke({ q: 'a' }), search?.invoke({ q: 'b' })]); + + const startIds = events.filter((e) => e.status === 'running').map((e) => e.call_id); + expect(new Set(startIds).size).toBe(2); + }); + + it('passes the invoke config through untouched', async () => { + const { emit } = collect(); + const seen: unknown[] = []; + const toolMap = new Map([ + [ + 'read_file', + createTool('read_file', async (_input, config) => { + seen.push(config); + return 'ok'; + }), + ], + ]); + + const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit }); + const config = { metadata: { run_tools_with_code: true } }; + await instrumented.get('read_file')?.invoke({ path: 'a.ts' }, config); + + expect(seen[0]).toBe(config); + }); + + it('leaves every other property readable on the wrapped tool', () => { + const { emit } = collect(); + const toolMap = new Map([ + [ + 'search_code_mcp_github', + createTool('search_code_mcp_github', async () => 'ok', { + mcp: true, + schema: { type: 'object' }, + }), + ], + ]); + + const instrumented = instrumentPtcToolMap({ toolMap, toolCallId: 'call_1', emit }); + const tool = instrumented.get('search_code_mcp_github') as StructuredToolInterface & { + mcp?: boolean; + }; + + expect(tool.name).toBe('search_code_mcp_github'); + expect(tool.mcp).toBe(true); + expect(tool.schema).toEqual({ type: 'object' }); + }); + + it('omits argument and failure previews when tool-argument filtering is on', async () => { + const { events, emit } = collect(); + const toolMap = new Map([ + [ + 'write_file', + createTool('write_file', async () => { + throw new Error('rejected value 555-01-0000'); + }), + ], + ]); + + const instrumented = instrumentPtcToolMap({ + toolMap, + toolCallId: 'call_1', + includePreviews: false, + emit, + }); + + await expect(instrumented.get('write_file')?.invoke({ ssn: '555-01-0000' })).rejects.toThrow(); + + /** Name, status and duration still report; nothing derived from the + * arguments or the failure text reaches the stream. */ + expect(events[0].args).toBeUndefined(); + expect(events[1].error).toBeUndefined(); + expect(events.map((e) => e.status)).toEqual(['running', 'error']); + expect(events[1].durationMs).toBeGreaterThanOrEqual(0); + expect(JSON.stringify(events)).not.toContain('555-01-0000'); + }); + + it('emits nothing at all for a tool whose name the policy filters', async () => { + const { events, emit } = collect(); + const toolMap = new Map([ + ['ok_tool', createTool('ok_tool', async () => 'ok')], + ['blocked_name_tool', createTool('blocked_name_tool', async () => 'ok')], + ]); + + const instrumented = instrumentPtcToolMap({ + toolMap, + toolCallId: 'call_1', + traceExclusions: new Set(['blocked_name_tool']), + emit, + }); + + /** Excluded tools still execute โ€” only their telemetry is suppressed. */ + await expect(instrumented.get('blocked_name_tool')?.invoke({ a: 1 })).resolves.toBe('ok'); + await instrumented.get('ok_tool')?.invoke({ a: 1 }); + + expect(events.map((e) => e.name)).toEqual(['ok_tool', 'ok_tool']); + expect(JSON.stringify(events)).not.toContain('blocked_name_tool'); + }); + + it('runs the inner call even when the emitter throws', async () => { + const toolMap = new Map([['read_file', createTool('read_file', async () => 'ok')]]); + const instrumented = instrumentPtcToolMap({ + toolMap, + toolCallId: 'call_1', + emit: () => { + throw new Error('stream closed'); + }, + }); + + await expect(instrumented.get('read_file')?.invoke({ path: 'a.ts' })).resolves.toBe('ok'); + }); +}); diff --git a/packages/api/src/agents/ptc.ts b/packages/api/src/agents/ptc.ts new file mode 100644 index 0000000000..aaa47d274f --- /dev/null +++ b/packages/api/src/agents/ptc.ts @@ -0,0 +1,200 @@ +import type { StructuredToolInterface } from '@librechat/agents/langchain/tools'; +import type { PtcToolCallEvent } from 'librechat-data-provider'; + +/** Whole-preview budget for one inner call's arguments. */ +const ARGS_PREVIEW_MAX_CHARS = 96; +/** Per-value budget, so one long string can't crowd out the other keys. */ +const ARGS_VALUE_MAX_CHARS = 40; +const ERROR_PREVIEW_MAX_CHARS = 160; +/** + * Collapsing whitespace can only shorten a string, so a window this many times + * the visible budget is always long enough to fill it. Slicing to the window + * before rewriting matters: this runs synchronously ahead of every inner + * `invoke`, and without it a multi-megabyte argument would be collapsed in + * full to produce a forty-character preview. + */ +const CLIP_OVERSCAN = 4; + +/** Bounded collapse-and-clip: never rewrites more of `input` than the budget + * can possibly need, and marks any truncation it performed. */ +const clip = (input: string, max: number): string => { + const window = input.length > max * CLIP_OVERSCAN ? input.slice(0, max * CLIP_OVERSCAN) : input; + const collapsed = window.replace(/\s+/g, ' ').trim(); + if (collapsed.length <= max && window.length === input.length) { + return collapsed; + } + return `${collapsed.slice(0, max)}โ€ฆ`; +}; + +/** + * Collapses an inner call's input into a single `key=value, key=value` line + * for the CLI-style trace. Values are clipped individually and iteration stops + * as soon as the joined preview can no longer grow, so a call with a large + * body or many keys costs the same as a small one. + */ +export function summarizePtcArgs(input: unknown): string { + if (input == null) { + return ''; + } + if (typeof input === 'string') { + return clip(input, ARGS_PREVIEW_MAX_CHARS); + } + if (typeof input !== 'object' || Array.isArray(input)) { + return clip(String(input), ARGS_PREVIEW_MAX_CHARS); + } + + const record = input as Record; + const entries: string[] = []; + let budget = ARGS_PREVIEW_MAX_CHARS; + /* Keys, not entries: `Object.entries` would materialize every value before + * the loop starts, so the budget check below could never skip the work it + * exists to skip. */ + for (const key of Object.keys(record)) { + if (budget <= 0) { + break; + } + const value = record[key]; + if (value == null || value === '') { + continue; + } + /* Strings are the values that get large (file bodies, request payloads), + * and `clip` bounds them without touching the tail. Everything else is + * small enough that serializing it first is cheaper than inspecting it. */ + const rendered = typeof value === 'string' ? value : safeStringify(value); + if (rendered === '') { + continue; + } + const entry = `${key}=${clip(rendered, ARGS_VALUE_MAX_CHARS)}`; + entries.push(entry); + budget -= entry.length + 2; + } + + return clip(entries.join(', '), ARGS_PREVIEW_MAX_CHARS); +} + +function safeStringify(value: unknown): string { + try { + return JSON.stringify(value) ?? ''; + } catch { + return ''; + } +} + +export interface InstrumentPtcToolMapParams { + /** The tool map the PTC runner resolves inner calls against. */ + toolMap: Map; + /** The PTC run step's tool call id โ€” the card the trace renders under. */ + toolCallId: string; + runId?: string; + /** + * Whether argument and failure previews may ride the stream. False when the + * deployment filters tool arguments for PII: inner calls never pass through + * `filteredToolArgumentsResult` (the sandbox bridge invokes them directly), + * so a preview would put values on the wire that the configured policy + * exists to keep off it โ€” and a failure message routinely quotes the very + * argument that caused it. The trace still reports name, status and duration. + */ + includePreviews?: boolean; + /** + * Inner tools whose *name* trips the deployment's PII policy. The event + * carries the tool name unconditionally, so a name the `name` filter would + * have blocked on a direct call cannot be allowed to ride the trace instead. + * These tools still execute โ€” they are simply left unwrapped, so no event + * about them is ever emitted. + */ + traceExclusions?: ReadonlySet; + emit: (event: PtcToolCallEvent) => void; +} + +/** + * Wraps every tool the PTC sandbox can reach so each inner invocation reports + * its lifecycle on the live stream. The runner (`executeTools` in + * `@librechat/agents`) resolves a tool by name and calls `invoke` on it, so a + * `Proxy` intercepting only `invoke` is enough โ€” `name`, `schema`, `mcp` and + * every other property the runner reads pass straight through to the real + * tool, and nothing about execution changes. + */ +export function instrumentPtcToolMap({ + toolMap, + toolCallId, + runId, + includePreviews = true, + traceExclusions, + emit, +}: InstrumentPtcToolMapParams): Map { + let sequence = 0; + /** Emission is telemetry: a dead stream must never fail the program. */ + const safeEmit = (event: PtcToolCallEvent): void => { + try { + emit(event); + } catch { + /* stream closed or transport rejected โ€” the run continues */ + } + }; + + const instrumented = new Map(); + for (const [name, tool] of toolMap) { + if (traceExclusions?.has(name)) { + instrumented.set(name, tool); + continue; + } + instrumented.set( + name, + new Proxy(tool, { + get(target, property) { + if (property !== 'invoke') { + /** `target` as the receiver, not the proxy: LangChain tools read + * private class state through their own getters. */ + return Reflect.get(target, property, target); + } + return async (input: unknown, config?: unknown): Promise => { + const callId = `${toolCallId}:${sequence++}`; + const startedAt = Date.now(); + safeEmit({ + tool_call_id: toolCallId, + call_id: callId, + name, + status: 'running', + ...(includePreviews ? { args: summarizePtcArgs(input) } : {}), + ...(runId != null ? { runId } : {}), + }); + try { + const result = await ( + target.invoke as (input: unknown, config?: unknown) => Promise + ).call(target, input, config); + safeEmit({ + tool_call_id: toolCallId, + call_id: callId, + name, + status: 'success', + durationMs: Date.now() - startedAt, + ...(runId != null ? { runId } : {}), + }); + return result; + } catch (error) { + safeEmit({ + tool_call_id: toolCallId, + call_id: callId, + name, + status: 'error', + ...(includePreviews + ? { + error: clip( + error instanceof Error ? error.message : String(error), + ERROR_PREVIEW_MAX_CHARS, + ), + } + : {}), + durationMs: Date.now() - startedAt, + ...(runId != null ? { runId } : {}), + }); + throw error; + } + }; + }, + }) as StructuredToolInterface, + ); + } + + return instrumented; +} diff --git a/packages/data-provider/src/types/runs.ts b/packages/data-provider/src/types/runs.ts index 69c7b18c8b..e8498bdd74 100644 --- a/packages/data-provider/src/types/runs.ts +++ b/packages/data-provider/src/types/runs.ts @@ -46,6 +46,7 @@ export enum StepEvents { ON_SUMMARIZE_COMPLETE = 'on_summarize_complete', ON_SUBAGENT_UPDATE = 'on_subagent_update', ON_SANDBOX_STARTING = 'on_sandbox_starting', + ON_PTC_TOOL_CALL = 'on_ptc_tool_call', } /** Payload for {@link StepEvents.ON_SANDBOX_STARTING} โ€” the stateful code @@ -55,6 +56,32 @@ export type SandboxStartingEvent = { runId?: string; }; +/** Lifecycle of one tool call made from inside a programmatic (PTC) program. */ +export type PtcToolCallStatus = 'running' | 'success' | 'error'; + +/** + * Payload for {@link StepEvents.ON_PTC_TOOL_CALL} โ€” one tool invocation the + * sandbox made on behalf of a programmatic tool-calling program. Emitted twice + * per inner call (`running`, then `success` / `error`) so the PTC card can + * render a live trace of what the code is doing under the code itself. + */ +export type PtcToolCallEvent = { + /** The PTC run step's tool call id โ€” the card this line belongs under. */ + tool_call_id: string; + /** Stable per-program id; the settle event reuses the start event's value. */ + call_id: string; + /** Inner tool id, e.g. `search_code_mcp_github`. */ + name: string; + status: PtcToolCallStatus; + /** `key=value` preview of the call's input. Start event only. */ + args?: string; + /** Truncated failure message. `error` status only. */ + error?: string; + /** Wall-clock time the inner call took. Settle events only. */ + durationMs?: number; + runId?: string; +}; + /** Token-tracking event names streamed to the client (separate from StepEvents dispatch). */ export enum UsageEvents { ON_CONTEXT_USAGE = 'on_context_usage',