mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-10-20 17:13:44 +00:00
🪙 fix: Correct Context Usage Gauge After Summarization (#13744)
* 🪙 fix: Persist Context Snapshot + Summary Marker After Summarization The post-summarization context is correctly compacted by the SDK, but the breakdown wasn't reliably reaching the client, leaving the gauge on the whole-history estimate (stuck at 100% forever once a conversation compacts). Two server changes in buildResponseMetadata: - Snapshot guard: persist the breakdown when a PRIMARY usage event follows the latest snapshot (tracked via contextUsageSink.latestUsageIndex, recorded in the on_context_usage handler) instead of a brittle snapshot-vs-primary count. A summarization detour adds an extra snapshot whose only following usage is tagged 'summarization', which the count guard could miscount and drop. - Summary marker: whenever a turn compacts (summaryTokens > 0), persist a lightweight metadata.summaryUsedTokens (the pre-invoke compacted context size) UNCONDITIONALLY — so even when the full snapshot can't be saved (interrupted final call) or never reaches the client, the per-message estimate has a signal to cap the discarded history. Tests: client.contextMetadata.spec (guard + marker, incl. marker-survives-drop) and a real-pipeline summarization integration test. * 🪙 fix: Cap the Context Estimate at the Summary Marker When the gauge falls back to the per-message estimate (no usable snapshot on the branch), sumBranch summed the ENTIRE branch history — after a summarization that discarded most of it, this over-counts and pins the gauge at 100% in perpetuity. sumBranch now stops at the deepest summarized response (metadata.summaryUsedTokens) and records it as summaryBaseline; the walk counts only post-summary messages, and useTokenUsage adds the baseline. So the estimate reflects the compacted context (summary + recent turns), not the discarded history. USD/default behavior unchanged when no marker is present. Test: sumBranch caps a huge pre-summary history at the compacted baseline. * 🪙 fix: Address Codex Review on the Summarization Marker - Branch cost/usage is no longer truncated at the summary marker — sumBranch caps only the CONTEXT-window count there and keeps accumulating provider usage/cost to the root (cumulative spend isn't discarded by compaction). - findBranchSnapshotAnchor stops at a summarized response with no snapshot of its own, so it can't recover a stale PRE-summary snapshot and show discarded history; the summary-baseline estimate is used instead. - Abort path: buildAbortedResponseMetadata now persists the summaryUsedTokens marker (pre-invoke, no completedOutputTokens ambiguity, so safe on abort) so a STOPPED summarized turn isn't re-summed on reload. - Marker baseline fallback now includes summaryTokens (a separate breakdown field) so it doesn't under-report the compacted size. DRY'd into a shared computeSummaryUsedTokens used by the completion and abort paths. - Estimate popover surfaces the summary baseline as a row so the displayed rows reconcile with the header total. Tests: sumBranch cost-not-truncated + anchor-stops-at-marker (client); computeSummaryUsedTokens fallback + abort marker (packages/api). * 🪙 fix: Attribute Persisted Context Usage to the Snapshot Run Match the post-snapshot primary usage to the latest snapshot's runId before persisting metadata.contextUsage. Parallel/direct runs interleave snapshots and usage (A snapshot → B snapshot → A usage → B no-usage); the prior index-only guard persisted B's snapshot with A's output. finalCallOutputTokens now filters completedOutputTokens to the snapshot's run. Untagged events (older lib/resume) match any run for back-compat. * 🪙 fix: Harden Summary Marker Against Tool-Loops, Stale Anchors, and Emit Races Codex round on the summarization marker: - Avoid double-counting earlier tool-loop outputs in the summary marker: those outputs sit in BOTH the latest snapshot's pre-invoke baseline AND the response message's tokenCount the client estimate adds on top. computeSummaryUsedTokens now subtracts the run's prior primary outputs (priorRunOutputTokens) — the live path bounds them by the snapshot's usage index, the abort path by all primaries (an interrupted final call emits none). Single-call turns subtract 0. - Stop treating pre-summary anchors as active: sumBranch no longer sets containsAnchor once the context is capped at a summary marker, so a stale pre-summary snapshot can't override the summary-baseline estimate. - Capture latestUsageIndex BEFORE awaiting emitEvent: a yield (resumable SSE / Redis) during parallel runs could let this call's own usage advance the index past the event that proves the snapshot completed, dropping a valid breakdown. * 🪙 fix: Subtract Summarization Output from the Summary Marker recordCollectedUsage folds the summarization call's completion into the response message's tokenCount, while the generated summary is also in the snapshot baseline as summaryTokens. The client estimate (summaryBaseline + responseTokenCount) thus counted the summary twice — inflating the gauge after compaction even on a single-call turn whenever the full snapshot is unavailable. priorRunOutputTokens now also counts summarization-tagged output (still excluding subagent/sequential, which recordCollectedUsage keeps out of the reported total), so the marker subtracts it. Updated unit + guard tests. * 🪙 fix: Refine Marker Subtraction for Summarization RunId and Abort Boundary Two Codex follow-ups on the marker-subtraction logic: - Subtract summarization output regardless of runId: the summarize detour is its own model-end call that may carry a distinct runId, but its output still lands in this response's tokenCount AND the snapshot baseline (summaryTokens). It is now counted unconditionally (still within the response's own usageEmitSink), while primaries keep the parallel-run runId filter. - Don't subtract primaries on the abort path: the job stores no snapshot/usage boundary, so a primary that completed AFTER the latest snapshot is NOT in the baseline; subtracting it would cancel real output and under-report. priorRun- OutputTokens gains an includePrimary flag (false for abort) — abort subtracts only the always-pre-snapshot summarization output. * 🪙 fix: Run-Scope Summary Subtraction and Stop Subtracting on Abort Two Codex follow-ups, resolved by reverting the round-4 detour: - Run-scope the summarization subtraction: the summarize detour inherits the graph run id (traceConfig spreads config.metadata.run_id), so its usage shares the answer snapshot's runId — it is NOT a distinct run. priorRunOutputTokens now filters summarization by runId like primaries, so a parallel sibling run's summary (different runId, in the sibling's baseline) is no longer subtracted from this branch's marker. Drops the includePrimary flag added last round. - Stop subtracting on the abort path: abort tokenCount is countTokens(text) (abortMiddleware) or absent (agents route) — it does not fold in summarization or earlier-call output the way recordCollectedUsage does, so the marker must keep the full baseline. buildAbortedResponseMetadata now subtracts nothing.
This commit is contained in:
Danny Avila
GitHub
parent
2350ebb24a
commit
44c253d48a
10 changed files with 801 additions and 41 deletions
|
|
@ -10,6 +10,8 @@ import {
|
|||
resolveAgentTokenConfig,
|
||||
buildPersistedContextUsage,
|
||||
buildAbortedResponseMetadata,
|
||||
computeSummaryUsedTokens,
|
||||
priorRunOutputTokens,
|
||||
} from './usage';
|
||||
|
||||
describe('recordCollectedUsage', () => {
|
||||
|
|
@ -1752,6 +1754,157 @@ describe('buildPersistedContextUsage', () => {
|
|||
it('omits completedOutputTokens when there are no primary calls', () => {
|
||||
expect(buildPersistedContextUsage(baseSnapshot, []).completedOutputTokens).toBeUndefined();
|
||||
});
|
||||
|
||||
it('attributes completedOutputTokens to the snapshot run, not a parallel run', () => {
|
||||
/** Parallel/direct runs interleave: this snapshot is run-1, but run-2 emits a
|
||||
* later primary usage. The persisted delta must be run-1's own output (40),
|
||||
* never run-2's trailing output (99). */
|
||||
const events: TTokenUsageEvent[] = [
|
||||
{
|
||||
input_tokens: 100,
|
||||
output_tokens: 40,
|
||||
total_tokens: 140,
|
||||
provider: 'openAI',
|
||||
runId: 'run-1',
|
||||
},
|
||||
{
|
||||
input_tokens: 200,
|
||||
output_tokens: 99,
|
||||
total_tokens: 299,
|
||||
provider: 'openAI',
|
||||
runId: 'run-2',
|
||||
},
|
||||
];
|
||||
const result = buildPersistedContextUsage(baseSnapshot, events);
|
||||
expect(result.completedOutputTokens).toBe(40);
|
||||
});
|
||||
|
||||
it('omits completedOutputTokens when only other runs emitted usage', () => {
|
||||
/** The snapshot run never completed (no matching primary); a sibling run's
|
||||
* output must not be borrowed — fall back to the per-message estimate. */
|
||||
const events: TTokenUsageEvent[] = [
|
||||
{
|
||||
input_tokens: 200,
|
||||
output_tokens: 99,
|
||||
total_tokens: 299,
|
||||
provider: 'openAI',
|
||||
runId: 'run-2',
|
||||
},
|
||||
];
|
||||
expect(buildPersistedContextUsage(baseSnapshot, events).completedOutputTokens).toBeUndefined();
|
||||
});
|
||||
|
||||
it('matches untagged usage events for back-compat (older lib / resume)', () => {
|
||||
/** Events without a runId predate run tagging; they match any snapshot so the
|
||||
* last primary (25) is still recorded. */
|
||||
const events: TTokenUsageEvent[] = [
|
||||
{ input_tokens: 100, output_tokens: 40, total_tokens: 140, provider: 'openAI' },
|
||||
{ input_tokens: 200, output_tokens: 25, total_tokens: 225, provider: 'openAI' },
|
||||
];
|
||||
expect(buildPersistedContextUsage(baseSnapshot, events).completedOutputTokens).toBe(25);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeSummaryUsedTokens', () => {
|
||||
const summarized = (over?: Partial<TContextUsageEvent>): TContextUsageEvent => ({
|
||||
runId: 'run-1',
|
||||
breakdown: {
|
||||
maxContextTokens: 1000,
|
||||
instructionTokens: 50,
|
||||
systemMessageTokens: 50,
|
||||
dynamicInstructionTokens: 0,
|
||||
toolSchemaTokens: 0,
|
||||
summaryTokens: 80,
|
||||
toolCount: 0,
|
||||
messageCount: 1,
|
||||
messageTokens: 20,
|
||||
availableForMessages: 900,
|
||||
},
|
||||
contextBudget: 1000,
|
||||
remainingContextTokens: 700,
|
||||
effectiveInstructionTokens: 50,
|
||||
...over,
|
||||
});
|
||||
|
||||
it('uses contextBudget − remainingContextTokens when available', () => {
|
||||
expect(computeSummaryUsedTokens(summarized())).toBe(300);
|
||||
});
|
||||
|
||||
it('falls back to instructions + summary + messages, including summaryTokens', () => {
|
||||
/** summaryTokens is a separate breakdown field, so the no-remaining fallback
|
||||
* must add it: 50 + 80 + 20 = 150, not 70. */
|
||||
expect(computeSummaryUsedTokens(summarized({ remainingContextTokens: undefined }))).toBe(150);
|
||||
});
|
||||
|
||||
it('returns undefined when the turn did not summarize', () => {
|
||||
expect(
|
||||
computeSummaryUsedTokens(
|
||||
summarized({ breakdown: { ...summarized().breakdown, summaryTokens: 0 } }),
|
||||
),
|
||||
).toBeUndefined();
|
||||
expect(computeSummaryUsedTokens(null)).toBeUndefined();
|
||||
expect(computeSummaryUsedTokens(undefined)).toBeUndefined();
|
||||
});
|
||||
|
||||
it('subtracts the response’s earlier tool-loop outputs from the marker', () => {
|
||||
/** 300 baseUsed − 90 earlier outputs = 210, so the client estimate
|
||||
* (summaryBaseline + full response tokenCount) doesn’t double-count them. */
|
||||
expect(computeSummaryUsedTokens(summarized(), 90)).toBe(210);
|
||||
});
|
||||
|
||||
it('clamps to undefined when the prior outputs exceed the baseline', () => {
|
||||
expect(computeSummaryUsedTokens(summarized(), 5000)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('priorRunOutputTokens', () => {
|
||||
const ev = (over: Partial<TTokenUsageEvent>): TTokenUsageEvent => ({
|
||||
input_tokens: 10,
|
||||
output_tokens: 0,
|
||||
total_tokens: 10,
|
||||
provider: 'openAI',
|
||||
...over,
|
||||
});
|
||||
|
||||
it('sums primary outputs before the index for the matching run', () => {
|
||||
const events = [
|
||||
ev({ output_tokens: 20, runId: 'run-1' }),
|
||||
ev({ output_tokens: 30, runId: 'run-1' }),
|
||||
ev({ output_tokens: 99, runId: 'run-1' }), // at/after the index — excluded
|
||||
];
|
||||
expect(priorRunOutputTokens(events, 2, 'run-1')).toBe(50);
|
||||
});
|
||||
|
||||
it('counts run-matched primary + summarization, skips subagent/sequential and other runs', () => {
|
||||
/** Both the primary and the summarization output are in this run's tokenCount
|
||||
* AND baseline, so both are subtracted; subagent/sequential are excluded from
|
||||
* the reported output total; a parallel run's primary is not this snapshot's. */
|
||||
const events = [
|
||||
ev({ output_tokens: 20, runId: 'run-1' }), // primary, matches
|
||||
ev({ output_tokens: 8, runId: 'run-1', usage_type: 'summarization' }), // counted
|
||||
ev({ output_tokens: 5, runId: 'run-1', usage_type: 'subagent' }), // skipped
|
||||
ev({ output_tokens: 7, runId: 'run-1', usage_type: 'sequential' }), // skipped
|
||||
ev({ output_tokens: 40, runId: 'run-2' }), // other-run primary — skipped
|
||||
];
|
||||
expect(priorRunOutputTokens(events, 5, 'run-1')).toBe(28);
|
||||
});
|
||||
|
||||
it('does not subtract a parallel sibling run’s summarization output', () => {
|
||||
/** The summarize detour inherits the graph run id (traceConfig), so a sibling
|
||||
* run's summary carries a DIFFERENT runId; its summary is in the sibling's
|
||||
* baseline, not this snapshot's, so subtracting it would under-report. */
|
||||
const events = [
|
||||
ev({ output_tokens: 20, runId: 'run-1' }),
|
||||
ev({ output_tokens: 8, runId: 'run-2', usage_type: 'summarization' }), // sibling — skipped
|
||||
];
|
||||
expect(priorRunOutputTokens(events, 2, 'run-1')).toBe(20);
|
||||
});
|
||||
|
||||
it('matches untagged events for back-compat and returns 0 with no prior calls', () => {
|
||||
const events = [ev({ output_tokens: 20 }), ev({ output_tokens: 30 })];
|
||||
expect(priorRunOutputTokens(events, 2, 'run-1')).toBe(50);
|
||||
expect(priorRunOutputTokens(events, 0, 'run-1')).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildAbortedResponseMetadata', () => {
|
||||
|
|
@ -1783,6 +1936,65 @@ describe('buildAbortedResponseMetadata', () => {
|
|||
expect(result).toEqual({ usage: { input: 100, output: 20, cacheWrite: 0, cacheRead: 0 } });
|
||||
expect((result as { contextUsage?: unknown }).contextUsage).toBeUndefined();
|
||||
});
|
||||
|
||||
const abortSnapshot: TContextUsageEvent = {
|
||||
runId: 'run-1',
|
||||
breakdown: {
|
||||
maxContextTokens: 1000,
|
||||
instructionTokens: 50,
|
||||
systemMessageTokens: 50,
|
||||
dynamicInstructionTokens: 0,
|
||||
toolSchemaTokens: 0,
|
||||
summaryTokens: 80,
|
||||
toolCount: 0,
|
||||
messageCount: 1,
|
||||
messageTokens: 20,
|
||||
availableForMessages: 900,
|
||||
},
|
||||
contextBudget: 1000,
|
||||
remainingContextTokens: 700,
|
||||
};
|
||||
|
||||
it('persists the summary marker (but not the full snapshot) for a stopped summarized turn', () => {
|
||||
/** A single-call stopped turn: the interrupted call emitted no usage, so the
|
||||
* marker is the full baseUsed (300) with nothing to subtract. */
|
||||
const result = buildAbortedResponseMetadata({
|
||||
tokenUsage: JSON.stringify([]),
|
||||
contextUsage: JSON.stringify(abortSnapshot),
|
||||
});
|
||||
expect(result?.summaryUsedTokens).toBe(300);
|
||||
/** The full snapshot stays off the abort path (completedOutputTokens ambiguity). */
|
||||
expect((result as { contextUsage?: unknown }).contextUsage).toBeUndefined();
|
||||
});
|
||||
|
||||
it('does not subtract any output from the marker on a stopped turn', () => {
|
||||
/** The abort tokenCount comes from countTokens(text) or is absent — it does
|
||||
* NOT fold in summarization/earlier-call output the way recordCollectedUsage
|
||||
* does. So the marker is the full baseUsed (300); subtracting the summarization
|
||||
* (8) or the primary (20) here would under-report after reload. */
|
||||
const events: TTokenUsageEvent[] = [
|
||||
{
|
||||
input_tokens: 100,
|
||||
output_tokens: 20,
|
||||
total_tokens: 120,
|
||||
provider: 'openAI',
|
||||
runId: 'run-1',
|
||||
},
|
||||
{
|
||||
input_tokens: 60,
|
||||
output_tokens: 8,
|
||||
total_tokens: 68,
|
||||
provider: 'openAI',
|
||||
runId: 'run-1',
|
||||
usage_type: 'summarization',
|
||||
},
|
||||
];
|
||||
const result = buildAbortedResponseMetadata({
|
||||
tokenUsage: JSON.stringify(events),
|
||||
contextUsage: JSON.stringify(abortSnapshot),
|
||||
});
|
||||
expect(result?.summaryUsedTokens).toBe(300);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveAgentTokenConfig', () => {
|
||||
|
|
|
|||
|
|
@ -247,16 +247,23 @@ function normalizeEventUnits(event: TTokenUsageEvent): {
|
|||
};
|
||||
}
|
||||
|
||||
/** Output tokens of the response's final primary model call — the call the
|
||||
* latest pre-invoke snapshot precedes. Persisted as the snapshot's
|
||||
* `completedOutputTokens` so a reloaded multi-call turn adds only this delta
|
||||
* (matching the live finalizer) instead of the full response `tokenCount`,
|
||||
* which the snapshot already counts for earlier steps. */
|
||||
function finalCallOutputTokens(events: ReadonlyArray<TTokenUsageEvent>): number {
|
||||
/** Output tokens of the final primary model call belonging to the snapshot's
|
||||
* run — the call the latest pre-invoke snapshot precedes. Persisted as the
|
||||
* snapshot's `completedOutputTokens` so a reloaded multi-call turn adds only
|
||||
* this delta (matching the live finalizer) instead of the full response
|
||||
* `tokenCount`, which the snapshot already counts for earlier steps. Filtering
|
||||
* by `runId` prevents a parallel run's later usage from being attributed to this
|
||||
* snapshot; untagged events (older lib / resume) match any run for back-compat. */
|
||||
function finalCallOutputTokens(events: ReadonlyArray<TTokenUsageEvent>, runId?: string): number {
|
||||
for (let i = events.length - 1; i >= 0; i--) {
|
||||
if (events[i].usage_type == null) {
|
||||
return normalizeEventUnits(events[i]).output;
|
||||
const event = events[i];
|
||||
if (event.usage_type != null) {
|
||||
continue;
|
||||
}
|
||||
if (runId != null && event.runId != null && event.runId !== runId) {
|
||||
continue;
|
||||
}
|
||||
return normalizeEventUnits(event).output;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -273,7 +280,7 @@ export function buildPersistedContextUsage(
|
|||
usageEvents: ReadonlyArray<TTokenUsageEvent> = [],
|
||||
): TContextUsageEvent {
|
||||
const { breakdown } = snapshot;
|
||||
const completedOutputTokens = finalCallOutputTokens(usageEvents);
|
||||
const completedOutputTokens = finalCallOutputTokens(usageEvents, snapshot.runId);
|
||||
let toolTokenCounts = breakdown.toolTokenCounts;
|
||||
if (toolTokenCounts != null) {
|
||||
const trimmed: Record<string, number> = {};
|
||||
|
|
@ -291,6 +298,83 @@ export function buildPersistedContextUsage(
|
|||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Sum of this response's output tokens already folded into a later snapshot's
|
||||
* pre-invoke baseline that the response message's `tokenCount` ALSO carries — the
|
||||
* overlap `computeSummaryUsedTokens` subtracts from the marker so the live-path
|
||||
* client estimate (`summaryBaseline + responseTokenCount`) doesn't double-count:
|
||||
* - earlier tool-loop PRIMARY calls: a multi-call turn's first output sits in the
|
||||
* kept-message context of the next call's snapshot AND in `tokenCount`.
|
||||
* - the SUMMARIZATION call's generated summary: it sits in the snapshot baseline
|
||||
* as `summaryTokens` AND in `tokenCount` (`recordCollectedUsage` folds
|
||||
* summarization completion into the reported output total; subagent/sequential
|
||||
* are kept out of that total, so they are excluded here too).
|
||||
*
|
||||
* Both are matched by `runId` and bounded by `beforeIndex` to the calls that
|
||||
* preceded the snapshot. The summarize detour inherits the graph run id
|
||||
* (`traceConfig` spreads `config.metadata.run_id`), so it shares the snapshot's
|
||||
* `runId`; a parallel sibling run's summary carries a DIFFERENT `runId` and must
|
||||
* NOT be subtracted (its summary lives in the sibling's baseline, not this one).
|
||||
* Untagged events (older lib / resume) match any run for back-compat.
|
||||
*
|
||||
* Only the live path (which builds `tokenCount` via `recordCollectedUsage`) calls
|
||||
* this; the abort path subtracts nothing — see {@link buildAbortedResponseMetadata}.
|
||||
*/
|
||||
export function priorRunOutputTokens(
|
||||
events: ReadonlyArray<TTokenUsageEvent>,
|
||||
beforeIndex: number,
|
||||
runId?: string,
|
||||
): number {
|
||||
let total = 0;
|
||||
const end = Math.min(beforeIndex, events.length);
|
||||
for (let i = 0; i < end; i++) {
|
||||
const event = events[i];
|
||||
if (event.usage_type != null && event.usage_type !== 'summarization') {
|
||||
continue;
|
||||
}
|
||||
if (runId != null && event.runId != null && event.runId !== runId) {
|
||||
continue;
|
||||
}
|
||||
total += normalizeEventUnits(event).output;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-invoke compacted context size for a summarized turn (instructions +
|
||||
* summary + kept messages), or `undefined` when the turn did not summarize.
|
||||
* Persisted as the lightweight `summaryUsedTokens` marker so the client estimate
|
||||
* fallback caps the discarded pre-summary history instead of re-summing it (the
|
||||
* gauge otherwise reads 100% in perpetuity after a compaction). Pre-invoke, so
|
||||
* it carries none of the `completedOutputTokens` ambiguity that keeps the full
|
||||
* snapshot off some save paths. `summaryTokens` is a SEPARATE breakdown field, so
|
||||
* the non-`remainingContextTokens` fallback adds it explicitly.
|
||||
*
|
||||
* `priorOutputTokens` (this response's earlier tool-loop outputs, see
|
||||
* {@link priorRunOutputTokens}) is subtracted: those tokens are inside the
|
||||
* baseline's kept messages AND in the response message's `tokenCount` the client
|
||||
* adds on top, so leaving them in the marker double-counts them on a tool-loop
|
||||
* summarized turn. Single-call turns pass 0 and are unaffected.
|
||||
*/
|
||||
export function computeSummaryUsedTokens(
|
||||
snapshot: TContextUsageEvent | null | undefined,
|
||||
priorOutputTokens = 0,
|
||||
): number | undefined {
|
||||
const summaryTokens = snapshot?.breakdown?.summaryTokens ?? 0;
|
||||
if (!snapshot || summaryTokens <= 0) {
|
||||
return undefined;
|
||||
}
|
||||
const maxTokens = snapshot.contextBudget ?? snapshot.breakdown.maxContextTokens ?? 0;
|
||||
const baseUsed =
|
||||
snapshot.remainingContextTokens != null
|
||||
? maxTokens - snapshot.remainingContextTokens
|
||||
: (snapshot.effectiveInstructionTokens ?? snapshot.breakdown.instructionTokens ?? 0) +
|
||||
summaryTokens +
|
||||
(snapshot.breakdown.messageTokens ?? 0);
|
||||
const adjusted = baseUsed - Math.max(0, priorOutputTokens);
|
||||
return adjusted > 0 ? Math.round(adjusted) : undefined;
|
||||
}
|
||||
|
||||
function parseUsageEvents(value?: string | null): TTokenUsageEvent[] {
|
||||
if (typeof value !== 'string' || value.length === 0) {
|
||||
return [];
|
||||
|
|
@ -309,21 +393,50 @@ function parseUsageEvents(value?: string | null): TTokenUsageEvent[] {
|
|||
* reload (finding: stopped responses otherwise lose cost). Shared by every abort
|
||||
* save path (agents abort route + legacy abort middleware).
|
||||
*
|
||||
* Deliberately persists ONLY `usage`, not `contextUsage`: unlike the live path,
|
||||
* the abort path can't tell whether the FINAL call (the one the latest snapshot
|
||||
* precedes) emitted usage — the job stores only the latest snapshot, not the
|
||||
* snapshot count. If the final call emitted none, `completedOutputTokens` would
|
||||
* reuse an earlier call's output the snapshot already counts → reload
|
||||
* over-reports. A stopped/incomplete response therefore falls back to the coarse
|
||||
* per-message gauge estimate on reload, which is both safe and apt for an
|
||||
* interrupted turn that never reached a clean pre-invoke breakdown.
|
||||
* Deliberately omits the full `contextUsage`: unlike the live path, the abort
|
||||
* path can't tell whether the FINAL call (the one the latest snapshot precedes)
|
||||
* emitted usage — the job stores only the latest snapshot, not the snapshot
|
||||
* count. If the final call emitted none, `completedOutputTokens` would reuse an
|
||||
* earlier call's output the snapshot already counts → reload over-reports. So a
|
||||
* stopped response falls back to the per-message gauge estimate on reload.
|
||||
*
|
||||
* It DOES persist the `summaryUsedTokens` marker when the stopped turn had
|
||||
* summarized: that marker is pre-invoke (no `completedOutputTokens` ambiguity),
|
||||
* and without it the fallback estimate re-sums the history the compaction
|
||||
* discarded — leaving a stopped summarized turn pinned at 100%. Unlike the live
|
||||
* path, the abort `tokenCount` comes from `countTokens(text)` (abortMiddleware) or
|
||||
* is absent (agents abort route) — it does NOT fold in the summarization or
|
||||
* earlier-call output the way `recordCollectedUsage` does. So the marker subtracts
|
||||
* NOTHING: the full pre-invoke baseline is correct, and the client adds only the
|
||||
* partial answer text on top (no overlap to cancel).
|
||||
*/
|
||||
export function buildAbortedResponseMetadata(
|
||||
job: { tokenUsage?: string | null } | null | undefined,
|
||||
): { usage?: TResponseUsage } | undefined {
|
||||
job: { tokenUsage?: string | null; contextUsage?: string | null } | null | undefined,
|
||||
): { usage?: TResponseUsage; summaryUsedTokens?: number } | undefined {
|
||||
const events = parseUsageEvents(job?.tokenUsage);
|
||||
const usage = aggregateEmittedUsage(events);
|
||||
return usage ? { usage } : undefined;
|
||||
|
||||
let snapshot: TContextUsageEvent | null = null;
|
||||
if (typeof job?.contextUsage === 'string' && job.contextUsage.length > 0) {
|
||||
try {
|
||||
snapshot = JSON.parse(job.contextUsage) as TContextUsageEvent;
|
||||
} catch {
|
||||
snapshot = null;
|
||||
}
|
||||
}
|
||||
/** Subtract nothing: the abort `tokenCount` (countTokens(text) or absent) does
|
||||
* not fold in summarization/earlier-call output, so the full baseline is the
|
||||
* marker and the client's partial-text addition has no overlap to cancel. */
|
||||
const summaryUsedTokens = computeSummaryUsedTokens(snapshot);
|
||||
|
||||
const metadata: { usage?: TResponseUsage; summaryUsedTokens?: number } = {};
|
||||
if (usage) {
|
||||
metadata.usage = usage;
|
||||
}
|
||||
if (summaryUsedTokens != null) {
|
||||
metadata.summaryUsedTokens = summaryUsedTokens;
|
||||
}
|
||||
return Object.keys(metadata).length > 0 ? metadata : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue