mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 04:07:05 +00:00
💾 feat: Persist Context Breakdown & Branch/Total Usage Cost (#13734)
* 💾 feat: Persist Context Breakdown & Branch/Total Usage Cost Persist the granular context breakdown and per-response usage/cost on the response message metadata, and re-derive branch + total usage/cost from a per-message index so the popover survives reloads and is branch-aware live. - Add aggregateEmittedUsage + buildPersistedContextUsage helpers in packages/api; capture the latest visible snapshot and every emitted on_token_usage payload via contextUsageSink/usageEmitSink. - Attach metadata.contextUsage (Part A) and metadata.usage (Part B) on the agents response message in sendCompletion. - Carry per-message usage on the token index; add sumTotalUsage/setEntryUsage and branch-scoped usage on sumBranch. - Repurpose the session accumulator into a single in-flight pending holder; flush it into the index at finalize; hydrate breakdowns on load. - Render branch cost with a conditional all-branches total in the breakdown. * 🧹 chore: Remove orphaned com_ui_session_cost i18n key * 🩹 fix: Address Codex review — normalize usage server-side, fix reload deltas - Persist per-event-normalized display units in metadata.usage (TResponseUsage) so reloaded mixed-provider turns match the live session; client reads them directly instead of re-normalizing with a single stamped provider (P2). - Persist completedOutputTokens (final call output) on metadata.contextUsage so a reloaded multi-call turn adds the post-snapshot delta, not the full tokenCount the snapshot already counts (P2). - buildIndex preserves a prior entry's immutable usage when a rebuilt cache message lacks metadata.usage, so a mid-session rebuild (regenerate) keeps a sibling branch's flushed cost (fixes the e2e regenerate failure). - Track costKnown so turns saved with contextCost off don't render $0.00 when cost display is later enabled (P3). - Use an epsilon for the all-branches cost comparison to avoid a spurious total row from float summation order (P3). - Update unit/integration/e2e tests for the new shapes; regenerate e2e asserts the all-branches total after reload (deterministic via persisted metadata). * 🩹 fix: Address Codex round 2 — pending leak, cost coverage, reload delta - Clear the in-flight pending usage on terminal abort/error (resetLive), so a stopped generation's tokens no longer merge into the next response (P2). - costKnown now means COMPLETE coverage (ANDed): a branch mixing cost-bearing and cost-less turns is flagged incomplete and the cost row is hidden rather than rendering an under-reported total (P2). - Drop the tokenCount fallback for completedOutputTokens on reload: only the persisted post-snapshot delta is used, so a multi-call turn whose provider emitted no usage_metadata no longer double-counts earlier output (P2). - Update tokens.spec for AND coverage semantics + incomplete-cost case. * 🩹 fix: Address Codex round 3 — no-usage snapshots, total coverage, provider-less cache - Skip persisting metadata.contextUsage when the response emitted no primary usage event: without a known post-snapshot output the granular gauge would undercount the reply on reload, so fall back to the coarse per-message estimate instead (P2). - Gate the all-branches cost row on totalUsage.costKnown so an incomplete total (a sibling saved without cost) never renders an under-reported figure (P2). - aggregateEmittedUsage/finalCallOutputTokens now normalize per-event with the client's magnitude fallback (normalizeEventUnits) instead of billing splitUsage, so provider-less cached events match live on reload (P2). - Add backend test for the provider-less cached case. * 🩹 fix: Address Codex round 4 — abort attribution, complete cost coverage - aggregateEmittedUsage persists cost only when EVERY call was priced; a partial pricing failure now omits cost so the client treats coverage as unknown rather than reading an under-reported sum as authoritative (P2). - finalizeUsage flushes pending into the response entry only when events were folded this session (eventCount > 0), so a late/second resumable subscriber carrying persisted metadata.usage keeps it instead of being overwritten with an empty pending record (P2). - On user stop, attribute the in-flight pending usage to the partial response (new attributePending handler) instead of discarding it in resetLive — the stopped reply's billed tokens are kept and still can't leak into the next response; resetLive's discard remains for the error path (P2). * 🐛 fix: Persist branch cost across branch switches via sticky usage history Branch cost vanished on switching to a sibling branch (until a new turn) — the cost analog of the granularity bug. buildIndex rebuilds the token index from the messages cache; a sibling generated this session whose cache message lacks metadata.usage (and is transiently dropped from the cache during regenerate) lost its live-flushed usage, so sumBranch found none and the cost row hid. Fix: a sticky per-response usage map (conversationId → messageId → usage), written by setEntryUsage and never rebuilt from the cache — the usage counterpart of snapshotsByAnchorFamily for the breakdown. buildIndex/upsertEntries restore an entry's usage from it when the message carries none; cleared on convo switch and migrated with the index. Add unit coverage for the drop-then-readd regression and an e2e assertion that branch cost survives a branch switch. * 🐛 fix: Re-index on branch switch so branch cost survives the switch The sticky usage history alone didn't fix the reported branch-switch cost drop: on a branch switch no cache `updated` event fires, so the index subscriber never re-ran, and the post-regenerate rebuild was skipped while `isSubmitting` was still true — leaving the index stale and missing the now-viewed branch's response entirely (sticky can only restore entries present in a rebuild). Re-index from the messages cache on every tail change (created/finalize AND branch switch), not just while submitting. The cache holds the full message set at switch time, so the viewed branch's response is re-added and its usage restored from metadata.usage or the sticky history → sumBranch finds it and the branch cost renders. Verified locally: the branch-switch e2e now passes (the cost section shows both the branch row and the all-branches total). Also fixed that e2e assertion to target a single cost value (strict-mode safe). * 🩹 fix: Handle stopped-stream usage — reset pending + persist abort metadata Codex round (stop/abort edges): - Resumable explicit-stop (intentional SSE close) reset UI state but never cleared pendingUsageFamily, so usage folded before the stop leaked into the next response in the conversation. Discard pending on intentional close (resetLive); a resume re-folds via backfillUsage, so nothing is lost. - The abort save path (abortMiddleware) persisted the stopped response without metadata.usage/contextUsage, so its cost + breakdown vanished on reload. Rebuild both from the job's persisted tokenUsage (emitted payloads incl. cost) and contextUsage snapshot — parity with the normal sendCompletion path; breakdown gated on a primary usage event like buildResponseMetadata. Deferred (per scope decision): mid-stream branch-switch transiently shows the streaming branch's pending on the viewed sibling (cosmetic, until finalize). * 🩹 fix: Persist abort metadata on the real agents route + tighten snapshot gate Codex round (corrects last round's wrong-path fixes): - Stopped AGENTS responses are saved by routes/agents/index.js (/chat/abort), not abortMiddleware — so last round's metadata fix never ran for them. Moved the rollup/snapshot builder into packages/api as buildAbortedResponseMetadata (shared, unit-tested) and applied it in BOTH abort save paths, so a stopped agent reply keeps its cost + breakdown on reload. - Persist the breakdown only when the FINAL visible call emitted usage: track a per-response snapshot count and require primaryUsageCount >= snapshotCount. Previously any earlier primary usage event passed the gate, so a multi-call turn whose final call emitted no usage_metadata used an earlier call's output as completedOutputTokens (already counted by the latest snapshot) → reload over-reported. Now it falls back to the coarse estimate. Resumable stop pending-reset (prior round, 3cde6fe035) already flows through clearAllSubmissions → SSE close → the intentional-close handler's resetLive. Deferred per scope: mid-stream branch-switch pending attribution (tracked). * 🩹 fix: Abort breakdown over-count + resume re-fold after pending discard Codex round (on the re-applied abort/snapshot work): - buildAbortedResponseMetadata now persists ONLY the usage/cost rollup, not the context breakdown. The abort path can't tell whether the final call emitted usage (the job stores only the latest snapshot, not a count), so persisting the breakdown risked reusing an earlier call's output as completedOutputTokens (already in the snapshot) → reload over-count. Stopped/incomplete responses now fall back to the coarse gauge estimate, which is safe and apt. - resetLive now also forgets the conversation's folded usage-event identities (clearUsageFolded). Discarding pending on a terminal/intentional close left the folded keys set, so a later resume's backfillUsage saw the persisted events as duplicates and never rebuilt pending — leaving the response's usage missing until a full reload. Clearing them lets the resume re-fold.
This commit is contained in:
Danny Avila
GitHub
parent
98704f28c1
commit
b03b2a0a29
19 changed files with 1321 additions and 90 deletions
|
|
@ -9,7 +9,11 @@ const {
|
|||
FakeChatModel,
|
||||
createContentAggregator,
|
||||
} = require('@librechat/agents');
|
||||
const { GenerationJobManager } = require('@librechat/api');
|
||||
const {
|
||||
GenerationJobManager,
|
||||
aggregateEmittedUsage,
|
||||
buildPersistedContextUsage,
|
||||
} = require('@librechat/api');
|
||||
const { getDefaultHandlers } = require('~/server/controllers/agents/callbacks');
|
||||
|
||||
jest.mock('nanoid', () => ({
|
||||
|
|
@ -112,7 +116,14 @@ const SECOND_CALL_USAGE = {
|
|||
|
||||
const MAX_CONTEXT_TOKENS = 8000;
|
||||
|
||||
async function runToolLoop({ res, streamId = null, collectedUsage }) {
|
||||
async function runToolLoop({
|
||||
res,
|
||||
streamId = null,
|
||||
collectedUsage,
|
||||
contextUsageSink = null,
|
||||
usageEmitSink = null,
|
||||
usageCost = null,
|
||||
}) {
|
||||
const { contentParts, aggregateContent } = createContentAggregator();
|
||||
const handlers = getDefaultHandlers({
|
||||
res,
|
||||
|
|
@ -120,6 +131,9 @@ async function runToolLoop({ res, streamId = null, collectedUsage }) {
|
|||
toolEndCallback: () => {},
|
||||
collectedUsage,
|
||||
streamId,
|
||||
contextUsageSink,
|
||||
usageEmitSink,
|
||||
usageCost,
|
||||
});
|
||||
|
||||
const run = await Run.create({
|
||||
|
|
@ -230,6 +244,63 @@ describe('usage events through the real agents pipeline', () => {
|
|||
expect(firstContextIndex).toBeLessThan(firstUsageIndex);
|
||||
});
|
||||
|
||||
test('captures the usage rollup + latest context snapshot for message persistence', () => {
|
||||
const res = createMockRes();
|
||||
const contextUsageSink = { latest: null };
|
||||
const usageEmitSink = [];
|
||||
return runToolLoop({ res, collectedUsage: [], contextUsageSink, usageEmitSink }).then(() => {
|
||||
/** Both model calls' emitted payloads are captured for the rollup */
|
||||
expect(usageEmitSink).toHaveLength(2);
|
||||
|
||||
const usage = aggregateEmittedUsage(usageEmitSink);
|
||||
/** Display units: openAI is cache-subset, so input excludes cache
|
||||
* (150−30−50=70); output is repaired completion */
|
||||
expect(usage).toEqual({
|
||||
input:
|
||||
FIRST_CALL_USAGE.input_tokens +
|
||||
(SECOND_CALL_USAGE.input_tokens -
|
||||
SECOND_CALL_USAGE.input_token_details.cache_creation -
|
||||
SECOND_CALL_USAGE.input_token_details.cache_read),
|
||||
output: FIRST_CALL_USAGE.output_tokens + SECOND_CALL_USAGE.output_tokens,
|
||||
cacheWrite: SECOND_CALL_USAGE.input_token_details.cache_creation,
|
||||
cacheRead: SECOND_CALL_USAGE.input_token_details.cache_read,
|
||||
});
|
||||
/** contextCost off → no cost folded into the rollup */
|
||||
expect(usage.cost).toBeUndefined();
|
||||
|
||||
if (hasContextUsageEvent) {
|
||||
expect(contextUsageSink.latest).not.toBeNull();
|
||||
const persisted = buildPersistedContextUsage(contextUsageSink.latest);
|
||||
expect(persisted.breakdown.maxContextTokens).toBe(MAX_CONTEXT_TOKENS);
|
||||
/** Zero-valued tool counts are trimmed from the persisted blob */
|
||||
for (const count of Object.values(persisted.breakdown.toolTokenCounts ?? {})) {
|
||||
expect(count).toBeGreaterThan(0);
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test('folds authoritative per-event cost into the rollup when contextCost is on', async () => {
|
||||
const res = createMockRes();
|
||||
const usageEmitSink = [];
|
||||
/** Stub pricing mirroring getMultiplier/getCacheMultiplier shape */
|
||||
const usageCost = {
|
||||
enabled: true,
|
||||
pricing: {
|
||||
getMultiplier: ({ tokenType }) => (tokenType === 'completion' ? 15 : 3),
|
||||
getCacheMultiplier: ({ cacheType }) => (cacheType === 'write' ? 3.75 : 0.3),
|
||||
},
|
||||
};
|
||||
await runToolLoop({ res, collectedUsage: [], usageEmitSink, usageCost });
|
||||
|
||||
for (const event of usageEmitSink) {
|
||||
expect(typeof event.cost).toBe('number');
|
||||
}
|
||||
const usage = aggregateEmittedUsage(usageEmitSink);
|
||||
expect(usage.cost).toBeGreaterThan(0);
|
||||
expect(usage.cost).toBeCloseTo(usageEmitSink.reduce((sum, e) => sum + e.cost, 0));
|
||||
});
|
||||
|
||||
test('persists usage and context snapshot for resume via GenerationJobManager', async () => {
|
||||
const streamId = `usage-e2e-stream-${Date.now()}`;
|
||||
await GenerationJobManager.createJob(streamId, 'user-1', 'convo-1');
|
||||
|
|
|
|||
|
|
@ -274,6 +274,11 @@ function feedSubagentAggregator(aggregator, event) {
|
|||
* @param {string | null} [options.streamId] - The stream ID for resumable mode, or null for standard mode.
|
||||
* @param {ToolExecuteOptions} [options.toolExecuteOptions] - Options for event-driven tool execution.
|
||||
* @param {UsageCostDeps} [options.usageCost] - Pricing context for authoritative per-event cost.
|
||||
* @param {{ latest: TContextUsageEvent | null, count: number }} [options.contextUsageSink] - Mutable
|
||||
* holder for the latest visible context snapshot + a count of visible snapshots (model calls),
|
||||
* used to persist the breakdown only when the final call emitted usage.
|
||||
* @param {Array<TTokenUsageEvent>} [options.usageEmitSink] - Array collecting each emitted
|
||||
* `on_token_usage` payload (incl. cost) so the response's usage rollup can be persisted.
|
||||
* @returns {Record<string, t.EventHandler>} The default handlers.
|
||||
* @throws {Error} If the request is not found.
|
||||
*/
|
||||
|
|
@ -288,6 +293,8 @@ function getDefaultHandlers({
|
|||
summarizationOptions = null,
|
||||
subagentAggregatorsByToolCallId = null,
|
||||
usageCost = null,
|
||||
contextUsageSink = null,
|
||||
usageEmitSink = null,
|
||||
}) {
|
||||
if (!res || !aggregateContent) {
|
||||
throw new Error(
|
||||
|
|
@ -313,6 +320,11 @@ function getDefaultHandlers({
|
|||
logger.warn('[getDefaultHandlers] Failed to compute usage cost', err);
|
||||
}
|
||||
}
|
||||
/** Collect the same payload the client folds so the response's usage rollup
|
||||
* persisted on `metadata.usage` reproduces the live branch/total + cost. */
|
||||
if (usageEmitSink) {
|
||||
usageEmitSink.push(payload);
|
||||
}
|
||||
return emitEvent(res, streamId, { event: UsageEvents.ON_TOKEN_USAGE, data: payload });
|
||||
};
|
||||
const handlers = {
|
||||
|
|
@ -521,6 +533,15 @@ function getDefaultHandlers({
|
|||
!metadata?.hide_sequential_outputs
|
||||
) {
|
||||
await emitEvent(res, streamId, { event, data });
|
||||
/** Capture the latest visible snapshot (last-wins) + count visible
|
||||
* snapshots (one per model call). The count lets the save path persist
|
||||
* the breakdown only when the FINAL call emitted usage (primary usage
|
||||
* events === snapshots), so completedOutputTokens is a real
|
||||
* post-snapshot delta and reload doesn't over-report. */
|
||||
if (contextUsageSink) {
|
||||
contextUsageSink.latest = data;
|
||||
contextUsageSink.count = (contextUsageSink.count ?? 0) + 1;
|
||||
}
|
||||
}
|
||||
},
|
||||
};
|
||||
|
|
|
|||
|
|
@ -23,6 +23,8 @@ const {
|
|||
recordCollectedUsage,
|
||||
sendEvent,
|
||||
computeUsageCostUSD,
|
||||
aggregateEmittedUsage,
|
||||
buildPersistedContextUsage,
|
||||
createSubagentUsageSink,
|
||||
isDeepSeekReasoningProvider,
|
||||
GenerationJobManager,
|
||||
|
|
@ -108,11 +110,22 @@ class AgentClient extends BaseClient {
|
|||
artifactPromises,
|
||||
maxContextTokens,
|
||||
subagentAggregatorsByToolCallId,
|
||||
contextUsageSink,
|
||||
usageEmitSink,
|
||||
...clientOptions
|
||||
} = options;
|
||||
|
||||
this.agentConfigs = agentConfigs;
|
||||
this.maxContextTokens = maxContextTokens;
|
||||
/** Latest visible context snapshot for this response, captured live by the
|
||||
* ON_CONTEXT_USAGE handler; persisted on `metadata.contextUsage`.
|
||||
* @type {{ latest: import('librechat-data-provider').TContextUsageEvent | null } | undefined} */
|
||||
this.contextUsageSink = contextUsageSink;
|
||||
/** Every emitted `on_token_usage` payload for this response (primary,
|
||||
* summarization, sequential, and subagent); aggregated into the rollup
|
||||
* persisted on `metadata.usage`.
|
||||
* @type {Array<import('librechat-data-provider').TTokenUsageEvent> | undefined} */
|
||||
this.usageEmitSink = usageEmitSink;
|
||||
/** @type {MessageContentComplex[]} */
|
||||
this.contentParts = contentParts;
|
||||
/** @type {Array<UsageMetadata>} */
|
||||
|
|
@ -828,11 +841,49 @@ class AgentClient extends BaseClient {
|
|||
});
|
||||
|
||||
const completion = filterMalformedContentParts(this.contentParts);
|
||||
const metadata = this.buildResponseMetadata();
|
||||
return metadata ? { completion, metadata } : { completion };
|
||||
}
|
||||
|
||||
/**
|
||||
* Assembles the response message `metadata`: Vertex thought signatures plus
|
||||
* the persisted context breakdown (Part A) and the usage/cost rollup (Part B),
|
||||
* which rebuild the gauge breakdown and branch/total cost across reloads.
|
||||
* Returns undefined when nothing was captured.
|
||||
* @returns {{
|
||||
* thoughtSignatures?: Record<string, string>,
|
||||
* contextUsage?: import('librechat-data-provider').TContextUsageEvent,
|
||||
* usage?: import('librechat-data-provider').TResponseUsage,
|
||||
* } | undefined}
|
||||
*/
|
||||
buildResponseMetadata() {
|
||||
/** @type {{
|
||||
* thoughtSignatures?: Record<string, string>,
|
||||
* contextUsage?: import('librechat-data-provider').TContextUsageEvent,
|
||||
* usage?: import('librechat-data-provider').TResponseUsage,
|
||||
* }} */
|
||||
const metadata = {};
|
||||
const signatures = this.collectedThoughtSignatures;
|
||||
if (!signatures || Object.keys(signatures).length === 0) {
|
||||
return { completion };
|
||||
if (signatures && Object.keys(signatures).length > 0) {
|
||||
metadata.thoughtSignatures = signatures;
|
||||
}
|
||||
return { completion, metadata: { thoughtSignatures: signatures } };
|
||||
const usageEvents = this.usageEmitSink ?? [];
|
||||
/** Persist the breakdown only when the FINAL visible call (the one the latest
|
||||
* snapshot precedes) emitted usage — i.e. as many primary usage events as
|
||||
* visible snapshots. If the final call emitted no usage_metadata (provider
|
||||
* gap, or interrupted after an earlier call did emit), `completedOutputTokens`
|
||||
* would be an earlier call's output the latest snapshot already counts, so
|
||||
* reload would over-report; fall back to the coarse per-message estimate. */
|
||||
const primaryUsageCount = usageEvents.filter((event) => event.usage_type == null).length;
|
||||
const snapshotCount = this.contextUsageSink?.count ?? 0;
|
||||
if (this.contextUsageSink?.latest && snapshotCount > 0 && primaryUsageCount >= snapshotCount) {
|
||||
metadata.contextUsage = buildPersistedContextUsage(this.contextUsageSink.latest, usageEvents);
|
||||
}
|
||||
const usage = aggregateEmittedUsage(usageEvents);
|
||||
if (usage) {
|
||||
metadata.usage = usage;
|
||||
}
|
||||
return Object.keys(metadata).length > 0 ? metadata : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -920,6 +971,12 @@ class AgentClient extends BaseClient {
|
|||
)
|
||||
: undefined,
|
||||
};
|
||||
/** Fold into the response's usage rollup (synchronously, regardless of
|
||||
* emit success) so the persisted total matches the live session, which
|
||||
* also folds subagent usage into its cost/totals. */
|
||||
if (this.usageEmitSink) {
|
||||
this.usageEmitSink.push(data);
|
||||
}
|
||||
/** The sink fires this without awaiting, so retain the promise and flush
|
||||
* it in chatCompletion's finally — emitChunk persists (HSET) before
|
||||
* publishing, and job cleanup must not race that persist or resumed
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ const {
|
|||
GenerationJobManager,
|
||||
recordCollectedUsage,
|
||||
sanitizeMessageForTransmit,
|
||||
buildAbortedResponseMetadata,
|
||||
} = require('@librechat/api');
|
||||
const { truncateText, smartTruncateText } = require('~/app/clients/prompts');
|
||||
const clearPendingReq = require('~/cache/clearPendingReq');
|
||||
|
|
@ -110,6 +111,14 @@ async function abortMessage(req, res) {
|
|||
tokenCount: completionTokens,
|
||||
};
|
||||
|
||||
/** Persist the usage/cost rollup + context breakdown for the stopped response
|
||||
* so its branch/total cost and granular rows survive a reload, matching the
|
||||
* normal completion path. */
|
||||
const abortMetadata = buildAbortedResponseMetadata(jobData);
|
||||
if (abortMetadata) {
|
||||
responseMessage.metadata = abortMetadata;
|
||||
}
|
||||
|
||||
// Spend tokens for ALL models from collectedUsage (handles parallel agents/addedConvo)
|
||||
if (collectedUsage && collectedUsage.length > 0) {
|
||||
await spendCollectedUsage({
|
||||
|
|
|
|||
|
|
@ -1,5 +1,10 @@
|
|||
const express = require('express');
|
||||
const { isEnabled, GenerationJobManager, hasPersistableAbortContent } = require('@librechat/api');
|
||||
const {
|
||||
isEnabled,
|
||||
GenerationJobManager,
|
||||
hasPersistableAbortContent,
|
||||
buildAbortedResponseMetadata,
|
||||
} = require('@librechat/api');
|
||||
const { createSseStreamTelemetry } = require('@librechat/api/telemetry');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const {
|
||||
|
|
@ -291,6 +296,15 @@ router.post('/chat/abort', async (req, res) => {
|
|||
user: userId,
|
||||
};
|
||||
|
||||
/** Persist the usage/cost rollup + context breakdown for the stopped
|
||||
* response (from the job's tracked tokenUsage/contextUsage) so its
|
||||
* branch/total cost and granular rows survive a reload — parity with the
|
||||
* normal completion path. */
|
||||
const abortMetadata = buildAbortedResponseMetadata(jobData);
|
||||
if (abortMetadata) {
|
||||
responseMessage.metadata = abortMetadata;
|
||||
}
|
||||
|
||||
try {
|
||||
await saveMessage(
|
||||
{
|
||||
|
|
|
|||
|
|
@ -252,6 +252,14 @@ const initializeClient = async ({ req, res, signal, endpointOption }) => {
|
|||
pricing: { getMultiplier: db.getMultiplier, getCacheMultiplier: db.getCacheMultiplier },
|
||||
};
|
||||
|
||||
/** Latest visible context snapshot + every emitted usage payload for this
|
||||
* response, captured by the handlers and persisted on the response message's
|
||||
* metadata so the breakdown and branch/total cost survive a reload.
|
||||
* @type {{ latest: import('librechat-data-provider').TContextUsageEvent | null, count: number }} */
|
||||
const contextUsageSink = { latest: null, count: 0 };
|
||||
/** @type {Array<import('librechat-data-provider').TTokenUsageEvent>} */
|
||||
const usageEmitSink = [];
|
||||
|
||||
const eventHandlers = getDefaultHandlers({
|
||||
res,
|
||||
toolExecuteOptions,
|
||||
|
|
@ -263,6 +271,8 @@ const initializeClient = async ({ req, res, signal, endpointOption }) => {
|
|||
streamId,
|
||||
subagentAggregatorsByToolCallId,
|
||||
usageCost,
|
||||
contextUsageSink,
|
||||
usageEmitSink,
|
||||
});
|
||||
|
||||
if (!endpointOption.agent) {
|
||||
|
|
@ -920,6 +930,10 @@ const initializeClient = async ({ req, res, signal, endpointOption }) => {
|
|||
/** Resolved endpoint token/pricing config so spending and cost reflect
|
||||
* configured rates for custom-endpoint agents instead of defaults. */
|
||||
endpointTokenConfig: primaryConfig.endpointTokenConfig,
|
||||
/** Capture sinks the handlers fill during the run; `sendCompletion` reads
|
||||
* them to persist the breakdown + usage rollup on the response message. */
|
||||
contextUsageSink,
|
||||
usageEmitSink,
|
||||
});
|
||||
|
||||
if (streamId) {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue