Commit graph

2 commits

Author SHA1 Message Date
Marco Beretta
e95e696d70
🧾 fix: Count the Tool Results a Tool-Limit Stop Retains (#15893)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 🧾 fix: Count the Tool Results a Tool-Limit Stop Retains

Context snapshots reach the client only through the SDK's pre-invoke
`ON_CONTEXT_USAGE`, so the results of the tools a call requests are never in that
call's snapshot — the next call's snapshot carries them as kept-message context.
A run that stops at the tool-call limit makes no next call, so the tool result it
retains lives in the response and in no snapshot: the gauge reported
`(budget − remaining) + completedOutputTokens` and left the retained result out
of used tokens and out of the tool-call share until the following turn.

The save path now counts those results with the run's own tokenizer and persists
them as `retainedToolTokens`, a second post-snapshot delta alongside
`completedOutputTokens` rather than a number folded into the provider-reconciled
`messageTokens`. `resolveRetainedToolTokens` owns the rule that only a tool-limit
stop retains anything, and the snapshot handler records where its content ended
so the count starts at the right boundary.

Counting had to avoid `Tokenizer.getTokenCount`, whose fallbacks would have put a
guess inside exact accounting: above 4 KiB it returns byte length, several times
the real count on ordinary text, and it estimates from character length while an
encoding loads. `countExactTokens` tokenizes in bounded slices cut on code-point
boundaries and returns nothing at all when the encoding is cold, so an
uncountable result withdraws the figure instead of inflating it.

The client adds the field to used tokens, subtracts it from the runway headroom
and widens the tool-call share, in the live snapshot after finalization and in
the persisted blob after a reload.

* 🧹 style: Wrap the Retained-Counter Assertion as Prettier Requires

* 🧮 fix: Address the Review of the Retained-Tool Count

Three findings from the first round, each a real defect in how the figure was
produced rather than a style point.

The boundary was a content index recorded mid-run, but completion reshapes the
array — skill cards are unshifted onto the front and `hide_sequential_outputs`
replaces it with a filtered one — so a saved index no longer means the same
position. The snapshot now records the tool-call ids it already accounts for, and
the save path counts the results of the calls missing from that set: ids survive
every reshape, and a filtered-away call is correctly left out.

Counting in 4 KiB slices was not exact either: a BPE merge spanning a seam is
charged twice, measured at ~1 token per slice, and the field exists precisely to
be an exact addend. `countExactTokens` now tokenizes the whole input — ~60 ms/MB,
paid once at the end of a stopped turn — and refuses content past 8 MiB rather
than estimating it.

The counter takes its exact-count function instead of reaching for the tokenizer
singleton, so `resolveRetainedToolTokens` owns the default (the run's own
encoding) and a caller or test can supply another. That also removes the mock of
global state from the specs.

`compactionReclaim` now includes the retained result in the total it subtracts the
kept exchange from. `latestExchangeTokens` already counts that result on the
other side, so leaving it out subtracted content the total never carried and
understated the savings — to zero on a large final result.

* 🧯 fix: Bound One Turn's Retained-Result Tokenization

The tokenizer refuses a single result past 8 MiB, but a final call that requested
several tools in parallel would pay that bound once per result. The counter now
holds a budget for the whole turn and withdraws its figure past it, so the save
path cannot be made to tokenize an unbounded pile of output.

* 🎚️ feat: Configure the Retained-Result Tokenization Budget

The exact count the gauge adds costs ~60 ms/MB of retained tool output, and the
ceiling on that work was hard-coded in two places. It is now one lever:
`endpoints.agents.maxRetainedToolCountChars`, defaulting to the 8 MiB that
reproduces today's behavior, shared by the schema and the save path through
`DEFAULT_MAX_RETAINED_TOOL_COUNT_CHARS`. Deployments whose tools legitimately
return more can raise it; slower hardware can lower it, or set `0` to withhold
the figure entirely.

`Tokenizer.countExactTokens` no longer carries a bound of its own — the caller
owns the budget — and `resolveRetainedToolTokens` passes the configured value to
the counter, which spends it across all of a final call's parallel results.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-13 22:20:25 -04:00
Danny Avila
44c253d48a
🪙 fix: Correct Context Usage Gauge After Summarization (#13744)
* 🪙 fix: Persist Context Snapshot + Summary Marker After Summarization

The post-summarization context is correctly compacted by the SDK, but the
breakdown wasn't reliably reaching the client, leaving the gauge on the
whole-history estimate (stuck at 100% forever once a conversation compacts).

Two server changes in buildResponseMetadata:
- Snapshot guard: persist the breakdown when a PRIMARY usage event follows the
  latest snapshot (tracked via contextUsageSink.latestUsageIndex, recorded in
  the on_context_usage handler) instead of a brittle snapshot-vs-primary count.
  A summarization detour adds an extra snapshot whose only following usage is
  tagged 'summarization', which the count guard could miscount and drop.
- Summary marker: whenever a turn compacts (summaryTokens > 0), persist a
  lightweight metadata.summaryUsedTokens (the pre-invoke compacted context size)
  UNCONDITIONALLY — so even when the full snapshot can't be saved (interrupted
  final call) or never reaches the client, the per-message estimate has a signal
  to cap the discarded history.

Tests: client.contextMetadata.spec (guard + marker, incl. marker-survives-drop)
and a real-pipeline summarization integration test.

* 🪙 fix: Cap the Context Estimate at the Summary Marker

When the gauge falls back to the per-message estimate (no usable snapshot on the
branch), sumBranch summed the ENTIRE branch history — after a summarization that
discarded most of it, this over-counts and pins the gauge at 100% in perpetuity.

sumBranch now stops at the deepest summarized response (metadata.summaryUsedTokens)
and records it as summaryBaseline; the walk counts only post-summary messages,
and useTokenUsage adds the baseline. So the estimate reflects the compacted
context (summary + recent turns), not the discarded history. USD/default
behavior unchanged when no marker is present.

Test: sumBranch caps a huge pre-summary history at the compacted baseline.

* 🪙 fix: Address Codex Review on the Summarization Marker

- Branch cost/usage is no longer truncated at the summary marker — sumBranch
  caps only the CONTEXT-window count there and keeps accumulating provider
  usage/cost to the root (cumulative spend isn't discarded by compaction).
- findBranchSnapshotAnchor stops at a summarized response with no snapshot of its
  own, so it can't recover a stale PRE-summary snapshot and show discarded
  history; the summary-baseline estimate is used instead.
- Abort path: buildAbortedResponseMetadata now persists the summaryUsedTokens
  marker (pre-invoke, no completedOutputTokens ambiguity, so safe on abort) so a
  STOPPED summarized turn isn't re-summed on reload.
- Marker baseline fallback now includes summaryTokens (a separate breakdown
  field) so it doesn't under-report the compacted size. DRY'd into a shared
  computeSummaryUsedTokens used by the completion and abort paths.
- Estimate popover surfaces the summary baseline as a row so the displayed rows
  reconcile with the header total.

Tests: sumBranch cost-not-truncated + anchor-stops-at-marker (client);
computeSummaryUsedTokens fallback + abort marker (packages/api).

* 🪙 fix: Attribute Persisted Context Usage to the Snapshot Run

Match the post-snapshot primary usage to the latest snapshot's runId before
persisting metadata.contextUsage. Parallel/direct runs interleave snapshots and
usage (A snapshot → B snapshot → A usage → B no-usage); the prior index-only
guard persisted B's snapshot with A's output. finalCallOutputTokens now filters
completedOutputTokens to the snapshot's run. Untagged events (older lib/resume)
match any run for back-compat.

* 🪙 fix: Harden Summary Marker Against Tool-Loops, Stale Anchors, and Emit Races

Codex round on the summarization marker:

- Avoid double-counting earlier tool-loop outputs in the summary marker: those
  outputs sit in BOTH the latest snapshot's pre-invoke baseline AND the response
  message's tokenCount the client estimate adds on top. computeSummaryUsedTokens
  now subtracts the run's prior primary outputs (priorRunOutputTokens) — the live
  path bounds them by the snapshot's usage index, the abort path by all primaries
  (an interrupted final call emits none). Single-call turns subtract 0.
- Stop treating pre-summary anchors as active: sumBranch no longer sets
  containsAnchor once the context is capped at a summary marker, so a stale
  pre-summary snapshot can't override the summary-baseline estimate.
- Capture latestUsageIndex BEFORE awaiting emitEvent: a yield (resumable SSE /
  Redis) during parallel runs could let this call's own usage advance the index
  past the event that proves the snapshot completed, dropping a valid breakdown.

* 🪙 fix: Subtract Summarization Output from the Summary Marker

recordCollectedUsage folds the summarization call's completion into the response
message's tokenCount, while the generated summary is also in the snapshot baseline
as summaryTokens. The client estimate (summaryBaseline + responseTokenCount) thus
counted the summary twice — inflating the gauge after compaction even on a
single-call turn whenever the full snapshot is unavailable. priorRunOutputTokens
now also counts summarization-tagged output (still excluding subagent/sequential,
which recordCollectedUsage keeps out of the reported total), so the marker
subtracts it. Updated unit + guard tests.

* 🪙 fix: Refine Marker Subtraction for Summarization RunId and Abort Boundary

Two Codex follow-ups on the marker-subtraction logic:

- Subtract summarization output regardless of runId: the summarize detour is its
  own model-end call that may carry a distinct runId, but its output still lands
  in this response's tokenCount AND the snapshot baseline (summaryTokens). It is
  now counted unconditionally (still within the response's own usageEmitSink),
  while primaries keep the parallel-run runId filter.
- Don't subtract primaries on the abort path: the job stores no snapshot/usage
  boundary, so a primary that completed AFTER the latest snapshot is NOT in the
  baseline; subtracting it would cancel real output and under-report. priorRun-
  OutputTokens gains an includePrimary flag (false for abort) — abort subtracts
  only the always-pre-snapshot summarization output.

* 🪙 fix: Run-Scope Summary Subtraction and Stop Subtracting on Abort

Two Codex follow-ups, resolved by reverting the round-4 detour:

- Run-scope the summarization subtraction: the summarize detour inherits the
  graph run id (traceConfig spreads config.metadata.run_id), so its usage shares
  the answer snapshot's runId — it is NOT a distinct run. priorRunOutputTokens now
  filters summarization by runId like primaries, so a parallel sibling run's
  summary (different runId, in the sibling's baseline) is no longer subtracted from
  this branch's marker. Drops the includePrimary flag added last round.
- Stop subtracting on the abort path: abort tokenCount is countTokens(text)
  (abortMiddleware) or absent (agents route) — it does not fold in summarization or
  earlier-call output the way recordCollectedUsage does, so the marker must keep
  the full baseline. buildAbortedResponseMetadata now subtracts nothing.
2026-06-14 18:23:30 -04:00