8 commits

Author	SHA1	Message	Date
Danny Avila	9618be6eb3	🌿 fix: Preserve Viewed Branch on Sibling-Tree Churn (#13732) ... * 🌿 fix: Preserve Viewed Branch on Sibling-Tree Churn Regenerating a message could snap the view to an unrelated newest branch. MultiMessage reset siblingIdx to 0 (newest) on any messagesTree.length change, but getRegenerateSubmissionMessages slices the flat message array during a regenerate — the streaming handlers render a tree missing unrelated sibling branches, then finalHandler restores the full set. That 2→1→2 child-count swing snapped unrelated forks to their newest sibling, so regenerating the latest response on an older branch jumped to a previously regenerated branch. Replace the indiscriminate reset with per-fork branch memory: a 'seen' set distinguishes a genuinely new sibling (submission/regeneration/edit here — focus it) from one transiently dropped and restored (preserve the user's branch). Decision extracted as the pure, unit-tested resolveSiblingSelection. - client/src/utils/messages.ts: resolveSiblingSelection + tests - MultiMessage: seen/selectedId refs, structural id-signature effect - e2e: regenerate-latest-on-older-branch keeps the viewed branch (fails on the old reset, passes now) * 🧪 test: Long-Thread Branch Preservation E2E Add the user-reported scenario: in a multi-turn thread, regenerate an earlier response (forking a root branch), switch back to the original, then regenerate a later response on it — the original branch must stay intact. Uses labeled prompts so each turn's unique reply is a reliable settle signal. Verified it fails on the original MultiMessage and passes with the fix. * 🎨 style: Fix import order in MultiMessage (react before recoil) * 🌿 fix: Keep Unrelated Branches in Regenerate Optimistic Render Regenerating a message used a flat `messages.slice(0, targetIndex)` for the optimistic render, which also drops unrelated sibling branches that merely sit later in the flat array. Mid-regenerate the thread briefly collapsed to a short branch (visible flash) and the scroll jumped to the shrunken content and didn't recover — the same flat-array root cause as the branch-reset bug. Remove only the regenerated response and its descendants, keeping unrelated branches. The thread (and scroll) stay put through the regenerate. This array is render-only — the server regenerates from parentMessageId and createPayload doesn't include it — so summing by subtree never affects the request. Verified via a small-viewport scroll trace: old collapses 903->295px / 8->2 renders mid-stream; fixed stays 903px / 8 renders, scroll held at bottom. Unit test covers the keep-unrelated-branches behavior (fails on the old slice). * 🌿 fix: Let an Explicit Branch Selection Survive Streaming ID Churn resolveSiblingSelection focused any unseen sibling id before checking the committed selection. When an in-flight response's id is replaced mid-stream (placeholder → server/run id, e.g. useStepHandler re-keys to runId) after the user switched to a different sibling, that swap looked like a brand-new sibling and stole focus back to the streaming branch. Reorder: the committed selection wins while still present; only focus a fresh sibling when the selection is gone (regenerated away, or its own placeholder id was just replaced — that's how a regen/edit still takes focus, since the slice removes the old response). Added unit tests for both churn directions. * 🌿 fix: Only Focus a New Sibling When the Fork Actually Grew The previous churn fix (selection-wins-first) was too aggressive: a genuinely new sibling ADDED while the prior selection is still present — e.g. a follow-up re-parented as a sibling after a generation-start failure — was no longer focused, so its reply never rendered (broke message-tree generation-start recovery e2e). Gate new-sibling focus on actual growth: resolveSiblingSelection now takes prevCount and only focuses a never-seen id when ids.length > prevCount. A same-count placeholder→server id swap (churn) or a restored already-seen sibling is not growth, so the committed selection still wins there. Covers follow-up/new-branch focus, churn steal-prevention, and self-churn follow. message-tree + chat e2e: 17 passed (incl. the recovered generation-start test). * 🌿 refactor: Drop MultiMessage Branch-Memory in Favor of the Slice Fix The regenerate-slice fix (keep unrelated branches in the optimistic render) is the true root cause: with no spurious tree collapse, the original setSiblingIdx(0)-on-length-change never misfires, so the branch-reset is fixed without per-fork memory. The earlier MultiMessage rewrite (seen/selectedId/ prevCount + resolveSiblingSelection) was a symptom patch added before the root cause was found, and its per-instance memory generated two edge-case findings (placeholder→server id churn; divergence from external siblingIdx writes like resume restore). Revert MultiMessage to the simple upstream version and remove resolveSiblingSelection (+ its tests). The slice fix + the existing branch e2e (chat.spec: switch-back, regenerate-latest, long-thread) cover the behavior; all 17 chat + message-tree branch specs pass with this version. * 🌿 fix: Focus the Regenerated Response When Its Fork Count Is Unchanged When a parent already has multiple sibling responses and the user switches to a non-latest one and regenerates it, the optimistic slice drops the target but keeps the other siblings, so the child count is unchanged. MultiMessage only resets the (reversed) sibling index on a length change, so the stale index kept pointing at the kept sibling and the regenerating response stayed hidden until the server restored the dropped sibling at finalize (count bump → reset). Explicitly focus the newest sibling (reversed index 0 = the appended response) of the regenerated fork in createdHandler. Position-based, fires only on the regenerate action, so it doesn't reintroduce the placeholder→server id churn or external-write fragility that a per-render selection memory had. E2E: new during-stream test (slow+counted reply marker) asserting the regenerating response is visible before finalize; negatively verified (fails without the focus call, passes with it). * 🌿 fix: Eliminate Pre-Created Flash by Focusing at the Optimistic Render The createdHandler focus removed the until-finalize bug, but a brief flash remained between clicking regenerate and the `created` event: useChatFunctions renders the optimistic placeholder first, and that render has the same unchanged-count problem, so the kept sibling showed until createdHandler fired. Extract the focus into a shared useFocusRegeneratedResponse hook and apply it at the optimistic render too (useChatFunctions) and on `created` (useEventHandlers). The placeholder is now focused from the first frame. E2E: gated pre-created test — holds the SSE stream GET (the chat POST returns a stream id; the stream is a separate GET) so `created` cannot arrive, leaving only the optimistic render, then asserts the kept sibling is already gone. This isolates the optimistic focus (createdHandler cannot mask it); negatively verified (fails without the optimistic focus call). * 🧪 test: Extend Store Mock for the Regenerate Focus Hook useChatFunctions.regenerate.spec.tsx mocks ~/store and recoil partially; the new useFocusRegeneratedResponse calls store.messagesSiblingIdxFamily via a recoil `set`, neither of which the mock provided (TypeError on regenerate). Add messagesSiblingIdxFamily to the store mock and `set` to the useRecoilCallback mock. Test-only; production code unchanged.	2026-06-14 09:38:06 -04:00
Danny Avila	db7011d567	📊 feat: Real-Time Context Window & Token Usage Tracking (#13670) ... * 📊 feat: Real-Time Context Window & Token Usage Tracking * 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps * 🩹 fix: Resolve Codex Findings for Context Usage Tracking * 📊 feat: Granular Tool Token Breakdown with Deferred Splits * 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors * 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated) * 🧪 test: Local Real-Provider Multi-Turn E2E Harness * 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate * 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events * 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output - carry the post-snapshot output estimate into the context snapshot at finalize so the gauge keeps the last response after live resets - accumulate per-rate billable units and price the session cost at render, so usage events arriving before the token-config load still count once it resolves - pass user-scoped token-config cache keys through loadConfigModels fetches and drop the controller's unscoped fallback to prevent serving another user's resolved config - tag emitted usage events with a per-run seq so resume dedupe never drops a distinct call with an identical payload - admit the static tokenConfig override in the custom endpoint schema so it survives zod parsing into req.config * 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics - classify cache tokens by provider (shared inputTokensIncludesCache from data-provider, consumed by both the backend billing path and the client) instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache is smaller than uncached input no longer under-bill input - mirror resolveCompletionTokens on the client so Vertex-style hidden thinking tokens are reflected in the Output row and session cost - prefer endpoint pricing over adapter-provider pricing so a custom endpoint can price a known model name without built-in rates shadowing it - carry static cacheRead/cacheWrite overrides through the tokenConfig schema and buildTokenConfigMap * 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness - initializeCustom now uses a static endpoint tokenConfig as the agent's endpointTokenConfig (billing + balance checks), not just the advertised UI config — previously the gauge showed admin rates while the agent billed against built-in tables - invalidate the token-config query alongside models on user-key add/ revoke so context windows and pricing refresh without a reload - include maxContextTokens in ChatForm's stabilized conversation memo so the gauge reflects a changed context-window setting immediately - feed the live output estimate from the legacy content path (direct and assistants streams), setting from cumulative part text rather than accumulating deltas * 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing - fold usage events idempotently by (runId, seq) so resume backfill no longer resets the conversation totals — a mid-stream reconnect keeps the usage of prompts already completed earlier in the session - tap replayed pending message/reasoning/content events so output streamed past the resume snapshot reaches the live estimate, not just the message - resolve cost against the agent's backing endpoint (Agents conversations report endpoint `agents` / provider `openAI`, neither of which keys a custom endpoint's tokenConfig) - getMultiplier/getCacheMultiplier fall back to the standard tables for models absent from a partial endpointTokenConfig, so a partial static override no longer bills non-listed models at defaultRate while the UI shows the correct pattern rate * 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup - live/completed gauge counts the repaired completion (normalized output), so under-reporting providers don't drop the response from used context - translate static tokenConfig cacheWrite/cacheRead onto the write/read keys getCacheMultiplier reads, so cache tokens bill at the configured rate instead of the prompt-rate fallback - clear the token index and usage atoms when leaving a conversation, so visited histories don't accumulate in memory for the tab's lifetime - wait for startupConfig before mounting the gauge, so a deployment with contextUsage disabled never briefly mounts it or fires the token-config query on first load * 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo - extract the token-config resolution (override gathering + cache lookup + buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule) - getConvoKey prefers the user message's real conversationId once the `created` event stamps it, so a new chat's first-response live gauge and totals land under the id TokenUsage subscribes to instead of NEW_CONVO * 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config - DEL the Redis job hash before re-creating it so a reused streamId can't inherit a prior run's contextUsage/tokenUsage and backfill stale usage - tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic streams) into the live estimate, not just the content path - copy a deduped fetch's token config to every sibling endpoint sharing the baseURL/key/headers, so /token-config resolves each by its own name * ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume) createJob is an idempotent join — a second replica calls it for the same streamId to share an in-flight stream's state. DELeting the hash wiped the prior replica's persisted created/usage state, so a joining replica missed the created event (GenerationJobManager cross-replica integration test). Reverts the F1 change from 2bfce0c34b; the stale-usage concern doesn't arise in practice (streamId is unique per generation). * 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage - wrap the ModelEndHandler usage emit in try/catch so a failed telemetry delivery (closed SSE / Redis publish error) can't abort the handler before thought-signature capture, which would break resumed tool calls - tag hidden sequential-agent usage as 'sequential' (non-primary) so the client folds it into session cost/totals but not the live context gauge, instead of letting an undefined usage_type inflate the visible gauge * 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup - useTokenConfigQuery refetches on mount when stale, so a user-key change that invalidates tokenConfig while the gauge is unmounted takes effect on return instead of serving the prior key's resolved config - normalize a Vertex-backed agent's provider (vertexai) to the google token-config key, so Gemini context windows and rates resolve instead of showing unknown context / $0 cost * ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge) Move usage-cost pricing to the single source of truth. The backend prices each model call with the same billing functions (premium tiers via getMultiplier(inputTokenCount), cache rates) and emits the USD cost on on_token_usage when interface.contextCost is enabled; the client sums emitted costs instead of re-deriving from base token-config rates. - computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend so the emitted cost matches what is billed (incl. premium thresholds) - getDefaultHandlers gains a usageCost pricing context; initialize.js wires db.getMultiplier/getCacheMultiplier gated on contextCost (agents path) - client UsageTotals carries a summed costUSD; retire the client-side rate lookups (costFromUnits/calcUsageCost) that drifted from backend pricing and produced the provider-keying / cache-key / Vertex / premium findings - keep normalizeUsageUnits for the displayed token counts; token-config is still used for the context-window meter Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1 above their input thresholds). * 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast - re-anchor the context snapshot from the user message to the response message at finalize. Regenerating a response branches off a shared user message, so anchoring on it made the snapshot read as "active" on both branches — switching to the sibling branch showed the wrong (other branch's) context. The response message is branch-unique, so sibling branches now correctly fall back to their own per-branch totals. - raise the gauge ring's track/fill contrast (muted track, prominent fill) so the used portion reads clearly as a fill-level indicator * 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors - tag hidden sequential-agent usage `usage_type: 'sequential'` on the COLLECTED usage (not just the emit), and treat it as non-primary in recordCollectedUsage (billed, excluded from the reported output total) so hidden intermediate output stops inflating the parent's tokenCount/pruning - emit on_token_usage from the subagent usage sink (tagged `subagent`, with authoritative cost when contextCost is on) so the gauge's session cost/totals include billed subagent usage; it stays out of the live meter - call resetLive on the resumable 404 and max-retry terminal branches so the gauge doesn't keep counting stale in-flight tokens after the stream ends * 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle - raise the popup breakdown's context progressbar contrast (muted surface-tertiary track, prominent text-primary fill) — that's the bar the contrast feedback was about - revert the gauge ring restyle (kept its original border-heavy track / text-secondary fill); the ring wasn't the element in question * 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo - a null-anchor context snapshot was treated as active on every branch, leaking one generation's granular breakdown onto sibling branches. Require a non-null (response-message) anchor on the viewed branch instead, so siblings without a matching snapshot fall back to their own totals. - revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds from shallow copies) so the memo was behaviorally identical, but it was the feature's only change to core branch-navigation selectors — removing it matches upstream and rules it out of branch-navigation debugging. * 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits Custom-endpoint agents resolve an endpointTokenConfig during agent init but it never reached the AgentClient, so spending, emitted cost, and runtime max-token resolution all fell back to default rates for those agents. - Surface options.endpointTokenConfig on the returned InitializedAgent. - Pass it to the AgentClient (this.options.endpointTokenConfig) so the spending path bills at configured rates. - Thread it through usageCost to computeUsageCostUSD so emitted per-event cost matches billing. - getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map for models absent from a partial override (matches buildTokenConfigMap); consolidates the duplicated fallback in pricing.ts. * 🪙 fix: Preserve Granular Breakdown Across Branch Switches The granular context breakdown lives only in the live on_context_usage snapshot — a single per-conversation slot, anchored to the latest response and overwritten by each generation. Switching to a branch generated earlier this session lost its tool/skill/system rows and fell back to coarse totals. Retain each generation's finalized snapshot in a per-conversation map keyed by its branch-unique response id (snapshotsByAnchorFamily). When the live snapshot is off the viewed branch, walk the branch tail for its deepest stored anchor and render that breakdown. Bounded by generation count and cleared on conversation switch; the live/just-generated path is unchanged. * 🪙 fix: Harden Resume Seeding and Subagent Usage Emission - useResumableSSE: skip the trailing-output live seed when the resume carries a context snapshot; the snapshot's messageTokens already counts produced output, so seeding it again inflated usage until the next reset. - AgentClient subagent emitter: await GenerationJobManager.emitChunk like every other caller (it persists before publishing), so a floating promise can't race job cleanup and a Redis/publish failure is caught by the emitter's try/catch instead of surfacing as an unhandled rejection. * 🧪 test: Playwright Coverage for Context Breakdown Granularity Add a test-only data-testid distinguishing the granular snapshot breakdown (context-breakdown) from the coarse message-history estimate (context-estimate), then assert granularity in the mock e2e harness: - renders the granular breakdown from the live on_context_usage snapshot (guards that the snapshot event actually reaches the popover, not just the usage totals). - preserves the granular breakdown after switching branches — regenerate to overwrite the single live snapshot, switch back, and confirm the rows survive via the per-anchor snapshot history map. Branch regenerate/sibling selectors mirror the existing chat.spec branch test. All three usage specs pass against the mock pipeline. * 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush Codex round on the prior commit: - countTrailingOutputChars now counts only output at the very END of the aggregated content (0 when the model paused at a tool call), and the resume path always seeds it. The earlier skip-trailing-tool-parts behavior plus the skip-seed-when-snapshot gate together over- or under-counted in-flight output on resume; one rule fixes both — pre-invoke snapshot budget is never double-counted, and genuine in-flight output is no longer dropped. - useTokenUsage re-indexes from the messages cache on tail change while submitting. The cache subscriber is muted during streaming, so without a context snapshot (non-agent streams) sumBranch missed the created tail and dropped history + prompt until finalize. Bounded — tailId only shifts on created/finalize/branch-switch. - AgentClient tracks subagent usage emit promises and flushes them in chatCompletion's finally. The sink fires the emitter without awaiting, and resume reads the usage emitChunk persists (HSET), so cleanup must not race it or resumed clients miss billed subagent usage.	2026-06-13 19:38:28 -04:00
Teresa Blanco	9628930958	✅ ci: Add mock e2e coverage for agents, prompts, MCP, and chat flows (#13589) ... * ✅ Add mock e2e coverage for agents, prompts, MCP, and chat flows * 🎯 fix: Change enforce modelSpecs to false --------- Co-authored-by: Danny Avila <danny@librechat.ai>	2026-06-10 09:06:52 -04:00
Danny Avila	2a956f143d	🪞 fix: Preserve Model Spec Icons Across Stream Resume and Abort (#13603)	2026-06-08 17:14:21 -04:00
Danny Avila	15108f0f2f	🌲 test: Add E2E Coverage for Message Tree Streaming (#13570) ... * add e2e message tree stream coverage * fix e2e message tree review findings * expand message tree e2e recovery coverage * fix stream-start failure recovery coverage	2026-06-07 09:21:27 -04:00
Danny Avila	21607ba3d7	📎 fix: Preserve Provider Document Uploads (#13550) ... * fix: Preserve provider document uploads * test: Add provider upload e2e coverage	2026-06-06 10:03:32 -04:00
Danny Avila	6357ea10c1	🧭 feat: Scope Model Spec Skills (#13522) ... * feat: scope model spec skills * style: format skill catalog limit * fix: serialize model spec skill resolution * test: satisfy model spec load config typing * fix: apply model spec skills to added conversations * fix: support alwaysApply frontmatter alias * fix: address model spec skills review	2026-06-05 10:22:02 -04:00
Danny Avila	a1bfa3b298	🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model (#13508) ... * 🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model Replace the standalone HTTP mock LLM server with an in-process fake model injected into the real createRun -> Run.create pipeline via run.Graph.overrideTestModel, so the mock suite exercises the agents integration end-to-end without a live provider or a separate server. - Bump @librechat/agents to 3.2.2 for the FakeChatModel/createFakeStreamingLLM exports - Add an env-gated applyTestRunHook seam in packages/api createRun (no /api changes) - Add e2e/setup/fake-model.js to drive default replies + the skill-authoring tool-call flow - Drop the mock-llm webServer from playwright.config.mock.ts and set LIBRECHAT_TEST_RUN_HOOK * 🧹 test: Retire Standalone Mock LLM Server From E2E Recorder Migrate the `--profile=mock` recorder onto the same in-process fake model as the Playwright mock suite, then delete the now-unused HTTP mock server so the fake-LLM logic lives in a single place. - Point record.js mock profile at the fake model via LIBRECHAT_TEST_RUN_HOOK - Remove the mock-llm-server spawn/wait and MOCK_LLM_PORT plumbing from record.js - Delete e2e/setup/mock-llm-server.js (e2e/setup/fake-model.js is now the only source) - Update e2e/README.md to describe the in-process fake LLM * 🏷️ ci: Rename Playwright Mock E2E Check to Playwright E2E Tests	2026-06-04 08:33:28 -04:00