mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-03 22:32:42 +00:00
📊 feat: Surface Message Feedback as Langfuse Scores (#13544)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: surface message feedback (thumbs up/down) as Langfuse scores When Langfuse tracing is enabled, the message feedback endpoint now posts a boolean `user-feedback` score (1/0 + tag/comment) to Langfuse for the assistant message's trace; clearing feedback deletes the score. Fire-and- forget, so the feedback UX never blocks on Langfuse. Linking is lookup-free: the run opts into deterministic Langfuse trace ids (`langfuse.deterministicTraceId`, passed to the agents Run), so the trace id is sha256(messageId)[:32]. The feedback route recomputes the same id and scores by it. - api/server/services/Langfuse.js: POST/DELETE /api/public/scores (env-gated) - api/server/utils/langfuseTrace.js: traceIdForMessage(messageId) - api/server/routes/messages.js: fire feedback score after the Mongo write - packages/api: pass langfuse.deterministicTraceId to the run - bump @librechat/agents to ^3.2.21 (adds LangfuseConfig.deterministicTraceId) Closes #13537 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: match Langfuse trace environment for feedback scores @librechat/agents passes no environment to its Langfuse tracer, so @langfuse/otel falls back to LANGFUSE_TRACING_ENVIRONMENT and otherwise to Langfuse's "default". The score helper instead fell back to NODE_ENV, so a deployment with only NODE_ENV=production filed scores under "production" while the trace stayed on "default" — the score never landed on the trace. Use LANGFUSE_TRACING_ENVIRONMENT only, and omit `environment` when unset so Langfuse defaults both score and trace to "default". Addresses Codex review on #13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't require LANGFUSE_BASE_URL to post feedback scores The agent tracer emits traces with just the public/secret keys (defaulting to Langfuse Cloud, or via the legacy LANGFUSE_BASEURL alias), but the score helper disabled itself unless LANGFUSE_BASE_URL was set — so an otherwise-traced deployment silently posted no scores. Resolve the base URL the same way the tracer does (LANGFUSE_BASE_URL -> LANGFUSE_BASEURL -> Cloud) and gate enablement on the credentials only. Addresses Codex review on #13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: only post feedback scores for agent-endpoint messages The feedback route is shared by all message types, but deterministic Langfuse trace IDs are only enabled for agent runs. Rating a message from a non-agent endpoint (with Langfuse configured) posted a user-feedback score for sha256(messageId) that no trace will ever match, leaving orphan scores. Gate scoring on isAgentsEndpoint(message.endpoint); `updateMessage` now returns `endpoint` so the route can check it. Addresses Codex review on #13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate feedback scoring by !isAssistantsEndpoint, not isAgentsEndpoint The previous gate used isAgentsEndpoint, which only matches the literal `agents` endpoint. But provider endpoints (anthropic, openai, custom, …) run through the agents runtime as ephemeral agents and DO emit deterministic AgentRun traces, so isAgentsEndpoint('anthropic') === false suppressed scoring for the common case. Only the OpenAI/Azure Assistants endpoints use a separate runtime with no agent trace, so gate on !isAssistantsEndpoint instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: sort message method imports * fix: honor Langfuse tracing gates for feedback scores * refactor: move Langfuse feedback logic to api package * fix: support Langfuse host for feedback scores * test: type Langfuse feedback fetch mock * chore: compact Langfuse feedback comment --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Danny Avila <danny@librechat.ai>
This commit is contained in:
parent
cb1d536874
commit
90ebecb254
8 changed files with 241 additions and 2 deletions
|
|
@ -1,8 +1,13 @@
|
|||
const express = require('express');
|
||||
const { v4: uuidv4 } = require('uuid');
|
||||
const { logger } = require('@librechat/data-schemas');
|
||||
const { ContentTypes } = require('librechat-data-provider');
|
||||
const { unescapeLaTeX, countTokens } = require('@librechat/api');
|
||||
const { ContentTypes, isAssistantsEndpoint } = require('librechat-data-provider');
|
||||
const {
|
||||
unescapeLaTeX,
|
||||
countTokens,
|
||||
sendFeedbackScore,
|
||||
traceIdForMessage,
|
||||
} = require('@librechat/api');
|
||||
const { findAllArtifacts, replaceArtifactContent } = require('~/server/services/Artifacts/update');
|
||||
const { requireJwtAuth, validateMessageReq } = require('~/server/middleware');
|
||||
const db = require('~/models');
|
||||
|
|
@ -391,6 +396,14 @@ router.put('/:conversationId/:messageId/feedback', validateMessageReq, async (re
|
|||
{ context: 'updateFeedback' },
|
||||
);
|
||||
|
||||
// Best-effort: Assistants messages do not have deterministic AgentRun traces.
|
||||
if (!isAssistantsEndpoint(updatedMessage.endpoint)) {
|
||||
sendFeedbackScore({
|
||||
traceId: traceIdForMessage(messageId),
|
||||
feedback: updatedMessage.feedback,
|
||||
}).catch((err) => logger.error('[langfuse] feedback score failed:', err));
|
||||
}
|
||||
|
||||
res.json({
|
||||
messageId,
|
||||
conversationId,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue