LibreChat/api/server/index.js
Danny Avila f43301f818
🔭 feat: Conversation Trace Viewer (#15869)
* 🔭 feat: Conversation Trace Viewer

Adds a trace view for a user's own conversations: a summary, a pinned
waterfall overview with interval focus, a hierarchical record ledger and
a record inspector, opened from the chat header (or the mobile overflow
menu) and read through a provider-neutral seam with Langfuse as the
first reader.

- interface.traceViewer (enabled, showInputOutput, maxRecords,
  maxContentLength, requestsPerMinute), off by default
- /api/traces/:conversationId availability, records and record routes,
  re-proving conversation ownership and narrowing the Langfuse session
  to traces the user's own sampled responses produced
- Langfuse reader over the v2 Observations API with cursor paging, time
  bounds, typed failures and server-side input/output withholding
- forks no longer copy the source messages' trace sampling record

* 🛂 fix: Address Trace Viewer Review Round 1

- interface.traceViewer.requestTimeoutMs bounds each Langfuse round trip
- availability is one existence query across every sampled response
  instead of the oldest one, run alongside the ownership check
- tree items report their position among visible siblings
- point-in-time Langfuse events read as completed, not running
- reads prefer the project holding the most responses when a tenant
  connection arrived mid-conversation

* 🎯 fix: Leave Focus With a Navigation That Closes the Trace

* 🧭 fix: Keep Trace Reads on One Project and Cancel Abandoned Reads

- page cursors and record details stay on the project that served the
  first page; pages report an opaque sourceId
- Open in Langfuse only appears when the linked project served every
  loaded page (the session link now reports its destinationId)
- client and server abort Langfuse reads when the viewer closes or the
  client disconnects
- reads never wait on the central project lookup, which the request
  timeout does not govern
- the cost total is withheld when any model call with usage is unpriced

* 🕰️ fix: Refresh Trace Pages After a Turn, Honor Clock Settings, Show Refresh Failures

- a settled run invalidates the cached trace pages, so reopening shows
  the new turn instead of pages fresh for another 30 seconds
- trace durations and clock times format in the app language and the
  user's 12/24-hour setting, as message timestamps do
- a failed refresh with every page loaded surfaces an alert and a retry

* 🔐 fix: Only Server-Authored Responses Grant Trace Access

- the message-create route and imports strip langfuseSampled and
  langfuseDestinationIds, and trace reference queries ignore client-authored
  rows (isUserSubmitted) and user turns, so a forged row cannot claim a trace
- the Langfuse reader receives its destination resolver and HTTP client
  from the route instead of defaulting to process-global ones
- availability asks the client to check again while central's project id
  is still resolving, instead of caching a false answer
- listRecords tries the next readable project when legacy responses left
  the first one empty
- a settled run also invalidates cached record details
- running is read from a record's status, not a missing end time

* 🏢 fix: Scope Trace Reads to the Tenant and Reach Every Project Holding a Turn

- ownership and both trace reference queries carry the requester's tenant
  (tenantless is its own scope), so a duplicate user and conversation id in
  another tenant cannot grant or supply trace references
- a read visits, page by page, every project that could hold a turn no
  earlier project provably holds, so legacy turns in another project stay
  reachable after a newer destination ranks first
- the trace read limiter takes its store from the route
- transient availability failures retry instead of hiding the control

* 🛟 fix: Fail Over Between Projects, Refresh One Page, Search Visible Labels

- a fresh read fails over to another project holding the same turns when the
  preferred one fails; the cursor carries the failed sources so later pages
  keep the rebuilt plan, and detail reads fail over the same way
- destinations for one project collapse into one source
- the trace read limiter keys each user within their tenant
- refresh and a settled run keep only the newest page before refetching, so
  older pages are not replayed through the limiter; retry repeats the read
  that failed
- search matches the localized kind and status labels a row shows

* 🧭 fix: Read the Newest Turn First, Report Failures Behind Empty Projects, Fit Trimmed Views

Rank trace sources by the newest sampled response they could hold before coverage and preference, so a conversation that moved projects shows its latest turn first.

Throw a list failure when a failed project could hold a turn that no answering project provably holds, instead of rendering an empty trace. Detail reads keep probing projects after an empty answer, skip ones whose turns an answering project provably holds, and surface a failure that could hide the record.

Refresh and settled runs trim the cache to the newest page; the viewer now fits a focused interval to the remaining records or clears it, and drops a selection whose record left with the older pages.

* 🧵 fix: Page Traces by Turn, Include Failed Runs, Keep the Active Row Mounted

List reads now page by turn segments instead of whole projects. A segment is the newest unread turn and up to 49 older consecutive turns one project could hold. The request filters Langfuse to those turns' run and title traces, so pages stay newest first when a conversation's responses alternate between projects. Legacy turns without destination ids are asked of the other projects in the segment. A project that failed for newer turns is retried when it alone holds older turns. A page whose rows all fail the observation schema is an upstream failure, not an empty trace.

A failed turn's error row now stores the sampling fields of the run that failed, plus langfuseRunId when that run was created, so the trace viewer and feedback scores follow the run's trace. langfuseRunId is stripped from client message writes and imports, dropped by forks, and hidden from client message projections.

The ledger keeps its active tree item mounted while virtualization scrolls it out of the window, so aria-activedescendant always names a rendered row.

* 🕰️ fix: Merge Multi-Project Trace Pages by Start Time, Keep Literal Content

A segment that reads more than one project now runs every read before returning. A read that fills the record budget cuts the page at the oldest start time all reads loaded, and the continuation carries that start time as an inclusive bound instead of a Langfuse cursor. Every project continues from the same point, so a newer turn held by another project is never left behind an older one, and failover works on any page. A page drawn from several projects no longer names a single source. A continuation that would repeat its own bound fails instead of looping.

Input and output that are literally the strings {} or null are kept; only an empty object or JSON null value is dropped.

* 🔎 fix: Read Each Trace Segment From a Project Shown to Hold It

Recorded destination ids say where a trace was eligible to go, not where it landed. A list read now looks up the turns more than one readable project could hold before choosing where to read them. Each project, preferred first, is asked which of those traces it holds a root observation for, and each turn is read from the preferred project that holds it, or from the preferred eligible one while a run's root is not exported yet. Turns only one project could hold need no lookup.

Segments are consecutive turns from one project, paged by that project's own Langfuse cursor. Records that share a start time page correctly, and the page budget bounds a single read. A failed project is dropped for the request. A turn only it could hold raises its failure at the turn's own place in page order, instead of leaving an empty or partial trace. Cursors carry the segment's trace-set hash and start over when the segment changed or Langfuse rejects them. Detail reads ask every project until the record turns up.

The trace summary withholds its cost total when any generation has no cost, with or without usage.

* 🧷 fix: Evidence per Trace, Continuations Without Replays, Stable Turn Order

A turn's run and title run are exported separately, so each trace id is looked up and read from a project shown to hold it. A trace no project shows (a title run the turn never had, or a run whose root is not exported yet) joins an adjacent segment whose project could hold it. Only a run that a failed project alone could hold raises that failure.

A continuation now stays on its project. Its failure is returned as is, and a cursor the segment no longer matches, or one Langfuse rejects, returns invalid_request instead of replaying the segment's first page. The viewer answers invalid_request on an older page by reloading from the newest page, with a message saying the trace changed.

Sampled responses sort by createdAt then _id, so responses saved in the same millisecond keep one order across page requests.

* 🪪 fix: Scope Trace Reads to the Exporting User, Order Split Title Runs

Trace ids derive from message ids, which a request can influence, so a colliding id must not return someone else's trace. Every Langfuse list, probe and detail read now also filters by userId: the requester's internal id, plus the configured langfuse.trace.userIdField value when set. The server stamps that id on the trace at export. The handler passes the requesting user into the trace query.

When a turn's run and title run sit in different projects, the probes' root start times decide which is read first, so a title generated after the response is not left behind Load older. Cursors name the trace a segment starts at within its turn.

The failed-turn trace lookup moves into packages/api as getFailedTurnTraceFields. The agent controller only passes the error row id, the run id and whether the run was created.

* 🔒 fix: Authorize Trace Reads by the Internal User Id Only

A configured langfuse.trace.userIdField can export a non-unique value (name, username) or one that repeats across tenants (email, provider ids) as the trace userId, so it cannot prove who a trace belongs to. Reads now require the internal user id alone. A deployment that exports another field reports no trace and logs why once, instead of trusting that value.

A failed read of a title run that a probe found in that project now raises its failure instead of dropping the title. The page contract now states that pages are ordered by turn and a turn's records may continue on the next page.

* 📚 fix: Page Sampled Responses With the Trace, Share Ingress Stripping

List reads load sampled response references a segment at a time: up to 50 ending at the cursor's turn, plus one older response that names the next page. Pages of a long conversation no longer rematerialize every sampled response. getConversationTraceRefs gains through and limit, anchored by createdAt and _id.

The message-create route and conversation imports strip trace sampling fields through withoutTraceRefs instead of their own field lists. A userIdField the exporter ignores counts as the internal id. A root lookup whose cursor repeats fails that project instead of paging forever.

In the viewer, Refresh also rereads the open record's detail, and closing the inspector after a filter removed every row returns focus to the search field.

* ⏱️ fix: One Round Trip per Trace Reference Read, Detail Reads Scoped to Their Turn

Cursors carry the anchor response's order key (creation time, then _id). The bounded sampled-response read then runs alongside the first-message lookup, with no anchor lookup ahead of it. A key that no longer names its response returns nothing, and the reader answers the cursor as changed.

Detail reads take the turn the list attributed the record to (?message=). They load only that sampled response instead of the conversation's whole history, and ask Langfuse within that turn's traces. The route rejects a detail read without it, and the client sends it from the listed record.

A list read whose Langfuse cursor repeats now fails instead of handing the next page the same records.

* 🧭 fix: Reject Out-of-range Trace Cursors, Open Any Listed Turn, Mark Copies Unsampled

A page cursor whose base-36 time parses to a number outside the Date range reached Mongoose as an Invalid Date and surfaced as a 500 the viewer retried; it now resumes nothing, which the reader reports as invalid_request so the viewer reloads its newest page.

The detail read capped the message id at the Langfuse record-id bound while the app stores response ids at any length, so a listed turn could refuse to open; message ids only name a stored row and are accepted at any length. Ids that reach a Langfuse filter keep their caps.

Copied and client-authored rows now carry langfuseSampled: false instead of no record, so a feedback score on a fork, import or posted message never recomputes sampling for a trace that was never made.

* 🧪 test: Type a Recordless Row for the Unsampled-copy Spec

An object literal with none of the trace fields fails TypeScript's weak-type check against the helper's all-optional constraint; the spec now passes a typed row that declares the optional field, as every stored message type does.

* 🚰 fix: Release Failed Langfuse Response Bodies

A non-success Langfuse response was turned into a TraceReadError without consuming or cancelling its body, and Node's fetch returns a connection to its pool only once the body is released, so a sustained upstream failure left sockets checked out until garbage collection. The body is now cancelled before the status becomes an error.
2026-09-13 00:56:52 -04:00

617 lines
23 KiB
JavaScript

require('../config/credentials');
const telemetry = require('./telemetry');
const fs = require('fs');
const path = require('path');
require('module-alias')({ base: path.resolve(__dirname, '..') });
const cors = require('cors');
const axios = require('axios');
const express = require('express');
const mongoose = require('mongoose');
const passport = require('passport');
const compression = require('compression');
const cookieParser = require('cookie-parser');
const mongoSanitize = require('express-mongo-sanitize');
const { logger, runAsSystem } = require('@librechat/data-schemas');
const {
isEnabled,
issueCsp,
apiNotFound,
createMetrics,
applyCspNonce,
createCspPolicy,
shellCacheHeaders,
escapeHtmlAttribute,
ErrorController,
memoryDiagnostics,
createSecurityHeaders,
performStartupChecks,
handleJsonParseError,
GenerationJobManager,
QUERY_DEVTOOLS_HEADER,
createStreamServices,
agentStartupIngressMiddleware,
agentStartupTelemetryMiddleware,
initializeFileStorage,
initializeDeploymentSkills,
initializeDeploymentPlugins,
getDeploymentPluginSkills,
getDeploymentPluginHookCapabilities,
registerDeploymentPluginHooks,
hasDeploymentPluginHooks,
hasDeploymentPluginToolApprovalHooks,
setPluginHookSource,
loadToolApprovalHooks,
maybeInjectQueryDevtoolsBootstrap,
preAuthTenantMiddleware,
requestContextMiddleware,
registerShutdownTask,
getRemainingShutdownMs,
configureServerTimeouts,
setupGracefulShutdown,
updateInterfacePermissions,
configureMessageFilterRegexValidator,
configureFileConfigRegexEngine,
configureAgentEventRuntime,
createAgentEventTerminalHandler,
createScheduleWriteGate,
startCodeEnvironmentLifecycleReconciler,
waitForKeyvRedisClient,
warnOnUnreachableDeliveryPaths,
createCodeApiUploadRegistry,
} = require('@librechat/api');
const { connectDb, indexSync } = require('~/db');
const {
updateAccessPermissions,
sweepOrphanedPreviews,
getRoleByName,
seedDatabase,
} = require('~/models');
const initializeOAuthReconnectManager = require('./services/initializeOAuthReconnectManager');
const { capabilityContextMiddleware } = require('./middleware/roles/capabilities');
const createValidateImageRequest = require('./middleware/validateImageRequest');
const { initializeGitHubSkillSync } = require('./services/Skills/sync');
const { initializeAgentTriggerService } = require('./services/Agents/triggers');
const { resumeAgentEventDetachedAction } = require('./services/Agents/detachedActionResume');
const { initializeScheduleEngine, recordExpiredScheduleApproval } = require('./services/Schedules');
const { jwtLogin, ldapLogin, passportLogin } = require('~/strategies');
const { startExpiredFileSweep } = require('./services/Files/process');
const { checkMigrations } = require('./services/start/migration');
const optionalJwtAuth = require('./middleware/optionalJwtAuth');
const initializeMCPs = require('./services/initializeMCPs');
const { configureSubagentTaskRouting } = require('./services/Endpoints/agents/subagentThreadStore');
const configureSocialLogins = require('./socialLogins');
const createSpaFallback = require('./utils/fallback');
const { getAppConfig } = require('./services/Config');
const staticCache = require('./utils/staticCache');
const noIndex = require('./middleware/noIndex');
const routes = require('./routes');
const agentEventMethods = require('~/models');
/** Route admin file-config MIME patterns through a linear-time engine (ReDoS-safe) on upload. */
configureFileConfigRegexEngine();
/** Reject messageFilter PII patterns the RE2 runtime engine cannot compile, at config load. */
configureMessageFilterRegexValidator();
const { PORT, HOST, ALLOW_SOCIAL_LOGIN, DISABLE_COMPRESSION, TRUST_PROXY } = process.env ?? {};
// Allow PORT=0 to be used for automatic free port assignment
const port = isNaN(Number(PORT)) ? 3080 : Number(PORT);
const host = HOST || 'localhost';
const trusted_proxy = Number(TRUST_PROXY) || 1; /* trust first proxy by default */
const app = express();
app.locals.codeApiUploadRegistry = createCodeApiUploadRegistry();
let serverReady = false;
/** @type {import('@librechat/api').ScheduleEngineState} */
let scheduleEngineState = 'starting';
const SERVER_NOT_READY_CODE = 'SERVER_NOT_READY';
const CHAT_START_RETRY_AFTER_SECONDS = '1';
const rejectChatStartsUntilReady = (req, res, next) => {
if (serverReady || req.method !== 'POST' || req.path === '/abort') {
return next();
}
res.set('Retry-After', CHAT_START_RETRY_AFTER_SECONDS);
return res.status(503).json({
code: SERVER_NOT_READY_CODE,
error: 'Server is still starting. Please retry shortly.',
});
};
const rejectScheduleWritesUntilReady = createScheduleWriteGate({
getState: () => scheduleEngineState,
retryAfterSeconds: CHAT_START_RETRY_AFTER_SECONDS,
});
const configureGenerationStreams = () => {
const streamServices = createStreamServices();
GenerationJobManager.configure({
...streamServices,
cleanupOnComplete: !isEnabled(process.env.STREAM_KEEP_COMPLETED_JOBS),
});
GenerationJobManager.setApprovalExpiredHandler(recordExpiredScheduleApproval);
GenerationJobManager.setTerminalHostActionHandler(
createAgentEventTerminalHandler(agentEventMethods, {
resumeDetachedAction: resumeAgentEventDetachedAction,
}),
);
GenerationJobManager.initialize();
// Stop active generations and close their SSE streams while the HTTP server drains.
registerShutdownTask(
'generation job manager prepare',
() => GenerationJobManager.prepareForShutdown(),
{
phase: 'pre-drain',
priority: 100,
},
);
/** Spend the shutdown budget that is actually left waiting for detached generations to
* record their own provider drains, holding back a reserve for the tasks after this one.
* Abandoning an unrecorded drain fences the next generation permanently. */
const destroyGenerationJobManager = () => {
const remaining = getRemainingShutdownMs();
return GenerationJobManager.destroy(
remaining == null
? undefined
: { settlementBudgetMs: Math.max(0, remaining - SHUTDOWN_TEARDOWN_RESERVE_MS) },
);
};
// Tear down stream resources before shared caches and telemetry exporters shut down.
registerShutdownTask('generation job manager', destroyGenerationJobManager, { priority: 100 });
};
/** Reserved for the shutdown tasks that run after the generation job manager. */
const SHUTDOWN_TEARDOWN_RESERVE_MS = 10_000;
const startServer = async () => {
await waitForKeyvRedisClient();
await configureSubagentTaskRouting();
const { metricsMiddleware, metricsRouter } = createMetrics({
collectAgentEventActorStorageMetrics: () =>
runAsSystem(async () => {
const now = new Date();
const [receiptMetrics, reconciliationMetrics] = await Promise.all([
agentEventMethods.getAgentEventActorReceiptStorageMetrics(now),
agentEventMethods.getAgentEventActorReconciliationStorageMetrics(now),
]);
return {
...receiptMetrics,
pendingReconciliations: reconciliationMetrics.pending,
oldestPendingAgeSeconds: reconciliationMetrics.oldestPendingAgeSeconds,
};
}),
});
if (!process.env.METRICS_SECRET) {
logger.warn('[metrics] METRICS_SECRET is not set - /metrics will return 401 for all requests');
}
if (typeof Bun !== 'undefined') {
axios.defaults.headers.common['Accept-Encoding'] = 'gzip';
}
await connectDb();
logger.info('Connected to MongoDB');
startCodeEnvironmentLifecycleReconciler({ mongoose });
indexSync().catch((err) => {
logger.error('[indexSync] Background sync failed:', err);
});
app.disable('x-powered-by');
app.set('trust proxy', trusted_proxy);
/* Registered ahead of every route so health checks carry the headers too. */
const securityHeaders = createSecurityHeaders();
if (securityHeaders) {
app.use(securityHeaders);
}
if (isEnabled(process.env.TRUST_TENANT_HEADER)) {
logger.warn(
'[Security] TRUST_TENANT_HEADER is active. Ensure your reverse proxy strips and sets ' +
'X-Tenant-Id — untrusted clients must not be able to supply it directly.',
);
} else if (isEnabled(process.env.TENANT_ISOLATION_STRICT)) {
logger.warn(
'[Security] TENANT_ISOLATION_STRICT is active while TRUST_TENANT_HEADER is disabled. ' +
'Pre-authentication tenant headers will be ignored.',
);
}
await runAsSystem(seedDatabase);
/* Recover stuck `status: 'pending'` records from a crash mid-render.
* `runAsSystem` is required — `File` is tenant-isolated and strict
* mode rejects unscoped queries. Lazy sweep in the preview endpoint
* covers anything younger than the boot cutoff. */
runAsSystem(sweepOrphanedPreviews).catch((err) => {
logger.error('[sweepOrphanedPreviews] Background sweep failed:', err);
});
const appConfig = await getAppConfig({ baseOnly: true });
configureAgentEventRuntime(appConfig?.endpoints?.agents?.eventDriven);
warnOnUnreachableDeliveryPaths(appConfig);
initializeFileStorage(appConfig);
const projectRoot = path.resolve(__dirname, '../..');
// Plugin hooks execute only when the operator opts in via DEPLOYMENT_PLUGIN_HOOKS;
// without it, declared hook documents load as parsed-but-inert with a warning.
await initializeDeploymentPlugins({
projectRoot,
hookCapabilities: getDeploymentPluginHookCapabilities(),
});
// Hand the run seam its plugin-hook source without a packages/api-internal
// agents -> plugins import (see agents/hooks/source.ts).
setPluginHookSource({
hasHooks: hasDeploymentPluginHooks,
hasToolApprovalHooks: hasDeploymentPluginToolApprovalHooks,
register: registerDeploymentPluginHooks,
});
await initializeDeploymentSkills({
projectRoot,
additionalSkills: getDeploymentPluginSkills(),
});
initializeGitHubSkillSync(appConfig);
startExpiredFileSweep({ appConfig, loadAppConfig: getAppConfig });
// Register any programmatic tool-approval policy hooks declared in
// `endpoints.agents.toolApproval.hooks`. Honor the `enabled` kill switch: when tool
// approval is off we pass no hooks, so a disabled endpoint imports/runs nothing (and any
// previously loaded batch is unregistered). Hooks are read from the BASE config only —
// they register once, process-wide; per-user/tenant differences belong inside the hook
// (via its context), not in per-override module lists.
const toolApproval = appConfig?.endpoints?.agents?.toolApproval;
await loadToolApprovalHooks(toolApproval?.enabled ? toolApproval.hooks : undefined, {
basePath: path.resolve(__dirname, '../..'),
});
await runAsSystem(async () => {
await performStartupChecks(appConfig);
await updateInterfacePermissions({ appConfig, getRoleByName, updateAccessPermissions });
});
const indexPath = path.join(appConfig.paths.dist, 'index.html');
let indexHTML = fs.readFileSync(indexPath, 'utf8');
// In order to provide support to serving the application in a sub-directory
// We need to update the base href if the DOMAIN_CLIENT is specified and not the root path
if (process.env.DOMAIN_CLIENT) {
const clientUrl = new URL(process.env.DOMAIN_CLIENT);
const baseHref = clientUrl.pathname.endsWith('/')
? clientUrl.pathname
: `${clientUrl.pathname}/`;
if (baseHref !== '/') {
logger.info(`Setting base href to ${baseHref}`);
indexHTML = indexHTML.replace(/base href="\/"/, `base href="${baseHref}"`);
}
}
const cspPolicy = createCspPolicy();
const shellCache = shellCacheHeaders(cspPolicy != null);
const sendIndexHtml = (req, res) => {
res.set(shellCache);
res.vary(QUERY_DEVTOOLS_HEADER);
const lang = req.cookies.lang || req.headers['accept-language']?.split(',')[0] || 'en-US';
const saneLang = escapeHtmlAttribute(lang);
let updatedIndexHtml = indexHTML.replace(/lang="en-US"/g, () => `lang="${saneLang}"`);
updatedIndexHtml = maybeInjectQueryDevtoolsBootstrap(updatedIndexHtml, req);
/* Nonce last: every injected script above must be stamped too. */
if (cspPolicy) {
const csp = issueCsp(cspPolicy);
res.set(csp.headerName, csp.headerValue);
updatedIndexHtml = applyCspNonce(updatedIndexHtml, csp.nonce);
}
res.type('html');
res.send(updatedIndexHtml);
};
app.get('/health', (_req, res) => res.status(200).send('OK'));
app.get('/livez', (_req, res) => res.status(200).send('OK'));
app.get('/readyz', (_req, res) => {
if (!serverReady) {
return res.status(503).send('NOT_READY');
}
return res.status(200).send('OK');
});
/* Middleware */
app.use(requestContextMiddleware);
app.use('/api/agents/chat', agentStartupIngressMiddleware);
app.use(metricsMiddleware);
app.use(noIndex);
app.use(express.json({ limit: '3mb' }));
app.use(express.urlencoded({ extended: true, limit: '3mb' }));
app.use(handleJsonParseError);
/**
* Express 5 Compatibility: Make req.query writable for mongoSanitize
* In Express 5, req.query is read-only by default, but express-mongo-sanitize needs to modify it
*/
app.use((req, _res, next) => {
Object.defineProperty(req, 'query', {
...Object.getOwnPropertyDescriptor(req, 'query'),
value: req.query,
writable: true,
});
next();
});
app.use(mongoSanitize());
app.use(cors());
app.use(cookieParser());
if (!isEnabled(DISABLE_COMPRESSION)) {
app.use(compression());
} else {
console.warn('Response compression has been disabled via DISABLE_COMPRESSION.');
}
app.get('/index.html', sendIndexHtml);
app.use(staticCache(appConfig.paths.dist));
app.use(staticCache(appConfig.paths.fonts));
app.use(staticCache(appConfig.paths.assets));
if (telemetry.enabled) {
app.use(telemetry.telemetryMiddleware);
}
app.use('/api/agents/chat', agentStartupTelemetryMiddleware);
if (!ALLOW_SOCIAL_LOGIN) {
console.warn('Social logins are disabled. Set ALLOW_SOCIAL_LOGIN=true to enable them.');
}
/* OAUTH */
app.use(passport.initialize());
passport.use(jwtLogin());
passport.use(passportLogin());
/* LDAP Auth */
if (process.env.LDAP_URL && process.env.LDAP_USER_SEARCH_BASE) {
passport.use(ldapLogin);
}
if (isEnabled(ALLOW_SOCIAL_LOGIN)) {
await configureSocialLogins(app);
}
/* Per-request capability cache — must be registered before any route that calls hasCapability */
app.use(capabilityContextMiddleware);
/* Pre-auth tenant context for unauthenticated routes that need tenant scoping.
* The reverse proxy / auth gateway sets `X-Tenant-Id` header for multi-tenant deployments. */
app.use('/oauth', preAuthTenantMiddleware, routes.oauth);
/* API Endpoints */
app.use('/api/auth', preAuthTenantMiddleware, routes.auth);
app.use('/api/insights', routes.insights);
app.use('/api/admin', routes.adminAuth);
app.use('/api/admin/config', routes.adminConfig);
app.use('/api/admin/code-environments', routes.adminCodeEnvironments);
app.use('/api/code-environments', routes.codeEnvironments);
app.use('/api/admin/langfuse', routes.adminLangfuse);
app.use('/api/admin/grants', routes.adminGrants);
app.use('/api/admin/groups', routes.adminGroups);
app.use('/api/admin/roles', routes.adminRoles);
app.use('/api/admin/skills', routes.adminSkills);
app.use('/api/admin/users', routes.adminUsers);
app.use('/api/admin/audit-log', routes.adminAuditLog);
app.use('/api/actions', routes.actions);
app.use('/api/keys', routes.keys);
app.use('/api/api-keys', routes.apiKeys);
app.use('/api/user', routes.user);
app.use('/api/search', routes.search);
app.use('/api/messages', routes.messages);
app.use('/api/convos', routes.convos);
app.use('/api/traces', routes.traces);
app.use('/api/presets', routes.presets);
app.use('/api/projects', routes.projects);
app.use('/api/prompts', routes.prompts);
app.use('/api/skills', routes.skills);
app.use('/api/categories', routes.categories);
app.use('/api/endpoints', routes.endpoints);
app.use('/api/balance', routes.balance);
app.use('/api/models', routes.models);
app.use('/api/config', preAuthTenantMiddleware, optionalJwtAuth, routes.config);
app.use('/api/assistants', routes.assistants);
app.use('/api/files', await routes.files.initialize());
app.use(
'/images/',
createValidateImageRequest({
secureImageLinks: appConfig.secureImageLinks,
}),
routes.staticRoute,
);
app.use('/api/share', preAuthTenantMiddleware, routes.share);
app.use('/api/roles', routes.roles);
app.use('/api/agents/chat', rejectChatStartsUntilReady);
app.use('/api/agents', routes.agents);
app.use('/api/banner', routes.banner);
app.use('/api/memories', routes.memories);
app.use('/api/schedules', rejectScheduleWritesUntilReady, routes.schedules);
app.use('/api/permissions', routes.accessPermissions);
app.use('/api/tags', routes.tags);
app.use('/api/mcp', routes.mcp);
app.use('/api/rum', routes.rum);
app.use('/metrics', metricsRouter);
/** 404 for unmatched API routes */
app.use('/api', apiNotFound);
/** SPA fallback - serve index.html for all unmatched routes */
app.use(createSpaFallback(sendIndexHtml));
/** Record trace errors before the final error controller. */
if (telemetry.enabled) {
app.use(telemetry.telemetryErrorMiddleware);
}
/** Error handler (must be last - Express identifies error middleware by its 4-arg signature) */
app.use(ErrorController);
configureGenerationStreams();
const server = app.listen(port, host, async (err) => {
if (err) {
logger.error('Failed to start server:', err);
process.exit(1);
}
if (host === '0.0.0.0') {
logger.info(
`Server listening on all interfaces at port ${port}. Use http://localhost:${port} to access it`,
);
} else {
logger.info(`Server listening at http://${host == '0.0.0.0' ? 'localhost' : host}:${port}`);
}
/**
* The listen callback is async, so any rejection from these awaits would
* otherwise be detached from `startServer().catch(...)` (which only
* catches errors that happen before `app.listen`). Without explicit
* handling, the global `unhandledRejection` handler would swallow init
* failures and leave the server listening but only partially
* initialized — passing liveness checks while serving broken requests.
*/
try {
await runAsSystem(async () => {
await initializeMCPs();
await initializeOAuthReconnectManager();
});
await checkMigrations();
const inspectFlags = process.execArgv.some((arg) => arg.startsWith('--inspect'));
if (inspectFlags || isEnabled(process.env.MEM_DIAG)) {
memoryDiagnostics.start();
}
await initializeAgentTriggerService({ address: server.address() });
const scheduleEngineArmed = (await initializeScheduleEngine()) != null;
scheduleEngineState = scheduleEngineArmed ? 'armed' : 'unavailable';
if (!scheduleEngineArmed) {
// Terminal, not transient: arming is attempted once, so schedule writes are refused
// for the life of this process. Logged at error level because the only other signal
// an operator gets is a 503 on every write — every other health signal stays green.
logger.error(
'[schedules] write routes are PERMANENTLY unavailable in this process: the engine did not arm. ' +
'Resolve the cause logged above and restart.',
);
}
serverReady = true;
logger.info('Server readiness checks passing.');
} catch (initErr) {
serverReady = false;
logger.error('Post-listen initialization failed:', initErr);
process.exit(1);
}
});
configureServerTimeouts(server);
logger.info('HTTP server timeout configuration', {
keepAliveTimeout: server.keepAliveTimeout,
keepAliveTimeoutBuffer: server.keepAliveTimeoutBuffer,
headersTimeout: server.headersTimeout,
requestTimeout: server.requestTimeout,
});
setupGracefulShutdown(server);
};
/**
* Boot rejections (e.g. `connectDb`, `getAppConfig`, `performStartupChecks`)
* must remain fail-fast: a half-initialized process with no listening HTTP
* server should die immediately so the orchestrator restarts it, instead of
* being kept alive by the `unhandledRejection` handler below until the
* liveness probe eventually times out. Mirrors the pattern in
* `experimental.js`.
*/
startServer().catch((err) => {
logger.error('Failed to start server:', err);
process.exit(1);
});
let messageCount = 0;
process.on('uncaughtException', (err) => {
if (!err.message.includes('fetch failed')) {
logger.error('There was an uncaught error:', err);
}
if (err.message && err.message?.toLowerCase()?.includes('abort')) {
logger.warn('There was an uncatchable abort error.');
return;
}
if (err.message.includes('GoogleGenerativeAI')) {
logger.warn(
'\n\n`GoogleGenerativeAI` errors cannot be caught due to an upstream issue, see: https://github.com/google-gemini/generative-ai-js/issues/303',
);
return;
}
if (err.message.includes('fetch failed')) {
if (messageCount === 0) {
logger.warn('Meilisearch error, search will be disabled');
messageCount++;
}
return;
}
if (err.message.includes('OpenAIError') || err.message.includes('ChatCompletionMessage')) {
logger.error(
'\n\nAn Uncaught `OpenAIError` error may be due to your reverse-proxy setup or stream configuration, or a bug in the `openai` node package.',
);
return;
}
if (err.stack && err.stack.includes('@librechat/agents')) {
logger.error(
'\n\nAn error occurred in the agents system. The error has been logged and the app will continue running.',
{
message: err.message,
stack: err.stack,
},
);
return;
}
if (isEnabled(process.env.CONTINUE_ON_UNCAUGHT_EXCEPTION)) {
logger.error('Unhandled error encountered. The app will continue running.', {
name: err?.name,
message: err?.message,
stack: err?.stack,
});
return;
}
process.exit(1);
});
/**
* Unhandled promise rejection handler.
*
* Node 15+ terminates the process by default when a promise rejection is
* unhandled. MCP OAuth reconnect storms and streamable-HTTP transport resets
* can produce transient fire-and-forget rejections (ECONNRESET, token refresh
* races) that are recoverable — the server should log and keep serving other
* requests rather than silently crash under load.
*
* Non-Error reasons are forwarded as-is so structured payloads (e.g.
* `{ code: "ECONNRESET", errno: -104 }`) survive instead of being collapsed to
* "[object Object]" by `String()`.
*/
process.on('unhandledRejection', (reason) => {
if (reason instanceof Error) {
logger.error('Unhandled promise rejection. The app will continue running.', {
name: reason.name,
message: reason.message,
stack: reason.stack,
cause: reason.cause,
});
return;
}
logger.error('Unhandled promise rejection. The app will continue running.', { reason });
});
/** Export app for easier testing purposes */
module.exports = app;