🗂️ feat: Audit MCP, Code Worker, Langfuse, Skill Credential, Passkey and Audit Log Access (#17054)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions

* fix: Remove Group Dependents Before Deleting the Group

* feat: Audit Code Worker Pairing and Langfuse Connection Changes

* feat: Audit Skill Sync Credential Changes

* feat: Audit Passkey Enrolment and Removal

* feat: Audit Audit Log Reads and CSV Exports

* feat: Audit Audit Log Retention Purges

* feat: Audit MCP Server Create, Update and Delete

* fix: Keep Purge Result When the Purge Audit Write Fails

* test: Cover Admin Integration Audit Scenarios

* fix: Tighten Admin Integration Audit Scope and Group Cleanup Order

* fix: Run Security Fences Before Awaiting Audit Writes

* fix: Move MCP Post-Commit Audit Ordering Into the Auditor and Audit Verify Reads

* fix: Fail Closed When Recording Audit Log Access

* fix: Audit Group Deletion Before the Cascade and Restore Dependents on Failure

* fix: Restore Only Config Overrides When Group Deletion Fails

* fix: Audit Worker Pairing and Export Before the Work and Append Failures

* fix: Record Group Deletion as Pending Without Delaying Revocation

* fix: Skip Config Restore When a Concurrent Delete Removed the Group

* fix: Restore Group Config When the Existence Recheck Fails

* fix: Keep Group Config Restricted When Delete State Is Unknown

* fix: Record Worker Pairing and Revocation as Pending Until Confirmed

* fix: Deliver the Pairing Code Before the Terminal Audit Write

* test: Assert Pairing Code Delivery Precedes the Terminal Audit Write

* fix: Send Revocation and Group Deletion Responses Before the Terminal Audit Write

* fix: Snapshot Group Config Before the Destructive Cascade Step

* refactor: Type the Group Config Snapshot Dependency Without Mongoose Shapes

* fix: Fence Shared MCP Users Before the Deletion Audit and Correlate Audit Attempts

* fix: Diff MCP Server Audits Against the Committed Config

* fix: Read the Committed MCP Config at Commit Time for the Audit Diff

* refactor: Start the Committed MCP Config Read Inside the Auditor

* fix: Audit Langfuse Patches When Activation Fails and Surface Failed Group Config Restores

* fix: Record Member Removals When Group Deletion Completes After a Failed Restore

* test: Serve the Committed Config on the MCP Update Audit Reread

* fix: Audit the Group Returned by the Compensating Deletion

* fix: Keep Member Removal Audits Fail-Open During Group Deletion

* fix: Retry ACL Cleanup After a Compensating Group Deletion

* fix: Mark Unavailable Audit Detail and Order Member Removals After the Pending Deletion

* fix: Guard Synchronous Committed-Config Read Failures in the MCP Auditor

* fix: Return the Persisted MCP Config and Audit Commits That Outlive Cache Invalidation

* style: Sort Imports in MCP Audit Files

* style: Sort Imports After Restack

* fix: Audit MCP Commits That Outlive Any Post-Write Step and Pend Audit Exports Until Complete

* test: Provide the Fail-Closed Setting in the Code Route Mock

* fix: Refresh MCP Read Caches When a Repository Step Fails After the Write

* fix: Finish Audit Exports on Response Completion and Keep Error Details Out of Messages and Logs

* fix: Watch Response Completion Before the Truncation Footer Write

* fix: Skip Audit for No-op MCP Deletes and Close Remaining Post-Failure Audit Gaps

* fix: Keep Causes Out of Structured Logs and Treat a Lost Compensating Delete as Not Found

* fix: Mark Failed MCP Post-Commit Steps and Null Langfuse Activation in Audits
This commit is contained in:
Marco Beretta 2026-10-15 00:30:26 +02:00 • committed by GitHub
parent 76192bdfd0
commit 9adabde72e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
36 changed files with 2758 additions and 147 deletions

View file

@ -20,4 +20,5 @@ module.exports = createPasskeyHandlers({
findPasskeysByUser: db.findPasskeysByUser,
countPasskeysByUser: db.countPasskeysByUser,
findPasskeyByCredentialId: db.findPasskeyByCredentialId,
recordAuditEntry: db.recordAuditEntry,
});

View file

@ -20,6 +20,8 @@ const {
redactServerSecrets,
sanitizeMcpIconPath,
redactAllServerSecrets,
createMCPServerAuditor,
settleMCPCommit,
getMCPErrorResponse,
prepareMCPServerOAuthDeletion,
cleanupDeletedMCPServerOAuthUsers,
@ -52,6 +54,8 @@ const { hasCapability } = require('~/server/middleware/roles/capabilities');
const { getMCPManager, getMCPServersRegistry } = require('~/config');
const db = require('~/models');
const auditMCPServer = createMCPServerAuditor({ recordAuditEntry: db.recordAuditEntry });
/**
* Handles MCP-specific errors and sends appropriate HTTP responses.
* @param {Error} error - The error to handle
@ -457,18 +461,30 @@ const createMCPServerController = async (req, res) => {
...new Set([...configNames, ...configNames.map(normalizeServerName)]),
];
const inspectionServerName = `temp_server_${randomUUID()}`;
let result;
let settled;
try {
result = await getMCPServersRegistry().addServer(
inspectionServerName,
validation.data,
'DB',
userId,
reservedServerNames,
settled = await settleMCPCommit(() =>
getMCPServersRegistry().addServer(
inspectionServerName,
validation.data,
'DB',
userId,
reservedServerNames,
),
);
} finally {
MCPConnection.clearCooldown(inspectionServerName);
}
const result = settled.committed;
await auditMCPServer(req, {
action: 'mcp_server.created',
serverName: result.serverName,
next: result.config,
postWriteFailed: settled.postWriteError != null,
});
if (settled.postWriteError) {
throw settled.postWriteError;
}
res.status(201).json({
serverName: result.serverName,
...redactServerSecrets(result.config, { canEdit: true }),
@ -572,8 +588,11 @@ const updateMCPServerController = async (req, res) => {
);
const retainedTools = await getMCPServerTools(userId, serverName, existingConfig);
await invalidateCachedTools({ userId, serverName });
let settled;
try {
await registry.commitServerUpdate(serverName, parsedConfig, 'DB', userId);
settled = await settleMCPCommit(() =>
registry.commitServerUpdate(serverName, parsedConfig, 'DB', userId),
);
} catch (error) {
await restoreRetainedServerCatalog({
userId,
@ -583,12 +602,28 @@ const updateMCPServerController = async (req, res) => {
});
throw error;
}
const committedConfig = settled.committed ?? parsedConfig;
/** Fence connections another replica could have created from the old DB
* config between the pre-commit fence and the committed update. */
await fenceCommittedMCPMutation({ userId, serverName });
await disconnectLocalMCPServer(userId, serverName);
await auditMCPServer(
req,
{
action: 'mcp_server.updated',
serverName,
previous: existingConfig,
next: committedConfig,
postWriteFailed: settled.postWriteError != null,
},
async () => {
await fenceCommittedMCPMutation({ userId, serverName });
await disconnectLocalMCPServer(userId, serverName);
},
);
if (settled.postWriteError) {
throw settled.postWriteError;
}
res.status(200).json(redactServerSecrets(parsedConfig, { canEdit: true }));
res.status(200).json(redactServerSecrets(committedConfig, { canEdit: true }));
} catch (error) {
logger.error('[updateMCPServer]', error);
const mcpErrorResponse = handleMCPError(error, res);
@ -631,8 +666,9 @@ const deleteMCPServerController = async (req, res, uninstallOAuthMCP) => {
getMCPServerTools(userId, serverName, existingConfig),
]);
await invalidateCachedTools({ userId, serverName });
let settled;
try {
await registry.removeServer(serverName, 'DB', userId);
settled = await settleMCPCommit(() => registry.removeServer(serverName, 'DB', userId));
} catch (error) {
await restoreRetainedServerCatalog({
userId,
@ -642,34 +678,51 @@ const deleteMCPServerController = async (req, res, uninstallOAuthMCP) => {
});
throw error;
}
/** Fence connections another replica could have created before deletion committed. */
await fenceCommittedMCPMutation({ userId, serverName });
await disconnectLocalMCPServer(userId, serverName);
try {
await cleanupDeletedMCPServerOAuthUsers({
ownerUserId: userId,
/** Fence connections another replica could have created before deletion committed,
* for every user holding a token, before the audit write is awaited. */
await auditMCPServer(
req,
{
action: 'mcp_server.deleted',
serverName,
serverConfig: existingConfig,
snapshot: oauthDeletionSnapshot,
getTokenUserIds,
getUserPrincipals: (candidateUserId) => db.getUserPrincipals({ userId: candidateUserId }),
resolveAllowlists: (candidateUserId) =>
registry.resolveAllowlists({ userId: candidateUserId }),
fenceAndDisconnectUser: async (candidateUserId) => {
if (candidateUserId === userId) {
return;
}
await fenceCommittedMCPMutation({ userId: candidateUserId, serverName });
await disconnectLocalMCPServer(candidateUserId, serverName);
},
uninstallOAuthMCP,
});
} catch (error) {
logger.warn(
`[deleteMCPServer] Server ${serverName} was deleted, but OAuth cleanup failed for user ${userId}:`,
error,
);
throw error;
previous: existingConfig,
postWriteFailed: settled.postWriteError != null,
noop: settled.committed === false,
},
async () => {
await fenceCommittedMCPMutation({ userId, serverName });
await disconnectLocalMCPServer(userId, serverName);
try {
await cleanupDeletedMCPServerOAuthUsers({
ownerUserId: userId,
serverName,
serverConfig: existingConfig,
snapshot: oauthDeletionSnapshot,
getTokenUserIds,
getUserPrincipals: (candidateUserId) =>
db.getUserPrincipals({ userId: candidateUserId }),
resolveAllowlists: (candidateUserId) =>
registry.resolveAllowlists({ userId: candidateUserId }),
fenceAndDisconnectUser: async (candidateUserId) => {
if (candidateUserId === userId) {
return;
}
await fenceCommittedMCPMutation({ userId: candidateUserId, serverName });
await disconnectLocalMCPServer(candidateUserId, serverName);
},
uninstallOAuthMCP,
});
} catch (error) {
logger.warn(
`[deleteMCPServer] Server ${serverName} was deleted, but OAuth cleanup failed for user ${userId}:`,
error,
);
throw error;
}
},
);
if (settled.postWriteError) {
throw settled.postWriteError;
}
res.status(200).json({ message: 'MCP server deleted successfully' });
} catch (error) {

View file

@ -137,6 +137,7 @@ jest.mock('~/models', () => ({
deleteTokens: jest.fn(),
findPluginAuthsByKeys: jest.fn(),
getRoleByName: jest.fn(),
recordAuditEntry: jest.fn(),
}));
jest.mock('~/server/services/Config', () => ({
@ -4063,6 +4064,63 @@ describe('MCP Routes', () => {
expect(MCPConnection.clearCooldown).toHaveBeenCalledWith(inspectionServerName);
});
it('records mcp_server.created with field names and no secret values', async () => {
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
const validConfig = {
type: 'sse',
url: 'https://mcp-server.example.com/sse',
title: 'Audited Server',
headers: { Authorization: 'Bearer super-secret-token' },
};
mockRegistryInstance.addServer.mockResolvedValue({
serverName: 'audited-server',
config: { ...validConfig, source: 'user' },
});
const response = await request(app).post('/api/mcp/servers').send({ config: validConfig });
expect(response.status).toBe(201);
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
const input = recordAuditEntry.mock.calls[0][0];
expect(input).toEqual(
expect.objectContaining({
action: 'mcp_server.created',
target: { type: 'mcp_server', id: 'audited-server', name: 'Audited Server' },
metadata: { source: 'DB', fields: 'headers,title,type,url' },
}),
);
expect(JSON.stringify(input)).not.toContain('super-secret-token');
});
it('audits a create that committed before the cache refresh failed, then returns 500', async () => {
const { MCPCommittedWriteError } = require('@librechat/api');
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
const validConfig = {
type: 'sse',
url: 'https://mcp-server.example.com/sse',
title: 'Audited Server',
};
mockRegistryInstance.addServer.mockRejectedValueOnce(
new MCPCommittedWriteError(
{ serverName: 'audited-server', config: { ...validConfig, source: 'user' } },
new Error('redis down'),
),
);
const response = await request(app).post('/api/mcp/servers').send({ config: validConfig });
expect(response.status).toBe(500);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.created',
target: { type: 'mcp_server', id: 'audited-server', name: 'Audited Server' },
metadata: expect.objectContaining({ postWrite: 'failed' }),
}),
);
});
it('should reserve config-managed server names when creating MCP server', async () => {
const validConfig = {
type: 'sse',
@ -4585,6 +4643,93 @@ describe('MCP Routes', () => {
);
});
it('records mcp_server.updated with only the changed field names', async () => {
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
const existing = {
type: 'sse',
url: 'https://old.example.com/sse',
title: 'Same Title',
headers: { Authorization: 'Bearer old-secret' },
};
const updated = {
...existing,
url: 'https://new.example.com/sse',
headers: { Authorization: 'Bearer new-secret' },
};
mockRegistryInstance.getServerConfig.mockResolvedValueOnce(existing);
mockRegistryInstance.inspectServerUpdate.mockResolvedValue({ ...updated, source: 'user' });
mockRegistryInstance.commitServerUpdate.mockResolvedValue({ ...updated, source: 'user' });
const response = await request(app)
.patch('/api/mcp/servers/test-server')
.send({ config: updated });
expect(response.status).toBe(200);
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
const input = recordAuditEntry.mock.calls[0][0];
expect(input).toEqual(
expect.objectContaining({
action: 'mcp_server.updated',
target: { type: 'mcp_server', id: 'test-server', name: 'Same Title' },
metadata: { source: 'DB', fields: 'headers,url' },
}),
);
expect(JSON.stringify(input)).not.toContain('secret');
});
it('audits the config this write persisted, not a later overlapping write', async () => {
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
const existing = { type: 'sse', url: 'https://old.example.com/sse', title: 'Same Title' };
const submitted = { ...existing, url: 'https://new.example.com/sse' };
mockRegistryInstance.getServerConfig.mockResolvedValue({
...existing,
url: 'https://other-writer.example.com/sse',
title: 'Other Writer',
});
mockRegistryInstance.getServerConfig.mockResolvedValueOnce(existing);
mockRegistryInstance.inspectServerUpdate.mockResolvedValue({ ...submitted, source: 'user' });
mockRegistryInstance.commitServerUpdate.mockResolvedValue({ ...submitted, source: 'user' });
const response = await request(app)
.patch('/api/mcp/servers/test-server')
.send({ config: submitted });
expect(response.status).toBe(200);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
target: { type: 'mcp_server', id: 'test-server', name: 'Same Title' },
metadata: { source: 'DB', fields: 'url' },
}),
);
});
it('audits an update that committed before the cache refresh failed, then returns 500', async () => {
const { MCPCommittedWriteError } = require('@librechat/api');
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
const existing = { type: 'sse', url: 'https://old.example.com/sse', title: 'Same Title' };
const submitted = { ...existing, url: 'https://new.example.com/sse' };
mockRegistryInstance.getServerConfig.mockResolvedValueOnce(existing);
mockRegistryInstance.inspectServerUpdate.mockResolvedValue({ ...submitted, source: 'user' });
mockRegistryInstance.commitServerUpdate.mockRejectedValueOnce(
new MCPCommittedWriteError({ ...submitted, source: 'user' }, new Error('redis down')),
);
const response = await request(app)
.patch('/api/mcp/servers/test-server')
.send({ config: submitted });
expect(response.status).toBe(500);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.updated',
metadata: { source: 'DB', fields: 'url', postWrite: 'failed' },
}),
);
});
it('should redact secrets from update response', async () => {
const validConfig = {
type: 'sse',
@ -4759,6 +4904,56 @@ describe('MCP Routes', () => {
);
});
it('records mcp_server.deleted only after the removal succeeds', async () => {
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
mockRegistryInstance.removeServer.mockRejectedValueOnce(new Error('Deletion failed'));
await request(app).delete('/api/mcp/servers/audited-server');
expect(recordAuditEntry).not.toHaveBeenCalled();
mockRegistryInstance.removeServer.mockResolvedValue(undefined);
const response = await request(app).delete('/api/mcp/servers/audited-server');
expect(response.status).toBe(200);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.deleted',
target: { type: 'mcp_server', id: 'audited-server' },
metadata: { source: 'DB' },
}),
);
});
it('audits a removal that committed before the cache refresh failed, then returns 500', async () => {
const { MCPCommittedWriteError } = require('@librechat/api');
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
mockRegistryInstance.removeServer.mockRejectedValueOnce(
new MCPCommittedWriteError(undefined, new Error('redis down')),
);
const response = await request(app).delete('/api/mcp/servers/audited-server');
expect(response.status).toBe(500);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.deleted',
metadata: { source: 'DB', postWrite: 'failed' },
}),
);
});
it('does not audit a delete that found nothing to remove', async () => {
const { recordAuditEntry } = require('~/models');
recordAuditEntry.mockClear();
mockRegistryInstance.removeServer.mockResolvedValueOnce(false);
const response = await request(app).delete('/api/mcp/servers/already-gone');
expect(response.status).toBe(200);
expect(recordAuditEntry).not.toHaveBeenCalled();
});
it('should return 500 when registry throws error', async () => {
mockRegistryInstance.removeServer.mockRejectedValue(new Error('Deletion failed'));

View file

@ -1,5 +1,5 @@
const express = require('express');
const { createAdminAuditLogHandlers } = require('@librechat/api');
const { createAdminAuditLogHandlers, isAuditFailClosed } = require('@librechat/api');
const { SystemCapabilities } = require('@librechat/data-schemas');
const { requireCapability } = require('~/server/middleware/roles/capabilities');
const { requireJwtAuth } = require('~/server/middleware');
@ -15,6 +15,8 @@ const handlers = createAdminAuditLogHandlers({
findAuditLogEntry: db.findAuditLogEntry,
streamAuditLogEntries: db.streamAuditLogEntries,
verifyAuditChain: db.verifyAuditChain,
recordAuditEntry: db.recordAuditEntry,
auditFailClosed: isAuditFailClosed(),
});
/**

View file

@ -1,9 +1,10 @@
const express = require('express');
const { createAdminCodeEnvironmentHandlers } = require('@librechat/api');
const { createAdminCodeEnvironmentHandlers, isAuditFailClosed } = require('@librechat/api');
const { SystemCapabilities } = require('@librechat/data-schemas');
const { requireCapability } = require('~/server/middleware/roles/capabilities');
const { getAppConfig } = require('~/server/services/Config');
const { requireJwtAuth } = require('~/server/middleware');
const db = require('~/models');
const router = express.Router();
const requireAdminAccess = requireCapability(SystemCapabilities.ACCESS_ADMIN);
@ -11,7 +12,11 @@ const requireCodeEnvironmentManage = requireCapability(
SystemCapabilities.MANAGE_CODE_ENVIRONMENTS,
{ platformOnly: true },
);
const handlers = createAdminCodeEnvironmentHandlers({ getAppConfig });
const handlers = createAdminCodeEnvironmentHandlers({
getAppConfig,
recordAuditEntry: db.recordAuditEntry,
auditFailClosed: isAuditFailClosed(),
});
router.use(requireJwtAuth, requireAdminAccess, requireCodeEnvironmentManage);
router.post('/:environmentId/pairings', handlers.createPairing);

View file

@ -30,6 +30,7 @@ jest.mock('@librechat/data-schemas', () => ({
jest.mock('@librechat/api', () => ({
createAdminCodeEnvironmentHandlers: jest.fn(() => mockHandlers),
isAuditFailClosed: jest.fn(() => true),
}));
jest.mock('~/server/middleware/roles/capabilities', () => ({
@ -44,6 +45,10 @@ jest.mock('~/server/services/Config', () => ({
getAppConfig: mockGetAppConfig,
}));
jest.mock('~/models', () => ({
recordAuditEntry: jest.fn(),
}));
function createApp() {
delete require.cache[require.resolve('./code')];
const router = require('./code');

View file

@ -24,6 +24,8 @@ const handlers = createAdminGroupsHandlers({
findUsers: db.findUsers,
deleteConfig: db.deleteConfig,
deleteAclEntries: db.deleteAclEntries,
restoreConfig: db.restoreConfig,
findConfigByPrincipal: db.findConfigByPrincipal,
recordAuditEntry: db.recordAuditEntry,
auditFailClosed: process.env.AUDIT_LOG_FAIL_CLOSED === 'true',
});

View file

@ -41,6 +41,7 @@ const handlers = createAdminLangfuseHandlers({
toggleConfigActive: db.toggleConfigActive,
getMessages: db.getMessages,
invalidateConfigCaches,
recordAuditEntry: db.recordAuditEntry,
});
// `configMiddleware` runs last so unauthorized callers are rejected before the

View file

@ -3,7 +3,11 @@ const { createAdminSkillsSyncAccess, createAdminSkillsSyncHandlers } = require('
const { SystemCapabilities } = require('@librechat/data-schemas');
const { hasCapability, requireCapability } = require('~/server/middleware/roles/capabilities');
const { requireJwtAuth } = require('~/server/middleware');
const { upsertSkillSyncCredential, deleteSkillSyncCredential } = require('~/models');
const {
upsertSkillSyncCredential,
deleteSkillSyncCredential,
recordAuditEntry,
} = require('~/models');
const { getGitHubSkillSyncRunnerForRequest } = require('~/server/services/Skills/sync');
const { getAppConfig } = require('~/server/services/Config');
const configMiddleware = require('~/server/middleware/config/app');
@ -20,6 +24,7 @@ const handlers = createAdminSkillsSyncHandlers({
getRunner: getGitHubSkillSyncRunnerForRequest,
upsertCredential: upsertSkillSyncCredential,
deleteCredential: deleteSkillSyncCredential,
recordAuditEntry,
});
router.use(

View file

@ -57,6 +57,7 @@ jest.mock('~/server/services/Config', () => ({
jest.mock('~/models', () => ({
upsertSkillSyncCredential: jest.fn(),
deleteSkillSyncCredential: jest.fn(),
recordAuditEntry: jest.fn(),
}));
jest.mock('~/server/services/Skills/sync', () => ({

View file

@ -0,0 +1,114 @@
import { resolve } from 'node:path';
import { expect, test } from '@playwright/test';
import { run, repoRoot, inOneProject } from './lint.helpers';
/**
* Admin integration changes (MCP servers, Langfuse, audit log reads and
* exports, group deletion) leave an audit trail that never carries secrets or
* searched-for text, and a group is only removed once its permissions are.
* Each probe drives the real handlers from the built @librechat/api with an
* in-memory recorder and prints the result as JSON.
*/
const PREAMBLE = `
const api = require('@librechat/api');
const { Types } = require('mongoose');
const out = (value) => { process.stdout.write('RESULT' + JSON.stringify(value)); process.exit(0); };
const makeRes = () => {
const res = { statusCode: 200, body: undefined, headers: {}, chunks: [] };
res.status = (code) => { res.statusCode = code; return res; };
res.json = (body) => { res.body = body; return res; };
res.setHeader = (key, value) => { res.headers[key] = value; };
res.write = (chunk) => { res.chunks.push(String(chunk)); return true; };
res.end = () => res;
res.once = () => res;
res.on = () => res;
return res;
};
const user = { _id: new Types.ObjectId(), name: 'Admin', email: 'admin@example.com' };
`;
function probe(body: string): unknown {
const result = run(
'node',
[
'-e',
`${PREAMBLE}\n(async () => {\n${body}\n})().catch((e) => { console.error(e); process.exit(1); });`,
],
{
cwd: resolve(repoRoot, 'api'),
},
);
expect(result.status, result.output).toBe(0);
return JSON.parse(result.stdout.slice(result.stdout.indexOf('RESULT') + 'RESULT'.length));
}
test.describe('admin integration auditing', () => {
test.beforeEach(() => inOneProject());
test('an MCP server update is audited by field name without the values @scenario:mcp-server-update-audit-lists-field-names-only', () => {
const entries = probe(`
const entries = [];
const audit = api.createMCPServerAuditor({ recordAuditEntry: async (input) => { entries.push(input); } });
await audit({ user }, {
action: 'mcp_server.updated',
serverName: 'srv',
previous: { title: 'Docs', url: 'https://old.example.com', headers: { Authorization: 'Bearer old-secret' } },
next: { title: 'Docs', url: 'https://old.example.com', headers: { Authorization: 'Bearer new-secret' } },
});
out(entries);
`) as Array<{ action: string; target: { id: string }; metadata: { fields?: string } }>;
expect(entries).toHaveLength(1);
expect(entries[0].action).toBe('mcp_server.updated');
expect(entries[0].target.id).toBe('srv');
expect(entries[0].metadata.fields).toBe('headers');
expect(JSON.stringify(entries)).not.toContain('secret');
});
test('an MCP server deletion is audited as a warning @scenario:mcp-server-delete-audit-is-a-warning', () => {
const entries = probe(`
const entries = [];
const audit = api.createMCPServerAuditor({ recordAuditEntry: async (input) => { entries.push(input); } });
await audit({ user }, { action: 'mcp_server.deleted', serverName: 'gone', previous: { title: 'Gone' } });
out(entries);
`) as Array<{ action: string; severity: string }>;
expect(entries[0].action).toBe('mcp_server.deleted');
expect(entries[0].severity).toBe('warning');
});
test('an audit write failure does not turn a deleted MCP server into an error @scenario:audit-write-failure-keeps-the-completed-change', () => {
const result = probe(`
const audit = api.createMCPServerAuditor({ recordAuditEntry: async () => { throw new Error('db down'); } });
let threw = false;
try {
await audit({ user }, { action: 'mcp_server.created', serverName: 'srv', next: { title: 'Docs' } });
} catch { threw = true; }
out({ threw });
`) as { threw: boolean };
expect(result.threw).toBe(false);
});
test('a group whose cleanup fails is kept with its config restored and reports GROUP_CLEANUP_FAILED @scenario:group-delete-keeps-the-group-when-cleanup-fails', () => {
const result = probe(`
const id = new Types.ObjectId().toString();
let deleted = false;
const restored = [];
const noop = async () => undefined;
const handlers = api.createAdminGroupsHandlers({
listGroups: noop, countGroups: noop, createGroup: noop, updateGroupById: noop,
findGroupById: async () => ({ _id: new Types.ObjectId(id), name: 'G', memberIds: [] }),
deleteGroup: async () => { deleted = true; return { _id: id }; },
restoreConfig: async () => { restored.push('config'); return true; },
deleteConfig: async () => ({ principalType: 'group', principalId: id, overrides: {} }),
deleteAclEntries: async () => { throw new Error('acl store down'); },
});
const res = makeRes();
await handlers.deleteGroup({ params: { id }, query: {}, body: {}, user }, res);
out({ status: res.statusCode, code: res.body && res.body.code, deleted, restored });
`) as { status: number; code?: string; deleted: boolean; restored: string[] };
expect(result.status).toBe(500);
expect(result.code).toBe('GROUP_CLEANUP_FAILED');
expect(result.deleted).toBe(false);
expect(result.restored).toEqual(['config']);
});
});

View file

@ -114,6 +114,7 @@ function createCsvContext(
}),
end: jest.fn(() => {
ended = true;
emitter.emit('finish');
}),
headersSent: false,
status: jest.fn().mockReturnValue({ json: jest.fn() }),
@ -351,6 +352,24 @@ describe('createAdminAuditLogHandlers', () => {
expect(status).toHaveBeenCalledWith(401);
});
it('records audit.read for an integrity verification', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createAdminAuditLogHandlers(createDeps({ recordAuditEntry }));
const { req, res } = createReqRes({
user: { _id: new Types.ObjectId(), role: 'admin' },
});
await handlers.verifyAuditLog(req, res);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'audit.read',
target: { type: 'audit_log' },
metadata: { operation: 'verify' },
}),
);
});
it('returns the verification result scoped to the caller tenant', async () => {
const verification = mockVerification({ ok: false, brokenAt: 2, reason: 'hash mismatch' });
const deps = createDeps({ verifyAuditChain: jest.fn().mockResolvedValue(verification) });
@ -488,4 +507,189 @@ describe('createAdminAuditLogHandlers', () => {
expect(ctx.endCalled()).toBe(true);
});
});
describe('audit entries', () => {
const user = { _id: new Types.ObjectId(), role: 'admin' };
it('records audit.read for a list with filters but not free-text values', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createAdminAuditLogHandlers(createDeps({ recordAuditEntry }));
const { req, res, status } = createReqRes({
user,
query: {
action: ['role.created', 'group.created'],
outcome: 'success',
from: '2025-01-01',
actorQuery: 'alice@example.com',
search: 'secret-term',
capability: 'private@example.com',
targetType: 'user-typed-text',
},
});
await handlers.listAuditLog(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
const input = recordAuditEntry.mock.calls[0][0];
expect(input).toEqual(
expect.objectContaining({
action: 'audit.read',
target: { type: 'audit_log' },
metadata: {
action: 'group.created,role.created',
outcome: 'success',
from: '2025-01-01T00:00:00.000Z',
freeTextFilters: 'search,actorQuery,targetType,capability',
},
}),
);
expect(JSON.stringify(input)).not.toContain('alice@example.com');
expect(JSON.stringify(input)).not.toContain('secret-term');
expect(JSON.stringify(input)).not.toContain('private@example.com');
expect(JSON.stringify(input)).not.toContain('user-typed-text');
});
it('withholds the rows and answers 500 when a fail-closed audit write fails', async () => {
const recordAuditEntry = jest.fn().mockRejectedValue(new Error('audit down'));
const listAuditLogPage = jest
.fn()
.mockResolvedValue({ entries: [mockEntry()], nextCursor: null });
const handlers = createAdminAuditLogHandlers(
createDeps({ recordAuditEntry, auditFailClosed: true, listAuditLogPage }),
);
const { req, res, status, json } = createReqRes({ user });
await handlers.listAuditLog(req, res);
expect(recordAuditEntry).toHaveBeenCalledWith(expect.anything(), { failClosed: true });
expect(status).toHaveBeenCalledWith(500);
expect(JSON.stringify(json.mock.calls)).not.toContain('entries');
});
it('records audit.read for a single entry only when it exists', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const found = createAdminAuditLogHandlers(
createDeps({
recordAuditEntry,
findAuditLogEntry: jest.fn().mockResolvedValue(mockEntry()),
}),
);
const missing = createAdminAuditLogHandlers(createDeps({ recordAuditEntry }));
const a = createReqRes({ user, params: { id: validObjectId } });
await missing.getAuditLogEntry(a.req, a.res);
expect(recordAuditEntry).not.toHaveBeenCalled();
const b = createReqRes({ user, params: { id: validObjectId } });
await found.getAuditLogEntry(b.req, b.res);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'audit.read',
target: { type: 'audit_log', id: validObjectId },
}),
);
});
it('records audit.exported before the first row is written', async () => {
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async () => {
order.push('audit');
});
const deps = createDeps({
recordAuditEntry,
streamAuditLogEntries: jest.fn().mockImplementation(async () => {
order.push('stream');
return { count: 0, truncated: false };
}),
});
const handlers = createAdminAuditLogHandlers(deps);
const ctx = createCsvContext({ user, query: { severity: 'critical' } });
await handlers.exportAuditLogCsv(ctx.req, ctx.res);
expect(order).toEqual(['audit', 'stream', 'audit']);
expect(recordAuditEntry).toHaveBeenNthCalledWith(
1,
expect.objectContaining({
action: 'audit.exported',
outcome: 'pending',
metadata: { format: 'csv', severity: 'critical' },
}),
);
const terminal = recordAuditEntry.mock.calls[recordAuditEntry.mock.calls.length - 1][0];
expect(terminal).toEqual(
expect.objectContaining({
action: 'audit.exported',
metadata: { format: 'csv', severity: 'critical', rows: 0, truncated: false },
}),
);
expect(terminal.outcome).toBeUndefined();
});
it('records a client that disconnects during the final flush as a failed export', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = createDeps({
recordAuditEntry,
streamAuditLogEntries: jest.fn().mockResolvedValue({ count: 0, truncated: false }),
});
const handlers = createAdminAuditLogHandlers(deps);
const ctx = createCsvContext({ user, query: { severity: 'critical' } });
(ctx.res.end as jest.Mock).mockImplementation(() => {
(ctx.res as unknown as EventEmitter).emit('close');
});
await handlers.exportAuditLogCsv(ctx.req, ctx.res);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({
outcome: 'failure',
metadata: expect.objectContaining({ code: 'client_aborted' }),
}),
);
});
it('appends a failure entry when the export fails after it was recorded', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = createDeps({
recordAuditEntry,
streamAuditLogEntries: jest.fn().mockRejectedValue(new Error('cursor failed')),
});
const handlers = createAdminAuditLogHandlers(deps);
const ctx = createCsvContext({ user, query: { severity: 'critical' } });
await handlers.exportAuditLogCsv(ctx.req, ctx.res);
expect(recordAuditEntry).toHaveBeenCalledTimes(2);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({
action: 'audit.exported',
outcome: 'failure',
metadata: { format: 'csv', severity: 'critical', code: 'export_failed' },
}),
);
});
it('does not record an export that fails filter validation', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createAdminAuditLogHandlers(createDeps({ recordAuditEntry }));
const ctx = createCsvContext({ user, query: { action: 'nope.nope' } });
await handlers.exportAuditLogCsv(ctx.req, ctx.res);
expect(recordAuditEntry).not.toHaveBeenCalled();
});
it('blocks the export when failClosed and the audit write fails', async () => {
const recordAuditEntry = jest.fn().mockRejectedValue(new Error('audit down'));
const deps = createDeps({ recordAuditEntry, auditFailClosed: true });
const handlers = createAdminAuditLogHandlers(deps);
const ctx = createCsvContext({ user });
await handlers.exportAuditLogCsv(ctx.req, ctx.res);
expect(deps.streamAuditLogEntries).not.toHaveBeenCalled();
expect(ctx.chunks).toHaveLength(0);
});
});
});

View file

@ -20,8 +20,12 @@ import type {
AuditOutcome,
AuditSeverity,
} from '@librechat/data-schemas';
import type { AuditMetadata } from '@librechat/data-schemas';
import type { Response } from 'express';
import type { ServerRequest } from '~/types/http';
import type { RecordAuditEntry } from './audit';
import { getSafeErrorMetadata } from '~/utils/errors';
import { createAdminAuditEmitter } from './audit';
const FORMULA_PREFIX = /^[=+\-@\t\r]/;
/** UTF-8 BOM, written first so Excel recognizes the encoding on import. Spelled
@ -64,6 +68,9 @@ export interface AdminAuditLogDeps {
tenantId: string | undefined,
options?: { isCancelled?: () => boolean; maxRows?: number },
) => Promise<AuditChainVerification>;
/** Optional audit emission for reads and exports; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
auditFailClosed?: boolean;
}
interface CallerContext {
@ -270,6 +277,32 @@ function parseFilters(query: AuditLogQuery): ParseResult<ParsedFilters> {
};
}
/**
* Describes a filter set for the audit trail without storing what was searched
* for: enum and date filters are recorded as given, free-text filters (which can
* carry names or email addresses) only by name.
*/
function describeFilters(filters: ParsedFilters): AuditMetadata {
const metadata: AuditMetadata = {};
const lists: Array<[string, string[] | undefined]> = [
['category', filters.category],
['action', filters.action],
['outcome', filters.outcome],
['severity', filters.severity],
];
for (const [key, values] of lists) {
if (values?.length) metadata[key] = [...values].sort().join(',');
}
if (filters.actorType) metadata.actorType = filters.actorType;
if (filters.from) metadata.from = filters.from.toISOString();
if (filters.to) metadata.to = filters.to.toISOString();
const freeText = (
['search', 'actorQuery', 'targetQuery', 'targetType', 'capability'] as const
).filter((key) => filters[key] != null);
if (freeText.length > 0) metadata.freeTextFilters = freeText.join(',');
return metadata;
}
export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
listAuditLog: (req: ServerRequest, res: Response) => Promise<Response>;
getAuditLogEntry: (req: ServerRequest, res: Response) => Promise<Response>;
@ -277,6 +310,17 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
exportAuditLogCsv: (req: ServerRequest, res: Response) => Promise<Response | void>;
} {
const { listAuditLogPage, findAuditLogEntry, streamAuditLogEntries, verifyAuditChain } = deps;
const emitAudit = createAdminAuditEmitter({
recordAuditEntry: deps.recordAuditEntry,
failClosed: deps.auditFailClosed,
logPrefix: '[adminAuditLog]',
});
/** Follow-up entries describe an attempt that already happened, so they never fail the caller. */
const emitFollowUp = createAdminAuditEmitter({
recordAuditEntry: deps.recordAuditEntry,
logPrefix: '[adminAuditLog]',
});
async function listAuditLogHandler(req: ServerRequest, res: Response) {
try {
@ -301,9 +345,14 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
cursor: cursor.value,
});
await emitAudit(req, {
action: 'audit.read',
target: { type: 'audit_log' },
metadata: describeFilters(filters.value),
});
return res.status(200).json(page);
} catch (err) {
logger.error('[adminAuditLog] list error:', err);
logger.error('[adminAuditLog] list error:', getSafeErrorMetadata(err));
return res.status(500).json({ error: 'Failed to fetch audit log' });
}
}
@ -320,9 +369,13 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
const entry = await findAuditLogEntry(caller.tenantId, id);
if (!entry) return res.status(404).json({ error: 'Not found' });
await emitAudit(req, {
action: 'audit.read',
target: { type: 'audit_log', id },
});
return res.status(200).json({ entry });
} catch (err) {
logger.error('[adminAuditLog] get error:', err);
logger.error('[adminAuditLog] get error:', getSafeErrorMetadata(err));
return res.status(500).json({ error: 'Failed to fetch audit log entry' });
}
}
@ -346,18 +399,25 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
});
if (clientAborted) return res;
await emitAudit(req, {
action: 'audit.read',
target: { type: 'audit_log' },
metadata: { operation: 'verify' },
});
return res.status(200).json(result);
} finally {
res.removeListener('close', markAborted);
req.removeListener('aborted', markAborted);
}
} catch (err) {
logger.error('[adminAuditLog] verify error:', err);
logger.error('[adminAuditLog] verify error:', getSafeErrorMetadata(err));
return res.status(500).json({ error: 'Failed to verify audit log integrity' });
}
}
async function exportAuditLogCsvHandler(req: ServerRequest, res: Response) {
let exportRecorded = false;
let exportFailed: ((code: string) => Promise<void>) | undefined;
try {
const caller = resolveCaller(req);
if (!caller) return res.status(401).json({ error: 'Authentication required' });
@ -365,11 +425,6 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
const filters = parseFilters(req.query as AuditLogQuery);
if (!filters.ok) return res.status(400).json({ error: filters.error });
const filenameStamp = new Date().toISOString().slice(0, 10);
res.setHeader('Content-Type', 'text/csv; charset=utf-8');
res.setHeader('Content-Disposition', `attachment; filename="audit-log-${filenameStamp}.csv"`);
res.setHeader('Cache-Control', 'no-store');
/**
* The socket-`close` listener is the canonical signal — it fires on client
* TCP RST as well as on graceful end. `req.aborted` (deprecated) is kept as
@ -384,6 +439,27 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
req.once('aborted', markAborted);
const isCancelled = () => clientAborted;
/** Recorded as pending before any row is streamed, so an export is never unaudited; `success` follows once the stream ends. */
const exportEvent = {
action: 'audit.exported' as const,
severity: 'warning' as const,
target: { type: 'audit_log' },
metadata: { format: 'csv', ...describeFilters(filters.value) },
};
await emitAudit(req, { ...exportEvent, outcome: 'pending' });
exportRecorded = true;
exportFailed = (code: string) =>
emitFollowUp(req, {
...exportEvent,
outcome: 'failure',
metadata: { ...exportEvent.metadata, code },
});
const filenameStamp = new Date().toISOString().slice(0, 10);
res.setHeader('Content-Type', 'text/csv; charset=utf-8');
res.setHeader('Content-Disposition', `attachment; filename="audit-log-${filenameStamp}.csv"`);
res.setHeader('Cache-Control', 'no-store');
/**
* Wait for `drain` when the socket buffer is full so we never queue an
* unbounded amount of CSV in Node memory for slow consumers. Race against
@ -411,7 +487,7 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
await writeChunk(formatCsvHeader());
await writeChunk('\r\n');
const { truncated } = await streamAuditLogEntries(
const { truncated, count } = await streamAuditLogEntries(
caller.tenantId,
filters.value,
async (entry) => {
@ -424,7 +500,19 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
res.removeListener('close', markAborted);
req.removeListener('aborted', markAborted);
if (clientAborted) {
await exportFailed('client_aborted');
}
if (!clientAborted) {
/** `end()` only queues the last bytes, so success waits for the response to finish. */
const settled = new Promise<boolean>((resolve) => {
if (res.destroyed) {
resolve(false);
return;
}
res.once('finish', () => resolve(true));
res.once('close', () => resolve(res.writableFinished === true));
});
/**
* `truncated` is true only when rows existed beyond the cap (an exact-cap
* match is reported as complete). Surfacing this is a compliance
@ -441,9 +529,20 @@ export function createAdminAuditLogHandlers(deps: AdminAuditLogDeps): {
);
}
res.end();
if (await settled) {
await emitFollowUp(req, {
...exportEvent,
metadata: { ...exportEvent.metadata, rows: count, truncated },
});
} else {
await exportFailed('client_aborted');
}
}
} catch (err) {
logger.error('[adminAuditLog] export error:', err);
logger.error('[adminAuditLog] export error:', getSafeErrorMetadata(err));
if (exportRecorded) {
await exportFailed?.('export_failed');
}
if (!res.headersSent) {
return res.status(500).json({ error: 'Failed to export audit log' });
}

View file

@ -339,4 +339,194 @@ describe('createAdminCodeEnvironmentHandlers', () => {
expect(response.statusCode).toBe(502);
expect(response.body).toEqual({ error: 'Code API returned an invalid revocation response' });
});
describe('audit entries', () => {
function build(fetchImpl: jest.Mock, recordAuditEntry: jest.Mock) {
return createAdminCodeEnvironmentHandlers({
getAppConfig: jest.fn().mockResolvedValue(config()),
readSecret: jest.fn().mockReturnValue('administrator-bootstrap-token'),
fetchImpl,
recordAuditEntry,
});
}
it('records code_worker.paired without the pairing code', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const fetchImpl = jest.fn().mockResolvedValue(
Response.json({
protocolVersion: 1,
workerId: 'vm-1',
code: 'one-time-code-value-that-is-long',
expiresAt: '2099-08-30T12:00:00.000Z',
}),
);
await build(fetchImpl, recordAuditEntry).createPairing(request(), mockResponse());
expect(recordAuditEntry.mock.calls.map(([entry]) => entry.outcome ?? 'success')).toEqual([
'pending',
'success',
]);
const input = recordAuditEntry.mock.calls[1][0];
expect(input).toEqual(
expect.objectContaining({
action: 'code_worker.paired',
target: { type: 'code_worker', id: 'vm-1', name: 'attached-vm' },
}),
);
expect(input.tenantId).toBeUndefined();
expect(input.metadata.boundTenantId).toEqual(expect.any(String));
expect(JSON.stringify(recordAuditEntry.mock.calls)).not.toContain(
'one-time-code-value-that-is-long',
);
expect(JSON.stringify(recordAuditEntry.mock.calls)).not.toContain(
'administrator-bootstrap-token',
);
});
it('records code_worker.revoked as pending, then success or a failure entry when it is not confirmed', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const ok = jest.fn().mockResolvedValue(Response.json({ protocolVersion: 1, revoked: true }));
const bad = jest
.fn()
.mockResolvedValue(Response.json({ protocolVersion: 1, revoked: false }));
await build(ok, recordAuditEntry).revokeWorker(request(), mockResponse());
expect(recordAuditEntry).toHaveBeenCalledTimes(2);
expect(recordAuditEntry).toHaveBeenNthCalledWith(
1,
expect.objectContaining({
action: 'code_worker.revoked',
outcome: 'pending',
target: { type: 'code_worker', id: 'vm-1', name: 'attached-vm' },
}),
);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({ action: 'code_worker.revoked', outcome: 'success' }),
);
recordAuditEntry.mockClear();
await build(bad, recordAuditEntry).revokeWorker(request(), mockResponse());
expect(recordAuditEntry).toHaveBeenCalledTimes(2);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({
outcome: 'failure',
metadata: { environmentId: 'attached-vm', code: 'invalid_response' },
}),
);
});
it('records the pairing before the bridge call and a failure entry when no code is minted', async () => {
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
order.push(`audit:${input.outcome ?? 'success'}`);
});
const fetchImpl = jest.fn().mockImplementation(async () => {
order.push('bridge');
return new Response('nope', { status: 500 });
});
await build(fetchImpl, recordAuditEntry).createPairing(request(), mockResponse());
expect(order).toEqual(['audit:pending', 'bridge', 'audit:failure']);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({ metadata: expect.objectContaining({ code: 'bridge_rejected' }) }),
);
});
it('sends the revocation without waiting for a slow pending audit write', async () => {
const order: string[] = [];
let release: () => void = () => undefined;
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
if (input.outcome === 'pending') {
await new Promise<void>((resolve) => (release = resolve));
}
order.push(`audit:${input.outcome ?? 'success'}`);
});
const fetchImpl = jest.fn().mockImplementation(async () => {
order.push('bridge');
release();
return Response.json({ protocolVersion: 1, revoked: true });
});
await build(fetchImpl, recordAuditEntry).revokeWorker(request(), mockResponse());
expect(order).toEqual(['bridge', 'audit:pending', 'audit:success']);
});
it('delivers the pairing code before the terminal audit write settles', async () => {
let release: () => void = () => undefined;
let deliveredAtTerminalWrite: unknown;
const response = mockResponse();
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
if (input.outcome === undefined || input.outcome === 'success') {
deliveredAtTerminalWrite = response.body;
await new Promise<void>((resolve) => (release = resolve));
}
});
const fetchImpl = jest.fn().mockResolvedValue(
Response.json({
protocolVersion: 1,
workerId: 'vm-1',
code: 'one-time-code-value-that-is-long',
expiresAt: '2099-08-30T12:00:00.000Z',
}),
);
const pending = build(fetchImpl, recordAuditEntry).createPairing(request(), response);
await new Promise((resolve) => setImmediate(resolve));
release();
await pending;
expect(deliveredAtTerminalWrite).toEqual(expect.objectContaining({ workerId: 'vm-1' }));
});
it('does not pair a worker when the pending audit write fails under fail-closed', async () => {
const fetchImpl = jest.fn();
const response = mockResponse();
await createAdminCodeEnvironmentHandlers({
getAppConfig: jest.fn().mockResolvedValue(config()),
readSecret: jest.fn().mockReturnValue('administrator-bootstrap-token'),
fetchImpl,
recordAuditEntry: jest.fn().mockRejectedValue(new Error('audit down')),
auditFailClosed: true,
})
.createPairing(request(), response)
.catch(() => undefined);
expect(fetchImpl).not.toHaveBeenCalled();
});
it('never blocks a revocation on an audit outage, even when fail-closed', async () => {
const ok = jest.fn().mockResolvedValue(Response.json({ protocolVersion: 1, revoked: true }));
const response = mockResponse();
await createAdminCodeEnvironmentHandlers({
getAppConfig: jest.fn().mockResolvedValue(config()),
readSecret: jest.fn().mockReturnValue('administrator-bootstrap-token'),
fetchImpl: ok,
recordAuditEntry: jest.fn().mockRejectedValue(new Error('audit down')),
auditFailClosed: true,
}).revokeWorker(request(), response);
expect(response.statusCode).toBe(200);
});
it('does not fail the pairing when the audit write fails (fail-open)', async () => {
const recordAuditEntry = jest.fn().mockRejectedValue(new Error('audit down'));
const fetchImpl = jest.fn().mockResolvedValue(
Response.json({
protocolVersion: 1,
workerId: 'vm-1',
code: 'one-time-code-value-that-is-long',
expiresAt: '2099-08-30T12:00:00.000Z',
}),
);
const response = mockResponse();
await build(fetchImpl, recordAuditEntry).createPairing(request(), response);
expect(response.statusCode).toBe(200);
});
});
});

View file

@ -4,8 +4,10 @@ import type { AppConfig } from '@librechat/data-schemas';
import type { Response } from 'express';
import type { GetAppConfigOptions } from '~/app/service';
import type { ServerRequest } from '~/types/http';
import type { RecordAuditEntry } from './audit';
import { getCodeApiTenantId } from '~/auth/codeapi';
import { createAdminAuditEmitter } from './audit';
const CODE_BRIDGE_REQUEST_TIMEOUT_MS = 10_000;
@ -36,6 +38,9 @@ export interface AdminCodeEnvironmentDeps {
resolveTenantId?: (req: ServerRequest) => string;
readSecret?: (name: string) => string | undefined;
fetchImpl?: FetchImpl;
/** Optional audit emission for pairing and revocation; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
auditFailClosed?: boolean;
}
function environmentId(req: ServerRequest): string {
@ -95,6 +100,17 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
} {
const fetchImpl = deps.fetchImpl ?? fetch;
const resolveTenantId = deps.resolveTenantId ?? getCodeApiTenantId;
const emitAudit = createAdminAuditEmitter({
recordAuditEntry: deps.recordAuditEntry,
failClosed: deps.auditFailClosed,
logPrefix: '[adminCode]',
});
/** Revoking a worker removes access, and terminal entries record an outcome that already
* happened, so an audit outage blocks neither. */
const emitRevocationAudit = createAdminAuditEmitter({
recordAuditEntry: deps.recordAuditEntry,
logPrefix: '[adminCode]',
});
const readSecret =
deps.readSecret ??
((name: string) =>
@ -144,6 +160,28 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
} catch {
return res.status(503).json({ error: 'Code environment tenant context is unavailable' });
}
/**
* Recorded as `pending` before the bridge mints the one-time code, so no audit latency
* sits between minting and delivering it and a rejected pairing is never counted as
* granted; `success` is appended once the bridge response validates and `failure`
* otherwise. Workers belong to the deployment's YAML environments, so entries go to
* the platform chain.
*/
const pairingEvent = {
action: 'code_worker.paired' as const,
severity: 'warning' as const,
target: { type: 'code_worker', id: resolved.pairing.workerId, name: resolved.id },
tenantId: undefined,
};
const pairingMetadata = { environmentId: resolved.id, boundTenantId: tenantId };
await emitAudit(req, { ...pairingEvent, outcome: 'pending', metadata: pairingMetadata });
const pairingFailed = async (code: string) => {
await emitRevocationAudit(req, {
...pairingEvent,
outcome: 'failure',
metadata: { ...pairingMetadata, code },
});
};
try {
const response = await fetchImpl(bridgeUrl(resolved.environment, '/bridge/pairings'), {
method: 'POST',
@ -162,6 +200,7 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
signal: AbortSignal.timeout(CODE_BRIDGE_REQUEST_TIMEOUT_MS),
});
if (!response.ok) {
await pairingFailed('bridge_rejected');
return res.status(502).json({
error: 'Code API rejected the pairing request',
upstreamStatus: response.status,
@ -169,16 +208,21 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
}
const payload = (await response.json()) as unknown;
if (!validPairingResponse(payload, resolved.pairing.workerId)) {
await pairingFailed('invalid_response');
return res.status(502).json({ error: 'Code API returned an invalid pairing response' });
}
return res.status(200).json({
/** The code is delivered first: it expires, so audit latency must not sit in front of it. */
res.status(200).json({
environmentId: resolved.id,
workerId: payload.workerId,
code: payload.code,
expiresAt: payload.expiresAt,
});
await emitRevocationAudit(req, { ...pairingEvent, metadata: pairingMetadata });
return res;
} catch (error) {
const timedOut = error instanceof Error && error.name === 'TimeoutError';
await pairingFailed(timedOut ? 'timeout' : 'request_failed');
return res.status(timedOut ? 504 : 502).json({
error: timedOut ? 'Code API pairing request timed out' : 'Code API pairing request failed',
});
@ -188,6 +232,32 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
async function revokeWorker(req: ServerRequest, res: Response): Promise<Response> {
const resolved = await resolve(req, res);
if ('statusCode' in resolved) return resolved;
/**
* Recorded as `pending` without being awaited, so audit latency never delays the
* revocation; later entries wait for it to keep the chain ordered. `success` is
* appended once the bridge acknowledges and `failure` otherwise.
*/
const revokeEvent = {
action: 'code_worker.revoked' as const,
severity: 'warning' as const,
target: { type: 'code_worker', id: resolved.pairing.workerId, name: resolved.id },
tenantId: undefined,
};
const revokeMetadata = { environmentId: resolved.id };
const attempt = emitRevocationAudit(req, {
...revokeEvent,
outcome: 'pending',
metadata: revokeMetadata,
});
const revokeSettled = async (outcome: 'success' | 'failure', code?: string) => {
await attempt;
await emitRevocationAudit(req, {
...revokeEvent,
outcome,
metadata: code ? { ...revokeMetadata, code } : revokeMetadata,
});
};
const revokeFailed = (code: string) => revokeSettled('failure', code);
try {
const workerId = encodeURIComponent(resolved.pairing.workerId);
const response = await fetchImpl(
@ -204,6 +274,7 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
},
);
if (!response.ok) {
await revokeFailed('bridge_rejected');
return res.status(502).json({
error: 'Code API rejected the revocation request',
upstreamStatus: response.status,
@ -211,15 +282,19 @@ export function createAdminCodeEnvironmentHandlers(deps: AdminCodeEnvironmentDep
}
const payload = (await response.json()) as unknown;
if (!validRevocationResponse(payload)) {
await revokeFailed('invalid_response');
return res.status(502).json({ error: 'Code API returned an invalid revocation response' });
}
return res.status(200).json({
res.status(200).json({
environmentId: resolved.id,
workerId: resolved.pairing.workerId,
revoked: true,
});
await revokeSettled('success');
return res;
} catch (error) {
const timedOut = error instanceof Error && error.name === 'TimeoutError';
await revokeFailed(timedOut ? 'timeout' : 'request_failed');
return res.status(timedOut ? 504 : 502).json({
error: timedOut
? 'Code API revocation request timed out'

View file

@ -0,0 +1,19 @@
import type { ServerRequest } from '~/types/http';
import { buildAuditContext } from './context';
describe('buildAuditContext', () => {
it('prefers the server-assigned request id over client headers', () => {
const req = {
requestId: 'server-id',
headers: { 'x-request-id': 'client-id' },
} as unknown as ServerRequest;
expect(buildAuditContext(req).requestId).toBe('server-id');
});
it('falls back to the correlation headers', () => {
const req = { headers: { 'x-correlation-id': 'client-id' } } as unknown as ServerRequest;
expect(buildAuditContext(req).requestId).toBe('client-id');
});
});

View file

@ -8,7 +8,7 @@ function firstHeaderValue(value: string | string[] | undefined): string | undefi
}
/** Extracts forensic request context (IP, user agent, correlation id) for an
* audit record. Fields are undefined when the data isn't available. */
* audit record. The server-assigned request id wins over client-supplied headers. Fields are undefined when the data isn't available. */
export function buildAuditContext(req: ServerRequest): AuditContext {
const headers = req.headers ?? {};
const forwarded = firstHeaderValue(headers['x-forwarded-for']);
@ -16,6 +16,8 @@ export function buildAuditContext(req: ServerRequest): AuditContext {
return {
ip: req.ip || forwardedIp || req.socket?.remoteAddress || undefined,
userAgent: firstHeaderValue(headers['user-agent']),
requestId: firstHeaderValue(headers['x-request-id'] ?? headers['x-correlation-id']),
requestId:
(req as { requestId?: string }).requestId ??
firstHeaderValue(headers['x-request-id'] ?? headers['x-correlation-id']),
};
}

View file

@ -812,7 +812,10 @@ describe('createAdminGroupsHandlers', () => {
describe('deleteGroup', () => {
it('deletes group and returns 200 with id', async () => {
const deps = createDeps({ deleteGroup: jest.fn().mockResolvedValue(mockGroup()) });
const deps = createDeps({
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteGroup: jest.fn().mockResolvedValue(mockGroup()),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status, json } = createReqRes({ params: { id: validId } });
@ -834,8 +837,8 @@ describe('createAdminGroupsHandlers', () => {
expect(json).toHaveBeenCalledWith({ error: 'Invalid group ID format' });
});
it('returns 404 when deleteGroup returns null', async () => {
const deps = createDeps({ deleteGroup: jest.fn().mockResolvedValue(null) });
it('returns 404 and cleans nothing when the group is missing', async () => {
const deps = createDeps({ findGroupById: jest.fn().mockResolvedValue(null) });
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status, json } = createReqRes({ params: { id: validId } });
@ -844,10 +847,12 @@ describe('createAdminGroupsHandlers', () => {
expect(status).toHaveBeenCalledWith(404);
expect(json).toHaveBeenCalledWith({ error: 'Group not found' });
expect(deps.deleteConfig).not.toHaveBeenCalled();
expect(deps.deleteGroup).not.toHaveBeenCalled();
});
it('returns 500 on error', async () => {
const deps = createDeps({
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteGroup: jest.fn().mockRejectedValue(new Error('db down')),
});
const handlers = createAdminGroupsHandlers(deps);
@ -859,9 +864,9 @@ describe('createAdminGroupsHandlers', () => {
expect(json).toHaveBeenCalledWith({ error: 'Failed to delete group' });
});
it('returns 200 even when cascade cleanup partially fails', async () => {
it('returns 500 and keeps the group when cascade cleanup fails', async () => {
const deps = createDeps({
deleteGroup: jest.fn().mockResolvedValue(mockGroup()),
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('cleanup failed')),
});
const handlers = createAdminGroupsHandlers(deps);
@ -869,17 +874,358 @@ describe('createAdminGroupsHandlers', () => {
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(json).toHaveBeenCalledWith({ success: true, id: validId });
expect(deps.deleteConfig).toHaveBeenCalledWith(PrincipalType.GROUP, validId);
expect(deps.deleteAclEntries).toHaveBeenCalledWith({
principalType: PrincipalType.GROUP,
principalId: new Types.ObjectId(validId),
expect(status).toHaveBeenCalledWith(500);
expect(json).toHaveBeenCalledWith(expect.objectContaining({ code: 'GROUP_CLEANUP_FAILED' }));
expect(deps.deleteGroup).not.toHaveBeenCalled();
});
describe('compensation and audit', () => {
const withUser = <T extends { req: ServerRequest }>(reqRes: T): T => {
(reqRes.req as unknown as { user: Partial<IUser> }).user = {
_id: new Types.ObjectId(),
name: 'Admin',
};
return reqRes;
};
const config = { principalType: 'group', principalId: 'p', overrides: {} };
function compensationDeps(overrides: Partial<AdminGroupsDeps> = {}) {
return createDeps({
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteConfig: jest.fn().mockResolvedValue(config),
restoreConfig: jest.fn().mockResolvedValue(true),
...overrides,
});
}
it('restores the config when the ACL removal fails after it', async () => {
const deps = compensationDeps({
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status, json } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(json).toHaveBeenCalledWith(
expect.objectContaining({ code: 'GROUP_CLEANUP_FAILED' }),
);
expect(deps.restoreConfig).toHaveBeenCalledWith(config);
expect(deps.deleteGroup).not.toHaveBeenCalled();
});
it('does not restore the config when a concurrent delete already removed the group', async () => {
const deps = compensationDeps({
findGroupById: jest.fn().mockResolvedValueOnce(mockGroup()).mockResolvedValueOnce(null),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.restoreConfig).not.toHaveBeenCalled();
});
it('restores the config when the group state recheck fails', async () => {
const deps = compensationDeps({
findGroupById: jest
.fn()
.mockResolvedValueOnce(mockGroup())
.mockRejectedValueOnce(new Error('db down')),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.restoreConfig).toHaveBeenCalledWith(config);
});
it('restores the config when the delete fails and the group state cannot be read', async () => {
const deps = compensationDeps({
findGroupById: jest
.fn()
.mockResolvedValueOnce(mockGroup())
.mockRejectedValueOnce(new Error('db down')),
deleteGroup: jest.fn().mockRejectedValue(new Error('delete failed')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.restoreConfig).toHaveBeenCalledWith(config);
});
it('restores the config from a pre-delete snapshot when the removal rejects after committing', async () => {
const deps = compensationDeps({
findConfigByPrincipal: jest.fn().mockResolvedValue(config),
deleteConfig: jest.fn().mockRejectedValue(new Error('ack lost')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.findConfigByPrincipal).toHaveBeenCalledWith('group', validId, {
includeInactive: true,
});
expect(deps.restoreConfig).toHaveBeenCalledWith(config);
expect(deps.deleteGroup).not.toHaveBeenCalled();
});
it('completes the deletion when the config restore itself fails', async () => {
const deps = compensationDeps({
restoreConfig: jest.fn().mockRejectedValue(new Error('restore down')),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(deps.deleteGroup).toHaveBeenCalledWith(validId);
expect(deps.deleteAclEntries).toHaveBeenCalledTimes(2);
expect(status).toHaveBeenCalledWith(200);
});
it('treats a compensating deletion that throws after committing as completed', async () => {
const group = mockGroup();
const deps = compensationDeps({
restoreConfig: jest.fn().mockRejectedValue(new Error('restore down')),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
findGroupById: jest
.fn()
.mockResolvedValueOnce(group)
.mockResolvedValueOnce(group)
.mockResolvedValue(null),
deleteGroup: jest.fn().mockRejectedValue(new Error('cache invalidation failed')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
});
it('reports not found when a concurrent delete won the race during the compensating deletion', async () => {
const deps = compensationDeps({
restoreConfig: jest.fn().mockRejectedValue(new Error('restore down')),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
deleteGroup: jest.fn().mockResolvedValue(null),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(404);
});
it('reports GROUP_RESTORE_FAILED when neither the restore nor the deletion succeeds', async () => {
const deps = compensationDeps({
restoreConfig: jest.fn().mockRejectedValue(new Error('restore down')),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
deleteGroup: jest.fn().mockRejectedValue(new Error('delete down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status, json } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(json).toHaveBeenCalledWith(
expect.objectContaining({ code: 'GROUP_RESTORE_FAILED' }),
);
});
it('restores the config but never the ACL entries when the group delete fails', async () => {
const deps = compensationDeps({
deleteGroup: jest.fn().mockRejectedValue(new Error('db down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.deleteAclEntries).toHaveBeenCalled();
expect(deps.restoreConfig).toHaveBeenCalledWith(config);
});
it('leaves the removed ACL entries removed when the config removal fails', async () => {
const deps = compensationDeps({
deleteConfig: jest.fn().mockRejectedValue(new Error('config down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(500);
expect(deps.restoreConfig).not.toHaveBeenCalled();
expect(deps.deleteGroup).not.toHaveBeenCalled();
});
it('does not restore dependents of a group that is already gone', async () => {
const deps = compensationDeps({
findGroupById: jest.fn().mockResolvedValueOnce(mockGroup()).mockResolvedValue(null),
deleteGroup: jest.fn().mockRejectedValue(new Error('db down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(deps.restoreConfig).not.toHaveBeenCalled();
});
it('records a pending group.deleted before any removal and a failure entry after a failed cleanup', async () => {
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
order.push(`audit:${input.outcome ?? 'success'}`);
});
const deps = compensationDeps({
recordAuditEntry,
deleteConfig: jest.fn().mockImplementation(async () => {
order.push('config');
return config;
}),
deleteAclEntries: jest.fn().mockRejectedValue(new Error('acl down')),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(order).toEqual(['audit:pending', 'config', 'audit:failure']);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({
action: 'group.deleted',
outcome: 'failure',
metadata: { code: 'GROUP_CLEANUP_FAILED' },
}),
);
});
it('appends a not_found failure entry when the request loses the race', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = compensationDeps({
recordAuditEntry,
deleteGroup: jest.fn().mockResolvedValue(null),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(404);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({ outcome: 'failure', metadata: { code: 'not_found' } }),
);
expect(deps.restoreConfig).not.toHaveBeenCalled();
});
it('does not wait for a slow pending audit write before removing dependents', async () => {
let releaseAudit: () => void = () => undefined;
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
if (input.outcome === 'pending') {
await new Promise<void>((resolve) => (releaseAudit = resolve));
}
order.push(`audit:${input.outcome ?? 'success'}`);
});
const deps = compensationDeps({
recordAuditEntry,
deleteGroup: jest.fn().mockImplementation(async () => {
order.push('group');
releaseAudit();
return mockGroup();
}),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(order).toEqual(['group', 'audit:pending', 'audit:success']);
});
it('still appends the terminal outcome when member-removal audits fail under fail-closed', async () => {
const recordAuditEntry = jest.fn().mockImplementation(async (input) => {
if (input.action === 'group.member_removed') {
throw new Error('audit down');
}
});
const group = mockGroup({ memberIds: ['m1'] });
const deps = compensationDeps({
recordAuditEntry,
auditFailClosed: true,
findGroupById: jest.fn().mockResolvedValue(group),
deleteGroup: jest.fn().mockResolvedValue(group),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({ action: 'group.deleted', outcome: 'success' }),
);
});
it('never blocks the delete on an audit outage, even when fail-closed', async () => {
const deps = compensationDeps({
recordAuditEntry: jest.fn().mockRejectedValue(new Error('audit down')),
auditFailClosed: true,
deleteGroup: jest.fn().mockResolvedValue(mockGroup()),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(deps.deleteGroup).toHaveBeenCalled();
});
});
it('removes dependents before the group', async () => {
const order: string[] = [];
const deps = createDeps({
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteConfig: jest.fn().mockImplementation(async () => void order.push('config')),
deleteAclEntries: jest.fn().mockImplementation(async () => void order.push('acl')),
deleteGroup: jest.fn().mockImplementation(async () => {
order.push('group');
return mockGroup();
}),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res } = createReqRes({ params: { id: validId } });
await handlers.deleteGroup(req, res);
expect(order[order.length - 1]).toBe('group');
expect(order).toHaveLength(3);
});
it('cleans up Config and AclEntry on group delete', async () => {
const deps = createDeps({ deleteGroup: jest.fn().mockResolvedValue(mockGroup()) });
const deps = createDeps({
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
deleteGroup: jest.fn().mockResolvedValue(mockGroup()),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = createReqRes({ params: { id: validId } });
@ -1521,16 +1867,20 @@ describe('createAdminGroupsHandlers', () => {
it('records group.deleted', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = createDeps({ recordAuditEntry });
const deps = createDeps({
recordAuditEntry,
findGroupById: jest.fn().mockResolvedValue(mockGroup()),
});
const handlers = createAdminGroupsHandlers(deps);
const { req, res, status } = withUser(createReqRes({ params: { id: validId } }));
await handlers.deleteGroup(req, res);
expect(status).toHaveBeenCalledWith(200);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect(recordAuditEntry).toHaveBeenLastCalledWith(
expect.objectContaining({
action: 'group.deleted',
outcome: 'success',
target: { type: 'group', id: validId, name: 'Test Group' },
}),
);
@ -1540,6 +1890,7 @@ describe('createAdminGroupsHandlers', () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = createDeps({
recordAuditEntry,
findGroupById: jest.fn().mockResolvedValue(mockGroup({ memberIds: ['m1', 'm2'] })),
deleteGroup: jest.fn().mockResolvedValue(mockGroup({ memberIds: ['m1', 'm2'] })),
});
const handlers = createAdminGroupsHandlers(deps);
@ -1629,7 +1980,7 @@ describe('createAdminGroupsHandlers', () => {
it('does not record when the group is missing', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const deps = createDeps({
deleteGroup: jest.fn().mockResolvedValue(null),
findGroupById: jest.fn().mockResolvedValue(null),
recordAuditEntry,
});
const handlers = createAdminGroupsHandlers(deps);

View file

@ -15,6 +15,7 @@ import type { ValidationError } from '~/types/error';
import type { ServerRequest } from '~/types/http';
import type { RecordAuditEntry } from './audit';
import { getSafeErrorMetadata } from '~/utils/errors';
import { cascadeStep, runCascade } from './cascade';
import { createAdminAuditEmitter } from './audit';
import { parsePagination } from './pagination';
@ -88,6 +89,16 @@ export interface AdminGroupsDeps {
principalType: PrincipalType;
principalId: string | Types.ObjectId;
}) => Promise<DeleteResult>;
/** Re-creates a removed config override in full, only if none exists, when the group
* deletion cannot finish. ACL entries are never restored. Resolves to whether it inserted. */
restoreConfig?: (snapshot: Record<string, unknown>) => Promise<boolean>;
/** Reads the override before it is removed, so a rejected removal whose commit was
* lost in transit can still be put back. */
findConfigByPrincipal?: (
principalType: PrincipalType,
principalId: string,
options?: { includeInactive?: boolean },
) => Promise<object | null>;
/** Optional audit emission for group mutations; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
/** When true, a failed audit write surfaces as a 5xx after the mutation has
@ -122,10 +133,16 @@ export function createAdminGroupsHandlers(deps: AdminGroupsDeps): {
findUsers,
deleteConfig,
deleteAclEntries,
restoreConfig,
findConfigByPrincipal,
recordAuditEntry,
auditFailClosed,
} = deps;
const emitRevocationAudit = createAdminAuditEmitter({
recordAuditEntry,
logPrefix: '[adminGroups]',
});
const emitAudit = createAdminAuditEmitter({
recordAuditEntry,
failClosed: auditFailClosed,
@ -355,44 +372,231 @@ export function createAdminGroupsHandlers(deps: AdminGroupsDeps): {
if (!isValidObjectIdString(id)) {
return res.status(400).json({ error: 'Invalid group ID format' });
}
const deleted = await deleteGroup(id);
if (!deleted) {
const existing = await findGroupById(id);
if (!existing) {
return res.status(404).json({ error: 'Group not found' });
}
const target = groupTarget(existing, id);
/**
* The attempt is recorded as `pending` before any removal so a crash mid-cascade
* still leaves a trail, but the write is started without being awaited: deleting a
* group revokes access, so audit latency must not delay the cleanup. Every
* later entry waits for it to settle to keep the chain ordered.
* A `success` or `failure` entry is appended once the outcome is known.
*/
const attempt = emitRevocationAudit(req, {
action: 'group.deleted',
outcome: 'pending',
severity: 'warning',
target,
});
const record = async (outcome: 'success' | 'failure', code?: string) => {
await attempt;
return emitRevocationAudit(req, {
action: 'group.deleted',
outcome,
severity: 'warning',
target,
...(code ? { metadata: { code } } : {}),
});
};
const recordFailure = (code: string) => record('failure', code);
/**
* Config overrides and ACL entries are keyed by group id, so they are
* removed before the group itself: if any removal fails the group stays
* and the admin can retry, instead of leaving orphaned entries that a later
* principal could inherit. Only the config override is put back (insert-only,
* so one written since is kept): removing it can widen what members may do,
* while ACL entries stay removed because that only narrows access and
* re-inserting them could resurrect a grant revoked in the meantime.
* deleteAclEntries is a raw deleteMany wrapper with no type casting.
* grantPermission stores group principalId as ObjectId, so we must
* cast here. deleteConfig normalizes internally.
*/
const cleanupResults = await Promise.allSettled([
deleteConfig(PrincipalType.GROUP, id),
deleteAclEntries({
let restoreFailed = false;
const cascade = await runCascade(
[
cascadeStep(
'config',
() => deleteConfig(PrincipalType.GROUP, id),
restoreConfig
? async (config) => {
if (!config) {
return;
}
const snapshot = (config as { toObject?: () => unknown }).toObject?.() ?? config;
let restored: boolean;
try {
restored = await restoreConfig(snapshot as Record<string, unknown>);
restoreFailed = false;
} catch (error) {
restoreFailed = true;
throw error;
}
if (!restored) {
logger.warn('[adminGroups] config restore skipped, override exists:', id);
}
}
: undefined,
findConfigByPrincipal
? () => findConfigByPrincipal(PrincipalType.GROUP, id, { includeInactive: true })
: undefined,
),
cascadeStep('acl', () =>
deleteAclEntries({
principalType: PrincipalType.GROUP,
principalId: new Types.ObjectId(id),
}),
),
],
'[adminGroups]',
);
/**
* Puts the config override back. When that restore itself fails the group would live on
* without its restriction, so the deletion is completed instead of retaining it.
*/
const compensate = async (): Promise<'restored' | 'stuck' | 'missing' | IGroup> => {
await cascade.compensate();
if (!restoreFailed) {
return 'restored';
}
logger.error('[adminGroups] config restore failed, completing the deletion:', id);
let removed: IGroup | null = null;
try {
removed = await deleteGroup(id);
if (!removed) {
/** Another request deleted the group first; this one removed nothing. */
return 'missing';
}
} catch (error) {
/** The delete may have committed before it threw (post-delete cache work); a confirmed
* absence is a completed deletion. */
const absent = await findGroupById(id).then(
(group) => group == null,
() => false,
);
if (!absent) {
logger.error(
'[adminGroups] group left without its config override:',
id,
getSafeErrorMetadata(error),
);
return 'stuck';
}
removed = existing;
}
/** The group is gone for good, so a leftover ACL entry is orphaned: retry its removal once. */
await deleteAclEntries({
principalType: PrincipalType.GROUP,
principalId: new Types.ObjectId(id),
}),
]);
for (const result of cleanupResults) {
if (result.status === 'rejected') {
logger.error('[adminGroups] cascade cleanup step failed for group:', id, result.reason);
}
}
}).catch((aclError) =>
logger.error(
'[adminGroups] acl cleanup retry failed after group deletion:',
getSafeErrorMetadata(aclError),
),
);
return removed;
};
/** Every member loses group-scoped access, so the affected accounts are recorded (chunked). */
const revoked = [...new Set(deleted.memberIds ?? [])];
for (let i = 0; i < revoked.length; i += AUDIT_MEMBER_CHUNK) {
const chunk = revoked.slice(i, i + AUDIT_MEMBER_CHUNK);
await emitAudit(req, {
action: 'group.member_removed',
severity: 'warning',
target: groupTarget(deleted, id),
metadata: { userIds: chunk.join(','), count: chunk.length, cause: 'group_deleted' },
const recordMemberRemovals = async (group: IGroup) => {
/** Keeps the chain causal: the pending attempt precedes the member entries. */
await attempt;
const revoked = [...new Set(group.memberIds ?? [])];
for (let i = 0; i < revoked.length; i += AUDIT_MEMBER_CHUNK) {
const chunk = revoked.slice(i, i + AUDIT_MEMBER_CHUNK);
await emitRevocationAudit(req, {
action: 'group.member_removed',
severity: 'warning',
target: groupTarget(group, id),
metadata: { userIds: chunk.join(','), count: chunk.length, cause: 'group_deleted' },
});
}
};
const compensatedResponse = async (outcome: 'stuck' | 'missing' | IGroup) => {
if (outcome === 'missing') {
await recordFailure('not_found');
return res.status(404).json({ error: 'Group not found' });
}
if (outcome !== 'stuck') {
await recordMemberRemovals(outcome);
res.status(200).json({ success: true, id });
await record('success');
return res;
}
await recordFailure('GROUP_RESTORE_FAILED');
return res.status(500).json({
error: 'Failed to delete group and its config override could not be restored',
code: 'GROUP_RESTORE_FAILED',
});
};
const cleanupFailedResponse = async () => {
await recordFailure('GROUP_CLEANUP_FAILED');
return res.status(500).json({
error: 'Failed to remove the group permissions and config; the group was not deleted',
code: 'GROUP_CLEANUP_FAILED',
});
};
if (cascade.failed.length > 0) {
/** A concurrent delete may have removed the group meanwhile; restoring its config would
* orphan it. When the lookup itself fails the group state is unknown, and putting the
* override back is the safe side: a leftover override only restricts, a lost one widens. */
const gone = await findGroupById(id).then(
(group) => group == null,
() => false,
);
if (!gone) {
const outcome = await compensate();
if (outcome !== 'restored') {
return compensatedResponse(outcome);
}
}
return cleanupFailedResponse();
}
await emitAudit(req, {
action: 'group.deleted',
severity: 'warning',
target: groupTarget(deleted, id),
});
return res.status(200).json({ success: true, id });
let deleted: IGroup | null;
try {
deleted = await deleteGroup(id);
} catch (error) {
logger.error(
'[adminGroups] deleteGroup failed after cascade cleanup:',
getSafeErrorMetadata(error),
);
/** The delete may have committed before it threw. Restoring dependents for a
* group that is gone would orphan them. */
const lookup = await findGroupById(id).then(
(group) => ({ known: true, group }),
() => ({ known: false, group: null }),
);
if (lookup.known && lookup.group) {
const outcome = await compensate();
if (outcome !== 'restored') {
return compensatedResponse(outcome);
}
await recordFailure('GROUP_CLEANUP_FAILED');
return res.status(500).json({ error: 'Failed to delete group' });
}
if (!lookup.known) {
/** Unknown state favors the restriction: a leftover override narrows access, a lost one widens it. */
const outcome = await compensate();
if (outcome !== 'restored') {
return compensatedResponse(outcome);
}
logger.error('[adminGroups] group state unverifiable after delete failure:', id);
await recordFailure('GROUP_STATE_UNVERIFIABLE');
return res.status(500).json({ error: 'Failed to delete group' });
}
/** Confirmed absent: the deletion did commit. */
deleted = existing;
}
if (!deleted) {
await recordFailure('not_found');
return res.status(404).json({ error: 'Group not found' });
}
await recordMemberRemovals(deleted);
res.status(200).json({ success: true, id });
await record('success');
return res;
} catch (error) {
logger.error('[adminGroups] deleteGroup error:', getSafeErrorMetadata(error));
return res.status(500).json({ error: 'Failed to delete group' });

View file

@ -547,6 +547,69 @@ describe('createAdminLangfuseHandlers', () => {
expect(JSON.stringify(deps.recordConnectionUpdate.mock.calls)).not.toContain('pk-lf-1');
});
it('records langfuse.updated with the changes and never the keys', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const { handlers } = createHandlers({ recordAuditEntry });
const res = mockRes();
await handlers.updateConnection(
mockReq({
body: {
enabled: true,
destination: 'eu',
publicKey: 'pk-lf-1',
secretKey: 'sk-lf-secret',
},
}),
res,
);
expect(res.statusCode).toBe(200);
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
const input = recordAuditEntry.mock.calls[0][0];
expect(input).toEqual(
expect.objectContaining({
action: 'langfuse.updated',
target: { type: 'langfuse', id: 'connection' },
metadata: { changes: 'created', enabled: true, destination: 'eu' },
}),
);
expect(JSON.stringify(input)).not.toContain('sk-lf-secret');
expect(JSON.stringify(input)).not.toContain('pk-lf-1');
});
it('starts cache invalidation before waiting on the audit write', async () => {
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async () => {
order.push('audit');
});
const invalidateConfigCaches = jest.fn().mockImplementation(async () => {
order.push('invalidate');
});
const { handlers } = createHandlers({ recordAuditEntry, invalidateConfigCaches });
await handlers.updateConnection(
mockReq({
body: { enabled: true, destination: 'eu', publicKey: 'pk-lf-1', secretKey: 'sk-lf-2' },
}),
mockRes(),
);
expect(order[0]).toBe('invalidate');
expect(order).toContain('audit');
});
it('records nothing when the update is rejected', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const { handlers } = createHandlers({ recordAuditEntry });
const res = mockRes();
await handlers.updateConnection(mockReq({ body: { enabled: true } }), res);
expect(res.statusCode).toBe(400);
expect(recordAuditEntry).not.toHaveBeenCalled();
});
it('requires a new secret when connection fields change', async () => {
const { handlers, deps } = createHandlers({
findConfigByPrincipal: jest
@ -771,6 +834,92 @@ describe('createAdminLangfuseHandlers', () => {
});
});
describe('updateConnection activation failure', () => {
it('still audits and invalidates caches when activation fails after the patch committed', async () => {
const inactive = {
...baseConfigDoc({
enabled: true,
destination: 'eu',
publicKey: 'pk-lf-1',
secretKey: encryptV3('sk-lf-secret'),
}),
isActive: false,
};
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const invalidateConfigCaches = jest.fn().mockResolvedValue(undefined);
const { handlers } = createHandlers({
findConfigByPrincipal: jest.fn().mockResolvedValue(inactive),
patchConfigFields: jest.fn().mockResolvedValue(inactive),
toggleConfigActive: jest.fn().mockRejectedValue(new Error('toggle failed')),
recordAuditEntry,
invalidateConfigCaches,
});
const res = mockRes();
await handlers.updateConnection(
mockReq({
body: {
enabled: true,
destination: 'eu',
publicKey: 'pk-lf-1',
secretKey: 'sk-lf-secret',
},
}),
res,
);
expect(res.statusCode).toBe(500);
expect(invalidateConfigCaches).toHaveBeenCalled();
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'langfuse.updated',
metadata: expect.objectContaining({ enabled: false, activation: 'failed' }),
}),
);
});
});
describe('updateConnection activation returning null', () => {
it('treats a vanished config as a failed activation', async () => {
const inactive = {
...baseConfigDoc({
enabled: true,
destination: 'eu',
publicKey: 'pk-lf-1',
secretKey: encryptV3('sk-lf-secret'),
}),
isActive: false,
};
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const { handlers } = createHandlers({
findConfigByPrincipal: jest.fn().mockResolvedValue(inactive),
patchConfigFields: jest.fn().mockResolvedValue(inactive),
toggleConfigActive: jest.fn().mockResolvedValue(null),
recordAuditEntry,
});
const res = mockRes();
await handlers.updateConnection(
mockReq({
body: {
enabled: true,
destination: 'eu',
publicKey: 'pk-lf-1',
secretKey: 'sk-lf-secret',
},
}),
res,
);
expect(res.statusCode).toBe(500);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
metadata: expect.objectContaining({ enabled: false, activation: 'failed' }),
}),
);
});
});
describe('testConnection', () => {
it('requires destination and publicKey', async () => {
const { handlers } = createHandlers();

View file

@ -14,6 +14,7 @@ import type { Types, ClientSession } from 'mongoose';
import type { Response } from 'express';
import type { LangfuseTenantDestination } from '~/langfuse/tenantDestinations';
import type { ServerRequest } from '~/types/http';
import type { RecordAuditEntry } from './audit';
import {
getLangfuseTenantDestinations,
resolveLangfuseTenantDestination,
@ -23,6 +24,7 @@ import { decryptConfigSecret, encryptConfigSecretFields } from './secrets';
import { scopeHeadersToDestination } from '~/langfuse/destinations';
import { isLangfuseConnectionAvailable } from '~/langfuse/policy';
import { resolveLangfuseSession } from '~/langfuse/session';
import { createAdminAuditEmitter } from './audit';
import { mergeHeaders } from '~/utils/headers';
const DEFAULT_PRIORITY = 10;
@ -46,7 +48,7 @@ export interface LangfuseConnectionEvent {
destination?: string;
change: LangfuseConnectionChange;
changes: LangfuseConnectionChange[];
verification_result: 'skipped' | 'success';
verification_result: 'skipped' | 'success' | 'failed';
}
export interface AdminLangfuseDeps {
@ -73,6 +75,9 @@ export interface AdminLangfuseDeps {
getMessages: MessageMethods['getMessages'];
invalidateConfigCaches?: (tenantId?: string) => Promise<void>;
recordConnectionUpdate?: (event: LangfuseConnectionEvent) => void;
/** Optional audit emission for connection updates; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
auditFailClosed?: boolean;
}
function getTenantId(req: ServerRequest): string | undefined {
@ -294,7 +299,14 @@ export function createAdminLangfuseHandlers(deps: AdminLangfuseDeps): {
invalidateConfigCaches,
recordConnectionUpdate = (event) =>
logger.info({ message: '[adminLangfuse] Connection updated', ...event }),
recordAuditEntry,
auditFailClosed,
} = deps;
const emitAudit = createAdminAuditEmitter({
recordAuditEntry,
failClosed: auditFailClosed,
logPrefix: '[adminLangfuse]',
});
function findBaseConfig(options?: { includeInactive?: boolean }): Promise<IConfig | null> {
return options
@ -443,8 +455,25 @@ export function createAdminLangfuseHandlers(deps: AdminLangfuseDeps): {
encryptConfigSecretFields(fields),
existing?.priority ?? DEFAULT_PRIORITY,
);
/** The patch has committed by now, so a failed activation must not skip the cache
* invalidation or the audit entry; it is rethrown once both have started. */
let activationError: unknown;
if (updated?.isActive === false) {
updated = await toggleConfigActive(PrincipalType.ROLE, BASE_CONFIG_PRINCIPAL_ID, true);
try {
const activated = await toggleConfigActive(
PrincipalType.ROLE,
BASE_CONFIG_PRINCIPAL_ID,
true,
);
if (activated) {
updated = activated;
} else {
/** A null result means the config vanished meanwhile (a concurrent delete). */
activationError = new Error('Langfuse base config no longer exists');
}
} catch (error) {
activationError = error;
}
}
const status = buildStatus(updated ?? existing);
@ -455,21 +484,42 @@ export function createAdminLangfuseHandlers(deps: AdminLangfuseDeps): {
publicKey,
secretKey,
);
let verificationResult: LangfuseConnectionEvent['verification_result'] = 'skipped';
if (activationError) {
verificationResult = 'failed';
} else if (connectionChanged) {
verificationResult = 'success';
}
recordConnectionUpdate({
event_name: 'librechat.langfuse.connection.changed',
tenant_id: getTenantId(req),
configured: status.configured,
enabled: status.enabled,
enabled: activationError ? false : status.enabled,
destination: status.destination,
change: changes[0],
changes,
verification_result: connectionChanged ? 'success' : 'skipped',
verification_result: verificationResult,
});
invalidateConfigCaches?.(getTenantId(req))?.catch((err) =>
logger.error('[adminLangfuse] Cache invalidation failed after update:', err),
);
/** Recorded after the cache invalidation starts so audit latency never delays it. Records which parts changed, never the keys themselves. */
await emitAudit(req, {
action: 'langfuse.updated',
severity: 'warning',
target: { type: 'langfuse', id: 'connection' },
metadata: {
changes: [...changes].sort().join(','),
enabled: activationError ? false : status.enabled,
...(activationError ? { activation: 'failed' } : {}),
...(status.destination ? { destination: status.destination } : {}),
},
});
if (activationError) {
throw activationError;
}
return res.status(200).json(status);
} catch (error) {
logger.error('[adminLangfuse] updateConnection error:', error);

View file

@ -576,3 +576,67 @@ describe('createAdminSkillsSyncAccess', () => {
expect(next).toHaveBeenCalledTimes(1);
});
});
describe('skill sync credential audit entries', () => {
function build(overrides: Record<string, jest.Mock> = {}) {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createAdminSkillsSyncHandlers({
runner: { getStatus: jest.fn(), runOnce: jest.fn() } as never,
upsertCredential: jest.fn().mockResolvedValue({
provider: 'github',
credentialKey: 'github-skills-prod',
credentialPresent: true,
tokenFingerprint: 'abcd',
}),
deleteCredential: jest.fn().mockResolvedValue({ deleted: true }),
recordAuditEntry,
...overrides,
});
return { handlers, recordAuditEntry };
}
const req = (extra: Record<string, unknown> = {}) =>
({
user: { id: 'admin-1', name: 'Admin', tenantId: 'tenant-a' },
params: { credentialKey: 'github-skills-prod' },
body: { token: 'ghp_supersecrettoken' },
headers: {},
...extra,
}) as never;
it('records skill_credential.set on the platform chain without the token', async () => {
const { handlers, recordAuditEntry } = build();
await handlers.setCredential(req(), createResponse());
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
const input = recordAuditEntry.mock.calls[0][0];
expect(input).toEqual(
expect.objectContaining({
action: 'skill_credential.set',
target: { type: 'skill_credential', id: 'github-skills-prod', name: 'github' },
tenantId: undefined,
}),
);
expect(JSON.stringify(input)).not.toContain('ghp_supersecrettoken');
});
it('records skill_credential.deleted only when something was deleted', async () => {
const missing = build({ deleteCredential: jest.fn().mockResolvedValue({ deleted: false }) });
await missing.handlers.deleteCredential(req(), createResponse());
expect(missing.recordAuditEntry).not.toHaveBeenCalled();
const { handlers, recordAuditEntry } = build();
await handlers.deleteCredential(req(), createResponse());
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({ action: 'skill_credential.deleted' }),
);
});
it('records nothing for an invalid credential key', async () => {
const { handlers, recordAuditEntry } = build();
await handlers.setCredential(req({ params: { credentialKey: 'bad key!' } }), createResponse());
expect(recordAuditEntry).not.toHaveBeenCalled();
});
});

View file

@ -17,6 +17,9 @@ import type {
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import type { Types } from 'mongoose';
import type { GitHubSkillSyncRunner } from '~/skills/sync';
import type { ServerRequest } from '~/types/http';
import type { RecordAuditEntry } from './audit';
import { createAdminAuditEmitter } from './audit';
export type AdminSkillsRequest = Request & {
user?: {
@ -61,6 +64,9 @@ export type AdminSkillSyncDeps = {
provider: SkillSyncProvider,
credentialKey: string,
) => Promise<{ deleted: boolean }>;
/** Optional audit emission for credential changes; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
auditFailClosed?: boolean;
};
export type AdminSkillSyncAccessDeps = {
@ -359,6 +365,12 @@ export function createAdminSkillsSyncAccess(deps: AdminSkillSyncAccessDeps): Adm
}
export function createAdminSkillsSyncHandlers(deps: AdminSkillSyncDeps): AdminSkillsSyncHandlers {
const emitAudit = createAdminAuditEmitter({
recordAuditEntry: deps.recordAuditEntry,
failClosed: deps.auditFailClosed,
logPrefix: '[adminSkills]',
});
function getRunner(req: Request): GitHubSkillSyncRunner {
const runner = deps.getRunner?.(req) ?? deps.runner;
if (!runner) {
@ -411,6 +423,13 @@ export function createAdminSkillsSyncHandlers(deps: AdminSkillSyncDeps): AdminSk
token: token.trim(),
userId: getUserObjectId(req),
});
/** Credentials are platform-level, so entries go to the platform chain. */
await emitAudit(req as unknown as ServerRequest, {
action: 'skill_credential.set',
severity: 'warning',
target: { type: 'skill_credential', id: credentialKey, name: 'github' },
tenantId: undefined,
});
return res.status(200).json(serializeCredential(credential));
}
@ -420,6 +439,14 @@ export function createAdminSkillsSyncHandlers(deps: AdminSkillSyncDeps): AdminSk
return res.status(400).json({ error: 'Invalid credential key' });
}
const result = await deps.deleteCredential('github', credentialKey);
if (result.deleted) {
await emitAudit(req as unknown as ServerRequest, {
action: 'skill_credential.deleted',
severity: 'warning',
target: { type: 'skill_credential', id: credentialKey, name: 'github' },
tenantId: undefined,
});
}
return res.status(200).json({ credentialKey, deleted: result.deleted });
}

View file

@ -397,6 +397,33 @@ describe('passkey registration provider enforcement', () => {
expect(await methods.countPasskeysByUser(user._id.toString())).toBe(0);
});
it('audits a credential that survives a failed rollback', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const countPasskeysByUser = jest
.fn(methods.countPasskeysByUser)
.mockResolvedValueOnce(0)
.mockRejectedValueOnce(new Error('Count unavailable'));
const deletePasskey = jest.fn().mockRejectedValue(new Error('Delete unavailable'));
const handlers = createPasskeyHandlers(
buildDeps({ countPasskeysByUser, deletePasskey, recordAuditEntry }),
);
const user = await createUser();
const res = buildRes();
await handlers.registerPasskeyVerify(
authedReq(user, { credential: attestation, password: PASSWORD }),
res,
);
expect(res.status).toHaveBeenCalledWith(500);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'passkey.registered',
metadata: expect.objectContaining({ rollback: 'failed' }),
}),
);
});
it('rolls back a credential that wins the enrollment race past the cap', async () => {
/** Both verifies observe the pre-enrollment count, as two concurrent
* ceremonies for the same account do, so both pass the early check. */
@ -474,6 +501,45 @@ describe('passkey registration provider enforcement', () => {
});
});
it('records passkey.registered with the user as actor', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createPasskeyHandlers(buildDeps({ recordAuditEntry }));
const user = await createUser();
const res = buildRes();
await handlers.registerPasskeyVerify(
authedReq(user, { credential: attestation, name: 'Work laptop', password: PASSWORD }),
res,
);
expect(res.status).toHaveBeenCalledWith(201);
const [stored] = await methods.findPasskeysByUser(user._id.toString());
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'passkey.registered',
actor: expect.objectContaining({ type: 'user', id: user._id.toString() }),
target: { type: 'passkey', id: stored.id, name: 'Work laptop' },
metadata: { deviceType: 'singleDevice', backedUp: false },
}),
);
});
it('does not record passkey.registered when registration is rejected', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createPasskeyHandlers(buildDeps({ recordAuditEntry }));
const user = await createUser();
const res = buildRes();
await handlers.registerPasskeyVerify(
authedReq(user, { credential: attestation, name: 'Key', password: 'wrong' }),
res,
);
expect(res.status).toHaveBeenCalledWith(403);
expect(recordAuditEntry).not.toHaveBeenCalled();
});
it('names an unnamed credential after its transports', async () => {
const handlers = createPasskeyHandlers(buildDeps());
const user = await createUser();
@ -710,6 +776,33 @@ describe('passkey rename', () => {
expect(res.json).toHaveBeenCalledWith({ message: 'Name is required' });
});
it('records passkey.removed only when a credential was deleted', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const handlers = createPasskeyHandlers(buildDeps({ recordAuditEntry }));
const owner = await createUser();
const other = await createUser();
const passkey = await createStoredPasskey(owner);
await handlers.removePasskey(
authedReq(other, { password: PASSWORD }, { passkeyId: passkey.id }),
buildRes(),
);
expect(recordAuditEntry).not.toHaveBeenCalled();
await handlers.removePasskey(
authedReq(owner, { password: PASSWORD }, { passkeyId: passkey.id }),
buildRes(),
);
expect(recordAuditEntry).toHaveBeenCalledTimes(1);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'passkey.removed',
actor: expect.objectContaining({ type: 'user', id: owner._id.toString() }),
target: { type: 'passkey', id: passkey.id },
}),
);
});
it('answers 404 for a credential the caller does not own', async () => {
const handlers = createPasskeyHandlers(buildDeps());
const owner = await createUser();

View file

@ -7,6 +7,8 @@ import type { AppConfigUserLike, GetAppConfigOptions } from '~/app/service';
import type { PasskeyConfig, PasskeyChallengeStore } from '~/auth/passkey';
import type { ComparePasswordDeps } from '~/auth/password';
import type { UserDocumentId } from '~/auth/verification';
import type { RecordAuditEntry } from '~/admin/audit';
import type { ServerRequest } from '~/types/http';
import {
getPasskeyConfig,
isPasskeyEnabled,
@ -18,6 +20,7 @@ import {
createPasskeyAuthenticationOptions,
} from '~/auth/passkey';
import { grandfatherLegacyEmailVerification } from '~/auth/verification';
import { createAdminAuditEmitter } from '~/admin/audit';
import { resolveStrictAppConfig } from '~/app/service';
import { comparePassword } from '~/auth/password';
import { isEnabled } from '~/utils';
@ -121,6 +124,9 @@ export interface PasskeyHandlersDeps {
/** Resolves effective configuration for the authenticated principal. */
getAppConfig?: (options: GetAppConfigOptions) => Promise<AppConfig | undefined>;
compare: ComparePasswordDeps['compare'];
/** Optional audit emission for enrolment and removal; fail-open unless `auditFailClosed`. */
recordAuditEntry?: RecordAuditEntry;
auditFailClosed?: boolean;
/** The login ban middleware; reports an internal failure through `next(err)`. */
checkBan: (
req: PasskeyRequest<AuthenticationBody>,
@ -250,8 +256,16 @@ export function createPasskeyHandlers(deps: PasskeyHandlersDeps): PasskeyHandler
getAppConfig,
countPasskeysByUser,
findPasskeyByCredentialId,
recordAuditEntry,
auditFailClosed,
} = deps;
const emitAudit = createAdminAuditEmitter({
recordAuditEntry,
failClosed: auditFailClosed,
logPrefix: '[passkey]',
});
const getChallengeStore = (): PasskeyChallengeStore =>
createPasskeyChallengeStore(getChallengeCache());
@ -446,18 +460,35 @@ export function createPasskeyHandlers(deps: PasskeyHandlersDeps): PasskeyHandler
* the write and rolling this credential back keeps the stored set within
* the cap without a transaction, at the cost of rejecting the whole race.
*/
const auditRegistered = (metadata: Record<string, unknown> = {}) =>
emitAudit(req as unknown as ServerRequest, {
action: 'passkey.registered',
severity: 'warning',
target: { type: 'passkey', id: passkey.id, name: passkey.name },
metadata: { deviceType: verified.deviceType, backedUp: verified.backedUp, ...metadata },
});
/** A credential that survives a failed rollback is a usable factor, so it is audited. */
const rollBack = async () => {
try {
await deletePasskey(passkey.id, req.user.id);
} catch (rollbackError) {
await auditRegistered({ rollback: 'failed' });
throw rollbackError;
}
};
let countAfterWrite: number;
try {
countAfterWrite = await countPasskeysByUser(req.user.id);
} catch (error) {
await deletePasskey(passkey.id, req.user.id);
await rollBack();
throw error;
}
if (countAfterWrite > maxPasskeys) {
await deletePasskey(passkey.id, req.user.id);
await rollBack();
return res.status(409).json({ message: 'Passkey limit reached' });
}
await auditRegistered();
return res.status(201).json({ passkey: serializePasskey(passkey) });
} catch (err) {
if (isDuplicateKeyError(err as MongoWriteError)) {
@ -529,6 +560,11 @@ export function createPasskeyHandlers(deps: PasskeyHandlersDeps): PasskeyHandler
if (!result?.deletedCount) {
return res.status(404).json({ message: 'Passkey not found' });
}
await emitAudit(req as unknown as ServerRequest, {
action: 'passkey.removed',
severity: 'warning',
target: { type: 'passkey', id: req.params.passkeyId },
});
return res.status(200).json({ message: 'Passkey deleted' });
} catch (err) {
logger.error('[removePasskey]', err);

View file

@ -29,6 +29,7 @@ export * from './mcp/oauth';
export * from './mcp/auth';
export * from './mcp/zod';
export * from './mcp/errors';
export * from './mcp/audit';
export * from './mcp/loading';
export * from './mcp/openid';
export * from './mcp/cache';

View file

@ -0,0 +1,193 @@
import type { ServerRequest } from '~/types/http';
import { createMCPServerAuditor, settleMCPCommit, diffMCPServerFields } from './audit';
import { MCPCommittedWriteError } from '~/mcp/errors';
const req = { user: { id: 'user-1', name: 'Alice' }, headers: {} } as unknown as ServerRequest;
describe('diffMCPServerFields', () => {
it('returns sorted names of changed user-input fields only', () => {
expect(
diffMCPServerFields(
{ url: 'https://a', title: 'T', headers: { A: '1' }, dbId: 'x' },
{ url: 'https://b', title: 'T', headers: { A: '2' }, dbId: 'y' },
),
).toEqual(['headers', 'url']);
});
it('ignores key order inside record fields', () => {
expect(
diffMCPServerFields({ headers: { A: '1', B: '2' } }, { headers: { B: '2', A: '1' } }),
).toEqual([]);
});
it('treats a missing side as all-new', () => {
expect(diffMCPServerFields(undefined, { type: 'sse', url: 'https://a' })).toEqual([
'type',
'url',
]);
});
});
describe('createMCPServerAuditor', () => {
it('lists no fields for a delete', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const audit = createMCPServerAuditor({ recordAuditEntry });
await audit(req, {
action: 'mcp_server.deleted',
serverName: 'srv',
previous: { url: 'https://a', title: 'Srv' },
});
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.deleted',
severity: 'warning',
target: { type: 'mcp_server', id: 'srv', name: 'Srv' },
metadata: { source: 'DB' },
}),
);
});
it('does not record a preserved OAuth secret as a change', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const audit = createMCPServerAuditor({ recordAuditEntry });
await audit(req, {
action: 'mcp_server.updated',
serverName: 'srv',
previous: { title: 'Old', oauth: { client_id: 'id', client_secret: 'stored' } },
next: { title: 'New', oauth: { client_id: 'id' } },
});
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({ metadata: { source: 'DB', fields: 'title' } }),
);
});
it('does not record omitted optional OAuth binding fields as a change', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const audit = createMCPServerAuditor({ recordAuditEntry });
await audit(req, {
action: 'mcp_server.updated',
serverName: 'srv',
previous: {
title: 'Old',
oauth: {
client_id: 'id',
client_secret: 'stored',
revocation_endpoint: 'https://idp.example/revoke',
token_endpoint_auth_methods_supported: ['client_secret_basic'],
},
},
next: { title: 'New', oauth: { client_id: 'id' } },
});
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({ metadata: { source: 'DB', fields: 'title' } }),
);
});
it('diffs against the persisted config, not the submission', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const audit = createMCPServerAuditor({ recordAuditEntry });
const headers = { Authorization: 'Bearer {{MCP_API_KEY_srv}}' };
await audit(req, {
action: 'mcp_server.updated',
serverName: 'srv',
previous: { title: 'Old', headers },
next: { title: 'New', headers },
});
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({ metadata: { source: 'DB', fields: 'title' } }),
);
});
it('marks the entry when the cache refresh failed after the commit', async () => {
const recordAuditEntry = jest.fn().mockResolvedValue(undefined);
const audit = createMCPServerAuditor({ recordAuditEntry });
await audit(req, {
action: 'mcp_server.deleted',
serverName: 'srv',
previous: { title: 'Old' },
postWriteFailed: true,
});
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({
action: 'mcp_server.deleted',
metadata: { source: 'DB', postWrite: 'failed' },
}),
);
});
describe('MCPCommittedWriteError', () => {
it('keeps the persisted config out of serialized and logged errors', () => {
const error = new MCPCommittedWriteError(
{ oauth: { client_secret: 'super-secret' } },
new Error('redis down'),
);
expect(error.committed).toEqual({ oauth: { client_secret: 'super-secret' } });
expect(JSON.stringify(error)).not.toContain('super-secret');
expect(Object.keys(error)).not.toContain('committed');
expect(Object.keys(error)).not.toContain('cause');
expect(error.cause).toBeInstanceOf(Error);
});
});
describe('settleMCPCommit', () => {
it('returns the committed result when the write succeeds', async () => {
await expect(settleMCPCommit(async () => ({ title: 'x' }))).resolves.toEqual({
committed: { title: 'x' },
});
});
it('returns the committed result together with an invalidation failure', async () => {
const failure = new MCPCommittedWriteError({ title: 'x' }, new Error('redis down'));
const settled = await settleMCPCommit(async () => {
throw failure;
});
expect(settled.committed).toEqual({ title: 'x' });
expect(settled.postWriteError).toBe(failure);
});
it('rethrows any other failure as an uncommitted write', async () => {
await expect(
settleMCPCommit(async () => {
throw new Error('write failed');
}),
).rejects.toThrow('write failed');
});
});
it('runs the post-commit steps before the audit write and still audits when they throw', async () => {
const order: string[] = [];
const recordAuditEntry = jest.fn().mockImplementation(async () => void order.push('audit'));
const audit = createMCPServerAuditor({ recordAuditEntry });
await expect(
audit(req, { action: 'mcp_server.deleted', serverName: 'srv' }, async () => {
order.push('fence');
throw new Error('fence failed');
}),
).rejects.toThrow('fence failed');
expect(order).toEqual(['fence', 'audit']);
expect(recordAuditEntry).toHaveBeenCalledWith(
expect.objectContaining({ metadata: { source: 'DB', postWrite: 'failed' } }),
);
});
it('is a no-op without a recorder', async () => {
await expect(
createMCPServerAuditor({})(req, { action: 'mcp_server.created', serverName: 'srv' }),
).resolves.toBeUndefined();
});
});

View file

@ -0,0 +1,165 @@
import { MCP_USER_INPUT_FIELDS } from 'librechat-data-provider';
import type { AdminAuditEmitter, RecordAuditEntry } from '~/admin/audit';
import { createAdminAuditEmitter } from '~/admin/audit';
import { MCPCommittedWriteError } from '~/mcp/errors';
type MCPServerAction = 'mcp_server.created' | 'mcp_server.updated' | 'mcp_server.deleted';
type ServerConfigLike = Record<string, unknown> | null | undefined;
export interface MCPServerAuditEvent {
action: MCPServerAction;
serverName: string;
/** Config before the change; omitted on create. */
previous?: ServerConfigLike;
/** Config after the change; omitted on delete. */
next?: ServerConfigLike;
/** The write committed but refreshing the read caches failed afterwards. */
postWriteFailed?: boolean;
/** The write found nothing to change (for example a delete that lost a race), so no entry is recorded. */
noop?: boolean;
}
/** JSON with object keys sorted, so entry order never reads as a change. */
function stableStringify(value: unknown): string {
return JSON.stringify(value, (_key, val) =>
val && typeof val === 'object' && !Array.isArray(val)
? Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b)))
: val,
);
}
/**
* Lists the user-input fields that differ between two server configs. Only
* field names are returned, so headers, env values and credentials never reach
* the audit trail.
*/
export function diffMCPServerFields(previous: ServerConfigLike, next: ServerConfigLike): string[] {
const changed: string[] = [];
for (const field of MCP_USER_INPUT_FIELDS) {
const before = previous?.[field];
const after = next?.[field];
if (before === undefined && after === undefined) {
continue;
}
if (stableStringify(before) !== stableStringify(after)) {
changed.push(field);
}
}
return changed.sort();
}
/** Optional OAuth fields the data layer also keeps when an update omits them alongside the secret. */
const OMITTED_OAUTH_BINDING_FIELDS = [
'token_endpoint_auth_methods_supported',
'revocation_endpoint',
'revocation_endpoint_auth_methods_supported',
] as const;
/**
* An update that omits `oauth.client_secret` keeps the stored one, so the secret
* is carried onto the incoming config before comparing; otherwise a title-only
* edit would be recorded as an oauth change.
*/
function withPreservedOAuthSecret(
previous: ServerConfigLike,
next: ServerConfigLike,
): ServerConfigLike {
const before = previous?.oauth as Record<string, unknown> | undefined;
const after = next?.oauth as Record<string, unknown> | undefined;
if (!next || !before?.client_secret || !after || after.client_secret) {
return next;
}
const carried = { ...after, client_secret: before.client_secret } as Record<string, unknown>;
for (const field of OMITTED_OAUTH_BINDING_FIELDS) {
if (carried[field] === undefined && before[field] !== undefined) {
carried[field] = before[field];
}
}
return { ...next, oauth: carried };
}
function serverTitle(config: ServerConfigLike): string | undefined {
const title = config?.title;
return typeof title === 'string' && title.length > 0 ? title : undefined;
}
/**
* Builds the MCP server audit writer. Fail-open unless `failClosed` is set,
* matching the admin handler factories.
*/
export function createMCPServerAuditor(options: {
recordAuditEntry?: RecordAuditEntry;
failClosed?: boolean;
}): (
req: Parameters<AdminAuditEmitter>[0],
event: MCPServerAuditEvent,
afterCommit?: () => Promise<void>,
) => Promise<void> {
const emit = createAdminAuditEmitter({ ...options, logPrefix: '[mcpServers]' });
async function record(
req: Parameters<AdminAuditEmitter>[0],
event: MCPServerAuditEvent,
): Promise<void> {
const { action, serverName, previous, next, postWriteFailed } = event;
const name = serverTitle(next) ?? serverTitle(previous);
const before = action === 'mcp_server.created' ? undefined : previous;
const changed =
action === 'mcp_server.deleted'
? []
: diffMCPServerFields(before, withPreservedOAuthSecret(before, next));
await emit(req, {
action,
severity: action === 'mcp_server.deleted' ? 'warning' : 'info',
target: { type: 'mcp_server', id: serverName, ...(name ? { name } : {}) },
metadata: {
source: 'DB',
...(changed.length > 0 ? { fields: changed.join(',') } : {}),
...(postWriteFailed ? { postWrite: 'failed' } : {}),
},
});
}
/**
* `afterCommit` carries the security-critical steps that follow a committed
* change (connection fences); they run first, and the audit write follows even
* when they throw, so audit latency never delays them and a failure never
* drops the entry.
*/
return async function auditMCPServer(req, event, afterCommit) {
let afterCommitFailed = false;
try {
await afterCommit?.();
} catch (error) {
afterCommitFailed = true;
throw error;
} finally {
if (!event.noop) {
await record(req, {
...event,
postWriteFailed: event.postWriteFailed || afterCommitFailed,
});
}
}
};
}
/**
* Runs a registry write and reports what it persisted. When the write committed but the
* cache refresh after it failed, the committed result is still returned together with that
* failure, so the caller can audit the change and then surface the error.
*/
export async function settleMCPCommit<T>(
write: () => Promise<T>,
): Promise<{ committed: T | undefined; postWriteError?: MCPCommittedWriteError<T> }> {
try {
return { committed: await write() };
} catch (error) {
if (error instanceof MCPCommittedWriteError) {
return {
committed: error.committed as T | undefined,
postWriteError: error as MCPCommittedWriteError<T>,
};
}
throw error;
}
}

View file

@ -527,3 +527,24 @@ export function createScheduledMCPTransportError(
agentId,
);
}
/**
* Thrown when a registry write committed but a step after it failed (permission
* bookkeeping, cache invalidation). The write stands; `committed` carries what was
* persisted (undefined for a removal) so a caller can still record it before surfacing
* the failure. The persisted config can hold decrypted secrets, so the property is
* non-enumerable and never reaches a serialized or logged error.
*/
export class MCPCommittedWriteError<T = unknown> extends Error {
public readonly committed!: T | undefined;
constructor(committed: T | undefined, cause: unknown) {
/** Fixed text: this message reaches API responses, and the cause may carry driver or cache details. */
super('MCP server change was saved but a step after it failed');
this.name = 'MCPCommittedWriteError';
Object.defineProperty(this, 'committed', { value: committed, enumerable: false });
/** Non-enumerable, so structured loggers never serialize driver or cache details. */
Object.defineProperty(this, 'cause', { value: cause, enumerable: false, configurable: true });
Object.setPrototypeOf(this, MCPCommittedWriteError.prototype);
}
}

View file

@ -14,7 +14,11 @@ import {
APP_CACHE_NAMESPACE,
CONFIG_CACHE_NAMESPACE,
} from './cache/ServerConfigsCacheFactory';
import { MCPInspectionFailedError, isMCPDomainNotAllowedError } from '~/mcp/errors';
import {
MCPCommittedWriteError,
MCPInspectionFailedError,
isMCPDomainNotAllowedError,
} from '~/mcp/errors';
import { normalizeLegacyHeaderMaps, normalizeLegacyHeaderMapsIn } from './compat';
import { canBackfillSharedServerInstructions, isUserSourced } from '~/mcp/utils';
import { ReadThroughAllCache } from './cache/ReadThroughAllCache';
@ -23,6 +27,7 @@ import { requireApiKeyReentryForRebinding } from './binding';
import { ReadThroughCache } from './cache/ReadThroughCache';
import { MCPServerInspector } from './MCPServerInspector';
import { ServerConfigsDB } from './db/ServerConfigsDB';
import { getSafeErrorMetadata } from '~/utils/errors';
import { cacheConfig } from '~/cache';
import { withTimeout } from '~/utils';
@ -672,22 +677,73 @@ export class MCPServersRegistry {
...parsedConfig,
source,
};
const result =
storageLocation === 'DB'
? await this.dbConfigsRepo.add(
serverName,
tagged,
userId,
await this.getOperatorManagedServerNames(reservedServerNames),
)
: await configRepo.add(serverName, tagged, userId);
await this.invalidateServerReadCaches(result.serverName, userId, storageLocation);
let result: t.AddServerResult;
try {
result =
storageLocation === 'DB'
? await this.dbConfigsRepo.add(
serverName,
tagged,
userId,
await this.getOperatorManagedServerNames(reservedServerNames),
)
: await configRepo.add(serverName, tagged, userId);
} catch (error) {
await this.invalidateAfterCommittedFailure(error, serverName, userId, storageLocation);
throw error;
}
await this.invalidateCommitted(result, () =>
this.invalidateServerReadCaches(result.serverName, userId, storageLocation),
);
if (storageLocation === 'CACHE') {
this.resetYamlServerNamesMemo();
}
return result;
}
/**
* A repository step after a committed write failed (for example permission bookkeeping),
* so the stored config changed even though the call rejects: refresh the read caches
* best-effort before the failure propagates, or they would keep serving the old state.
*/
private async invalidateAfterCommittedFailure(
error: unknown,
serverName: string,
userId: string | undefined,
storageLocation: 'CACHE' | 'DB',
): Promise<void> {
if (!(error instanceof MCPCommittedWriteError)) {
return;
}
const committed = error.committed as t.AddServerResult | undefined;
await this.invalidateServerReadCaches(
committed?.serverName ?? serverName,
userId,
storageLocation,
).catch((invalidationError) =>
logger.error(
'[MCPServersRegistry] cache invalidation failed after a committed write error:',
getSafeErrorMetadata(invalidationError),
),
);
}
/**
* Runs the cache invalidation that follows a committed write. A failure is rethrown as
* `MCPCommittedWriteError` carrying what was persisted, so the caller can record
* the committed change instead of mistaking it for a write that never happened.
*/
private async invalidateCommitted<T>(
committed: T | undefined,
invalidate: () => Promise<void>,
): Promise<void> {
try {
await invalidate();
} catch (error) {
throw new MCPCommittedWriteError(committed, error);
}
}
/**
* Backfills the inspector-derived `resolvedInstructions` for a server whose
* operator explicitly deferred startup inspection. An enabled
@ -1059,9 +1115,11 @@ export class MCPServersRegistry {
userId?: string,
): Promise<t.ParsedServerConfig> {
const configRepo = this.getConfigRepository(storageLocation);
await configRepo.update(serverName, parsedConfig, userId);
await this.invalidateServerReadCaches(serverName, userId, storageLocation);
return parsedConfig;
const persisted = (await configRepo.update(serverName, parsedConfig, userId)) ?? parsedConfig;
await this.invalidateCommitted(persisted, () =>
this.invalidateServerReadCaches(serverName, userId, storageLocation),
);
return persisted;
}
public async updateServer(
@ -1323,13 +1381,22 @@ export class MCPServersRegistry {
serverName: string,
storageLocation: 'CACHE' | 'DB',
userId?: string,
): Promise<void> {
): Promise<boolean> {
const configRepo = this.getConfigRepository(storageLocation);
await configRepo.remove(serverName, userId);
await this.invalidateServerReadCaches(serverName, userId, storageLocation);
let removed: boolean | void;
try {
removed = await configRepo.remove(serverName, userId);
} catch (error) {
await this.invalidateAfterCommittedFailure(error, serverName, userId, storageLocation);
throw error;
}
await this.invalidateCommitted(removed !== false, () =>
this.invalidateServerReadCaches(serverName, userId, storageLocation),
);
if (storageLocation === 'CACHE') {
this.resetYamlServerNamesMemo();
}
return removed !== false;
}
private getConfigRepository(storageLocation: 'CACHE' | 'DB'): IServerConfigsRepositoryInterface {

View file

@ -7,7 +7,12 @@ export interface IServerConfigsRepositoryInterface {
add(serverName: string, config: ParsedServerConfig, userId?: string): Promise<AddServerResult>;
//ACL Entry check if update is possible
update(serverName: string, config: ParsedServerConfig, userId?: string): Promise<void>;
/** Resolves to the config as persisted when the store can report it. */
update(
serverName: string,
config: ParsedServerConfig,
userId?: string,
): Promise<ParsedServerConfig | void>;
/** Atomic add-or-update without requiring callers to inspect error messages. */
upsert(serverName: string, config: ParsedServerConfig, userId?: string): Promise<void>;
@ -45,7 +50,8 @@ export interface IServerConfigsRepositoryInterface {
): Promise<ParsedServerConfig | undefined>;
//ACL Entry check if remove is possible
remove(serverName: string, userId?: string): Promise<void>;
/** Resolves to false when the store knew no such server, so a no-op is distinguishable. */
remove(serverName: string, userId?: string): Promise<boolean | void>;
//ACL Entry check if read is possible
get(serverName: string, userId?: string): Promise<ParsedServerConfig | undefined>;

View file

@ -14,9 +14,9 @@ import {
MCPConfigInitializationCanceledError,
} from '~/mcp/registry/MCPServersRegistry';
import { ServerConfigsCacheInMemory } from '~/mcp/registry/cache/ServerConfigsCacheInMemory';
import { MCPCommittedWriteError, MCPInspectionFailedError } from '~/mcp/errors';
import { getMCPAppToolsPublicationGeneration } from '~/mcp/toolsChanged';
import { MCPServerInspector } from '~/mcp/registry/MCPServerInspector';
import { MCPInspectionFailedError } from '~/mcp/errors';
import { processMCPEnv } from '~/utils/env';
// Mock MCPServerInspector to avoid actual server connections
@ -843,6 +843,67 @@ describe('MCPServersRegistry', () => {
});
});
describe('cache invalidation failure after a committed write', () => {
const failInvalidation = () =>
jest
.spyOn(
registry as unknown as { invalidateServerReadCaches: () => Promise<void> },
'invalidateServerReadCaches',
)
.mockRejectedValue(new Error('redis down'));
it('reports what addServer persisted', async () => {
failInvalidation();
const failure = await registry
.addServer('cache_server', testParsedConfig, 'CACHE')
.catch((error) => error);
expect(failure).toBeInstanceOf(MCPCommittedWriteError);
expect(failure.committed.serverName).toBe('cache_server');
expect(await registry['cacheConfigsRepo'].get('cache_server')).toBeDefined();
});
it('reports what commitServerUpdate persisted', async () => {
await registry.addServer('cache_server', testParsedConfig, 'CACHE');
failInvalidation();
const updated = { ...testParsedConfig, command: 'python' } as t.ParsedServerConfig;
const failure = await registry
.commitServerUpdate('cache_server', updated, 'CACHE')
.catch((error) => error);
expect(failure).toBeInstanceOf(MCPCommittedWriteError);
expect(failure.committed.command).toBe('python');
});
it('refreshes the read caches when a repository step after the write fails', async () => {
await registry.addServer('cache_server', testParsedConfig, 'CACHE');
const invalidate = jest.spyOn(
registry as unknown as { invalidateServerReadCaches: () => Promise<void> },
'invalidateServerReadCaches',
);
jest
.spyOn(registry['cacheConfigsRepo'], 'remove')
.mockRejectedValueOnce(new MCPCommittedWriteError(undefined, new Error('acl down')));
await expect(registry.removeServer('cache_server', 'CACHE')).rejects.toBeInstanceOf(
MCPCommittedWriteError,
);
expect(invalidate).toHaveBeenCalledWith('cache_server', undefined, 'CACHE');
});
it('still reports a removal that committed', async () => {
await registry.addServer('cache_server', testParsedConfig, 'CACHE');
failInvalidation();
await expect(registry.removeServer('cache_server', 'CACHE')).rejects.toBeInstanceOf(
MCPCommittedWriteError,
);
expect(await registry['cacheConfigsRepo'].get('cache_server')).toBeUndefined();
});
});
describe('Invalid storage location', () => {
it('should throw error for unsupported storage location in addServer', async () => {
await expect(

View file

@ -204,6 +204,37 @@ describe('ServerConfigsDB', () => {
expect(retrieved?.description).toBe('Updated description');
});
it('returns the config as persisted, so an overlapping update is not mistaken for this one', async () => {
const created = await serverConfigsDB.add(
'temp-name',
createSSEConfig('Title', 'First'),
userId,
);
const firstWrite = await serverConfigsDB.update(
created.serverName,
createSSEConfig('Title', 'Second'),
userId,
);
await serverConfigsDB.update(created.serverName, createSSEConfig('Title', 'Third'), userId);
expect(firstWrite?.description).toBe('Second');
expect((await serverConfigsDB.get(created.serverName, userId))?.description).toBe('Third');
});
it('rejects an update when the server was deleted after the lookup', async () => {
const created = await serverConfigsDB.add(
'temp-name',
createSSEConfig('Title', 'First'),
userId,
);
await serverConfigsDB.remove(created.serverName, userId);
await expect(
serverConfigsDB.update(created.serverName, createSSEConfig('Title', 'Second'), userId),
).rejects.toThrow('no longer exists');
});
it('should preserve oauth.client_secret when not provided in update', async () => {
const config: ParsedServerConfig = {
type: 'sse',
@ -1427,12 +1458,40 @@ describe('ServerConfigsDB', () => {
});
});
describe('post-write failures', () => {
it('reports the created server when granting its owner access fails', async () => {
jest
.spyOn(serverConfigsDB['_aclService'], 'grantPermission')
.mockRejectedValueOnce(new Error('acl down'));
const failure = await serverConfigsDB
.add('temp-name', createSSEConfig('Orphan Risk'), userId)
.catch((error) => error);
expect(failure.name).toBe('MCPCommittedWriteError');
expect(failure.committed.config.title).toBe('Orphan Risk');
expect(await mongoose.models.MCPServer.countDocuments({})).toBe(1);
});
it('reports the removal when cleaning up its permissions fails', async () => {
const created = await serverConfigsDB.add('temp-name', createSSEConfig('Remove Me'), userId);
jest
.spyOn(serverConfigsDB['_aclService'], 'removeAllPermissions')
.mockRejectedValueOnce(new Error('acl down'));
await expect(serverConfigsDB.remove(created.serverName, userId)).rejects.toMatchObject({
name: 'MCPCommittedWriteError',
});
expect(await mongoose.models.MCPServer.countDocuments({})).toBe(0);
});
});
describe('remove()', () => {
it('should delete server from database', async () => {
const config = createSSEConfig('Delete Test');
const created = await serverConfigsDB.add('temp-name', config, userId);
await serverConfigsDB.remove(created.serverName, userId);
await expect(serverConfigsDB.remove(created.serverName, userId)).resolves.toBe(true);
const MCPServer = mongoose.models.MCPServer;
const server = await MCPServer.findOne({ serverName: created.serverName });
@ -1461,8 +1520,8 @@ describe('ServerConfigsDB', () => {
});
it('should handle non-existent server gracefully', async () => {
// Should not throw
await expect(serverConfigsDB.remove('non-existent-server', userId)).resolves.toBeUndefined();
// Should not throw, and reports that nothing was removed
await expect(serverConfigsDB.remove('non-existent-server', userId)).resolves.toBe(false);
});
});

View file

@ -11,8 +11,8 @@ import type { IServerConfigsRepositoryInterface } from '~/mcp/registry/ServerCon
import type { ParsedServerConfig, AddServerResult } from '~/mcp/types';
import type { ResolvedPrincipal } from '~/types/principal';
import { getUserApiKeyVariable, requireApiKeyReentryForRebinding } from '~/mcp/registry/binding';
import { MCPCommittedWriteError, MCPOAuthSecretReentryRequiredError } from '~/mcp/errors';
import { normalizeLegacyHeaderMaps } from '~/mcp/registry/compat';
import { MCPOAuthSecretReentryRequiredError } from '~/mcp/errors';
import { AccessControlService } from '~/acl/accessControlService';
import { isGeneratedUserApiKeyVariable } from '~/mcp/headers';
@ -316,18 +316,24 @@ export class ServerConfigsDB implements IServerConfigsRepositoryInterface {
author: userId,
reservedServerNames,
});
await this._aclService.grantPermission({
principalType: PrincipalType.USER,
principalId: userId,
resourceType: ResourceType.MCPSERVER,
resourceId: createdServer._id,
accessRoleId: AccessRoleIds.MCPSERVER_OWNER,
grantedBy: userId,
});
return {
const created = {
serverName: createdServer.serverName,
config: await this.mapDBServerToParsedConfig(createdServer),
};
try {
await this._aclService.grantPermission({
principalType: PrincipalType.USER,
principalId: userId,
resourceType: ResourceType.MCPSERVER,
resourceId: createdServer._id,
accessRoleId: AccessRoleIds.MCPSERVER_OWNER,
grantedBy: userId,
});
} catch (error) {
/** The server exists already, so the failure must not read as a create that never happened. */
throw new MCPCommittedWriteError(created, error);
}
return created;
}
/**
@ -340,7 +346,7 @@ export class ServerConfigsDB implements IServerConfigsRepositoryInterface {
serverName: string,
config: ParsedServerConfig,
userId?: string,
): Promise<void> {
): Promise<ParsedServerConfig> {
if (!userId) {
throw new Error(
'[ServerConfigsDB.update] User ID is required to update a database-stored MCP server.',
@ -405,7 +411,11 @@ export class ServerConfigsDB implements IServerConfigsRepositoryInterface {
};
}
await this._dbMethods.updateMCPServer(serverName, { config: configToSave });
const saved = await this._dbMethods.updateMCPServer(serverName, { config: configToSave });
if (!saved) {
throw new Error(`[ServerConfigsDB.update] MCP server "${serverName}" no longer exists.`);
}
return await this.mapDBServerToParsedConfig(saved);
}
/**
@ -424,7 +434,7 @@ export class ServerConfigsDB implements IServerConfigsRepositoryInterface {
'Config-source servers should use configCacheRepo, not dbConfigsRepo.',
);
}
return this.update(serverName, config, userId);
await this.update(serverName, config, userId);
}
/**
@ -432,18 +442,23 @@ export class ServerConfigsDB implements IServerConfigsRepositoryInterface {
* @param serverName - The serverName of the server to remove
* @param userId - User performing the deletion (for logging)
*/
public async remove(serverName: string, userId?: string): Promise<void> {
public async remove(serverName: string, userId?: string): Promise<boolean> {
logger.debug(`[ServerConfigsDB.remove] removing ${serverName}. UserId: ${userId}`);
const deletedServer = await this._dbMethods.deleteMCPServer(serverName);
if (deletedServer && deletedServer._id) {
logger.debug(`[ServerConfigsDB.remove] removing all permissions entries of ${serverName}.`);
await this._aclService.removeAllPermissions({
resourceType: ResourceType.MCPSERVER,
resourceId: deletedServer._id!,
});
return;
try {
await this._aclService.removeAllPermissions({
resourceType: ResourceType.MCPSERVER,
resourceId: deletedServer._id!,
});
} catch (error) {
throw new MCPCommittedWriteError(true, error);
}
return true;
}
logger.warn(`[ServerConfigsDB.remove] server with serverName ${serverName} does not exist`);
return false;
}
/**

View file

@ -523,6 +523,59 @@ describe('auditLog methods', () => {
expect(remaining.map((r) => r.seq)).toEqual([1, 2, 3, 4]);
});
it('records audit.purged after a purge that removed rows, and not after a no-op', async () => {
for (let i = 0; i < 4; i++) {
await methods.recordAuditEntry(baseInput());
await new Promise((resolve) => setTimeout(resolve, 5));
}
const third = await AuditLog.findOne({ chainKey: CK_A, seq: 3 }).lean();
await methods.purgeAuditLogEntries('tenant-a', {
before: new Date(Date.now() - 60_000),
confirm: true,
});
expect(await AuditLog.countDocuments({ chainKey: CK_A, action: 'audit.purged' })).toBe(0);
const result = await methods.purgeAuditLogEntries('tenant-a', {
before: third!.createdAt,
confirm: true,
});
const entry = await AuditLog.findOne({ chainKey: CK_A, action: 'audit.purged' }).lean();
expect(entry).toMatchObject({
category: 'audit',
severity: 'critical',
actor: { type: 'system', name: 'audit-retention' },
target: { type: 'audit_log' },
metadata: { deletedCount: result.deletedCount, throughSeq: result.checkpoint?.throughSeq },
});
const verified = await methods.verifyAuditChain('tenant-a', {
trustedCheckpoint: result.checkpoint,
});
expect(verified.ok).toBe(true);
});
it('still returns the purge result when the audit write rejects', async () => {
for (let i = 0; i < 4; i++) {
await methods.recordAuditEntry(baseInput());
await new Promise((resolve) => setTimeout(resolve, 5));
}
const third = await AuditLog.findOne({ chainKey: CK_A, seq: 3 }).lean();
const spy = jest.spyOn(AuditLog, 'create').mockRejectedValue(new Error('audit store down'));
try {
const result = await methods.purgeAuditLogEntries('tenant-a', {
before: third!.createdAt,
confirm: true,
});
expect(result.deletedCount).toBeGreaterThanOrEqual(1);
expect(result.checkpoint).toBeDefined();
} finally {
spy.mockRestore();
}
expect(await AuditLog.countDocuments({ chainKey: CK_A, action: 'audit.purged' })).toBe(0);
});
it('does not mint a checkpoint when a confirmed purge removed nothing', async () => {
await seed(2);
const result = await methods.purgeAuditLogEntries('tenant-a', {

View file

@ -710,6 +710,29 @@ export function createAuditLogMethods(
.select('seq prevHash')
.lean<{ seq: number; prevHash: string }>();
/** The purge itself is audited after the checkpoint is read, so the new
* entry cannot shift the boundary; a failed write is logged, not thrown. */
try {
await recordAuditEntry({
action: 'audit.purged',
severity: 'critical',
actor: { type: 'system', name: 'audit-retention' },
target: { type: 'audit_log' },
metadata: {
deletedCount,
before: options.before.toISOString(),
...(newFirst ? { throughSeq: newFirst.seq - 1 } : {}),
},
tenantId,
});
} catch (err) {
logger.error('[auditLog] failed to record retention purge', {
chainKey,
deletedCount,
error: err instanceof Error ? err.name : 'unknown error',
});
}
return {
deletedCount,
/** Boundary for `verifyAuditChain`: the new earliest entry resumes the