🗃️ fix: Restore Authorization Uniqueness and Revoke Duplicate Grants (#18258)

* fix: repair duplicate authorization records and revoke every matching grant

* test: verify standalone authorization repair command and keep bootstrap isolated

* test: distinguish supported aggregate disk options in DocumentDB guard
This commit is contained in:
Danny Avila 2026-11-17 09:52:22 -05:00 • committed by GitHub
parent cb133ebf09
commit 8a7fe7b9f3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 767 additions and 1 deletions

View file

@ -180,3 +180,89 @@ possible latency spike. Do not use the broad migration on a live database.
The original offline procedure remains available when all writers can be stopped:
`npm run migrate:tenant-indexes`. It also covers collections beyond `mcpnames`.
### Repairing duplicate roles, system grants, and agent categories
A failed unique-index build can leave legacy duplicate authorization/configuration
records. Updating the application does not remove those records or repair the
missing constraint. The corrected grant revocation removes every matching grant
in the exact requested tenant/platform scope, even before the index is repaired.
It does not revoke grants for a different tenant or remove platform grants when
revoking a tenant grant. Deploy the correction to every API replica first.
Use the targeted repair below for `systemgrants`, `roles`, and `agentcategories`.
It does **not** repair files, messages, MCP names, or all tenant indexes. It runs
only when invoked explicitly, never at startup.
1. Take a database snapshot and rehearse on a restored copy using the deployment's
actual DocumentDB engine/version. Local regression tests use real MongoDB;
they do not establish DocumentDB compatibility. Check archive, CRUD, aggregate,
and index-build privileges before the production window. Inspect which regional
deployments share a database; run once per independent writer database, not
once per API region connected to the same database.
2. From one maintenance container with the corrected image, the writer `MONGO_URI`
and the deployment's TLS settings, run from `/app`:
```sh
node config/repair-authorization-duplicates.js --dry-run
```
Preview is also the default for `node config/repair-authorization-duplicates.js`. Output contains
aggregate counts and index status only. Conflicting settings produce exit code
2 in preview; operational failures produce exit code 1. A live preview is not
a consistent snapshot and is not permission or index-build validation.
3. Gate and drain **all writes to these three collections**, including admin grant
changes, role creation/edit/rename/deletion and their grant cascades, category
changes, tenant onboarding/seeding, account/group erasure and maintenance
scripts. Audit fork-specific writers. Freeze rollouts, restarts and automatic
scale-up that could run `initializeRoles`, `seedSystemGrants` or
`ensureDefaultCategories`. Role reads can create a missing built-in role:
existing chat can continue only when its role records already exist. Gate new
tenant provisioning and requests that would create missing roles as well.
This is a deployment-controlled gate; there is no built-in maintenance toggle.
Blocking only `/api/admin/grants` is insufficient. If the deployment cannot
isolate these writers, stop writers for the repair instead of assuming a safe
online migration. Reads and unrelated message/chat writes need not be stopped.
4. Run one migrator, keeping the gate closed until every index is verified:
```sh
node config/repair-authorization-duplicates.js --apply --writers-paused
```
`--writers-paused` acknowledges the operational gate; it does not enforce it.
The command inspects all three collections before cleanup and refuses conflicting
records. It retains the lowest `_id` in each equivalent group without modifying
it, archives every complete removed record in `authorization_duplicate_archive`,
then deletes the duplicate. Role/category settings and unknown fields must agree;
only `_id`, `__v`, `createdAt`, and `updatedAt` are ignored for comparison. Grants
also ignore `grantedAt` and `grantedBy` provenance, which the archive preserves;
different `expiresAt` values or tenant-field representations are conflicts.
No permissions are unioned and no tenant is reassigned. Conflict resolution
requires a separate reviewed decision; do not simply keep the most permissive
role or restore every archived grant.
5. Require exit zero and all three `uniqueIndex` values to be true. Verify full,
non-sparse, non-partial unique indexes on `(principalType, principalId,
capability, tenantId)`, `(name, tenantId)`, and `(value, tenantId)` respectively.
The repair requests background index builds serially and drops no existing/custom
indexes. Amazon DocumentDB defaults to foreground builds; background builds
allow reads to continue ([AWS index creation guidance](https://docs.aws.amazon.com/documentdb/latest/devguide/index-creation.html)).
Rehearse and watch database load/latency; background does not mean zero impact.
Re-run preview, verify zero duplicate groups, smoke-test grant/revoke plus role
and category operations in staging, then reopen the affected writes.
Every removal is archived with majority acknowledgement and read back before
source deletion. Keep that collection access-restricted with the source data;
there is no TTL or automatic restore. A database snapshot is still required.
A failed/interrupted run may have archived/deleted some records or built some
indexes. Keep the gate closed and rerun preview before resuming apply. The archive
is idempotent; an existing archive that differs from its source blocks deletion.
Apply takes a single-migrator lock in `authorization_repair_locks`. A hard-killed
process leaves the `authorization` lock in place intentionally. Before removing
that lock manually, verify the previous process and its database operations have
finished; do not expire or steal it while an index build may still be running.
Duplicate scans use no disk spill, a 60-second server timeout, and a 1,000-record
per-group bound. Oversized/slow scans fail rather than performing an unbounded
repair. These bounds are maintenance-library options, not request-path settings.

View file

@ -0,0 +1,85 @@
const path = require('node:path');
const { spawnSync } = require('node:child_process');
const mongoose = require('mongoose');
const { MongoMemoryServer } = require('mongodb-memory-server');
const root = path.resolve(__dirname, '../..');
let mongo;
let connection;
function run(...args) {
return spawnSync(process.execPath, ['config/repair-authorization-duplicates.js', ...args], {
cwd: root,
env: {
...process.env,
MONGO_URI: mongo.getUri('authorization_cli'),
MONGO_AUTO_INDEX: 'true',
MONGO_AUTO_CREATE: 'true',
},
encoding: 'utf8',
timeout: 20_000,
});
}
beforeAll(async () => {
mongo = await MongoMemoryServer.create();
connection = await mongoose
.createConnection(mongo.getUri('authorization_cli'), {
autoIndex: false,
autoCreate: false,
})
.asPromise();
});
afterAll(async () => {
await connection?.close();
await mongo?.stop();
});
beforeEach(async () => {
await connection.dropDatabase();
});
it('previews by default without starting seeders or creating indexes, then applies explicitly', async () => {
const roles = connection.db.collection('roles');
await roles.insertMany([
{ name: 'CUSTOM', permissions: {} },
{ name: 'CUSTOM', permissions: {} },
]);
const preview = run();
expect(preview.status).toBe(0);
expect(JSON.parse(preview.stdout.trim())).toMatchObject({ dryRun: true });
expect(await roles.countDocuments()).toBe(2);
expect((await roles.indexes()).map((index) => index.name)).toEqual(['_id_']);
expect((await connection.db.listCollections().toArray()).map((item) => item.name)).toEqual([
'roles',
]);
expect(run('--apply').status).toBe(1);
expect(run('--apply', '--dry-run', '--writers-paused').status).toBe(1);
expect(run('--aply').status).toBe(1);
expect(await roles.countDocuments()).toBe(2);
const applied = run('--apply', '--writers-paused');
expect(applied.status).toBe(0);
expect(JSON.parse(applied.stdout.trim())).toMatchObject({ dryRun: false });
expect(await roles.countDocuments()).toBe(1);
expect(await connection.db.collection('authorization_duplicate_archive').countDocuments()).toBe(
1,
);
expect((await roles.indexes()).find((index) => index.name === 'name_1_tenantId_1').unique).toBe(
true,
);
});
it('signals conflicts with aggregate-only output and leaves conflicting records intact', async () => {
const roles = connection.db.collection('roles');
await roles.insertMany([
{ name: 'PRIVATE_ROLE_VALUE', permissions: { allow: true } },
{ name: 'PRIVATE_ROLE_VALUE', permissions: { allow: false } },
]);
const preview = run('--dry-run');
expect(preview.status).toBe(2);
expect(preview.stdout).not.toContain('PRIVATE_ROLE_VALUE');
const applied = run('--apply', '--writers-paused');
expect(applied.status).toBe(1);
expect(applied.stderr).not.toContain('PRIVATE_ROLE_VALUE');
expect(await roles.countDocuments()).toBe(2);
});

View file

@ -0,0 +1,50 @@
require('dotenv').config();
process.env.MONGO_AUTO_INDEX = 'false';
process.env.MONGO_AUTO_CREATE = 'false';
const mongoose = require('mongoose');
const { repairAuthorizationDuplicates } = require('@librechat/data-schemas');
(async () => {
// Preview is the default. Apply needs both an explicit mode and a write-gate acknowledgement.
const args = process.argv.slice(2);
const allowed = new Set(['--dry-run', '--apply', '--writers-paused']);
if (
args.some((arg) => !allowed.has(arg)) ||
(args.includes('--apply') && args.includes('--dry-run')) ||
(args.includes('--apply') && !args.includes('--writers-paused'))
) {
console.error('Use --dry-run or --apply --writers-paused. See UPGRADING.md.');
process.exitCode = 1;
return;
}
try {
if (!process.env.MONGO_URI) {
throw new Error('MONGO_URI is required');
}
// Maintenance owns one primary connection; do not import application bootstrapping or seeders.
await mongoose.connect(process.env.MONGO_URI, {
autoIndex: false,
autoCreate: false,
readPreference: 'primary',
maxPoolSize: 1,
serverSelectionTimeoutMS: 10_000,
});
const dryRun = !args.includes('--apply');
const reports = await repairAuthorizationDuplicates(mongoose.connection, {
dryRun,
writersPaused: args.includes('--writers-paused'),
});
console.log(JSON.stringify({ dryRun, reports }));
if (reports.some((report) => report.conflictingGroups > 0)) {
process.exitCode = 2;
}
} catch {
// Database errors may contain principal identities or source documents.
console.error(
'Authorization repair failed. Keep affected writes gated; rerun dry-run and inspect database state securely.',
);
process.exitCode = 1;
} finally {
await mongoose.disconnect();
}
})();

View file

@ -212,3 +212,5 @@ export * from './postgres/import';
export * from './postgres/mongo';
export { migrateMCPReservationIndex } from './migrations/mcpReservations';
export { repairAuthorizationDuplicates } from './migrations/authorization';

View file

@ -531,6 +531,30 @@ function isUnjudgedDottedWhere(node: ts.Node): boolean {
return !assignsCode;
}
/** AWS supports this option on aggregate, but not find. Judge only direct option
* properties in the aggregate call's second argument; nested pipeline fields and
* unproven aliases retain the conservative token verdict.
* https://docs.aws.amazon.com/documentdb/latest/devguide/functional-differences.html#functional-differences.lookup
*/
function isAggregateDiskOption(node: ts.Node): boolean {
const property = node.parent;
if (!ts.isPropertyAssignment(property) || property.name !== node) {
return false;
}
const options = property.parent;
if (!ts.isObjectLiteralExpression(options)) {
return false;
}
const use = outermostWrapper(options).parent;
return (
ts.isCallExpression(use) &&
ts.isPropertyAccessExpression(use.expression) &&
calleeName(use) === 'aggregate' &&
use.arguments.length >= 2 &&
unwrapExpression(use.arguments[1]) === options
);
}
function findForbiddenTokens(sourceFile: ts.SourceFile): string[] {
if (OPERATOR_GUARDS.has(sourceFile.fileName)) {
return [];
@ -543,6 +567,9 @@ function findForbiddenTokens(sourceFile: ts.SourceFile): string[] {
(ts.isIdentifier(node) && !ts.isPropertySignature(node.parent)))
) {
for (const token of FORBIDDEN_TOKENS) {
if (token === 'allowDiskUse' && node.text === token && isAggregateDiskOption(node)) {
continue;
}
if (node.text === token || node.text.startsWith(`${token}.`)) {
offenses.push(offenseAt(sourceFile, node, token));
}
@ -1056,6 +1083,24 @@ describe('Amazon DocumentDB compatibility', () => {
expect(findMixedSelectStrings(parse('fixture.ts', source))).toEqual([]);
});
it.each([
`collection.aggregate([], { allowDiskUse: false });`,
`collection.aggregate([], { 'allowDiskUse': true });`,
`collection.aggregate([], ({ allowDiskUse: false } as AggregateOptions));`,
])('accepts the supported aggregation disk option: %s', (source) => {
expect(findForbiddenTokens(parse('fixture.ts', source))).toEqual([]);
});
it.each([
`collection.find({}, { allowDiskUse: false });`,
`collection.find({}).allowDiskUse(false);`,
`collection.aggregate([{ $project: { allowDiskUse: false } }]);`,
`collection.aggregate([], { nested: { allowDiskUse: false } });`,
`const options = { allowDiskUse: false }; collection.find({}, options);`,
])('retains the disk-option guard outside proven aggregate options: %s', (source) => {
expect(findForbiddenTokens(parse('fixture.ts', source))).not.toEqual([]);
});
it('flags forbidden operators in code but not in prose or type members', () => {
expect(
findForbiddenTokens(parse('fixture.ts', `const projection = { x: '$$REMOVE' };`)),

View file

@ -344,6 +344,53 @@ describe('systemGrant methods', () => {
expect(grant).toBeNull();
});
it('revokes every legacy duplicate without touching another tenant or capability', async () => {
const keys = { principalType: 1, principalId: 1, capability: 1, tenantId: 1 } as const;
await SystemGrant.collection.dropIndex(
'principalType_1_principalId_1_capability_1_tenantId_1',
);
try {
const principalId = new Types.ObjectId();
const base = {
principalType: PrincipalType.USER,
principalId,
capability: SystemCapabilities.READ_USERS,
};
await SystemGrant.collection.insertMany([
{ ...base },
{ ...base },
{ ...base, tenantId: 'a' },
{ ...base, tenantId: 'a' },
{ ...base, tenantId: 'b' },
{ ...base, capability: SystemCapabilities.READ_USAGE },
]);
expect(
await methods.revokeCapability({
...base,
principalId: principalId.toString(),
tenantId: 'a',
}),
).toEqual({ deletedCount: 2 });
expect(await methods.revokeCapability(base)).toEqual({ deletedCount: 2 });
expect(
await methods.hasCapabilityForPrincipals({
principals: [{ principalType: PrincipalType.USER, principalId }],
capability: base.capability,
tenantId: 'a',
}),
).toBe(false);
const remaining = await SystemGrant.find({ principalId }).lean();
expect(remaining).toHaveLength(2);
expect(remaining.some((grant) => grant.tenantId === 'b')).toBe(true);
expect(remaining.some((grant) => grant.capability === SystemCapabilities.READ_USAGE)).toBe(
true,
);
} finally {
await SystemGrant.deleteMany({});
await SystemGrant.collection.createIndex(keys, { unique: true });
}
});
it('is a no-op when the grant does not exist', async () => {
await expect(
methods.revokeCapability({

View file

@ -432,7 +432,8 @@ export function createSystemGrantMethods(mongoose: typeof import('mongoose')): {
};
const options = session ? { session } : {};
const result = await SystemGrant.deleteOne(filter, options);
// Legacy databases can contain duplicates when the unique index failed to build.
const result = await SystemGrant.deleteMany(filter, options);
return { deletedCount: result.deletedCount ?? 0 };
}

View file

@ -0,0 +1,195 @@
import mongoose from 'mongoose';
import { MongoMemoryServer } from 'mongodb-memory-server';
import { repairAuthorizationDuplicates } from './authorization';
let mongo: MongoMemoryServer;
const db = () => mongoose.connection.db!;
const collection = (name: string) => db().collection(name);
const apply = () =>
repairAuthorizationDuplicates(mongoose.connection, { dryRun: false, writersPaused: true });
beforeAll(async () => {
mongo = await MongoMemoryServer.create();
await mongoose.connect(mongo.getUri(), { autoIndex: false, autoCreate: false });
});
afterAll(async () => {
await mongoose.disconnect();
await mongo.stop();
});
beforeEach(async () => {
await mongoose.connection.dropDatabase();
});
it('defaults to a preview without creating collections or indexes', async () => {
const reports = await repairAuthorizationDuplicates(mongoose.connection);
expect(reports).toHaveLength(3);
expect(reports.every((report) => report.duplicateGroups === 0 && !report.uniqueIndex)).toBe(true);
expect(await db().listCollections().toArray()).toEqual([]);
await expect(
repairAuthorizationDuplicates(mongoose.connection, { dryRun: false }),
).rejects.toThrow('paused');
});
it('archives equivalent duplicates, preserves survivors and tenant boundaries, and is resumable', async () => {
const grants = collection('systemgrants');
const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' };
const first = { ...grant, grantedAt: new Date(1), grantedBy: new mongoose.Types.ObjectId() };
const second = { ...grant, grantedAt: new Date(2), grantedBy: new mongoose.Types.ObjectId() };
await grants.insertMany([first, second, { ...grant, tenantId: 'b' }]);
const role = { name: 'CUSTOM', permissions: { USERS: { READ: true } } };
await collection('roles').insertMany([role, { ...role }, { ...role, tenantId: 'b' }]);
const category = { value: 'general', label: 'General', isActive: true };
await collection('agentcategories').insertMany([category, { ...category }]);
await collection('roles').createIndex({ name: 1 }, { name: 'custom_lookup' });
const before = await grants.find().sort({ _id: 1 }).toArray();
const preview = await repairAuthorizationDuplicates(mongoose.connection);
expect(preview.map((report) => report.excessDocuments)).toEqual([1, 1, 1]);
expect(await grants.countDocuments()).toBe(3);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0);
const result = await apply();
expect(result.every((report) => report.uniqueIndex && report.removedDocuments === 1)).toBe(true);
expect(await grants.findOne({ _id: before[0]._id })).toEqual(before[0]);
expect(await grants.countDocuments()).toBe(2);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(3);
const archive = await collection('authorization_duplicate_archive').findOne({
'_id.collection': 'systemgrants',
});
expect(archive?.original).toEqual(before[1]);
expect(await collection('roles').indexExists('custom_lookup')).toBe(true);
await expect(grants.insertOne({ ...grant })).rejects.toMatchObject({ code: 11000 });
await grants.insertOne({ ...grant, tenantId: 'c' });
expect(
(await apply()).every((report) => report.removedDocuments === 0 && report.uniqueIndex),
).toBe(true);
});
it.each([
[
'roles',
{ name: 'CUSTOM', permissions: { allowed: true } },
{ name: 'CUSTOM', permissions: { allowed: false } },
],
['agentcategories', { value: 'general', label: 'First' }, { value: 'general', label: 'Second' }],
[
'systemgrants',
{ principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' },
{
principalType: 'role',
principalId: 'CUSTOM',
capability: 'read:users',
expiresAt: new Date(),
},
],
['roles', { name: 'CUSTOM' }, { name: 'CUSTOM', tenantId: null }],
])(
'refuses conflicting %s data before deleting from any collection',
async (name, first, second) => {
await collection(name).insertMany([first, second]);
const duplicate = { principalType: 'role', principalId: 'OTHER', capability: 'read:users' };
await collection('systemgrants').insertMany([{ ...duplicate }, { ...duplicate }]);
const report = await repairAuthorizationDuplicates(mongoose.connection);
expect(report.find((item) => item.collection === name)?.conflictingGroups).toBe(1);
await expect(apply()).rejects.toThrow('Conflicting');
expect(await collection('systemgrants').countDocuments({ principalId: 'OTHER' })).toBe(2);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0);
},
);
it('does not delete a source when archive persistence fails', async () => {
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
const spy = jest
.spyOn(mongoose.mongo.Collection.prototype, 'updateOne')
.mockRejectedValueOnce(new Error('unavailable'));
try {
await expect(apply()).rejects.toThrow('unavailable');
expect(await collection('roles').countDocuments()).toBe(2);
} finally {
spy.mockRestore();
}
});
it('resumes safely when a run stops after archival but before deletion', async () => {
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
const spy = jest
.spyOn(mongoose.mongo.Collection.prototype, 'deleteOne')
.mockRejectedValueOnce(new Error('interrupted'));
try {
await expect(apply()).rejects.toThrow('interrupted');
} finally {
spy.mockRestore();
}
expect(await collection('roles').countDocuments()).toBe(2);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
await apply();
expect(await collection('roles').countDocuments()).toBe(1);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
});
it('refuses an incompatible index and leaves duplicates untouched', async () => {
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
await collection('roles').createIndex({ name: 1, tenantId: 1 });
await expect(apply()).rejects.toThrow('Unexpected roles');
expect(await collection('roles').countDocuments()).toBe(2);
});
it('bounds duplicate group memory and refuses deletion', async () => {
await collection('roles').insertMany([
{ name: 'CUSTOM' },
{ name: 'CUSTOM' },
{ name: 'CUSTOM' },
]);
await expect(
repairAuthorizationDuplicates(mongoose.connection, { maxGroupSize: 2 }),
).rejects.toThrow('bound');
expect(await collection('roles').countDocuments()).toBe(3);
});
it('refuses a second migrator without releasing its lock', async () => {
const locks = db().collection<{ _id: string; owner: string }>('authorization_repair_locks');
await locks.insertOne({ _id: 'authorization', owner: 'first-process' });
await expect(apply()).rejects.toMatchObject({ code: 11000 });
expect(await locks.findOne({ _id: 'authorization' })).toMatchObject({ owner: 'first-process' });
});
it('keeps archived records and can resume after a failed index build', async () => {
const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' };
await collection('systemgrants').insertMany([{ ...grant }, { ...grant }]);
const spy = jest
.spyOn(mongoose.mongo.Collection.prototype, 'createIndex')
.mockRejectedValueOnce(
Object.assign(new Error('sensitive duplicate identity'), { code: 11000 }),
);
try {
await expect(apply()).rejects.toThrow('Authorization index build failed');
} finally {
spy.mockRestore();
}
expect(await collection('authorization_repair_locks').countDocuments()).toBe(0);
expect(await collection('systemgrants').countDocuments()).toBe(1);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
await apply();
expect(
await collection('systemgrants').indexExists(
'principalType_1_principalId_1_capability_1_tenantId_1',
),
).toBe(true);
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
});
it('never overwrites an archive with different source content', async () => {
const roles = collection('roles');
await roles.insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
const records = await roles.find().sort({ _id: 1 }).toArray();
await db()
.collection<{
_id: { collection: string; sourceId: mongoose.Types.ObjectId };
original: object;
}>('authorization_duplicate_archive')
.insertOne({
_id: { collection: 'roles', sourceId: records[1]._id },
original: { name: 'DIFFERENT' },
});
await expect(apply()).rejects.toThrow('archive differs');
expect(await roles.countDocuments()).toBe(2);
});

View file

@ -0,0 +1,255 @@
import { randomUUID } from 'node:crypto';
import { isDeepStrictEqual } from 'node:util';
import type { Document, IndexDescriptionInfo, ObjectId } from 'mongodb';
import type { Connection } from 'mongoose';
import { buildIndexWithRetry } from '~/utils/retry';
interface Definition {
collection: string;
keys: Record<string, 1>;
ignored: readonly string[];
}
const DEFINITIONS: readonly Definition[] = [
{
collection: 'systemgrants',
keys: { principalType: 1, principalId: 1, capability: 1, tenantId: 1 },
ignored: ['grantedAt', 'grantedBy'],
},
{ collection: 'roles', keys: { name: 1, tenantId: 1 }, ignored: [] },
{ collection: 'agentcategories', keys: { value: 1, tenantId: 1 }, ignored: [] },
];
const ARCHIVE = 'authorization_duplicate_archive';
interface ArchivedDuplicate {
_id: { collection: string; sourceId: ObjectId };
original: Document;
survivorId: ObjectId;
archivedAt: Date;
}
interface CollectionReport {
collection: string;
duplicateGroups: number;
excessDocuments: number;
conflictingGroups: number;
removedDocuments: number;
uniqueIndex: boolean;
}
export interface AuthorizationRepairOptions {
dryRun?: boolean;
/** Operator acknowledgement, not an application fence. Includes seeders and background writers. */
writersPaused?: boolean;
/** Maintenance-only scan bounds; not request-path configuration. */
maxTimeMS?: number;
maxGroupSize?: number;
}
function content(document: Document, ignored: readonly string[]): Document {
return Object.fromEntries(
Object.entries(document).filter(
([key]) => !['_id', '__v', 'createdAt', 'updatedAt', ...ignored].includes(key),
),
);
}
function sameIndex(index: IndexDescriptionInfo, keys: Definition['keys']): boolean {
return (
isDeepStrictEqual(Object.entries(index.key), Object.entries(keys)) &&
index.unique === true &&
!index.sparse &&
!index.partialFilterExpression &&
!index.collation
);
}
/**
* Explicit, resumable maintenance for three small authorization/configuration collections.
* Uses native collections to cover every tenant without compiling models or triggering seeders.
* Never merges conflicting settings, rewrites survivors, drops indexes, or prints identities.
* All affected writers must be gated and drained throughout apply; unrelated chat writes may continue.
*/
export async function repairAuthorizationDuplicates(
connection: Connection,
{
dryRun = true,
writersPaused = false,
maxTimeMS = 60_000,
maxGroupSize = 1_000,
}: AuthorizationRepairOptions = {},
): Promise<CollectionReport[]> {
if (!dryRun && !writersPaused) {
throw new Error('Authorization repair requires paused and drained collection writers');
}
if (
!Number.isSafeInteger(maxTimeMS) ||
maxTimeMS <= 0 ||
!Number.isSafeInteger(maxGroupSize) ||
maxGroupSize < 2
) {
throw new Error('Invalid authorization repair scan bounds');
}
const db = connection.db;
if (!db) {
throw new Error('Authorization repair requires an open database connection');
}
async function indexes(definition: Definition): Promise<IndexDescriptionInfo[]> {
try {
return await db!
.collection(definition.collection, { readPreference: 'primary' })
.listIndexes({ maxTimeMS })
.toArray();
} catch (error) {
if (error instanceof Error && 'code' in error && error.code === 26) {
return [];
}
throw error;
}
}
async function scan(
definition: Definition,
report: CollectionReport,
apply: boolean,
): Promise<void> {
const collection = db!.collection(definition.collection, { readPreference: 'primary' });
const archive = db!.collection<ArchivedDuplicate>(ARCHIVE, { readPreference: 'primary' });
const identity = Object.fromEntries(
Object.keys(definition.keys).map((key) => [key, { $ifNull: [`$${key}`, null] }]),
);
const cursor = collection.aggregate<{ _id: Document; count: number }>(
[{ $group: { _id: identity, count: { $sum: 1 } } }, { $match: { count: { $gt: 1 } } }],
{ allowDiskUse: false, maxTimeMS, batchSize: 50 },
);
try {
for await (const group of cursor) {
report.duplicateGroups++;
report.excessDocuments += group.count - 1;
if (group.count > maxGroupSize) {
throw new Error('Authorization duplicate group exceeds the configured bound');
}
// Null matches both absent and explicit-null fields, as the unique index does.
const records = await collection
.find(
Object.fromEntries(
Object.entries(group._id).map(([key, value]) => [key, { $eq: value }]),
),
{ maxTimeMS },
)
.sort({ _id: 1 })
.limit(maxGroupSize + 1)
.toArray();
if (records.length !== group.count) {
throw new Error('Authorization data changed during repair; verify the write gate');
}
const [survivor, ...duplicates] = records;
const settings = content(survivor, definition.ignored);
if (
duplicates.some(
(record) => !isDeepStrictEqual(content(record, definition.ignored), settings),
)
) {
report.conflictingGroups++;
continue;
}
if (!apply) {
continue;
}
for (const record of duplicates) {
// Preserve complete provenance before deletion. The archive has no TTL and is not read by auth.
const archiveId = { collection: definition.collection, sourceId: record._id };
await archive.updateOne(
{ _id: archiveId },
{
$setOnInsert: { original: record, survivorId: survivor._id, archivedAt: new Date() },
},
{ upsert: true, writeConcern: { w: 'majority' } },
);
const archived = await archive.findOne({ _id: archiveId }, { maxTimeMS });
if (!archived || !isDeepStrictEqual(archived.original, record)) {
throw new Error('Authorization archive differs from source; refusing deletion');
}
const removed = await collection.deleteOne(
{ _id: record._id },
{ writeConcern: { w: 'majority' } },
);
if (removed.deletedCount !== 1) {
throw new Error('Authorization data changed during repair; verify the write gate');
}
report.removedDocuments++;
}
}
} finally {
await cursor.close();
}
}
const locks = db.collection<{ _id: string; owner: string }>('authorization_repair_locks');
const owner = randomUUID();
if (!dryRun) {
// No expiry: a slow index build must never allow a second migrator to take over.
await locks.insertOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } });
}
try {
const reports: CollectionReport[] = [];
// Inspect every collection before deleting anything, so known conflicts fail without partial cleanup.
for (const definition of DEFINITIONS) {
const current = await indexes(definition);
const name = Object.keys(definition.keys)
.map((key) => `${key}_1`)
.join('_');
if (current.some((index) => index.name === name && !sameIndex(index, definition.keys))) {
throw new Error(`Unexpected ${definition.collection} constraint; inspect before repair`);
}
const report: CollectionReport = {
collection: definition.collection,
duplicateGroups: 0,
excessDocuments: 0,
conflictingGroups: 0,
removedDocuments: 0,
uniqueIndex: current.some((index) => sameIndex(index, definition.keys)),
};
await scan(definition, report, false);
reports.push(report);
}
if (dryRun) {
return reports;
}
if (reports.some((report) => report.conflictingGroups > 0)) {
throw new Error('Conflicting authorization settings; inspect dry-run counts before repair');
}
for (const [i, definition] of DEFINITIONS.entries()) {
const report = reports[i];
const applied = { ...report, duplicateGroups: 0, excessDocuments: 0, conflictingGroups: 0 };
await scan(definition, applied, true);
report.removedDocuments = applied.removedDocuments;
if (applied.conflictingGroups > 0) {
throw new Error('Authorization settings changed after preflight; verify the write gate');
}
if (!report.uniqueIndex) {
await buildIndexWithRetry(async () => {
try {
return await db
.collection(definition.collection)
.createIndex(definition.keys, { unique: true, background: true });
} catch (error) {
// The shared retry helper logs errors; never give it a duplicate-key identity or raw query.
const code = error instanceof Error && 'code' in error ? error.code : undefined;
throw Object.assign(new Error('Authorization index build failed'), { code });
}
}, `createIndex(${definition.collection})`);
}
if (!(await indexes(definition)).some((index) => sameIndex(index, definition.keys))) {
throw new Error('Authorization unique constraint could not be verified');
}
report.uniqueIndex = true;
}
return reports;
} finally {
if (!dryRun) {
await locks.deleteOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } });
}
}
}