diff --git a/UPGRADING.md b/UPGRADING.md index 85d49f38f3..239b7c9019 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -180,3 +180,89 @@ possible latency spike. Do not use the broad migration on a live database. The original offline procedure remains available when all writers can be stopped: `npm run migrate:tenant-indexes`. It also covers collections beyond `mcpnames`. + +### Repairing duplicate roles, system grants, and agent categories + +A failed unique-index build can leave legacy duplicate authorization/configuration +records. Updating the application does not remove those records or repair the +missing constraint. The corrected grant revocation removes every matching grant +in the exact requested tenant/platform scope, even before the index is repaired. +It does not revoke grants for a different tenant or remove platform grants when +revoking a tenant grant. Deploy the correction to every API replica first. + +Use the targeted repair below for `systemgrants`, `roles`, and `agentcategories`. +It does **not** repair files, messages, MCP names, or all tenant indexes. It runs +only when invoked explicitly, never at startup. + +1. Take a database snapshot and rehearse on a restored copy using the deployment's + actual DocumentDB engine/version. Local regression tests use real MongoDB; + they do not establish DocumentDB compatibility. Check archive, CRUD, aggregate, + and index-build privileges before the production window. Inspect which regional + deployments share a database; run once per independent writer database, not + once per API region connected to the same database. +2. From one maintenance container with the corrected image, the writer `MONGO_URI` + and the deployment's TLS settings, run from `/app`: + + ```sh + node config/repair-authorization-duplicates.js --dry-run + ``` + + Preview is also the default for `node config/repair-authorization-duplicates.js`. Output contains + aggregate counts and index status only. Conflicting settings produce exit code + 2 in preview; operational failures produce exit code 1. A live preview is not + a consistent snapshot and is not permission or index-build validation. + +3. Gate and drain **all writes to these three collections**, including admin grant + changes, role creation/edit/rename/deletion and their grant cascades, category + changes, tenant onboarding/seeding, account/group erasure and maintenance + scripts. Audit fork-specific writers. Freeze rollouts, restarts and automatic + scale-up that could run `initializeRoles`, `seedSystemGrants` or + `ensureDefaultCategories`. Role reads can create a missing built-in role: + existing chat can continue only when its role records already exist. Gate new + tenant provisioning and requests that would create missing roles as well. + This is a deployment-controlled gate; there is no built-in maintenance toggle. + Blocking only `/api/admin/grants` is insufficient. If the deployment cannot + isolate these writers, stop writers for the repair instead of assuming a safe + online migration. Reads and unrelated message/chat writes need not be stopped. +4. Run one migrator, keeping the gate closed until every index is verified: + + ```sh + node config/repair-authorization-duplicates.js --apply --writers-paused + ``` + + `--writers-paused` acknowledges the operational gate; it does not enforce it. + The command inspects all three collections before cleanup and refuses conflicting + records. It retains the lowest `_id` in each equivalent group without modifying + it, archives every complete removed record in `authorization_duplicate_archive`, + then deletes the duplicate. Role/category settings and unknown fields must agree; + only `_id`, `__v`, `createdAt`, and `updatedAt` are ignored for comparison. Grants + also ignore `grantedAt` and `grantedBy` provenance, which the archive preserves; + different `expiresAt` values or tenant-field representations are conflicts. + No permissions are unioned and no tenant is reassigned. Conflict resolution + requires a separate reviewed decision; do not simply keep the most permissive + role or restore every archived grant. + +5. Require exit zero and all three `uniqueIndex` values to be true. Verify full, + non-sparse, non-partial unique indexes on `(principalType, principalId, +capability, tenantId)`, `(name, tenantId)`, and `(value, tenantId)` respectively. + The repair requests background index builds serially and drops no existing/custom + indexes. Amazon DocumentDB defaults to foreground builds; background builds + allow reads to continue ([AWS index creation guidance](https://docs.aws.amazon.com/documentdb/latest/devguide/index-creation.html)). + Rehearse and watch database load/latency; background does not mean zero impact. + Re-run preview, verify zero duplicate groups, smoke-test grant/revoke plus role + and category operations in staging, then reopen the affected writes. + +Every removal is archived with majority acknowledgement and read back before +source deletion. Keep that collection access-restricted with the source data; +there is no TTL or automatic restore. A database snapshot is still required. +A failed/interrupted run may have archived/deleted some records or built some +indexes. Keep the gate closed and rerun preview before resuming apply. The archive +is idempotent; an existing archive that differs from its source blocks deletion. + +Apply takes a single-migrator lock in `authorization_repair_locks`. A hard-killed +process leaves the `authorization` lock in place intentionally. Before removing +that lock manually, verify the previous process and its database operations have +finished; do not expire or steal it while an index build may still be running. +Duplicate scans use no disk spill, a 60-second server timeout, and a 1,000-record +per-group bound. Oversized/slow scans fail rather than performing an unbounded +repair. These bounds are maintenance-library options, not request-path settings. diff --git a/config/__tests__/repair-authorization-duplicates.spec.js b/config/__tests__/repair-authorization-duplicates.spec.js new file mode 100644 index 0000000000..e260dceabc --- /dev/null +++ b/config/__tests__/repair-authorization-duplicates.spec.js @@ -0,0 +1,85 @@ +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); +const mongoose = require('mongoose'); +const { MongoMemoryServer } = require('mongodb-memory-server'); + +const root = path.resolve(__dirname, '../..'); +let mongo; +let connection; + +function run(...args) { + return spawnSync(process.execPath, ['config/repair-authorization-duplicates.js', ...args], { + cwd: root, + env: { + ...process.env, + MONGO_URI: mongo.getUri('authorization_cli'), + MONGO_AUTO_INDEX: 'true', + MONGO_AUTO_CREATE: 'true', + }, + encoding: 'utf8', + timeout: 20_000, + }); +} + +beforeAll(async () => { + mongo = await MongoMemoryServer.create(); + connection = await mongoose + .createConnection(mongo.getUri('authorization_cli'), { + autoIndex: false, + autoCreate: false, + }) + .asPromise(); +}); +afterAll(async () => { + await connection?.close(); + await mongo?.stop(); +}); +beforeEach(async () => { + await connection.dropDatabase(); +}); + +it('previews by default without starting seeders or creating indexes, then applies explicitly', async () => { + const roles = connection.db.collection('roles'); + await roles.insertMany([ + { name: 'CUSTOM', permissions: {} }, + { name: 'CUSTOM', permissions: {} }, + ]); + const preview = run(); + expect(preview.status).toBe(0); + expect(JSON.parse(preview.stdout.trim())).toMatchObject({ dryRun: true }); + expect(await roles.countDocuments()).toBe(2); + expect((await roles.indexes()).map((index) => index.name)).toEqual(['_id_']); + expect((await connection.db.listCollections().toArray()).map((item) => item.name)).toEqual([ + 'roles', + ]); + + expect(run('--apply').status).toBe(1); + expect(run('--apply', '--dry-run', '--writers-paused').status).toBe(1); + expect(run('--aply').status).toBe(1); + expect(await roles.countDocuments()).toBe(2); + const applied = run('--apply', '--writers-paused'); + expect(applied.status).toBe(0); + expect(JSON.parse(applied.stdout.trim())).toMatchObject({ dryRun: false }); + expect(await roles.countDocuments()).toBe(1); + expect(await connection.db.collection('authorization_duplicate_archive').countDocuments()).toBe( + 1, + ); + expect((await roles.indexes()).find((index) => index.name === 'name_1_tenantId_1').unique).toBe( + true, + ); +}); + +it('signals conflicts with aggregate-only output and leaves conflicting records intact', async () => { + const roles = connection.db.collection('roles'); + await roles.insertMany([ + { name: 'PRIVATE_ROLE_VALUE', permissions: { allow: true } }, + { name: 'PRIVATE_ROLE_VALUE', permissions: { allow: false } }, + ]); + const preview = run('--dry-run'); + expect(preview.status).toBe(2); + expect(preview.stdout).not.toContain('PRIVATE_ROLE_VALUE'); + const applied = run('--apply', '--writers-paused'); + expect(applied.status).toBe(1); + expect(applied.stderr).not.toContain('PRIVATE_ROLE_VALUE'); + expect(await roles.countDocuments()).toBe(2); +}); diff --git a/config/repair-authorization-duplicates.js b/config/repair-authorization-duplicates.js new file mode 100644 index 0000000000..8450c60743 --- /dev/null +++ b/config/repair-authorization-duplicates.js @@ -0,0 +1,50 @@ +require('dotenv').config(); +process.env.MONGO_AUTO_INDEX = 'false'; +process.env.MONGO_AUTO_CREATE = 'false'; +const mongoose = require('mongoose'); +const { repairAuthorizationDuplicates } = require('@librechat/data-schemas'); + +(async () => { + // Preview is the default. Apply needs both an explicit mode and a write-gate acknowledgement. + const args = process.argv.slice(2); + const allowed = new Set(['--dry-run', '--apply', '--writers-paused']); + if ( + args.some((arg) => !allowed.has(arg)) || + (args.includes('--apply') && args.includes('--dry-run')) || + (args.includes('--apply') && !args.includes('--writers-paused')) + ) { + console.error('Use --dry-run or --apply --writers-paused. See UPGRADING.md.'); + process.exitCode = 1; + return; + } + try { + if (!process.env.MONGO_URI) { + throw new Error('MONGO_URI is required'); + } + // Maintenance owns one primary connection; do not import application bootstrapping or seeders. + await mongoose.connect(process.env.MONGO_URI, { + autoIndex: false, + autoCreate: false, + readPreference: 'primary', + maxPoolSize: 1, + serverSelectionTimeoutMS: 10_000, + }); + const dryRun = !args.includes('--apply'); + const reports = await repairAuthorizationDuplicates(mongoose.connection, { + dryRun, + writersPaused: args.includes('--writers-paused'), + }); + console.log(JSON.stringify({ dryRun, reports })); + if (reports.some((report) => report.conflictingGroups > 0)) { + process.exitCode = 2; + } + } catch { + // Database errors may contain principal identities or source documents. + console.error( + 'Authorization repair failed. Keep affected writes gated; rerun dry-run and inspect database state securely.', + ); + process.exitCode = 1; + } finally { + await mongoose.disconnect(); + } +})(); diff --git a/packages/data-schemas/src/index.ts b/packages/data-schemas/src/index.ts index 12ae518203..969de71b20 100644 --- a/packages/data-schemas/src/index.ts +++ b/packages/data-schemas/src/index.ts @@ -212,3 +212,5 @@ export * from './postgres/import'; export * from './postgres/mongo'; export { migrateMCPReservationIndex } from './migrations/mcpReservations'; + +export { repairAuthorizationDuplicates } from './migrations/authorization'; diff --git a/packages/data-schemas/src/methods/documentdb.spec.ts b/packages/data-schemas/src/methods/documentdb.spec.ts index 5d2a3826f3..c528b81f44 100644 --- a/packages/data-schemas/src/methods/documentdb.spec.ts +++ b/packages/data-schemas/src/methods/documentdb.spec.ts @@ -531,6 +531,30 @@ function isUnjudgedDottedWhere(node: ts.Node): boolean { return !assignsCode; } +/** AWS supports this option on aggregate, but not find. Judge only direct option + * properties in the aggregate call's second argument; nested pipeline fields and + * unproven aliases retain the conservative token verdict. + * https://docs.aws.amazon.com/documentdb/latest/devguide/functional-differences.html#functional-differences.lookup + */ +function isAggregateDiskOption(node: ts.Node): boolean { + const property = node.parent; + if (!ts.isPropertyAssignment(property) || property.name !== node) { + return false; + } + const options = property.parent; + if (!ts.isObjectLiteralExpression(options)) { + return false; + } + const use = outermostWrapper(options).parent; + return ( + ts.isCallExpression(use) && + ts.isPropertyAccessExpression(use.expression) && + calleeName(use) === 'aggregate' && + use.arguments.length >= 2 && + unwrapExpression(use.arguments[1]) === options + ); +} + function findForbiddenTokens(sourceFile: ts.SourceFile): string[] { if (OPERATOR_GUARDS.has(sourceFile.fileName)) { return []; @@ -543,6 +567,9 @@ function findForbiddenTokens(sourceFile: ts.SourceFile): string[] { (ts.isIdentifier(node) && !ts.isPropertySignature(node.parent))) ) { for (const token of FORBIDDEN_TOKENS) { + if (token === 'allowDiskUse' && node.text === token && isAggregateDiskOption(node)) { + continue; + } if (node.text === token || node.text.startsWith(`${token}.`)) { offenses.push(offenseAt(sourceFile, node, token)); } @@ -1056,6 +1083,24 @@ describe('Amazon DocumentDB compatibility', () => { expect(findMixedSelectStrings(parse('fixture.ts', source))).toEqual([]); }); + it.each([ + `collection.aggregate([], { allowDiskUse: false });`, + `collection.aggregate([], { 'allowDiskUse': true });`, + `collection.aggregate([], ({ allowDiskUse: false } as AggregateOptions));`, + ])('accepts the supported aggregation disk option: %s', (source) => { + expect(findForbiddenTokens(parse('fixture.ts', source))).toEqual([]); + }); + + it.each([ + `collection.find({}, { allowDiskUse: false });`, + `collection.find({}).allowDiskUse(false);`, + `collection.aggregate([{ $project: { allowDiskUse: false } }]);`, + `collection.aggregate([], { nested: { allowDiskUse: false } });`, + `const options = { allowDiskUse: false }; collection.find({}, options);`, + ])('retains the disk-option guard outside proven aggregate options: %s', (source) => { + expect(findForbiddenTokens(parse('fixture.ts', source))).not.toEqual([]); + }); + it('flags forbidden operators in code but not in prose or type members', () => { expect( findForbiddenTokens(parse('fixture.ts', `const projection = { x: '$$REMOVE' };`)), diff --git a/packages/data-schemas/src/methods/systemGrant.spec.ts b/packages/data-schemas/src/methods/systemGrant.spec.ts index 5f6d1130d2..70d550c5fb 100644 --- a/packages/data-schemas/src/methods/systemGrant.spec.ts +++ b/packages/data-schemas/src/methods/systemGrant.spec.ts @@ -344,6 +344,53 @@ describe('systemGrant methods', () => { expect(grant).toBeNull(); }); + it('revokes every legacy duplicate without touching another tenant or capability', async () => { + const keys = { principalType: 1, principalId: 1, capability: 1, tenantId: 1 } as const; + await SystemGrant.collection.dropIndex( + 'principalType_1_principalId_1_capability_1_tenantId_1', + ); + try { + const principalId = new Types.ObjectId(); + const base = { + principalType: PrincipalType.USER, + principalId, + capability: SystemCapabilities.READ_USERS, + }; + await SystemGrant.collection.insertMany([ + { ...base }, + { ...base }, + { ...base, tenantId: 'a' }, + { ...base, tenantId: 'a' }, + { ...base, tenantId: 'b' }, + { ...base, capability: SystemCapabilities.READ_USAGE }, + ]); + expect( + await methods.revokeCapability({ + ...base, + principalId: principalId.toString(), + tenantId: 'a', + }), + ).toEqual({ deletedCount: 2 }); + expect(await methods.revokeCapability(base)).toEqual({ deletedCount: 2 }); + expect( + await methods.hasCapabilityForPrincipals({ + principals: [{ principalType: PrincipalType.USER, principalId }], + capability: base.capability, + tenantId: 'a', + }), + ).toBe(false); + const remaining = await SystemGrant.find({ principalId }).lean(); + expect(remaining).toHaveLength(2); + expect(remaining.some((grant) => grant.tenantId === 'b')).toBe(true); + expect(remaining.some((grant) => grant.capability === SystemCapabilities.READ_USAGE)).toBe( + true, + ); + } finally { + await SystemGrant.deleteMany({}); + await SystemGrant.collection.createIndex(keys, { unique: true }); + } + }); + it('is a no-op when the grant does not exist', async () => { await expect( methods.revokeCapability({ diff --git a/packages/data-schemas/src/methods/systemGrant.ts b/packages/data-schemas/src/methods/systemGrant.ts index 76eea9dad7..5c536d9329 100644 --- a/packages/data-schemas/src/methods/systemGrant.ts +++ b/packages/data-schemas/src/methods/systemGrant.ts @@ -432,7 +432,8 @@ export function createSystemGrantMethods(mongoose: typeof import('mongoose')): { }; const options = session ? { session } : {}; - const result = await SystemGrant.deleteOne(filter, options); + // Legacy databases can contain duplicates when the unique index failed to build. + const result = await SystemGrant.deleteMany(filter, options); return { deletedCount: result.deletedCount ?? 0 }; } diff --git a/packages/data-schemas/src/migrations/authorization.spec.ts b/packages/data-schemas/src/migrations/authorization.spec.ts new file mode 100644 index 0000000000..71d3d14316 --- /dev/null +++ b/packages/data-schemas/src/migrations/authorization.spec.ts @@ -0,0 +1,195 @@ +import mongoose from 'mongoose'; +import { MongoMemoryServer } from 'mongodb-memory-server'; +import { repairAuthorizationDuplicates } from './authorization'; + +let mongo: MongoMemoryServer; +const db = () => mongoose.connection.db!; +const collection = (name: string) => db().collection(name); +const apply = () => + repairAuthorizationDuplicates(mongoose.connection, { dryRun: false, writersPaused: true }); + +beforeAll(async () => { + mongo = await MongoMemoryServer.create(); + await mongoose.connect(mongo.getUri(), { autoIndex: false, autoCreate: false }); +}); +afterAll(async () => { + await mongoose.disconnect(); + await mongo.stop(); +}); +beforeEach(async () => { + await mongoose.connection.dropDatabase(); +}); + +it('defaults to a preview without creating collections or indexes', async () => { + const reports = await repairAuthorizationDuplicates(mongoose.connection); + expect(reports).toHaveLength(3); + expect(reports.every((report) => report.duplicateGroups === 0 && !report.uniqueIndex)).toBe(true); + expect(await db().listCollections().toArray()).toEqual([]); + await expect( + repairAuthorizationDuplicates(mongoose.connection, { dryRun: false }), + ).rejects.toThrow('paused'); +}); + +it('archives equivalent duplicates, preserves survivors and tenant boundaries, and is resumable', async () => { + const grants = collection('systemgrants'); + const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' }; + const first = { ...grant, grantedAt: new Date(1), grantedBy: new mongoose.Types.ObjectId() }; + const second = { ...grant, grantedAt: new Date(2), grantedBy: new mongoose.Types.ObjectId() }; + await grants.insertMany([first, second, { ...grant, tenantId: 'b' }]); + const role = { name: 'CUSTOM', permissions: { USERS: { READ: true } } }; + await collection('roles').insertMany([role, { ...role }, { ...role, tenantId: 'b' }]); + const category = { value: 'general', label: 'General', isActive: true }; + await collection('agentcategories').insertMany([category, { ...category }]); + await collection('roles').createIndex({ name: 1 }, { name: 'custom_lookup' }); + const before = await grants.find().sort({ _id: 1 }).toArray(); + const preview = await repairAuthorizationDuplicates(mongoose.connection); + expect(preview.map((report) => report.excessDocuments)).toEqual([1, 1, 1]); + expect(await grants.countDocuments()).toBe(3); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0); + + const result = await apply(); + expect(result.every((report) => report.uniqueIndex && report.removedDocuments === 1)).toBe(true); + expect(await grants.findOne({ _id: before[0]._id })).toEqual(before[0]); + expect(await grants.countDocuments()).toBe(2); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(3); + const archive = await collection('authorization_duplicate_archive').findOne({ + '_id.collection': 'systemgrants', + }); + expect(archive?.original).toEqual(before[1]); + expect(await collection('roles').indexExists('custom_lookup')).toBe(true); + await expect(grants.insertOne({ ...grant })).rejects.toMatchObject({ code: 11000 }); + await grants.insertOne({ ...grant, tenantId: 'c' }); + expect( + (await apply()).every((report) => report.removedDocuments === 0 && report.uniqueIndex), + ).toBe(true); +}); + +it.each([ + [ + 'roles', + { name: 'CUSTOM', permissions: { allowed: true } }, + { name: 'CUSTOM', permissions: { allowed: false } }, + ], + ['agentcategories', { value: 'general', label: 'First' }, { value: 'general', label: 'Second' }], + [ + 'systemgrants', + { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' }, + { + principalType: 'role', + principalId: 'CUSTOM', + capability: 'read:users', + expiresAt: new Date(), + }, + ], + ['roles', { name: 'CUSTOM' }, { name: 'CUSTOM', tenantId: null }], +])( + 'refuses conflicting %s data before deleting from any collection', + async (name, first, second) => { + await collection(name).insertMany([first, second]); + const duplicate = { principalType: 'role', principalId: 'OTHER', capability: 'read:users' }; + await collection('systemgrants').insertMany([{ ...duplicate }, { ...duplicate }]); + const report = await repairAuthorizationDuplicates(mongoose.connection); + expect(report.find((item) => item.collection === name)?.conflictingGroups).toBe(1); + await expect(apply()).rejects.toThrow('Conflicting'); + expect(await collection('systemgrants').countDocuments({ principalId: 'OTHER' })).toBe(2); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0); + }, +); + +it('does not delete a source when archive persistence fails', async () => { + await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]); + const spy = jest + .spyOn(mongoose.mongo.Collection.prototype, 'updateOne') + .mockRejectedValueOnce(new Error('unavailable')); + try { + await expect(apply()).rejects.toThrow('unavailable'); + expect(await collection('roles').countDocuments()).toBe(2); + } finally { + spy.mockRestore(); + } +}); + +it('resumes safely when a run stops after archival but before deletion', async () => { + await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]); + const spy = jest + .spyOn(mongoose.mongo.Collection.prototype, 'deleteOne') + .mockRejectedValueOnce(new Error('interrupted')); + try { + await expect(apply()).rejects.toThrow('interrupted'); + } finally { + spy.mockRestore(); + } + expect(await collection('roles').countDocuments()).toBe(2); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1); + await apply(); + expect(await collection('roles').countDocuments()).toBe(1); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1); +}); + +it('refuses an incompatible index and leaves duplicates untouched', async () => { + await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]); + await collection('roles').createIndex({ name: 1, tenantId: 1 }); + await expect(apply()).rejects.toThrow('Unexpected roles'); + expect(await collection('roles').countDocuments()).toBe(2); +}); + +it('bounds duplicate group memory and refuses deletion', async () => { + await collection('roles').insertMany([ + { name: 'CUSTOM' }, + { name: 'CUSTOM' }, + { name: 'CUSTOM' }, + ]); + await expect( + repairAuthorizationDuplicates(mongoose.connection, { maxGroupSize: 2 }), + ).rejects.toThrow('bound'); + expect(await collection('roles').countDocuments()).toBe(3); +}); + +it('refuses a second migrator without releasing its lock', async () => { + const locks = db().collection<{ _id: string; owner: string }>('authorization_repair_locks'); + await locks.insertOne({ _id: 'authorization', owner: 'first-process' }); + await expect(apply()).rejects.toMatchObject({ code: 11000 }); + expect(await locks.findOne({ _id: 'authorization' })).toMatchObject({ owner: 'first-process' }); +}); + +it('keeps archived records and can resume after a failed index build', async () => { + const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' }; + await collection('systemgrants').insertMany([{ ...grant }, { ...grant }]); + const spy = jest + .spyOn(mongoose.mongo.Collection.prototype, 'createIndex') + .mockRejectedValueOnce( + Object.assign(new Error('sensitive duplicate identity'), { code: 11000 }), + ); + try { + await expect(apply()).rejects.toThrow('Authorization index build failed'); + } finally { + spy.mockRestore(); + } + expect(await collection('authorization_repair_locks').countDocuments()).toBe(0); + expect(await collection('systemgrants').countDocuments()).toBe(1); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1); + await apply(); + expect( + await collection('systemgrants').indexExists( + 'principalType_1_principalId_1_capability_1_tenantId_1', + ), + ).toBe(true); + expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1); +}); + +it('never overwrites an archive with different source content', async () => { + const roles = collection('roles'); + await roles.insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]); + const records = await roles.find().sort({ _id: 1 }).toArray(); + await db() + .collection<{ + _id: { collection: string; sourceId: mongoose.Types.ObjectId }; + original: object; + }>('authorization_duplicate_archive') + .insertOne({ + _id: { collection: 'roles', sourceId: records[1]._id }, + original: { name: 'DIFFERENT' }, + }); + await expect(apply()).rejects.toThrow('archive differs'); + expect(await roles.countDocuments()).toBe(2); +}); diff --git a/packages/data-schemas/src/migrations/authorization.ts b/packages/data-schemas/src/migrations/authorization.ts new file mode 100644 index 0000000000..9bed166398 --- /dev/null +++ b/packages/data-schemas/src/migrations/authorization.ts @@ -0,0 +1,255 @@ +import { randomUUID } from 'node:crypto'; +import { isDeepStrictEqual } from 'node:util'; +import type { Document, IndexDescriptionInfo, ObjectId } from 'mongodb'; +import type { Connection } from 'mongoose'; +import { buildIndexWithRetry } from '~/utils/retry'; + +interface Definition { + collection: string; + keys: Record; + ignored: readonly string[]; +} + +const DEFINITIONS: readonly Definition[] = [ + { + collection: 'systemgrants', + keys: { principalType: 1, principalId: 1, capability: 1, tenantId: 1 }, + ignored: ['grantedAt', 'grantedBy'], + }, + { collection: 'roles', keys: { name: 1, tenantId: 1 }, ignored: [] }, + { collection: 'agentcategories', keys: { value: 1, tenantId: 1 }, ignored: [] }, +]; +const ARCHIVE = 'authorization_duplicate_archive'; +interface ArchivedDuplicate { + _id: { collection: string; sourceId: ObjectId }; + original: Document; + survivorId: ObjectId; + archivedAt: Date; +} + +interface CollectionReport { + collection: string; + duplicateGroups: number; + excessDocuments: number; + conflictingGroups: number; + removedDocuments: number; + uniqueIndex: boolean; +} + +export interface AuthorizationRepairOptions { + dryRun?: boolean; + /** Operator acknowledgement, not an application fence. Includes seeders and background writers. */ + writersPaused?: boolean; + /** Maintenance-only scan bounds; not request-path configuration. */ + maxTimeMS?: number; + maxGroupSize?: number; +} + +function content(document: Document, ignored: readonly string[]): Document { + return Object.fromEntries( + Object.entries(document).filter( + ([key]) => !['_id', '__v', 'createdAt', 'updatedAt', ...ignored].includes(key), + ), + ); +} + +function sameIndex(index: IndexDescriptionInfo, keys: Definition['keys']): boolean { + return ( + isDeepStrictEqual(Object.entries(index.key), Object.entries(keys)) && + index.unique === true && + !index.sparse && + !index.partialFilterExpression && + !index.collation + ); +} + +/** + * Explicit, resumable maintenance for three small authorization/configuration collections. + * Uses native collections to cover every tenant without compiling models or triggering seeders. + * Never merges conflicting settings, rewrites survivors, drops indexes, or prints identities. + * All affected writers must be gated and drained throughout apply; unrelated chat writes may continue. + */ +export async function repairAuthorizationDuplicates( + connection: Connection, + { + dryRun = true, + writersPaused = false, + maxTimeMS = 60_000, + maxGroupSize = 1_000, + }: AuthorizationRepairOptions = {}, +): Promise { + if (!dryRun && !writersPaused) { + throw new Error('Authorization repair requires paused and drained collection writers'); + } + if ( + !Number.isSafeInteger(maxTimeMS) || + maxTimeMS <= 0 || + !Number.isSafeInteger(maxGroupSize) || + maxGroupSize < 2 + ) { + throw new Error('Invalid authorization repair scan bounds'); + } + const db = connection.db; + if (!db) { + throw new Error('Authorization repair requires an open database connection'); + } + + async function indexes(definition: Definition): Promise { + try { + return await db! + .collection(definition.collection, { readPreference: 'primary' }) + .listIndexes({ maxTimeMS }) + .toArray(); + } catch (error) { + if (error instanceof Error && 'code' in error && error.code === 26) { + return []; + } + throw error; + } + } + + async function scan( + definition: Definition, + report: CollectionReport, + apply: boolean, + ): Promise { + const collection = db!.collection(definition.collection, { readPreference: 'primary' }); + const archive = db!.collection(ARCHIVE, { readPreference: 'primary' }); + const identity = Object.fromEntries( + Object.keys(definition.keys).map((key) => [key, { $ifNull: [`$${key}`, null] }]), + ); + const cursor = collection.aggregate<{ _id: Document; count: number }>( + [{ $group: { _id: identity, count: { $sum: 1 } } }, { $match: { count: { $gt: 1 } } }], + { allowDiskUse: false, maxTimeMS, batchSize: 50 }, + ); + try { + for await (const group of cursor) { + report.duplicateGroups++; + report.excessDocuments += group.count - 1; + if (group.count > maxGroupSize) { + throw new Error('Authorization duplicate group exceeds the configured bound'); + } + // Null matches both absent and explicit-null fields, as the unique index does. + const records = await collection + .find( + Object.fromEntries( + Object.entries(group._id).map(([key, value]) => [key, { $eq: value }]), + ), + { maxTimeMS }, + ) + .sort({ _id: 1 }) + .limit(maxGroupSize + 1) + .toArray(); + if (records.length !== group.count) { + throw new Error('Authorization data changed during repair; verify the write gate'); + } + const [survivor, ...duplicates] = records; + const settings = content(survivor, definition.ignored); + if ( + duplicates.some( + (record) => !isDeepStrictEqual(content(record, definition.ignored), settings), + ) + ) { + report.conflictingGroups++; + continue; + } + if (!apply) { + continue; + } + for (const record of duplicates) { + // Preserve complete provenance before deletion. The archive has no TTL and is not read by auth. + const archiveId = { collection: definition.collection, sourceId: record._id }; + await archive.updateOne( + { _id: archiveId }, + { + $setOnInsert: { original: record, survivorId: survivor._id, archivedAt: new Date() }, + }, + { upsert: true, writeConcern: { w: 'majority' } }, + ); + const archived = await archive.findOne({ _id: archiveId }, { maxTimeMS }); + if (!archived || !isDeepStrictEqual(archived.original, record)) { + throw new Error('Authorization archive differs from source; refusing deletion'); + } + const removed = await collection.deleteOne( + { _id: record._id }, + { writeConcern: { w: 'majority' } }, + ); + if (removed.deletedCount !== 1) { + throw new Error('Authorization data changed during repair; verify the write gate'); + } + report.removedDocuments++; + } + } + } finally { + await cursor.close(); + } + } + + const locks = db.collection<{ _id: string; owner: string }>('authorization_repair_locks'); + const owner = randomUUID(); + if (!dryRun) { + // No expiry: a slow index build must never allow a second migrator to take over. + await locks.insertOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } }); + } + try { + const reports: CollectionReport[] = []; + // Inspect every collection before deleting anything, so known conflicts fail without partial cleanup. + for (const definition of DEFINITIONS) { + const current = await indexes(definition); + const name = Object.keys(definition.keys) + .map((key) => `${key}_1`) + .join('_'); + if (current.some((index) => index.name === name && !sameIndex(index, definition.keys))) { + throw new Error(`Unexpected ${definition.collection} constraint; inspect before repair`); + } + const report: CollectionReport = { + collection: definition.collection, + duplicateGroups: 0, + excessDocuments: 0, + conflictingGroups: 0, + removedDocuments: 0, + uniqueIndex: current.some((index) => sameIndex(index, definition.keys)), + }; + await scan(definition, report, false); + reports.push(report); + } + if (dryRun) { + return reports; + } + if (reports.some((report) => report.conflictingGroups > 0)) { + throw new Error('Conflicting authorization settings; inspect dry-run counts before repair'); + } + + for (const [i, definition] of DEFINITIONS.entries()) { + const report = reports[i]; + const applied = { ...report, duplicateGroups: 0, excessDocuments: 0, conflictingGroups: 0 }; + await scan(definition, applied, true); + report.removedDocuments = applied.removedDocuments; + if (applied.conflictingGroups > 0) { + throw new Error('Authorization settings changed after preflight; verify the write gate'); + } + if (!report.uniqueIndex) { + await buildIndexWithRetry(async () => { + try { + return await db + .collection(definition.collection) + .createIndex(definition.keys, { unique: true, background: true }); + } catch (error) { + // The shared retry helper logs errors; never give it a duplicate-key identity or raw query. + const code = error instanceof Error && 'code' in error ? error.code : undefined; + throw Object.assign(new Error('Authorization index build failed'), { code }); + } + }, `createIndex(${definition.collection})`); + } + if (!(await indexes(definition)).some((index) => sameIndex(index, definition.keys))) { + throw new Error('Authorization unique constraint could not be verified'); + } + report.uniqueIndex = true; + } + return reports; + } finally { + if (!dryRun) { + await locks.deleteOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } }); + } + } +}