mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-20 06:51:59 +00:00
🗃️ fix: Restore Authorization Uniqueness and Revoke Duplicate Grants (#18258)
* fix: repair duplicate authorization records and revoke every matching grant * test: verify standalone authorization repair command and keep bootstrap isolated * test: distinguish supported aggregate disk options in DocumentDB guard
This commit is contained in:
parent
cb133ebf09
commit
8a7fe7b9f3
9 changed files with 767 additions and 1 deletions
86
UPGRADING.md
86
UPGRADING.md
|
|
@ -180,3 +180,89 @@ possible latency spike. Do not use the broad migration on a live database.
|
|||
|
||||
The original offline procedure remains available when all writers can be stopped:
|
||||
`npm run migrate:tenant-indexes`. It also covers collections beyond `mcpnames`.
|
||||
|
||||
### Repairing duplicate roles, system grants, and agent categories
|
||||
|
||||
A failed unique-index build can leave legacy duplicate authorization/configuration
|
||||
records. Updating the application does not remove those records or repair the
|
||||
missing constraint. The corrected grant revocation removes every matching grant
|
||||
in the exact requested tenant/platform scope, even before the index is repaired.
|
||||
It does not revoke grants for a different tenant or remove platform grants when
|
||||
revoking a tenant grant. Deploy the correction to every API replica first.
|
||||
|
||||
Use the targeted repair below for `systemgrants`, `roles`, and `agentcategories`.
|
||||
It does **not** repair files, messages, MCP names, or all tenant indexes. It runs
|
||||
only when invoked explicitly, never at startup.
|
||||
|
||||
1. Take a database snapshot and rehearse on a restored copy using the deployment's
|
||||
actual DocumentDB engine/version. Local regression tests use real MongoDB;
|
||||
they do not establish DocumentDB compatibility. Check archive, CRUD, aggregate,
|
||||
and index-build privileges before the production window. Inspect which regional
|
||||
deployments share a database; run once per independent writer database, not
|
||||
once per API region connected to the same database.
|
||||
2. From one maintenance container with the corrected image, the writer `MONGO_URI`
|
||||
and the deployment's TLS settings, run from `/app`:
|
||||
|
||||
```sh
|
||||
node config/repair-authorization-duplicates.js --dry-run
|
||||
```
|
||||
|
||||
Preview is also the default for `node config/repair-authorization-duplicates.js`. Output contains
|
||||
aggregate counts and index status only. Conflicting settings produce exit code
|
||||
2 in preview; operational failures produce exit code 1. A live preview is not
|
||||
a consistent snapshot and is not permission or index-build validation.
|
||||
|
||||
3. Gate and drain **all writes to these three collections**, including admin grant
|
||||
changes, role creation/edit/rename/deletion and their grant cascades, category
|
||||
changes, tenant onboarding/seeding, account/group erasure and maintenance
|
||||
scripts. Audit fork-specific writers. Freeze rollouts, restarts and automatic
|
||||
scale-up that could run `initializeRoles`, `seedSystemGrants` or
|
||||
`ensureDefaultCategories`. Role reads can create a missing built-in role:
|
||||
existing chat can continue only when its role records already exist. Gate new
|
||||
tenant provisioning and requests that would create missing roles as well.
|
||||
This is a deployment-controlled gate; there is no built-in maintenance toggle.
|
||||
Blocking only `/api/admin/grants` is insufficient. If the deployment cannot
|
||||
isolate these writers, stop writers for the repair instead of assuming a safe
|
||||
online migration. Reads and unrelated message/chat writes need not be stopped.
|
||||
4. Run one migrator, keeping the gate closed until every index is verified:
|
||||
|
||||
```sh
|
||||
node config/repair-authorization-duplicates.js --apply --writers-paused
|
||||
```
|
||||
|
||||
`--writers-paused` acknowledges the operational gate; it does not enforce it.
|
||||
The command inspects all three collections before cleanup and refuses conflicting
|
||||
records. It retains the lowest `_id` in each equivalent group without modifying
|
||||
it, archives every complete removed record in `authorization_duplicate_archive`,
|
||||
then deletes the duplicate. Role/category settings and unknown fields must agree;
|
||||
only `_id`, `__v`, `createdAt`, and `updatedAt` are ignored for comparison. Grants
|
||||
also ignore `grantedAt` and `grantedBy` provenance, which the archive preserves;
|
||||
different `expiresAt` values or tenant-field representations are conflicts.
|
||||
No permissions are unioned and no tenant is reassigned. Conflict resolution
|
||||
requires a separate reviewed decision; do not simply keep the most permissive
|
||||
role or restore every archived grant.
|
||||
|
||||
5. Require exit zero and all three `uniqueIndex` values to be true. Verify full,
|
||||
non-sparse, non-partial unique indexes on `(principalType, principalId,
|
||||
capability, tenantId)`, `(name, tenantId)`, and `(value, tenantId)` respectively.
|
||||
The repair requests background index builds serially and drops no existing/custom
|
||||
indexes. Amazon DocumentDB defaults to foreground builds; background builds
|
||||
allow reads to continue ([AWS index creation guidance](https://docs.aws.amazon.com/documentdb/latest/devguide/index-creation.html)).
|
||||
Rehearse and watch database load/latency; background does not mean zero impact.
|
||||
Re-run preview, verify zero duplicate groups, smoke-test grant/revoke plus role
|
||||
and category operations in staging, then reopen the affected writes.
|
||||
|
||||
Every removal is archived with majority acknowledgement and read back before
|
||||
source deletion. Keep that collection access-restricted with the source data;
|
||||
there is no TTL or automatic restore. A database snapshot is still required.
|
||||
A failed/interrupted run may have archived/deleted some records or built some
|
||||
indexes. Keep the gate closed and rerun preview before resuming apply. The archive
|
||||
is idempotent; an existing archive that differs from its source blocks deletion.
|
||||
|
||||
Apply takes a single-migrator lock in `authorization_repair_locks`. A hard-killed
|
||||
process leaves the `authorization` lock in place intentionally. Before removing
|
||||
that lock manually, verify the previous process and its database operations have
|
||||
finished; do not expire or steal it while an index build may still be running.
|
||||
Duplicate scans use no disk spill, a 60-second server timeout, and a 1,000-record
|
||||
per-group bound. Oversized/slow scans fail rather than performing an unbounded
|
||||
repair. These bounds are maintenance-library options, not request-path settings.
|
||||
|
|
|
|||
85
config/__tests__/repair-authorization-duplicates.spec.js
Normal file
85
config/__tests__/repair-authorization-duplicates.spec.js
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
const path = require('node:path');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const mongoose = require('mongoose');
|
||||
const { MongoMemoryServer } = require('mongodb-memory-server');
|
||||
|
||||
const root = path.resolve(__dirname, '../..');
|
||||
let mongo;
|
||||
let connection;
|
||||
|
||||
function run(...args) {
|
||||
return spawnSync(process.execPath, ['config/repair-authorization-duplicates.js', ...args], {
|
||||
cwd: root,
|
||||
env: {
|
||||
...process.env,
|
||||
MONGO_URI: mongo.getUri('authorization_cli'),
|
||||
MONGO_AUTO_INDEX: 'true',
|
||||
MONGO_AUTO_CREATE: 'true',
|
||||
},
|
||||
encoding: 'utf8',
|
||||
timeout: 20_000,
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
mongo = await MongoMemoryServer.create();
|
||||
connection = await mongoose
|
||||
.createConnection(mongo.getUri('authorization_cli'), {
|
||||
autoIndex: false,
|
||||
autoCreate: false,
|
||||
})
|
||||
.asPromise();
|
||||
});
|
||||
afterAll(async () => {
|
||||
await connection?.close();
|
||||
await mongo?.stop();
|
||||
});
|
||||
beforeEach(async () => {
|
||||
await connection.dropDatabase();
|
||||
});
|
||||
|
||||
it('previews by default without starting seeders or creating indexes, then applies explicitly', async () => {
|
||||
const roles = connection.db.collection('roles');
|
||||
await roles.insertMany([
|
||||
{ name: 'CUSTOM', permissions: {} },
|
||||
{ name: 'CUSTOM', permissions: {} },
|
||||
]);
|
||||
const preview = run();
|
||||
expect(preview.status).toBe(0);
|
||||
expect(JSON.parse(preview.stdout.trim())).toMatchObject({ dryRun: true });
|
||||
expect(await roles.countDocuments()).toBe(2);
|
||||
expect((await roles.indexes()).map((index) => index.name)).toEqual(['_id_']);
|
||||
expect((await connection.db.listCollections().toArray()).map((item) => item.name)).toEqual([
|
||||
'roles',
|
||||
]);
|
||||
|
||||
expect(run('--apply').status).toBe(1);
|
||||
expect(run('--apply', '--dry-run', '--writers-paused').status).toBe(1);
|
||||
expect(run('--aply').status).toBe(1);
|
||||
expect(await roles.countDocuments()).toBe(2);
|
||||
const applied = run('--apply', '--writers-paused');
|
||||
expect(applied.status).toBe(0);
|
||||
expect(JSON.parse(applied.stdout.trim())).toMatchObject({ dryRun: false });
|
||||
expect(await roles.countDocuments()).toBe(1);
|
||||
expect(await connection.db.collection('authorization_duplicate_archive').countDocuments()).toBe(
|
||||
1,
|
||||
);
|
||||
expect((await roles.indexes()).find((index) => index.name === 'name_1_tenantId_1').unique).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it('signals conflicts with aggregate-only output and leaves conflicting records intact', async () => {
|
||||
const roles = connection.db.collection('roles');
|
||||
await roles.insertMany([
|
||||
{ name: 'PRIVATE_ROLE_VALUE', permissions: { allow: true } },
|
||||
{ name: 'PRIVATE_ROLE_VALUE', permissions: { allow: false } },
|
||||
]);
|
||||
const preview = run('--dry-run');
|
||||
expect(preview.status).toBe(2);
|
||||
expect(preview.stdout).not.toContain('PRIVATE_ROLE_VALUE');
|
||||
const applied = run('--apply', '--writers-paused');
|
||||
expect(applied.status).toBe(1);
|
||||
expect(applied.stderr).not.toContain('PRIVATE_ROLE_VALUE');
|
||||
expect(await roles.countDocuments()).toBe(2);
|
||||
});
|
||||
50
config/repair-authorization-duplicates.js
Normal file
50
config/repair-authorization-duplicates.js
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
require('dotenv').config();
|
||||
process.env.MONGO_AUTO_INDEX = 'false';
|
||||
process.env.MONGO_AUTO_CREATE = 'false';
|
||||
const mongoose = require('mongoose');
|
||||
const { repairAuthorizationDuplicates } = require('@librechat/data-schemas');
|
||||
|
||||
(async () => {
|
||||
// Preview is the default. Apply needs both an explicit mode and a write-gate acknowledgement.
|
||||
const args = process.argv.slice(2);
|
||||
const allowed = new Set(['--dry-run', '--apply', '--writers-paused']);
|
||||
if (
|
||||
args.some((arg) => !allowed.has(arg)) ||
|
||||
(args.includes('--apply') && args.includes('--dry-run')) ||
|
||||
(args.includes('--apply') && !args.includes('--writers-paused'))
|
||||
) {
|
||||
console.error('Use --dry-run or --apply --writers-paused. See UPGRADING.md.');
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
try {
|
||||
if (!process.env.MONGO_URI) {
|
||||
throw new Error('MONGO_URI is required');
|
||||
}
|
||||
// Maintenance owns one primary connection; do not import application bootstrapping or seeders.
|
||||
await mongoose.connect(process.env.MONGO_URI, {
|
||||
autoIndex: false,
|
||||
autoCreate: false,
|
||||
readPreference: 'primary',
|
||||
maxPoolSize: 1,
|
||||
serverSelectionTimeoutMS: 10_000,
|
||||
});
|
||||
const dryRun = !args.includes('--apply');
|
||||
const reports = await repairAuthorizationDuplicates(mongoose.connection, {
|
||||
dryRun,
|
||||
writersPaused: args.includes('--writers-paused'),
|
||||
});
|
||||
console.log(JSON.stringify({ dryRun, reports }));
|
||||
if (reports.some((report) => report.conflictingGroups > 0)) {
|
||||
process.exitCode = 2;
|
||||
}
|
||||
} catch {
|
||||
// Database errors may contain principal identities or source documents.
|
||||
console.error(
|
||||
'Authorization repair failed. Keep affected writes gated; rerun dry-run and inspect database state securely.',
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
await mongoose.disconnect();
|
||||
}
|
||||
})();
|
||||
|
|
@ -212,3 +212,5 @@ export * from './postgres/import';
|
|||
export * from './postgres/mongo';
|
||||
|
||||
export { migrateMCPReservationIndex } from './migrations/mcpReservations';
|
||||
|
||||
export { repairAuthorizationDuplicates } from './migrations/authorization';
|
||||
|
|
|
|||
|
|
@ -531,6 +531,30 @@ function isUnjudgedDottedWhere(node: ts.Node): boolean {
|
|||
return !assignsCode;
|
||||
}
|
||||
|
||||
/** AWS supports this option on aggregate, but not find. Judge only direct option
|
||||
* properties in the aggregate call's second argument; nested pipeline fields and
|
||||
* unproven aliases retain the conservative token verdict.
|
||||
* https://docs.aws.amazon.com/documentdb/latest/devguide/functional-differences.html#functional-differences.lookup
|
||||
*/
|
||||
function isAggregateDiskOption(node: ts.Node): boolean {
|
||||
const property = node.parent;
|
||||
if (!ts.isPropertyAssignment(property) || property.name !== node) {
|
||||
return false;
|
||||
}
|
||||
const options = property.parent;
|
||||
if (!ts.isObjectLiteralExpression(options)) {
|
||||
return false;
|
||||
}
|
||||
const use = outermostWrapper(options).parent;
|
||||
return (
|
||||
ts.isCallExpression(use) &&
|
||||
ts.isPropertyAccessExpression(use.expression) &&
|
||||
calleeName(use) === 'aggregate' &&
|
||||
use.arguments.length >= 2 &&
|
||||
unwrapExpression(use.arguments[1]) === options
|
||||
);
|
||||
}
|
||||
|
||||
function findForbiddenTokens(sourceFile: ts.SourceFile): string[] {
|
||||
if (OPERATOR_GUARDS.has(sourceFile.fileName)) {
|
||||
return [];
|
||||
|
|
@ -543,6 +567,9 @@ function findForbiddenTokens(sourceFile: ts.SourceFile): string[] {
|
|||
(ts.isIdentifier(node) && !ts.isPropertySignature(node.parent)))
|
||||
) {
|
||||
for (const token of FORBIDDEN_TOKENS) {
|
||||
if (token === 'allowDiskUse' && node.text === token && isAggregateDiskOption(node)) {
|
||||
continue;
|
||||
}
|
||||
if (node.text === token || node.text.startsWith(`${token}.`)) {
|
||||
offenses.push(offenseAt(sourceFile, node, token));
|
||||
}
|
||||
|
|
@ -1056,6 +1083,24 @@ describe('Amazon DocumentDB compatibility', () => {
|
|||
expect(findMixedSelectStrings(parse('fixture.ts', source))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
`collection.aggregate([], { allowDiskUse: false });`,
|
||||
`collection.aggregate([], { 'allowDiskUse': true });`,
|
||||
`collection.aggregate([], ({ allowDiskUse: false } as AggregateOptions));`,
|
||||
])('accepts the supported aggregation disk option: %s', (source) => {
|
||||
expect(findForbiddenTokens(parse('fixture.ts', source))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
`collection.find({}, { allowDiskUse: false });`,
|
||||
`collection.find({}).allowDiskUse(false);`,
|
||||
`collection.aggregate([{ $project: { allowDiskUse: false } }]);`,
|
||||
`collection.aggregate([], { nested: { allowDiskUse: false } });`,
|
||||
`const options = { allowDiskUse: false }; collection.find({}, options);`,
|
||||
])('retains the disk-option guard outside proven aggregate options: %s', (source) => {
|
||||
expect(findForbiddenTokens(parse('fixture.ts', source))).not.toEqual([]);
|
||||
});
|
||||
|
||||
it('flags forbidden operators in code but not in prose or type members', () => {
|
||||
expect(
|
||||
findForbiddenTokens(parse('fixture.ts', `const projection = { x: '$$REMOVE' };`)),
|
||||
|
|
|
|||
|
|
@ -344,6 +344,53 @@ describe('systemGrant methods', () => {
|
|||
expect(grant).toBeNull();
|
||||
});
|
||||
|
||||
it('revokes every legacy duplicate without touching another tenant or capability', async () => {
|
||||
const keys = { principalType: 1, principalId: 1, capability: 1, tenantId: 1 } as const;
|
||||
await SystemGrant.collection.dropIndex(
|
||||
'principalType_1_principalId_1_capability_1_tenantId_1',
|
||||
);
|
||||
try {
|
||||
const principalId = new Types.ObjectId();
|
||||
const base = {
|
||||
principalType: PrincipalType.USER,
|
||||
principalId,
|
||||
capability: SystemCapabilities.READ_USERS,
|
||||
};
|
||||
await SystemGrant.collection.insertMany([
|
||||
{ ...base },
|
||||
{ ...base },
|
||||
{ ...base, tenantId: 'a' },
|
||||
{ ...base, tenantId: 'a' },
|
||||
{ ...base, tenantId: 'b' },
|
||||
{ ...base, capability: SystemCapabilities.READ_USAGE },
|
||||
]);
|
||||
expect(
|
||||
await methods.revokeCapability({
|
||||
...base,
|
||||
principalId: principalId.toString(),
|
||||
tenantId: 'a',
|
||||
}),
|
||||
).toEqual({ deletedCount: 2 });
|
||||
expect(await methods.revokeCapability(base)).toEqual({ deletedCount: 2 });
|
||||
expect(
|
||||
await methods.hasCapabilityForPrincipals({
|
||||
principals: [{ principalType: PrincipalType.USER, principalId }],
|
||||
capability: base.capability,
|
||||
tenantId: 'a',
|
||||
}),
|
||||
).toBe(false);
|
||||
const remaining = await SystemGrant.find({ principalId }).lean();
|
||||
expect(remaining).toHaveLength(2);
|
||||
expect(remaining.some((grant) => grant.tenantId === 'b')).toBe(true);
|
||||
expect(remaining.some((grant) => grant.capability === SystemCapabilities.READ_USAGE)).toBe(
|
||||
true,
|
||||
);
|
||||
} finally {
|
||||
await SystemGrant.deleteMany({});
|
||||
await SystemGrant.collection.createIndex(keys, { unique: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('is a no-op when the grant does not exist', async () => {
|
||||
await expect(
|
||||
methods.revokeCapability({
|
||||
|
|
|
|||
|
|
@ -432,7 +432,8 @@ export function createSystemGrantMethods(mongoose: typeof import('mongoose')): {
|
|||
};
|
||||
|
||||
const options = session ? { session } : {};
|
||||
const result = await SystemGrant.deleteOne(filter, options);
|
||||
// Legacy databases can contain duplicates when the unique index failed to build.
|
||||
const result = await SystemGrant.deleteMany(filter, options);
|
||||
return { deletedCount: result.deletedCount ?? 0 };
|
||||
}
|
||||
|
||||
|
|
|
|||
195
packages/data-schemas/src/migrations/authorization.spec.ts
Normal file
195
packages/data-schemas/src/migrations/authorization.spec.ts
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
import mongoose from 'mongoose';
|
||||
import { MongoMemoryServer } from 'mongodb-memory-server';
|
||||
import { repairAuthorizationDuplicates } from './authorization';
|
||||
|
||||
let mongo: MongoMemoryServer;
|
||||
const db = () => mongoose.connection.db!;
|
||||
const collection = (name: string) => db().collection(name);
|
||||
const apply = () =>
|
||||
repairAuthorizationDuplicates(mongoose.connection, { dryRun: false, writersPaused: true });
|
||||
|
||||
beforeAll(async () => {
|
||||
mongo = await MongoMemoryServer.create();
|
||||
await mongoose.connect(mongo.getUri(), { autoIndex: false, autoCreate: false });
|
||||
});
|
||||
afterAll(async () => {
|
||||
await mongoose.disconnect();
|
||||
await mongo.stop();
|
||||
});
|
||||
beforeEach(async () => {
|
||||
await mongoose.connection.dropDatabase();
|
||||
});
|
||||
|
||||
it('defaults to a preview without creating collections or indexes', async () => {
|
||||
const reports = await repairAuthorizationDuplicates(mongoose.connection);
|
||||
expect(reports).toHaveLength(3);
|
||||
expect(reports.every((report) => report.duplicateGroups === 0 && !report.uniqueIndex)).toBe(true);
|
||||
expect(await db().listCollections().toArray()).toEqual([]);
|
||||
await expect(
|
||||
repairAuthorizationDuplicates(mongoose.connection, { dryRun: false }),
|
||||
).rejects.toThrow('paused');
|
||||
});
|
||||
|
||||
it('archives equivalent duplicates, preserves survivors and tenant boundaries, and is resumable', async () => {
|
||||
const grants = collection('systemgrants');
|
||||
const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' };
|
||||
const first = { ...grant, grantedAt: new Date(1), grantedBy: new mongoose.Types.ObjectId() };
|
||||
const second = { ...grant, grantedAt: new Date(2), grantedBy: new mongoose.Types.ObjectId() };
|
||||
await grants.insertMany([first, second, { ...grant, tenantId: 'b' }]);
|
||||
const role = { name: 'CUSTOM', permissions: { USERS: { READ: true } } };
|
||||
await collection('roles').insertMany([role, { ...role }, { ...role, tenantId: 'b' }]);
|
||||
const category = { value: 'general', label: 'General', isActive: true };
|
||||
await collection('agentcategories').insertMany([category, { ...category }]);
|
||||
await collection('roles').createIndex({ name: 1 }, { name: 'custom_lookup' });
|
||||
const before = await grants.find().sort({ _id: 1 }).toArray();
|
||||
const preview = await repairAuthorizationDuplicates(mongoose.connection);
|
||||
expect(preview.map((report) => report.excessDocuments)).toEqual([1, 1, 1]);
|
||||
expect(await grants.countDocuments()).toBe(3);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0);
|
||||
|
||||
const result = await apply();
|
||||
expect(result.every((report) => report.uniqueIndex && report.removedDocuments === 1)).toBe(true);
|
||||
expect(await grants.findOne({ _id: before[0]._id })).toEqual(before[0]);
|
||||
expect(await grants.countDocuments()).toBe(2);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(3);
|
||||
const archive = await collection('authorization_duplicate_archive').findOne({
|
||||
'_id.collection': 'systemgrants',
|
||||
});
|
||||
expect(archive?.original).toEqual(before[1]);
|
||||
expect(await collection('roles').indexExists('custom_lookup')).toBe(true);
|
||||
await expect(grants.insertOne({ ...grant })).rejects.toMatchObject({ code: 11000 });
|
||||
await grants.insertOne({ ...grant, tenantId: 'c' });
|
||||
expect(
|
||||
(await apply()).every((report) => report.removedDocuments === 0 && report.uniqueIndex),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
'roles',
|
||||
{ name: 'CUSTOM', permissions: { allowed: true } },
|
||||
{ name: 'CUSTOM', permissions: { allowed: false } },
|
||||
],
|
||||
['agentcategories', { value: 'general', label: 'First' }, { value: 'general', label: 'Second' }],
|
||||
[
|
||||
'systemgrants',
|
||||
{ principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' },
|
||||
{
|
||||
principalType: 'role',
|
||||
principalId: 'CUSTOM',
|
||||
capability: 'read:users',
|
||||
expiresAt: new Date(),
|
||||
},
|
||||
],
|
||||
['roles', { name: 'CUSTOM' }, { name: 'CUSTOM', tenantId: null }],
|
||||
])(
|
||||
'refuses conflicting %s data before deleting from any collection',
|
||||
async (name, first, second) => {
|
||||
await collection(name).insertMany([first, second]);
|
||||
const duplicate = { principalType: 'role', principalId: 'OTHER', capability: 'read:users' };
|
||||
await collection('systemgrants').insertMany([{ ...duplicate }, { ...duplicate }]);
|
||||
const report = await repairAuthorizationDuplicates(mongoose.connection);
|
||||
expect(report.find((item) => item.collection === name)?.conflictingGroups).toBe(1);
|
||||
await expect(apply()).rejects.toThrow('Conflicting');
|
||||
expect(await collection('systemgrants').countDocuments({ principalId: 'OTHER' })).toBe(2);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(0);
|
||||
},
|
||||
);
|
||||
|
||||
it('does not delete a source when archive persistence fails', async () => {
|
||||
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
|
||||
const spy = jest
|
||||
.spyOn(mongoose.mongo.Collection.prototype, 'updateOne')
|
||||
.mockRejectedValueOnce(new Error('unavailable'));
|
||||
try {
|
||||
await expect(apply()).rejects.toThrow('unavailable');
|
||||
expect(await collection('roles').countDocuments()).toBe(2);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('resumes safely when a run stops after archival but before deletion', async () => {
|
||||
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
|
||||
const spy = jest
|
||||
.spyOn(mongoose.mongo.Collection.prototype, 'deleteOne')
|
||||
.mockRejectedValueOnce(new Error('interrupted'));
|
||||
try {
|
||||
await expect(apply()).rejects.toThrow('interrupted');
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
expect(await collection('roles').countDocuments()).toBe(2);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
|
||||
await apply();
|
||||
expect(await collection('roles').countDocuments()).toBe(1);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
|
||||
});
|
||||
|
||||
it('refuses an incompatible index and leaves duplicates untouched', async () => {
|
||||
await collection('roles').insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
|
||||
await collection('roles').createIndex({ name: 1, tenantId: 1 });
|
||||
await expect(apply()).rejects.toThrow('Unexpected roles');
|
||||
expect(await collection('roles').countDocuments()).toBe(2);
|
||||
});
|
||||
|
||||
it('bounds duplicate group memory and refuses deletion', async () => {
|
||||
await collection('roles').insertMany([
|
||||
{ name: 'CUSTOM' },
|
||||
{ name: 'CUSTOM' },
|
||||
{ name: 'CUSTOM' },
|
||||
]);
|
||||
await expect(
|
||||
repairAuthorizationDuplicates(mongoose.connection, { maxGroupSize: 2 }),
|
||||
).rejects.toThrow('bound');
|
||||
expect(await collection('roles').countDocuments()).toBe(3);
|
||||
});
|
||||
|
||||
it('refuses a second migrator without releasing its lock', async () => {
|
||||
const locks = db().collection<{ _id: string; owner: string }>('authorization_repair_locks');
|
||||
await locks.insertOne({ _id: 'authorization', owner: 'first-process' });
|
||||
await expect(apply()).rejects.toMatchObject({ code: 11000 });
|
||||
expect(await locks.findOne({ _id: 'authorization' })).toMatchObject({ owner: 'first-process' });
|
||||
});
|
||||
|
||||
it('keeps archived records and can resume after a failed index build', async () => {
|
||||
const grant = { principalType: 'role', principalId: 'CUSTOM', capability: 'read:users' };
|
||||
await collection('systemgrants').insertMany([{ ...grant }, { ...grant }]);
|
||||
const spy = jest
|
||||
.spyOn(mongoose.mongo.Collection.prototype, 'createIndex')
|
||||
.mockRejectedValueOnce(
|
||||
Object.assign(new Error('sensitive duplicate identity'), { code: 11000 }),
|
||||
);
|
||||
try {
|
||||
await expect(apply()).rejects.toThrow('Authorization index build failed');
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
expect(await collection('authorization_repair_locks').countDocuments()).toBe(0);
|
||||
expect(await collection('systemgrants').countDocuments()).toBe(1);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
|
||||
await apply();
|
||||
expect(
|
||||
await collection('systemgrants').indexExists(
|
||||
'principalType_1_principalId_1_capability_1_tenantId_1',
|
||||
),
|
||||
).toBe(true);
|
||||
expect(await collection('authorization_duplicate_archive').countDocuments()).toBe(1);
|
||||
});
|
||||
|
||||
it('never overwrites an archive with different source content', async () => {
|
||||
const roles = collection('roles');
|
||||
await roles.insertMany([{ name: 'CUSTOM' }, { name: 'CUSTOM' }]);
|
||||
const records = await roles.find().sort({ _id: 1 }).toArray();
|
||||
await db()
|
||||
.collection<{
|
||||
_id: { collection: string; sourceId: mongoose.Types.ObjectId };
|
||||
original: object;
|
||||
}>('authorization_duplicate_archive')
|
||||
.insertOne({
|
||||
_id: { collection: 'roles', sourceId: records[1]._id },
|
||||
original: { name: 'DIFFERENT' },
|
||||
});
|
||||
await expect(apply()).rejects.toThrow('archive differs');
|
||||
expect(await roles.countDocuments()).toBe(2);
|
||||
});
|
||||
255
packages/data-schemas/src/migrations/authorization.ts
Normal file
255
packages/data-schemas/src/migrations/authorization.ts
Normal file
|
|
@ -0,0 +1,255 @@
|
|||
import { randomUUID } from 'node:crypto';
|
||||
import { isDeepStrictEqual } from 'node:util';
|
||||
import type { Document, IndexDescriptionInfo, ObjectId } from 'mongodb';
|
||||
import type { Connection } from 'mongoose';
|
||||
import { buildIndexWithRetry } from '~/utils/retry';
|
||||
|
||||
interface Definition {
|
||||
collection: string;
|
||||
keys: Record<string, 1>;
|
||||
ignored: readonly string[];
|
||||
}
|
||||
|
||||
const DEFINITIONS: readonly Definition[] = [
|
||||
{
|
||||
collection: 'systemgrants',
|
||||
keys: { principalType: 1, principalId: 1, capability: 1, tenantId: 1 },
|
||||
ignored: ['grantedAt', 'grantedBy'],
|
||||
},
|
||||
{ collection: 'roles', keys: { name: 1, tenantId: 1 }, ignored: [] },
|
||||
{ collection: 'agentcategories', keys: { value: 1, tenantId: 1 }, ignored: [] },
|
||||
];
|
||||
const ARCHIVE = 'authorization_duplicate_archive';
|
||||
interface ArchivedDuplicate {
|
||||
_id: { collection: string; sourceId: ObjectId };
|
||||
original: Document;
|
||||
survivorId: ObjectId;
|
||||
archivedAt: Date;
|
||||
}
|
||||
|
||||
interface CollectionReport {
|
||||
collection: string;
|
||||
duplicateGroups: number;
|
||||
excessDocuments: number;
|
||||
conflictingGroups: number;
|
||||
removedDocuments: number;
|
||||
uniqueIndex: boolean;
|
||||
}
|
||||
|
||||
export interface AuthorizationRepairOptions {
|
||||
dryRun?: boolean;
|
||||
/** Operator acknowledgement, not an application fence. Includes seeders and background writers. */
|
||||
writersPaused?: boolean;
|
||||
/** Maintenance-only scan bounds; not request-path configuration. */
|
||||
maxTimeMS?: number;
|
||||
maxGroupSize?: number;
|
||||
}
|
||||
|
||||
function content(document: Document, ignored: readonly string[]): Document {
|
||||
return Object.fromEntries(
|
||||
Object.entries(document).filter(
|
||||
([key]) => !['_id', '__v', 'createdAt', 'updatedAt', ...ignored].includes(key),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
function sameIndex(index: IndexDescriptionInfo, keys: Definition['keys']): boolean {
|
||||
return (
|
||||
isDeepStrictEqual(Object.entries(index.key), Object.entries(keys)) &&
|
||||
index.unique === true &&
|
||||
!index.sparse &&
|
||||
!index.partialFilterExpression &&
|
||||
!index.collation
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Explicit, resumable maintenance for three small authorization/configuration collections.
|
||||
* Uses native collections to cover every tenant without compiling models or triggering seeders.
|
||||
* Never merges conflicting settings, rewrites survivors, drops indexes, or prints identities.
|
||||
* All affected writers must be gated and drained throughout apply; unrelated chat writes may continue.
|
||||
*/
|
||||
export async function repairAuthorizationDuplicates(
|
||||
connection: Connection,
|
||||
{
|
||||
dryRun = true,
|
||||
writersPaused = false,
|
||||
maxTimeMS = 60_000,
|
||||
maxGroupSize = 1_000,
|
||||
}: AuthorizationRepairOptions = {},
|
||||
): Promise<CollectionReport[]> {
|
||||
if (!dryRun && !writersPaused) {
|
||||
throw new Error('Authorization repair requires paused and drained collection writers');
|
||||
}
|
||||
if (
|
||||
!Number.isSafeInteger(maxTimeMS) ||
|
||||
maxTimeMS <= 0 ||
|
||||
!Number.isSafeInteger(maxGroupSize) ||
|
||||
maxGroupSize < 2
|
||||
) {
|
||||
throw new Error('Invalid authorization repair scan bounds');
|
||||
}
|
||||
const db = connection.db;
|
||||
if (!db) {
|
||||
throw new Error('Authorization repair requires an open database connection');
|
||||
}
|
||||
|
||||
async function indexes(definition: Definition): Promise<IndexDescriptionInfo[]> {
|
||||
try {
|
||||
return await db!
|
||||
.collection(definition.collection, { readPreference: 'primary' })
|
||||
.listIndexes({ maxTimeMS })
|
||||
.toArray();
|
||||
} catch (error) {
|
||||
if (error instanceof Error && 'code' in error && error.code === 26) {
|
||||
return [];
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function scan(
|
||||
definition: Definition,
|
||||
report: CollectionReport,
|
||||
apply: boolean,
|
||||
): Promise<void> {
|
||||
const collection = db!.collection(definition.collection, { readPreference: 'primary' });
|
||||
const archive = db!.collection<ArchivedDuplicate>(ARCHIVE, { readPreference: 'primary' });
|
||||
const identity = Object.fromEntries(
|
||||
Object.keys(definition.keys).map((key) => [key, { $ifNull: [`$${key}`, null] }]),
|
||||
);
|
||||
const cursor = collection.aggregate<{ _id: Document; count: number }>(
|
||||
[{ $group: { _id: identity, count: { $sum: 1 } } }, { $match: { count: { $gt: 1 } } }],
|
||||
{ allowDiskUse: false, maxTimeMS, batchSize: 50 },
|
||||
);
|
||||
try {
|
||||
for await (const group of cursor) {
|
||||
report.duplicateGroups++;
|
||||
report.excessDocuments += group.count - 1;
|
||||
if (group.count > maxGroupSize) {
|
||||
throw new Error('Authorization duplicate group exceeds the configured bound');
|
||||
}
|
||||
// Null matches both absent and explicit-null fields, as the unique index does.
|
||||
const records = await collection
|
||||
.find(
|
||||
Object.fromEntries(
|
||||
Object.entries(group._id).map(([key, value]) => [key, { $eq: value }]),
|
||||
),
|
||||
{ maxTimeMS },
|
||||
)
|
||||
.sort({ _id: 1 })
|
||||
.limit(maxGroupSize + 1)
|
||||
.toArray();
|
||||
if (records.length !== group.count) {
|
||||
throw new Error('Authorization data changed during repair; verify the write gate');
|
||||
}
|
||||
const [survivor, ...duplicates] = records;
|
||||
const settings = content(survivor, definition.ignored);
|
||||
if (
|
||||
duplicates.some(
|
||||
(record) => !isDeepStrictEqual(content(record, definition.ignored), settings),
|
||||
)
|
||||
) {
|
||||
report.conflictingGroups++;
|
||||
continue;
|
||||
}
|
||||
if (!apply) {
|
||||
continue;
|
||||
}
|
||||
for (const record of duplicates) {
|
||||
// Preserve complete provenance before deletion. The archive has no TTL and is not read by auth.
|
||||
const archiveId = { collection: definition.collection, sourceId: record._id };
|
||||
await archive.updateOne(
|
||||
{ _id: archiveId },
|
||||
{
|
||||
$setOnInsert: { original: record, survivorId: survivor._id, archivedAt: new Date() },
|
||||
},
|
||||
{ upsert: true, writeConcern: { w: 'majority' } },
|
||||
);
|
||||
const archived = await archive.findOne({ _id: archiveId }, { maxTimeMS });
|
||||
if (!archived || !isDeepStrictEqual(archived.original, record)) {
|
||||
throw new Error('Authorization archive differs from source; refusing deletion');
|
||||
}
|
||||
const removed = await collection.deleteOne(
|
||||
{ _id: record._id },
|
||||
{ writeConcern: { w: 'majority' } },
|
||||
);
|
||||
if (removed.deletedCount !== 1) {
|
||||
throw new Error('Authorization data changed during repair; verify the write gate');
|
||||
}
|
||||
report.removedDocuments++;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await cursor.close();
|
||||
}
|
||||
}
|
||||
|
||||
const locks = db.collection<{ _id: string; owner: string }>('authorization_repair_locks');
|
||||
const owner = randomUUID();
|
||||
if (!dryRun) {
|
||||
// No expiry: a slow index build must never allow a second migrator to take over.
|
||||
await locks.insertOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } });
|
||||
}
|
||||
try {
|
||||
const reports: CollectionReport[] = [];
|
||||
// Inspect every collection before deleting anything, so known conflicts fail without partial cleanup.
|
||||
for (const definition of DEFINITIONS) {
|
||||
const current = await indexes(definition);
|
||||
const name = Object.keys(definition.keys)
|
||||
.map((key) => `${key}_1`)
|
||||
.join('_');
|
||||
if (current.some((index) => index.name === name && !sameIndex(index, definition.keys))) {
|
||||
throw new Error(`Unexpected ${definition.collection} constraint; inspect before repair`);
|
||||
}
|
||||
const report: CollectionReport = {
|
||||
collection: definition.collection,
|
||||
duplicateGroups: 0,
|
||||
excessDocuments: 0,
|
||||
conflictingGroups: 0,
|
||||
removedDocuments: 0,
|
||||
uniqueIndex: current.some((index) => sameIndex(index, definition.keys)),
|
||||
};
|
||||
await scan(definition, report, false);
|
||||
reports.push(report);
|
||||
}
|
||||
if (dryRun) {
|
||||
return reports;
|
||||
}
|
||||
if (reports.some((report) => report.conflictingGroups > 0)) {
|
||||
throw new Error('Conflicting authorization settings; inspect dry-run counts before repair');
|
||||
}
|
||||
|
||||
for (const [i, definition] of DEFINITIONS.entries()) {
|
||||
const report = reports[i];
|
||||
const applied = { ...report, duplicateGroups: 0, excessDocuments: 0, conflictingGroups: 0 };
|
||||
await scan(definition, applied, true);
|
||||
report.removedDocuments = applied.removedDocuments;
|
||||
if (applied.conflictingGroups > 0) {
|
||||
throw new Error('Authorization settings changed after preflight; verify the write gate');
|
||||
}
|
||||
if (!report.uniqueIndex) {
|
||||
await buildIndexWithRetry(async () => {
|
||||
try {
|
||||
return await db
|
||||
.collection(definition.collection)
|
||||
.createIndex(definition.keys, { unique: true, background: true });
|
||||
} catch (error) {
|
||||
// The shared retry helper logs errors; never give it a duplicate-key identity or raw query.
|
||||
const code = error instanceof Error && 'code' in error ? error.code : undefined;
|
||||
throw Object.assign(new Error('Authorization index build failed'), { code });
|
||||
}
|
||||
}, `createIndex(${definition.collection})`);
|
||||
}
|
||||
if (!(await indexes(definition)).some((index) => sameIndex(index, definition.keys))) {
|
||||
throw new Error('Authorization unique constraint could not be verified');
|
||||
}
|
||||
report.uniqueIndex = true;
|
||||
}
|
||||
return reports;
|
||||
} finally {
|
||||
if (!dryRun) {
|
||||
await locks.deleteOne({ _id: 'authorization', owner }, { writeConcern: { w: 'majority' } });
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue