LibreChat/packages/data-schemas
Marco Beretta 92a8058f02
🛟 fix: Isolate Invalid Skills During GitHub Sync (#14735)
* fix: treat unrecognized SKILL.md frontmatter keys as warnings

An unknown key in one SKILL.md failed that skill outright, and because the
GitHub sync runner marks a source failed on any validation error, a single
stray key took down every other skill in the repository. Syncing
github.com/cloudflare/skills failed entirely because 2 of its 13 skills
carry a `references:` key.

UNKNOWN_KEY is now a warning, so the skill is stored (unknown keys and all)
and the issue is surfaced rather than fatal. `references` joins the allowed
set with a shallow JSON-safety check instead of a strict kind match: real
files use a string, a list of strings, a list of objects, and a map, and
pinning one shape would reintroduce the same failure.

Malformed frontmatter stays fatal: INVALID_TYPE, INVALID_SHAPE and the
non-plain-object check are unchanged.

* fix: skip individual skills instead of failing a whole sync source

Any error inside the discovery or commit loop reached the outer catch and
marked the entire source failed, so one unusable SKILL.md, one oversized
blob, or one duplicate name cost every other skill in the repository.

Each skill now runs inside its own boundary and a failure is recorded
against that skill. Errors that mean nothing else in the run can succeed
(lock loss, GitHub auth failures, rate limiting) still abort the source
rather than being charged to whichever skill hit them first. Skills are
marked seen before the attempt, so the reconcile pass cannot mirror-delete
the previously synced copy of a skill a later run can repair, and duplicate
names now drop the whole colliding group instead of letting tree order pick
an arbitrary winner.

Status gains `partial` (published some, skipped others) plus a capped
sample of the skipped skills with the reason for each. A run that publishes
nothing and skips something is still `failed`, carrying the first skip's
error. The skipped entries name repository paths, so they follow the same
visibility rule as owner/repo/paths; the bare count does not.

Sync warnings are logged too: a background run has no user-facing surface,
so the log is the only place a maintainer sees why an upstream SKILL.md
looks off.

* test: cover skill sync warnings reaching the log

An unrecognized frontmatter key no longer fails the skill, so a background
sync has nowhere to report it except the log. Every mock in this spec
returned an empty warning list, which left that path unexercised.

* fix: describe nested frontmatter values in the shared skill type

`SkillFrontmatterValue` allowed only scalars and string arrays, while the
server has always stored `hooks` and `metadata` as JSON-safe objects, and now
`references` too. A skill carrying any of them could not be represented by
`TSkill`, `TCreateSkill` or `TUpdateSkillPayload` without a cast.

The type stays free of `any` and `unknown`: values remain JSON-safe by
construction, and the server keeps bounding depth, string length and array
size when it validates them.

* fix: protect moved mirrors and rolled-back counts when a skill is skipped

Continuing past a failed skill exposed two problems that aborting the whole
source used to hide.

A moved skill's mirror keeps its old upstream id until the update lands, and
only the new path was marked as seen, so the reconcile pass read the mirror as
stale and deleted the very copy the skip path exists to preserve. The old id is
now marked as seen too.

Deletion counters were incremented when a stale name-conflicting mirror was
removed, but never undone when the following commit failed and the mirror was
restored. The run no longer stops there, so the status persisted a deletion
that did not happen and the reconcile pass counted the restored row again.
Counters are now rolled back when the restore succeeds.

* fix: bound unknown frontmatter values and keep moved mirrors through duplicates

Tolerating an unrecognized key meant its value skipped the shared JSON-safety
check, so a deeply nested or oversized payload was accepted and persisted under
a key nobody validates. The key stays non-blocking; the value is now held to
the same depth, array and string bounds as every structured key.

A skill that moves into a name another discovered skill also claims is dropped
with the rest of its duplicate group before the sync path can reuse its mirror,
which left the still-published copy unmarked and reconciled away. Both paths now
mark the moved mirror through one helper.

* fix: end the source when a skipped skill fails to roll back

A skill that fails and rolls back cleanly is just a skipped skill. One whose
restore or delete also fails leaves a mirror with half-rewritten files or a
half-created row, and the run now continues past it, so the source could report
partial success while that mirror stayed inconsistent and its pre-marked
upstream id kept reconciliation away from it.

Failed rollbacks now raise a source-fatal error carrying the original failure,
which stops the source the way a lost lock or a refused GitHub token does.

* test: cover a skipped skill discovered at the repository root

A repository-level SKILL.md is discovered with an empty path, so this pins
that a skip recorded against it still persists with the rest of the partial
status rather than taking the whole status row down with it.

* docs: describe unknown skill frontmatter warnings

* fix: preserve mirrors after partial skill sync

* fix: preserve skill validation details during sync

* fix: fail sync when mirror identity cannot be restored

* fix: harden skill sync failure boundaries

* fix: preserve skipped skills on fatal sync

* fix: surface skill sync diagnostics and rollback failures

* fix: preserve skill frontmatter extension keys

* fix: reject skill frontmatter keys that collide when normalized

Frontmatter keys are matched case-insensitively against the canonical
key list, so "Name" and "name" both resolve to "name". Every call site
normalized independently, and the last key in iteration order silently
won, meaning the effective value depended on YAML ordering rather than
on anything the author could see.

Centralize the normalization in normalizeSkillFrontmatterKeys and have
it fail when two recognized keys resolve to the same canonical key,
rather than picking one. parse.ts, deployment.ts and the agent handler
now surface that as a parse error; createSkill and updateSkill surface
it as a blocking DUPLICATE_KEY validation issue. Unrecognized keys are
still passed through untouched so extension frontmatter survives.

deriveStructuredFrontmatterFields and both write paths now run on the
normalized map, so a "Disable-Model-Invocation" key derives the same
column a lowercase one does.

* fix: harden github skill sync against dropped requests and failed cleanup

Three failure paths in the GitHub sync could leave a source looking
healthier than it was.

githubJson only handled HTTP-level errors. A fetch that rejected before
producing a response (DNS failure, socket reset, abort) escaped as a
raw TypeError, so the sync reported a generic crash instead of a typed
sync error. Wrap it as GITHUB_REQUEST_FAILED and add that code to the
fatal set, since a source whose requests never complete cannot be
partially synced.

When a synced file failed to persist, the orphaned upload was cleaned
up on a best-effort basis and the cleanup error was only logged. If the
cleanup itself failed, the source still ended with the original error
and left a real orphan behind. Promote that to a rollback failure so
the source reports SYNC_ROLLBACK_FAILED with the triggering error.

Skill warnings were logged inside commitRemoteSkill, before the file
sync and viewer setup that can still roll the skill back. A skill that
never survived publication therefore emitted warnings as though it had.
Return the warnings from the commit and log them once the skill is
fully published.

* fix: report skipped github skills before credential errors

serializeErrorMessage checked isCredentialError first, and that check
matches on the error text. A skipped skill whose path happens to
contain a credential-ish word, for example skills/credential-helper,
was therefore redacted to "GitHub skill sync credentials are not
available" for admins without credential-metadata access, hiding a
parse failure behind a wrong diagnosis.

Check the promoted skipped-skill case first, since it is identified by
error code rather than by text and is the more specific match. The
credential redaction still applies to everything else.
2026-08-12 13:22:06 -04:00
..
misc 🛡️ feat: Add Batched MCP Authority Proofs (#14688) 2026-08-07 12:23:25 -04:00
src 🛟 fix: Isolate Invalid Skills During GitHub Sync (#14735) 2026-08-12 13:22:06 -04:00
.gitignore
babel.config.cjs
jest.config.mjs
LICENSE
package.json 📦 chore: bump @librechat/agents to v3.4.2 and npm audit (#14702) 2026-08-08 12:24:06 -04:00
README.md
tsconfig.build.json
tsconfig.json 🔧 chore: Enforce isolatedDeclarations in data-schemas tsconfig (#13593) 2026-06-08 09:55:20 -04:00
tsconfig.spec.json 📦 chore: Update TypeScript Config for TS v7 (#12794) 2026-04-23 12:51:03 -04:00
tsdown.config.mjs 🌀 ci: Deterministic Circular Dependency Checks (#14579) 2026-08-01 14:43:26 -04:00

LibreChat Data Schemas Package

This package provides the database schemas, models, types, and methods for LibreChat using Mongoose ODM.

📁 Package Structure

packages/data-schemas/
├── src/
│   ├── schema/         # Mongoose schema definitions
│   ├── models/         # Model factory functions
│   ├── types/          # TypeScript type definitions
│   ├── methods/        # Database operation methods
│   ├── common/         # Shared constants and enums
│   ├── config/         # Configuration files (winston, etc.)
│   └── index.ts        # Main package exports

🏗️ Architecture Patterns

1. Schema Files (src/schema/)

Schema files define the Mongoose schema structure. They follow these conventions:

  • Naming: Use lowercase filenames (e.g., user.ts, accessRole.ts)
  • Imports: Import types from ~/types for TypeScript support
  • Exports: Export only the schema as default

Example:

import { Schema } from 'mongoose';
import type { IUser } from '~/types';

const userSchema = new Schema<IUser>(
  {
    name: { type: String },
    email: { type: String, required: true },
    // ... other fields
  },
  { timestamps: true }
);

export default userSchema;

2. Type Definitions (src/types/)

Type files define TypeScript interfaces and types. They follow these conventions:

  • Base Type: Define a plain type without Mongoose Document properties
  • Document Interface: Extend the base type with Document and _id
  • Enums/Constants: Place related enums in the type file or common/ if shared

Example:

import type { Document, Types } from 'mongoose';

export type User = {
  name?: string;
  email: string;
  // ... other fields
};

export type IUser = User &
  Document & {
    _id: Types.ObjectId;
  };

3. Model Factory Functions (src/models/)

Model files create Mongoose models using factory functions. They follow these conventions:

  • Function Name: create[EntityName]Model
  • Singleton Pattern: Check if model exists before creating
  • Type Safety: Use the corresponding interface from types

Example:

import userSchema from '~/schema/user';
import type * as t from '~/types';

export function createUserModel(mongoose: typeof import('mongoose')) {
  return mongoose.models.User || mongoose.model<t.IUser>('User', userSchema);
}

4. Database Methods (src/methods/)

Method files contain database operations for each entity. They follow these conventions:

  • Function Name: create[EntityName]Methods
  • Return Type: Export a type for the methods object
  • Operations: Include CRUD operations and entity-specific queries

Example:

import type { Model } from 'mongoose';
import type { IUser } from '~/types';

export function createUserMethods(mongoose: typeof import('mongoose')) {
  async function findUserById(userId: string): Promise<IUser | null> {
    const User = mongoose.models.User as Model<IUser>;
    return await User.findById(userId).lean();
  }

  async function createUser(userData: Partial<IUser>): Promise<IUser> {
    const User = mongoose.models.User as Model<IUser>;
    return await User.create(userData);
  }

  return {
    findUserById,
    createUser,
    // ... other methods
  };
}

export type UserMethods = ReturnType<typeof createUserMethods>;

5. Main Exports (src/index.ts)

The main index file exports:

  • createModels() - Factory function for all models
  • createMethods() - Factory function for all methods
  • Type exports from ~/types
  • Shared utilities and constants

🚀 Adding a New Entity

To add a new entity to the data-schemas package, follow these steps:

Step 1: Create the Type Definition

Create src/types/[entityName].ts:

import type { Document, Types } from 'mongoose';

export type EntityName = {
  /** Field description */
  fieldName: string;
  // ... other fields
};

export type IEntityName = EntityName &
  Document & {
    _id: Types.ObjectId;
  };

Step 2: Update Types Index

Add to src/types/index.ts:

export * from './entityName';

Step 3: Create the Schema

Create src/schema/[entityName].ts:

import { Schema } from 'mongoose';
import type { IEntityName } from '~/types';

const entityNameSchema = new Schema<IEntityName>(
  {
    fieldName: { type: String, required: true },
    // ... other fields
  },
  { timestamps: true }
);

export default entityNameSchema;

Step 4: Create the Model Factory

Create src/models/[entityName].ts:

import entityNameSchema from '~/schema/entityName';
import type * as t from '~/types';

export function createEntityNameModel(mongoose: typeof import('mongoose')) {
  return (
    mongoose.models.EntityName || 
    mongoose.model<t.IEntityName>('EntityName', entityNameSchema)
  );
}

Step 5: Update Models Index

Add to src/models/index.ts:

  1. Import the factory function:
import { createEntityNameModel } from './entityName';
  1. Add to the return object in createModels():
EntityName: createEntityNameModel(mongoose),

Step 6: Create Database Methods

Create src/methods/[entityName].ts:

import type { Model, Types } from 'mongoose';
import type { IEntityName } from '~/types';

export function createEntityNameMethods(mongoose: typeof import('mongoose')) {
  async function findEntityById(id: string | Types.ObjectId): Promise<IEntityName | null> {
    const EntityName = mongoose.models.EntityName as Model<IEntityName>;
    return await EntityName.findById(id).lean();
  }

  // ... other methods

  return {
    findEntityById,
    // ... other methods
  };
}

export type EntityNameMethods = ReturnType<typeof createEntityNameMethods>;

Step 7: Update Methods Index

Add to src/methods/index.ts:

  1. Import the methods:
import { createEntityNameMethods, type EntityNameMethods } from './entityName';
  1. Add to the return object in createMethods():
...createEntityNameMethods(mongoose),
  1. Add to the AllMethods type:
export type AllMethods = UserMethods &
  // ... other methods
  EntityNameMethods;

📝 Best Practices

  1. Consistent Naming: Use lowercase for filenames, PascalCase for types/interfaces
  2. Type Safety: Always use TypeScript types, avoid any
  3. JSDoc Comments: Document complex fields and methods
  4. Indexes: Define database indexes in schema files for query performance
  5. Validation: Use Mongoose schema validation for data integrity
  6. Lean Queries: Use .lean() for read operations when you don't need Mongoose document methods

🔧 Common Patterns

Enums and Constants

Place shared enums in src/common/:

// src/common/permissions.ts
export enum PermissionBits {
  VIEW = 1,
  EDIT = 2,
  DELETE = 4,
  SHARE = 8,
}

Compound Indexes

For complex queries, add compound indexes:

schema.index({ field1: 1, field2: 1 });
schema.index(
  { uniqueField: 1 },
  { 
    unique: true, 
    partialFilterExpression: { uniqueField: { $exists: true } }
  }
);

Virtual Properties

Add computed properties using virtuals:

schema.virtual('fullName').get(function() {
  return `${this.firstName} ${this.lastName}`;
});

🧪 Testing

When adding new entities, ensure:

  • Types compile without errors
  • Models can be created successfully
  • Methods handle edge cases (null checks, validation)
  • Indexes are properly defined for query patterns

📚 Resources