LibreChat/packages/api/src/agents/selection.ts
Danny Avila d5819becf2
🎯 feat: Per-Tool Intent Labels for Model Specs (#14526)
* 🎯 feat: Per-Tool Intent Labels for Model Specs

A model spec could only turn intent labels on for ALL of its eligible
tools. Saved agents have had per-tool control since the capability landed
(`tool_options[id].describe_intent`, with the builder toggle following in
the UI slice), but a model spec is admin YAML that produces an ephemeral
agent — there is no agent document to hold per-tool options, so
`describeIntent: true` synthesized an entry for every eligible tool.

`describeIntent` now accepts a string array alongside the boolean,
matching the `skills` field already in the same schema:

  describeIntent: true                                  # every eligible tool
  describeIntent: ['web_search', 'search_code_mcp_github']

This matters because the label costs schema tokens on every request, so
an admin may want it on a handful of illegible calls rather than the
whole toolset.

- Named tools still pass eligibility, so an excluded tool cannot be
  forced on by listing it.
- An empty array reads as disabled.
- Names that are not eligible or not equipped on the spec are logged
  rather than silently skipped — a typo in a spec would otherwise be
  undiagnosable.
- The ephemeral toggle stays boolean and stays global even when a spec
  list is present: it has no per-tool UI to drive it, so narrowing it
  would silently cover fewer tools than the user asked for.

`runInBackground` has the same all-or-nothing limitation and could take
the same shape; left alone here to keep this change reviewable.

* 🎯 feat: Per-Tool Background Dispatch for Model Specs

Gives `runInBackground` the same `boolean | string[]` shape as
`describeIntent`, so the two per-tool capabilities are configured
identically from a model spec:

  runInBackground: true                          # every eligible tool
  runInBackground: ['slow_report_mcp_analytics']  # only this one

Selectivity matters more here than for intent labels. An intent label is
inert — it costs tokens and nothing else. Backgrounding changes execution
semantics: the model gets a synthetic handle and must poll. Letting an
admin detach one slow MCP call without making every other tool in the
spec detachable is the difference between a usable setting and an
all-or-nothing one.

Same guarantees as the intent equivalent:
- Named tools still pass eligibility, so the exclusion list still holds —
  a list cannot force on web_search, file_search, image gen, the HITL
  tool, or anything whose attachments/artifact continuity would break.
- Empty array reads as disabled.
- Unmatched names are logged rather than silently skipped.
- The ephemeral toggle stays boolean and global; it has no per-tool UI.

Also fixes a latent no-op: this function did not skip the lazily-expanded
`mcp_all` placeholder, so a spec with an overlay MCP server recorded an
option under a name `applyBackgroundToolCalls` can never match. The
intent equivalent already skipped it; now both do.

* ♻️ refactor: One Definition of the mcp_all Placeholder Guard

Fixing the background no-op left the placeholder prefix declared twice —
once per capability synthesizer — which is the same duplicated-literal
shape that made the intent label marker fragile: two copies that must
agree, with drift producing a silent no-op rather than an error.

`MCP_ALL_PLACEHOLDER_PREFIX` and `isMCPAllPlaceholder` now live beside
`mcpToolPattern` in mcp/utils, so both synthesizers cannot disagree about
which tool entries to ignore, and anything added later that keys per-tool
config by exact name has an obvious guard to reach for.

Audited the rest of the capability family while here: only background and
intent synthesize per-tool options from a model spec. `defer_loading` and
`allowed_callers` have no model-spec path at all, so neither can carry
this bug. Both synthesizers now have an explicit regression test naming
the placeholder.

* 🧯 fix: Treat a describeIntent List as a Selection Policy, Not a Filter

Two build/behavior defects from the previous commits.

**Narrowing did not actually narrow.** Omitting a tool from the
synthesized options is not the same as opting it out, because intent has
two default-on paths background does not: `isIntentOptedIn` treats every
NATIVE_INTENT_TOOL_NAMES member as enabled when it finds no entry, and
`sanitizeIntentLabels` keeps an SDK-native label unless it sees an
explicit `describe_intent: false`. So `describeIntent: ['web_search']`
still labelled `set_memory`, and an empty list — the most explicit way to
say "none" — disabled nothing at all. A list is now a selection policy:
selected eligible tools get true, unselected get an explicit false.
Ineligible tools still get no entry at all.

Background needs no equivalent change: it opts in on
`run_in_background === true` only, with no default-on set, so omission
there genuinely means off.

**Fixed the CI build break.** `MCP_ALL_PLACEHOLDER_PREFIX` was exported
without a type annotation, which `tsc --noEmit` accepts but the package
build rejects under `--isolatedDeclarations` (TS9010) — the same reason
`mcpToolPattern` beside it is annotated `: RegExp`. Verified with an
actual `npm run build` this time, not just a typecheck.

* 🧯 fix: Propagate Intent Selection Through Capability Marker Expansion

A model spec's `tools` carries capability MARKERS, not the definition
names initialization actually registers, so an option recorded under a
marker never matches the tool it becomes — the same silent no-op as an
`mcp_all` placeholder entry.

Harmless for an opt-IN (the tool keeps its default) but not for the
opt-OUTs the previous commit introduced: `memory` becomes `set_memory` +
`delete_memory`, both default-on natives, and `execute_code` becomes
`bash_tool`, which carries an SDK-native label that survives unless
sanitize sees an explicit false. So `describeIntent: ['web_search']` on a
spec with `memory: true` still labelled both memory tools, and `[]`
disabled neither.

`expandIntentToolOptions` propagates a marker's value to the names it
expands into, mirroring `expandCodeToolOptions` in background.ts which
solves the same problem for the code marker. Applied at APPLY time rather
than synthesis, so hand-edited saved agents that key options by marker
benefit too, not just synthesized specs. An explicit per-tool entry always
wins — expansion only fills names the caller did not already decide — and
it runs in both `applyIntentLabels` and `sanitizeIntentLabels`, since the
SDK-native strip reads the same options.

Verified with an actual package build, not just a typecheck.

* ♻️ refactor: Resolve Spec Tool Selections Against Final Definitions

Three review rounds hit the same root cause from three directions: a
per-tool option synthesized at load time is keyed by names that may not
exist at injection time. Spec tools carry capability markers
(execute_code, memory), skills never reach the tools array at all, and
lazy MCP servers expand after synthesis - every mismatch was a silent
no-op, and each fix added another hand-maintained marker map that the
next case leaked past.

This removes the name-space gap instead of bridging it per case:

- Synthesis records the selection as a policy: a wildcard '*' entry
  carries the default (true for "every tool", false for "only the named
  ones") and listed names are recorded verbatim. No load-time
  enumeration, eligibility checks, or placeholder special-casing.
- Resolution happens at injection time, per final definition:
  explicit name -> capability marker projection -> wildcard. The
  projection maps a marker onto the names its registration actually
  produced this run - the registrars report their own tool names and
  initializeAgent accumulates them - so the mapping cannot drift from
  what gets registered.
- The unmatched-name diagnosis moves to the apply passes, the one place
  the real definitions are known, so a selection naming a marker whose
  runtime expansion is entirely ineligible (runInBackground: ['memory'])
  is now warned about instead of recorded as a dead success.

Covers all four round-3 findings: code opt-outs now reach
create_file/edit_file/read_file, skill definitions are governed by
narrowing selections, the memory marker is rejected and diagnosed for
backgrounding, and the mcp_all placeholder predicate is no longer
consulted for selections at all (its one remaining consumer is the
definitions loader that defines the convention).

* 🧯 fix: Reject Dead Ask-Tool Selections and the Reserved Wildcard

Two review findings on the selection policy, both fixed at the point
where they are knowably invalid:

- ask_user_question joins EXCLUDED_INTENT_TOOL_NAMES: createRun strips
  its provisional definition and rebuilds the graph tool from its own
  Zod schema, so definition-level injection never reaches the model. A
  describeIntent selection naming it now warns as ineligible instead of
  crediting a label that gets discarded. Real intent support for the
  ask tool lands with the HITL slice via the interrupt payload.

- A literal '*' in a describeIntent/runInBackground list is dropped
  with a warning at synthesis: it would overwrite the wildcard opt-out
  default and silently enable the capability for every eligible tool
  instead of selecting one named tool. The wildcard is reserved for the
  internal policy; boolean true is the supported way to cover
  everything.
2026-07-30 13:32:04 -04:00

214 lines
8.3 KiB
TypeScript

/**
* @fileoverview Tool-selection policy for capability-injected per-tool options.
*
* A model spec / ephemeral agent expresses per-tool capability choices
* (`describeIntent`, `runInBackground`) against its LOAD-TIME tool entries:
* capability markers (`execute_code`, `memory`), lazy `mcp_all` placeholders,
* and fields that never reach the `tools` array at all (`skills`). The
* definitions those choices must govern only exist at INITIALIZATION time,
* after markers expand and late registrations (file authoring, the skill
* catalog) run — so any option pre-keyed by a load-time name risks being a
* silent no-op against the final definition set.
*
* This module removes that name-space gap instead of bridging it per case:
*
* - Synthesis records the selection as a POLICY, not per-name entries: a
* wildcard `*` entry carries the default (`true` for "every tool", `false`
* for "only the named ones"), and listed names are recorded verbatim as
* explicit opt-ins.
* - Resolution happens at injection time, per final definition:
* explicit name → capability projection → wildcard → caller's default.
* The capability projection maps a marker entry onto the definition names
* its registration ACTUALLY produced this run (reported by the registrars
* themselves in `initializeAgent`), so the mapping cannot drift from what
* gets registered.
*
* Saved agents are untouched by the wildcard (their `tool_options` come from
* the builder, keyed by real names or markers) but gain the same projection,
* replacing the hand-maintained marker→names constants that kept leaking
* (create_file/edit_file, the skill catalog, memory-in-background).
*
* @module packages/api/src/agents/selection
*/
import { logger } from '@librechat/data-schemas';
import type { AgentToolOptions } from 'librechat-data-provider';
/**
* Reserved `tool_options` key holding a synthesized selection's default for
* every tool without a more specific entry. Provider tool-name grammars
* (`^[a-zA-Z0-9_-]+$`) cannot produce it, and every other consumer of
* `tool_options` reads by exact definition name, so the entry is inert
* outside the resolution below. Never persisted — it only appears on
* synthesized ephemeral/model-spec options.
*/
export const TOOL_SELECTION_WILDCARD = '*';
/**
* Capability marker → tool definition names its registration produced for
* this run. Built by `initializeAgent` from the registrars' own reports, so
* an option keyed by a marker (a spec selection or a hand-edited saved
* agent) projects onto exactly what got registered.
*/
export type CapabilityToolNames = ReadonlyMap<string, readonly string[]>;
type SelectionField = 'describe_intent' | 'run_in_background';
export interface ResolvedToolOption {
value: boolean;
/** The `tool_options` key that decided the value: the definition's own
* name, a capability marker that registered it, or the wildcard. */
source: string;
}
/**
* Resolves one selection field for a definition name. Precedence: an
* explicit entry under the definition's own name, then capability markers
* whose registrations include the name (an opting-in marker wins over an
* opting-out one when two capabilities register the same tool, e.g.
* `read_file` under both code and skills), then the wildcard default.
* Returns undefined when no policy speaks, leaving the caller's default
* (native intent tools, background's opt-in-only) in force.
*/
export function resolveToolOption(
name: string,
field: SelectionField,
toolOptions: AgentToolOptions | undefined,
capabilityToolNames?: CapabilityToolNames,
): ResolvedToolOption | undefined {
if (!toolOptions) {
return undefined;
}
const explicit = toolOptions[name]?.[field];
if (explicit != null) {
return { value: explicit === true, source: name };
}
let markerOptOut: ResolvedToolOption | undefined;
if (capabilityToolNames) {
for (const [marker, names] of capabilityToolNames) {
const markerValue = toolOptions[marker]?.[field];
if (markerValue == null || !names.includes(name)) {
continue;
}
if (markerValue === true) {
return { value: true, source: marker };
}
markerOptOut ??= { value: false, source: marker };
}
}
if (markerOptOut) {
return markerOptOut;
}
const wildcard = toolOptions[TOOL_SELECTION_WILDCARD]?.[field];
if (wildcard != null) {
return { value: wildcard === true, source: TOOL_SELECTION_WILDCARD };
}
return undefined;
}
/**
* Records a spec/ephemeral selection as policy entries. `true` (or the
* ephemeral toggle, which has no per-tool UI and therefore never narrows)
* becomes a wildcard opt-in; a list becomes a wildcard opt-out plus verbatim
* opt-ins for the named entries — names are NOT validated here, because the
* definitions they must match only exist at injection time, where
* {@link resolveToolOption} consumes them and the apply pass diagnoses
* selections that never took effect. The one exception is the reserved
* wildcard itself: a literal `*` list entry would overwrite the opt-out
* default and silently enable the capability for EVERY eligible tool, so it
* is dropped and warned about here, where it is knowably invalid.
*/
export function synthesizeSelectionToolOptions(
field: SelectionField,
selection: boolean | string[] | undefined,
ephemeralEnabled: boolean,
label: string,
): AgentToolOptions | undefined {
const selectedNames = Array.isArray(selection) ? selection : undefined;
if (!ephemeralEnabled && selection !== true && selectedNames == null) {
return undefined;
}
if (ephemeralEnabled || selection === true) {
return { [TOOL_SELECTION_WILDCARD]: { [field]: true } };
}
const toolOptions: AgentToolOptions = { [TOOL_SELECTION_WILDCARD]: { [field]: false } };
for (const name of selectedNames ?? []) {
if (name === TOOL_SELECTION_WILDCARD) {
logger.warn(
`${label} contains the reserved wildcard "${TOOL_SELECTION_WILDCARD}"; ignoring it. Set the option to true to cover every eligible tool.`,
);
continue;
}
toolOptions[name] = { [field]: true };
}
return toolOptions;
}
/**
* The names a narrowing selection opted in, or undefined when no narrowing
* policy is present (boolean modes, saved agents). Presence of a wildcard
* opt-out is the discriminator: only synthesized list selections carry one,
* so saved agents with stale hand-edited entries are never warned about.
*/
export function getSelectionNames(
toolOptions: AgentToolOptions | undefined,
field: SelectionField,
): Set<string> | undefined {
if (toolOptions?.[TOOL_SELECTION_WILDCARD]?.[field] !== false) {
return undefined;
}
const names = new Set<string>();
for (const [name, options] of Object.entries(toolOptions)) {
if (name !== TOOL_SELECTION_WILDCARD && options?.[field] === true) {
names.add(name);
}
}
return names;
}
/**
* Warns about selection names that never took effect on any definition — a
* typo, a tool the spec doesn't equip, or a name whose every definition is
* ineligible (e.g. `runInBackground: ['memory']`, whose expansion is
* entirely background-excluded). A silent no-op would leave the misspelled
* or unsupported entry undiagnosable, since the symptom is simply "nothing
* happened".
*/
export function warnUnmatchedSelectionNames(
selectionNames: ReadonlySet<string> | undefined,
effectiveSources: ReadonlySet<string>,
label: string,
): void {
if (selectionNames == null || selectionNames.size === 0) {
return;
}
const unmatched = [...selectionNames].filter((name) => !effectiveSources.has(name));
if (unmatched.length === 0) {
return;
}
logger.warn(
`${label} named ${unmatched.length} tool(s) that are not eligible or not equipped on this spec: ${unmatched.join(', ')}`,
);
}
/**
* Deep-merges two synthesized `tool_options` maps per tool key, so the
* ephemeral background and intent toggles compose instead of overwriting
* each other's per-tool entries.
*/
export function mergeSynthesizedToolOptions(
base: AgentToolOptions | undefined,
extra: AgentToolOptions | undefined,
): AgentToolOptions | undefined {
if (!extra) {
return base;
}
if (!base) {
return extra;
}
const merged: AgentToolOptions = { ...base };
for (const [name, options] of Object.entries(extra)) {
merged[name] = { ...merged[name], ...options };
}
return merged;
}