mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-04 14:57:42 +00:00
a07c0e4ae8
Grades fast-model activity-label headers against a fixed corpus so instruction changes are measured rather than eyeballed on one conversation. This existed untracked while the continuity work was developed; committing it because it is the only reproducible record of WHY `ACTIVITY_INSTRUCTION` is ordered and capped the way it is. - captured.json: 9 real production payloads pulled verbatim from Langfuse with the headers that shipped. Irreplaceable — traces age out. - corpus.js: 17 cases / 28 steps. The captured run replays as one sequence, plus synthetic cases for the modes it never exercised (all-failed, partial, parallel batches, rapid near-duplicates, entry overflow, truncated output, error-shaped success). Multi-step cases chain each generated label into the next step's context, which is what makes cross-batch redundancy measurable at all. - prompt.js: faithful port of the SDK's buildActivityLabelPrompt so synthetic cases render the bytes production sends, plus a previousLabelCap knob for continuity-window experiments. - variants.js: single-factor instruction variants. The baseline is read from the BUILT package (workspace resolution, then dist, then LABEL_EVAL_DIST) so a variant can never be graded against a stale copy of the shipped instruction. - checks.js: length/punctuation/markdown/tool-echo/count-echo, plus overlap split into `restate` (adds nothing over an earlier header) vs `template` (same frame, new payload — often fine). - run.js / rescore.js: live runner on the production wire shape (max_tokens 256) and an offline re-grader, so metric fixes never require re-spending on the API. Results are gitignored — regenerable, and 292K of the 364K. A full sweep is ~$0.03 per variant and ~45s. Findings are recorded in the README, two of them counter-intuitive: enumerating acceptable opening verbs ANCHORED the model rather than diversifying it (Confirmed 18→23, opener diversity halved), and diverse examples alone changed nothing. Sentence order is load-bearing, so a tidying reshuffle of ACTIVITY_INSTRUCTION regresses real output. |
||
|---|---|---|
| .. | ||
| captured.json | ||
| checks.js | ||
| corpus.js | ||
| prompt.js | ||
| README.md | ||
| report.js | ||
| rescore.js | ||
| run.js | ||
| variants.js | ||
Activity-label eval harness
Measures the prose quality of fast-model activity-label headers (the collapsed
group headers produced by activityLabel). Instruction changes are graded
against a fixed corpus instead of eyeballed on one conversation — the two
hypotheses that felt most obvious when this was built both turned out wrong
under measurement (see Findings).
node scripts/activity-labels/run.js # every variant, 1 sample
node scripts/activity-labels/run.js --samples 3 # 3 samples each
node scripts/activity-labels/run.js --variants baseline,shipping --cases fib-rapid
node scripts/activity-labels/run.js --dry --cases mega-batch # render prompts, no API calls
node scripts/activity-labels/rescore.js # re-grade stored results, no re-spend
Requires ANTHROPIC_API_KEY (env or .env). A full sweep is roughly $0.03 per
variant and ~45s. Results land in results/ (gitignored): a timestamped JSON
of every record plus latest.md with per-case tables.
Why the tables matter more than the aggregate
The mechanical checks catch format violations and lexical repetition, but the
failure this feature actually had in production — headers that were
informationally redundant while lexically varied — scores below the overlap
threshold. results/latest.md read by eye is the recall instrument; the
aggregate is the regression guard.
Layout
| File | Role |
|---|---|
captured.json |
9 real production payloads pulled verbatim from Langfuse, with the headers that shipped. Irreplaceable — traces age out. |
corpus.js |
17 cases / 28 steps: the captured run as one sequence, plus synthetic cases for modes it never hit (all-failed, partial, parallel, rapid duplicates, entry overflow, truncated output, error-shaped success). Multi-step cases chain each generated label into the next step's context. |
prompt.js |
Faithful port of the SDK's buildActivityLabelPrompt, so synthetic cases render the bytes production would send. Adds previousLabelCap for continuity-window experiments. |
variants.js |
Single-factor instruction variants. baseline is read from the built packages/api dist, so drift from the shipped instruction is impossible. |
checks.js |
Mechanical grading: length, punctuation, markdown, tool-name echo, count echo, and overlap split into restate (adds nothing) vs template (same frame, new payload). |
run.js / rescore.js |
Live runner (production wire shape, max_tokens: 256) and offline re-grader. |
Findings this produced
- Sentence order is load-bearing. Moving format constraints after content
rules cut length violations and register collapse measurably. This is why
ACTIVITY_INSTRUCTIONis ordered the way it is — a "tidying" reshuffle regresses real output. - Enumerating verbs backfires. An instruction listing acceptable opening
verbs anchored the model:
Confirmedwent from 18 to 23 occurrences and opener diversity halved. - Diverse examples alone changed nothing. Register collapse is task-shaped, not example-seeded.
- Continuity context fixed the real defect. Feeding committed headers back eliminated restatement and, unexpectedly, stopped setup batches from being labeled with conclusions their tools had not yet established.
- A 3-label window is enough. Unbounded history scored no better —
restatement is inherently a recency problem — while prompt growth is linear
(+82 input tokens by batch 9, extrapolating to ~+250 at the
activityMaxPerRundefault of 20).