mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 04:07:05 +00:00
🧱 refactor: typed CodeEnvRef + kind discriminator + principal-aware sandbox cache (#12960)
* 🧱 refactor: typed CodeEnvRef + kind discriminator + tenant-aware sandbox cache Final cutover for the LibreChat ↔ codeapi sandbox file identity. Replaces the magic string `${session_id}/${file_id}?entity_id=...` with a typed, discriminated `CodeEnvRef`. Pre-release lockstep deploy with codeapi #1455 and agents #148; no legacy aliases retained. ## Final shape ```ts type CodeEnvRef = | { kind: 'skill'; id: string; storage_session_id: string; file_id: string; version: number } | { kind: 'agent'; id: string; storage_session_id: string; file_id: string } | { kind: 'user'; id: string; storage_session_id: string; file_id: string }; ``` `kind` drives codeapi's sessionKey: `<tenant>:<kind>:<id>[✌️<version>]` for shared kinds, `<tenant>:user:<userId>` for user-private (auth context provides `userId`). `version` is statically required for `kind: 'skill'` and forbidden otherwise via discriminated union — constraint holds at compile time on every consumer, not just codeapi's runtime validator. `id` is sessionKey-meaningful for `'skill'` / `'agent'`; informational only for `'user'` (codeapi resolves user identity from auth context). ## What changed - `packages/data-provider/src/codeEnvRef.ts` — discriminated union + `CODE_ENV_KINDS` const-tuple keeps the runtime list and TS union locked together. - Schemas: `metadata.codeEnvRef` and `SkillFile.codeEnvRef` enums tightened to `['skill', 'agent', 'user']`. - `primeSkillFiles` writes `kind: 'skill'`, `id: skill._id`, `version: skill.version`. Cache-hit path reads `codeEnvRef` directly. Bumping `skill.version` on edit naturally invalidates the prior cache entry under the new sessionKey. - `processCodeOutput` writes `kind: 'user'`, `id: req.user.id`. Output bucket is always user-scoped, regardless of which skill the execution invoked. New regression test pins the asymmetry. - `primeFiles` reupload preserves `kind`/`id`/`version?` from the existing ref so a skill-cache-miss reupload doesn't silently demote to user bucket. - `crud.js` upload functions (`uploadCodeEnvFile` / `batchUploadCodeEnvFiles`) thread `kind`/`id`/`version?` to the multipart form (codeapi #1455 option α). Without these on the wire, codeapi falls back to user bucketing and skill-cache invalidation never fires. Client-side validation mirrors codeapi's validator. - `Files/process.js` — chat attachments use `kind: 'user'`; agent setup files use `kind: 'agent'`. - Drops `entity_id` everywhere (struct, schema sub-docs, write paths, upload form fields). Drops `'system'` from the kind enum (no emitter ever existed). ## Test plan - [x] `cd packages/data-provider && npx jest src/codeEnvRef.spec` — 4 / 4 - [x] `cd packages/data-schemas && npx jest` — 1447 / 1447 - [x] `cd packages/api && npx jest src/agents` — 81 / 81 in skillFiles + handlers + resources - [x] `cd api && npx jest server/services/Files server/controllers/agents` — 436 / 436 - [x] `cd api && npx jest server/services/Files/Code` — 98 / 98 (incl. new "outputs are user-scoped regardless of which skill the execution invoked" regression and "reupload forwards kind/id/version from existing ref") - [x] `npx tsc --noEmit -p packages/data-{provider,schemas}/tsconfig.json && npx tsc --noEmit -p packages/api/tsconfig.json` — clean (only pre-existing unrelated dev errors in storage/balance, untouched here) ## Deploy notes - **24h cache-miss burst** on first deploy. Inputs (skill caches re-prime under new sessionKey shape) and outputs (any pre-Phase C skill-output cached files become unreadable). Bounded by codeapi's 24h TTL. - **Lockstep with codeapi #1455 and agents #148.** Either repo can land first since no aliases to drain, but the three deploys must overlap within the same maintenance window. - **`@librechat/agents` bump to `3.1.79-dev.0`** required after agents #148 lands and is published. ## What this enables Auth bridge work (JWT-based tenant/user identity between LC and codeapi) — codeapi now derives sessionKey purely from `req.codeApiAuthContext.{ tenantId, userId}`, so the next chapter is replacing the header-asserted user identity with a verified-claim path. * 🩹 fix: persist execute_code uploads under codeEnvRef metadata key Codex review P1 (chatgpt-codex-connector). `Files/process.js` was storing the upload result under `metadata.fileIdentifier` even though: - `uploadCodeEnvFile` now returns `{ storage_session_id, file_id }`, not the legacy magic string. - The post-cutover schema (`File.metadata.codeEnvRef`) only declares `codeEnvRef` — mongoose strict mode silently strips unknown keys. - All readers (`primeFiles`, `getCodeFilesByIds`, `categorizeFileForToolResources`, controller filtering) check `metadata.codeEnvRef`. Net effect of the bug: chat-attached and agent-setup execute_code files would lose their sandbox reference on save, and primeFiles would skip them on subsequent code-execution turns — the file blob would still be available locally but never re-mounted in the sandbox. Fix: construct the full `CodeEnvRef` (`{ kind, id, storage_session_id, file_id }`) at the write site and persist under `metadata.codeEnvRef`. `BaseClient`'s "is this a code-env file" presence check accepts the new shape alongside the legacy `fileIdentifier` for back-compat with any pre-cutover records still in the database. Mirrors the same change in `processAttachments.spec.ts` (which re-implements the BaseClient logic for testability). New regression tests in `process.spec.js` cover three cases: - chat attachments (`messageAttachment=true`) → `kind: 'user'` - agent setup (`messageAttachment=false`) → `kind: 'agent'` - legacy `fileIdentifier` key is NOT persisted (would be schema-stripped) * 🩹 fix: read storage_session_id on primed file refs (Codex P1) Codex review (chatgpt-codex-connector). After Phase B's per-file `session_id` → `storage_session_id` rename, `primeFiles` emits the new field — but `seedCodeFilesIntoSessions` was still reading `files[0].session_id` for the representative session and `f.session_id` for the dedupe key. In runs with only primed attachments (no skill seed), `representativeSessionId` was `undefined`, the function returned the unchanged map, and `seedCodeFilesIntoSessions` silently dropped the entire batch. The first `execute_code` call then started without `_injected_files` and the agent couldn't see prior-turn artifacts. Fix: - `codeFilesSession.ts`: read `f.storage_session_id` for both the dedupe key and the representative session id. JSDoc updated to match the new field name. - `callbacks.js`: the two output-file persistence paths read `file.session_id` to pass to `processCodeOutput` — switch to `file.storage_session_id`. The original comment explicitly says this should be the STORAGE session, which is exactly the field Phase B renamed. - `codeFilesSession.spec.ts`: fixture builder uses `storage_session_id` and `kind: 'user'` to match the post-cutover `CodeEnvFile` shape. Lockstep coordination: this matches the post-bump shape of `@librechat/agents` 3.1.79+. CI tsc errors against the currently-pinned 3.1.78 are expected and resolve when the dep bumps in this PR before merge. * 📦 chore: Bump `@librechat/agents` to version 3.1.80-dev.0 in package-lock and package.json files * 🪪 fix: thread kind/id/version through codeapi /download URLs (Phase C α) Symmetric fix for the upload-side wire change in 537725a. Codeapi's `sessionAuth` middleware now requires `kind`/`id`/`version?` on every download/freshness URL — without them it 400s with "kind must be one of: skill, agent, user" before serving the file. Three sites construct codeapi-side URLs that go through `sessionAuth`: - `processCodeOutput` (`Files/Code/process.js`): `/download/<sess>/<id>` for freshly-generated sandbox outputs. Always `kind: 'user'` + `id: req.user.id` — code-output files are always user-private, regardless of which skill the run invoked. - `getSessionInfo` (`Files/Code/process.js`): `/sessions/<sess>/objects/<id>` for the 23h freshness check. Pulls kind/id/version straight off the `codeEnvRef` already in scope — skill files stay skill-bucketed, user files stay user-bucketed. - `/code/download/:session_id/:fileId` LC route (`routes/files/files.js`): proxies to codeapi for manual downloads. Code-output files only on this route, so `kind: 'user'` + `id: req.user.id`. The `getCodeOutputDownloadStream` helper in `crud.js` now takes an `identity` param, validated by a `buildCodeEnvDownloadQuery` helper that mirrors `appendCodeEnvFileIdentity`'s shape rules: kind required from the closed `{skill, agent, user}` set, version required for 'skill' and forbidden otherwise. Bad callers fail fast on the client instead of round-tripping a 400. Also cleans up two log-noise sources reported alongside the 400: - `logAxiosError` in `packages/api/src/utils/axios.ts` was dumping `error.response.data` raw. With `responseType: 'arraybuffer'` that's a `Buffer` (~4 chars per byte after JSON-serialization); with `responseType: 'stream'` it's a `Readable` whose internal state serializes the entire ring buffer + socket. New `renderResponseData` decodes small buffers as UTF-8 (truncated past 2KB) and stubs streams as `'[stream]'`. Diagnostics stay useful, log lines stop being megabytes. - `/code/download` route's catch was bare `logger.error('...', error)`, bypassing the redactor. Switched to `logAxiosError` so it benefits from the same buffer/stream handling. Tests updated to match the new contract: - crud.spec: `getCodeOutputDownloadStream` fixtures pass `userIdentity`; new cases cover skill identity (with version), bad kind rejection, skill-without-version rejection. - process.spec: `getSessionInfo` test passes a full `codeEnvRef` object. * ♻️ refactor: extract codeEnv identity helpers into packages/api Per the project convention that new backend code lives in TypeScript under `packages/api`, moves `appendCodeEnvFileIdentity` and `buildCodeEnvDownloadQuery` from `api/server/services/Files/Code/crud.js` into a new `packages/api/src/files/code/identity.ts` module. Both helpers are pure validators that mirror codeapi's `parseUploadSessionKeyInput` server-side rules (closed kind set, `version` required for `'skill'` and forbidden otherwise) — they deserve TS support and a dedicated spec rather than living as JSDoc-typed helpers in the legacy `/api` workspace. The new module: - Exports a `CodeEnvIdentity` interface using the `librechat-data-provider` `CodeEnvKind` discriminated union. - Adds 13 unit tests in `identity.spec.ts` covering the validation matrix (skill+version, agent, user, and every rejection path) plus URL encoding for the download query. - Re-exported from `packages/api/src/files/code/index.ts` alongside `classify`, `extract`, and `form`. Consumer updates: - `api/server/services/Files/Code/crud.js`: drops the local helpers and imports them from `@librechat/api`. Net -64 lines. - `api/server/services/Files/Code/process.js`: same. - Test mocks for `@librechat/api` in three spec files now stub the helpers' validation behavior locally rather than pulling them through `requireActual` (which would drag in provider-config init-time side effects). The package's `exports` field only surfaces the root barrel, so leaf imports aren't reachable from legacy `/api` test setup. No runtime behavior change. Identity validation rules and emitted form/query shapes are byte-for-byte identical pre/post. * 🪪 fix: emit resource_id alongside id on _injected_files (skill 403 fix) Companion to codeapi #1455 fix and agents 3.1.80-dev.1 — the wire shape for shared-kind files now requires `resource_id` distinct from the storage `id`. Without this LC change, codeapi's sessionKey re-derivation on every shared-kind /exec rejects with 403 session_key_mismatch: cached: legacy:skill:69dcf561...✌️59 (signed at upload, skill _id) derived: legacy:skill:ysPwEURuPk-...✌️59 (storage nanoid) Emit sites updated: - `primeInvokedSkills` cache-hit path: `resource_id: ref.id` (the persisted skill `_id` from `codeEnvRef.id`); `id: ref.file_id` unchanged (storage uuid). - `primeInvokedSkills` fresh-upload path: `resource_id: skill._id.toString()` on every primed file (the `allPrimedFiles` builder type now carries the field). - `processCodeOutput`'s `pushFile` (Code/process.js): `resource_id: ref.id` — for `kind: 'user'` this is informational (codeapi derives sessionKey from auth context) but emitted for shape uniformity with shared kinds. Bumps `@librechat/agents` to `^3.1.80-dev.1` (the version that ships the matching `CodeEnvFile.resource_id` field). ## Test plan - [x] `cd packages/api && npx jest src/agents` — 67 / 67 pass (skillFiles fixtures updated to assert `resource_id` on the emitted CodeSessionContext.files). - [x] `cd api && npx jest server/services/Files server/controllers/agents` — 445 / 445 pass (process.spec fixtures updated for the reupload + cache-hit emission). - [x] `npx tsc --noEmit -p packages/api/tsconfig.json` — clean. * fix(skill-tool-call): carry resource_id through primeSkillFiles → artifact Codeapi was 400ing every /exec following a `handle_skill` tool call with `resource_id is invalid` (`type: 'undefined'`). Both code paths in `primeSkillFiles` (cache-hit + fresh-upload) returned files without `resource_id`/`kind`/`version`, and the artifact in `handlers.ts` forwarded the stripped shape into `tc.codeSessionContext.files` → `_injected_files`. `primeInvokedSkills` (the NL-detected loader) had already been fixed end-to-end; this commit aligns the tool-invoked path with the same contract: `resource_id` = `skill._id.toString()`, `kind: 'skill'`, `version` = the skill's monotonic counter. Tests added to `skillFiles.spec.ts` lock the contract on `primeSkillFiles` directly so future refactors can't silently drop the resource identity again. * fix(handlers.spec): align session_id → storage_session_id rename + kind discriminator Pre-existing TS errors against the post-rename `CodeEnvFile` shape: the test file still used `session_id` on per-file objects (renamed to `storage_session_id` in agents Phase B/C) and was missing the `kind` discriminator the discriminated union requires. Both inputs and the matching `expect.toEqual(...)` mirrors updated together so the runtime equality check still holds. Lines 723-732 stay as-is — they sit behind `as unknown as ToolCallRequest` and TS already skipped them. * chore: fix `@librechat/agents`, correct version to 3.1.80-dev.0 in package.json files * chore: bump `@librechat/agents` to version 3.1.80-dev.1 in package.json and package-lock.json * chore: bump `@librechat/agents` to version 3.1.80-dev.2 * feat(observability): trace file priming chain from primeCodeFiles to _injected_files Diagnosing the user-upload "files=[] on first /exec" bug requires seeing where in the LC chain a file ref disappears. Prior to this patch the chain (primeCodeFiles → primedCodeFiles → initialSessions → CodeSessionContext → _injected_files) was opaque end-to-end: - primeCodeFiles silently dropped files without `metadata.codeEnvRef` - reuploadFile catches all errors and continues with no signal - the handlers.ts handoff to codeapi never logged what it was sending After this patch, a single grep on `[primeCodeFiles]` plus `[code-env:inject]` shows the full per-file path: [primeCodeFiles] in: file_ids=N resourceFiles=M [primeCodeFiles] file=<id> path=skip reason=no-codeenvref filename=... [primeCodeFiles] file=<id> path=cache-hit-by-session storage_session_id=... [primeCodeFiles] file=<id> path=reupload reason=no-uploadtime ... [primeCodeFiles] file=<id> path=reupload reason=stale ... [primeCodeFiles] file=<id> path=reupload-success oldSession=... newSession=... newFileId=... [primeCodeFiles] file=<id> path=reupload-failed session=... [primeCodeFiles] file=<id> path=fresh-active storage_session_id=... [primeCodeFiles] out: returned=N skippedNoRef=M reuploadFailures=K [code-env:inject] tool=<name> files=N missingResourceId=K (debug) [code-env:inject] M/N files missing resource_id ... (warn) [code-env:inject] tool=<name> _injected_files=0 ... (warn) The boundary log warns when LC sends zero injected files on a code-execution tool call — that's the user's actual symptom showing up at the LC side instead of having to correlate against codeapi's `Request received { files: [] }`. Tag chosen as `[code-env:inject]` rather than `[handoff:exec]` to avoid collision with the app-level "handoff" semantic (subagent handoff workflow). Structural cleanup in primeFiles: replaced the `if (ref) { ... }` nesting with an early `if (!ref) continue` so the per-path instrumentation hooks land at top-level scope instead of indented inside a conditional. Behavior unchanged; pushFile / reuploadFile identical. Spec fixtures (handlers.spec.ts, codeFilesSession.spec.ts) updated to include `resource_id` on `CodeEnvFile` literals — required by the post-3.1.80-dev.2 type now installed. ## Test plan - [x] `cd packages/api && npx jest src/agents/handlers.spec.ts src/agents/codeFilesSession.spec.ts src/agents/skillFiles.spec.ts` — 69/69 pass - [x] `cd api && npx jest server/services/Files/Code/process.spec.js` — 84/84 pass - [x] `npx tsc --noEmit -p packages/api` — clean - [x] `npx eslint` on all four touched files — clean * chore: add CONSOLE_JSON_STRING_LENGTH to .env.example for JSON log string length configuration * fix(files): align codeapi upload filename with LC's sanitized DB filename User-attached files for code execution were uploading to codeapi under `file.originalname` (raw upload filename, may contain spaces / special chars) while LC's DB record stored the sanitized form (`sanitizeFilename(file.originalname)`, underscores). Codeapi preserves whatever filename the upload sent, so the sandbox saw `/mnt/data/<originalname>` while LC's `primeFiles` toolContext text + `_injected_files.name` referenced `file.filename` (sanitized). Visible failure: agent gets system prompt saying /mnt/data/librechat_code_api_-_active_customer_-_2025-11-05.xlsx …tries that path, hits `FileNotFoundError`, then notices the sandbox's actual `Available files` line says /mnt/data/librechat code api - active customer - 2025-11-05.xlsx …retries with spaces, succeeds. Wastes a tool call per upload and leaks raw filenames into model context. Fix: sanitize once and use the sanitized form in both the codeapi upload AND the LC DB record. Sandbox path = LC toolContext text = in-memory ref name. No drift. Reupload path (`Code/process.js` line 867 `filename: file.filename`) already uses the sanitized DB name, so it stays consistent with the fresh-upload path after this change. ## Test plan - [x] `cd api && npx jest server/services/Files/process` — 32/32 pass - [x] `npx eslint` on the touched file — clean * chore: bump `@librechat/agents` to version 3.1.80-dev.3 in package.json and package-lock.json
This commit is contained in:
Danny Avila
parent
9441563b95
commit
93c4ef4ba8
40 changed files with 1937 additions and 456 deletions
|
|
@ -16,6 +16,7 @@ const {
|
|||
extractCodeArtifactText,
|
||||
getExtractedTextFormat,
|
||||
getStorageMetadata,
|
||||
buildCodeEnvDownloadQuery,
|
||||
} = require('@librechat/api');
|
||||
const {
|
||||
Tools,
|
||||
|
|
@ -286,7 +287,7 @@ const runPreviewFinalize = ({ finalize, fileId, previewRevision, onResolved }) =
|
|||
/**
|
||||
* Process code execution output files — downloads and saves both images
|
||||
* and non-image files. All files are saved to local storage with
|
||||
* `fileIdentifier` metadata for code env re-upload.
|
||||
* `codeEnvRef` metadata for code env re-upload.
|
||||
*
|
||||
* Returns a two-part shape so callers can ship the attachment to the
|
||||
* client immediately and run preview extraction in the background:
|
||||
|
|
@ -334,9 +335,15 @@ const processCodeOutput = async ({
|
|||
|
||||
try {
|
||||
const formattedDate = currentDate.toISOString();
|
||||
/* Code-output files are always user-private — no skill execution
|
||||
* produces a skill-scoped output bucket. The download URL must
|
||||
* carry `?kind=user&id=<userId>` so codeapi's `sessionAuth`
|
||||
* resolves the matching `<tenant>:user:<userId>` sessionKey. See
|
||||
* codeapi #1455 / Phase C. */
|
||||
const downloadQuery = buildCodeEnvDownloadQuery({ kind: 'user', id: req.user.id });
|
||||
const response = await axios({
|
||||
method: 'get',
|
||||
url: `${baseURL}/download/${session_id}/${id}`,
|
||||
url: `${baseURL}/download/${session_id}/${id}${downloadQuery}`,
|
||||
responseType: 'arraybuffer',
|
||||
headers: {
|
||||
'User-Agent': 'LibreChat/1.0',
|
||||
|
|
@ -366,7 +373,15 @@ const processCodeOutput = async ({
|
|||
};
|
||||
}
|
||||
|
||||
const fileIdentifier = `${session_id}/${id}`;
|
||||
/* Code-output files belong to the user who ran the execution.
|
||||
* SessionKey on codeapi will be `<tenant>:user:<userId>` for these,
|
||||
* so cache and access stay user-private. */
|
||||
const codeEnvRef = {
|
||||
kind: 'user',
|
||||
id: req.user.id,
|
||||
storage_session_id: session_id,
|
||||
file_id: id,
|
||||
};
|
||||
|
||||
/* `safeName` keeps the directory structure (`a/b/file.txt` -> `a/b/file.txt`)
|
||||
* so the next prime() can place the file at the same nested path in the
|
||||
|
|
@ -444,7 +459,7 @@ const processCodeOutput = async ({
|
|||
updatedAt: formattedDate,
|
||||
source: appConfig.fileStrategy,
|
||||
context: FileContext.execute_code,
|
||||
metadata: { fileIdentifier },
|
||||
metadata: { codeEnvRef },
|
||||
};
|
||||
await createFile(file, true);
|
||||
return { file: Object.assign(file, { messageId, toolCallId }) };
|
||||
|
|
@ -542,7 +557,7 @@ const processCodeOutput = async ({
|
|||
tenantId: req.user.tenantId,
|
||||
bytes: buffer.length,
|
||||
updatedAt: formattedDate,
|
||||
metadata: { fileIdentifier },
|
||||
metadata: { codeEnvRef },
|
||||
source: appConfig.fileStrategy,
|
||||
context: FileContext.execute_code,
|
||||
usage: isUpdate ? (claimed.usage ?? 0) + 1 : 1,
|
||||
|
|
@ -651,26 +666,31 @@ function checkIfActive(dateString) {
|
|||
/**
|
||||
* Retrieves the `lastModified` time string for a specified file from Code Execution Server.
|
||||
*
|
||||
* @param {string} fileIdentifier - The identifier for the file (e.g., "session_id/fileId").
|
||||
* @param {import('librechat-data-provider').CodeEnvRef} ref - Typed pointer
|
||||
* into codeapi storage. Carries kind/id/storage_session_id/file_id;
|
||||
* codeapi resolves the sessionKey from the request's auth context.
|
||||
*
|
||||
* @returns {Promise<string|null>}
|
||||
* A promise that resolves to the `lastModified` time string of the file if successful, or null if there is an
|
||||
* error in initialization or fetching the info.
|
||||
*/
|
||||
async function getSessionInfo(fileIdentifier) {
|
||||
async function getSessionInfo(ref) {
|
||||
try {
|
||||
const baseURL = getCodeBaseURL();
|
||||
const [path, queryString] = fileIdentifier.split('?');
|
||||
const [session_id, fileId] = path.split('/');
|
||||
let queryParams = {};
|
||||
if (queryString) {
|
||||
queryParams = Object.fromEntries(new URLSearchParams(queryString).entries());
|
||||
}
|
||||
|
||||
/* `/sessions/.../objects/...` is gated by codeapi's `sessionAuth`
|
||||
* middleware (post-Phase C). The middleware reconstructs the
|
||||
* sessionKey from the URL query (`kind`/`id`/`version?`) plus the
|
||||
* requester's auth context, then matches it against the cached
|
||||
* sessionKey on the storage bucket. We have the full `codeEnvRef`
|
||||
* here, so pass kind+id (+version when skill) directly. */
|
||||
const query = buildCodeEnvDownloadQuery({
|
||||
kind: ref.kind,
|
||||
id: ref.id,
|
||||
...(ref.kind === 'skill' ? { version: ref.version } : {}),
|
||||
});
|
||||
const response = await axios({
|
||||
method: 'get',
|
||||
url: `${baseURL}/sessions/${session_id}/objects/${fileId}`,
|
||||
params: queryParams,
|
||||
url: `${baseURL}/sessions/${ref.storage_session_id}/objects/${ref.file_id}${query}`,
|
||||
headers: {
|
||||
'User-Agent': 'LibreChat/1.0',
|
||||
},
|
||||
|
|
@ -706,6 +726,15 @@ const primeFiles = async (options) => {
|
|||
const agentResourceIds = new Set(file_ids);
|
||||
const resourceFiles = tool_resources?.[EToolResources.execute_code]?.files ?? [];
|
||||
|
||||
/* Step 1 of the priming trace: input volume. Pair with the
|
||||
* per-file `[primeCodeFiles] file=...` lines and the final
|
||||
* `[primeCodeFiles] returned=...` line below to locate which
|
||||
* layer drops a file the sandbox doesn't end up seeing. */
|
||||
logger.debug(
|
||||
`[primeCodeFiles] in: file_ids=${file_ids.length} resourceFiles=${resourceFiles.length}`,
|
||||
{ agentId, file_ids, resourceFileIds: resourceFiles.map((f) => f?.file_id) },
|
||||
);
|
||||
|
||||
// Get all files first
|
||||
const allFiles = (await getFiles({ file_id: { $in: file_ids } }, null, { text: 0 })) ?? [];
|
||||
|
||||
|
|
@ -728,146 +757,195 @@ const primeFiles = async (options) => {
|
|||
const sessions = new Map();
|
||||
let toolContext = '';
|
||||
|
||||
/* Per-file path counters — emitted at the bottom so a single
|
||||
* grep on `[primeCodeFiles]` shows the input volume, the per-file
|
||||
* paths taken, and the final dispatch summary in one trace. */
|
||||
let skippedNoRef = 0;
|
||||
let reuploadFailures = 0;
|
||||
|
||||
for (let i = 0; i < dbFiles.length; i++) {
|
||||
const file = dbFiles[i];
|
||||
if (!file) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (file.metadata.fileIdentifier) {
|
||||
const [path, queryString] = file.metadata.fileIdentifier.split('?');
|
||||
const [session_id, id] = path.split('/');
|
||||
|
||||
let queryParams = {};
|
||||
if (queryString) {
|
||||
queryParams = Object.fromEntries(new URLSearchParams(queryString).entries());
|
||||
}
|
||||
|
||||
/**
|
||||
* `pushFile` accepts optional overrides so the reupload path can
|
||||
* push the FRESH `(session_id, id, entity_id)` parsed off the new
|
||||
* `fileIdentifier`. Without these overrides, the closure would
|
||||
* capture the stale pre-reupload refs from the outer loop and
|
||||
* the in-memory `files` array (now consumed by
|
||||
* `buildInitialToolSessions` to seed `Graph.sessions`) would
|
||||
* point at a sandbox object that no longer exists. The DB record
|
||||
* gets the new identifier via `updateFile`, but the seed would
|
||||
* still inject the old one — bash_tool / read_file would 404
|
||||
* trying to mount the file until the next turn re-reads metadata.
|
||||
*
|
||||
* `entity_id` is forwarded so codeapi can resolve sessionKey
|
||||
* per-file, allowing one execute to mix files uploaded under
|
||||
* different entities (e.g. a skill bundle plus a user attachment).
|
||||
*/
|
||||
const pushFile = (overrideSessionId, overrideId, overrideEntityId) => {
|
||||
if (!toolContext) {
|
||||
toolContext = `- Note: The following files are available in the "${Tools.execute_code}" tool environment:`;
|
||||
}
|
||||
|
||||
let fileSuffix = '';
|
||||
if (!agentResourceIds.has(file.file_id)) {
|
||||
fileSuffix =
|
||||
file.context === FileContext.execute_code
|
||||
? ' (from previous code execution)'
|
||||
: ' (attached by user)';
|
||||
}
|
||||
|
||||
const entity_id = overrideEntityId ?? queryParams.entity_id;
|
||||
|
||||
/* Surface the preview lifecycle so the LLM knows when a
|
||||
* prior-turn artifact's rich preview didn't materialize. The
|
||||
* file blob is always available (`processCodeOutput` persists
|
||||
* it before returning), so the model can still tell the user
|
||||
* "you can download it" even when the preview never resolved.
|
||||
* Absent status means legacy or non-office — render normally. */
|
||||
let previewSuffix = '';
|
||||
if (file.status === 'pending') {
|
||||
previewSuffix = ' (preview not yet generated)';
|
||||
} else if (file.status === 'failed') {
|
||||
previewSuffix = file.previewError
|
||||
? ` (preview unavailable: ${file.previewError})`
|
||||
: ' (preview unavailable)';
|
||||
}
|
||||
|
||||
toolContext += `\n\t- /mnt/data/${file.filename}${fileSuffix}${previewSuffix}`;
|
||||
files.push({
|
||||
id: overrideId ?? id,
|
||||
session_id: overrideSessionId ?? session_id,
|
||||
name: file.filename,
|
||||
...(entity_id ? { entity_id } : {}),
|
||||
});
|
||||
};
|
||||
|
||||
if (sessions.has(session_id)) {
|
||||
pushFile();
|
||||
continue;
|
||||
}
|
||||
|
||||
const reuploadFile = async () => {
|
||||
try {
|
||||
const { getDownloadStream } = getStrategyFunctions(file.source);
|
||||
const { handleFileUpload: uploadCodeEnvFile } = getStrategyFunctions(
|
||||
FileSources.execute_code,
|
||||
);
|
||||
const stream = await getDownloadStream(options.req, file.filepath);
|
||||
const fileIdentifier = await uploadCodeEnvFile({
|
||||
req: options.req,
|
||||
stream,
|
||||
filename: file.filename,
|
||||
entity_id: queryParams.entity_id,
|
||||
});
|
||||
|
||||
// Preserve existing metadata when adding fileIdentifier
|
||||
const updatedMetadata = {
|
||||
...file.metadata, // Preserve existing metadata (like S3 storage info)
|
||||
fileIdentifier, // Add fileIdentifier
|
||||
};
|
||||
|
||||
await updateFile({
|
||||
file_id: file.file_id,
|
||||
metadata: updatedMetadata,
|
||||
});
|
||||
/**
|
||||
* Parse the FRESH fileIdentifier returned by the reupload and
|
||||
* route it through both the dedupe Map and the in-memory
|
||||
* `files` list. The original `(session_id, id)` parsed at the
|
||||
* top of this iteration refer to the old, expired/missing
|
||||
* sandbox object — using them here would silently re-introduce
|
||||
* the bug `Graph.sessions` seeding is supposed to fix.
|
||||
*
|
||||
* `entity_id` survives the round-trip: the upload was tagged
|
||||
* with `queryParams.entity_id` above, so the new identifier
|
||||
* carries the same scope.
|
||||
*/
|
||||
const [newPath, newQuery] = fileIdentifier.split('?');
|
||||
const [newSessionId, newId] = newPath.split('/');
|
||||
const newQueryParams = newQuery
|
||||
? Object.fromEntries(new URLSearchParams(newQuery).entries())
|
||||
: {};
|
||||
sessions.set(newSessionId, true);
|
||||
pushFile(newSessionId, newId, newQueryParams.entity_id);
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
`Error re-uploading file ${id} in session ${session_id}: ${error.message}`,
|
||||
error,
|
||||
);
|
||||
}
|
||||
};
|
||||
const uploadTime = await getSessionInfo(file.metadata.fileIdentifier);
|
||||
if (!uploadTime) {
|
||||
logger.warn(`Failed to get upload time for file ${id} in session ${session_id}`);
|
||||
await reuploadFile();
|
||||
continue;
|
||||
}
|
||||
if (!checkIfActive(uploadTime)) {
|
||||
await reuploadFile();
|
||||
continue;
|
||||
}
|
||||
sessions.set(session_id, true);
|
||||
pushFile();
|
||||
const ref = file.metadata?.codeEnvRef;
|
||||
if (!ref) {
|
||||
skippedNoRef += 1;
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=skip reason=no-codeenvref filename=${file.filename}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const session_id = ref.storage_session_id;
|
||||
const id = ref.file_id;
|
||||
|
||||
/**
|
||||
* `pushFile` accepts optional overrides so the reupload path can
|
||||
* push the FRESH `(storage_session_id, file_id)` from the new
|
||||
* `codeEnvRef`. Without these overrides, the closure would
|
||||
* capture the stale pre-reupload refs from the outer loop and
|
||||
* the in-memory `files` array (now consumed by
|
||||
* `buildInitialToolSessions` to seed `Graph.sessions`) would
|
||||
* point at a sandbox object that no longer exists. The DB record
|
||||
* gets the new ref via `updateFile`, but the seed would still
|
||||
* inject the old one — bash_tool / read_file would 404 trying to
|
||||
* mount the file until the next turn re-reads metadata.
|
||||
*
|
||||
* `kind`, `id`, `version` are preserved on the in-memory ref so
|
||||
* codeapi can resolve sessionKey per-file (kind switch +
|
||||
* tenant prefix from auth context).
|
||||
*/
|
||||
const pushFile = (overrideSessionId, overrideId) => {
|
||||
if (!toolContext) {
|
||||
toolContext = `- Note: The following files are available in the "${Tools.execute_code}" tool environment:`;
|
||||
}
|
||||
|
||||
let fileSuffix = '';
|
||||
if (!agentResourceIds.has(file.file_id)) {
|
||||
fileSuffix =
|
||||
file.context === FileContext.execute_code
|
||||
? ' (from previous code execution)'
|
||||
: ' (attached by user)';
|
||||
}
|
||||
|
||||
/* Surface the preview lifecycle so the LLM knows when a
|
||||
* prior-turn artifact's rich preview didn't materialize. The
|
||||
* file blob is always available (`processCodeOutput` persists
|
||||
* it before returning), so the model can still tell the user
|
||||
* "you can download it" even when the preview never resolved.
|
||||
* Absent status means legacy or non-office — render normally. */
|
||||
let previewSuffix = '';
|
||||
if (file.status === 'pending') {
|
||||
previewSuffix = ' (preview not yet generated)';
|
||||
} else if (file.status === 'failed') {
|
||||
previewSuffix = file.previewError
|
||||
? ` (preview unavailable: ${file.previewError})`
|
||||
: ' (preview unavailable)';
|
||||
}
|
||||
|
||||
toolContext += `\n\t- /mnt/data/${file.filename}${fileSuffix}${previewSuffix}`;
|
||||
/* `id` is the storage file_id (drives codeapi's upload-key
|
||||
* existence check), `resource_id` is the entity that owns
|
||||
* the storage session (drives sessionKey re-derivation). For
|
||||
* code-output files this is `kind: 'user'` and `resource_id`
|
||||
* is informational (codeapi ignores it for user kind), but
|
||||
* we still send it for shape uniformity with shared kinds. */
|
||||
files.push({
|
||||
id: overrideId ?? id,
|
||||
resource_id: ref.id,
|
||||
storage_session_id: overrideSessionId ?? session_id,
|
||||
name: file.filename,
|
||||
kind: ref.kind,
|
||||
...(ref.kind === 'skill' ? { version: ref.version } : {}),
|
||||
});
|
||||
};
|
||||
|
||||
if (sessions.has(session_id)) {
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=cache-hit-by-session storage_session_id=${session_id}`,
|
||||
);
|
||||
pushFile();
|
||||
continue;
|
||||
}
|
||||
|
||||
const reuploadFile = async () => {
|
||||
try {
|
||||
const { getDownloadStream } = getStrategyFunctions(file.source);
|
||||
const { handleFileUpload: uploadCodeEnvFile } = getStrategyFunctions(
|
||||
FileSources.execute_code,
|
||||
);
|
||||
const stream = await getDownloadStream(options.req, file.filepath);
|
||||
/* Reupload preserves the resource identity from the existing
|
||||
* ref so codeapi re-buckets under the same sessionKey shape
|
||||
* (skill stays skill, user stays user). Without this, a
|
||||
* skill-cache-miss reupload would land in the user bucket
|
||||
* and never re-shareable cross-user. */
|
||||
const uploaded = await uploadCodeEnvFile({
|
||||
req: options.req,
|
||||
stream,
|
||||
filename: file.filename,
|
||||
kind: ref.kind,
|
||||
id: ref.id,
|
||||
...(ref.kind === 'skill' ? { version: ref.version } : {}),
|
||||
});
|
||||
|
||||
/**
|
||||
* Use the FRESH `(storage_session_id, file_id)` from the
|
||||
* reupload response and route it through the dedupe Map, the
|
||||
* persisted record, and the in-memory `files` list. The
|
||||
* original ref captured at the top of this iteration refers
|
||||
* to the old, expired/missing sandbox object — using it here
|
||||
* would silently re-introduce the bug `Graph.sessions`
|
||||
* seeding is supposed to fix.
|
||||
*
|
||||
* `kind`, `id`, `version` survive the round-trip: the
|
||||
* upload preserves the resource identity, only the storage
|
||||
* pointer changes.
|
||||
*/
|
||||
const newRef = {
|
||||
kind: ref.kind,
|
||||
id: ref.id,
|
||||
storage_session_id: uploaded.storage_session_id,
|
||||
file_id: uploaded.file_id,
|
||||
...(ref.kind === 'skill' ? { version: ref.version } : {}),
|
||||
};
|
||||
|
||||
const updatedMetadata = {
|
||||
...file.metadata,
|
||||
codeEnvRef: newRef,
|
||||
};
|
||||
|
||||
await updateFile({
|
||||
file_id: file.file_id,
|
||||
metadata: updatedMetadata,
|
||||
});
|
||||
sessions.set(newRef.storage_session_id, true);
|
||||
pushFile(newRef.storage_session_id, newRef.file_id);
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=reupload-success ` +
|
||||
`oldSession=${session_id} newSession=${newRef.storage_session_id} newFileId=${newRef.file_id}`,
|
||||
);
|
||||
} catch (error) {
|
||||
reuploadFailures += 1;
|
||||
logger.error(
|
||||
`[primeCodeFiles] file=${file.file_id} path=reupload-failed session=${session_id}: ${error.message}`,
|
||||
error,
|
||||
);
|
||||
}
|
||||
};
|
||||
const uploadTime = await getSessionInfo(ref);
|
||||
if (!uploadTime) {
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=reupload reason=no-uploadtime ` +
|
||||
`storage_session_id=${session_id}`,
|
||||
);
|
||||
await reuploadFile();
|
||||
continue;
|
||||
}
|
||||
if (!checkIfActive(uploadTime)) {
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=reupload reason=stale ` +
|
||||
`uploadTime=${uploadTime} storage_session_id=${session_id}`,
|
||||
);
|
||||
await reuploadFile();
|
||||
continue;
|
||||
}
|
||||
sessions.set(session_id, true);
|
||||
logger.debug(
|
||||
`[primeCodeFiles] file=${file.file_id} path=fresh-active storage_session_id=${session_id}`,
|
||||
);
|
||||
pushFile();
|
||||
}
|
||||
|
||||
/* Dispatch summary — emitted unconditionally so a single grep on
|
||||
* `[primeCodeFiles] out` always shows the final state, not only
|
||||
* the per-path trail leading up to it. */
|
||||
logger.debug(
|
||||
`[primeCodeFiles] out: returned=${files.length} ` +
|
||||
`skippedNoRef=${skippedNoRef} reuploadFailures=${reuploadFailures}`,
|
||||
);
|
||||
|
||||
return { files, toolContext };
|
||||
};
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue