Commit graph

4 commits

Author SHA1 Message Date
Danny Avila
65bb64bc68
🛟 fix: Recover CodeAPI File Uploads After Throttling (#15784)
* fix: recover CodeAPI uploads after throttling

* fix: cover eager CodeAPI uploads

* fix: scope CodeAPI upload recovery

* test: clarify upload recovery deadlines

* fix: isolate CodeAPI recovery state

* fix: cancel CodeAPI recovery waits

* test: provide CodeAPI registry in handler fixture

* fix: propagate CodeAPI recovery controls

* fix: bound CodeAPI recovery by run

* fix: preserve skill upload cancellation

* test: update deferred loader signal contract

* fix: preserve recovery cancellation

* fix: cancel recovery source reads

* fix: classify Axios upload cancellation

* fix: preserve lazy provisioning cancellation

* fix: close resource recovery boundaries

* fix: preserve provisioning failures compatibly
2026-09-09 15:48:11 -04:00
Danny Avila
8f6228095f
🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format (#15453)
* 🖼️ fix: Stop Reporting a Failed Sandbox Image Read as an Unreadable Format

`read_file` on a `/mnt/data` image collapsed every failure — rate limit,
missing path, truncated transfer, transport error — into the binary-file
hint, whose image branch reads as a permanent capability limit ("is an
image file and cannot be read as text"). The model stopped retrying, and
the operator had no way to tell which failure had actually occurred.

The failure that surfaced it: the reader windows the bytes through
`/exec` stdout, one call per window, against the Code API's per-user
execution limiter (20 requests / 30s in the reference deployment). At a
fixed 32KB window a 122KB chart cost 4 calls and a 1MB image cost 33 —
more than a whole limiter window, so anything over ~640KB could never
complete. A turn that generates several charts and reads them back
exhausts the budget partway through and reports the 429s as unreadable
images.

- `handlers.ts`: failed reads name the real cause and the affordance that
  matches it — rate-limited waits and retries, a missing path lists the
  directory, a truncated transfer retries. The "cannot be read as text"
  phrasing is now reserved for bytes that are not a supported image. An
  image past the inline budget says to downscale and read the smaller
  copy rather than pointing at `bash_tool`, which cannot show an image.
- `process.js`: the window is derived from the runner's stdout budget
  (48,960B vs 32,768B, so a 122KB chart is 3 calls and 1MB is 22), halves
  and re-reads the same offset when a runner truncates anyway (remembered
  per Code API, with the env remedy logged), waits out a rate limit it
  can afford instead of discarding the bytes already pulled, and stops at
  a 24-call ceiling so no read stalls a turn across limiter windows.
- `utils/code.ts`: shared `getCodeApiRetryAfterMs` (header, then the
  `retry_after_seconds` body); `skillFiles.ts` drops its private copy.

* ♻️ refactor: Move the Sandbox Image Reader Into TypeScript

Addresses the review round on #15453.

- P1: windowing, window sizing, the in-sandbox reader script, response
  parsing and byte assembly move to `packages/api/src/files/code/image.ts`,
  and the Code API rate-limit policy to `packages/api/src/utils/code.ts`.
  `process.js` keeps only the `/exec` transport it owns. Both new modules
  are tested against their real implementations (19 + 13 cases), including
  the round-trip ceiling, the changed-file guard and the shell-injection
  guard on the reader script.
- P2: a configured stdout budget below the old 8KB window floor produced a
  window the runner could never emit, and narrowing was pinned to the same
  floor, so no retry could succeed. The window now aligns to the base64
  group size instead, and the round-trip ceiling bounds what a small budget
  costs.
- P2: `.env.example` documented a 32,768-byte default that no longer holds
  and never mentioned the stdout budget. It now describes
  `LIBRECHAT_CODE_SANDBOX_OUTPUT_MAX_SIZE` as the setting to tune, with the
  chunk override as the escape hatch.
- P3: `process.spec.js` no longer reimplements the retry parser in its
  module mock. Behaviour is tested where it lives; the mock is a declared
  passthrough and the spec covers what the transport still owns — session
  forwarding, the profile header, parser hand-off and error logging.
- P3: `getCodeApiRetryAfterMs` reads the response body through `in`
  narrowing rather than an `as` assertion.
- P3: the window-sizing JSDoc no longer claims a multiple of 3 in the one
  case that is not — an explicit `LIBRECHAT_CODE_IMAGE_CHUNK_BYTES`.

* 🎯 fix: Size Sandbox Image Reads to What the Limiter Can Deliver

Addresses review round 2 on #15453.

- Narrowing now halves the caller's own failing window instead of the
  shared learned value, and the learned value only moves down to the
  narrowest size any read needed. Reads run concurrently (one per tool
  call), so two reads failing at the same size were compounding into a
  window two steps smaller than the runner would have accepted, and
  persisting it for every later read.
- Narrowing continues until a runner accepts a window or a floor below
  which no runner could serve an image at all. Two halvings stopped at
  12,240 bytes — still ~16KB of base64 — so an 8-10KB stdout cap that was
  not configured up front stayed unreachable.
- The round-trip ceiling is now one limiter window (20), and a read that
  cannot finish within it says so from the FIRST response rather than
  spending the whole budget rediscovering it: the first window reveals
  both the file size and what a call really delivers. A near-limit image
  no longer drains the turn's executions before failing, which is what
  made the wait budget look too small.
- The round-trip message names the size this deployment can actually
  inline, so the model has a downscale target instead of a guess.
- `python3: not found` from the sandbox reader was matching the generic
  "not found" in `isSandboxMissingFileError` and being reported as a
  missing image path, sending the model to `ls /mnt/data` while hiding a
  runner dependency. Image reads now classify only path-specific ENOENT
  forms as a missing file.

* 📐 fix: Size a Truncated Read's Retry From the Cap It Revealed

Addresses review round 3 on #15453.

A ~1KB sandbox stdout cap is a real runner default, and the 1,536-byte
narrowing floor sat above it: discovery stopped one step short and
declared every image unreadable, even though small ones fit comfortably
in the 576-byte window the configured path already derives.

Narrowing now runs until no smaller window exists. It also no longer has
to grope its way down: a truncated response emitted exactly as much as
the runner allows, so its length measures the cap this deployment never
declared, and the retry is derived from it directly — one round-trip
instead of the eight blind halvings it takes to fall from ~49KB to ~1KB.
Blind halving remains the fallback and bounds the result either way, so
every retry is strictly smaller than the window that just failed.
2026-09-01 09:08:01 -04:00
Danny Avila
6bc75d24c8
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown

Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the
@librechat/api package build, mirroring the merged data-schemas migration.

- Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps,
  bundle first-party `~/` + relative imports)
- Annotate exports for isolatedDeclarations (codefix-driven). Collapse the
  tokens.ts model->token maps to Record<string, Record<string, number>> and
  switch validation.ts's runtime `files` field from z.any() to z.unknown()
  so no explicit `any` is introduced
- Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output
- Add src/telemetry.ts entry shim so the two index.ts entries don't collide
  in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts})
- Delete rollup.config.js

Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass,
both entries load via require(), legacy /api consumes them unchanged.

* 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys

The build-api cache keys hashed `packages/api/server-rollup.config.js`,
which never existed (api used `rollup.config.js`, now removed) — a copy-paste
artifact from the data-provider key that matched no file. Replace it with the
new `packages/api/tsdown.config.mjs` so edits to the build config (entry,
format, externals) bust the api build cache, matching the data-schemas key.
2026-06-08 10:54:48 -04:00
Danny Avila
39f5f83a8a
🔌 fix: Isolate Code-Server HTTP Agents to Prevent Socket Pool Contamination (#12311)
* 🔧 fix: Isolate HTTP agents for code-server axios requests

Prevents socket hang up after 5s on Node 19+ when code executor has
file attachments. follow-redirects (axios dep) leaks `socket.destroy`
as a timeout listener on TCP sockets; with Node 19+ defaulting to
keepAlive: true, tainted sockets re-enter the global pool and destroy
active node-fetch requests in CodeExecutor after the idle timeout.

Uses dedicated http/https agents with keepAlive: false for all axios
calls targeting CODE_BASEURL in crud.js and process.js.

Closes #12298

* ♻️ refactor: Extract code-server HTTP agents to shared module

- Move duplicated agent construction from crud.js and process.js into
  a shared agents.js module to eliminate DRY violation
- Switch process.js from raw `require('axios')` to `createAxiosInstance()`
  for proxy configuration parity with crud.js
- Fix import ordering in process.js (agent constants no longer split imports)
- Add 120s timeout to uploadCodeEnvFile (was the only code-server call
  without a timeout)

* ✅ test: Add regression tests for code-server socket isolation

- Add crud.spec.js covering getCodeOutputDownloadStream and
  uploadCodeEnvFile (agent options, timeout, URL, error handling)
- Add socket pool isolation tests to process.spec.js asserting
  keepAlive:false agents are forwarded to axios
- Update process.spec.js mocks for createAxiosInstance() migration

* ♻️ refactor: Move code-server agents to packages/api

Relocate agents.js from api/server/services/Files/Code/ to
packages/api/src/utils/code.ts per workspace conventions. Consumers
now import codeServerHttpAgent/codeServerHttpsAgent from @librechat/api.
2026-03-19 16:16:57 -04:00