Two review findings on the import job path.
The strategies default basePath to images, which on the local strategy is client/public/images: served statically, with authentication off unless secureImageLinks is set, so an imported PDF or audio file was retrievable by URL without a session. Non-image assets now take the uploads base every other document upload uses, and the base path travels with the backend so the image and document strategies each get the one they mean.
The per-user limit also said nothing about aggregate work: enough accounts each within their own limit still parse an export apiece in one heap, and inspection runs unbounded in the upload request besides. Both stages now sit under a node-wide ceiling, CONVERSATION_IMPORT_MAX_CONCURRENT, defaulting to three; requests over it get a 429 they can retry rather than a slot.
The claude/grok service specs stubbed maybeFlush as async () => undefined, which no longer matches the sink's Promise<boolean> and broke the type check. The three supertest-only import route tests carried no expect() call, which eslint's jest/expect-expect flags and CI treats as an error at --max-warnings 0; they now assert on the response body, and the user-scoping test also checks the owner still sees the job.
A conversation is buffered when it is converted; the flush is what writes
it. Claiming its assets at buffer time meant a flush that rejected left
those files behind, referenced by a conversation that never landed and
skipped by the cleanup. Claims are held pending and promoted only when a
flush actually commits, so maybeFlush now reports whether it ran.
The pre-scan also never consulted the cancel flag, so a cancelled job kept
inflating and parsing every remaining shard after the user was told it had
stopped.
If saveBuffer succeeded and createFile then rejected, the object was in
storage with no row pointing at it - and the caller only learns an asset
exists once ingestOne resolves, so nothing could ever find it to clean up.
A Claude local_resource references bytes the export does not ship, same as
an image, so it counts as unavailable. Rendering its label is not the same
as having it.
index.size undercounted it: directory records return before the check, and a
repeated filename overwrites the same map key. Either lets an archive carry
far more records than the cap allows, each costing real central-directory
traversal, while the limit believes the archive is nearly empty.
Removes com_ui_import_assets_one, com_ui_import_errors_one,
com_ui_import_report_assets_one, com_ui_import_report_one and
com_ui_import_stat_conversations_one, and returns the call sites to
positional interpolation so no key is left referencing {{count}}.
Under retentionMode all the batch builder gives imported conversations and
messages a deadline, but attachments got none - so an attachment outlived
the chat that showed it, in exactly the deployments that require everything
to expire. disableTTL only suppresses the short upload TTL, which is a
different field.
Re-importing a finished export is all skips, and the skip path advanced the
counter without publishing it, so the bar sat at its old value for the whole
run and then jumped to completed.
A ChatGPT run reads every shard twice, once to scan for assets and once to
convert. Charging both passes meant a legitimate export holding more than
half the decompressed limit failed partway through the second pass with a
zip-bomb error, despite passing the same limit at index time.
The budget bounds how much distinct data the archive can yield; a re-read
yields nothing new and each read is still bounded by the per-entry cap. The
aggregate guard still catches an archive that understates its sizes, which
is now tested by doctoring a central directory rather than by re-reading.
Passing a user replaced the id filter rather than narrowing it, so
deleteFiles([oneId], user) deleted every file that user owned. Reads exactly
like the opposite of what it did, and the import's asset cleanup called it
that way - releasing a single unreferenced attachment would have wiped the
account's files while leaving their storage objects behind.
Passing no ids with a user still means everything that user owns, which is
how account deletion calls it, and an unscoped call now refuses rather than
emptying the collection.
Three gaps where the implementation could break with a green suite. The
drag listeners and their enter/leave depth counter had no test at all, so
replacing the counter with a boolean passed everything. The Confirm and
Cancel buttons were only asserted on for labels and disabled state, so the
panel could have called mutate with a stale id or nothing at all. And the
fixtures' own specs only checked archive layout, so deleting the U+E202
citation marker - which renders as nothing in an editor and in a diff -
would have left every citation assertion downstream passing vacuously.
A single-item import rendered "1 conversations", "1 attachment(s)", and
"1 items could not be imported". The counts now use the i18next plural
convention already established by com_ui_tools_count, with the secondary
numbers left bare where they read correctly either way.
The seven job types were declared twice, once here and once in
librechat-data-provider, with nothing enforcing agreement - and the route
that serves them is untyped JS, so a drift's first symptom would be a client
polling a phase it does not recognise forever. They are aliased now, and
ImportJob is TImportJob plus the two fields the route strips, which makes
that strip provably exhaustive rather than a comment.
The barrel also re-exported every internal helper into a published package.
It now names what /api actually consumes.
A shard is buffered, decoded, and JSON-parsed, which peaks at roughly 3.2x
its size in heap, so a small crafted archive reaches the same ceiling a real
export does. The default drops to 256 MiB, far above any shard a real export
ships, with an env var for the deployment that proves otherwise. 512 MiB
stays the hard ceiling: V8 cannot build a longer string.
A deployment can point fileStrategies.image and fileStrategies.document at
different places, and an export mixes images with audio, video, and PDFs.
Resolving isImage: true once for the whole archive sent every non-image
attachment to the image backend, so it landed somewhere the deployment never
meant it to live and could be unavailable after a restart.
Both backends are still resolved up front, so a misconfigured strategy
fails the job at setup rather than surfacing as a per-asset error halfway
through; only the choice between them is per asset. saveBuffer now reports
which backend took the bytes so the file row records it.
onProgress fires once per conversation and once per asset, and each call is
a read and a write against the job store. On a Redis-backed deployment a
10k-conversation import is tens of thousands of serialized round trips,
which at managed-Redis latency costs more wall time than the import itself.
The client polls every two seconds, so a 500ms write cadence is invisible.
Also passes the storage strategy's delete through to the run, so it can
release the assets no conversation ended up referencing.
The asset phase runs to completion before the first conversation is
written, so cancelling in between left every ingested file referenced by
nothing: the rows are created with the TTL disabled and no sweep knows they
exist. Failing anywhere after ingestion had the same effect.
The run now records which pointers a buffered conversation claimed and
releases the rest on every exit path. expiresAt is not an alternative - it
is a MongoDB TTL index that drops the row and leaves the storage object
behind.
A systematically broken export produced one error string per conversation
and per asset, and the whole array is serialized into the job's cache
record and returned in full on every poll.
The skip set was also a snapshot taken once at job start, so a conversation
id repeated within one export - across shards, or because the user
concatenated two exports - imported twice.
The route emitted the raw env var, so an unset variable became 0 and the
client read that as "no limit" - which is every default install. Its
pre-flight size check never ran, and a too-large export was only rejected
after the whole upload, as a 413 with no way for the UI to have warned
first. The default is 1 GiB, not unlimited.
Only POST /import carried the import limiters, and /start launched an
unawaited run with no cap. The default budget allows 50 uploads per user per
15 minutes, so one account could park 50 archives and start them all at
once; each run peaks at several times its shard size in heap, which is an
OOM rather than a slow import.
/start now carries both limiters and refuses a second concurrent run per
user with 429, handing the job back to awaiting_confirmation so it can be
started once the running one finishes. A replay of the job that is already
running still gets 409 from the phase transition.
The run also no longer holds the Express request across its lifetime, its
failure patch is best-effort, and the detached promise has a terminal catch
- without it a rejection while recording a failure became an unhandled
rejection in a process serving live chat streams.
role="status" is implicitly aria-atomic, so the region is re-announced in
full whenever anything inside it changes. The counter moves every two
seconds for the length of a multi-minute import, which floods the polite
queue and starves every other announcement in the app. The heading and the
final outcome stay announced; the numbers move to the progress bar's
aria-valuetext, where a screen reader reads them on demand.
Nothing cleared the recorded id on a terminal phase, so it outlived the
import. Reopening Settings showed a stale report instead of the import
control, and once the 24h server TTL expired the row's resting state became
the "job lost" card for anyone who had ever run an import. The report still
stays on screen for the mount that saw it finish.
The hook focused on mount and on every poll-driven phase change, without
regard for where focus actually was. The settings search mounts each
matching setting as you type, so one keystroke matching "Import" pulled the
caret out of the search box, and a phase change every two seconds took focus
from whatever else was being operated in the dialog.
The files slot only existed on the sequential branch, so the parallel and
edit branches still dropped a non-image attachment. Edit mode also
double-rendered it: that branch emits one EditTextPart per text and think
part, and each wraps its output in a Container that rendered the files
again. The slot is now built once and rendered by all three branches, and
the edit containers opt out.
The import filter became extension-only, which rejected a valid JSON export
sent without a .json suffix even when its MIME type said application/json.
Content is inspected before anything is imported, so the filter only needs
to decide whether the bytes are worth writing to disk.
sanitizeFilename now takes the byte budget it truncates against. The import
storage prepends a 37-byte upload id, and at the default budget a long
export name sanitizes to a full 255 bytes and pushes the path component
past NAME_MAX, so multer fails with ENAMETOOLONG.
JSON.parse succeeds for null, an array, and nested values, so a
conversation_asset_file_names.json of the wrong shape reached
originalLeafName and threw on split - dropping the attachment over a
cosmetic file the import does not need. Only string values are kept now,
and anything else falls back to the same empty map malformed JSON uses.
A shard listed in the manifest but absent from the zip was filtered out
silently, so the surviving shards were treated as the whole export:
inspection undercounted it and the job reported success having skipped
every conversation in the missing file.
tether_browsing_display and tether_quote store their content in result,
text, or content rather than a parts array. isEmitted saves those messages
anyway, so every browsed page and quotation in an export imported as a
blank bubble.
breakCycles allocated a Set per message and re-walked the whole ancestor
chain, measuring 1.06s for a single 5,000-message conversation and 5.1s at
10,000 - sizes a long real thread reaches. Colouring each node once makes it
linear, and it now roots the message that actually closes the loop instead
of whichever one the outer iteration started from, so a descendant of a
cycle keeps its parent.
enforceOrdering walked its queue with shift(), which re-indexes the whole
remaining array each step and turns a single BFS quadratic on a wide tree.
Conversion is synchronous, so that stalls unrelated requests.
Citation and knowledge-block URLs went from the archive straight onto
message.attachments, and Web/Sources renders them as href/src with no
transform of its own. React 18 emits a javascript: URL rather than blocking
it, and a shared conversation carries its search results verbatim to every
viewer, so an uploaded export could script in the origin of anyone who
opened the share.
Filtering where the source is built covers all three importers and both
sinks a link reaches: the anchor list and the markdown fallback.
jest.spyOn resolves against Keyv's array-key overload, so the single-key
mock implementation did not fit and tsc failed on the spec. tsconfig.json
includes src/**/*, and backend-review runs exactly this check.
Container is only reached by messages that carry no content, so an
assistant turn with both reasoning and a generated file rendered the
reasoning and dropped the file: images have an image_file part, everything
else lives only on message.files. ContentParts now renders the non-image
half of that list.
A single dropped request or 5xx put the job query into an error state that
permanently disabled its interval, with retry, reconnect, focus and mount
refetching all off - so the panel declared a still-running import lost and
never saw it finish. Only a 404 now ends the poll; everything else retries
and keeps polling.
Two related gaps alongside it: a /start whose response is lost left the
client on the confirmation screen for an import that had already begun, so
the job is refetched on error to find out which happened; and the sidebar
was only refreshed for a completed job, though a failed or cancelled run
keeps every conversation it flushed before it stopped.
sweepStaleTempUploads deletes temp uploads by mtime alone and its age
cutoff matches the job TTL, so a job confirmed just short of a day after
upload could have its archive removed mid-run. Touching the file when the
run claims it restarts the clock the sweep reads.
The upload limit is 1 GiB because it is sized for a zip, whose shards are
each well under the 512 MiB per-entry cap. A bare .json between the two
cleared the client-side check and multer, then failed after being streamed
in full, reported as an oversized archive.
It now fails on the stat, with a message naming the workaround. The cap
itself stays where it is: V8 refuses to build a string longer than
536,870,888 characters, so a larger entry could never be parsed anyway.
default_model_slug is a historical ChatGPT identifier - auto, research,
gpt-5-t - that no endpoint serves, and it was being written as the
conversation's own model, which is what the next prompt is sent with. The
messages keep their historical slug for display; the conversation resolves
to the configured default, as the Claude and Grok converters already do.
The scan collected pointers from every conversation in the export and the
asset phase ran to completion before the conversion loop reached its
existing-id check. Re-uploading a finished export therefore wrote a second
copy of every attachment to storage, all of them unreferenced once the
conversations themselves were skipped, while the report showed zero
imported.
The background run reads the cancel flag and writes progress in separate
round trips, so a DELETE landing between one of those reads and its write
was overwritten by the stale snapshot the read returned: the job went back
to active, the next isCancelled said so, and the import kept going.
Every mutation now goes through the per-key lock that only confirmStart
held, and a job that has reached a terminal status no longer accepts a
status or phase change - only the partial report describing what it wrote
before it stopped.
Grok jobs resolve to the OpenAI endpoint and a grok importedFrom source, so
their external ids stay in their own namespace. getImporter recognises a bare
prod-grok-backend.json and delegates to the same converter the zip path uses,
and the confirmation summary names the provider instead of showing the slug.
Adds a grok converter alongside chatgpt and claude. A Grok export ships a
single prod-grok-backend.json nested under a per-export uuid, so the layout
resolver finds it by name rather than by position, and detection keys on the
{ conversation, responses } envelope since Grok is the only format whose root
is an object.
The converter rebuilds the tree from parent_response_id, compares sender
case-insensitively (a real export mixes human, assistant and ASSISTANT),
reads MongoDB Extended JSON timestamps, keeps the raw model slug per message
and skips the aborted generations that carry no text, re-parenting their
children through the shared tree ordering.
Constants.NO_PARENT infers the enum type, so assigning a generated message id
to it failed type-check while every test stayed green. Also records the
Claude support report.
The summary interpolated the raw source value, so a Claude import read
"Detected claude export". Maps the reachable sources to localized provider
labels and falls back to the raw value for anything else.
A Claude .zip failed inspectExport because only the ChatGPT conversation
shape was recognised, and the legacy fallback is (correctly) restricted to
non-zip uploads, so the real export returned 400 and could not be imported.
inspectExport now detects the format from the shard's element shape and
runImport dispatches on it, so a Claude zip and a bare conversations.json
both flow through upload, inspect, summary, confirm, progress and report.
The job's endpoint, default model and importedFrom source now follow the
detected format, and importers.js delegates its Claude path to the same
converter instead of building a linear chain of its own.
Adds a claude/ converter beside the ChatGPT one: the message tree is built
from parent_message_uuid so branches survive, tool_use/tool_result pairs
become finished tool_call content parts, attachment extractions are carried
in as fenced blocks, and citation offsets are applied back to front into
highlight markers backed by a merged web_search attachment.
The cycle-breaking and parent-before-child ordering both formats need move
to a shared tree module, and the converted-message types and batch-sink
interfaces move to shared modules so neither format imports the other.
conversation_asset_file_names.json (and metadata.attachments[].name) map
each .dat entry to an original filename that can carry its nested export
location, e.g. <conv-id>/audio/<file>.wav. assets.ts built the storage
filename as `${fileId}-${originalName}` and handed it straight to
saveBuffer, so any separator in that name became a missing intermediate
directory and the write failed with ENOENT -- 274 audio/video assets were
silently dropped from a real 778 MB export while the job still reported
completed. The same unsanitized value is a path-traversal vector for a
crafted export using ../ segments.
Extract the leaf segment (tolerating both / and \) for the human-readable
display name, and run that leaf through the existing sanitizeFilename
helper before it becomes part of the storage path. The per-asset uuidv4()
prefix already guarantees two names that sanitize to the same leaf can't
overwrite each other.
Verified against the real export: assetsImported rose from 1,306 to 1,580
with zero storage errors, and the 274 previously-failing audio files now
land flat with no extra directory level.
openReadStream's default decompressing path silently stalls on Node
24.16.0 with yauzl 3.2.1 for any entry whose decompressed output needs
more than a single internal zlib chunk (~64 KB) — the stream never
emits data, end, or error, so archive.read() hangs forever. Every
fixture in the existing suite was small enough (and, incidentally,
stored rather than deflated) to never hit this path, so all 132 tests
passed while the feature was completely broken against a real export.
Read entries raw (decompress: false) and inflate with zlib.inflateRaw
ourselves instead. This also strengthens the per-entry cap: zlib now
enforces it during inflation via maxOutputLength instead of us
counting decompressed chunks after the fact, mapping its
ERR_BUFFER_TOO_LARGE onto the existing ZipBombError.
Verified against a real 778 MB export: 2,929 conversations and 1,666
assets inspected in ~1s instead of hanging indefinitely.
