📬 feat: Move Agent Queued Turns Server-Side (#15370)

* feat: move agent queued turns server-side

* test: update queued turn composition coverage

* fix: harden queued-turn lifecycle ownership

* fix: centralize queued-turn lifecycle ownership

* fix: close queued-turn lifecycle gaps

* fix: unify queued turn lifecycle transitions

* fix: fence queued-turn deletion admission

* fix: preserve queued-turn lifecycle ownership

* fix: close queued-turn reconciliation races

* fix: reconcile quarantined queued-turn admissions

* fix: annotate admission evidence default

* fix: deepen queued turn lifecycle admission

* fix: fence queued turn provider admission
This commit is contained in:
Danny Avila 2026-08-30 21:53:58 -04:00 • committed by GitHub
parent 4de0a9341b
commit c7f0a8f247
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
61 changed files with 10471 additions and 253 deletions

View file

@ -13,6 +13,7 @@
- **Subagent completion wakeup**: a durable internal `continue` trigger pre-registered before detached child execution so a process crash cannot lose the wakeup. Delivery defers until the child's terminal transcript is persisted, targets the initiating agent and exact parent response branch, carries task metadata rather than child output, waits for the parent generation to settle, and starts the parent turn that collects the result through the existing task store.
- **Agent continuation preparation**: the single source-dispatch seam that resolves a durable `continue` delivery immediately before admission. Bound Event Actor work selects its binding adapter; internal completion work selects an adapter by stable source identity. Preparation may resolve authoritative input and branch state or settle already-consumed work, but it does not own source result truth, delivery ordering, or generation execution.
- **Warm terminal steer continuation**: a queued steer accepted before a generation's terminal boundary may continue the same SDK `Run` without creating a replacement generation. After parallel Stop hooks fold, the serialized StopFinalize phase tells the job store whether another continuation is already planned or terminal progress is forbidden. The store atomically chooses among claiming the current protocol-v2 FIFO batch, keeping empty admission open for an already-planned segment, and sealing admission so every racing or later message becomes an ordinary follow-up. Claimed steer receipts remain the crash-recovery authority; protocol-v1 generations always seal because they cannot recover an ambiguous terminal claim. Tool-batch, preemption, and terminal boundaries share one durable apply-and-inject adapter, while the SDK owns the bounded Stop-continuation loop.
- **Agent queued turn**: a server-owned ordinary follow-up accepted while an Agent generation is active. Its Mongo row is the sole FIFO, payload, and lifecycle authority; the trigger delivery is only a replayable wakeup and Agent execution enrollment is only an execution adapter. The lifecycle reserves the deterministic delivery identity before publication, admits only after the captured branch has a clean durable predecessor outcome, and commits the source-owned generation receipt only after provider invocation has been enrolled. An accepted or deduplicated loopback response requires that exact receipt; a process death before it remains explicit admission-indeterminate evidence instead of silently consuming the text. Admission reconciliation is leased, backoff-scheduled work: exact generation evidence may repair legacy ambiguity, while current source-owned ambiguity never infers provider outcome from transient job state. Aborted or failed predecessors and exhausted admission attempts remain visible terminal rows. Transport-ambiguous enqueue outcomes stay non-resendable until exact request-identity reconciliation succeeds; an elapsed client observation window may hide the warning locally but never converts ambiguity into permission to submit again. Conversation deletion cancels the rows, retires their deliveries, and removes their payloads before deleting the conversation wave.
- **Subagent activity stream**: an observational, task-scoped live projection of bounded child progress for the currently open private panel. It may cross API replicas through Redis, never carries hidden reasoning text, and never controls or settles execution. The durable child thread remains canonical and its existing polling view is the fallback for missed or unavailable live events.
- **Agent event handling outcome**: the durable, generation-fenced result of a previously accepted event delivery. `started` proves generation admission; terminal states distinguish verified tool application, clean completion without action, failure, and cancellation. Transport success remains separate so an accepted event cannot masquerade as completed work.
- **Agent event expected action**: an optional source-declared tool name and bounded argument subset evaluated against host-observed completed run steps. It is evidence policy, not authorization and not a model-authored success claim.

View file

@ -76,6 +76,8 @@ const mockExemptFromConcurrencyLimiter = jest.fn();
const mockRecordScheduleOutcome = jest.fn();
const mockIsScheduleLive = jest.fn();
const mockDeleteAgentCheckpoint = jest.fn();
const mockSettleAgentQueuedTurnExecutionAdmission = jest.fn();
const mockVerifyAgentQueuedTurnExecutionAdmission = jest.fn();
const mockExecuteAgentEventActor = jest.fn();
const mockResumeAgentEventActor = jest.fn();
const mockCreateAgentEventActorTurn = jest.fn((input, dependencies) => {
@ -379,6 +381,13 @@ jest.mock('~/server/services/Schedules', () => ({
isScheduleLive: (...args) => mockIsScheduleLive(...args),
}));
jest.mock('~/server/services/Agents/triggers', () => ({
settleAgentQueuedTurnExecutionAdmission: (...args) =>
mockSettleAgentQueuedTurnExecutionAdmission(...args),
verifyAgentQueuedTurnExecutionAdmission: (...args) =>
mockVerifyAgentQueuedTurnExecutionAdmission(...args),
}));
const AgentController = require('../request');
const { ErrorTypes } = require('librechat-data-provider');
const { disposeClient: mockDisposeClient } = require('~/server/cleanup');
@ -485,6 +494,8 @@ describe('ResumableAgentController resume metadata', () => {
mockSaveMessage.mockResolvedValue({});
mockSaveConvo.mockResolvedValue({});
mockDeleteAgentCheckpoint.mockResolvedValue(undefined);
mockSettleAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
mockVerifyAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
mockGetAgentEventActorSnapshot.mockResolvedValue({ state: null, reconciliations: [] });
mockCommitAgentEventActorState.mockResolvedValue({ status: 'stale' });
mockBeginAgentEventActorLegacyTurn.mockResolvedValue(true);
@ -1135,6 +1146,131 @@ describe('ResumableAgentController resume metadata', () => {
);
});
it('keeps a queued-turn source nonterminal when provider initialization fails', async () => {
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
const admissionSource = {
source: 'agent-queued-turn',
sourceId: 'queued-turn-1',
claimId: 'queued-delivery-1',
claimBy: 'queued-worker-1',
};
const req = {
_isAgentTrigger: true,
user: { id: 'user-123', tenantId: 'tenant-1' },
body: {
text: 'Run the queued turn.',
messageId: 'queued-user-message',
parentMessageId: 'persisted-response_',
conversationId: 'conversation-123',
clientRequestId: 'queued-delivery-1',
agentContinuationAdmission: admissionSource,
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
},
config: {},
};
const res = createResumableResponse();
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before provider'));
await AgentController(req, res, jest.fn(), initializeClient, null);
expect(initializeClient).toHaveBeenCalledTimes(1);
expect(mockSettleAgentQueuedTurnExecutionAdmission).not.toHaveBeenCalled();
expect(res.status).toHaveBeenCalledWith(500);
});
it('commits a queued-turn receipt after provider invocation and before accepting HTTP', async () => {
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
const admissionSource = {
source: 'agent-queued-turn',
sourceId: 'queued-turn-1',
claimId: 'queued-delivery-1',
claimBy: 'queued-worker-1',
};
const req = {
_isAgentTrigger: true,
user: { id: 'user-123', tenantId: 'tenant-1' },
body: {
text: 'Run the queued turn.',
messageId: 'queued-user-message',
parentMessageId: 'persisted-response_',
conversationId: 'conversation-123',
clientRequestId: 'queued-delivery-1',
agentContinuationAdmission: admissionSource,
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
},
config: {},
};
const sendMessage = jest.fn(() => new Promise(() => {}));
const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
const res = createResumableResponse();
await AgentController(req, res, jest.fn(), initializeClient, null);
await nextTick();
expect(mockSettleAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
userId: 'user-123',
tenantId: 'tenant-1',
conversationId: 'conversation-123',
clientRequestId: 'queued-delivery-1',
generationId: 'conversation-123',
generationCreatedAt: 1000,
});
expect(initializeClient.mock.invocationCallOrder[0]).toBeLessThan(
sendMessage.mock.invocationCallOrder[0],
);
expect(sendMessage.mock.invocationCallOrder[0]).toBeLessThan(
mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0],
);
expect(mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0]).toBeLessThan(
res.json.mock.invocationCallOrder[0],
);
});
it('fails closed after provider invocation when the queued-turn receipt is unavailable', async () => {
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
mockSettleAgentQueuedTurnExecutionAdmission.mockRejectedValue(new Error('mongo unavailable'));
const req = {
_isAgentTrigger: true,
user: { id: 'user-123' },
body: {
text: 'Run the queued turn.',
messageId: 'queued-user-message',
parentMessageId: 'persisted-response_',
conversationId: 'conversation-123',
clientRequestId: 'queued-delivery-1',
agentContinuationAdmission: {
source: 'agent-queued-turn',
sourceId: 'queued-turn-1',
claimId: 'queued-delivery-1',
claimBy: 'queued-worker-1',
},
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
},
config: {},
};
const sendMessage = jest.fn(() => new Promise(() => {}));
const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
const res = createResumableResponse();
await AgentController(req, res, jest.fn(), initializeClient, null);
await nextTick();
expect(mockGenerationJobManager.beginProviderExecution).toHaveBeenCalledTimes(1);
expect(sendMessage).toHaveBeenCalledTimes(1);
expect(res.status).toHaveBeenCalledWith(500);
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
'conversation-123',
'mongo unavailable',
1000,
expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
);
expect(mockGenerationJobManager.markProviderExecutionDrained).toHaveBeenCalledWith(
'conversation-123',
1000,
'provider-segment-1',
);
});
it('prefetches conversation state before admission and joins it with job metadata', async () => {
let resolveConversation;
let signalMetadataStarted;
@ -2042,6 +2178,61 @@ describe('ResumableAgentController resume metadata', () => {
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
});
it('fails closed when a deduplicated queued generation lacks its source receipt', async () => {
const reacquired = wonGenerationClaim({
streamId: 'conversation-123',
conversationId: 'conversation-123',
claimToken: 'reacquired-token',
});
mockGenerationJobManager.claimGeneration.mockResolvedValue(reacquired);
mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue({
...reacquired.existing,
startedAt: 42,
});
mockVerifyAgentQueuedTurnExecutionAdmission.mockRejectedValue(
new Error('source receipt missing'),
);
const admissionSource = {
source: 'agent-queued-turn',
sourceId: 'queued-turn-1',
claimId: 'queued-delivery-1',
claimBy: 'queued-worker-1',
};
const req = {
_isAgentTrigger: true,
user: { id: 'user-123', tenantId: 'tenant-1' },
body: {
text: 'Retry a queued generation.',
messageId: 'user-msg',
parentMessageId: 'assistant-1',
clientRequestId: 'queued-delivery-1',
conversationId: 'conversation-123',
agentContinuationAdmission: admissionSource,
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
},
config: {},
};
const res = createResumableResponse();
await AgentController(req, res, jest.fn(), jest.fn(), null);
expect(mockVerifyAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
userId: 'user-123',
tenantId: 'tenant-1',
conversationId: 'conversation-123',
clientRequestId: 'queued-delivery-1',
generationId: 'conversation-123',
generationCreatedAt: 42,
});
expect(res.status).toHaveBeenCalledWith(503);
expect(res.json).toHaveBeenCalledWith(
expect.objectContaining({
code: 'SERVER_NOT_READY',
}),
);
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
});
it('caps a reacquired v2 lease to the immutable v1 live-job protocol', async () => {
const reacquired = wonGenerationClaim({
streamId: 'conversation-123',
@ -2090,7 +2281,7 @@ describe('ResumableAgentController resume metadata', () => {
conversationId: 'orig-stream',
claimedAt: Date.now() - 60000,
claimToken: 'existing-token',
startedAt: Date.now() - 59000,
startedAt: 1000,
},
});
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
@ -2112,6 +2303,7 @@ describe('ResumableAgentController resume metadata', () => {
expect(res.json).toHaveBeenCalledWith({
streamId: 'orig-stream',
conversationId: 'orig-stream',
generationCreatedAt: 1000,
status: 'resumed',
generationProtocolVersion: 1,
});
@ -2127,7 +2319,7 @@ describe('ResumableAgentController resume metadata', () => {
conversationId: 'orig-stream',
claimedAt: Date.now() - 60_000,
claimToken: 'existing-token',
startedAt: Date.now() - 59_000,
startedAt: 2000,
generationProtocolVersion: 2,
},
});
@ -2150,6 +2342,7 @@ describe('ResumableAgentController resume metadata', () => {
expect(res.json).toHaveBeenCalledWith({
conversationId: 'orig-stream',
generationCreatedAt: 2000,
status: 'settled',
generationProtocolVersion: 2,
});

View file

@ -0,0 +1,80 @@
const {
handleAgentQueuedTurnEnqueue,
handleAgentQueuedTurnList,
handleAgentQueuedTurnCancel,
} = require('@librechat/api');
const { logger } = require('@librechat/data-schemas');
const db = require('~/models');
const { createAgentAccessCheck } = require('./steer');
const dependencies = (req) => {
/** Keep trigger-service composition lazy so importing the route for unrelated
* middleware probes does not initialize or require the worker graph. */
const {
scheduleAgentQueuedTurn,
cancelAgentQueuedTurn,
} = require('~/server/services/Agents/triggers');
return {
methods: db,
lifecycle: {
schedule: scheduleAgentQueuedTurn,
cancel: cancelAgentQueuedTurn,
},
getFiles: db.getFiles,
updateFilesUsage: db.updateFilesUsage,
checkAgentAccess: createAgentAccessCheck(req),
isPrincipalActive: db.isAgentTriggerPrincipalActive,
};
};
const send = (res, result) => res.status(result.status).json(result.body);
const AgentQueuedTurnEnqueueController = async (req, res) => {
try {
return send(
res,
await handleAgentQueuedTurnEnqueue(req.user ?? {}, req.body ?? {}, dependencies(req)),
);
} catch (error) {
logger.error('[AgentQueuedTurns] Failed to enqueue turn', error);
return res.status(500).json({ code: 'QUEUED_TURN_FAILED' });
}
};
const AgentQueuedTurnListController = async (req, res) => {
try {
return send(
res,
await handleAgentQueuedTurnList(
req.user ?? {},
req.query?.conversationId,
dependencies(req),
req.query?.clientRequestIds,
),
);
} catch (error) {
logger.error('[AgentQueuedTurns] Failed to list turns', error);
return res.status(500).json({ code: 'QUEUED_TURN_LIST_FAILED' });
}
};
const AgentQueuedTurnCancelController = async (req, res) => {
try {
return send(
res,
await handleAgentQueuedTurnCancel(
req.user ?? {},
req.params?.queuedTurnId,
dependencies(req),
),
);
} catch (error) {
logger.error('[AgentQueuedTurns] Failed to cancel turn', error);
return res.status(500).json({ code: 'QUEUED_TURN_CANCEL_FAILED' });
}
};
module.exports = {
AgentQueuedTurnEnqueueController,
AgentQueuedTurnListController,
AgentQueuedTurnCancelController,
};

View file

@ -523,20 +523,30 @@ function sendSettledGeneration(
conversationId,
startupTelemetry,
generationProtocolVersion,
generationCreatedAt,
) {
startupTelemetry?.end('deduplicated');
if (generationProtocolVersion < GENERATION_PROTOCOL_V2) {
return sendGenerationJson(
res,
200,
{ streamId, conversationId, status: 'resumed' },
{
streamId,
conversationId,
...(generationCreatedAt != null && { generationCreatedAt }),
status: 'resumed',
},
generationProtocolVersion,
);
}
return sendGenerationJson(
res,
200,
{ conversationId, status: 'settled' },
{
conversationId,
...(generationCreatedAt != null && { generationCreatedAt }),
status: 'settled',
},
generationProtocolVersion,
);
}
@ -753,6 +763,55 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
req._isAgentTrigger === true &&
!isNewConvo &&
(parentMessageId !== Constants.NO_PARENT || isBoundEventContinuation);
const queuedTurnAdmissionSource = isTriggerContinuation
? req.body?.agentContinuationAdmission
: undefined;
const hasQueuedTurnAdmissionSource = queuedTurnAdmissionSource != null;
const verifyQueuedTurnAdmission = async (generationId, generationCreatedAt) => {
if (!hasQueuedTurnAdmissionSource) {
return true;
}
if (
typeof clientRequestId !== 'string' ||
!Number.isSafeInteger(generationCreatedAt) ||
generationCreatedAt < 0
) {
return false;
}
try {
const {
verifyAgentQueuedTurnExecutionAdmission,
} = require('~/server/services/Agents/triggers');
const confirmed = await verifyAgentQueuedTurnExecutionAdmission(queuedTurnAdmissionSource, {
userId,
...(tenantId != null && { tenantId }),
conversationId,
clientRequestId,
generationId,
generationCreatedAt,
});
return confirmed === true;
} catch (error) {
logger.warn(
'[ResumableAgentController] Deduplicated queued-turn admission is not confirmed',
error,
);
return false;
}
};
const rejectUnconfirmedQueuedTurnAdmission = () => {
res.set('Retry-After', '1');
startupTelemetry?.end('deduplicated');
return sendGenerationJson(
res,
503,
{
code: 'SERVER_NOT_READY',
error: 'Queued turn execution is still being confirmed. Please retry shortly.',
},
generationProtocolVersion,
);
};
if (
await isUnpersistedPreliminaryParent({
@ -942,6 +1001,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
? GENERATION_PROTOCOL_V2
: 1,
);
if (
!(await verifyQueuedTurnAdmission(
existingLiveGeneration.streamId,
existingLiveGeneration.startedAt,
))
) {
return rejectUnconfirmedQueuedTurnAdmission();
}
startupTelemetry?.end('deduplicated');
return sendGenerationJson(
res,
@ -1006,12 +1073,16 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
// generation (usually fast completion + cleanup), never an abandoned
// pre-create lease that may be taken over and billed again. There is no
// attachable stream; the settled response refetches persisted history.
if (!(await verifyQueuedTurnAdmission(existingStreamId, claim.existing.startedAt))) {
return rejectUnconfirmedQueuedTurnAdmission();
}
return sendSettledGeneration(
res,
existingStreamId,
claim.existing.conversationId,
startupTelemetry,
generationProtocolVersion,
claim.existing.startedAt,
);
}
if (!liveJob && isLegacyTokenlessClaim && claimAgeMs >= IDEMPOTENCY_STARTUP_GRACE_MS) {
@ -1020,6 +1091,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
* attach/refetch path: this covers fast completion without starting a
* second billed generation, while an abandoned pre-create claim ages
* out under the old server's bounded TTL. */
if (!(await verifyQueuedTurnAdmission(existingStreamId, claim.existing.startedAt))) {
return rejectUnconfirmedQueuedTurnAdmission();
}
return sendSettledGeneration(
res,
existingStreamId,
@ -1108,6 +1182,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
(startedAt != null && liveJob.createdAt !== startedAt) ||
(liveClientRequestId != null && liveClientRequestId !== clientRequestId);
if (replacedGeneration) {
if (!(await verifyQueuedTurnAdmission(existingStreamId, startedAt))) {
return rejectUnconfirmedQueuedTurnAdmission();
}
// streamId === conversationId, so a later turn reuses the same route.
// Never pair this stale POST's optimistic submission with that newer
// job's SSE snapshot. If the replacement is still active, distinguish
@ -1141,6 +1218,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
claim.existing.conversationId,
startupTelemetry,
generationProtocolVersion,
claim.existing.startedAt,
);
}
if (liveClientRequestId == null && !isLegacyTokenlessClaim) {
@ -1164,6 +1242,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
clientRequestId,
streamId: existingStreamId,
});
if (!(await verifyQueuedTurnAdmission(existingStreamId, liveJob.createdAt))) {
return rejectUnconfirmedQueuedTurnAdmission();
}
startupTelemetry?.end('deduplicated');
return sendGenerationJson(
res,
@ -1518,7 +1599,6 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
status: 409,
});
}
acceptAgentStartupTelemetry(req, streamId);
startupTelemetry?.mark('metadata_persisted');
req._resumableStreamId = streamId;
@ -1543,14 +1623,29 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
recoveredSteerCommitted = true;
};
// Send JSON response IMMEDIATELY so client can connect to SSE stream
// This is critical: tool loading (MCP OAuth) may emit events that the client needs to receive
sendGenerationJson(
res,
200,
{ streamId, conversationId, generationCreatedAt: jobCreatedAt, status: 'started' },
generationProtocolVersion,
);
// Ordinary clients receive the stream id immediately so they can attach
// before tool loading emits events. Source-owned loopback work delays only
// until its provider invocation and Mongo receipt exist.
let generationStartResponseSent = false;
const sendGenerationStarted = () => {
if (generationStartResponseSent || res.headersSent) {
return;
}
generationStartResponseSent = true;
sendGenerationJson(
res,
200,
{ streamId, conversationId, generationCreatedAt: jobCreatedAt, status: 'started' },
generationProtocolVersion,
);
};
/** Ordinary clients need the stream id before tool discovery. A queued
* source instead keeps its local loopback response open until the provider
* invocation exists, so an accepted HTTP result can never retire text that
* died between job creation and provider startup. */
if (!hasQueuedTurnAdmissionSource) {
sendGenerationStarted();
}
await attachConversationCreatedAt(req, conversationId, conversationAnchorPromise).then(() =>
startupTelemetry?.mark('conversation_resolved'),
@ -2298,6 +2393,21 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
})
: client.sendMessage(text, messageOptions);
if (hasQueuedTurnAdmissionSource) {
const {
settleAgentQueuedTurnExecutionAdmission,
} = require('~/server/services/Agents/triggers');
await settleAgentQueuedTurnExecutionAdmission(queuedTurnAdmissionSource, {
userId,
...(tenantId != null && { tenantId }),
conversationId,
clientRequestId,
generationId: streamId,
generationCreatedAt: jobCreatedAt,
});
sendGenerationStarted();
}
if (titleEligible && titleTiming === 'immediate') {
immediateTitlePromise = addTitle(req, {
text: text || getAttachmentTitleText(req.body.files),
@ -2780,6 +2890,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
job.abortController.signal.removeEventListener('abort', abortTitleOnJobAbort);
acceptsTitleEvents = false;
resolveConvoReady();
if (!res.headersSent) {
sendGenerationJson(
res,
500,
{ error: error.message || 'Failed to start generation' },
generationProtocolVersion,
);
}
try {
await recordEventActorPersistenceFailure(error);
} catch (reconciliationError) {
@ -2899,6 +3017,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
`[ResumableAgentController] Unhandled error in background generation: ${err.message}`,
);
startupTelemetry?.end('error', err);
if (!res.headersSent) {
sendGenerationJson(
res,
500,
{ error: err.message || 'Failed to start generation' },
generationProtocolVersion,
);
}
let errorFinalized = false;
if (!pausePersistenceFailed) {
errorFinalized =

View file

@ -203,3 +203,4 @@ module.exports = SteerController;
module.exports.SteerDeliveryController = SteerDeliveryController;
module.exports.SteerCancelController = SteerCancelController;
module.exports.SteerArmController = SteerArmController;
module.exports.createAgentAccessCheck = createAgentAccessCheck;

View file

@ -67,6 +67,11 @@ jest.mock('~/server/controllers/agents/steer', () => {
controller.SteerArmController = (_req, _res, next) => next();
return controller;
});
jest.mock('~/server/controllers/agents/queuedTurns', () => ({
AgentQueuedTurnEnqueueController: (_req, res) => res.status(202).json({ queued: true }),
AgentQueuedTurnListController: (_req, res) => res.status(200).json({ queuedTurns: [] }),
AgentQueuedTurnCancelController: (_req, res) => res.status(200).json({ cancelled: true }),
}));
jest.mock('~/models', () => ({}));
jest.mock('~/server/services/Schedules', () => ({}));
@ -150,4 +155,29 @@ describe('start-generation idempotency before message limiters', () => {
expect(mockIpLimiter).not.toHaveBeenCalled();
expect(mockUserLimiter).not.toHaveBeenCalled();
});
it('keeps read-only queued-turn polling outside message admission limiters', async () => {
const responses = await Promise.all(
Array.from({ length: 3 }, () =>
request(app).get('/agents/chat/queued-turns').query({ conversationId: 'conversation-1' }),
),
);
expect(responses.map((response) => response.status)).toEqual([200, 200, 200]);
expect(responses[0].body).toEqual({ queuedTurns: [] });
expect(mockIpLimiter).not.toHaveBeenCalled();
expect(mockUserLimiter).not.toHaveBeenCalled();
});
it.each([
['enqueue', () => request(app).post('/agents/chat/queued-turns').send({ text: 'next' })],
['cancel', () => request(app).delete('/agents/chat/queued-turns/queued-turn-1')],
])('keeps queued-turn %s mutations behind message admission limiters', async (_label, send) => {
const response = await send();
expect(response.status).toBe(429);
expect(response.body).toEqual({ limited: 'ip' });
expect(mockIpLimiter).toHaveBeenCalledTimes(1);
expect(mockUserLimiter).not.toHaveBeenCalled();
});
});

View file

@ -35,6 +35,11 @@ const {
messageUserLimiter,
} = require('~/server/middleware');
const SteerController = require('~/server/controllers/agents/steer');
const {
AgentQueuedTurnEnqueueController,
AgentQueuedTurnListController,
AgentQueuedTurnCancelController,
} = require('~/server/controllers/agents/queuedTurns');
const {
GENERATION_PROTOCOL_HEADER,
GENERATION_PROTOCOL_V2,
@ -228,7 +233,12 @@ router.get('/chat/stream/:streamId', async (req, res) => {
logger.warn(`[AgentStream] Refusing stream with invalid generation identity: ${streamId}`);
return sendGenerationJson(res, 403, { error: 'Unauthorized' }, requestProtocolVersion);
}
const streamTelemetry = createSseStreamTelemetry({ req, res, streamId, isResume });
const streamTelemetry = createSseStreamTelemetry({
req,
res,
streamId,
isResume,
});
res.setHeader('Content-Encoding', 'identity');
res.setHeader('Content-Type', 'text/event-stream');
@ -383,7 +393,9 @@ router.get('/chat/stream/:streamId', async (req, res) => {
final: true,
reconcile: true,
reconcileReason: generationReplaced ? 'generation_replaced' : 'terminal_payload_missing',
...(expectedGenerationTerminal && { terminalStatus: currentJob.status }),
...(expectedGenerationTerminal && {
terminalStatus: currentJob.status,
}),
generationCreatedAt: authorizedGenerationCreatedAt,
conversation: {
conversationId: currentJob?.conversationId ?? job.conversationId ?? streamId,
@ -975,7 +987,9 @@ router.post('/chat/abort', configMiddleware, async (req, res, next) => {
// Steers that never reached an injection boundary — restored client-side
// as queued chips so the user's words aren't dropped with the abort.
...(!abortResult.persistenceFailed &&
abortResult.pendingSteers?.length > 0 && { pendingSteers: abortResult.pendingSteers }),
abortResult.pendingSteers?.length > 0 && {
pendingSteers: abortResult.pendingSteers,
}),
});
}
@ -1076,6 +1090,28 @@ router.post(
SteerController.SteerArmController,
);
router.post(
'/chat/queued-turns',
configMiddleware,
...steerLimiters,
createMessageFilterPii({
getConfig: (req) => req.config?.messageFilter?.pii,
getFilters: (req) => req.config?.filters,
getFiles,
}),
moderateText,
AgentQueuedTurnEnqueueController,
);
/** Synchronizing durable queue state is read-only and polled while work is
* pending. It must not consume the model-submission admission budget. */
router.get('/chat/queued-turns', configMiddleware, AgentQueuedTurnListController);
router.delete(
'/chat/queued-turns/:queuedTurnId',
configMiddleware,
...steerLimiters,
AgentQueuedTurnCancelController,
);
router.use('/', v1);
const chatRouter = express.Router();

View file

@ -6,10 +6,20 @@ const {
SUBAGENT_COMPLETION_SOURCE,
createBackgroundToolCompletionWakeupResolver,
BACKGROUND_TOOL_COMPLETION_SOURCE,
createAgentQueuedTurnLifecycle,
AGENT_QUEUED_TURN_SOURCE,
GenerationJobManager,
} = require('@librechat/api');
const methods = require('~/models');
const getGenerationAdmissionEvidence = (userId, clientRequestId, streamId, conversationId) =>
GenerationJobManager.getGenerationAdmissionEvidence(
userId,
clientRequestId,
streamId,
conversationId,
);
const subagentCompletionAdapter = createSubagentCompletionWakeupResolver({
methods,
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
@ -22,23 +32,48 @@ const eventActorAdapter = createAgentEventContinueResolver({
methods,
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
});
let service;
const queuedTurnLifecycle = createAgentQueuedTurnLifecycle({
methods,
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
getGenerationAdmissionEvidence,
enqueue: (...args) => service.enqueue(...args),
retireDelivery: (...args) => service.retire(...args),
getDelivery: (...args) => service.getDelivery(...args),
});
const service = createAgentTriggerService({
service = createAgentTriggerService({
methods,
isPrincipalActive: methods.isAgentTriggerPrincipalActive,
supportsDetachedActionCompletion: () => GenerationJobManager.supportsDetachedAgentEventActions,
settleSourceBeforeDeadLetter: queuedTurnLifecycle.settleBeforeDeadLetter,
prepareContinue: createAgentContinuationResolver({
eventActor: eventActorAdapter,
internalSources: new Map([
[SUBAGENT_COMPLETION_SOURCE, subagentCompletionAdapter],
[BACKGROUND_TOOL_COMPLETION_SOURCE, backgroundToolCompletionAdapter],
[AGENT_QUEUED_TURN_SOURCE, queuedTurnLifecycle.prepareContinue],
]),
}),
});
const initializeAgentTriggerService = async (options) => {
await service.initialize(options);
await queuedTurnLifecycle.initialize();
};
const stopAgentTriggerService = async () => {
await queuedTurnLifecycle.stop();
await service.stop();
};
const purgeAgentTriggerDeliveriesForUser = async (userId) => {
await service.purgeUser(userId);
};
module.exports = {
initializeAgentTriggerService: service.initialize,
stopAgentTriggerService: service.stop,
initializeAgentTriggerService,
stopAgentTriggerService,
dispatchAgentTrigger: service.dispatch,
enqueueAgentTrigger: service.enqueue,
getAgentTriggerDelivery: service.getDelivery,
@ -50,5 +85,9 @@ module.exports = {
drainAgentTriggerDeliveriesForUser: service.drainUser,
prepareAgentTriggerUserPurge: service.prepareUserPurge,
cancelAgentTriggerUserPurge: service.cancelUserPurge,
purgeAgentTriggerDeliveriesForUser: service.purgeUser,
purgeAgentTriggerDeliveriesForUser,
scheduleAgentQueuedTurn: queuedTurnLifecycle.schedule,
cancelAgentQueuedTurn: queuedTurnLifecycle.cancel,
settleAgentQueuedTurnExecutionAdmission: queuedTurnLifecycle.recordExecutionAdmission,
verifyAgentQueuedTurnExecutionAdmission: queuedTurnLifecycle.verifyExecutionAdmission,
};

View file

@ -2,6 +2,17 @@ const mockCreateAgentTriggerService = jest.fn();
const mockGenerationJobManager = {
supportsDetachedAgentEventActions: true,
getJob: jest.fn(),
getGenerationAdmissionEvidence: jest.fn(),
};
const mockQueuedTurnLifecycle = {
prepareContinue: jest.fn(),
settleBeforeDeadLetter: jest.fn(),
recordExecutionAdmission: jest.fn(),
verifyExecutionAdmission: jest.fn(),
initialize: jest.fn(),
stop: jest.fn(),
schedule: jest.fn(),
cancel: jest.fn(),
};
jest.mock('@librechat/api', () => ({
@ -10,8 +21,10 @@ jest.mock('@librechat/api', () => ({
createAgentEventContinueResolver: jest.fn(() => jest.fn()),
createBackgroundToolCompletionWakeupResolver: jest.fn(() => jest.fn()),
createSubagentCompletionWakeupResolver: jest.fn(() => jest.fn()),
createAgentQueuedTurnLifecycle: jest.fn(() => mockQueuedTurnLifecycle),
BACKGROUND_TOOL_COMPLETION_SOURCE: 'background-tool-completion',
SUBAGENT_COMPLETION_SOURCE: 'subagent-completion',
AGENT_QUEUED_TURN_SOURCE: 'agent-queued-turn',
GenerationJobManager: mockGenerationJobManager,
}));

View file

@ -2,7 +2,18 @@ import { memo, useMemo, useRef, useState, useCallback } from 'react';
import { useAtomValue } from 'jotai';
import { useRecoilValue } from 'recoil';
import { useToastContext } from '@librechat/client';
import { X, Zap, Send, Clock, Pencil, Trash2, Paperclip, RotateCcw, TextQuote } from 'lucide-react';
import {
X,
Zap,
Send,
Clock,
Pencil,
Trash2,
Paperclip,
RotateCcw,
TextQuote,
TriangleAlert,
} from 'lucide-react';
import type { TMessage } from 'librechat-data-provider';
import type { SteeringControls, QueuedMessageContext } from '~/hooks/Chat/useSteering';
import type { PendingSteer, QueuedMessage } from '~/store/families';
@ -91,6 +102,14 @@ function QueuedRow({
const fileCount = message.files?.length ?? 0;
const quoteCount = message.quotes?.length ?? 0;
const isRecovered = message.recoverySteerId != null;
const isRejected = message.server?.status === 'rejected';
const isUnconfirmed =
message.server?.status === 'uncertain' && message.server.reconciliationExpired === true;
const requiresDiscard = isRecovered || message.server?.id != null;
const serverActionable =
message.server == null ||
message.server.status === 'rejected' ||
(message.server.id != null && message.server.status === 'queued');
const actionPendingRef = useRef(false);
const [actionPending, setActionPending] = useState(false);
/** A recovered item has a replayable parked source. Edit/remove must first
@ -128,7 +147,8 @@ function QueuedRow({
// generation. Re-steering it would leave or duplicate the parked source;
// Edit/remove are safe because `afterDiscard` tombstones that source first.
const canSteerNow = steering.duringRunActive && steering.canSteer && !isRecovered;
const showPrimary = canSteerNow || (!steering.duringRunActive && steering.canSendQueuedNow);
const showPrimary =
serverActionable && (canSteerNow || (!steering.duringRunActive && steering.canSendQueuedNow));
/** `canSteer` is defined as false while paused on approval, but the
* escalation control must stay visible-and-disabled there — hiding it
* during the pause is exactly the discoverability gap this button fixes. */
@ -140,10 +160,13 @@ function QueuedRow({
key: 'edit',
label: localize('com_ui_edit_message'),
icon: <Pencil className="h-4 w-4" aria-hidden="true" />,
disabled: actionPending,
disabled: actionPending || !serverActionable,
onClick: () => {
const context = { quotes: message.quotes, manualSkills: message.manualSkills };
if (!isRecovered) {
const context = {
quotes: message.quotes,
manualSkills: message.manualSkills,
};
if (!requiresDiscard) {
steering.removeQueued(message.id);
onEditToComposer(message.text, message.files, {
quotes: message.quotes,
@ -159,7 +182,10 @@ function QueuedRow({
conversationId,
);
if (!restored) {
showToast({ message: localize('com_ui_steer_edit_queued'), status: 'info' });
showToast({
message: localize('com_ui_steer_edit_queued'),
status: 'info',
});
return false;
}
steering.removeQueued(message.id);
@ -172,7 +198,11 @@ function QueuedRow({
return (
<div role="listitem" className={ROW_CLASS} data-testid="queued-message-row">
<Clock className="h-4 w-4 shrink-0 text-cyan-500" aria-hidden="true" />
{isRejected || isUnconfirmed ? (
<TriangleAlert className="h-4 w-4 shrink-0 text-text-warning" aria-hidden="true" />
) : (
<Clock className="h-4 w-4 shrink-0 text-cyan-500" aria-hidden="true" />
)}
<span className="min-w-0 flex-1 truncate" title={message.text}>
{message.text}
</span>
@ -182,13 +212,22 @@ function QueuedRow({
/>
<AttachmentCount
count={fileCount}
label={localize('com_ui_queued_attachment_count', { 0: String(fileCount) })}
label={localize('com_ui_queued_attachment_count', {
0: String(fileCount),
})}
/>
{(isRejected || isUnconfirmed) && (
<span className="shrink-0 text-xs text-text-warning">
{localize(
isUnconfirmed ? 'com_ui_steer_delivery_unconfirmed' : 'com_ui_queued_turn_failed',
)}
</span>
)}
{showPrimary && (
<button
type="button"
className={PRIMARY_BTN_CLASS}
disabled={actionPending}
disabled={actionPending || !serverActionable}
onClick={() => steering.sendQueuedNow(message)}
>
{canSteerNow ? (
@ -208,15 +247,23 @@ function QueuedRow({
<EscalateNowButton
surface="queued"
messageText={message.text}
disabled={steering.pausedOnApproval || interruptPending || actionPending}
disabled={
steering.pausedOnApproval || interruptPending || actionPending || !serverActionable
}
onClick={() => steering.sendQueuedNow(message, { preempt: true })}
/>
)}
<button
type="button"
aria-label={localize('com_ui_remove_queued')}
disabled={actionPending}
aria-label={localize(
isUnconfirmed ? 'com_ui_dismiss_unconfirmed_delivery' : 'com_ui_remove_queued',
)}
disabled={actionPending || (!serverActionable && !isUnconfirmed)}
onClick={() => {
if (isUnconfirmed) {
steering.removeQueued(message.id);
return;
}
const remove = () => {
/* Same safety net as the in-flight cancel: once removal is safely
* settled, return the words to the composer when it is free (the
@ -230,7 +277,7 @@ function QueuedRow({
steering.removeQueued(message.id);
return true;
};
if (!isRecovered) {
if (!requiresDiscard) {
remove();
return;
}
@ -310,7 +357,9 @@ function FailedSteerRow({
</span>
<QuoteCount
count={steer.quotes?.length ?? 0}
label={localize('com_ui_queued_quote_count', { 0: String(steer.quotes?.length ?? 0) })}
label={localize('com_ui_queued_quote_count', {
0: String(steer.quotes?.length ?? 0),
})}
/>
<span className="shrink-0 text-xs text-red-500">
{localize(

View file

@ -163,6 +163,48 @@ describe('PendingSteerChips — queued primary availability', () => {
});
});
describe('PendingSteerChips — terminal server state', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('shows a durable rejection as failed work', () => {
renderChips([
{
id: 'rejected-turn',
text: 'could not be admitted',
createdAt: 1,
server: { status: 'rejected', errorCode: 'ADMISSION_FAILED' },
},
]);
expect(screen.getByText('com_ui_queued_turn_failed')).toBeInTheDocument();
});
it('dismisses an expired ambiguous receipt without restoring or resending it', () => {
renderChips([
{
id: 'uncertain-turn',
text: 'delivery outcome unknown',
createdAt: 1,
server: {
status: 'uncertain',
uncertainSince: 1,
reconciliationExpired: true,
},
},
]);
expect(screen.getByText('com_ui_steer_delivery_unconfirmed')).toBeInTheDocument();
fireEvent.click(screen.getByLabelText('com_ui_dismiss_unconfirmed_delivery'));
expect(mockRemoveQueued).toHaveBeenCalledWith('uncertain-turn');
expect(mockDiscardQueued).not.toHaveBeenCalled();
expect(mockRestoreToComposer).not.toHaveBeenCalled();
expect(mockSendQueuedNow).not.toHaveBeenCalled();
});
});
describe('PendingSteerChips — queued trash', () => {
beforeEach(() => {
jest.clearAllMocks();
@ -240,6 +282,32 @@ describe('PendingSteerChips — queued trash', () => {
expect(mockRemoveQueued).toHaveBeenCalledWith('q-recovered');
});
it('discards a dead server row before restoring and removing it', async () => {
const dead = {
id: 'q-dead',
text: 'recover dead work',
createdAt: 1,
clientRequestId: 'client-dead',
server: {
id: 'server-dead',
status: 'rejected' as const,
errorCode: 'ADMISSION_FAILED',
},
};
renderChips([dead]);
fireEvent.click(screen.getByLabelText('com_ui_remove_queued'));
await waitFor(() => expect(mockDiscardQueued).toHaveBeenCalledWith(dead));
expect(mockRestoreToComposer).toHaveBeenCalledWith(
'recover dead work',
undefined,
{ quotes: undefined, manualSkills: undefined },
CONVO_ID,
);
expect(mockRemoveQueued).toHaveBeenCalledWith('q-dead');
});
it('offers Edit for a recovered row and leaves it untouched when discard is refused', async () => {
const user = userEvent.setup();
mockDiscardQueued.mockResolvedValueOnce(false);

View file

@ -0,0 +1,119 @@
const mockListAgentQueuedTurns = jest.fn();
const mockEnqueueAgentQueuedTurn = jest.fn();
const mockCancelAgentQueuedTurn = jest.fn();
jest.mock('librechat-data-provider', () => ({
QueryKeys: { agentQueuedTurns: 'agentQueuedTurns' },
MutationKeys: {
enqueueAgentQueuedTurn: 'enqueueAgentQueuedTurn',
cancelAgentQueuedTurn: 'cancelAgentQueuedTurn',
},
dataService: {
listAgentQueuedTurns: (...args: unknown[]) => mockListAgentQueuedTurns(...args),
enqueueAgentQueuedTurn: (...args: unknown[]) => mockEnqueueAgentQueuedTurn(...args),
cancelAgentQueuedTurn: (...args: unknown[]) => mockCancelAgentQueuedTurn(...args),
},
}));
import {
cancelAgentQueuedTurn,
enqueueAgentQueuedTurn,
fetchAgentQueuedTurns,
isDefiniteQueuedTurnRejection,
isDefiniteQueuedTurnsUnsupported,
} from '../queuedTurns';
describe('Agent queued-turn data adapter', () => {
beforeEach(() => {
jest.clearAllMocks();
});
it('posts the stable request identity and exact visible parent', async () => {
const input = {
conversationId: 'conversation/one',
clientRequestId: 'client-1',
parentMessageId: 'assistant-1',
text: 'next turn',
};
mockEnqueueAgentQueuedTurn.mockResolvedValueOnce({
receipt: { queuedTurnId: 'queued-1' },
});
await expect(enqueueAgentQueuedTurn(input)).resolves.toEqual({
queuedTurnId: 'queued-1',
});
expect(mockEnqueueAgentQueuedTurn).toHaveBeenCalledWith(input);
});
it('unwraps the shared list and cancel responses', async () => {
mockListAgentQueuedTurns.mockResolvedValueOnce({ queuedTurns: [] });
mockCancelAgentQueuedTurn.mockResolvedValueOnce({
receipt: { queuedTurnId: 'queued/two', status: 'cancelled' },
});
await expect(
fetchAgentQueuedTurns('conversation/one', ['request/one', 'request,two']),
).resolves.toEqual([]);
await expect(
cancelAgentQueuedTurn({
conversationId: 'conversation/one',
queuedTurnId: 'queued/two',
}),
).resolves.toEqual({ queuedTurnId: 'queued/two', status: 'cancelled' });
expect(mockListAgentQueuedTurns).toHaveBeenCalledWith('conversation/one', [
'request/one',
'request,two',
]);
expect(mockCancelAgentQueuedTurn).toHaveBeenCalledWith('queued/two');
});
it('distinguishes an old route from an application-level 404', () => {
expect(isDefiniteQueuedTurnsUnsupported({ response: { status: 404 } })).toBe(true);
expect(
isDefiniteQueuedTurnsUnsupported({
response: { status: 404, data: { code: 'CONVERSATION_NOT_FOUND' } },
}),
).toBe(false);
});
it('keeps the explicit priority fallback on the legacy local path', () => {
expect(
isDefiniteQueuedTurnsUnsupported({
response: { status: 501, data: { code: 'QUEUED_TURN_PRIORITY_UNSUPPORTED' } },
}),
).toBe(true);
});
it.each([400, 409, 500, undefined])(
'keeps status %s server-owned because it is not an unsupported route',
(status) => {
const error = status == null ? new Error('network') : { response: { status } };
expect(isDefiniteQueuedTurnsUnsupported(error)).toBe(false);
},
);
it.each([400, 403, 409, 413, 429])(
'classifies %s as a definite non-commit rejection',
(status) => {
expect(isDefiniteQueuedTurnRejection({ response: { status } })).toBe(true);
},
);
it('classifies a coded conversation 404 as a definite rejection', () => {
expect(
isDefiniteQueuedTurnRejection({
response: { status: 404, data: { code: 'CONVERSATION_NOT_FOUND' } },
}),
).toBe(true);
});
it.each([408, 425, 500, 503, undefined])(
'keeps %s outcome-ambiguous for request-id reconciliation',
(status) => {
const error = status == null ? new Error('network') : { response: { status } };
expect(isDefiniteQueuedTurnRejection(error)).toBe(false);
},
);
});

View file

@ -1,3 +1,4 @@
export * from './queries';
export * from './mutations';
export * from './protocol';
export * from './queuedTurns';

View file

@ -0,0 +1,126 @@
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
import { dataService, MutationKeys, QueryKeys } from 'librechat-data-provider';
import type {
TAgentQueuedTurnReceipt,
TEnqueueAgentQueuedTurnRequest,
} from 'librechat-data-provider';
export type AgentQueuedTurnReceipt = TAgentQueuedTurnReceipt;
export type EnqueueAgentQueuedTurnRequest = TEnqueueAgentQueuedTurnRequest;
export const agentQueuedTurnsQueryKey = (
conversationId: string,
clientRequestIds: readonly string[] = [],
) => [QueryKeys.agentQueuedTurns, conversationId, ...clientRequestIds] as const;
function queuedTurnErrorResponse(error: unknown): { status?: number; code?: string } {
const response = (
error as { response?: { status?: unknown; data?: { code?: unknown } } } | undefined
)?.response;
return {
...(typeof response?.status === 'number' && { status: response.status }),
...(typeof response?.data?.code === 'string' && { code: response.data.code }),
};
}
export function isDefiniteQueuedTurnsUnsupported(error: unknown): boolean {
const { status, code } = queuedTurnErrorResponse(error);
return (
(status === 404 && code == null) ||
(status === 501 &&
(code == null ||
code === 'QUEUED_TURNS_UNSUPPORTED' ||
code === 'QUEUED_TURN_PRIORITY_UNSUPPORTED'))
);
}
/** A bounded origin 4xx proves the queued row was not committed. Timeouts and
* early-data responses can be generated while the origin request continues,
* so those remain outcome-ambiguous and must reconcile by request identity. */
export function isDefiniteQueuedTurnRejection(error: unknown): boolean {
const { status } = queuedTurnErrorResponse(error);
return (
!isDefiniteQueuedTurnsUnsupported(error) &&
typeof status === 'number' &&
status >= 400 &&
status < 500 &&
status !== 408 &&
status !== 425
);
}
export async function fetchAgentQueuedTurns(
conversationId: string,
clientRequestIds?: string[],
): Promise<AgentQueuedTurnReceipt[]> {
const response = await dataService.listAgentQueuedTurns(conversationId, clientRequestIds);
return response.queuedTurns;
}
export async function enqueueAgentQueuedTurn(
input: EnqueueAgentQueuedTurnRequest,
): Promise<AgentQueuedTurnReceipt> {
const response = await dataService.enqueueAgentQueuedTurn(input);
return response.receipt;
}
export async function cancelAgentQueuedTurn(input: {
conversationId: string;
queuedTurnId: string;
}): Promise<AgentQueuedTurnReceipt> {
const response = await dataService.cancelAgentQueuedTurn(input.queuedTurnId);
return response.receipt;
}
export function useAgentQueuedTurns(
conversationId: string,
enabled: boolean,
clientRequestIds: string[] = [],
reconcileUntil?: number,
) {
const knownIds = [...new Set(clientRequestIds)].sort();
return useQuery({
queryKey: agentQueuedTurnsQueryKey(conversationId, knownIds),
queryFn: () => fetchAgentQueuedTurns(conversationId, knownIds),
enabled: enabled && conversationId.length > 0,
staleTime: 1_000,
refetchOnMount: true,
refetchOnWindowFocus: true,
refetchInterval: (receipts) => {
return (reconcileUntil != null && Date.now() < reconcileUntil) ||
(Array.isArray(receipts) &&
receipts.some((item) => item.status === 'queued' || item.status === 'claimed'))
? 2_000
: false;
},
retry: false,
});
}
export function useEnqueueAgentQueuedTurnMutation() {
const queryClient = useQueryClient();
return useMutation({
mutationKey: [MutationKeys.enqueueAgentQueuedTurn],
mutationFn: enqueueAgentQueuedTurn,
/** A retry can encounter admission middleware while the first request is
* still committing. Reconcile ambiguous outcomes through the read-only
* known-id projection instead of issuing a second mutating POST. */
retry: false,
onSettled: (_receipt, _error, input) =>
queryClient.invalidateQueries({
queryKey: [QueryKeys.agentQueuedTurns, input.conversationId],
}),
});
}
export function useCancelAgentQueuedTurnMutation() {
const queryClient = useQueryClient();
return useMutation({
mutationKey: [MutationKeys.cancelAgentQueuedTurn],
mutationFn: cancelAgentQueuedTurn,
onSuccess: (_receipt, input) =>
queryClient.invalidateQueries({
queryKey: [QueryKeys.agentQueuedTurns, input.conversationId],
}),
});
}

View file

@ -24,10 +24,10 @@ function setup(
const setters: {
setRunEnd?: (value: RunEnd | null) => void;
setIsSubmitting?: (value: boolean) => void;
setQueue?: (value: { id: string; text: string; createdAt: number }[]) => void;
setNewConvoQueue?: (value: { id: string; text: string; createdAt: number }[]) => void;
setQueue?: (value: QueuedMessage[]) => void;
setNewConvoQueue?: (value: QueuedMessage[]) => void;
setInterruptFlag?: (value: DrainAfterAbort | false) => void;
queueRef?: { current: { id: string; text: string; createdAt: number }[] };
queueRef?: { current: QueuedMessage[] };
} = {};
function Harness() {
@ -66,7 +66,11 @@ const emptyOverrides = expect.objectContaining({
overrideQueuedMessageOrigin: expect.any(Object),
});
const queuedMessage = (id: string, text: string) => ({ id, text, createdAt: Date.now() });
const queuedMessage = (id: string, text: string) => ({
id,
text,
createdAt: Date.now(),
});
const runEnd = (overrides: Partial<RunEnd> = {}): RunEnd => ({
conversationId: CONVO_ID,
@ -97,6 +101,55 @@ describe('useQueueDrain', () => {
expect(ask).toHaveBeenCalledWith({ text: 'first follow-up' }, emptyOverrides);
});
it('does not locally drain or renew server-owned Agent rows', async () => {
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{
...queuedMessage('q-server', 'the server starts this turn'),
files: [{ file_id: 'server-held-file' }],
clientRequestId: 'client-request-1',
server: { id: 'server-queue-1', status: 'queued', revision: 1 },
},
]);
});
act(() => {
setters.setRunEnd!(runEnd());
});
await new Promise((resolve) => setTimeout(resolve, 25));
expect(ask).not.toHaveBeenCalled();
expect(mockMarkFilesUsage).not.toHaveBeenCalledWith({
file_ids: ['server-held-file'],
});
});
it('retains the terminal boundary until server authority releases a local successor', async () => {
const localSuccessor = queuedMessage('q-local', 'legacy successor');
const queue: QueuedMessage[] = [
{
...queuedMessage('q-server', 'server-owned predecessor'),
server: { id: 'server-queue-1', status: 'queued', revision: 1 },
},
localSuccessor,
];
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), queue);
});
act(() => {
setters.setRunEnd!(runEnd());
});
await new Promise((resolve) => setTimeout(resolve, 25));
expect(ask).not.toHaveBeenCalled();
act(() => {
setters.setQueue!([localSuccessor]);
});
await waitFor(() => expect(ask).toHaveBeenCalledTimes(1));
expect(ask).toHaveBeenCalledWith({ text: 'legacy successor' }, emptyOverrides);
});
it('parks a mismatched signal instead of draining into the wrong conversation', async () => {
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [queuedMessage('q1', 'stay put')]);
@ -152,12 +205,18 @@ describe('useQueueDrain', () => {
it('renews the TTL hold on attachments that stay queued', async () => {
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'first'), files: [{ file_id: 'sent', type: 'image/png' }] },
{
...queuedMessage('q1', 'first'),
files: [{ file_id: 'sent', type: 'image/png' }],
},
{
...queuedMessage('q2', 'second'),
files: [{ file_id: 'still-queued', type: 'image/png' }],
},
{ ...queuedMessage('q3', 'third'), files: [{ file_id: 'also-queued', type: 'image/png' }] },
{
...queuedMessage('q3', 'third'),
files: [{ file_id: 'also-queued', type: 'image/png' }],
},
]);
});
@ -176,7 +235,10 @@ describe('useQueueDrain', () => {
it('does not renew when nothing with attachments stays queued', async () => {
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'only one'), files: [{ file_id: 'sent', type: 'image/png' }] },
{
...queuedMessage('q1', 'only one'),
files: [{ file_id: 'sent', type: 'image/png' }],
},
]);
});
@ -193,7 +255,10 @@ describe('useQueueDrain', () => {
* enqueue-time hold. */
it('renews every queued attachment across multiple capped batches', async () => {
const manyFiles = (prefix: string, n: number) =>
Array.from({ length: n }, (_, i) => ({ file_id: `${prefix}-${i}`, type: 'image/png' }));
Array.from({ length: n }, (_, i) => ({
file_id: `${prefix}-${i}`,
type: 'image/png',
}));
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'first'), files: manyFiles('sent', 2) },
@ -222,7 +287,10 @@ describe('useQueueDrain', () => {
it('renews a restored item when the send is refused', async () => {
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'refused'), files: [{ file_id: 'restored', type: 'image/png' }] },
{
...queuedMessage('q1', 'refused'),
files: [{ file_id: 'restored', type: 'image/png' }],
},
]);
});
ask.mockReturnValue(false);
@ -243,7 +311,10 @@ describe('useQueueDrain', () => {
try {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'waiting'), files: [{ file_id: 'held', type: 'image/png' }] },
{
...queuedMessage('q1', 'waiting'),
files: [{ file_id: 'held', type: 'image/png' }],
},
]);
});
@ -287,10 +358,16 @@ describe('useQueueDrain', () => {
it('renews the pre-migration NEW_CONVO queue alongside the active one', async () => {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(Constants.NEW_CONVO), [
{ ...queuedMessage('n1', 'queued pre-migration'), files: [{ file_id: 'pending-migrate' }] },
{
...queuedMessage('n1', 'queued pre-migration'),
files: [{ file_id: 'pending-migrate' }],
},
]);
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'queued after'), files: [{ file_id: 'already-migrated' }] },
{
...queuedMessage('q1', 'queued after'),
files: [{ file_id: 'already-migrated' }],
},
]);
});
@ -303,7 +380,10 @@ describe('useQueueDrain', () => {
it('does not double-count the queue before migration', async () => {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(Constants.NEW_CONVO), [
{ ...queuedMessage('n1', 'new convo'), files: [{ file_id: 'only-once' }] },
{
...queuedMessage('n1', 'new convo'),
files: [{ file_id: 'only-once' }],
},
]);
}, Constants.NEW_CONVO as string);
@ -558,7 +638,12 @@ describe('useQueueDrain', () => {
{ id: 'ordinary-before-url', text: 'ordinary follow-up', createdAt: 1 },
]);
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ id: 'interrupt-after-url', text: 'interrupt next', createdAt: 2, priority: true },
{
id: 'interrupt-after-url',
text: 'interrupt next',
createdAt: 2,
priority: true,
},
]);
set(store.drainAfterAbortByIndex(INDEX), {
conversationId: CONVO_ID,

File diff suppressed because it is too large Load diff

View file

@ -16,6 +16,9 @@ const QUEUE_USAGE_RENEW_INTERVAL_MS = 30 * 60 * 1000;
const collectQueuedFileIds = (items: QueuedMessage[]): string[] => {
const fileIds: string[] = [];
for (const item of items) {
if (item.server != null) {
continue;
}
for (const file of item.files ?? []) {
if (typeof file.file_id === 'string' && file.file_id.length > 0) {
fileIds.push(file.file_id);
@ -86,6 +89,7 @@ export default function useQueueDrain(
* during the first turn stay keyed here until that run ends. Renewing only
* the active id would skip them for the whole of that run. */
const newConvoQueue = useRecoilValue(store.queuedMessagesByConvoId(Constants.NEW_CONVO));
const hasServerOwnedQueue = [...ownQueue, ...newConvoQueue].some((item) => item.server != null);
/* Deduped because the two subscriptions are the same atom before migration.
* Keyed by id list so the effect re-runs when the held set changes, not
@ -146,7 +150,9 @@ export default function useQueueDrain(
}
set(store.pendingRunEndByConvoId(end.conversationId), {
...end,
...((end.interruptArmed === true || interruptArmed) && { interruptArmed: true }),
...((end.interruptArmed === true || interruptArmed) && {
interruptArmed: true,
}),
});
set(store.runEndByIndex(index), null);
return true;
@ -188,7 +194,9 @@ export default function useQueueDrain(
}
set(store.pendingRunEndByConvoId(end.conversationId), {
...end,
...((end.interruptArmed === true || interruptArmed) && { interruptArmed: true }),
...((end.interruptArmed === true || interruptArmed) && {
interruptArmed: true,
}),
});
set(store.runEndByIndex(index), null);
return null;
@ -205,19 +213,8 @@ export default function useQueueDrain(
if (end == null) {
return null;
}
// Consume the signal first — a hard double-fire guard even if the
// effect re-runs before Recoil propagates.
if (fromParked && activeConversationId) {
set(store.pendingRunEndByConvoId(activeConversationId), null);
} else {
set(store.runEndByIndex(index), null);
}
const indexArmed = snapshot.getLoadable(store.drainAfterAbortByIndex(index)).getValue();
const matchingIndexArm = matchesInterruptArm(indexArmed, end);
if (matchingIndexArm) {
set(store.drainAfterAbortByIndex(index), false);
}
const interruptArmed = matchingIndexArm || end.interruptArmed === true;
const conversationId = end.conversationId;
@ -241,6 +238,30 @@ export default function useQueueDrain(
: ownQueue;
const shouldDrain = end.outcome === 'completed' || interruptArmed;
/** A server-owned Agent row means the backend owns the next fresh-turn
* admission. Do not let a legacy/recovered local row race or overtake
* it. Keep this terminal boundary available until the authoritative
* queue snapshot proves whether a server-started successor now owns it. */
const serverOwnsBoundary = merged.some((item) => item.server != null);
if (serverOwnsBoundary) {
/** Keep the one-shot terminal signal until the authoritative snapshot
* removes the server row. If it was admitted, its own later terminal
* signal orders the remaining local queue; if it was cancelled/dead,
* this signal still lets the legacy successor make progress. */
return null;
}
// Consume only after server authority has yielded the boundary — a
// hard double-fire guard even if the effect re-runs before propagation.
if (fromParked && activeConversationId) {
set(store.pendingRunEndByConvoId(activeConversationId), null);
} else {
set(store.runEndByIndex(index), null);
}
if (matchingIndexArm) {
set(store.drainAfterAbortByIndex(index), false);
}
const next = shouldDrain ? (merged[0] ?? null) : null;
const remainder = next ? merged.slice(1) : merged;
@ -340,6 +361,7 @@ export default function useQueueDrain(
drainNext,
restoreQueued,
markFilesUsage,
hasServerOwnedQueue,
ask,
]);
}

View file

@ -2,16 +2,30 @@ import { useCallback, useEffect, useMemo, useRef } from 'react';
import { v4 } from 'uuid';
import { useToastContext } from '@librechat/client';
import { useRecoilValue, useSetRecoilState, useRecoilCallback } from 'recoil';
import { Constants, ContentTypes, isAssistantsEndpoint } from 'librechat-data-provider';
import type { TMessage, TConversation, TMessageContentParts } from 'librechat-data-provider';
import {
Constants,
ContentTypes,
isAgentsEndpoint,
isAssistantsEndpoint,
} from 'librechat-data-provider';
import type {
TAgentQueuedTurnFileRef,
TMessage,
TConversation,
TMessageContentParts,
} from 'librechat-data-provider';
import type { RunEnd, PendingSteer, QueuedMessage, QueuedMessageOrigin } from '~/store/families';
import type { GenerationProtocolVersion } from '~/data-provider';
import type { AgentQueuedTurnReceipt, GenerationProtocolVersion } from '~/data-provider';
import type { ExtendedFile, FileSetter } from '~/common';
import {
useGetMessagesByConvoId,
useCancelSteerMutation,
useSteerMessageMutation,
useMarkFilesUsageMutation,
useAgentQueuedTurns,
useCancelAgentQueuedTurnMutation,
useEnqueueAgentQueuedTurnMutation,
isDefiniteQueuedTurnRejection,
isDefiniteQueuedTurnsUnsupported,
supportsGenerationProtocolV2,
} from '~/data-provider';
import {
@ -23,6 +37,7 @@ import {
mergeRestagedQuotes,
} from '~/utils';
import useSteerConvert from '~/hooks/Chat/useSteerConvert';
import { useLatestMessage } from '~/hooks/Messages';
import { useSetFilesToDelete } from '~/hooks/Files';
import useLocalize from '~/hooks/useLocalize';
import store from '~/store';
@ -42,6 +57,10 @@ export interface QueuedMessageContext {
/** Server-side cap on a usage touch (mirrors `FILES_USAGE_MAX_IDS`). */
const QUEUE_USAGE_MAX_FILES = 10;
/** Bound transport-outcome reconciliation while still guaranteeing several
* list reads after the enqueue promise settles. Focus/remount remains a later
* reconciliation path for exceptionally slow intermediaries. */
const QUEUED_TURN_RECONCILIATION_MS = 60_000;
type SteerErrorCode =
| 'NO_ACTIVE_RUN'
@ -97,30 +116,139 @@ function isSameRunEpoch(a: RunEnd | null, b: RunEnd): boolean {
/** True when the latest assistant message carries an unresolved tool approval —
* the run is (or is about to be) paused, so a steer POST would 409. */
function hasLiveToolApproval(messages: TMessage[] | undefined): boolean {
if (!messages || messages.length === 0) {
function hasLiveToolApproval(message: TMessage | null): boolean {
if (message?.isCreatedByUser !== false) {
return false;
}
for (let i = messages.length - 1; i >= 0; i--) {
const message = messages[i];
if (message.isCreatedByUser !== false) {
continue;
}
const content = message.content;
if (!Array.isArray(content)) {
const content = message.content;
if (!Array.isArray(content)) {
return false;
}
return content.some((part: TMessageContentParts | undefined) => {
if (part?.type !== ContentTypes.TOOL_CALL) {
return false;
}
return content.some((part: TMessageContentParts | undefined) => {
if (part?.type !== ContentTypes.TOOL_CALL) {
return false;
}
const toolCall = part[ContentTypes.TOOL_CALL] as
| { approval?: unknown; output?: string | null }
| undefined;
return toolCall?.approval != null && (toolCall.output?.length ?? 0) === 0;
});
const toolCall = part[ContentTypes.TOOL_CALL] as
| { approval?: unknown; output?: string | null }
| undefined;
return toolCall?.approval != null && (toolCall.output?.length ?? 0) === 0;
});
}
interface LiveMessageState {
approval: boolean;
parentMessageId?: string;
}
function selectLiveMessageState(message: TMessage | null): LiveMessageState {
const parentMessageId =
message?.isCreatedByUser === false && typeof message.messageId === 'string'
? message.messageId
: undefined;
return { approval: hasLiveToolApproval(message), parentMessageId };
}
function compareQueuedMessages(a: QueuedMessage, b: QueuedMessage): number {
const priority = Number(b.priority ?? false) - Number(a.priority ?? false);
if (priority !== 0) {
return priority;
}
return false;
const aServerOrder = a.server?.revision;
const bServerOrder = b.server?.revision;
if (aServerOrder != null && bServerOrder != null && aServerOrder !== bServerOrder) {
return aServerOrder - bServerOrder;
}
if (a.server != null && b.server == null) {
return -1;
}
if (a.server == null && b.server != null) {
return 1;
}
return a.createdAt - b.createdAt;
}
function queuedTurnCreatedAt(receipt: AgentQueuedTurnReceipt): number {
const createdAt = Date.parse(receipt.createdAt);
return Number.isFinite(createdAt) ? createdAt : 0;
}
function toQueuedTurnFileRefs(files: TMessage['files']): TAgentQueuedTurnFileRef[] | undefined {
const refs = (files ?? []).flatMap((file): TAgentQueuedTurnFileRef[] => {
if (typeof file.file_id !== 'string' || file.file_id.length === 0) {
return [];
}
return [
{
file_id: file.file_id,
...(file.type != null && { type: file.type }),
...(file.filepath != null && { filepath: file.filepath }),
...(file.filename != null && { filename: file.filename }),
...(file.height != null && { height: file.height }),
...(file.width != null && { width: file.width }),
...(file.bytes != null && { bytes: file.bytes }),
},
];
});
return refs.length > 0 ? refs : undefined;
}
function reconcileServerQueuedTurns(
previous: QueuedMessage[],
receipts: AgentQueuedTurnReceipt[],
): QueuedMessage[] {
const previousByClientRequestId = new Map(
previous.flatMap((item) =>
item.clientRequestId != null ? [[item.clientRequestId, item] as const] : [],
),
);
const observedClientRequestIds = new Set(receipts.map((receipt) => receipt.clientRequestId));
const projected = receipts.flatMap((receipt): QueuedMessage[] => {
if (receipt.status === 'admitted' || receipt.status === 'cancelled') {
return [];
}
const optimistic = previousByClientRequestId.get(receipt.clientRequestId);
return [
{
id: optimistic?.id ?? receipt.clientRequestId,
text: receipt.text,
createdAt: queuedTurnCreatedAt(receipt),
clientRequestId: receipt.clientRequestId,
parentMessageId: receipt.parentMessageId,
...(receipt.expectedPredecessorCreatedAt != null && {
expectedPredecessorCreatedAt: receipt.expectedPredecessorCreatedAt,
}),
...(receipt.files != null && receipt.files.length > 0 && { files: receipt.files }),
...(receipt.quotes != null && receipt.quotes.length > 0 && { quotes: receipt.quotes }),
...(receipt.manualSkills != null &&
receipt.manualSkills.length > 0 && {
manualSkills: receipt.manualSkills,
}),
...(receipt.priority === true && { priority: true }),
server: {
id: receipt.queuedTurnId,
status: receipt.status === 'dead' ? 'rejected' : receipt.status,
revision: receipt.revision,
...(receipt.position != null && { position: receipt.position }),
...(receipt.failure?.code != null && { errorCode: receipt.failure.code }),
...(receipt.failure?.message != null && { errorMessage: receipt.failure.message }),
},
},
];
});
const retained = previous.filter((item) => {
if (item.server == null) {
return true;
}
if (item.clientRequestId == null || observedClientRequestIds.has(item.clientRequestId)) {
return false;
}
return (
item.server.status === 'sending' ||
item.server.status === 'uncertain' ||
item.server.status === 'rejected'
);
});
return [...retained, ...projected].sort(compareQueuedMessages);
}
export interface UseSteeringParams {
@ -177,6 +305,8 @@ export default function useSteering({
const { mutate: steerMessage } = useSteerMessageMutation();
const { mutateAsync: cancelSteer } = useCancelSteerMutation();
const { mutate: markFilesUsage } = useMarkFilesUsageMutation();
const { mutate: enqueueAgentQueuedTurn } = useEnqueueAgentQueuedTurnMutation();
const { mutateAsync: cancelAgentQueuedTurn } = useCancelAgentQueuedTurnMutation();
const defaultAction = useRecoilValue<DuringRunAction>(store.duringRunDefaultAction);
const setDefaultAction = useSetRecoilState(store.duringRunDefaultAction);
const steerInterruptsByDefault = useRecoilValue(store.steerInterruptsByDefault);
@ -185,15 +315,87 @@ export default function useSteering({
const steerable = !isAssistantsEndpoint(endpoint);
const hasRealConvoId =
conversationId != null && conversationId !== '' && conversationId !== Constants.NEW_CONVO;
const serverQueueEnabled = isAgentsEndpoint(endpoint) && hasRealConvoId;
/** v1 gates the during-run UI to the primary composer, like the HITL popover. */
const enabled = steerable && index === 0;
const queueKey = hasRealConvoId ? conversationId : Constants.NEW_CONVO;
const queuedMessages = useRecoilValue(store.queuedMessagesByConvoId(queueKey));
const setQueuedMessages = useSetRecoilState(store.queuedMessagesByConvoId(queueKey));
const knownClientRequestIds = useMemo(
() =>
Array.from(
new Set(
queuedMessages.flatMap((item) =>
item.server != null && item.clientRequestId != null ? [item.clientRequestId] : [],
),
),
).slice(0, 100),
[queuedMessages],
);
const reconciliationUntil = queuedMessages.reduce<number | undefined>((latest, item) => {
if (item.server?.status !== 'uncertain' || item.server.uncertainSince == null) {
return latest;
}
const until = item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS;
return latest == null ? until : Math.max(latest, until);
}, undefined);
const nextReconciliationExpiry = queuedMessages.reduce<number | undefined>((earliest, item) => {
if (
item.server?.status !== 'uncertain' ||
item.server.uncertainSince == null ||
item.server.reconciliationExpired === true
) {
return earliest;
}
const expiry = item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS;
return earliest == null ? expiry : Math.min(earliest, expiry);
}, undefined);
const { data: serverQueuedTurns } = useAgentQueuedTurns(
conversationId,
serverQueueEnabled,
knownClientRequestIds,
reconciliationUntil,
);
const activeGenerationCreatedAt = useRecoilValue(
store.activeGenerationCreatedAtByConvoId(queueKey),
);
const activeGenerationProtocolVersion = useRecoilValue(
store.activeGenerationProtocolVersionByConvoId(queueKey),
);
useEffect(() => {
if (!serverQueueEnabled || serverQueuedTurns == null) {
return;
}
setQueuedMessages((previous) => reconcileServerQueuedTurns(previous, serverQueuedTurns));
}, [serverQueueEnabled, serverQueuedTurns, setQueuedMessages]);
useEffect(() => {
if (!serverQueueEnabled || nextReconciliationExpiry == null) {
return;
}
const expire = () => {
const observedAt = Date.now();
setQueuedMessages((previous) =>
previous.map((item) => {
if (
item.server?.status !== 'uncertain' ||
item.server.uncertainSince == null ||
item.server.reconciliationExpired === true ||
item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS > observedAt
) {
return item;
}
return {
...item,
server: { ...item.server, reconciliationExpired: true },
};
}),
);
};
const delay = Math.max(0, nextReconciliationExpiry - Date.now());
const timer = window.setTimeout(expire, delay);
return () => window.clearTimeout(timer);
}, [nextReconciliationExpiry, serverQueueEnabled, setQueuedMessages]);
/** The start POST installs this epoch before any live mutation may be sent.
* `isSubmitting` flips earlier so the user can keep typing; during that
* bounded interval submits degrade to the local queue and Stop/steer refuse. */
@ -299,22 +501,17 @@ export default function useSteering({
}
}, [pendingSteers, queueKey]);
/** Boolean `select` so streaming deltas don't notify this subscription:
* structural sharing only re-renders the composer when the flag flips. */
const { data: liveToolApproval } = useGetMessagesByConvoId<boolean>(
hasRealConvoId ? conversationId : '',
{
enabled: hasRealConvoId,
select: hasLiveToolApproval,
},
);
/** The exact visible assistant tail is captured into a server queued turn,
* while the approval bit keeps the steering controls honest. */
const latestMessage = useLatestMessage(index, hasRealConvoId ? conversationId : null);
const liveMessageState = useMemo(() => selectLiveMessageState(latestMessage), [latestMessage]);
/** Both approval cards and `ask_user_question` suspend the current
* generation while keeping its submission slot occupied. Answer mode hides
* the ordinary during-run composer, but waiting-message controls still need
* to know a live generation exists so they remain discoverable and disabled
* instead of looking as though the action disappeared. */
const pausedOnApproval =
enabled && isSubmitting ? answerModeActive || (liveToolApproval ?? false) : false;
enabled && isSubmitting ? answerModeActive || (liveMessageState?.approval ?? false) : false;
/** Whether a steer can reach the live run right now — independent of the
* user's default action, so the per-send menu can always override to
@ -523,6 +720,22 @@ export default function useSteering({
[markFilesUsage],
);
const updateQueuedMessage = useRecoilCallback(
({ set }) =>
(id: string, update: (item: QueuedMessage) => QueuedMessage | null) => {
set(store.queuedMessagesByConvoId(queueKey), (previous) =>
previous.flatMap((item) => {
if (item.id !== id) {
return [item];
}
const next = update(item);
return next == null ? [] : [next];
}),
);
},
[queueKey],
);
const enqueue = useRecoilCallback(
({ set }) =>
(
@ -545,11 +758,23 @@ export default function useSteering({
if (trimmed.length === 0) {
return;
}
const item = {
const parentMessageId = liveMessageState?.parentMessageId;
/** The active epoch is the authority-transfer fence. During startup an
* existing branch tail may already be visible while no generation owns
* it yet, so keep those turns local until the epoch is concrete. */
const serverOwned =
serverQueueEnabled && parentMessageId != null && activeGenerationCreatedAt != null;
const generatedClientRequestId = options?.clientRequestId == null;
const clientRequestId = options?.clientRequestId ?? (serverOwned ? v4() : undefined);
const item: QueuedMessage = {
id: options?.id ?? v4(),
text: trimmed,
createdAt: options?.createdAt ?? Date.now(),
...(options?.clientRequestId && { clientRequestId: options.clientRequestId }),
...(clientRequestId != null && { clientRequestId }),
...(serverOwned && {
parentMessageId,
server: { status: 'sending' },
}),
...((options?.expectedPredecessorCreatedAt ?? activeGenerationCreatedAt) != null && {
expectedPredecessorCreatedAt:
options?.expectedPredecessorCreatedAt ?? activeGenerationCreatedAt ?? undefined,
@ -557,21 +782,117 @@ export default function useSteering({
...(options?.files && options.files.length > 0 && { files: options.files }),
...(options?.quotes && options.quotes.length > 0 && { quotes: options.quotes }),
...(options?.manualSkills &&
options.manualSkills.length > 0 && { manualSkills: options.manualSkills }),
options.manualSkills.length > 0 && {
manualSkills: options.manualSkills,
}),
...(options?.front && { priority: true }),
};
set(store.queuedMessagesByConvoId(queueKey), (prev) =>
[...prev, item].sort(
(a, b) =>
Number(b.priority ?? false) - Number(a.priority ?? false) ||
a.createdAt - b.createdAt,
),
[...prev, item].sort(compareQueuedMessages),
);
if (options?.skipUsageMark !== true) {
markQueuedFilesUsage(options?.files);
}
if (!serverOwned || clientRequestId == null) {
return;
}
const serverFiles = toQueuedTurnFileRefs(item.files);
/** Commit the optimistic Recoil row before mutation callbacks can
* reconcile it. This also makes synchronous test/adaptor completions
* obey the same ordering as a real network response. */
queueMicrotask(() =>
enqueueAgentQueuedTurn(
{
conversationId,
clientRequestId,
parentMessageId,
text: item.text,
...(serverFiles != null && { files: serverFiles }),
...(item.quotes != null && item.quotes.length > 0 && { quotes: item.quotes }),
...(item.manualSkills != null &&
item.manualSkills.length > 0 && {
manualSkills: item.manualSkills,
}),
...(item.priority === true && { priority: true }),
...(item.expectedPredecessorCreatedAt != null && {
expectedPredecessorCreatedAt: item.expectedPredecessorCreatedAt,
}),
},
{
onSuccess: (receipt) => {
updateQueuedMessage(item.id, (current) => {
if (receipt.status === 'admitted' || receipt.status === 'cancelled') {
return null;
}
return {
...current,
text: receipt.text,
createdAt: queuedTurnCreatedAt(receipt),
parentMessageId: receipt.parentMessageId,
server: {
id: receipt.queuedTurnId,
status: receipt.status === 'dead' ? 'rejected' : receipt.status,
revision: receipt.revision,
...(receipt.position != null && { position: receipt.position }),
...(receipt.failure?.code != null && { errorCode: receipt.failure.code }),
...(receipt.failure?.message != null && {
errorMessage: receipt.failure.message,
}),
},
};
});
},
onError: (error) => {
updateQueuedMessage(item.id, (current) => {
if (isDefiniteQueuedTurnsUnsupported(error)) {
const {
server: _server,
parentMessageId: _parentMessageId,
clientRequestId: fallbackClientRequestId,
...legacy
} = current;
return {
...legacy,
...(!generatedClientRequestId && fallbackClientRequestId != null
? { clientRequestId: fallbackClientRequestId }
: {}),
};
}
if (!isDefiniteQueuedTurnRejection(error)) {
return {
...current,
server: {
...current.server,
status: 'uncertain',
uncertainSince: current.server?.uncertainSince ?? Date.now(),
},
};
}
const code = getSteerErrorCode(error);
return {
...current,
server: {
...current.server,
status: 'rejected',
...(code != null && { errorCode: code }),
},
};
});
},
},
),
);
},
[queueKey, markQueuedFilesUsage, activeGenerationCreatedAt],
[
queueKey,
conversationId,
serverQueueEnabled,
liveMessageState?.parentMessageId,
markQueuedFilesUsage,
activeGenerationCreatedAt,
enqueueAgentQueuedTurn,
updateQueuedMessage,
],
);
/** Consumes completed composer attachments into message file refs (clearing
@ -678,7 +999,10 @@ export default function useSteering({
return rest;
}
const { quotes: _originQuotes, ...originItem } = rest.queuedOrigin.item;
return { ...rest, queuedOrigin: { ...rest.queuedOrigin, item: originItem } };
return {
...rest,
queuedOrigin: { ...rest.queuedOrigin, item: originItem },
};
});
return changed ? next : prev;
});
@ -709,6 +1033,27 @@ export default function useSteering({
[queueKey],
);
const downgradeServerQueuedTurn = useRecoilCallback(
({ snapshot, set }) =>
(id: string): boolean => {
const queue = snapshot.getLoadable(store.queuedMessagesByConvoId(queueKey)).getValue();
let found = false;
const next = queue.map((item) => {
if (item.id !== id) {
return item;
}
found = true;
const { server: _server, parentMessageId: _parentMessageId, ...local } = item;
return local;
});
if (found) {
set(store.queuedMessagesByConvoId(queueKey), next);
}
return found;
},
[queueKey],
);
/** Once a parked source is discarded it must never be retried as a recovery
* attempt. Downgrade the row in place so a guarded Edit that finds a newer
* draft can leave the same words, context, identity, and queue position as
@ -747,11 +1092,46 @@ export default function useSteering({
* terminal source generation, not whichever run now occupies the chat. */
const discardQueued = useCallback(
async (item: QueuedMessage): Promise<boolean> => {
if (item.server != null) {
if (item.server.status === 'rejected' && item.server.id == null) {
return downgradeServerQueuedTurn(item.id);
}
if (item.server.id == null || !hasRealConvoId) {
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
try {
const receipt = await cancelAgentQueuedTurn({
conversationId,
queuedTurnId: item.server.id,
});
if (receipt.status !== 'cancelled') {
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
return downgradeServerQueuedTurn(item.id);
} catch {
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
}
if (item.recoverySteerId == null) {
return true;
}
if (item.recoveryClientSteerId == null || !hasRealConvoId) {
showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
try {
@ -761,16 +1141,31 @@ export default function useSteering({
clientSteerId: item.recoveryClientSteerId,
});
if (removed !== true) {
showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
return downgradeQueuedRecovery(item.id);
} catch {
showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
showToast({
message: localize('com_ui_steer_cancel_failed'),
status: 'error',
});
return false;
}
},
[cancelSteer, conversationId, downgradeQueuedRecovery, hasRealConvoId, localize, showToast],
[
cancelSteer,
cancelAgentQueuedTurn,
conversationId,
downgradeQueuedRecovery,
downgradeServerQueuedTurn,
hasRealConvoId,
localize,
showToast,
],
);
/** Capture-then-remove, including the item's neighbours, so any refused send
@ -967,8 +1362,12 @@ export default function useSteering({
expectedPredecessorCreatedAt: targetGenerationCreatedAt,
...(files && { files }),
...carried,
...(context?.clientRequestId && { clientRequestId: context.clientRequestId }),
...(context?.recoverySteerId && { recoverySteerId: context.recoverySteerId }),
...(context?.clientRequestId && {
clientRequestId: context.clientRequestId,
}),
...(context?.recoverySteerId && {
recoverySteerId: context.recoverySteerId,
}),
},
beforeIds: [],
afterIds: [],
@ -1050,7 +1449,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
...(opts?.queuedOrigin && {
queuedOrigin: opts.queuedOrigin,
}),
...carried,
});
} else {
@ -1080,7 +1481,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
...(opts?.queuedOrigin && {
queuedOrigin: opts.queuedOrigin,
}),
...carried,
} satisfies PendingSteer;
if (acknowledgeSteer(conversationId, localId, acknowledged)) {
@ -1137,11 +1540,18 @@ export default function useSteering({
return;
}
restoreRejectedSteer();
showToast({ message: localize('com_ui_steer_paused_queued'), status: 'info' });
showToast({
message: localize('com_ui_steer_paused_queued'),
status: 'info',
});
return;
}
const errorData = (
error as { response?: { data?: { generationProtocolVersion?: unknown } } }
error as {
response?: {
data?: { generationProtocolVersion?: unknown };
};
}
)?.response?.data;
if (
code === 'RUN_REPLACED' &&
@ -1154,7 +1564,10 @@ export default function useSteering({
* drain regardless of this tab's local submission state. */
replaceSteerChip(conversationId, localId, null);
restoreRejectedSteer();
showToast({ message: localize('com_ui_steer_paused_queued'), status: 'info' });
showToast({
message: localize('com_ui_steer_paused_queued'),
status: 'info',
});
return;
}
if (isDefiniteSteerRejection(error)) {
@ -1173,7 +1586,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
...(opts?.queuedOrigin && {
queuedOrigin: opts.queuedOrigin,
}),
...carried,
});
return;
@ -1190,7 +1605,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
...(opts?.queuedOrigin && {
queuedOrigin: opts.queuedOrigin,
}),
...carried,
});
} finally {
@ -1256,7 +1673,10 @@ export default function useSteering({
if (trimmed.length === 0 || filesLoading) {
return false;
}
enqueue(trimmed, { files: takeComposerFiles(), ...takeComposerContext() });
enqueue(trimmed, {
files: takeComposerFiles(),
...takeComposerContext(),
});
takeComposerDraft();
return true;
},
@ -1274,7 +1694,10 @@ export default function useSteering({
) => {
/** A failed interrupt-steer must retry AS an interrupt — resubmitting it
* as an ordinary steer would silently let generation run on. */
submitSteer(text, steerFiles, context, { ...opts, clientSteerId: steerId });
submitSteer(text, steerFiles, context, {
...opts,
clientSteerId: steerId,
});
},
[submitSteer],
);
@ -1356,7 +1779,7 @@ export default function useSteering({
* restore context so a degraded steer requeues/sends with them intact.
* `preempt` escalates the steer to interrupt at the next safe token
* boundary; it only means something on the live-run path. */
const sendQueuedNow = useCallback(
const sendLocalQueuedNow = useCallback(
(item: QueuedMessage, opts?: { preempt?: boolean }) => {
/** In answer mode (and any other submission-owned non-steerable state)
* there is no immediate path. Refuse before touching queue state so a
@ -1366,7 +1789,11 @@ export default function useSteering({
}
/** UI callers always find the item; a stale/direct caller has no original
* neighbours, so restoration falls back to the queue's priority split. */
const origin = takeQueued(item.id) ?? { item, beforeIds: [], afterIds: [] };
const origin = takeQueued(item.id) ?? {
item,
beforeIds: [],
afterIds: [],
};
const taken = origin.item;
if (duringRunActive && canSteer) {
const consumed = submitSteer(
@ -1432,6 +1859,30 @@ export default function useSteering({
],
);
const queuedActionClaimsRef = useRef(new Set<string>());
const sendQueuedNow = useCallback(
(item: QueuedMessage, opts?: { preempt?: boolean }) => {
if (item.server == null) {
sendLocalQueuedNow(item, opts);
return;
}
if (queuedActionClaimsRef.current.has(item.id)) {
return;
}
queuedActionClaimsRef.current.add(item.id);
void discardQueued(item)
.then((discarded) => {
if (discarded) {
sendLocalQueuedNow(item, opts);
}
})
.finally(() => {
queuedActionClaimsRef.current.delete(item.id);
});
},
[discardQueued, sendLocalQueuedNow],
);
/** Abort the current run and auto-send this text once the abort settles. */
const interruptAndSend = useCallback(
(text: string): boolean => {
@ -1439,7 +1890,11 @@ export default function useSteering({
if (trimmed.length === 0 || filesLoading || !canControlGeneration) {
return false;
}
enqueue(trimmed, { front: true, files: takeComposerFiles(), ...takeComposerContext() });
enqueue(trimmed, {
front: true,
files: takeComposerFiles(),
...takeComposerContext(),
});
takeComposerDraft();
armDrainAfterAbort();
stopGenerating();

View file

@ -1790,6 +1790,7 @@
"com_ui_queued_attachment_count": "{{0}} attachments queued with this message",
"com_ui_queued_messages": "Queued messages",
"com_ui_queued_quote_count": "{{0}} quoted excerpts included with this message",
"com_ui_queued_turn_failed": "Queued message failed",
"com_ui_quote_selections": "{{0}} selections",
"com_ui_quotes_queued": "Quotes added for your next message",
"com_ui_ran_n_agents": "Ran {{0}} agents",
@ -1829,6 +1830,7 @@
"com_ui_remove_all_quotes": "Remove all selections",
"com_ui_remove_from_project": "Remove from project",
"com_ui_remove_queued": "Remove message",
"com_ui_dismiss_unconfirmed_delivery": "Dismiss unconfirmed delivery",
"com_ui_remove_quote": "Remove quote",
"com_ui_remove_skill_var": "Remove {{0}}",
"com_ui_remove_user": "Remove {{0}}",

View file

@ -74,7 +74,11 @@ const conversationByIndex = atomFamily<TConversation | null, string | number>({
({ onSet, node }) => {
onSet(async (newValue, oldValue) => {
const index = Number(node.key.split('__')[1]);
logger.log('conversation', 'Setting conversation:', { index, newValue, oldValue });
logger.log('conversation', 'Setting conversation:', {
index,
newValue,
oldValue,
});
if (newValue?.assistant_id != null && newValue.assistant_id) {
localStorage.setItem(
`${LocalStorageKeys.ASST_ID_PREFIX}${index}${newValue.endpoint}`,
@ -392,9 +396,33 @@ export type QueuedMessage = {
id: string;
text: string;
createdAt: number;
/** Stable only for this queued recovery attempt and its transport retries.
* A failed generation re-converts the durable source with a fresh key. */
/** Server authority for an Agent queued turn. Absence means the row remains
* on the legacy mounted-client drain path (including a definite old-server
* fallback). `uncertain` is deliberately still server-owned: falling back
* after an ambiguous POST could submit the same words twice. */
server?: {
id?: string;
status: 'sending' | 'uncertain' | 'rejected' | 'queued' | 'claimed';
errorCode?: string;
errorMessage?: string;
/** Observation time for a transport-ambiguous enqueue. The logical item
* may be much older than the request that just became uncertain. */
uncertainSince?: number;
/** The bounded reconciliation window elapsed without authoritative
* evidence. The outcome remains ambiguous and must never become resendable. */
reconciliationExpired?: boolean;
/** Current one-based projection; server sequence remains the stable
* fallback when predecessors settle and positions close up. */
position?: number;
revision?: number;
};
/** Stable identity for server enqueue/retry. Recovered steer rows also use
* it to dismiss their parked source; a later recovery attempt gets a fresh
* identity. */
clientRequestId?: string;
/** Exact visible branch leaf captured when this turn entered the server
* queue. The server revalidates it before admitting the fresh successor. */
parentMessageId?: string;
/** Correlation used only to durably dismiss/reclaim the parked source. */
recoveryClientSteerId?: string;
recoverySteerId?: string;

View file

@ -67,3 +67,5 @@ export * from './view';
export * from './reasoningLabels';
export * from './toolValidation';
export * from './remote';
export * from './queuedTurns';
export * from './queuedTurnHttp';

View file

@ -0,0 +1,204 @@
import { Types } from 'mongoose';
import { AgentQueuedTurnLaneRetiredError } from '@librechat/data-schemas';
import type { AgentQueuedTurnMethods, AgentQueuedTurnRecord } from '@librechat/data-schemas';
import type { AgentQueuedTurnHttpDeps } from './queuedTurnHttp';
import {
handleAgentQueuedTurnCancel,
handleAgentQueuedTurnEnqueue,
handleAgentQueuedTurnList,
} from './queuedTurnHttp';
const USER_ID = '507f191e810c19729de860ea';
function turn(status: AgentQueuedTurnRecord['status']): AgentQueuedTurnRecord {
return {
queuedTurnId: 'queued-turn-1',
user: new Types.ObjectId(USER_ID),
conversationId: 'conversation-1',
agentId: 'agent_1',
parentMessageId: 'assistant-1',
clientRequestId: 'client-request-1',
fingerprint: 'fingerprint-1',
sequence: 1,
status,
priority: false,
text: 'follow up',
attempts: status === 'admitted' ? 1 : 0,
availableAt: new Date('2026-08-30T12:00:00Z'),
createdAt: new Date('2026-08-30T12:00:00Z'),
};
}
function requestBody() {
return {
conversationId: 'conversation-1',
parentMessageId: 'assistant-1',
clientRequestId: 'client-request-1',
text: 'follow up',
};
}
describe('Agent queued-turn HTTP admission receipts', () => {
it('rejects an enqueue after conversation deletion closes its lane', async () => {
const methods = {
getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
getAgentQueuedTurnByClientRequestId: jest.fn(async () => null),
enqueueAgentQueuedTurn: jest.fn(async () => {
throw new AgentQueuedTurnLaneRetiredError();
}),
};
const deps = {
methods: methods as unknown as AgentQueuedTurnMethods & {
getConvo: typeof methods.getConvo;
},
lifecycle: { schedule: jest.fn(), cancel: jest.fn() },
checkAgentAccess: jest.fn(async () => true),
} satisfies AgentQueuedTurnHttpDeps;
await expect(
handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
).resolves.toEqual({
status: 409,
body: { code: 'QUEUED_TURN_CONVERSATION_DELETING' },
});
});
it('resolves a scheduling-pending response through an exact same-body terminal replay', async () => {
const enqueueAgentQueuedTurn = jest
.fn()
.mockResolvedValueOnce({ turn: turn('queued'), replayed: false });
const getAgentQueuedTurnByClientRequestId = jest
.fn()
.mockResolvedValueOnce(null)
.mockResolvedValueOnce(turn('admitted'));
const schedule = jest.fn().mockRejectedValueOnce(new Error('scheduler unavailable'));
const methods = {
getConvo: jest.fn(async () => ({
agent_id: 'agent_1',
endpoint: 'agents',
})),
enqueueAgentQueuedTurn,
getAgentQueuedTurnByClientRequestId,
listActiveAgentQueuedTurns: jest.fn(async () => []),
};
const deps = {
methods: methods as unknown as AgentQueuedTurnMethods & {
getConvo: typeof methods.getConvo;
},
lifecycle: { schedule, cancel: jest.fn() },
checkAgentAccess: jest.fn(async () => true),
} satisfies AgentQueuedTurnHttpDeps;
await expect(
handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
).resolves.toMatchObject({
status: 503,
body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' },
});
await expect(
handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
).resolves.toMatchObject({
status: 200,
body: {
receipt: {
queuedTurnId: 'queued-turn-1',
clientRequestId: 'client-request-1',
status: 'admitted',
},
},
});
expect(getAgentQueuedTurnByClientRequestId).toHaveBeenCalledTimes(2);
expect(getAgentQueuedTurnByClientRequestId.mock.calls[1][0]).toEqual(
getAgentQueuedTurnByClientRequestId.mock.calls[0][0],
);
expect(enqueueAgentQueuedTurn).toHaveBeenCalledTimes(1);
expect(methods.getConvo).toHaveBeenCalledTimes(1);
expect(schedule).toHaveBeenCalledTimes(1);
});
it('surfaces a dead receipt and lets the user dismiss its delivery', async () => {
const dead = {
...turn('dead'),
deliveryKey: 'delivery-1',
settledAt: new Date('2026-08-30T12:01:00Z'),
terminalReceipt: {
outcome: 'dead' as const,
settledAt: new Date('2026-08-30T12:01:00Z'),
failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
},
};
const cancelled = {
...dead,
status: 'cancelled' as const,
terminalReceipt: {
outcome: 'cancelled' as const,
settledAt: new Date('2026-08-30T12:02:00Z'),
},
};
const methods = {
getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
listAgentQueuedTurnReceipts: jest.fn(async () => [dead]),
};
const cancel = jest.fn(async () => ({ outcome: 'cancelled' as const, turn: cancelled }));
const deps = {
methods: methods as unknown as AgentQueuedTurnMethods & {
getConvo: typeof methods.getConvo;
},
lifecycle: { schedule: jest.fn(), cancel },
checkAgentAccess: jest.fn(async () => true),
} satisfies AgentQueuedTurnHttpDeps;
await expect(
handleAgentQueuedTurnList({ id: USER_ID }, 'conversation-1', deps),
).resolves.toMatchObject({
status: 200,
body: {
queuedTurns: [
{
queuedTurnId: 'queued-turn-1',
status: 'dead',
failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
},
],
},
});
await expect(
handleAgentQueuedTurnCancel({ id: USER_ID }, 'queued-turn-1', deps),
).resolves.toMatchObject({
status: 200,
body: { receipt: { queuedTurnId: 'queued-turn-1', status: 'cancelled' } },
});
expect(cancel).toHaveBeenCalledWith(expect.objectContaining({ queuedTurnId: 'queued-turn-1' }));
});
it('retires a cancelled source after its published delivery receipt expires', async () => {
const cancelled = {
...turn('cancelled'),
deliveryKey: 'delivery-expired',
deliveryState: 'published' as const,
terminalReceipt: {
outcome: 'cancelled' as const,
settledAt: new Date('2026-08-30T12:02:00Z'),
},
};
const methods = {
getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
};
const cancel = jest.fn(async () => ({
outcome: 'already_cancelled' as const,
turn: cancelled,
}));
const deps = {
methods: methods as unknown as AgentQueuedTurnMethods & {
getConvo: typeof methods.getConvo;
},
lifecycle: { schedule: jest.fn(), cancel },
} satisfies AgentQueuedTurnHttpDeps;
await expect(
handleAgentQueuedTurnCancel({ id: USER_ID }, 'queued-turn-1', deps),
).resolves.toMatchObject({ status: 200 });
expect(cancel).toHaveBeenCalledTimes(1);
});
});

View file

@ -0,0 +1,452 @@
import { Types } from 'mongoose';
import {
enqueueAgentQueuedTurnSchema,
isAgentsEndpoint,
isEphemeralAgentId,
} from 'librechat-data-provider';
import {
AgentQueuedTurnCapacityError,
AgentQueuedTurnConflictError,
AgentQueuedTurnLaneRetiredError,
} from '@librechat/data-schemas';
import type {
AgentQueuedTurnActiveRecord,
AgentQueuedTurnMethods,
AgentQueuedTurnRecord,
IMongoFile,
} from '@librechat/data-schemas';
import type {
TAgentQueuedTurnFileRef,
TAgentQueuedTurnReceipt,
TFile,
} from 'librechat-data-provider';
import type { AgentQueuedTurnLifecycle } from './queuedTurns';
import type { SteerFileFetcher } from './steering/request';
import type { SteerRequestUser } from './steering/refs';
import { buildOwnerFilter, collectFileIds, toSteerFileRef } from './steering/refs';
import { getReferencedQuotes } from '~/utils';
const MAX_QUEUED_TURN_LENGTH = 16_000;
const MAX_QUEUED_TURN_FILES = 10;
const CAPABILITY = { supported: true, durability: 'durable' } as const;
interface QueuedTurnConversation {
agent_id?: string;
endpoint?: string;
tenantId?: string;
}
interface QueuedTurnHttpMethods extends AgentQueuedTurnMethods {
getConvo: (userId: string, conversationId: string) => Promise<QueuedTurnConversation | null>;
}
export interface AgentQueuedTurnHttpDeps {
methods: QueuedTurnHttpMethods;
lifecycle: Pick<AgentQueuedTurnLifecycle, 'schedule' | 'cancel'>;
getFiles?: SteerFileFetcher;
updateFilesUsage?: (
files: Array<{ file_id: string }>,
fileIds?: string[],
options?: { user?: string; tenantId?: string | null },
) => Promise<unknown[]>;
checkAgentAccess?: (run: { agentId?: string; endpoint?: string }) => Promise<boolean>;
isPrincipalActive?: (userId: string) => boolean | Promise<boolean>;
}
export interface AgentQueuedTurnHttpResult {
status: number;
body: Record<string, unknown>;
}
function owner(user: SteerRequestUser): { user: Types.ObjectId; tenantId?: string } | null {
if (typeof user.id !== 'string' || !Types.ObjectId.isValid(user.id)) {
return null;
}
return {
user: new Types.ObjectId(user.id),
...(user.tenantId != null && { tenantId: user.tenantId }),
};
}
function sameTenant(actual: unknown, expected: string | undefined): boolean {
return expected == null ? actual == null : actual === expected;
}
function receipt(
turn: AgentQueuedTurnRecord | AgentQueuedTurnActiveRecord,
position?: number,
): TAgentQueuedTurnReceipt {
const updatedAt = turn.updatedAt ?? turn.createdAt;
return {
queuedTurnId: turn.queuedTurnId,
conversationId: turn.conversationId,
parentMessageId: turn.parentMessageId,
clientRequestId: turn.clientRequestId,
text: turn.text,
...(turn.files != null && { files: turn.files }),
...(turn.quotes != null && { quotes: turn.quotes }),
...(turn.manualSkills != null && { manualSkills: turn.manualSkills }),
priority: turn.priority,
...(turn.expectedPredecessorCreatedAt != null && {
expectedPredecessorCreatedAt: turn.expectedPredecessorCreatedAt,
}),
status: turn.status,
...(position != null && { position }),
revision: turn.sequence,
createdAt: turn.createdAt.toISOString(),
updatedAt: updatedAt.toISOString(),
...(turn.terminalReceipt?.failure != null && {
failure: {
code: turn.terminalReceipt.failure.code,
message: turn.terminalReceipt.failure.message,
},
}),
};
}
function parseClientRequestIds(raw: unknown): string[] | null {
if (raw == null) {
return [];
}
const values = Array.isArray(raw) ? raw : [raw];
if (values.length > 100 || values.some((value) => typeof value !== 'string')) {
return null;
}
const normalized = [...new Set((values as string[]).map((value) => value.trim()))];
return normalized.some((value) => value.length === 0 || value.length > 128) ? null : normalized;
}
function sameStrings(left: readonly string[] | undefined, right: readonly string[] | undefined) {
const a = left ?? [];
const b = right ?? [];
return a.length === b.length && a.every((value, index) => value === b[index]);
}
function uniqueStrings(values: readonly string[] | undefined): string[] | undefined {
return values == null ? undefined : [...new Set(values)];
}
function matchesReplayIntent(
turn: AgentQueuedTurnRecord,
input: {
parentMessageId: string;
clientRequestId: string;
files?: readonly TAgentQueuedTurnFileRef[];
manualSkills?: readonly string[];
expectedPredecessorCreatedAt?: number;
},
text: string,
quotes: readonly string[] | undefined,
): boolean {
return (
turn.parentMessageId === input.parentMessageId &&
turn.clientRequestId === input.clientRequestId &&
turn.text === text &&
sameStrings(
turn.files?.map((file) => file.file_id),
uniqueStrings(input.files?.map((file) => file.file_id)),
) &&
sameStrings(turn.quotes, quotes) &&
sameStrings(turn.manualSkills, uniqueStrings(input.manualSkills)) &&
turn.expectedPredecessorCreatedAt === input.expectedPredecessorCreatedAt
);
}
async function authorizeConversation(
user: SteerRequestUser,
conversationId: string,
deps: AgentQueuedTurnHttpDeps,
): Promise<
| { status: 200; conversation: QueuedTurnConversation & { agent_id: string } }
| AgentQueuedTurnHttpResult
> {
if (user.id == null) {
return { status: 401, body: { code: 'UNAUTHORIZED' } };
}
const conversation = await deps.methods.getConvo(user.id, conversationId);
if (conversation == null || !sameTenant(conversation.tenantId, user.tenantId)) {
return { status: 404, body: { code: 'CONVERSATION_NOT_FOUND' } };
}
const agentId = conversation.agent_id;
if (
!isAgentsEndpoint(conversation.endpoint) ||
typeof agentId !== 'string' ||
isEphemeralAgentId(agentId)
) {
return { status: 501, body: { code: 'QUEUED_TURNS_UNSUPPORTED' } };
}
if (
deps.checkAgentAccess != null &&
!(await deps.checkAgentAccess({ agentId, endpoint: conversation.endpoint }))
) {
return { status: 403, body: { code: 'FORBIDDEN' } };
}
return { status: 200, conversation: { ...conversation, agent_id: agentId } };
}
async function resolveFiles(
raw: unknown,
user: SteerRequestUser,
deps: AgentQueuedTurnHttpDeps,
): Promise<{ files?: TAgentQueuedTurnFileRef[]; error?: AgentQueuedTurnHttpResult }> {
if (raw == null) {
return {};
}
if (!Array.isArray(raw) || raw.length > MAX_QUEUED_TURN_FILES) {
return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
}
const requested = raw.map(toSteerFileRef);
if (requested.some((file) => file == null)) {
return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
}
const files = requested as Partial<TFile>[];
if (files.length === 0) {
return {};
}
if (deps.getFiles == null || deps.updateFilesUsage == null) {
return {
error: { status: 503, body: { code: 'FILE_STORAGE_UNAVAILABLE' } },
};
}
const ids = collectFileIds(files);
const filter = buildOwnerFilter(ids, user);
if (filter == null) {
return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
}
const docs = (await deps.getFiles(filter, {}, {})) ?? [];
const byId = new Map(docs.map((doc: IMongoFile) => [doc.file_id, doc]));
const resolved = ids.flatMap((id): TAgentQueuedTurnFileRef[] => {
const ref = toSteerFileRef(byId.get(id));
return ref == null ? [] : [{ ...ref, file_id: id }];
});
if (resolved.length !== ids.length) {
return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
}
const retained = await deps.updateFilesUsage(
ids.map((file_id) => ({ file_id })),
undefined,
{ user: user.id, tenantId: user.tenantId },
);
const retainedIds = new Set<string>();
for (const file of retained) {
if (
file != null &&
typeof file === 'object' &&
'file_id' in file &&
typeof file.file_id === 'string'
) {
retainedIds.add(file.file_id);
}
}
if (!ids.every((id) => retainedIds.has(id))) {
return { error: { status: 503, body: { code: 'FILE_RETENTION_FAILED' } } };
}
return { files: resolved };
}
export async function handleAgentQueuedTurnEnqueue(
user: SteerRequestUser,
body: unknown,
deps: AgentQueuedTurnHttpDeps,
): Promise<AgentQueuedTurnHttpResult> {
const parsed = enqueueAgentQueuedTurnSchema.safeParse(body);
if (!parsed.success) {
return { status: 400, body: { code: 'INVALID_QUEUED_TURN' } };
}
const input = parsed.data;
const text = input.text.replace(/\0/g, '').trim();
if (text.length === 0) {
return { status: 400, body: { code: 'EMPTY_TEXT' } };
}
if (text.length > MAX_QUEUED_TURN_LENGTH) {
return {
status: 413,
body: { code: 'QUEUED_TURN_TOO_LONG', maxLength: MAX_QUEUED_TURN_LENGTH },
};
}
/** Interrupt-and-send remains the existing warm steer path; a future lane
* arbiter can add durable front insertion without weakening FIFO here. */
if (input.priority === true) {
return { status: 501, body: { code: 'QUEUED_TURN_PRIORITY_UNSUPPORTED' } };
}
const scope = owner(user);
if (scope == null) {
return { status: 401, body: { code: 'UNAUTHORIZED' } };
}
if (deps.isPrincipalActive != null && !(await deps.isPrincipalActive(user.id!))) {
return { status: 409, body: { code: 'USER_DELETION_IN_PROGRESS' } };
}
const quotes = getReferencedQuotes(input.quotes) ?? undefined;
/** Request identity is the durable receipt address. Resolve it before
* mutable conversation/agent/file preconditions so an accepted turn remains
* observable even if those resources change before a lost-response replay. */
const existing = await deps.methods.getAgentQueuedTurnByClientRequestId({
...scope,
conversationId: input.conversationId,
clientRequestId: input.clientRequestId,
});
if (existing != null) {
if (!matchesReplayIntent(existing, input, text, quotes)) {
return { status: 409, body: { code: 'QUEUED_TURN_IDEMPOTENCY_CONFLICT' } };
}
if (existing.status !== 'queued' && existing.status !== 'claimed') {
return {
status: 200,
body: { receipt: receipt(existing), capability: CAPABILITY },
};
}
try {
await deps.lifecycle.schedule(existing);
} catch {
return { status: 503, body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' } };
}
const active = await deps.methods.listActiveAgentQueuedTurns({
...scope,
conversationId: input.conversationId,
});
const position = active.findIndex((turn) => turn.queuedTurnId === existing.queuedTurnId);
return {
status: 202,
body: {
receipt: receipt(existing, position >= 0 ? position + 1 : undefined),
capability: CAPABILITY,
},
};
}
const authorized = await authorizeConversation(user, input.conversationId, deps);
if ('body' in authorized) {
return authorized;
}
const resolvedFiles = await resolveFiles(input.files, user, deps);
if (resolvedFiles.error != null) {
return resolvedFiles.error;
}
try {
const queued = await deps.methods.enqueueAgentQueuedTurn({
...scope,
conversationId: input.conversationId,
agentId: authorized.conversation.agent_id,
parentMessageId: input.parentMessageId,
clientRequestId: input.clientRequestId,
text,
...(resolvedFiles.files != null && { files: resolvedFiles.files }),
...(quotes != null && { quotes }),
...(input.manualSkills != null && { manualSkills: input.manualSkills }),
priority: false,
...(input.expectedPredecessorCreatedAt != null && {
expectedPredecessorCreatedAt: input.expectedPredecessorCreatedAt,
}),
});
/** A same-body replay is the transport-independent receipt lookup. It can
* arrive after the original row already settled, in which case no new
* scheduling side effect is valid or necessary. */
if (queued.replayed && queued.turn.status !== 'queued' && queued.turn.status !== 'claimed') {
return {
status: 200,
body: { receipt: receipt(queued.turn), capability: CAPABILITY },
};
}
try {
await deps.lifecycle.schedule(queued.turn);
} catch {
/** The row is the outbox source of truth; periodic recovery repairs the
* record-to-delivery seam. A retry with the same clientRequestId replays. */
return { status: 503, body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' } };
}
const active = await deps.methods.listActiveAgentQueuedTurns({
...scope,
conversationId: input.conversationId,
});
const position = active.findIndex((turn) => turn.queuedTurnId === queued.turn.queuedTurnId);
return {
status: 202,
body: {
receipt: receipt(queued.turn, position >= 0 ? position + 1 : undefined),
capability: CAPABILITY,
},
};
} catch (error) {
if (error instanceof AgentQueuedTurnCapacityError) {
return { status: 429, body: { code: 'QUEUED_TURN_QUEUE_FULL' } };
}
if (error instanceof AgentQueuedTurnConflictError) {
return {
status: 409,
body: { code: 'QUEUED_TURN_IDEMPOTENCY_CONFLICT' },
};
}
if (error instanceof AgentQueuedTurnLaneRetiredError) {
return { status: 409, body: { code: 'QUEUED_TURN_CONVERSATION_DELETING' } };
}
throw error;
}
}
export async function handleAgentQueuedTurnList(
user: SteerRequestUser,
conversationId: unknown,
deps: AgentQueuedTurnHttpDeps,
rawClientRequestIds?: unknown,
): Promise<AgentQueuedTurnHttpResult> {
if (typeof conversationId !== 'string' || conversationId.length === 0) {
return { status: 400, body: { code: 'INVALID_CONVERSATION' } };
}
const scope = owner(user);
if (scope == null) {
return { status: 401, body: { code: 'UNAUTHORIZED' } };
}
const clientRequestIds = parseClientRequestIds(rawClientRequestIds);
if (clientRequestIds == null) {
return { status: 400, body: { code: 'INVALID_CLIENT_REQUEST_IDS' } };
}
const authorized = await authorizeConversation(user, conversationId, deps);
if ('body' in authorized) {
return authorized;
}
const turns = await deps.methods.listAgentQueuedTurnReceipts({
...scope,
conversationId,
clientRequestIds,
});
let activePosition = 0;
return {
status: 200,
body: {
queuedTurns: turns.map((turn) => {
if (turn.status !== 'queued' && turn.status !== 'claimed') {
return receipt(turn);
}
activePosition += 1;
return receipt(turn, activePosition);
}),
capability: CAPABILITY,
revision: turns.reduce((latest, turn) => Math.max(latest, turn.sequence), 0),
},
};
}
export async function handleAgentQueuedTurnCancel(
user: SteerRequestUser,
queuedTurnId: unknown,
deps: AgentQueuedTurnHttpDeps,
): Promise<AgentQueuedTurnHttpResult> {
if (typeof queuedTurnId !== 'string' || queuedTurnId.length === 0) {
return { status: 400, body: { code: 'INVALID_QUEUED_TURN' } };
}
const scope = owner(user);
if (scope == null) {
return { status: 401, body: { code: 'UNAUTHORIZED' } };
}
const cancelled = await deps.lifecycle.cancel({
...scope,
queuedTurnId,
settledAt: new Date(),
});
if (cancelled.outcome === 'not_found') {
return { status: 404, body: { code: 'QUEUED_TURN_NOT_FOUND' } };
}
if (cancelled.outcome === 'not_cancellable') {
return { status: 409, body: { code: 'QUEUED_TURN_ALREADY_ADMITTING' } };
}
return { status: 200, body: { receipt: receipt(cancelled.turn) } };
}

View file

@ -0,0 +1,942 @@
import { Types } from 'mongoose';
import type {
AgentQueuedTurnClaim,
AgentQueuedTurnMethods,
AgentQueuedTurnRecord,
ConversationMethods,
IMessage,
MessageMethods,
} from '@librechat/data-schemas';
import type { AgentQueuedTurnResolverDeps, AgentQueuedTurnSchedulerDeps } from './queuedTurns';
import type { AgentContinueTriggerEnvelope } from './triggers/envelope';
import { AGENT_QUEUED_TURN_SOURCE, createAgentQueuedTurnLifecycle } from './queuedTurns';
import { getAgentTriggerIdempotencyKey } from './triggers/envelope';
import { AgentTriggerExecutionError } from './triggers/host';
const NOW = Date.parse('2026-08-30T12:00:00Z');
const USER_ID = '507f191e810c19729de860ea';
function envelope(): AgentContinueTriggerEnvelope {
return {
version: 1,
mode: 'continue',
requestId: 'request-1',
deliveryId: 'queued-turn-1',
receivedAt: NOW,
principal: { userId: USER_ID, tenantId: 'tenant-1' },
event: {
id: 'queued-turn-1',
type: 'agent.queued-turn',
occurredAt: NOW,
source: { id: AGENT_QUEUED_TURN_SOURCE, type: 'internal' },
payload: { queuedTurnId: 'queued-turn-1' },
},
target: {
agentId: 'agent-1',
conversationId: 'conversation-1',
parentMessageId: 'assistant-1',
},
input: 'queued words',
};
}
function claim(): AgentQueuedTurnClaim {
return {
queuedTurnId: 'queued-turn-1',
user: new Types.ObjectId(USER_ID),
tenantId: 'tenant-1',
conversationId: 'conversation-1',
agentId: 'agent-1',
parentMessageId: 'assistant-1',
clientRequestId: 'client-1',
fingerprint: 'fingerprint-1',
sequence: 1,
status: 'claimed',
priority: false,
text: 'queued words',
files: [{ file_id: 'file-1' }],
quotes: ['quote'],
manualSkills: ['skill-1'],
attempts: 1,
availableAt: new Date(NOW),
claimId: 'trigger-1',
claimBy: 'worker-1',
claimUntil: new Date(NOW + 60_000),
createdAt: new Date(NOW - 1_000),
};
}
function persistedMessage(
value: Pick<IMessage, 'messageId' | 'parentMessageId' | 'isCreatedByUser'> &
Partial<Pick<IMessage, 'createdAt' | 'unfinished' | 'error'>>,
): IMessage {
return value as unknown as IMessage;
}
function resolverMethods() {
const turn = claim();
const methods = {
getConvo: jest.fn(async () => ({ tenantId: 'tenant-1', agent_id: 'agent-1' })),
getMessages: jest.fn(
async (..._args: Parameters<MessageMethods['getMessages']>): Promise<IMessage[]> => [
persistedMessage({
messageId: 'assistant-1',
parentMessageId: 'user-1',
isCreatedByUser: false,
createdAt: new Date(NOW - 500),
}),
persistedMessage({
messageId: 'assistant-2',
parentMessageId: 'assistant-1',
isCreatedByUser: false,
createdAt: new Date(NOW),
}),
],
),
claimNextAgentQueuedTurn: jest.fn(
async (..._args: Parameters<AgentQueuedTurnMethods['claimNextAgentQueuedTurn']>) => ({
outcome: 'acquired' as const,
claim: turn,
}),
),
releaseAgentQueuedTurn: jest.fn(async () => ({ outcome: 'released' as const, turn })),
beginAgentQueuedTurnAdmission: jest.fn(async () => ({
outcome: 'started' as const,
turn,
})),
getEffectiveAgentQueuedTurnPredecessor: jest.fn(
async (
..._args: Parameters<AgentQueuedTurnMethods['getEffectiveAgentQueuedTurnPredecessor']>
): Promise<number | undefined> => undefined,
),
markAgentQueuedTurnAdmitted: jest.fn(async () => ({
outcome: 'admitted' as const,
turn: { ...turn, status: 'admitted' as const },
})),
hasAgentQueuedTurnAdmissionReceipt: jest.fn(async () => true),
};
return {
methods: methods as unknown as AgentQueuedTurnMethods &
Pick<ConversationMethods, 'getConvo'> &
Pick<MessageMethods, 'getMessages'>,
spies: methods,
};
}
function createAgentQueuedTurnResolver(deps: AgentQueuedTurnResolverDeps) {
return createAgentQueuedTurnLifecycle({
...deps,
getGenerationAdmissionEvidence: async () => null,
enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
}).prepareContinue;
}
function createAgentQueuedTurnDeadLetterSettlement({
methods,
getGenerationAdmissionEvidence = async () => null,
now,
}: {
methods: Pick<AgentQueuedTurnMethods, 'deadLetterAgentQueuedTurn'>;
getGenerationAdmissionEvidence?: AgentQueuedTurnSchedulerDeps['getGenerationAdmissionEvidence'];
now?: () => number;
}) {
return createAgentQueuedTurnLifecycle({
methods: methods as AgentQueuedTurnSchedulerDeps['methods'] &
AgentQueuedTurnResolverDeps['methods'],
getGenerationJob: async () => null,
getGenerationAdmissionEvidence,
enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
...(now != null && { now }),
}).settleBeforeDeadLetter;
}
function createAgentQueuedTurnScheduler(deps: AgentQueuedTurnSchedulerDeps) {
return createAgentQueuedTurnLifecycle({
...deps,
methods: deps.methods as AgentQueuedTurnSchedulerDeps['methods'] &
AgentQueuedTurnResolverDeps['methods'],
getGenerationJob: async () => null,
});
}
describe('Agent queued-turn continuation', () => {
it('dead-letters a delivery while preserving an admission-indeterminate source', async () => {
const deadLetterAgentQueuedTurn = jest.fn(async () => ({
outcome: 'admission_indeterminate' as const,
turn: claim(),
}));
const settle = createAgentQueuedTurnDeadLetterSettlement({
methods: { deadLetterAgentQueuedTurn },
now: () => NOW,
});
await expect(
settle(envelope(), {
code: 'ATTEMPTS_EXHAUSTED',
message: 'admission receipt unavailable',
certainty: 'ambiguous',
retryable: true,
attemptedAt: new Date(NOW),
}),
).resolves.toBeUndefined();
expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
deliveryKey: getAgentTriggerIdempotencyKey(envelope()),
}),
);
});
it('passes exact live-generation evidence into dead-letter reconciliation', async () => {
const deliveryKey = getAgentTriggerIdempotencyKey(envelope());
const deadLetterAgentQueuedTurn = jest.fn(async () => ({
outcome: 'admission_reconciled' as const,
turn: { ...claim(), status: 'admitted' as const },
}));
const settle = createAgentQueuedTurnDeadLetterSettlement({
methods: { deadLetterAgentQueuedTurn },
getGenerationAdmissionEvidence: async () => ({
generationId: 'generation-1',
generationCreatedAt: NOW + 1,
}),
now: () => NOW,
});
await settle(envelope(), {
code: 'ATTEMPTS_EXHAUSTED',
message: 'admission receipt unavailable',
certainty: 'ambiguous',
retryable: true,
attemptedAt: new Date(NOW),
});
expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
deliveryKey,
admissionEvidence: {
generationId: 'generation-1',
generationCreatedAt: NOW + 1,
},
}),
);
});
it('defers without claiming while the predecessor generation remains active', async () => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => ({ status: 'running' }),
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
code: 'PARENT_NOT_READY',
retryable: true,
deferWithoutAttempt: true,
});
expect(spies.claimNextAgentQueuedTurn).not.toHaveBeenCalled();
});
it('preserves queued context and settles only after fresh-turn admission', async () => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
expect(prepared).toMatchObject({
status: 'ready',
input: 'queued words',
parentMessageId: 'assistant-2',
files: [{ file_id: 'file-1' }],
quotes: ['quote'],
manualSkills: ['skill-1'],
admissionSource: {
source: AGENT_QUEUED_TURN_SOURCE,
sourceId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
},
});
if (prepared?.status !== 'ready') {
throw new Error('Expected a ready queued turn');
}
expect(spies.markAgentQueuedTurnAdmitted).not.toHaveBeenCalled();
expect(spies.beginAgentQueuedTurnAdmission).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
admissionId: 'trigger-1',
admissionProtocolVersion: 2,
}),
);
await prepared.settleOnAdmission?.({
mode: 'continue',
status: 'started',
conversationId: 'conversation-1',
streamId: 'stream-1',
generationCreatedAt: NOW + 1,
});
expect(spies.markAgentQueuedTurnAdmitted).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
admissionId: 'trigger-1',
admissionMode: 'ordinary',
generationId: 'stream-1',
generationCreatedAt: NOW + 1,
}),
);
});
it('records execution admission through the source-owned lifecycle seam', async () => {
const { methods, spies } = resolverMethods();
const lifecycle = createAgentQueuedTurnLifecycle({
methods,
getGenerationJob: async () => null,
getGenerationAdmissionEvidence: async () => null,
enqueue: async () => ({ deliveryKey: 'queued-delivery-1' }),
});
await expect(
lifecycle.recordExecutionAdmission(
{
source: AGENT_QUEUED_TURN_SOURCE,
sourceId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
},
{
userId: USER_ID,
tenantId: 'tenant-1',
conversationId: 'conversation-1',
clientRequestId: 'trigger-1',
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
},
),
).resolves.toBe(true);
expect(spies.markAgentQueuedTurnAdmitted).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
admissionId: 'trigger-1',
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
}),
);
});
it('requires the exact durable source receipt before deduplicated success', async () => {
const { methods, spies } = resolverMethods();
const lifecycle = createAgentQueuedTurnLifecycle({
methods,
getGenerationJob: async () => null,
getGenerationAdmissionEvidence: async () => null,
enqueue: async () => ({ deliveryKey: 'queued-delivery-1' }),
});
const source = {
source: AGENT_QUEUED_TURN_SOURCE,
sourceId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
};
const admission = {
userId: USER_ID,
tenantId: 'tenant-1',
conversationId: 'conversation-1',
clientRequestId: 'trigger-1',
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
};
await expect(lifecycle.verifyExecutionAdmission(source, admission)).resolves.toBe(true);
expect(spies.hasAgentQueuedTurnAdmissionReceipt).toHaveBeenCalledWith({
user: new Types.ObjectId(USER_ID),
tenantId: 'tenant-1',
conversationId: 'conversation-1',
queuedTurnId: 'queued-turn-1',
admissionId: 'trigger-1',
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
});
spies.hasAgentQueuedTurnAdmissionReceipt.mockResolvedValueOnce(false);
await expect(lifecycle.verifyExecutionAdmission(source, admission)).rejects.toThrow(
'not yet confirmed',
);
});
it('retires cancellation through the same lifecycle interface', async () => {
const { methods: resolver } = resolverMethods();
const cancelled = {
...claim(),
status: 'cancelled' as const,
deliveryKey: 'delivery-cancelled',
terminalReceipt: {
outcome: 'cancelled' as const,
settledAt: new Date(NOW),
},
};
const cancelAgentQueuedTurn = jest.fn(async () => ({
outcome: 'cancelled' as const,
turn: cancelled,
}));
const markAgentQueuedTurnDeliveryRetired = jest.fn(async () => true);
const retireDelivery = jest.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true);
const lifecycle = createAgentQueuedTurnLifecycle({
methods: {
...resolver,
cancelAgentQueuedTurn,
markAgentQueuedTurnDeliveryRetired,
},
getGenerationJob: async () => null,
getGenerationAdmissionEvidence: async () => null,
enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
retireDelivery,
});
await expect(
lifecycle.cancel({
user: new Types.ObjectId(USER_ID),
tenantId: 'tenant-1',
queuedTurnId: 'queued-turn-1',
}),
).resolves.toMatchObject({ outcome: 'cancelled' });
expect(retireDelivery).toHaveBeenNthCalledWith(
1,
'delivery-cancelled',
AGENT_QUEUED_TURN_SOURCE,
'queued_turn_cancelled',
);
expect(retireDelivery).toHaveBeenNthCalledWith(
2,
'delivery-cancelled',
AGENT_QUEUED_TURN_SOURCE,
'queued_turn_cancelled',
{ onlyIfDead: true },
);
expect(markAgentQueuedTurnDeliveryRetired).toHaveBeenCalledWith({
deliveryKey: 'delivery-cancelled',
});
});
it('settles an obsolete-lane delivery without crossing ordinary admission', async () => {
const { methods, spies } = resolverMethods();
(
spies.beginAgentQueuedTurnAdmission as unknown as jest.MockedFunction<
AgentQueuedTurnMethods['beginAgentQueuedTurnAdmission']
>
).mockResolvedValueOnce({
outcome: 'retired',
turn: { ...claim(), status: 'cancelled' },
});
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
status: 'settled',
});
expect(spies.markAgentQueuedTurnAdmitted).not.toHaveBeenCalled();
});
it('uses the latest admitted queued generation as the effective predecessor epoch', async () => {
const { methods, spies } = resolverMethods();
const claimed = claim();
claimed.expectedPredecessorCreatedAt = NOW;
spies.claimNextAgentQueuedTurn.mockResolvedValueOnce({ outcome: 'acquired', claim: claimed });
spies.getEffectiveAgentQueuedTurnPredecessor.mockResolvedValueOnce(NOW + 250);
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toMatchObject({
status: 'ready',
expectedPredecessorCreatedAt: NOW + 250,
});
expect(spies.getEffectiveAgentQueuedTurnPredecessor).toHaveBeenCalledWith({
user: new Types.ObjectId(USER_ID),
tenantId: 'tenant-1',
conversationId: 'conversation-1',
sequence: 1,
expectedPredecessorCreatedAt: NOW,
});
});
it.each([
['aborted', 'PREDECESSOR_ABORTED'],
['error', 'PREDECESSOR_FAILED'],
] as const)('dead-letters a queued turn whose predecessor is %s', async (status, code) => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => ({ status }),
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
status: 'settled',
});
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
disposition: 'dead',
failure: expect.objectContaining({ code }),
}),
);
expect(spies.getMessages).toHaveBeenCalledTimes(1);
});
it.each([
[{ unfinished: true }, 'PREDECESSOR_ABORTED'],
[{ error: true }, 'PREDECESSOR_FAILED'],
] as const)(
'uses durable response history after the transient predecessor job is cleaned up',
async (terminal, code) => {
const { methods, spies } = resolverMethods();
spies.getMessages.mockResolvedValueOnce([
persistedMessage({
messageId: 'assistant-1',
parentMessageId: 'user-1',
isCreatedByUser: false,
createdAt: new Date(NOW),
...terminal,
}),
]);
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
status: 'settled',
});
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
disposition: 'dead',
failure: expect.objectContaining({ code }),
}),
);
},
);
it('allows an explicitly prioritized interrupt successor after its predecessor is unfinished', async () => {
const { methods, spies } = resolverMethods();
const prioritized = claim();
prioritized.priority = true;
spies.claimNextAgentQueuedTurn.mockResolvedValueOnce({
outcome: 'acquired',
claim: prioritized,
});
spies.getMessages.mockResolvedValueOnce([
persistedMessage({
messageId: 'assistant-1',
parentMessageId: 'user-1',
isCreatedByUser: false,
createdAt: new Date(NOW),
unfinished: true,
}),
]);
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toMatchObject({
status: 'ready',
parentMessageId: 'assistant-1',
});
});
it('dead-letters the queue row when admission is definitely rejected', async () => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
if (prepared?.status !== 'ready') {
throw new Error('Expected a ready queued turn');
}
await prepared.releaseOnDefiniteFailure?.(
new AgentTriggerExecutionError('forbidden', {
mode: 'continue',
certainty: 'definite',
retryable: false,
status: 403,
code: 'FORBIDDEN',
}),
);
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
disposition: 'dead',
failure: { code: 'FORBIDDEN', message: 'forbidden' },
}),
);
});
it('releases a claim when admission defers with PARENT_NOT_READY', async () => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
if (prepared?.status !== 'ready') {
throw new Error('Expected a ready queued turn');
}
await prepared.releaseOnDefiniteFailure?.(
new AgentTriggerExecutionError('parent raced', {
mode: 'continue',
certainty: 'definite',
retryable: true,
deferWithoutAttempt: true,
status: 409,
code: 'PARENT_NOT_READY',
}),
);
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
disposition: 'retry',
}),
);
});
it('dead-letters a retryable admission rejection on the final delivery attempt', async () => {
const { methods, spies } = resolverMethods();
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
const prepared = await resolve(envelope(), {
idempotencyKey: 'trigger-1',
attempt: 3,
maxAttempts: 3,
});
if (prepared?.status !== 'ready') {
throw new Error('Expected a ready queued turn');
}
await prepared.releaseOnDefiniteFailure?.(
new AgentTriggerExecutionError('still busy', {
mode: 'continue',
certainty: 'definite',
retryable: true,
status: 503,
code: 'ADMISSION_BUSY',
}),
);
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
disposition: 'dead',
failure: { code: 'ADMISSION_BUSY', message: 'still busy' },
}),
);
});
it('releases an acquired claim when branch preparation is temporarily unavailable', async () => {
const { methods, spies } = resolverMethods();
spies.getMessages.mockRejectedValueOnce(new Error('read unavailable'));
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
code: 'QUEUED_TURN_PREPARATION_UNAVAILABLE',
retryable: true,
deferWithoutAttempt: true,
});
expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-1',
claimId: 'trigger-1',
claimBy: 'worker-1',
disposition: 'retry',
}),
);
});
it('keeps a failed preparation release retryable under the same claim fence', async () => {
const { methods, spies } = resolverMethods();
spies.getMessages.mockRejectedValueOnce(new Error('read unavailable'));
spies.releaseAgentQueuedTurn.mockRejectedValueOnce(new Error('write unavailable'));
const resolve = createAgentQueuedTurnResolver({
methods,
getGenerationJob: async () => null,
now: () => NOW,
claimBy: 'worker-1',
});
await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
code: 'QUEUED_TURN_PREPARATION_RELEASE_FAILED',
retryable: true,
deferWithoutAttempt: true,
});
});
it('uses a replica-unique default claim owner that remains stable for the process', async () => {
const first = resolverMethods();
const second = resolverMethods();
const firstResolve = createAgentQueuedTurnResolver({
methods: first.methods,
getGenerationJob: async () => null,
now: () => NOW,
});
const secondResolve = createAgentQueuedTurnResolver({
methods: second.methods,
getGenerationJob: async () => null,
now: () => NOW,
});
await firstResolve(envelope(), { idempotencyKey: 'trigger-1' });
await secondResolve(envelope(), { idempotencyKey: 'trigger-1' });
const firstOwner = first.spies.claimNextAgentQueuedTurn.mock.calls[0][0].claimBy;
const secondOwner = second.spies.claimNextAgentQueuedTurn.mock.calls[0][0].claimBy;
expect(firstOwner).toMatch(/^agent-queued-turn:\d+:[0-9a-f-]{36}$/);
expect(secondOwner).toBe(firstOwner);
});
});
describe('Agent queued-turn delivery scheduling', () => {
function queuedTurn(id: string, sequence: number): AgentQueuedTurnRecord {
const { claimId: _claimId, claimBy: _claimBy, claimUntil: _claimUntil, ...record } = claim();
return {
...record,
queuedTurnId: id,
sequence,
status: 'queued',
};
}
it('uses independent delivery lanes so publication order cannot invert queue order', async () => {
const enqueue = jest.fn(async (value: unknown) => ({
deliveryKey: getAgentTriggerIdempotencyKey(value as AgentContinueTriggerEnvelope),
}));
const reserveAgentQueuedTurnDelivery = jest.fn(async (input) => ({
outcome: 'reserved' as const,
turn: { ...queuedTurn(input.queuedTurnId, 1), deliveryKey: input.deliveryKey },
}));
const markQueuedTurnScheduled = jest.fn(async (input) => ({
outcome: 'scheduled' as const,
turn: queuedTurn(input.queuedTurnId, 1),
}));
const scheduler = createAgentQueuedTurnScheduler({
methods: {
reserveAgentQueuedTurnDelivery,
markQueuedTurnScheduled,
} as unknown as AgentQueuedTurnMethods,
enqueue,
getGenerationAdmissionEvidence: async () => null,
});
await scheduler.schedule(queuedTurn('queued-turn-2', 2));
await scheduler.schedule(queuedTurn('queued-turn-1', 1));
expect(enqueue).toHaveBeenNthCalledWith(
1,
expect.anything(),
expect.objectContaining({ orderingKey: 'agent-queued-turn-delivery:queued-turn-2' }),
);
expect(enqueue).toHaveBeenNthCalledWith(
2,
expect.anything(),
expect.objectContaining({ orderingKey: 'agent-queued-turn-delivery:queued-turn-1' }),
);
expect(reserveAgentQueuedTurnDelivery).toHaveBeenCalledTimes(2);
expect(reserveAgentQueuedTurnDelivery.mock.invocationCallOrder[0]).toBeLessThan(
enqueue.mock.invocationCallOrder[0],
);
expect(reserveAgentQueuedTurnDelivery.mock.calls[0]?.[0].deliveryKey).toBe(
getAgentTriggerIdempotencyKey(enqueue.mock.calls[0]?.[0] as AgentContinueTriggerEnvelope),
);
});
it('reconciles quarantined admission receipts during durable recovery', async () => {
const turn = {
...queuedTurn('queued-turn-indeterminate', 1),
status: 'dead' as const,
deliveryKey: 'delivery-indeterminate',
deliveryState: 'published' as const,
admissionId: 'delivery-indeterminate',
admissionStartedAt: new Date(NOW - 1_000),
reconciliationClaimId: 'reconciliation-claim',
reconciliationClaimBy: 'reconciler-1',
terminalReceipt: {
outcome: 'dead' as const,
settledAt: new Date(NOW),
failure: {
code: 'ADMISSION_INDETERMINATE',
message: 'The queued turn may have been admitted',
},
},
};
const deadLetterAgentQueuedTurn = jest.fn(async () => ({
outcome: 'admission_reconciled' as const,
turn: { ...turn, status: 'admitted' as const },
}));
const getGenerationAdmissionEvidence = jest.fn(async () => ({
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
}));
const scheduler = createAgentQueuedTurnScheduler({
methods: {
findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
{
...turn,
reconciliationClaimId: input.claimId,
reconciliationClaimBy: input.claimBy,
},
]),
deadLetterAgentQueuedTurn,
} as unknown as AgentQueuedTurnMethods,
enqueue: jest.fn(),
getGenerationAdmissionEvidence,
});
await expect(scheduler.recover()).resolves.toBe(1);
expect(getGenerationAdmissionEvidence).toHaveBeenCalledWith(
USER_ID,
'delivery-indeterminate',
'conversation-1',
'conversation-1',
);
expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-indeterminate',
deliveryKey: 'delivery-indeterminate',
admissionEvidence: {
generationId: 'conversation-1',
generationCreatedAt: NOW + 1,
},
}),
);
});
it('rotates source-fenced ambiguity without trusting transient job-store evidence', async () => {
const turn = {
...queuedTurn('queued-turn-source-fenced', 1),
status: 'dead' as const,
deliveryKey: 'delivery-source-fenced',
deliveryState: 'published' as const,
admissionId: 'delivery-source-fenced',
admissionStartedAt: new Date(NOW - 1_000),
admissionProtocolVersion: 2 as const,
terminalReceipt: {
outcome: 'dead' as const,
settledAt: new Date(NOW),
failure: {
code: 'ADMISSION_INDETERMINATE',
message: 'The queued turn may have been admitted',
},
},
};
const deferAgentQueuedTurnAdmissionReconciliation = jest.fn(async () => true);
const getGenerationAdmissionEvidence = jest.fn(async () => ({
generationId: 'stale-generation',
generationCreatedAt: NOW,
}));
const scheduler = createAgentQueuedTurnScheduler({
methods: {
findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
{
...turn,
reconciliationClaimId: input.claimId,
reconciliationClaimBy: input.claimBy,
},
]),
deferAgentQueuedTurnAdmissionReconciliation,
} as unknown as AgentQueuedTurnMethods,
enqueue: jest.fn(),
getGenerationAdmissionEvidence,
});
await expect(scheduler.recover()).resolves.toBe(0);
expect(getGenerationAdmissionEvidence).not.toHaveBeenCalled();
expect(deferAgentQueuedTurnAdmissionReconciliation).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-source-fenced',
deliveryKey: 'delivery-source-fenced',
}),
);
});
it('quarantines an expired provider-admission owner before any replay', async () => {
const turn = {
...queuedTurn('queued-turn-owner-lost', 1),
status: 'claimed' as const,
claimId: 'delivery-owner-lost',
claimBy: 'worker-gone',
claimUntil: new Date(NOW - 1),
deliveryKey: 'delivery-owner-lost',
deliveryState: 'published' as const,
admissionId: 'delivery-owner-lost',
admissionStartedAt: new Date(NOW - 1_000),
admissionProtocolVersion: 2 as const,
};
const deadLetterAgentQueuedTurn = jest.fn(async () => ({
outcome: 'admission_indeterminate' as const,
turn: {
...turn,
status: 'dead' as const,
terminalReceipt: {
outcome: 'dead' as const,
settledAt: new Date(NOW),
failure: { code: 'ADMISSION_INDETERMINATE', message: 'owner disappeared' },
},
},
}));
const getGenerationAdmissionEvidence = jest.fn(async () => null);
const scheduler = createAgentQueuedTurnScheduler({
methods: {
findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
{
...turn,
reconciliationClaimId: input.claimId,
reconciliationClaimBy: input.claimBy,
},
]),
deadLetterAgentQueuedTurn,
} as unknown as AgentQueuedTurnMethods,
enqueue: jest.fn(),
getGenerationAdmissionEvidence,
});
await expect(scheduler.recover()).resolves.toBe(1);
expect(getGenerationAdmissionEvidence).not.toHaveBeenCalled();
expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
expect.objectContaining({
queuedTurnId: 'queued-turn-owner-lost',
deliveryKey: 'delivery-owner-lost',
failure: expect.objectContaining({ code: 'ADMISSION_INDETERMINATE' }),
}),
);
});
});

File diff suppressed because it is too large Load diff

View file

@ -8,6 +8,9 @@ import { getAgentTriggerIdempotencyKey, parseAgentTriggerEnvelope } from './enve
export interface AgentTriggerDispatchContext {
idempotencyKey: string;
/** Durable delivery attempt metadata, when dispatched by the queue engine. */
attempt?: number;
maxAttempts?: number;
signal?: AbortSignal;
}
@ -41,7 +44,7 @@ export interface AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerR
export function dispatchAgentTrigger<FireResult, ContinueResult, SteerResult>(
envelope: unknown,
handlers: AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerResult>,
options?: { signal?: AbortSignal },
options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
): Promise<ContinueResult | FireResult | SteerResult> {
let normalized: AgentTriggerEnvelope;
try {
@ -51,6 +54,8 @@ export function dispatchAgentTrigger<FireResult, ContinueResult, SteerResult>(
}
const context: AgentTriggerDispatchContext = {
idempotencyKey: getAgentTriggerIdempotencyKey(normalized),
...(options?.attempt != null && { attempt: options.attempt }),
...(options?.maxAttempts != null && { maxAttempts: options.maxAttempts }),
...(options?.signal != null && { signal: options.signal }),
};
if (normalized.mode === 'fire') {

View file

@ -1,4 +1,8 @@
import type { AgentTriggerDeliveryRecord, AgentTriggerDeliveryStore } from './engine';
import type {
AgentTriggerDeliveryFailure,
AgentTriggerDeliveryRecord,
AgentTriggerDeliveryStore,
} from './engine';
import type { AgentTriggerExecutionResult } from './host';
import { AgentTriggerDeliveryDeferredError, createAgentTriggerDeliveryEngine } from './engine';
import { createAgentTriggerEnvelope } from './envelope';
@ -67,7 +71,10 @@ describe('createAgentTriggerDeliveryEngine', () => {
await expect(engine.runTick()).resolves.toBe(1);
expect(dispatch).toHaveBeenCalledWith({ version: 1 }, { signal: expect.any(AbortSignal) });
expect(dispatch).toHaveBeenCalledWith(
{ version: 1 },
{ signal: expect.any(AbortSignal), attempt: 1, maxAttempts: 8 },
);
expect(store.beginAttempt).toHaveBeenCalledWith({
id: 'delivery-row-1',
workerId: 'worker-1',
@ -463,11 +470,21 @@ describe('createAgentTriggerDeliveryEngine', () => {
});
it('dead-letters invalid envelopes without retrying', async () => {
const store = storeWith();
const transitions: string[] = [];
const store = storeWith({
dead: jest.fn(async () => {
transitions.push('delivery-dead');
return true;
}),
});
const settleSourceBeforeDeadLetter = jest.fn(async () => {
transitions.push('source-dead');
});
const engine = createAgentTriggerDeliveryEngine(
{
store,
dispatch: async () => Promise.reject(new AgentTriggerDispatchError('invalid envelope')),
settleSourceBeforeDeadLetter,
now: () => START,
},
{ concurrency: 1 },
@ -485,9 +502,76 @@ describe('createAgentTriggerDeliveryEngine', () => {
}),
}),
);
expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
{ version: 1 },
expect.objectContaining({ code: 'INVALID_ENVELOPE' }),
);
expect(transitions).toEqual(['source-dead', 'delivery-dead']);
expect(store.retry).not.toHaveBeenCalled();
});
it('bounds one terminal failure before source and delivery settlement', async () => {
const oversized = 'x'.repeat(3_000);
const store = storeWith();
const settleSourceBeforeDeadLetter = jest.fn(
async (_envelope: unknown, _failure: AgentTriggerDeliveryFailure) => undefined,
);
const engine = createAgentTriggerDeliveryEngine(
{
store,
dispatch: async () =>
Promise.reject(
new AgentTriggerExecutionError(oversized, {
mode: 'fire',
certainty: 'definite',
retryable: false,
code: oversized,
}),
),
settleSourceBeforeDeadLetter,
now: () => START,
},
{ concurrency: 1 },
);
await engine.runTick();
const sourceFailure = settleSourceBeforeDeadLetter.mock.calls[0]?.[1];
const deliveryFailure = (store.dead as jest.Mock).mock.calls[0]?.[0]?.error;
expect(sourceFailure).toMatchObject({
code: 'x'.repeat(128),
message: 'x'.repeat(2_048),
});
expect(deliveryFailure).toEqual(sourceFailure);
});
it('releases the delivery when source terminalization fails before dead-lettering', async () => {
const store = storeWith();
const settleSourceBeforeDeadLetter = jest.fn(async () => {
throw new Error('source write unavailable');
});
const engine = createAgentTriggerDeliveryEngine(
{
store,
dispatch: async () => Promise.reject(new AgentTriggerDispatchError('invalid envelope')),
settleSourceBeforeDeadLetter,
now: () => START,
workerId: 'worker-1',
},
{ concurrency: 1 },
);
await engine.runTick();
expect(store.dead).not.toHaveBeenCalled();
expect(store.release).toHaveBeenCalledWith({
id: 'delivery-row-1',
workerId: 'worker-1',
claimToken: 'claim-1',
availableAt: START,
});
});
it('dead-letters an exhausted row without dispatching again', async () => {
const store = storeWith({
claimNext: jest.fn(async () => delivery({ attempts: 8 })),
@ -505,6 +589,77 @@ describe('createAgentTriggerDeliveryEngine', () => {
expect(store.dead).toHaveBeenCalledWith(expect.objectContaining({ claimToken: 'claim-1' }));
});
it('bounds a persisted last failure before exhausting its source', async () => {
const oversized = 'x'.repeat(3_000);
const store = storeWith({
claimNext: jest.fn(async () =>
delivery({
attempts: 8,
lastError: {
code: oversized,
message: oversized,
certainty: 'ambiguous',
retryable: true,
attemptedAt: START,
},
}),
),
});
const settleSourceBeforeDeadLetter = jest.fn(async () => undefined);
const engine = createAgentTriggerDeliveryEngine(
{ store, dispatch: jest.fn(), settleSourceBeforeDeadLetter, now: () => START },
{ concurrency: 1, maxAttempts: 8 },
);
await engine.runTick();
expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
{ version: 1 },
expect.objectContaining({ code: 'x'.repeat(128), message: 'x'.repeat(2_048) }),
);
expect(store.dead).toHaveBeenCalledWith(
expect.objectContaining({
error: expect.objectContaining({
code: 'x'.repeat(128),
message: 'x'.repeat(2_048),
}),
}),
);
});
it('does not dead-letter an exhausted row until its source is terminal', async () => {
const store = storeWith({
claimNext: jest.fn(async () => delivery({ attempts: 8 })),
});
const settleSourceBeforeDeadLetter = jest.fn(async () => {
throw new Error('source write unavailable');
});
const engine = createAgentTriggerDeliveryEngine(
{
store,
dispatch: jest.fn(async () => successResult()),
settleSourceBeforeDeadLetter,
now: () => START,
workerId: 'worker-1',
},
{ concurrency: 1, maxAttempts: 8 },
);
await engine.runTick();
expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
{ version: 1 },
expect.objectContaining({ retryable: true }),
);
expect(store.dead).not.toHaveBeenCalled();
expect(store.release).toHaveBeenCalledWith({
id: 'delivery-row-1',
workerId: 'worker-1',
claimToken: 'claim-1',
availableAt: START,
});
});
it('rechecks a leased predecessor promptly instead of waiting for its full lease', async () => {
const store = storeWith({
findEarlierUnsettled: jest.fn(async () => ({

View file

@ -16,6 +16,8 @@ const ORDERING_RECHECK_MS = 250;
const ACTIVE_HANDLING_RECHECK_MS = 5_000;
const DEFAULT_DEFER_MS = 5_000;
const MAX_RETRY_AFTER_MS = 24 * 60 * 60_000;
const MAX_FAILURE_CODE_LENGTH = 128;
const MAX_FAILURE_MESSAGE_LENGTH = 2048;
function startedHandling(
delivery: Pick<AgentTriggerDeliveryRecord, 'envelope'>,
@ -200,8 +202,14 @@ export interface AgentTriggerDeliveryEngineDeps {
store: AgentTriggerDeliveryStore;
dispatch: (
envelope: unknown,
options?: { signal?: AbortSignal },
options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
) => Promise<AgentTriggerExecutionResult>;
/** Source-owned terminalization must commit before its delivery can become
* dead, including recovery after a crash that exhausted the attempt budget. */
settleSourceBeforeDeadLetter?: (
envelope: unknown,
failure: AgentTriggerDeliveryFailure,
) => Promise<void>;
now?: () => Date;
random?: () => number;
workerId?: string;
@ -261,6 +269,19 @@ function failure(error: unknown, attemptedAt: Date): AgentTriggerDeliveryFailure
};
}
function normalizeFailure(failure: AgentTriggerDeliveryFailure): AgentTriggerDeliveryFailure {
const code = failure.code.trim();
const message = failure.message.trim();
return {
...failure,
code: (code.length === 0 ? 'DELIVERY_FAILED' : code).slice(0, MAX_FAILURE_CODE_LENGTH),
message: (message.length === 0 ? 'Agent trigger delivery failed' : message).slice(
0,
MAX_FAILURE_MESSAGE_LENGTH,
),
};
}
function retryAt(
error: unknown,
attempt: number,
@ -362,9 +383,25 @@ export function createAgentTriggerDeliveryEngine(
}
if (delivery.attempts >= maxAttempts) {
const recorded =
const recorded = normalizeFailure(
delivery.lastError ??
failure(new Error('Delivery attempt limit was already exhausted'), now());
failure(new Error('Delivery attempt limit was already exhausted'), now()),
);
try {
await deps.settleSourceBeforeDeadLetter?.(delivery.envelope, recorded);
} catch (error) {
logger.error('[agent-triggers] source terminalization failed before dead-lettering', {
deliveryKey: delivery.deliveryKey,
error: error instanceof Error ? error.message : String(error),
});
await deps.store.release({
id: delivery.id,
workerId,
claimToken: delivery.claimToken,
availableAt: now(),
});
return;
}
const deadLettered = await deps.store.dead({
id: delivery.id,
workerId,
@ -426,7 +463,11 @@ export function createAgentTriggerDeliveryEngine(
members.length === 0
? delivery.envelope
: createAgentTriggerBatchEnvelope(delivery, members);
result = await deps.dispatch(dispatchEnvelope, { signal: controller.signal });
result = await deps.dispatch(dispatchEnvelope, {
signal: controller.signal,
attempt,
maxAttempts,
});
} catch (error) {
const attemptedAt = now();
const deletionCancelled = controller.signal.aborted && cancelledUsers.has(userId);
@ -464,8 +505,26 @@ export function createAgentTriggerDeliveryEngine(
}
return;
}
const recorded = failure(error, attemptedAt);
const recorded = normalizeFailure(failure(error, attemptedAt));
if (!recorded.retryable || attempt >= maxAttempts) {
try {
await deps.settleSourceBeforeDeadLetter?.(delivery.envelope, recorded);
} catch (settlementError) {
logger.error('[agent-triggers] source terminalization failed before dead-lettering', {
deliveryKey: delivery.deliveryKey,
error:
settlementError instanceof Error
? settlementError.message
: String(settlementError),
});
await deps.store.release({
id: delivery.id,
workerId,
claimToken: delivery.claimToken,
availableAt: attemptedAt,
});
return;
}
const deadLettered = await deps.store.dead({
id: delivery.id,
workerId,

View file

@ -216,8 +216,12 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
const bodies = fetcher.mock.calls.map(([, init]) => JSON.parse(String(init?.body)) as unknown);
expect(bodies).toEqual([
expect.objectContaining({ clientRequestId: getAgentTriggerIdempotencyKey(envelope) }),
expect.objectContaining({ clientRequestId: getAgentTriggerIdempotencyKey(envelope) }),
expect.objectContaining({
clientRequestId: getAgentTriggerIdempotencyKey(envelope),
}),
expect.objectContaining({
clientRequestId: getAgentTriggerIdempotencyKey(envelope),
}),
]);
});
@ -419,7 +423,10 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
resolveToken('signed-token');
resolveTimezone('UTC');
await expect(pending).resolves.toMatchObject({ mode: 'fire', status: 'started' });
await expect(pending).resolves.toMatchObject({
mode: 'fire',
status: 'started',
});
});
it('observes caller cancellation while asynchronous setup is pending', async () => {
@ -429,7 +436,9 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
deps(fetcher, { mintToken: () => new Promise<string>(() => undefined) }),
);
const controller = new AbortController();
const pending = host.dispatch(createFireEnvelope(), { signal: controller.signal });
const pending = host.dispatch(createFireEnvelope(), {
signal: controller.signal,
});
controller.abort();
await pending.catch((error: unknown) => {
@ -579,6 +588,97 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
});
});
it('carries a prepared queued-turn payload and settles it after admission', async () => {
const envelope = createContinueEnvelope();
const admitted = {
mode: 'continue' as const,
streamId: 'conversation-1',
conversationId: 'conversation-1',
generationCreatedAt: 50,
status: 'started' as const,
};
const settleOnAdmission = jest.fn(async () => undefined);
const getBaseUrl = jest.fn(() => 'http://127.0.0.1:3080');
const admissionSource = {
source: 'agent-queued-turn',
sourceId: 'queued-turn-1',
claimId: 'queued-delivery-1',
claimBy: 'queued-worker-1',
};
const prepareContinue = jest.fn(async () => ({
status: 'ready' as const,
input: 'queued user turn',
parentMessageId: 'latest-response',
expectedPredecessorCreatedAt: 49,
files: [{ file_id: 'file-1' }],
quotes: ['quoted context'],
manualSkills: ['research'],
admissionSource,
settleOnAdmission,
}));
const fetcher = fetchMock(async () => response(admitted));
const host = createAgentTriggerExecutionHost(
deps(fetcher, {
prepareContinue,
getBaseUrl,
}),
);
await expect(host.dispatch(envelope, { attempt: 3, maxAttempts: 3 })).resolves.toEqual(
admitted,
);
expect(JSON.parse(String(fetcher.mock.calls[0][1]?.body))).toMatchObject({
text: 'queued user turn',
parentMessageId: 'latest-response',
expectedPredecessorCreatedAt: 49,
files: [{ file_id: 'file-1' }],
quotes: ['quoted context'],
manualSkills: ['research'],
agentContinuationAdmission: admissionSource,
});
expect(getBaseUrl).toHaveBeenCalledWith({ localOnly: true });
expect(prepareContinue).toHaveBeenCalledWith(envelope, {
idempotencyKey: getAgentTriggerIdempotencyKey(envelope),
attempt: 3,
maxAttempts: 3,
});
expect(settleOnAdmission).toHaveBeenCalledWith(admitted);
});
it('keeps a prepared queued turn claimed when admission settlement is unavailable', async () => {
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
const host = createAgentTriggerExecutionHost(
deps(
fetchMock(async () =>
response({
mode: 'continue',
streamId: 'conversation-1',
conversationId: 'conversation-1',
status: 'started',
}),
),
{
prepareContinue: async () => ({
status: 'ready',
input: 'queued user turn',
parentMessageId: 'response-1',
releaseOnDefiniteFailure,
settleOnAdmission: async () => {
throw new Error('mongo unavailable');
},
}),
},
),
);
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
certainty: 'ambiguous',
retryable: true,
code: 'PREPARATION_SETTLEMENT_FAILED',
});
expect(releaseOnDefiniteFailure).not.toHaveBeenCalled();
});
it('carries server-resolved binding identity only on bound child continuations', async () => {
const base = createContinueEnvelope();
if (base.mode !== 'continue') {
@ -653,7 +753,10 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
id: completion.taskId,
type: EVENT_ACTOR_DETACHED_COMPLETION_TYPE,
occurredAt: completion.generationCreatedAt + 1,
source: { id: EVENT_ACTOR_DETACHED_COMPLETION_SOURCE, type: 'internal' },
source: {
id: EVENT_ACTOR_DETACHED_COMPLETION_SOURCE,
type: 'internal',
},
payload: completion,
},
expectedAction: { toolName: 'submit_move' },
@ -787,7 +890,10 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
deps(
fetchMock(async () =>
response(
{ code: 'PARENT_STATE_UNAVAILABLE', error: 'Parent state is unavailable.' },
{
code: 'PARENT_STATE_UNAVAILABLE',
error: 'Parent state is unavailable.',
},
{ status: 503, headers: { 'retry-after': '1' } },
),
),
@ -841,7 +947,11 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
const host = createAgentTriggerExecutionHost(
deps(
fetchMock(async () =>
response({ streamId: 'other', conversationId: 'other', status: 'started' }),
response({
streamId: 'other',
conversationId: 'other',
status: 'started',
}),
),
),
);

View file

@ -1,5 +1,6 @@
import { logger, tenantStorage } from '@librechat/data-schemas';
import { Constants, EModelEndpoint } from 'librechat-data-provider';
import type { TFile } from 'librechat-data-provider';
import type {
AgentContinueTriggerEnvelope,
AgentFireTriggerEnvelope,
@ -37,14 +38,33 @@ type FireStatus = 'started' | 'resumed' | 'replaced' | 'settled';
export type AgentTriggerFetch = (input: string | URL, init?: RequestInit) => Promise<Response>;
export interface AgentContinuationAdmissionSource {
source: string;
sourceId: string;
claimId: string;
claimBy: string;
}
export type AgentTriggerContinuePreparation =
| {
status: 'ready';
input: string;
parentMessageId: string;
expectedPredecessorCreatedAt?: number;
/** Complete user-turn context for internal continuations that must start
* a fresh ordinary Agent turn rather than inject into an existing run. */
files?: Partial<TFile>[];
quotes?: string[];
manualSkills?: string[];
/** Trusted source identity committed by execution enrollment before the
* provider-start fence opens. */
admissionSource?: AgentContinuationAdmissionSource;
/** Compensates a durable pre-admission claim only when the host knows
* that no generation was admitted. Ambiguous outcomes retain the claim. */
releaseOnDefiniteFailure?: () => MaybePromise<void>;
releaseOnDefiniteFailure?: (error?: AgentTriggerExecutionError) => MaybePromise<void>;
/** Commits the source handoff after generation admission. Failure is
* outcome-ambiguous: the same delivery retries with the same request id. */
settleOnAdmission?: (result: AgentTriggerContinueResult) => MaybePromise<void>;
}
| { status: 'settled' };
@ -146,7 +166,7 @@ export interface AgentTriggerExecutionHostDeps {
export interface AgentTriggerExecutionHost {
dispatch: (
envelope: unknown,
options?: { signal?: AbortSignal },
options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
) => Promise<AgentTriggerExecutionResult>;
}
@ -302,7 +322,11 @@ async function readResponseBody(response: Response): Promise<BoundedResponseBody
const remaining = MAX_RESPONSE_BODY_BYTES - size;
if (chunk.value.byteLength > remaining) {
if (remaining > 0) {
parts.push(decoder.decode(chunk.value.subarray(0, remaining), { stream: true }));
parts.push(
decoder.decode(chunk.value.subarray(0, remaining), {
stream: true,
}),
);
}
parts.push(decoder.decode());
await reader.cancel().catch(() => undefined);
@ -587,7 +611,12 @@ async function startRun(
context.signal,
),
setupValue(
() => deps.getBaseUrl(detachedCompletion == null ? undefined : { localOnly: true }),
() =>
deps.getBaseUrl(
detachedCompletion == null && readyPreparation?.admissionSource == null
? undefined
: { localOnly: true },
),
mode,
scope,
context.signal,
@ -625,13 +654,28 @@ async function startRun(
endpoint: EModelEndpoint.agents,
agent_id: envelope.target.agentId,
parentMessageId,
...(readyPreparation?.expectedPredecessorCreatedAt != null && {
expectedPredecessorCreatedAt: readyPreparation.expectedPredecessorCreatedAt,
}),
...(envelope.mode === 'continue' && {
conversationId: envelope.target.conversationId,
}),
...(readyPreparation?.files != null && {
files: readyPreparation.files,
}),
...(readyPreparation?.quotes != null && {
quotes: readyPreparation.quotes,
}),
...(readyPreparation?.manualSkills != null && {
manualSkills: readyPreparation.manualSkills,
}),
isContinued: false,
isRegenerate: false,
clientRequestId: context.idempotencyKey,
generationProtocolVersion: 2,
...(readyPreparation?.admissionSource != null && {
agentContinuationAdmission: readyPreparation.admissionSource,
}),
...(envelope.mode === 'continue' &&
envelope.target.bindingId != null && {
agentEventDelivery: {
@ -670,7 +714,9 @@ async function startRun(
...(run?.conversationId != null && {
newConversationId: run.conversationId,
}),
...(run?.chatProjectId != null && { chatProjectId: run.chatProjectId }),
...(run?.chatProjectId != null && {
chatProjectId: run.chatProjectId,
}),
...(run?.files != null && { files: run.files }),
...(typeof timezone === 'string' && timezone.trim().length > 0
? { timezone: timezone.trim() }
@ -772,6 +818,28 @@ async function startRun(
status: response.status,
});
}
if (
mode === 'continue' &&
result.mode === 'continue' &&
readyPreparation?.settleOnAdmission != null
) {
try {
await readyPreparation.settleOnAdmission(result);
} catch (error) {
throw executionError(
`Agent trigger continue admitted its generation but could not settle its prepared source: ${
error instanceof Error ? error.message : String(error)
}`,
{
mode,
certainty: 'ambiguous',
retryable: true,
code: 'PREPARATION_SETTLEMENT_FAILED',
status: response.status,
},
);
}
}
return result;
} catch (error) {
if (
@ -781,7 +849,7 @@ async function startRun(
canReleasePreparedResult(error)
) {
try {
await preparation.releaseOnDefiniteFailure();
await preparation.releaseOnDefiniteFailure(error);
} catch (releaseError) {
throw executionError(
`Agent trigger ${mode} could not release its rejected preparation: ${
@ -792,6 +860,7 @@ async function startRun(
certainty: 'definite',
retryable: true,
code: 'PREPARATION_RELEASE_FAILED',
deferWithoutAttempt: true,
},
);
}

View file

@ -1,6 +1,7 @@
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
getTenantId,
SYSTEM_TENANT_ID,
} from '@librechat/data-schemas';
@ -180,6 +181,7 @@ describe('durable agent trigger service', () => {
expect.objectContaining({
workerCapabilities: [
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
],
}),
@ -215,7 +217,10 @@ describe('durable agent trigger service', () => {
expect(methods.claimNextAgentTriggerDelivery).toHaveBeenCalledWith(
expect.objectContaining({
workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1],
workerCapabilities: [
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
],
}),
);
await service.stop();

View file

@ -1,6 +1,7 @@
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
logger,
runAsSystem,
} from '@librechat/data-schemas';
@ -11,6 +12,7 @@ import type {
import type {
AgentTriggerDeliveryFailure,
AgentTriggerDeliveryEngine,
AgentTriggerDeliveryEngineDeps,
AgentTriggerDeliveryEngineOptions,
AgentTriggerDeliveryRecord,
AgentTriggerDeliveryStore,
@ -54,6 +56,7 @@ export interface AgentTriggerServiceDeps {
purgeRecoveryIntervalMs?: number;
purgeRecoveryLimit?: number;
supportsDetachedActionCompletion?: () => boolean;
settleSourceBeforeDeadLetter?: AgentTriggerDeliveryEngineDeps['settleSourceBeforeDeadLetter'];
}
export interface AgentTriggerDeliveryReceipt {
@ -180,6 +183,7 @@ function createDeliveryStore(
...input,
workerCapabilities: [
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
...(supportsDetachedActionCompletion()
? [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1]
: []),
@ -272,7 +276,9 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
((principal) => generateAgentTriggerToken(principal.userId, AGENT_TRIGGER_TOKEN_TTL)),
...(deps.fetch != null && { fetch: deps.fetch }),
...(deps.getTimezone != null && { getTimezone: deps.getTimezone }),
...(deps.prepareContinue != null && { prepareContinue: deps.prepareContinue }),
...(deps.prepareContinue != null && {
prepareContinue: deps.prepareContinue,
}),
...(deps.timeoutMs != null && { timeoutMs: deps.timeoutMs }),
});
@ -304,7 +310,7 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
const dispatchForActivePrincipal = async (
envelope: unknown,
options?: { signal?: AbortSignal },
options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
): Promise<AgentTriggerExecutionResult> => {
const parsed = parseAgentTriggerEnvelope(envelope);
await requireActivePrincipal(parsed.principal.userId);
@ -440,6 +446,9 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
{
store: createDeliveryStore(methods, supportsDetachedActionCompletion),
dispatch: dispatchForActivePrincipal,
...(deps.settleSourceBeforeDeadLetter != null && {
settleSourceBeforeDeadLetter: deps.settleSourceBeforeDeadLetter,
}),
},
deps.deliveryOptions,
);

View file

@ -2939,6 +2939,51 @@ class GenerationJobManagerClass {
);
}
/** Returns immutable proof that one exact request crossed generation
* admission, even after the generation job itself has completed or been
* replaced. Built-in stores retain this started receipt for the bounded
* idempotency horizon; legacy custom stores fall back to the live job. */
async getGenerationAdmissionEvidence(
userId: string,
clientRequestId: string,
streamId: string,
conversationId: string = streamId,
): Promise<{ generationId: string; generationCreatedAt: number } | null> {
if (!CLIENT_REQUEST_ID_PATTERN.test(clientRequestId)) {
return null;
}
const getClaim = this.jobStore.getIdempotencyClaim;
if (getClaim != null) {
const claim = await getClaim.call(
this.jobStore,
this.generationClaimKey(userId, clientRequestId, streamId),
);
if (claim == null) {
return null;
}
const normalized = normalizeTokenClaim(claim, 'admission evidence');
assertClaimMatchesRequest(normalized, streamId, conversationId);
if (normalized.startedAt == null) {
return null;
}
return {
generationId: normalized.streamId,
generationCreatedAt: normalized.startedAt,
};
}
const job = await this.jobStore.getJob(streamId);
if (
job == null ||
job.userId !== userId ||
(job.conversationId ?? streamId) !== conversationId ||
job.idempotencyClientRequestId !== clientRequestId
) {
return null;
}
return { generationId: streamId, generationCreatedAt: job.createdAt };
}
/** Atomically prevents an unpublished automatic continuation from starting
* after its delivery has been retired for manual recovery. If recovery wins
* the unstarted idempotency claim (directly or through the existing takeover

View file

@ -64,6 +64,20 @@ describe('InMemoryJobStore.claimIdempotencyKey', () => {
await expect(store.hasIdempotencyKey('user:missing')).resolves.toBe(false);
});
it('reads an idempotency receipt without creating a missing claim', async () => {
const receipt = {
streamId: 's1',
conversationId: 'c1',
claimToken: 'token-1',
claimedAt: 1,
startedAt: 2,
};
await expect(store.getIdempotencyClaim('user:missing')).resolves.toBeNull();
await store.claimIdempotencyKey('user:existing', receipt, 1200);
await expect(store.getIdempotencyClaim('user:existing')).resolves.toEqual(receipt);
await expect(store.hasIdempotencyKey('user:missing')).resolves.toBe(false);
});
it('does not report an expired claim as existing', async () => {
jest.useFakeTimers();
try {
@ -234,6 +248,27 @@ describe('GenerationJobManager start-generation claim', () => {
await expect(manager.hasGenerationClaim('user-2', 'req-1')).resolves.toBe(false);
});
it('reads durable admission evidence after the live job is gone', async () => {
await store.claimIdempotencyKey(
'{stream-a}:user-1:req-1',
{
streamId: 'stream-a',
conversationId: 'convo-a',
claimToken: 'claim-token',
claimedAt: 1,
startedAt: 42,
},
1200,
);
await expect(
manager.getGenerationAdmissionEvidence('user-1', 'req-1', 'stream-a', 'convo-a'),
).resolves.toEqual({ generationId: 'stream-a', generationCreatedAt: 42 });
await expect(
manager.getGenerationAdmissionEvidence('user-1', 'req-2', 'stream-a', 'convo-a'),
).resolves.toBeNull();
});
it('claims the exact legacy key before the same-slot primary with staggered TTLs', async () => {
const claimSpy = jest.spyOn(store, 'claimIdempotencyKey');
const result = await manager.claimGeneration(

View file

@ -906,6 +906,18 @@ export class InMemoryJobStore implements IJobStoreV2 {
return false;
}
async getIdempotencyClaim(key: string): Promise<IdempotencyClaimValue | null> {
const existing = this.idempotencyClaims.get(key);
if (existing == null) {
return null;
}
if (existing.expiresAt <= Date.now()) {
this.idempotencyClaims.delete(key);
return null;
}
return { ...existing.value };
}
async takeoverIdempotencyKey(
key: string,
expected: IdempotencyClaimValue,

View file

@ -2755,6 +2755,18 @@ export class RedisJobStore implements IJobStoreV2 {
return (await this.redis.exists(KEYS.idempotency(key))) === 1;
}
async getIdempotencyClaim(key: string): Promise<IdempotencyClaimValue | null> {
const value = await this.redis.get(KEYS.idempotency(key));
if (value == null) {
return null;
}
try {
return JSON.parse(value) as IdempotencyClaimValue;
} catch {
throw new Error('Invalid generation idempotency claim');
}
}
async takeoverIdempotencyKey(
key: string,
expected: IdempotencyClaimValue,

View file

@ -903,6 +903,10 @@ export interface IJobStore {
* where every request remains subject to the limiter. */
hasIdempotencyKey?(key: string): Promise<boolean>;
/** Read-only claim receipt used by durable source reconcilers. Optional
* stores fall back to inspecting the current generation only. */
getIdempotencyClaim?(key: string): Promise<IdempotencyClaimValue | null>;
deleteJob(streamId: string, expectedCreatedAt?: number): Promise<boolean>;
hasJob(streamId: string): Promise<boolean>;
getRunningJobs(): Promise<SerializableJobData[]>;

View file

@ -1,7 +1,12 @@
/**
* @jest-environment jsdom
*/
import { buildLoginRedirectUrl, getSharedLinks } from '../src/api-endpoints';
import {
agentQueuedTurn,
buildLoginRedirectUrl,
getSharedLinks,
agentQueuedTurnsByConversation,
} from '../src/api-endpoints';
describe('buildLoginRedirectUrl', () => {
afterEach(() => {
@ -88,3 +93,27 @@ describe('getSharedLinks', () => {
);
});
});
describe('agent queued turns', () => {
it('encodes conversation identity exactly once', () => {
expect(agentQueuedTurnsByConversation('conversation/a b')).toBe(
'/api/agents/chat/queued-turns?conversationId=conversation%2Fa%20b',
);
});
it('encodes deduplicated known request identities as repeated bounded query values', () => {
expect(
agentQueuedTurnsByConversation('conversation/one', [
'request/one',
'request,two',
'request/one',
]),
).toBe(
'/api/agents/chat/queued-turns?conversationId=conversation%2Fone&clientRequestIds=request%2Fone&clientRequestIds=request%2Ctwo',
);
});
it('encodes queued-turn identity as one path segment', () => {
expect(agentQueuedTurn('turn/a b')).toBe('/api/agents/chat/queued-turns/turn%2Fa%20b');
});
});

View file

@ -313,6 +313,22 @@ export const agents = ({ path = '', options }: { path?: string; options?: object
export const activeJobs = () => `${BASE_URL}/api/agents/chat/active`;
const agentQueuedTurnsRoot = `${BASE_URL}/api/agents/chat/queued-turns`;
export const agentQueuedTurns = () => agentQueuedTurnsRoot;
export const agentQueuedTurnsByConversation = (
conversationId: string,
clientRequestIds: string[] = [],
) => {
const uniqueIds = Array.from(new Set(clientRequestIds)).slice(0, 100);
const knownIds =
uniqueIds.length > 0
? `&${uniqueIds.map((id) => `clientRequestIds=${encodeURIComponent(id)}`).join('&')}`
: '';
return `${agentQueuedTurnsRoot}?conversationId=${encodeURIComponent(conversationId)}${knownIds}`;
};
export const agentQueuedTurn = (queuedTurnId: string) =>
`${agentQueuedTurnsRoot}/${encodeURIComponent(queuedTurnId)}`;
export const mcp = {
tools: `${BASE_URL}/api/mcp/tools`,
servers: `${BASE_URL}/api/mcp/servers`,

View file

@ -6,6 +6,7 @@ import * as permissions from './accessPermissions';
import * as endpoints from './api-endpoints';
import { uploadEventStream } from './upload';
import * as mcp from './types/mcpServers';
import * as qt from './types/queuedTurns';
import * as sch from './types/schedules';
import * as a from './types/assistants';
import * as m from './types/mutations';
@ -72,7 +73,9 @@ export function getFavorites(): Promise<q.TUserFavorite[]> {
}
export function updateFavorites(favorites: q.TUserFavorite[]): Promise<q.TUserFavorite[]> {
return request.post(`${endpoints.apiBaseUrl()}/api/user/settings/favorites`, { favorites });
return request.post(`${endpoints.apiBaseUrl()}/api/user/settings/favorites`, {
favorites,
});
}
/** Tool favorites — starred marketplace items (builtins, tools, MCP servers, skills). */
@ -135,7 +138,10 @@ export function updateSharedLink(
targetMessageId?: string,
snapshotFiles?: boolean,
): Promise<t.TSharedLinkResponse> {
return request.patch(endpoints.updateSharedLink(shareId), { targetMessageId, snapshotFiles });
return request.patch(endpoints.updateSharedLink(shareId), {
targetMessageId,
snapshotFiles,
});
}
export function deleteSharedLink(shareId: string): Promise<m.TDeleteSharedLinkResponse> {
@ -882,7 +888,9 @@ export function forkSharedConversation(
}
export function deleteConversation(payload: t.TDeleteConversationRequest) {
return request.deleteWithOptions(endpoints.deleteConversation(), { data: { arg: payload } });
return request.deleteWithOptions(endpoints.deleteConversation(), {
data: { arg: payload },
});
}
export function clearAllConversations(): Promise<unknown> {
@ -944,7 +952,9 @@ export function assignConversationToProject(
payload: t.TAssignConversationToProjectRequest,
): Promise<t.TAssignConversationToProjectResponse> {
const { conversationId, projectId } = payload;
return request.put(endpoints.projectConversation(conversationId), { projectId });
return request.put(endpoints.projectConversation(conversationId), {
projectId,
});
}
export function pinConversation(
@ -967,7 +977,9 @@ export function updateMessage(payload: t.TUpdateMessageRequest): Promise<unknown
throw new Error('conversationId is required');
}
return request.put(endpoints.messages({ conversationId, messageId }), { text });
return request.put(endpoints.messages({ conversationId, messageId }), {
text,
});
}
export function updateMessageContent(payload: t.TUpdateMessageContent): Promise<unknown> {
@ -976,7 +988,10 @@ export function updateMessageContent(payload: t.TUpdateMessageContent): Promise<
throw new Error('conversationId is required');
}
return request.put(endpoints.messages({ conversationId, messageId }), { text, index });
return request.put(endpoints.messages({ conversationId, messageId }), {
text,
index,
});
}
export const editArtifact = async ({
@ -1108,6 +1123,25 @@ export function getSchedules(): Promise<sch.TSchedulesResponse> {
return request.get(endpoints.schedules());
}
export function enqueueAgentQueuedTurn(
payload: qt.TEnqueueAgentQueuedTurnRequest,
): Promise<qt.TEnqueueAgentQueuedTurnResponse> {
return request.post(endpoints.agentQueuedTurns(), payload);
}
export function listAgentQueuedTurns(
conversationId: string,
clientRequestIds?: string[],
): Promise<qt.TListAgentQueuedTurnsResponse> {
return request.get(endpoints.agentQueuedTurnsByConversation(conversationId, clientRequestIds));
}
export function cancelAgentQueuedTurn(
queuedTurnId: string,
): Promise<qt.TCancelAgentQueuedTurnResponse> {
return request.delete(endpoints.agentQueuedTurn(queuedTurnId));
}
export function getSchedule(id: string): Promise<sch.TSchedule> {
return request.get(endpoints.schedule(id));
}
@ -1448,7 +1482,10 @@ export const updateMemory = (
originalKey?: string,
agentId?: string,
): Promise<q.UpdateMemoryResponse> => {
return request.patch(endpoints.memory(originalKey || key, agentId), { key, value });
return request.patch(endpoints.memory(originalKey || key, agentId), {
key,
value,
});
};
export const updateMemoryById = (

View file

@ -37,6 +37,7 @@ export * from './types/web';
export * from './types/graph';
export * from './types/insights';
export * from './types/subagents';
export * from './types/queuedTurns';
/* access permissions */
export * from './accessPermissions';
/* query/mutation keys */

View file

@ -95,6 +95,7 @@ export enum QueryKeys {
schedule = 'schedule',
parentSubagents = 'parentSubagents',
subagentThread = 'subagentThread',
agentQueuedTurns = 'agentQueuedTurns',
}
// Dynamic query keys that require parameters
@ -104,6 +105,8 @@ export const DynamicQueryKeys = {
export enum MutationKeys {
subagentControl = 'subagentControl',
enqueueAgentQueuedTurn = 'enqueueAgentQueuedTurn',
cancelAgentQueuedTurn = 'cancelAgentQueuedTurn',
updateLangfuseConnection = 'updateLangfuseConnection',
testLangfuseConnection = 'testLangfuseConnection',
createAgentApiKey = 'createAgentApiKey',

View file

@ -1,3 +1,4 @@
export * from './queries';
export * from './mcpServers';
export * from './subagents';
export * from './queuedTurns';

View file

@ -0,0 +1,109 @@
import {
agentQueuedTurnCapabilitySchema,
agentQueuedTurnReceiptSchema,
cancelAgentQueuedTurnSchema,
enqueueAgentQueuedTurnSchema,
listAgentQueuedTurnsSchema,
} from './queuedTurns';
const request = {
conversationId: 'conversation-1',
parentMessageId: 'message-1',
clientRequestId: 'request-1',
text: 'Continue with this context',
files: [{ file_id: 'file-1', filename: 'context.txt' }],
quotes: ['quoted context'],
manualSkills: ['research'],
expectedPredecessorCreatedAt: 42,
};
describe('agent queued turn schemas', () => {
it('accepts the complete enqueue contract', () => {
expect(enqueueAgentQueuedTurnSchema.parse(request)).toEqual(request);
});
it('allows attachment-only queued turns', () => {
expect(
enqueueAgentQueuedTurnSchema.parse({
conversationId: 'conversation-1',
parentMessageId: 'message-1',
clientRequestId: 'request-1',
text: '',
files: [{ file_id: 'file-1' }],
}),
).toEqual({
conversationId: 'conversation-1',
parentMessageId: 'message-1',
clientRequestId: 'request-1',
text: '',
files: [{ file_id: 'file-1' }],
});
});
it('rejects an invalid predecessor epoch', () => {
expect(() =>
enqueueAgentQueuedTurnSchema.parse({
...request,
expectedPredecessorCreatedAt: -1,
}),
).toThrow();
});
it('validates list and cancel identities', () => {
expect(
listAgentQueuedTurnsSchema.parse({
conversationId: 'conversation-1',
clientRequestIds: ['request-1', 'request-1', 'request-2'],
}),
).toEqual({
conversationId: 'conversation-1',
clientRequestIds: ['request-1', 'request-2'],
});
expect(cancelAgentQueuedTurnSchema.parse({ queuedTurnId: 'turn-1' })).toEqual({
queuedTurnId: 'turn-1',
});
expect(() => listAgentQueuedTurnsSchema.parse({ conversationId: '' })).toThrow();
expect(() => cancelAgentQueuedTurnSchema.parse({ queuedTurnId: '' })).toThrow();
});
it('parses a durable queued receipt', () => {
const receipt = {
...request,
queuedTurnId: 'turn-1',
status: 'queued',
position: 0,
revision: 1,
createdAt: '2026-08-30T12:00:00.000Z',
updatedAt: '2026-08-30T12:00:00.000Z',
};
expect(agentQueuedTurnReceiptSchema.parse(receipt)).toEqual(receipt);
});
it('parses a bounded terminal failure projection', () => {
const receipt = {
...request,
queuedTurnId: 'turn-1',
status: 'dead',
revision: 1,
failure: { code: 'PARENT_NOT_FOUND', message: 'The selected branch is unavailable' },
createdAt: '2026-08-30T12:00:00.000Z',
updatedAt: '2026-08-30T12:01:00.000Z',
};
expect(agentQueuedTurnReceiptSchema.parse(receipt)).toEqual(receipt);
});
it('requires durability only when the capability is supported', () => {
expect(agentQueuedTurnCapabilitySchema.parse({ supported: false })).toEqual({
supported: false,
});
expect(
agentQueuedTurnCapabilitySchema.parse({
supported: true,
durability: 'process_local',
}),
).toEqual({ supported: true, durability: 'process_local' });
expect(() => agentQueuedTurnCapabilitySchema.parse({ supported: true })).toThrow();
});
});

View file

@ -0,0 +1,93 @@
import { z } from 'zod';
export const agentQueuedTurnStatuses = [
'queued',
'claimed',
'admitted',
'cancelled',
'dead',
] as const;
export const agentQueuedTurnDurability = ['process_local', 'durable'] as const;
export const agentQueuedTurnFileRefSchema = z.object({
file_id: z.string().trim().min(1),
type: z.string().optional(),
filepath: z.string().optional(),
filename: z.string().optional(),
height: z.number().optional(),
width: z.number().optional(),
bytes: z.number().nonnegative().optional(),
});
export type TAgentQueuedTurnFileRef = z.infer<typeof agentQueuedTurnFileRefSchema>;
export const enqueueAgentQueuedTurnSchema = z.object({
conversationId: z.string().trim().min(1),
parentMessageId: z.string().trim().min(1),
clientRequestId: z.string().trim().min(1).max(128),
text: z.string(),
files: z.array(agentQueuedTurnFileRefSchema).optional(),
quotes: z.array(z.string()).optional(),
manualSkills: z.array(z.string().trim().min(1)).optional(),
priority: z.boolean().optional(),
expectedPredecessorCreatedAt: z.number().int().nonnegative().optional(),
});
export type TEnqueueAgentQueuedTurnRequest = z.infer<typeof enqueueAgentQueuedTurnSchema>;
export const listAgentQueuedTurnsSchema = z.object({
conversationId: z.string().trim().min(1),
clientRequestIds: z
.array(z.string().trim().min(1).max(128))
.max(100)
.transform((ids) => Array.from(new Set(ids)))
.optional(),
});
export type TListAgentQueuedTurnsRequest = z.infer<typeof listAgentQueuedTurnsSchema>;
export const cancelAgentQueuedTurnSchema = z.object({
queuedTurnId: z.string().trim().min(1),
});
export type TCancelAgentQueuedTurnRequest = z.infer<typeof cancelAgentQueuedTurnSchema>;
export const agentQueuedTurnReceiptSchema = enqueueAgentQueuedTurnSchema.extend({
queuedTurnId: z.string().trim().min(1),
status: z.enum(agentQueuedTurnStatuses),
position: z.number().int().nonnegative().optional(),
revision: z.number().int().nonnegative(),
failure: z
.object({
code: z.string().trim().min(1).max(128),
message: z.string().max(2048).optional(),
})
.optional(),
createdAt: z.string(),
updatedAt: z.string(),
});
export type TAgentQueuedTurnReceipt = z.infer<typeof agentQueuedTurnReceiptSchema>;
export const agentQueuedTurnCapabilitySchema = z.discriminatedUnion('supported', [
z.object({ supported: z.literal(false) }),
z.object({
supported: z.literal(true),
durability: z.enum(agentQueuedTurnDurability),
}),
]);
export type TAgentQueuedTurnCapability = z.infer<typeof agentQueuedTurnCapabilitySchema>;
export const enqueueAgentQueuedTurnResponseSchema = z.object({
receipt: agentQueuedTurnReceiptSchema,
capability: agentQueuedTurnCapabilitySchema,
});
export type TEnqueueAgentQueuedTurnResponse = z.infer<typeof enqueueAgentQueuedTurnResponseSchema>;
export const listAgentQueuedTurnsResponseSchema = z.object({
queuedTurns: z.array(agentQueuedTurnReceiptSchema),
capability: agentQueuedTurnCapabilitySchema,
revision: z.number().int().nonnegative(),
});
export type TListAgentQueuedTurnsResponse = z.infer<typeof listAgentQueuedTurnsResponseSchema>;
export const cancelAgentQueuedTurnResponseSchema = z.object({
receipt: agentQueuedTurnReceiptSchema,
});
export type TCancelAgentQueuedTurnResponse = z.infer<typeof cancelAgentQueuedTurnResponseSchema>;

View file

@ -35,6 +35,9 @@ export {
MAX_AUDIT_VERIFY_ROWS,
MAX_TOOL_FAVORITES,
AgentTriggerDeliveryConflictError,
AgentQueuedTurnCapacityError,
AgentQueuedTurnConflictError,
AgentQueuedTurnLaneRetiredError,
recordAgentEventActorReceiptMetric,
setAgentEventActorReceiptMetricObserver,
MCPAuthorityProofError,
@ -64,6 +67,7 @@ export {
export {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
} from './types/triggerDelivery';
export type * from './types';
export type * from './methods';

View file

@ -1767,6 +1767,59 @@ describe('Conversation Operations', () => {
});
describe('deleteConvos', () => {
it('retires queued-turn work before each conversation deletion wave', async () => {
const conversationId = uuidv4();
await Conversation.create({
conversationId,
user: 'user123',
tenantId: 'tenant-1',
endpoint: EModelEndpoint.agents,
});
const transitions: string[] = [];
const deleteAgentQueuedTurns = jest.fn(async () => {
transitions.push('queue-retired');
});
const scopedMethods = createConversationMethods(mongoose, {
getMessages,
deleteMessages,
deleteAgentQueuedTurns,
});
await scopedMethods.deleteConvos(
'user123',
{ conversationId },
{
beforeDelete: async () => {
transitions.push('generation-drained');
},
},
);
expect(deleteAgentQueuedTurns).toHaveBeenCalledWith('user123', [
{ conversationId, tenantId: 'tenant-1' },
]);
expect(transitions).toEqual(['queue-retired', 'generation-drained']);
});
it('fails closed before deleting a conversation when queued-turn retirement fails', async () => {
const conversationId = uuidv4();
await Conversation.create({
conversationId,
user: 'user123',
endpoint: EModelEndpoint.agents,
});
const scopedMethods = createConversationMethods(mongoose, {
getMessages,
deleteMessages,
deleteAgentQueuedTurns: async () => Promise.reject(new Error('retirement unavailable')),
});
await expect(scopedMethods.deleteConvos('user123', { conversationId })).rejects.toThrow(
'retirement unavailable',
);
expect(await Conversation.findOne({ conversationId })).not.toBeNull();
});
it('should delete conversations and associated messages', async () => {
await Conversation.create({
conversationId: mockConversationData.conversationId,

View file

@ -466,17 +466,25 @@ export interface ConversationMethods {
archiveAllConvos(user: string): Promise<{ archivedCount: number }>;
}
export interface ConversationMethodDeps
extends Pick<MessageMethods, 'getMessages' | 'deleteMessages'> {
deleteAgentQueuedTurns?: (
user: string,
conversations: Array<{ conversationId: string; tenantId?: string; allTenants?: true }>,
) => Promise<void>;
}
export function createConversationMethods(
mongoose: typeof import('mongoose'),
messageMethods?: Pick<MessageMethods, 'getMessages' | 'deleteMessages'>,
deps?: ConversationMethodDeps,
): ConversationMethods {
let legacyReceiptExpiryCursor: Types.ObjectId | undefined;
function getMessageMethods() {
if (!messageMethods) {
if (!deps) {
throw new Error('Message methods not injected into conversation methods');
}
return messageMethods;
return deps;
}
function getVisibleConversationRetentionFilter(): FilterQuery<IConversation> {
@ -2865,7 +2873,10 @@ export function createConversationMethods(
const Conversation = mongoose.models.Conversation as Model<IConversation>;
const { deleteMessages, getMessages } = getMessageMethods();
const userFilter = { ...filter, user };
type DeletionConversation = Pick<IConversation, 'conversationId' | 'chatProjectId' | 'tags'>;
type DeletionConversation = Pick<
IConversation,
'conversationId' | 'tenantId' | 'chatProjectId' | 'tags'
>;
const retryCascadeOperation = async <T>(operation: () => PromiseLike<T> | T): Promise<T> => {
let lastError: unknown;
for (let attempt = 1; attempt <= 3; attempt += 1) {
@ -2881,7 +2892,7 @@ export function createConversationMethods(
throw lastError;
};
let conversations = await Conversation.find(userFilter)
.select('conversationId chatProjectId tags')
.select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>();
const recoveryConversationIds: string[] = [];
if (!conversations.length && typeof filter.conversationId === 'string') {
@ -2895,7 +2906,7 @@ export function createConversationMethods(
user,
'subagentThread.rootConversationId': filter.conversationId,
})
.select('conversationId chatProjectId tags')
.select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>(),
),
getMessages({ user, conversationId: filter.conversationId }, '_id', { limit: 1 }),
@ -2970,6 +2981,13 @@ export function createConversationMethods(
break;
}
const waveIds = wave.map((conversation) => conversation.conversationId);
await deps?.deleteAgentQueuedTurns?.(
user,
wave.map((conversation) => ({
conversationId: conversation.conversationId,
...(conversation.tenantId != null && { tenantId: conversation.tenantId }),
})),
);
await options?.beforeDelete?.(waveIds);
const result = await Conversation.deleteMany({ user, conversationId: { $in: waveIds } });
acknowledged &&= result.acknowledged;
@ -2984,7 +3002,7 @@ export function createConversationMethods(
user,
'subagentThread.parentConversationId': { $in: waveIds },
})
.select('conversationId chatProjectId tags')
.select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>(),
);
}
@ -2994,6 +3012,16 @@ export function createConversationMethods(
...deletedConversations.map((conversation) => conversation.conversationId),
];
if (recoveryConversationIds.length > 0) {
await deps?.deleteAgentQueuedTurns?.(
user,
recoveryConversationIds.map((conversationId) => ({
conversationId,
allTenants: true,
})),
);
}
const deleteConvoResult: DeleteResult = { acknowledged, deletedCount };
/**

View file

@ -123,6 +123,13 @@ import {
type ValidationIssue,
} from './skill';
import { createScheduleMethods, type ScheduleMethods } from './schedule';
import {
createAgentQueuedTurnMethods,
AgentQueuedTurnCapacityError,
AgentQueuedTurnConflictError,
AgentQueuedTurnLaneRetiredError,
type AgentQueuedTurnMethods,
} from './queuedTurn';
import {
createAgentTriggerDeliveryMethods,
AgentTriggerDeliveryConflictError,
@ -191,6 +198,11 @@ export {
export { AUDIT_SCHEMA_VERSION, MAX_AUDIT_EXPORT_ROWS, MAX_AUDIT_LOG_LIMIT, MAX_AUDIT_VERIFY_ROWS };
export { MAX_TOOL_FAVORITES };
export { AgentTriggerDeliveryConflictError };
export {
AgentQueuedTurnCapacityError,
AgentQueuedTurnConflictError,
AgentQueuedTurnLaneRetiredError,
};
export type AllMethods = UserMethods &
SessionMethods &
@ -230,6 +242,7 @@ export type AllMethods = UserMethods &
SkillMethods &
SkillSyncMethods &
AgentTriggerDeliveryMethods &
AgentQueuedTurnMethods &
ScheduleMethods &
AgentMethods &
ConfigMethods &
@ -283,9 +296,77 @@ export function createMethods(
const messageMethods = createMessageMethods(mongoose);
const agentQueuedTurnMethods = createAgentQueuedTurnMethods(mongoose);
const agentTriggerDeliveryMethods = createAgentTriggerDeliveryMethods(mongoose, {
purgeQueuedTurnsForUser: (user) =>
agentQueuedTurnMethods.deleteAllAgentQueuedTurnsForUser({
user: typeof user === 'string' ? new mongoose.Types.ObjectId(user) : user,
}),
});
const conversationMethods = createConversationMethods(mongoose, {
getMessages: messageMethods.getMessages,
deleteMessages: messageMethods.deleteMessages,
deleteAgentQueuedTurns: async (user, conversations) => {
/** Queued-turn ownership is ObjectId-backed. Conversation methods also
* support synthetic/non-ObjectId owners in embedded integrations and
* tests; those owners cannot have queued-turn rows to retire. */
if (!mongoose.isObjectIdOrHexString(user)) {
return;
}
const owner = new mongoose.Types.ObjectId(user);
const settledAt = new Date();
const deliveryKeys = await agentQueuedTurnMethods.prepareAgentQueuedTurnConversationDeletion({
user: owner,
targets: conversations,
settledAt,
});
await Promise.all(
deliveryKeys.map(async (deliveryKey) => {
const retirement = {
deliveryKey,
sourceId: 'agent-queued-turn',
reason: 'queued_turn_conversation_deleted',
settledAt,
};
let retired = await agentTriggerDeliveryMethods.retireAgentTriggerDelivery(retirement);
if (!retired) {
/** A delivery can exhaust immediately before owner deletion wins.
* Convert that operator-requeueable dead row into the same terminal
* retirement receipt before removing its source record. */
retired = await agentTriggerDeliveryMethods.retireAgentTriggerDelivery({
...retirement,
onlyIfDead: true,
});
}
if (!retired) {
/** Successful and dead delivery receipts have bounded retention.
* A source in `published` proves the delivery existed, so a missing
* receipt after both retirement attempts is terminal absence. */
const fenced =
await agentQueuedTurnMethods.beginAgentQueuedTurnMissingDeliveryRetirement({
deliveryKey,
});
if (fenced) {
const delivery =
await agentTriggerDeliveryMethods.getAgentTriggerDelivery(deliveryKey);
if (delivery == null) {
retired = await agentQueuedTurnMethods.markAgentQueuedTurnMissingDeliveryRetired({
deliveryKey,
});
}
}
}
if (retired) {
await agentQueuedTurnMethods.markAgentQueuedTurnDeliveryRetired({ deliveryKey });
}
}),
);
await agentQueuedTurnMethods.deletePreparedAgentQueuedTurnConversations({
user: owner,
targets: conversations,
});
},
});
// ACL entry methods (used internally for removeAllPermissions)
@ -341,7 +422,6 @@ export function createMethods(
isExternalSkillId: deps.isExternalSkillId,
};
const agentMethods = createAgentMethods(mongoose, agentDeps);
return {
...createUserMethods(mongoose, { getCache: deps.getCache }),
...createSessionMethods(mongoose),
@ -383,7 +463,8 @@ export function createMethods(
...promptMethods,
...skillMethods,
...createSkillSyncMethods(mongoose),
...createAgentTriggerDeliveryMethods(mongoose),
...agentTriggerDeliveryMethods,
...agentQueuedTurnMethods,
...createScheduleMethods(mongoose),
/* Tier 5 */
...agentMethods,
@ -455,6 +536,7 @@ export type {
UpsertSkillSyncCredentialInput,
SkillSyncMethods,
AgentTriggerDeliveryMethods,
AgentQueuedTurnMethods,
AgentTriggerProducerLeaseStatus,
AgentEventActorReceiptMetric,
AgentEventActorReceiptStorageMetrics,

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -639,6 +639,50 @@ describe('agent trigger delivery methods', () => {
).resolves.toMatchObject({ id: successor.delivery.id });
});
it('treats an already successful internal delivery as terminal retirement evidence', async () => {
const source = { id: 'agent-queued-turn', type: 'internal' };
const queued = await methods.enqueueAgentTriggerDelivery(
enqueueInput({
deliveryKey: 'queued-turn-already-succeeded',
envelope: { event: { source } },
}),
);
const claim = await methods.claimNextAgentTriggerDelivery({
workerId: 'queued-turn-worker',
claimToken: 'queued-turn-claim',
now: START,
leaseUntil: new Date(START.getTime() + 60_000),
});
const attempt = await methods.beginAgentTriggerDeliveryAttempt({
id: claim!.id,
workerId: 'queued-turn-worker',
claimToken: 'queued-turn-claim',
now: START,
});
await methods.completeAgentTriggerDelivery({
id: claim!.id,
workerId: 'queued-turn-worker',
claimToken: 'queued-turn-claim',
attempt: attempt!,
result: { status: 'settled' },
settledAt: START,
});
await expect(
methods.retireAgentTriggerDelivery({
deliveryKey: queued.delivery.deliveryKey,
sourceId: source.id,
settledAt: new Date(START.getTime() + 1),
reason: 'source already terminalized',
}),
).resolves.toBe(true);
await expect(Delivery.findById(queued.delivery.id).lean()).resolves.toMatchObject({
status: 'succeeded',
result: { status: 'settled' },
settledAt: START,
});
});
it('does not retire a completion already leased by a resolver for a manual poll', async () => {
const user = new mongoose.Types.ObjectId();
const source = { id: 'background-tool-completion', type: 'internal' };
@ -4213,6 +4257,43 @@ describe('agent trigger delivery methods', () => {
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
});
it('retains trigger purge state when queued-turn cleanup fails', async () => {
const user = new mongoose.Types.ObjectId();
const fenceStartedAt = new Date(START);
let unavailable = true;
const purgeQueuedTurnsForUser = jest.fn(async () => {
if (unavailable) {
throw new Error('queued-turn store unavailable');
}
});
const purgeMethods = createAgentTriggerDeliveryMethods(mongoose, {
purgeQueuedTurnsForUser,
});
await User.create({
_id: user,
email: 'purge-callback@example.com',
provider: 'local',
agentTriggerDeletionStartedAt: fenceStartedAt,
});
await purgeMethods.enqueueAgentTriggerDelivery(enqueueInput({ user }));
await purgeMethods.prepareAgentTriggerUserPurge(user, fenceStartedAt);
await User.deleteOne({ _id: user });
await expect(purgeMethods.deleteAgentTriggerDeliveriesByUser(user)).rejects.toThrow(
'queued-turn store unavailable',
);
expect(await Delivery.countDocuments({ user })).toBe(1);
expect(await LaneSequence.countDocuments({ user })).toBe(1);
expect(await UserPurge.countDocuments({ _id: user })).toBe(1);
unavailable = false;
await expect(purgeMethods.deleteAgentTriggerDeliveriesByUser(user)).resolves.toBeUndefined();
expect(await Delivery.countDocuments({ user })).toBe(0);
expect(await LaneSequence.countDocuments({ user })).toBe(0);
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
expect(purgeQueuedTurnsForUser).toHaveBeenCalledTimes(2);
});
it('recovers an armed purge after the user deletion commits', async () => {
const user = new mongoose.Types.ObjectId();
const fenceStartedAt = new Date(START);
@ -4236,6 +4317,42 @@ describe('agent trigger delivery methods', () => {
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
});
it('retains a recovery marker when queued-turn purge cannot be confirmed', async () => {
const user = new mongoose.Types.ObjectId();
const fenceStartedAt = new Date(START);
let unavailable = true;
const purgeQueuedTurnsForUser = jest.fn(async () => {
if (unavailable) {
throw new Error('queued-turn store unavailable');
}
});
const purgeMethods = createAgentTriggerDeliveryMethods(mongoose, {
purgeQueuedTurnsForUser,
});
await User.create({
_id: user,
email: 'purge-recovery-callback@example.com',
provider: 'local',
agentTriggerDeletionStartedAt: fenceStartedAt,
});
await purgeMethods.enqueueAgentTriggerDelivery(enqueueInput({ user }));
await purgeMethods.prepareAgentTriggerUserPurge(user, fenceStartedAt);
await User.deleteOne({ _id: user });
await expect(purgeMethods.recoverAgentTriggerUserPurges()).rejects.toThrow(
'queued-turn store unavailable',
);
expect(await Delivery.countDocuments({ user })).toBe(1);
expect(await LaneSequence.countDocuments({ user })).toBe(1);
expect(await UserPurge.countDocuments({ _id: user })).toBe(1);
unavailable = false;
await expect(purgeMethods.recoverAgentTriggerUserPurges()).resolves.toBe(1);
expect(await Delivery.countDocuments({ user })).toBe(0);
expect(await LaneSequence.countDocuments({ user })).toBe(0);
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
});
it('rotates active purge markers so a later committed purge is recovered', async () => {
const users = Array.from({ length: 3 }, () => new mongoose.Types.ObjectId());
for (const [index, user] of users.entries()) {

View file

@ -19,6 +19,7 @@ import type {
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
} from '~/types/triggerDelivery';
import { createIndexesWithRetry } from '~/utils/retry';
import logger from '~/config/winston';
@ -412,15 +413,25 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {
...(delivery.history != null && { history: delivery.history }),
...(delivery.settledAt != null && { settledAt: delivery.settledAt }),
...(delivery.expiresAt != null && { expiresAt: delivery.expiresAt }),
...(delivery.requeueCount != null && { requeueCount: delivery.requeueCount }),
...(delivery.requeueCount != null && {
requeueCount: delivery.requeueCount,
}),
...(delivery.updatedAt != null && { updatedAt: delivery.updatedAt }),
...(delivery.envelopeBytes != null && { envelopeBytes: delivery.envelopeBytes }),
...(delivery.envelopeBytes != null && {
envelopeBytes: delivery.envelopeBytes,
}),
...(delivery.coalesceKey != null && { coalesceKey: delivery.coalesceKey }),
...(delivery.coalesceFrom != null && { coalesceFrom: delivery.coalesceFrom }),
...(delivery.coalesceUntil != null && { coalesceUntil: delivery.coalesceUntil }),
...(delivery.coalesceFrom != null && {
coalesceFrom: delivery.coalesceFrom,
}),
...(delivery.coalesceUntil != null && {
coalesceUntil: delivery.coalesceUntil,
}),
...(delivery.batchSize != null && { batchSize: delivery.batchSize }),
...(delivery.batchBytes != null && { batchBytes: delivery.batchBytes }),
...(delivery.batchMemberIds != null && { batchMemberIds: delivery.batchMemberIds }),
...(delivery.batchMemberIds != null && {
batchMemberIds: delivery.batchMemberIds,
}),
...(delivery.batchRootId != null && { batchRootId: delivery.batchRootId }),
...(delivery.batchMembersSettledAt != null && {
batchMembersSettledAt: delivery.batchMembersSettledAt,
@ -429,7 +440,9 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {
awaitTerminalHandling: delivery.awaitTerminalHandling,
}),
...(delivery.handling != null && { handling: delivery.handling }),
...(delivery.actorReceipt != null && { actorReceipt: delivery.actorReceipt }),
...(delivery.actorReceipt != null && {
actorReceipt: delivery.actorReceipt,
}),
};
}
@ -451,6 +464,9 @@ function requireClaim(delivery: IAgentTriggerDelivery | null): AgentTriggerDeliv
export function createAgentTriggerDeliveryMethods(
mongoose: typeof import('mongoose'),
deps: {
purgeQueuedTurnsForUser?: (user: string | Types.ObjectId) => Promise<unknown>;
} = {},
): AgentTriggerDeliveryMethods {
const Delivery = () =>
mongoose.models.AgentTriggerDelivery as Model<IAgentTriggerDeliveryDocument>;
@ -490,7 +506,11 @@ export function createAgentTriggerDeliveryMethods(
const released = await LaneSequence().updateOne(publisherFence, {
$inc: { value: -1 },
$set: { cleanupRequestedAt: new Date() },
$unset: { publisherDeliveryId: 1, publisherRequeueCount: 1, publisherStartedAt: 1 },
$unset: {
publisherDeliveryId: 1,
publisherRequeueCount: 1,
publisherStartedAt: 1,
},
});
return released.modifiedCount === 1;
}
@ -524,7 +544,10 @@ export function createAgentTriggerDeliveryMethods(
return false;
}
}
const publicationLane = { ...lane, publisherRequeueCount: publicationGeneration };
const publicationLane = {
...lane,
publisherRequeueCount: publicationGeneration,
};
let batchRoot: IAgentTriggerDelivery | null = null;
if (
staged?._id != null &&
@ -713,7 +736,11 @@ export function createAgentTriggerDeliveryMethods(
if (publicationCommitted) {
const released = await LaneSequence().updateOne(publisherFence, {
$set: { tailDeliveryId: publisherDeliveryId },
$unset: { publisherDeliveryId: 1, publisherRequeueCount: 1, publisherStartedAt: 1 },
$unset: {
publisherDeliveryId: 1,
publisherRequeueCount: 1,
publisherStartedAt: 1,
},
});
return published.modifiedCount === 1 || released.modifiedCount === 1;
}
@ -933,7 +960,8 @@ export function createAgentTriggerDeliveryMethods(
if (
input.requiredWorkerCapability != null &&
input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1 &&
input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1
input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1 &&
input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1
) {
throw new TypeError('Agent trigger delivery requires an unsupported worker capability');
}
@ -1080,7 +1108,10 @@ export function createAgentTriggerDeliveryMethods(
.lean<IAgentTriggerDelivery[]>()
: [];
const staged = [
...legacyStaged.map((delivery) => ({ ...delivery, stagingRecoveryAt: legacyCursor })),
...legacyStaged.map((delivery) => ({
...delivery,
stagingRecoveryAt: legacyCursor,
})),
...indexedStaged,
];
const stagingRecoveryCursor = new Date();
@ -1444,7 +1475,9 @@ export function createAgentTriggerDeliveryMethods(
: earlier.availableAt,
...(leaseUntil != null && { leaseUntil }),
...(earlier.status === 'succeeded' &&
earlier.handling?.status === 'started' && { reason: 'active_handling' as const }),
earlier.handling?.status === 'started' && {
reason: 'active_handling' as const,
}),
};
}
@ -1601,7 +1634,10 @@ export function createAgentTriggerDeliveryMethods(
leaseUntil: 1,
claimToken: 1,
...(input.status === 'succeeded'
? { lastError: 1, ...(awaitsTerminalHandling && { expiresAt: 1 }) }
? {
lastError: 1,
...(awaitsTerminalHandling && { expiresAt: 1 }),
}
: { result: 1, expiresAt: 1 }),
},
$push: {
@ -1644,7 +1680,11 @@ export function createAgentTriggerDeliveryMethods(
}
}
await Delivery().updateOne(
{ _id: root._id, status: input.status, batchMembersSettledAt: { $exists: false } },
{
_id: root._id,
status: input.status,
batchMembersSettledAt: { $exists: false },
},
{ $set: { batchMembersSettledAt: input.settledAt } },
{ timestamps: false },
);
@ -1800,11 +1840,18 @@ export function createAgentTriggerDeliveryMethods(
}
const update = {
$inc: { attempts: -1 },
$set: { availableAt: input.availableAt, claimAvailableAt: input.availableAt },
$set: {
availableAt: input.availableAt,
claimAvailableAt: input.availableAt,
},
$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },
};
const shieldResult = await Delivery().updateOne(
{ _id: input.id, ...shieldCapabilityFence(input), attempts: input.attempt },
{
_id: input.id,
...shieldCapabilityFence(input),
attempts: input.attempt,
},
{
$inc: update.$inc,
$set: {
@ -1827,7 +1874,11 @@ export function createAgentTriggerDeliveryMethods(
return true;
}
const capabilityResult = await Delivery().updateOne(
{ _id: input.id, ...legacyCapabilityFence(input), attempts: input.attempt },
{
_id: input.id,
...legacyCapabilityFence(input),
attempts: input.attempt,
},
{
...update,
$set: { ...update.$set, status: 'capability_pending' },
@ -2042,7 +2093,10 @@ export function createAgentTriggerDeliveryMethods(
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
status: 'succeeded',
'result.backgroundToolCompletionRetired': true,
$or: [
{ handling: { $exists: false } },
{ 'handling.status': { $in: ['applied', 'completed_no_action', 'failed', 'cancelled'] } },
],
})) != null
);
}
@ -2071,7 +2125,9 @@ export function createAgentTriggerDeliveryMethods(
requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
status: { $in: ['pending', 'leased', 'capability_pending', 'capability_leased'] },
status: {
$in: ['pending', 'leased', 'capability_pending', 'capability_leased'],
},
capabilityStatus: { $ne: 'dead' },
settledAt: { $exists: false },
},
@ -2878,8 +2934,14 @@ export function createAgentTriggerDeliveryMethods(
actorReceipt: { $exists: false },
},
{
$set: { status: 'succeeded', settledAt: input.settledAt, actorReceipt },
$max: { expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS) },
$set: {
status: 'succeeded',
settledAt: input.settledAt,
actorReceipt,
},
$max: {
expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS),
},
$unset: { lastError: 1 },
},
{ new: true },
@ -2959,11 +3021,17 @@ export function createAgentTriggerDeliveryMethods(
now: Date,
): Promise<AgentEventActorReceiptStorageMetrics> {
const [byResolution, expiryEligible, retryDeliveries, deadDeliveries] = await Promise.all([
Delivery().aggregate<{ _id: AgentEventActorReceipt['resolution']; count: number }>([
Delivery().aggregate<{
_id: AgentEventActorReceipt['resolution'];
count: number;
}>([
{ $match: { actorReceipt: { $exists: true } } },
{ $group: { _id: '$actorReceipt.resolution', count: { $sum: 1 } } },
]),
Delivery().countDocuments({ actorReceipt: { $exists: true }, expiresAt: { $lte: now } }),
Delivery().countDocuments({
actorReceipt: { $exists: true },
expiresAt: { $lte: now },
}),
Delivery().countDocuments({
$or: [
{ status: { $in: ['pending', 'capability_pending'] } },
@ -2994,7 +3062,12 @@ export function createAgentTriggerDeliveryMethods(
retainedByResolution[row._id] = row.count;
}
}
return { retainedByResolution, expiryEligible, retryDeliveries, deadDeliveries };
return {
retainedByResolution,
expiryEligible,
retryDeliveries,
deadDeliveries,
};
}
async function retryAgentTriggerDelivery(
@ -3012,7 +3085,13 @@ export function createAgentTriggerDeliveryMethods(
claimAvailableAt: input.availableAt,
lastError: error,
},
$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1, settledAt: 1, expiresAt: 1 },
$unset: {
leaseBy: 1,
leaseUntil: 1,
claimToken: 1,
settledAt: 1,
expiresAt: 1,
},
$push: {
history: {
$each: [
@ -3454,12 +3533,19 @@ export function createAgentTriggerDeliveryMethods(
continue;
}
await Delivery().updateMany(
{ user: marker._id, actorActionAdmissionClosedAt: marker.fenceStartedAt },
{
user: marker._id,
actorActionAdmissionClosedAt: marker.fenceStartedAt,
},
{ $unset: { actorActionAdmissionClosedAt: 1 } },
);
await UserPurge().deleteOne({ _id: marker._id, fenceStartedAt: marker.fenceStartedAt });
await UserPurge().deleteOne({
_id: marker._id,
fenceStartedAt: marker.fenceStartedAt,
});
continue;
}
await deps.purgeQueuedTurnsForUser?.(marker._id);
await Promise.all([
Delivery().deleteMany({ user: marker._id }),
LaneSequence().deleteMany({ user: marker._id }),
@ -3474,6 +3560,7 @@ export function createAgentTriggerDeliveryMethods(
}
async function deleteAgentTriggerDeliveriesByUser(user: string | Types.ObjectId): Promise<void> {
await deps.purgeQueuedTurnsForUser?.(user);
await Promise.all([Delivery().deleteMany({ user }), LaneSequence().deleteMany({ user })]);
await UserPurge().deleteOne({ _id: user });
}

View file

@ -1,3 +1,4 @@
import { createAgentQueuedTurnModel, createAgentQueuedTurnSequenceModel } from './queuedTurn';
import { createAgentTriggerLaneSequenceModel } from './triggerLaneSequence';
import { createScheduleModel, createScheduleRunModel } from './schedule';
import { createSkillSyncCredentialModel } from './skillSyncCredential';
@ -89,6 +90,8 @@ export function createModels(mongoose: typeof import('mongoose')): {
AgentTriggerDelivery: ReturnType<typeof createAgentTriggerDeliveryModel>;
AgentTriggerLaneSequence: ReturnType<typeof createAgentTriggerLaneSequenceModel>;
AgentTriggerUserPurge: ReturnType<typeof createAgentTriggerUserPurgeModel>;
AgentQueuedTurn: ReturnType<typeof createAgentQueuedTurnModel>;
AgentQueuedTurnSequence: ReturnType<typeof createAgentQueuedTurnSequenceModel>;
Schedule: ReturnType<typeof createScheduleModel>;
ScheduleRun: ReturnType<typeof createScheduleRunModel>;
RefreshTokenBridge: ReturnType<typeof createRefreshTokenBridgeModel>;
@ -136,6 +139,8 @@ export function createModels(mongoose: typeof import('mongoose')): {
AgentTriggerDelivery: createAgentTriggerDeliveryModel(mongoose),
AgentTriggerLaneSequence: createAgentTriggerLaneSequenceModel(mongoose),
AgentTriggerUserPurge: createAgentTriggerUserPurgeModel(mongoose),
AgentQueuedTurn: createAgentQueuedTurnModel(mongoose),
AgentQueuedTurnSequence: createAgentQueuedTurnSequenceModel(mongoose),
Schedule: createScheduleModel(mongoose),
ScheduleRun: createScheduleRunModel(mongoose),
RefreshTokenBridge: createRefreshTokenBridgeModel(mongoose),

View file

@ -0,0 +1,31 @@
import type { Model } from 'mongoose';
import type {
IAgentQueuedTurnDocument,
IAgentQueuedTurnSequenceDocument,
} from '~/types/queuedTurn';
import { applyTenantIsolation } from '~/models/plugins/tenantIsolation';
import queuedTurnSequenceSchema from '~/schema/queuedTurnSequence';
import queuedTurnSchema from '~/schema/queuedTurn';
export function createAgentQueuedTurnModel(
mongoose: typeof import('mongoose'),
): Model<IAgentQueuedTurnDocument> {
applyTenantIsolation(queuedTurnSchema);
return (
mongoose.models.AgentQueuedTurn ||
mongoose.model<IAgentQueuedTurnDocument>('AgentQueuedTurn', queuedTurnSchema)
);
}
export function createAgentQueuedTurnSequenceModel(
mongoose: typeof import('mongoose'),
): Model<IAgentQueuedTurnSequenceDocument> {
applyTenantIsolation(queuedTurnSequenceSchema);
return (
mongoose.models.AgentQueuedTurnSequence ||
mongoose.model<IAgentQueuedTurnSequenceDocument>(
'AgentQueuedTurnSequence',
queuedTurnSequenceSchema,
)
);
}

View file

@ -38,5 +38,7 @@ export { default as configSchema } from './config';
export { default as triggerDeliverySchema } from './triggerDelivery';
export { default as triggerLaneSequenceSchema } from './triggerLaneSequence';
export { default as triggerUserPurgeSchema } from './triggerUserPurge';
export { default as queuedTurnSchema } from './queuedTurn';
export { default as queuedTurnSequenceSchema } from './queuedTurnSequence';
export { default as scheduleSchema } from './schedule';
export { default as scheduleRunSchema } from './scheduleRun';

View file

@ -0,0 +1,156 @@
import { Schema } from 'mongoose';
import type { IAgentQueuedTurnDocument } from '~/types/queuedTurn';
const fileRefSchema = new Schema(
{
file_id: { type: String, required: true, maxlength: 256 },
type: { type: String, maxlength: 256 },
filepath: { type: String, maxlength: 2048 },
filename: { type: String, maxlength: 1024 },
height: { type: Number, min: 0 },
width: { type: Number, min: 0 },
bytes: { type: Number, min: 0 },
},
{ _id: false },
);
const failureSchema = new Schema(
{
code: { type: String, required: true, maxlength: 128 },
message: { type: String, required: true, maxlength: 2048 },
},
{ _id: false },
);
const terminalReceiptSchema = new Schema(
{
outcome: {
type: String,
enum: ['admitted', 'cancelled', 'dead'],
required: true,
},
settledAt: { type: Date, required: true },
admissionId: { type: String, maxlength: 128 },
admissionMode: { type: String, enum: ['warm', 'ordinary'] },
generationId: { type: String, maxlength: 256 },
generationCreatedAt: { type: Number, min: 0 },
failure: { type: failureSchema },
},
{ _id: false },
);
const queuedTurnSchema: Schema<IAgentQueuedTurnDocument> = new Schema(
{
user: {
type: Schema.Types.ObjectId,
ref: 'User',
required: true,
index: true,
},
tenantId: { type: String, index: true },
conversationId: { type: String, required: true, maxlength: 256 },
agentId: { type: String, required: true, maxlength: 256 },
parentMessageId: { type: String, required: true, maxlength: 256 },
clientRequestId: { type: String, required: true, maxlength: 128 },
fingerprint: { type: String, required: true, minlength: 43, maxlength: 64 },
laneId: { type: String, required: true, maxlength: 128 },
sequence: { type: Number, min: 1 },
reservationWriterId: { type: String, maxlength: 128 },
activeSlot: { type: Number, min: 0, max: 99 },
status: {
type: String,
enum: ['reserving', 'queued', 'claimed', 'admitted', 'cancelled', 'dead'],
required: true,
default: 'reserving',
},
priority: { type: Boolean, required: true, default: false },
text: { type: String, required: true, maxlength: 32_768 },
files: { type: [fileRefSchema], default: undefined },
quotes: { type: [String], default: undefined },
manualSkills: { type: [String], default: undefined },
expectedPredecessorCreatedAt: { type: Number, min: 0 },
attempts: { type: Number, required: true, default: 0, min: 0 },
availableAt: { type: Date, required: true },
deliveryKey: { type: String, maxlength: 128 },
deliveryState: {
type: String,
enum: ['pending', 'publishing', 'published', 'retiring', 'retired'],
required: true,
default: 'pending',
},
scheduledAt: { type: Date },
claimId: { type: String, maxlength: 128 },
claimBy: { type: String, maxlength: 256 },
claimUntil: { type: Date },
admissionStartedAt: { type: Date },
admissionId: { type: String, maxlength: 128 },
admissionProtocolVersion: { type: Number, enum: [2] },
reconciliationAvailableAt: { type: Date },
reconciliationClaimId: { type: String, maxlength: 128 },
reconciliationClaimBy: { type: String, maxlength: 256 },
reconciliationClaimUntil: { type: Date },
reconciliationAttempts: { type: Number, min: 0 },
terminalReceipt: { type: terminalReceiptSchema },
},
{ timestamps: true },
);
queuedTurnSchema.index(
{ tenantId: 1, user: 1, conversationId: 1, clientRequestId: 1 },
{ unique: true },
);
queuedTurnSchema.index(
{ tenantId: 1, user: 1, conversationId: 1, sequence: 1 },
{
name: 'agent_queued_turn_sequence',
unique: true,
partialFilterExpression: { sequence: { $exists: true } },
},
);
queuedTurnSchema.index(
{ tenantId: 1, user: 1, conversationId: 1, activeSlot: 1 },
{
name: 'agent_queued_turn_active_capacity',
unique: true,
partialFilterExpression: { activeSlot: { $exists: true } },
},
);
queuedTurnSchema.index(
{
tenantId: 1,
user: 1,
conversationId: 1,
status: 1,
priority: -1,
availableAt: 1,
sequence: 1,
},
{ name: 'agent_queued_turn_claim' },
);
queuedTurnSchema.index(
{
tenantId: 1,
user: 1,
conversationId: 1,
status: 1,
priority: -1,
sequence: 1,
},
{ name: 'agent_queued_turn_active' },
);
queuedTurnSchema.index({ tenantId: 1, user: 1, status: 1, claimUntil: 1 });
queuedTurnSchema.index({ status: 1, createdAt: 1, _id: 1 });
queuedTurnSchema.index({
status: 1,
reconciliationAvailableAt: 1,
reconciliationClaimUntil: 1,
_id: 1,
});
queuedTurnSchema.index({
status: 1,
scheduledAt: 1,
availableAt: 1,
sequence: 1,
});
export default queuedTurnSchema;

View file

@ -0,0 +1,28 @@
import { Schema } from 'mongoose';
import type { IAgentQueuedTurnSequenceDocument } from '~/types/queuedTurn';
const queuedTurnSequenceSchema: Schema<IAgentQueuedTurnSequenceDocument> = new Schema(
{
_id: { type: String, required: true },
user: {
type: Schema.Types.ObjectId,
ref: 'User',
required: true,
index: true,
},
tenantId: { type: String, index: true },
conversationId: { type: String, required: true, maxlength: 256 },
laneId: { type: String, required: true, maxlength: 128 },
value: { type: Number, required: true, min: 0 },
reservationId: { type: String, maxlength: 24 },
writerId: { type: String, maxlength: 128 },
writerUntil: { type: Date },
retiredAt: { type: Date },
expiresAt: { type: Date },
},
{ timestamps: true },
);
queuedTurnSequenceSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 });
export default queuedTurnSequenceSchema;

View file

@ -34,6 +34,7 @@ export * from './prompts';
export * from './skill';
export * from './skillSync';
export * from './triggerDelivery';
export * from './queuedTurn';
export * from './schedule';
/* Access Control */
export * from './accessRole';

View file

@ -0,0 +1,154 @@
import type { Document, Types } from 'mongoose';
export type AgentQueuedTurnStatus =
| 'reserving'
| 'queued'
| 'claimed'
| 'admitted'
| 'cancelled'
| 'dead';
export type AgentQueuedTurnDeliveryState =
| 'pending'
| 'publishing'
| 'published'
| 'retiring'
| 'retired';
export interface AgentQueuedTurnFileRef {
file_id: string;
type?: string;
filepath?: string;
filename?: string;
height?: number;
width?: number;
bytes?: number;
}
export interface AgentQueuedTurnFailure {
code: string;
message: string;
}
export interface AgentQueuedTurnTerminalReceipt {
outcome: 'admitted' | 'cancelled' | 'dead';
settledAt: Date;
admissionId?: string;
admissionMode?: 'warm' | 'ordinary';
generationId?: string;
generationCreatedAt?: number;
failure?: AgentQueuedTurnFailure;
}
export interface IAgentQueuedTurn {
_id?: Types.ObjectId;
user: Types.ObjectId;
tenantId?: string;
conversationId: string;
agentId: string;
parentMessageId: string;
clientRequestId: string;
fingerprint: string;
/** Immutable generation of the conversation lane that admitted this row. */
laneId?: string;
/** Assigned after the row is durably visible as `reserving`. */
sequence?: number;
/** Fences a reserving row to the lane writer allowed to assign its sequence. */
reservationWriterId?: string;
/** Bounded active-lane capacity token. Present only while queued/claimed. */
activeSlot?: number;
status: AgentQueuedTurnStatus;
priority: boolean;
text: string;
files?: AgentQueuedTurnFileRef[];
quotes?: string[];
manualSkills?: string[];
expectedPredecessorCreatedAt?: number;
attempts: number;
availableAt: Date;
deliveryKey?: string;
deliveryState?: AgentQueuedTurnDeliveryState;
scheduledAt?: Date;
claimId?: string;
claimBy?: string;
claimUntil?: Date;
/** Durable proof that ordinary admission may have crossed the HTTP boundary. */
admissionStartedAt?: Date;
admissionId?: string;
/** Version 2 requires accepted and deduplicated execution responses to prove
* the exact post-invocation source receipt. */
admissionProtocolVersion?: 2;
reconciliationAvailableAt?: Date;
reconciliationClaimId?: string;
reconciliationClaimBy?: string;
reconciliationClaimUntil?: Date;
reconciliationAttempts?: number;
terminalReceipt?: AgentQueuedTurnTerminalReceipt;
createdAt?: Date;
updatedAt?: Date;
}
export interface IAgentQueuedTurnDocument extends Omit<IAgentQueuedTurn, '_id'>, Document {}
export interface IAgentQueuedTurnSequence {
_id: string;
user: Types.ObjectId;
tenantId?: string;
conversationId: string;
/** Changes whenever a fully retired lane is recreated. */
laneId: string;
value: number;
/** Visible reservation currently owning `value`; recovery completes it. */
reservationId?: string;
writerId?: string;
writerUntil?: Date;
retiredAt?: Date;
expiresAt?: Date;
createdAt?: Date;
updatedAt?: Date;
}
export interface IAgentQueuedTurnSequenceDocument
extends Omit<IAgentQueuedTurnSequence, '_id'>,
Document<string> {}
export interface AgentQueuedTurnRecord
extends Omit<IAgentQueuedTurn, '_id' | 'createdAt' | 'sequence' | 'status'> {
queuedTurnId: string;
sequence: number;
status: Exclude<AgentQueuedTurnStatus, 'reserving'>;
createdAt: Date;
}
export type AgentQueuedTurnActiveRecord = Pick<
AgentQueuedTurnRecord,
| 'queuedTurnId'
| 'conversationId'
| 'agentId'
| 'parentMessageId'
| 'clientRequestId'
| 'sequence'
| 'activeSlot'
| 'status'
| 'priority'
| 'text'
| 'files'
| 'quotes'
| 'manualSkills'
| 'expectedPredecessorCreatedAt'
| 'attempts'
| 'availableAt'
| 'deliveryKey'
| 'deliveryState'
| 'scheduledAt'
| 'createdAt'
| 'updatedAt'
| 'terminalReceipt'
>;
export interface AgentQueuedTurnClaim extends AgentQueuedTurnRecord {
status: 'claimed';
claimId: string;
claimBy: string;
claimUntil: Date;
}

View file

@ -15,6 +15,7 @@ export type AgentTriggerDeliveryStatus =
export const AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1 = 'event_actor_detached_action_v1';
export const AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1 =
'background_tool_completion_v1';
export const AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1 = 'agent_queued_turn_v1';
export type AgentTriggerDeliveryOutcome = 'succeeded' | 'retry' | 'dead';
export interface AgentTriggerHandlingState {