mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 04:07:05 +00:00
🪃 feat: Wake Parent Agents on Child Completion (#14975)
* feat: wake parent agents on child completion * wip: harden child completion wakeup lifecycle * fix: close the completion-wakeup static failures Type the durable-claim store fixture, the continue-envelope test helper, and the terminal message's task metadata so the wakeup suites compile against the shapes they actually exercise. Replace `Array.prototype.at`, which the package target library does not provide. Capture the prepared child thread in a non-optional local before the provider callback closes over it, and narrow the trigger envelope itself on `mode === 'continue'` rather than a separately copied mode, so reading the continue target is sound. Lift the parent-message fallback out of a nested ternary into a named resolver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * test: cover the active-predecessor admission fence The Redis job-creation call gained a thirteenth scalar argument, so the spec helper reconstructed the HSET pairs one slot early and rebuilt an invalid job hash; three creation tests failed on that alone. Give the fence itself direct coverage in both store adapters, which it had none of despite deciding whether an automatic continuation may replace a live parent turn. Each proves a running and a requires_action predecessor are refused with the state a controller needs for a finite 409, that an absent or settled predecessor is admitted, and that an ordinary user turn without the policy still replaces its predecessor. The Redis case also asserts a refused continuation leaves the parent's durable job and chunks untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ * fix: harden completion wakeup rollout and claims * fix: close completion wakeup race windows * test: keep the child store fixture exact * fix: close final subagent wakeup gaps * fix: preserve ambiguous completion claims * fix: release pre-admission wakeup claims * fix: stabilize subagent completion recovery --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
parent
d175741010
commit
5e3c680761
34 changed files with 2526 additions and 126 deletions
|
|
@ -1211,11 +1211,17 @@ OPENWEATHER_API_KEY=
|
|||
# Agent Trigger Delivery #
|
||||
#===========================#
|
||||
|
||||
# Base URL used by trusted in-process event producers to dispatch agent fires and steers.
|
||||
# Base URL used by trusted in-process event producers to dispatch agent fires, continuations,
|
||||
# and steers.
|
||||
# Defaults to this process's bound listener. Set only when internal trigger admission must
|
||||
# traverse another trusted HTTP origin, such as a TLS front door.
|
||||
# AGENT_TRIGGERS_SELF_URL=http://127.0.0.1:3080
|
||||
|
||||
# Automatically continue a saved parent agent after a detached subagent settles.
|
||||
# Rolling-deploy safety: deploy support with this disabled first, wait until every API
|
||||
# replica is upgraded, then enable it in a subsequent rollout.
|
||||
# ENABLE_SUBAGENT_COMPLETION_WAKEUPS=false
|
||||
|
||||
# Trusted event adapters enqueue through the shared durable trigger service. Mongo-backed
|
||||
# leases make its workers safe across replicas; successful delivery records expire after
|
||||
# 90 days, while dead letters remain available for explicit operator requeue.
|
||||
|
|
|
|||
|
|
@ -3,4 +3,5 @@
|
|||
- **Agent run envelope**: the versioned, JSON-safe request contract created after ingress authentication and protocol validation but before agent, provider, tool, or MCP initialization. It carries only the validated protocol payload and the minimum trusted principal identifiers. The execution host rehydrates all runtime state from those identifiers.
|
||||
- **Subagent thread**: a durable, view-only child conversation owned by one parent conversation and subagent identity. A parent agent may continue it by stable `threadId`; each continuation uses a fresh execution lease restored from the canonical child transcript. It is not an ordinary human-writable chat.
|
||||
- **Live subagent task owner**: the one API process holding a detached child execution, its abort controller, and its bounded control queue. Redis may route trusted poll/control envelopes to that owner, but it does not migrate or persist the executor; Mongo persists only the logical child thread and its continuation fence.
|
||||
- **Subagent completion wakeup**: a durable internal `continue` trigger pre-registered before detached child execution so a process crash cannot lose the wakeup. Delivery defers until the child's terminal transcript is persisted, targets the initiating agent and exact parent response branch, carries task metadata rather than child output, waits for the parent generation to settle, and starts the parent turn that collects the result through the existing task store.
|
||||
- **Theme definition**: a versioned, data-only description of LibreChat semantic colors and shared appearance roles, optionally specialized by light or dark mode. The theme module validates and resolves partial definitions against bundled defaults before adapters apply them. A theme definition does not contain arbitrary CSS, application behavior, or alternate feature layouts.
|
||||
|
|
|
|||
|
|
@ -519,6 +519,120 @@ describe('ResumableAgentController resume metadata', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('defers a trusted trigger resume while its parent generation is still active', async () => {
|
||||
const conversationId = 'conversation-123';
|
||||
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
||||
mockGenerationJobManager.getJob.mockResolvedValue({
|
||||
status: 'running',
|
||||
metadata: { userId: 'user-123' },
|
||||
});
|
||||
const initializeClient = jest.fn();
|
||||
const req = {
|
||||
_isAgentTrigger: true,
|
||||
user: { id: 'user-123' },
|
||||
body: {
|
||||
text: 'Collect the completed child.',
|
||||
messageId: 'wakeup-user-message',
|
||||
parentMessageId: 'persisted-response_',
|
||||
conversationId,
|
||||
clientRequestId: 'trigger_resume_1',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
},
|
||||
config: {},
|
||||
};
|
||||
const res = createResumableResponse();
|
||||
|
||||
await AgentController(req, res, jest.fn(), initializeClient, null);
|
||||
|
||||
expect(res.status).toHaveBeenCalledWith(409);
|
||||
expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ code: 'PARENT_NOT_READY' }));
|
||||
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
||||
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
||||
expect(initializeClient).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('labels a trigger parent-state lookup failure as provably pre-admission', async () => {
|
||||
const conversationId = 'conversation-123';
|
||||
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
||||
mockGenerationJobManager.getJob.mockRejectedValue(new Error('redis unavailable'));
|
||||
const initializeClient = jest.fn();
|
||||
const req = {
|
||||
_isAgentTrigger: true,
|
||||
user: { id: 'user-123' },
|
||||
body: {
|
||||
text: 'Collect the completed child.',
|
||||
messageId: 'wakeup-user-message',
|
||||
parentMessageId: 'persisted-response_',
|
||||
conversationId,
|
||||
clientRequestId: 'trigger_resume_1',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
},
|
||||
config: {},
|
||||
};
|
||||
const res = createResumableResponse();
|
||||
|
||||
await AgentController(req, res, jest.fn(), initializeClient, null);
|
||||
|
||||
expect(res.set).toHaveBeenCalledWith('Retry-After', '1');
|
||||
expect(res.status).toHaveBeenCalledWith(503);
|
||||
expect(res.json).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ code: 'PARENT_STATE_UNAVAILABLE' }),
|
||||
);
|
||||
expect(mockGenerationJobManager.claimGeneration).not.toHaveBeenCalled();
|
||||
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
||||
expect(initializeClient).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('deduplicates the active continuation whose admission response was lost', async () => {
|
||||
const conversationId = 'conversation-123';
|
||||
mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
|
||||
mockGenerationJobManager.getJob.mockResolvedValue({
|
||||
createdAt: 1000,
|
||||
status: 'requires_action',
|
||||
metadata: {
|
||||
userId: 'user-123',
|
||||
idempotencyClientRequestId: 'trigger_resume_1',
|
||||
},
|
||||
});
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue({
|
||||
claimed: false,
|
||||
existing: {
|
||||
streamId: conversationId,
|
||||
conversationId,
|
||||
claimedAt: 100,
|
||||
claimToken: 'existing-token',
|
||||
startedAt: 1000,
|
||||
},
|
||||
});
|
||||
const req = {
|
||||
_isAgentTrigger: true,
|
||||
user: { id: 'user-123' },
|
||||
body: {
|
||||
text: 'Collect the completed child.',
|
||||
messageId: 'wakeup-user-message',
|
||||
parentMessageId: 'persisted-response_',
|
||||
conversationId,
|
||||
clientRequestId: 'trigger_resume_1',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
},
|
||||
config: {},
|
||||
};
|
||||
const res = createResumableResponse();
|
||||
|
||||
await AgentController(req, res, jest.fn(), jest.fn(), null);
|
||||
|
||||
expect(res.status).not.toHaveBeenCalledWith(409);
|
||||
expect(res.json).toHaveBeenCalledWith({
|
||||
streamId: conversationId,
|
||||
conversationId,
|
||||
generationCreatedAt: 1000,
|
||||
status: 'resumed',
|
||||
generationProtocolVersion: 1,
|
||||
});
|
||||
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
|
||||
expect(mockCheckAndIncrementPendingRequest).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('creates the job with the in-flight turn before MCP initialization can emit OAuth', async () => {
|
||||
const conversationId = 'conversation-123';
|
||||
const initializeClient = jest.fn().mockRejectedValue(new Error('stop before tool loading'));
|
||||
|
|
|
|||
|
|
@ -301,6 +301,7 @@ function rejectPreliminaryParentMessageId(res, generationProtocolVersion) {
|
|||
res,
|
||||
409,
|
||||
{
|
||||
code: 'PARENT_NOT_READY',
|
||||
error:
|
||||
'Cannot submit a follow-up while the selected parent response is still being saved. Please wait and try again.',
|
||||
},
|
||||
|
|
@ -308,6 +309,18 @@ function rejectPreliminaryParentMessageId(res, generationProtocolVersion) {
|
|||
);
|
||||
}
|
||||
|
||||
function rejectMissingTriggerParentMessageId(res, generationProtocolVersion) {
|
||||
return sendGenerationJson(
|
||||
res,
|
||||
404,
|
||||
{
|
||||
code: 'PARENT_NOT_FOUND',
|
||||
error: 'The selected parent response is no longer available.',
|
||||
},
|
||||
generationProtocolVersion,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resumable Agent Controller - Generation runs independently of HTTP connection.
|
||||
* Returns streamId immediately, client subscribes separately via SSE.
|
||||
|
|
@ -449,6 +462,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
: undefined,
|
||||
});
|
||||
|
||||
const isTriggerContinuation =
|
||||
req._isAgentTrigger === true && !isNewConvo && parentMessageId !== Constants.NO_PARENT;
|
||||
|
||||
if (
|
||||
await isUnpersistedPreliminaryParent({
|
||||
userId,
|
||||
|
|
@ -457,6 +473,38 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
getMessages,
|
||||
})
|
||||
) {
|
||||
if (isTriggerContinuation) {
|
||||
let parentJob;
|
||||
try {
|
||||
parentJob = await GenerationJobManager.getJob(conversationId);
|
||||
} catch (error) {
|
||||
logger.warn('[ResumableAgentController] Trigger parent lookup failed', error);
|
||||
res.set('Retry-After', '1');
|
||||
startupTelemetry?.end('rejected');
|
||||
return sendGenerationJson(
|
||||
res,
|
||||
503,
|
||||
{ code: 'PARENT_STATE_UNAVAILABLE', error: 'Parent generation state is unavailable.' },
|
||||
generationProtocolVersion,
|
||||
);
|
||||
}
|
||||
if (
|
||||
parentJob != null &&
|
||||
liveJobBelongsToRequester(parentJob, req.user) &&
|
||||
(parentJob.status === 'running' ||
|
||||
parentJob.status === 'requires_action' ||
|
||||
parentJob.metadata?.terminalPersistencePending === true) &&
|
||||
!(
|
||||
typeof clientRequestId === 'string' &&
|
||||
parentJob.metadata?.idempotencyClientRequestId === clientRequestId
|
||||
)
|
||||
) {
|
||||
startupTelemetry?.end('rejected');
|
||||
return rejectPreliminaryParentMessageId(res, generationProtocolVersion);
|
||||
}
|
||||
startupTelemetry?.end('rejected');
|
||||
return rejectMissingTriggerParentMessageId(res, generationProtocolVersion);
|
||||
}
|
||||
startupTelemetry?.end('rejected');
|
||||
return rejectPreliminaryParentMessageId(res, generationProtocolVersion);
|
||||
}
|
||||
|
|
@ -472,6 +520,51 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
const streamId = conversationId;
|
||||
req.body.conversationId = conversationId;
|
||||
|
||||
/** A durable continuation trigger appends below a completed parent response. If
|
||||
* that response belongs to a still-running or paused generation, admitting
|
||||
* another generation on the same conversation stream would replace it.
|
||||
* Defer without claiming the continuation idempotency key so the delivery engine
|
||||
* can retry after the parent reaches a terminal state. */
|
||||
if (isTriggerContinuation) {
|
||||
let parentJob;
|
||||
try {
|
||||
parentJob = await GenerationJobManager.getJob(streamId);
|
||||
} catch (error) {
|
||||
logger.warn('[ResumableAgentController] Trigger continuation parent lookup failed', error);
|
||||
res.set('Retry-After', '1');
|
||||
startupTelemetry?.end('rejected');
|
||||
return sendGenerationJson(
|
||||
res,
|
||||
503,
|
||||
{
|
||||
code: 'PARENT_STATE_UNAVAILABLE',
|
||||
error: 'Parent generation state is temporarily unavailable.',
|
||||
},
|
||||
generationProtocolVersion,
|
||||
);
|
||||
}
|
||||
if (
|
||||
parentJob != null &&
|
||||
liveJobBelongsToRequester(parentJob, req.user) &&
|
||||
(parentJob.status === 'running' ||
|
||||
parentJob.status === 'requires_action' ||
|
||||
parentJob.metadata?.terminalPersistencePending === true) &&
|
||||
!(
|
||||
typeof clientRequestId === 'string' &&
|
||||
parentJob.metadata?.idempotencyClientRequestId === clientRequestId
|
||||
)
|
||||
) {
|
||||
res.set('Retry-After', '1');
|
||||
startupTelemetry?.end('rejected');
|
||||
return sendGenerationJson(
|
||||
res,
|
||||
409,
|
||||
{ code: 'PARENT_NOT_READY', error: 'The parent generation has not settled yet.' },
|
||||
generationProtocolVersion,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Idempotency: a lost/reset start-generation response makes the client re-POST the
|
||||
// identical payload, which would otherwise start a second fully-billed generation.
|
||||
// Claim the submission's clientRequestId before creating the job so a retry attaches
|
||||
|
|
@ -881,6 +974,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
...(recoveredSteerId && { recoveredSteerId }),
|
||||
...(recoveredSteerPayload && { recoveredSteerPayload }),
|
||||
...(expectedPredecessorCreatedAt != null && { expectedPredecessorCreatedAt }),
|
||||
...(isTriggerContinuation && { rejectActivePredecessor: true }),
|
||||
...(ownedIdempotencyClaim?.claimToken && {
|
||||
idempotencyClientRequestId: clientRequestId,
|
||||
idempotencyClaimToken: ownedIdempotencyClaim.claimToken,
|
||||
|
|
@ -1908,31 +2002,44 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
if (error?.code === 'GENERATION_PREDECESSOR_MISMATCH') {
|
||||
const currentJob = error.currentJob;
|
||||
const currentStatus = currentJob?.status;
|
||||
const predecessorVerified =
|
||||
currentJob != null &&
|
||||
Number.isSafeInteger(currentJob.createdAt) &&
|
||||
currentJob.createdAt >= 0 &&
|
||||
currentJob.verified !== false;
|
||||
sendGenerationJson(
|
||||
res,
|
||||
409,
|
||||
{
|
||||
status: 'predecessor_mismatch',
|
||||
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
||||
error: predecessorVerified
|
||||
? 'A newer generation became current before this request could start.'
|
||||
: 'The prior generation could not be verified. Please retry.',
|
||||
streamId,
|
||||
conversationId: currentJob?.conversationId ?? conversationId,
|
||||
generationCreatedAt: currentJob?.createdAt,
|
||||
predecessorVerified,
|
||||
active:
|
||||
typeof currentJob?.active === 'boolean'
|
||||
? currentJob.active
|
||||
: currentStatus === 'running' || currentStatus === 'requires_action',
|
||||
},
|
||||
generationProtocolVersion,
|
||||
);
|
||||
if (isTriggerContinuation && currentJob?.active === true) {
|
||||
res.set('Retry-After', '1');
|
||||
sendGenerationJson(
|
||||
res,
|
||||
409,
|
||||
{
|
||||
code: 'PARENT_NOT_READY',
|
||||
error: 'Another generation became active before the continuation could start.',
|
||||
},
|
||||
generationProtocolVersion,
|
||||
);
|
||||
} else {
|
||||
const predecessorVerified =
|
||||
currentJob != null &&
|
||||
Number.isSafeInteger(currentJob.createdAt) &&
|
||||
currentJob.createdAt >= 0 &&
|
||||
currentJob.verified !== false;
|
||||
sendGenerationJson(
|
||||
res,
|
||||
409,
|
||||
{
|
||||
status: 'predecessor_mismatch',
|
||||
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
||||
error: predecessorVerified
|
||||
? 'A newer generation became current before this request could start.'
|
||||
: 'The prior generation could not be verified. Please retry.',
|
||||
streamId,
|
||||
conversationId: currentJob?.conversationId ?? conversationId,
|
||||
generationCreatedAt: currentJob?.createdAt,
|
||||
predecessorVerified,
|
||||
active:
|
||||
typeof currentJob?.active === 'boolean'
|
||||
? currentJob.active
|
||||
: currentStatus === 'running' || currentStatus === 'requires_action',
|
||||
},
|
||||
generationProtocolVersion,
|
||||
);
|
||||
}
|
||||
} else if (error?.code === 'RECOVERY_PAYLOAD_MISMATCH') {
|
||||
sendGenerationJson(
|
||||
res,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,17 @@
|
|||
const { createAgentTriggerService } = require('@librechat/api');
|
||||
const {
|
||||
createAgentTriggerService,
|
||||
createSubagentCompletionWakeupResolver,
|
||||
GenerationJobManager,
|
||||
} = require('@librechat/api');
|
||||
const methods = require('~/models');
|
||||
|
||||
const service = createAgentTriggerService({
|
||||
methods,
|
||||
isPrincipalActive: methods.isAgentTriggerPrincipalActive,
|
||||
prepareContinue: createSubagentCompletionWakeupResolver({
|
||||
methods,
|
||||
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
|
||||
}),
|
||||
});
|
||||
|
||||
module.exports = {
|
||||
|
|
|
|||
|
|
@ -1,12 +1,20 @@
|
|||
const {
|
||||
cacheConfig,
|
||||
ioredisClient,
|
||||
isEnabled,
|
||||
registerShutdownTask,
|
||||
duplicateIoRedisClient,
|
||||
createSubagentThreadTaskStore,
|
||||
createSubagentCompletionWakeupHandler,
|
||||
RedisSubagentTaskControlTransport,
|
||||
} = require('@librechat/api');
|
||||
const db = require('~/models');
|
||||
const { enqueueAgentTrigger } = require('../../Agents/triggers');
|
||||
|
||||
/** Keep producers off for the first rollout so older trigger workers cannot
|
||||
* permanently reject the new `continue` envelope. Enable only after every API
|
||||
* replica runs a release that understands completion wakeups. */
|
||||
const completionWakeupsEnabled = isEnabled(process.env.ENABLE_SUBAGENT_COMPLETION_WAKEUPS);
|
||||
|
||||
/** Durable logical threads use normal LibreChat conversations/messages. Mongo
|
||||
* fences continuation; optional Redis routing reaches the live owning process. */
|
||||
|
|
@ -14,6 +22,7 @@ const subagentThreadTaskStore = createSubagentThreadTaskStore(
|
|||
{
|
||||
acquireSubagentThreadLease: db.acquireSubagentThreadLease,
|
||||
claimSubagentTaskResult: db.claimSubagentTaskResult,
|
||||
releaseSubagentTaskResultClaim: db.releaseSubagentTaskResultClaim,
|
||||
countActiveSubagentThreadLeases: db.countActiveSubagentThreadLeases,
|
||||
deleteConvos: db.deleteConvos,
|
||||
deleteMessages: db.deleteMessages,
|
||||
|
|
@ -31,6 +40,9 @@ const subagentThreadTaskStore = createSubagentThreadTaskStore(
|
|||
fenceOwnerAdmission: db.fenceSubagentAdmission,
|
||||
renewOwnerAdmission: db.renewSubagentAdmission,
|
||||
releaseOwnerAdmission: db.releaseSubagentAdmission,
|
||||
...(completionWakeupsEnabled && {
|
||||
onTaskPrepared: createSubagentCompletionWakeupHandler(enqueueAgentTrigger),
|
||||
}),
|
||||
},
|
||||
);
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ export * from './skills';
|
|||
export * from './phases';
|
||||
export * from './startup';
|
||||
export * from './subagentThreads';
|
||||
export * from './subagentCompletionWakeup';
|
||||
export * from './subagentTaskRouting';
|
||||
export * from './skillConfigurable';
|
||||
export * from './skillFiles';
|
||||
|
|
|
|||
483
packages/api/src/agents/subagentCompletionWakeup.spec.ts
Normal file
483
packages/api/src/agents/subagentCompletionWakeup.spec.ts
Normal file
|
|
@ -0,0 +1,483 @@
|
|||
import type { IMessage } from '@librechat/data-schemas';
|
||||
import type { AgentContinueTriggerEnvelope } from './triggers/envelope';
|
||||
import type { SubagentTaskWakeupRegistration } from './subagentThreads';
|
||||
import type { EnqueueAgentTrigger } from './subagentCompletionWakeup';
|
||||
import {
|
||||
createAgentTriggerEnvelope,
|
||||
getAgentTriggerIdempotencyKey,
|
||||
parseAgentTriggerEnvelope,
|
||||
} from './triggers/envelope';
|
||||
import {
|
||||
createSubagentCompletionWakeupHandler,
|
||||
createSubagentCompletionWakeupResolver,
|
||||
} from './subagentCompletionWakeup';
|
||||
|
||||
const NOW = 1_775_000_000_000;
|
||||
|
||||
function enqueueMock(): jest.MockedFunction<EnqueueAgentTrigger> {
|
||||
return jest.fn<ReturnType<EnqueueAgentTrigger>, Parameters<EnqueueAgentTrigger>>(async () => ({
|
||||
id: 'delivery-1',
|
||||
}));
|
||||
}
|
||||
|
||||
function registration(
|
||||
overrides: Partial<SubagentTaskWakeupRegistration> = {},
|
||||
): SubagentTaskWakeupRegistration {
|
||||
return {
|
||||
userId: 'user-1',
|
||||
tenantId: 'tenant-1',
|
||||
parentConversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
parentAgentId: 'agent_parent_1',
|
||||
taskId: 'task-1',
|
||||
threadId: 'thread-1',
|
||||
subagentType: 'researcher',
|
||||
createdAt: NOW - 10,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('createSubagentCompletionWakeupHandler', () => {
|
||||
beforeEach(() => {
|
||||
jest.useFakeTimers().setSystemTime(NOW);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
jest.useRealTimers();
|
||||
});
|
||||
|
||||
it('pre-registers a bounded continuation on the exact parent branch', async () => {
|
||||
const enqueue = enqueueMock();
|
||||
const notify = createSubagentCompletionWakeupHandler(enqueue);
|
||||
|
||||
await notify(registration());
|
||||
|
||||
expect(enqueue).toHaveBeenCalledTimes(1);
|
||||
const [envelopeValue, options] = enqueue.mock.calls[0]!;
|
||||
const envelope = parseAgentTriggerEnvelope(envelopeValue);
|
||||
expect(envelope).toMatchObject({
|
||||
version: 1,
|
||||
mode: 'continue',
|
||||
principal: { userId: 'user-1', tenantId: 'tenant-1' },
|
||||
target: {
|
||||
agentId: 'agent_parent_1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
},
|
||||
event: {
|
||||
id: 'task-1',
|
||||
type: 'subagent.completion',
|
||||
source: { id: 'subagent-completion', type: 'internal' },
|
||||
payload: {
|
||||
taskId: 'task-1',
|
||||
threadId: 'thread-1',
|
||||
subagentType: 'researcher',
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(envelope.input).toContain('waiting to complete');
|
||||
expect(options).toEqual({
|
||||
orderingKey: 'subagent-completion:conversation-1',
|
||||
availableAt: new Date(NOW + 250),
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps one idempotency identity across duplicate registration callbacks', async () => {
|
||||
const enqueue = enqueueMock();
|
||||
const notify = createSubagentCompletionWakeupHandler(enqueue);
|
||||
const event = registration();
|
||||
|
||||
await notify(event);
|
||||
await notify(event);
|
||||
|
||||
const first = parseAgentTriggerEnvelope(enqueue.mock.calls[0]![0]);
|
||||
const retry = parseAgentTriggerEnvelope(enqueue.mock.calls[1]![0]);
|
||||
expect(first.requestId).not.toBe(retry.requestId);
|
||||
expect(first.deliveryId).toBe('task-1');
|
||||
expect(getAgentTriggerIdempotencyKey(first)).toBe(getAgentTriggerIdempotencyKey(retry));
|
||||
expect(first.input).toContain('waiting to complete');
|
||||
});
|
||||
|
||||
it('does not enqueue without a stable initiating agent', async () => {
|
||||
const enqueue = enqueueMock();
|
||||
const notify = createSubagentCompletionWakeupHandler(enqueue);
|
||||
|
||||
await notify(registration({ parentAgentId: undefined }));
|
||||
|
||||
expect(enqueue).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not enqueue for an ephemeral initiating agent', async () => {
|
||||
const enqueue = enqueueMock();
|
||||
const notify = createSubagentCompletionWakeupHandler(enqueue);
|
||||
|
||||
await notify(registration({ parentAgentId: 'openAI__gpt-4o___GPT-4o____1' }));
|
||||
|
||||
expect(enqueue).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
function wakeupEnvelope(): AgentContinueTriggerEnvelope {
|
||||
const envelope = createAgentTriggerEnvelope({
|
||||
mode: 'continue',
|
||||
requestId: 'request-1',
|
||||
deliveryId: 'task-1',
|
||||
receivedAt: NOW,
|
||||
principal: { id: 'user-1', tenantId: 'tenant-1' },
|
||||
event: {
|
||||
id: 'task-1',
|
||||
type: 'subagent.completion',
|
||||
occurredAt: NOW,
|
||||
source: { id: 'subagent-completion', type: 'internal' },
|
||||
payload: { taskId: 'task-1', threadId: 'thread-1', subagentType: 'researcher' },
|
||||
},
|
||||
target: {
|
||||
agentId: 'agent_parent_1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
},
|
||||
input: 'pending',
|
||||
});
|
||||
if (envelope.mode !== 'continue') {
|
||||
throw new Error('Expected a continue envelope.');
|
||||
}
|
||||
return envelope;
|
||||
}
|
||||
|
||||
function resolverMethods() {
|
||||
const subagentTask: IMessage['subagentTask'] = {
|
||||
attemptKey: 'attempt-1',
|
||||
parentRunId: 'response-1',
|
||||
status: 'completed',
|
||||
};
|
||||
const terminal = {
|
||||
messageId: 'task-1:assistant',
|
||||
conversationId: 'thread-1',
|
||||
parentMessageId: 'task-1:user',
|
||||
sender: 'researcher',
|
||||
text: 'Child result',
|
||||
isCreatedByUser: false,
|
||||
createdAt: new Date(NOW),
|
||||
updatedAt: new Date(NOW),
|
||||
subagentTask,
|
||||
};
|
||||
const methods = {
|
||||
getConvo: jest.fn(async (_userId: string, conversationId: string) =>
|
||||
conversationId === 'conversation-1'
|
||||
? { conversationId, tenantId: 'tenant-1' }
|
||||
: {
|
||||
conversationId,
|
||||
tenantId: 'tenant-1',
|
||||
subagentThread: {
|
||||
parentConversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
parentAgentId: 'agent_parent_1',
|
||||
subagentType: 'researcher',
|
||||
},
|
||||
},
|
||||
),
|
||||
getMessages: jest.fn(async (filter: { conversationId: string }) =>
|
||||
filter.conversationId === 'conversation-1'
|
||||
? [
|
||||
{
|
||||
messageId: 'response-1',
|
||||
parentMessageId: 'user-1',
|
||||
isCreatedByUser: false,
|
||||
createdAt: new Date(NOW - 30),
|
||||
},
|
||||
{
|
||||
messageId: 'wakeup-user',
|
||||
parentMessageId: 'response-1',
|
||||
isCreatedByUser: true,
|
||||
createdAt: new Date(NOW - 20),
|
||||
},
|
||||
{
|
||||
messageId: 'wakeup-response',
|
||||
parentMessageId: 'wakeup-user',
|
||||
isCreatedByUser: false,
|
||||
createdAt: new Date(NOW - 10),
|
||||
},
|
||||
]
|
||||
: [
|
||||
{
|
||||
messageId: 'task-1:user',
|
||||
conversationId: 'thread-1',
|
||||
isCreatedByUser: true,
|
||||
},
|
||||
terminal,
|
||||
],
|
||||
),
|
||||
claimSubagentTaskResult: jest.fn(async () => ({ status: 'acquired', message: terminal })),
|
||||
releaseSubagentTaskResultClaim: jest.fn(async () => true),
|
||||
};
|
||||
return { methods, terminal };
|
||||
}
|
||||
|
||||
describe('createSubagentCompletionWakeupResolver', () => {
|
||||
it('defers without claiming while the parent generation is active', async () => {
|
||||
const { methods } = resolverMethods();
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => ({ status: 'running' }),
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).rejects.toMatchObject({
|
||||
code: 'PARENT_NOT_READY',
|
||||
retryable: true,
|
||||
deferWithoutAttempt: true,
|
||||
});
|
||||
expect(methods.claimSubagentTaskResult).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('lets a lost-receipt retry reach HTTP dedup for its own active continuation', async () => {
|
||||
const { methods } = resolverMethods();
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => ({
|
||||
status: 'requires_action',
|
||||
metadata: { idempotencyClientRequestId: 'trigger_claim_1' },
|
||||
}),
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).resolves.toMatchObject({ status: 'ready' });
|
||||
});
|
||||
|
||||
it('bounds a persisted child result before rendering model input', async () => {
|
||||
const { methods, terminal } = resolverMethods();
|
||||
terminal.text = 'x'.repeat(200_000);
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
const prepared = await resolve(wakeupEnvelope(), {
|
||||
idempotencyKey: 'trigger_claim_1',
|
||||
} as never);
|
||||
|
||||
expect(prepared).toMatchObject({ status: 'ready' });
|
||||
expect(prepared?.status === 'ready' && prepared.input.length).toBeLessThan(110_000);
|
||||
});
|
||||
|
||||
it('dead-letters a child whose process disappeared after the task timeout grace', async () => {
|
||||
const { methods } = resolverMethods();
|
||||
methods.getMessages.mockImplementation(async (filter: { conversationId: string }) =>
|
||||
filter.conversationId === 'conversation-1'
|
||||
? [
|
||||
{
|
||||
messageId: 'response-1',
|
||||
parentMessageId: 'user-1',
|
||||
isCreatedByUser: false,
|
||||
createdAt: new Date(NOW - 30),
|
||||
},
|
||||
]
|
||||
: [
|
||||
{
|
||||
messageId: 'task-1:user',
|
||||
conversationId: 'thread-1',
|
||||
isCreatedByUser: true,
|
||||
},
|
||||
],
|
||||
);
|
||||
const fresh = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
now: () => NOW + 60_000,
|
||||
});
|
||||
const stale = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
now: () => NOW + 36 * 60_000,
|
||||
});
|
||||
|
||||
await expect(
|
||||
fresh(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).rejects.toMatchObject({
|
||||
code: 'CHILD_NOT_READY',
|
||||
retryable: true,
|
||||
deferWithoutAttempt: true,
|
||||
});
|
||||
await expect(
|
||||
stale(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).rejects.toMatchObject({ code: 'CHILD_TASK_ABANDONED', retryable: false, status: 410 });
|
||||
expect(methods.claimSubagentTaskResult).not.toHaveBeenCalled();
|
||||
expect(
|
||||
methods.getMessages.mock.calls.filter(
|
||||
([filter]) => filter.conversationId === 'conversation-1',
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('resolves a crash-retry terminal by logical attempt without blocking its ordered lane', async () => {
|
||||
const { methods, terminal } = resolverMethods();
|
||||
const supersedingTerminal = {
|
||||
...terminal,
|
||||
messageId: 'task-2:assistant',
|
||||
parentMessageId: 'task-1:user',
|
||||
};
|
||||
methods.getMessages.mockImplementation(async (filter: Record<string, unknown>) => {
|
||||
if (filter.conversationId === 'conversation-1') {
|
||||
return [
|
||||
{
|
||||
messageId: 'response-1',
|
||||
parentMessageId: 'user-1',
|
||||
isCreatedByUser: false,
|
||||
createdAt: new Date(NOW - 30),
|
||||
},
|
||||
];
|
||||
}
|
||||
if (filter['subagentTask.attemptKey'] === 'attempt-1') {
|
||||
return [supersedingTerminal];
|
||||
}
|
||||
return [
|
||||
{
|
||||
messageId: 'task-1:user',
|
||||
conversationId: 'thread-1',
|
||||
isCreatedByUser: true,
|
||||
subagentTask: {
|
||||
attemptKey: 'attempt-1',
|
||||
parentRunId: 'response-1',
|
||||
status: 'running',
|
||||
},
|
||||
},
|
||||
];
|
||||
});
|
||||
methods.claimSubagentTaskResult.mockResolvedValueOnce({
|
||||
status: 'acquired',
|
||||
message: supersedingTerminal,
|
||||
});
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
const prepared = await resolve(wakeupEnvelope(), {
|
||||
idempotencyKey: 'trigger_claim_1',
|
||||
} as never);
|
||||
|
||||
expect(prepared).toMatchObject({
|
||||
status: 'ready',
|
||||
input: expect.stringContaining('"background_task_id":"task-2"'),
|
||||
});
|
||||
expect(methods.claimSubagentTaskResult).toHaveBeenCalledWith({
|
||||
userId: 'user-1',
|
||||
conversationId: 'thread-1',
|
||||
taskId: 'task-2',
|
||||
kind: 'wakeup',
|
||||
claimId: 'trigger_claim_1',
|
||||
});
|
||||
});
|
||||
|
||||
it('validates a continued child against its per-task parent instead of original lineage', async () => {
|
||||
const { methods } = resolverMethods();
|
||||
methods.getConvo.mockImplementation(async (_userId: string, conversationId: string) =>
|
||||
conversationId === 'conversation-1'
|
||||
? { conversationId, tenantId: 'tenant-1' }
|
||||
: {
|
||||
conversationId,
|
||||
tenantId: 'tenant-1',
|
||||
subagentThread: {
|
||||
parentConversationId: 'conversation-1',
|
||||
parentMessageId: 'original-response',
|
||||
parentAgentId: 'agent_parent_1',
|
||||
subagentType: 'researcher',
|
||||
},
|
||||
},
|
||||
);
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).resolves.toMatchObject({ status: 'ready' });
|
||||
});
|
||||
|
||||
it('claims the durable result and chains onto the latest assistant descendant', async () => {
|
||||
const { methods } = resolverMethods();
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).resolves.toMatchObject({
|
||||
status: 'ready',
|
||||
parentMessageId: 'wakeup-response',
|
||||
input: expect.stringContaining('Child result'),
|
||||
});
|
||||
expect(methods.claimSubagentTaskResult).toHaveBeenCalledWith({
|
||||
userId: 'user-1',
|
||||
conversationId: 'thread-1',
|
||||
taskId: 'task-1',
|
||||
kind: 'wakeup',
|
||||
claimId: 'trigger_claim_1',
|
||||
});
|
||||
|
||||
const prepared = await resolve(wakeupEnvelope(), {
|
||||
idempotencyKey: 'trigger_claim_1',
|
||||
} as never);
|
||||
expect(prepared?.status).toBe('ready');
|
||||
if (prepared?.status === 'ready') {
|
||||
await prepared.releaseOnDefiniteFailure?.();
|
||||
}
|
||||
expect(methods.releaseSubagentTaskResultClaim).toHaveBeenCalledWith({
|
||||
userId: 'user-1',
|
||||
conversationId: 'thread-1',
|
||||
taskId: 'task-1',
|
||||
kind: 'wakeup',
|
||||
claimId: 'trigger_claim_1',
|
||||
});
|
||||
});
|
||||
|
||||
it('settles without starting a turn when a manual poll already claimed the result', async () => {
|
||||
const { methods, terminal } = resolverMethods();
|
||||
methods.claimSubagentTaskResult.mockResolvedValueOnce({
|
||||
status: 'claimed',
|
||||
message: {
|
||||
...terminal,
|
||||
subagentTask: {
|
||||
...terminal.subagentTask,
|
||||
resultClaim: { kind: 'manual', claimId: 'poll-1', claimedAt: new Date(NOW) },
|
||||
},
|
||||
},
|
||||
});
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).resolves.toEqual({ status: 'settled' });
|
||||
});
|
||||
|
||||
it('releases a cancelled wakeup result for later explicit collection', async () => {
|
||||
const { methods, terminal } = resolverMethods();
|
||||
terminal.subagentTask = { ...terminal.subagentTask!, status: 'cancelled' };
|
||||
methods.claimSubagentTaskResult.mockResolvedValueOnce({
|
||||
status: 'acquired',
|
||||
message: terminal,
|
||||
});
|
||||
const resolve = createSubagentCompletionWakeupResolver({
|
||||
methods: methods as never,
|
||||
getGenerationJob: async () => null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolve(wakeupEnvelope(), { idempotencyKey: 'trigger_claim_1' } as never),
|
||||
).resolves.toEqual({ status: 'settled' });
|
||||
expect(methods.releaseSubagentTaskResultClaim).toHaveBeenCalledWith({
|
||||
userId: 'user-1',
|
||||
conversationId: 'thread-1',
|
||||
taskId: 'task-1',
|
||||
kind: 'wakeup',
|
||||
claimId: 'trigger_claim_1',
|
||||
});
|
||||
});
|
||||
});
|
||||
425
packages/api/src/agents/subagentCompletionWakeup.ts
Normal file
425
packages/api/src/agents/subagentCompletionWakeup.ts
Normal file
|
|
@ -0,0 +1,425 @@
|
|||
import { randomUUID } from 'node:crypto';
|
||||
import { isEphemeralAgentId } from 'librechat-data-provider';
|
||||
import type { ConversationMethods, IMessage, MessageMethods } from '@librechat/data-schemas';
|
||||
import type {
|
||||
AgentTriggerContinuePreparation,
|
||||
AgentTriggerExecutionHostDeps,
|
||||
} from './triggers/host';
|
||||
import type { SubagentTaskWakeupRegistration } from './subagentThreads';
|
||||
import type { AgentContinueTriggerEnvelope } from './triggers/envelope';
|
||||
import type { AgentTriggerDispatchContext } from './triggers/dispatch';
|
||||
import type { AgentTriggerEnqueueOptions } from './triggers/delivery';
|
||||
import { boundedSubagentTaskResult } from './subagentTaskRouting';
|
||||
import { createAgentTriggerEnvelope } from './triggers/envelope';
|
||||
import { AgentTriggerExecutionError } from './triggers/host';
|
||||
|
||||
const WAKEUP_ADMISSION_DELAY_MS = 250;
|
||||
/** SDK tasks time out after 30 minutes; this grace covers terminal persistence. */
|
||||
const CHILD_READY_WAIT_MS = 35 * 60_000;
|
||||
const SOURCE_ID = 'subagent-completion';
|
||||
const EVENT_TYPE = 'subagent.completion';
|
||||
const MESSAGE_SELECT = 'messageId parentMessageId isCreatedByUser createdAt';
|
||||
const TASK_SELECT =
|
||||
'messageId conversationId parentMessageId sender text error createdAt updatedAt +subagentTask';
|
||||
|
||||
export type EnqueueAgentTrigger = (
|
||||
envelope: unknown,
|
||||
options?: AgentTriggerEnqueueOptions,
|
||||
) => Promise<unknown>;
|
||||
|
||||
type WakeupMethods = Pick<ConversationMethods, 'getConvo'> &
|
||||
Pick<
|
||||
MessageMethods,
|
||||
'claimSubagentTaskResult' | 'getMessages' | 'releaseSubagentTaskResultClaim'
|
||||
>;
|
||||
|
||||
interface GenerationState {
|
||||
status?: unknown;
|
||||
metadata?: {
|
||||
idempotencyClientRequestId?: unknown;
|
||||
terminalPersistencePending?: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
export interface SubagentCompletionWakeupResolverDeps {
|
||||
methods: WakeupMethods;
|
||||
getGenerationJob: (conversationId: string) => Promise<GenerationState | null>;
|
||||
now?: () => number;
|
||||
}
|
||||
|
||||
function payloadRegistration(
|
||||
envelope: AgentContinueTriggerEnvelope,
|
||||
): Pick<SubagentTaskWakeupRegistration, 'taskId' | 'threadId' | 'subagentType'> | null | undefined {
|
||||
if (
|
||||
envelope.event.source.type !== 'internal' ||
|
||||
envelope.event.source.id !== SOURCE_ID ||
|
||||
envelope.event.type !== EVENT_TYPE
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const payload = envelope.event.payload;
|
||||
if (payload == null || typeof payload !== 'object' || Array.isArray(payload)) {
|
||||
return null;
|
||||
}
|
||||
const { taskId, threadId, subagentType } = payload;
|
||||
if (
|
||||
typeof taskId !== 'string' ||
|
||||
taskId.length === 0 ||
|
||||
taskId.length > 256 ||
|
||||
typeof threadId !== 'string' ||
|
||||
threadId.length === 0 ||
|
||||
threadId.length > 256 ||
|
||||
typeof subagentType !== 'string' ||
|
||||
subagentType.length === 0 ||
|
||||
subagentType.length > 256
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return { taskId, threadId, subagentType };
|
||||
}
|
||||
|
||||
function executionError(
|
||||
message: string,
|
||||
options: {
|
||||
code: string;
|
||||
retryable: boolean;
|
||||
deferWithoutAttempt?: boolean;
|
||||
status?: number;
|
||||
retryAfter?: string;
|
||||
},
|
||||
): AgentTriggerExecutionError {
|
||||
return new AgentTriggerExecutionError(message, {
|
||||
mode: 'continue',
|
||||
certainty: 'definite',
|
||||
...options,
|
||||
});
|
||||
}
|
||||
|
||||
function isParentActive(job: GenerationState | null): boolean {
|
||||
return (
|
||||
job?.status === 'running' ||
|
||||
job?.status === 'requires_action' ||
|
||||
job?.metadata?.terminalPersistencePending === true
|
||||
);
|
||||
}
|
||||
|
||||
function sameTenant(actual: string | undefined, expected: string | undefined): boolean {
|
||||
return actual === expected;
|
||||
}
|
||||
|
||||
function timestamp(message: Pick<IMessage, 'createdAt'>): number {
|
||||
const value = message.createdAt;
|
||||
if (value instanceof Date) {
|
||||
return value.getTime();
|
||||
}
|
||||
const parsed = value == null ? Number.NaN : new Date(value).getTime();
|
||||
return Number.isFinite(parsed) ? parsed : 0;
|
||||
}
|
||||
|
||||
/** Selects the newest persisted assistant on the branch below the original
|
||||
* parent. Re-resolving for every ordered delivery serializes sibling child
|
||||
* completions onto the branch produced by the preceding wakeup. */
|
||||
function latestAssistantDescendant(messages: IMessage[], anchorId: string): string | undefined {
|
||||
const byId = new Map(messages.map((message) => [message.messageId, message]));
|
||||
if (!byId.has(anchorId)) {
|
||||
return;
|
||||
}
|
||||
const memo = new Map<string, boolean>([[anchorId, true]]);
|
||||
const reachesAnchor = (message: IMessage, visiting = new Set<string>()): boolean => {
|
||||
const known = memo.get(message.messageId);
|
||||
if (known != null) {
|
||||
return known;
|
||||
}
|
||||
if (visiting.has(message.messageId)) {
|
||||
memo.set(message.messageId, false);
|
||||
return false;
|
||||
}
|
||||
visiting.add(message.messageId);
|
||||
const parent =
|
||||
typeof message.parentMessageId === 'string' ? byId.get(message.parentMessageId) : undefined;
|
||||
const reachable = parent != null && reachesAnchor(parent, visiting);
|
||||
visiting.delete(message.messageId);
|
||||
memo.set(message.messageId, reachable);
|
||||
return reachable;
|
||||
};
|
||||
const descendants = messages
|
||||
.filter((message) => message.isCreatedByUser === false && reachesAnchor(message))
|
||||
.sort((left, right) => {
|
||||
const time = timestamp(left) - timestamp(right);
|
||||
return time === 0 ? left.messageId.localeCompare(right.messageId) : time;
|
||||
});
|
||||
return descendants[descendants.length - 1]?.messageId;
|
||||
}
|
||||
|
||||
function renderWakeupInput(
|
||||
registration: Pick<SubagentTaskWakeupRegistration, 'threadId' | 'subagentType'>,
|
||||
resultTaskId: string,
|
||||
terminal: IMessage,
|
||||
): string {
|
||||
const status = terminal.subagentTask?.status ?? 'error';
|
||||
return [
|
||||
`A detached subagent task has ${status}. Continue the parent task using its durable result below.`,
|
||||
JSON.stringify({
|
||||
background_task_id: resultTaskId,
|
||||
subagent_thread_id: registration.threadId,
|
||||
subagent_type: registration.subagentType,
|
||||
status,
|
||||
result: boundedSubagentTaskResult(terminal.text ?? ''),
|
||||
}),
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
/** Resolves a pre-registered completion delivery immediately before dispatch.
|
||||
* The durable result claim elects exactly one consumer (manual poll or this
|
||||
* delivery), while the branch lookup chains ordered sibling completions. */
|
||||
export function createSubagentCompletionWakeupResolver({
|
||||
methods,
|
||||
getGenerationJob,
|
||||
now = Date.now,
|
||||
}: SubagentCompletionWakeupResolverDeps): NonNullable<
|
||||
AgentTriggerExecutionHostDeps['prepareContinue']
|
||||
> {
|
||||
return async (
|
||||
envelope: AgentContinueTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
): Promise<AgentTriggerContinuePreparation | undefined> => {
|
||||
const registration = payloadRegistration(envelope);
|
||||
if (registration === undefined) {
|
||||
return;
|
||||
}
|
||||
if (registration === null) {
|
||||
throw executionError('The subagent completion wakeup payload is invalid.', {
|
||||
code: 'INVALID_SUBAGENT_WAKEUP',
|
||||
retryable: false,
|
||||
});
|
||||
}
|
||||
|
||||
let parentJob: GenerationState | null;
|
||||
try {
|
||||
parentJob = await getGenerationJob(envelope.target.conversationId);
|
||||
} catch (error) {
|
||||
throw executionError(
|
||||
`Parent generation state is temporarily unavailable: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
{ code: 'PARENT_STATE_UNAVAILABLE', retryable: true },
|
||||
);
|
||||
}
|
||||
if (
|
||||
isParentActive(parentJob) &&
|
||||
parentJob?.metadata?.idempotencyClientRequestId !== context.idempotencyKey
|
||||
) {
|
||||
throw executionError('The parent generation has not settled yet.', {
|
||||
code: 'PARENT_NOT_READY',
|
||||
retryable: true,
|
||||
status: 409,
|
||||
retryAfter: '1',
|
||||
deferWithoutAttempt: true,
|
||||
});
|
||||
}
|
||||
|
||||
const userId = envelope.principal.userId;
|
||||
const tenantId = envelope.principal.tenantId;
|
||||
const [parent, child, taskMessages] = await Promise.all([
|
||||
methods.getConvo(userId, envelope.target.conversationId),
|
||||
methods.getConvo(userId, registration.threadId),
|
||||
methods.getMessages(
|
||||
{
|
||||
user: userId,
|
||||
conversationId: registration.threadId,
|
||||
messageId: { $in: [`${registration.taskId}:user`, `${registration.taskId}:assistant`] },
|
||||
},
|
||||
TASK_SELECT,
|
||||
{ sort: { createdAt: 1, _id: 1 } },
|
||||
),
|
||||
]);
|
||||
if (parent == null || !sameTenant(parent.tenantId, tenantId)) {
|
||||
throw executionError('The parent conversation is no longer available.', {
|
||||
code: 'PARENT_NOT_FOUND',
|
||||
retryable: false,
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
const lineage = child?.subagentThread;
|
||||
if (
|
||||
child == null ||
|
||||
!sameTenant(child.tenantId, tenantId) ||
|
||||
lineage?.parentConversationId !== envelope.target.conversationId ||
|
||||
lineage.parentAgentId !== envelope.target.agentId ||
|
||||
lineage.subagentType !== registration.subagentType
|
||||
) {
|
||||
throw executionError('The child task lineage is no longer available.', {
|
||||
code: 'CHILD_TASK_MISSING',
|
||||
retryable: false,
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
let resultTaskId = registration.taskId;
|
||||
let terminal = taskMessages.find(
|
||||
(message) =>
|
||||
message.messageId === `${registration.taskId}:assistant` &&
|
||||
message.subagentTask?.status !== 'running',
|
||||
);
|
||||
const started = taskMessages.find(
|
||||
(message) => message.messageId === `${registration.taskId}:user`,
|
||||
);
|
||||
/** A worker can persist the input, lose its lease, and then have a retry
|
||||
* close the same logical attempt under the retry's runtime task id. Resolve
|
||||
* that terminal by the durable attempt identity so the earlier ordered
|
||||
* delivery cannot block the repaired delivery behind it for the full
|
||||
* abandonment grace period. */
|
||||
if (terminal == null && started?.subagentTask?.attemptKey != null) {
|
||||
const [supersedingTerminal] = await methods.getMessages(
|
||||
{
|
||||
user: userId,
|
||||
conversationId: registration.threadId,
|
||||
'subagentTask.attemptKey': started.subagentTask.attemptKey,
|
||||
'subagentTask.status': { $in: ['completed', 'error', 'cancelled'] },
|
||||
},
|
||||
TASK_SELECT,
|
||||
{ sort: { createdAt: -1, _id: -1 }, limit: 1 },
|
||||
);
|
||||
if (supersedingTerminal?.messageId.endsWith(':assistant') === true) {
|
||||
terminal = supersedingTerminal;
|
||||
resultTaskId = supersedingTerminal.messageId.slice(0, -':assistant'.length);
|
||||
}
|
||||
}
|
||||
if (terminal == null) {
|
||||
if (started != null) {
|
||||
if (now() - envelope.event.occurredAt > CHILD_READY_WAIT_MS) {
|
||||
throw executionError('The child task owner disappeared before settlement.', {
|
||||
code: 'CHILD_TASK_ABANDONED',
|
||||
retryable: false,
|
||||
status: 410,
|
||||
});
|
||||
}
|
||||
throw executionError('The child task has not settled yet.', {
|
||||
code: 'CHILD_NOT_READY',
|
||||
retryable: true,
|
||||
status: 409,
|
||||
retryAfter: '1',
|
||||
deferWithoutAttempt: true,
|
||||
});
|
||||
}
|
||||
throw executionError('The child task no longer exists.', {
|
||||
code: 'CHILD_TASK_MISSING',
|
||||
retryable: false,
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
if (terminal.subagentTask?.parentRunId !== envelope.target.parentMessageId) {
|
||||
throw executionError('The child task lineage is no longer available.', {
|
||||
code: 'CHILD_TASK_MISSING',
|
||||
retryable: false,
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
|
||||
const parentMessages = await methods.getMessages(
|
||||
{ user: userId, conversationId: envelope.target.conversationId },
|
||||
MESSAGE_SELECT,
|
||||
{ sort: { createdAt: 1, _id: 1 } },
|
||||
);
|
||||
|
||||
const parentMessageId = latestAssistantDescendant(
|
||||
parentMessages,
|
||||
envelope.target.parentMessageId,
|
||||
);
|
||||
if (parentMessageId == null) {
|
||||
throw executionError('The parent conversation branch is no longer available.', {
|
||||
code: 'PARENT_NOT_FOUND',
|
||||
retryable: false,
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
|
||||
const claim = await methods.claimSubagentTaskResult({
|
||||
userId,
|
||||
conversationId: registration.threadId,
|
||||
taskId: resultTaskId,
|
||||
kind: 'wakeup',
|
||||
claimId: context.idempotencyKey,
|
||||
});
|
||||
if (claim.status !== 'acquired') {
|
||||
return { status: 'settled' };
|
||||
}
|
||||
if (claim.message.subagentTask?.status === 'cancelled') {
|
||||
const released = await methods.releaseSubagentTaskResultClaim({
|
||||
userId,
|
||||
conversationId: registration.threadId,
|
||||
taskId: resultTaskId,
|
||||
kind: 'wakeup',
|
||||
claimId: context.idempotencyKey,
|
||||
});
|
||||
if (!released) {
|
||||
throw executionError('The cancelled child result claim could not be released.', {
|
||||
code: 'RESULT_CLAIM_RELEASE_FAILED',
|
||||
retryable: true,
|
||||
});
|
||||
}
|
||||
return { status: 'settled' };
|
||||
}
|
||||
return {
|
||||
status: 'ready',
|
||||
parentMessageId,
|
||||
input: renderWakeupInput(registration, resultTaskId, claim.message),
|
||||
releaseOnDefiniteFailure: async () => {
|
||||
await methods.releaseSubagentTaskResultClaim({
|
||||
userId,
|
||||
conversationId: registration.threadId,
|
||||
taskId: resultTaskId,
|
||||
kind: 'wakeup',
|
||||
claimId: context.idempotencyKey,
|
||||
});
|
||||
},
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
/** Pre-registers the idempotent delivery before child provider work starts.
|
||||
* A process crash can therefore delay a wakeup but cannot lose it; dispatch
|
||||
* simply defers until the terminal child message exists. */
|
||||
export function createSubagentCompletionWakeupHandler(
|
||||
enqueue: EnqueueAgentTrigger,
|
||||
): (registration: SubagentTaskWakeupRegistration) => Promise<void> {
|
||||
return async (registration) => {
|
||||
const parentAgentId = registration.parentAgentId?.trim();
|
||||
if (parentAgentId == null || parentAgentId === '' || isEphemeralAgentId(parentAgentId)) {
|
||||
return;
|
||||
}
|
||||
const eventId = registration.taskId;
|
||||
const envelope = createAgentTriggerEnvelope({
|
||||
mode: 'continue',
|
||||
requestId: randomUUID(),
|
||||
deliveryId: eventId,
|
||||
receivedAt: Date.now(),
|
||||
principal: {
|
||||
id: registration.userId,
|
||||
...(registration.tenantId == null ? {} : { tenantId: registration.tenantId }),
|
||||
},
|
||||
event: {
|
||||
id: eventId,
|
||||
type: EVENT_TYPE,
|
||||
occurredAt: registration.createdAt,
|
||||
source: { id: SOURCE_ID, type: 'internal' },
|
||||
payload: {
|
||||
taskId: registration.taskId,
|
||||
threadId: registration.threadId,
|
||||
subagentType: registration.subagentType,
|
||||
},
|
||||
},
|
||||
target: {
|
||||
agentId: parentAgentId,
|
||||
conversationId: registration.parentConversationId,
|
||||
parentMessageId: registration.parentMessageId,
|
||||
},
|
||||
input: 'A detached subagent task is waiting to complete.',
|
||||
});
|
||||
await enqueue(envelope, {
|
||||
orderingKey: `subagent-completion:${registration.parentConversationId}`,
|
||||
availableAt: new Date(
|
||||
Math.max(Date.now(), registration.createdAt) + WAKEUP_ADMISSION_DELAY_MS,
|
||||
),
|
||||
});
|
||||
};
|
||||
}
|
||||
|
|
@ -289,6 +289,11 @@ export function boundedTaskList(tasks: SubagentTaskSnapshot[]): SubagentTaskSnap
|
|||
return [...keptRunning, ...settled.slice(-remaining)].sort(byCreatedAt);
|
||||
}
|
||||
|
||||
/** Applies the shared model-facing bound to a durable child result. */
|
||||
export function boundedSubagentTaskResult(result: string): string {
|
||||
return truncateMiddle(result, MAX_RESULT_CHARS);
|
||||
}
|
||||
|
||||
/** Applies the routed result and snapshot bounds to a claim from any source. */
|
||||
export function boundedClaim(claim: SubagentTaskClaim): SubagentTaskClaim {
|
||||
if (claim.status === 'not_found') {
|
||||
|
|
@ -296,7 +301,7 @@ export function boundedClaim(claim: SubagentTaskClaim): SubagentTaskClaim {
|
|||
}
|
||||
const task = boundedSnapshot(claim.task);
|
||||
if (claim.status === 'completed') {
|
||||
return { status: 'completed', task, result: truncateMiddle(claim.result, MAX_RESULT_CHARS) };
|
||||
return { status: 'completed', task, result: boundedSubagentTaskResult(claim.result) };
|
||||
}
|
||||
if (claim.status === 'error' || claim.status === 'cancelled') {
|
||||
return { status: claim.status, task, error: truncateMiddle(claim.error, MAX_ERROR_CHARS) };
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ import type {
|
|||
SubagentTaskControlHandler,
|
||||
SubagentTaskControlTransport,
|
||||
} from './subagentTaskRouting';
|
||||
import type { SubagentTaskWakeupRegistration } from './subagentThreads';
|
||||
import type { UsageMetadata } from '~/stream/interfaces/IJobStore';
|
||||
import {
|
||||
buildSubagentThreadTaskConfig,
|
||||
|
|
@ -285,6 +286,68 @@ describe('SubagentThreadTaskStore', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('registers a host-safe wakeup before child provider work begins', async () => {
|
||||
const userId = 'wakeup-user';
|
||||
const parentConversationId = randomUUID();
|
||||
await saveParent(userId, parentConversationId);
|
||||
const run = jest.fn(taskRequest('').run);
|
||||
const onTaskPrepared = jest.fn(async (registration: SubagentTaskWakeupRegistration) => {
|
||||
const messages = await methods.getMessages({
|
||||
user: userId,
|
||||
conversationId: registration.threadId,
|
||||
messageId: `${registration.taskId}:assistant`,
|
||||
});
|
||||
expect(messages).toHaveLength(0);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
const store = new SubagentThreadTaskStore(methods, { onTaskPrepared });
|
||||
const config = buildSubagentThreadTaskConfig(store, { userId, parentConversationId });
|
||||
const started = store.start(
|
||||
taskRequest(config.scopeId, {
|
||||
parentRunId: 'parent-response-1',
|
||||
parentAgentId: 'agent_parent_1',
|
||||
run,
|
||||
}),
|
||||
);
|
||||
await waitForSettled(store, config.scopeId, started);
|
||||
|
||||
const settledTask = store.get(config.scopeId, requireAccepted(started).task.taskId);
|
||||
expect(settledTask?.error).toBeUndefined();
|
||||
expect(settledTask).toMatchObject({
|
||||
status: 'completed',
|
||||
});
|
||||
expect(onTaskPrepared).toHaveBeenCalledWith({
|
||||
userId,
|
||||
parentConversationId,
|
||||
parentMessageId: 'parent-response-1',
|
||||
parentAgentId: 'agent_parent_1',
|
||||
taskId: requireAccepted(started).task.taskId,
|
||||
threadId: requireThreadId(started),
|
||||
subagentType: 'researcher-agent',
|
||||
createdAt: expect.any(Number),
|
||||
});
|
||||
});
|
||||
|
||||
it('fails before provider work and keeps the durable failure collectable when registration fails', async () => {
|
||||
const userId = 'wakeup-failure-user';
|
||||
const parentConversationId = randomUUID();
|
||||
await saveParent(userId, parentConversationId);
|
||||
const run = jest.fn(taskRequest('').run);
|
||||
const store = new SubagentThreadTaskStore(methods, {
|
||||
onTaskPrepared: async () => Promise.reject(new Error('trigger queue unavailable')),
|
||||
});
|
||||
const config = buildSubagentThreadTaskConfig(store, { userId, parentConversationId });
|
||||
const started = store.start(taskRequest(config.scopeId, { run }));
|
||||
await waitForSettled(store, config.scopeId, started);
|
||||
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
await expect(
|
||||
store.claimTask(config.scopeId, requireAccepted(started).task.taskId),
|
||||
).resolves.toMatchObject({
|
||||
status: 'error',
|
||||
});
|
||||
});
|
||||
|
||||
it('waits for initial parent persistence before creating the first child', async () => {
|
||||
const userId = 'parent-gate-user';
|
||||
const parentConversationId = randomUUID();
|
||||
|
|
@ -452,22 +515,35 @@ describe('SubagentThreadTaskStore', () => {
|
|||
const userId = 'durable-idempotency-user';
|
||||
const parentConversationId = randomUUID();
|
||||
await saveParent(userId, parentConversationId);
|
||||
const firstWorker = new SubagentThreadTaskStore(methods);
|
||||
const secondWorker = new SubagentThreadTaskStore(methods);
|
||||
const firstWakeup = jest.fn(async (_registration: SubagentTaskWakeupRegistration) => undefined);
|
||||
const replayWakeup = jest.fn(
|
||||
async (_registration: SubagentTaskWakeupRegistration) => undefined,
|
||||
);
|
||||
const firstWorker = new SubagentThreadTaskStore(methods, { onTaskPrepared: firstWakeup });
|
||||
const secondWorker = new SubagentThreadTaskStore(methods, { onTaskPrepared: replayWakeup });
|
||||
const config = buildSubagentThreadTaskConfig(firstWorker, { userId, parentConversationId });
|
||||
const firstRun = jest.fn(async () => ({
|
||||
content: 'Original durable result.',
|
||||
messages: [new HumanMessage('Run once.'), new AIMessage('Original durable result.')],
|
||||
}));
|
||||
const firstParentRunId = 'original-parent-response';
|
||||
const first = firstWorker.start(
|
||||
taskRequest(config.scopeId, {
|
||||
idempotencyKey: 'cross-worker-attempt',
|
||||
parentRunId: firstParentRunId,
|
||||
requestFingerprint: 'same-inputs',
|
||||
input: 'Run once.',
|
||||
run: firstRun,
|
||||
}),
|
||||
);
|
||||
await waitForSettled(firstWorker, config.scopeId, first);
|
||||
const durableAttempt = await methods.getMessages(
|
||||
{ user: userId, conversationId: requireThreadId(first) },
|
||||
'+subagentTask',
|
||||
);
|
||||
expect(durableAttempt[durableAttempt.length - 1]?.subagentTask?.parentRunId).toBe(
|
||||
firstParentRunId,
|
||||
);
|
||||
|
||||
const replayRun = jest.fn(taskRequest(config.scopeId).run);
|
||||
const replay = secondWorker.start(
|
||||
|
|
@ -483,6 +559,18 @@ describe('SubagentThreadTaskStore', () => {
|
|||
|
||||
expect(firstRun).toHaveBeenCalledTimes(1);
|
||||
expect(replayRun).not.toHaveBeenCalled();
|
||||
expect(firstWakeup).toHaveBeenCalledTimes(1);
|
||||
const firstRegistration = firstWakeup.mock.calls[0]?.[0];
|
||||
const replayRegistration = replayWakeup.mock.calls[0]?.[0];
|
||||
expect(firstRegistration?.createdAt).toBe(durableAttempt[0]?.createdAt?.getTime());
|
||||
expect(replayRegistration?.createdAt).toBe(firstRegistration?.createdAt);
|
||||
expect(replayWakeup).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
taskId: requireAccepted(first).task.taskId,
|
||||
parentMessageId: firstParentRunId,
|
||||
createdAt: firstRegistration?.createdAt,
|
||||
}),
|
||||
);
|
||||
expect(secondWorker.claim(config.scopeId, requireAccepted(replay).task.taskId)).toMatchObject({
|
||||
status: 'completed',
|
||||
result: 'Original durable result.',
|
||||
|
|
|
|||
|
|
@ -101,10 +101,14 @@ interface PreparedThread {
|
|||
initialMessages: BaseMessage[];
|
||||
initialStoredMessages: StoredMessage[];
|
||||
attemptKey: string;
|
||||
/** Stable source-occurrence time shared by first delivery and every replay. */
|
||||
taskCreatedAt: number;
|
||||
userMessageId?: string;
|
||||
replay?: {
|
||||
status: 'completed' | 'error' | 'cancelled';
|
||||
content: string;
|
||||
taskId: string;
|
||||
parentRunId: string;
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -141,6 +145,19 @@ export interface SubagentThreadTaskStoreOptions extends InMemorySubagentTaskStor
|
|||
fenceOwnerAdmission?: (userId: string, token: string, fencedUntil: Date) => Promise<void>;
|
||||
renewOwnerAdmission?: (userId: string, token: string, fencedUntil: Date) => Promise<boolean>;
|
||||
releaseOwnerAdmission?: (userId: string, token: string) => Promise<void>;
|
||||
onTaskPrepared?: (registration: SubagentTaskWakeupRegistration) => Promise<void> | void;
|
||||
}
|
||||
|
||||
export interface SubagentTaskWakeupRegistration {
|
||||
userId: string;
|
||||
parentConversationId: string;
|
||||
parentMessageId: string;
|
||||
parentAgentId?: string;
|
||||
tenantId?: string;
|
||||
taskId: string;
|
||||
threadId: string;
|
||||
subagentType: string;
|
||||
createdAt: number;
|
||||
}
|
||||
|
||||
function positiveInteger(value: number | undefined, fallback: number): number {
|
||||
|
|
@ -191,6 +208,14 @@ function matchesTenant(actual: string | undefined, expected: string | undefined)
|
|||
return actual === expected;
|
||||
}
|
||||
|
||||
function durableMessageTime(message: Pick<IMessage, 'createdAt'>, missingMessage: string): number {
|
||||
const value = message.createdAt?.getTime();
|
||||
if (!Number.isSafeInteger(value) || value == null || value < 0) {
|
||||
throw new Error(missingMessage);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function assertParentPersistence(
|
||||
value: unknown,
|
||||
scope: SubagentThreadScope,
|
||||
|
|
@ -387,6 +412,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
) => Promise<boolean>;
|
||||
|
||||
private readonly releaseOwnerAdmission?: (userId: string, token: string) => Promise<void>;
|
||||
private readonly onTaskPrepared?: SubagentThreadTaskStoreOptions['onTaskPrepared'];
|
||||
private taskControlTransport?: SubagentTaskControlTransport;
|
||||
|
||||
constructor(
|
||||
|
|
@ -418,6 +444,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
this.fenceOwnerAdmission = options.fenceOwnerAdmission;
|
||||
this.renewOwnerAdmission = options.renewOwnerAdmission;
|
||||
this.releaseOwnerAdmission = options.releaseOwnerAdmission;
|
||||
this.onTaskPrepared = options.onTaskPrepared;
|
||||
}
|
||||
|
||||
/** Enables optional cross-replica lookup after the host's Redis service is ready. */
|
||||
|
|
@ -496,6 +523,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
lease.taskId = runtime.taskId;
|
||||
lease.running = true;
|
||||
const detachedUsage: UsageMetadata[] = [];
|
||||
let prepared: PreparedThread | undefined;
|
||||
try {
|
||||
if (runtime.signal.aborted) {
|
||||
throw runtime.signal.reason ?? new Error('Subagent task was cancelled.');
|
||||
|
|
@ -510,7 +538,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
this.taskRoutingTtlMs,
|
||||
);
|
||||
await parentReady;
|
||||
const prepared = await this.prepareThread(
|
||||
prepared = await this.prepareThread(
|
||||
request.scopeId,
|
||||
scope,
|
||||
threadId,
|
||||
|
|
@ -519,6 +547,11 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
runtime.taskId,
|
||||
lease,
|
||||
);
|
||||
await this.registerTaskWakeup(scope, prepared.conversation.conversationId, request, {
|
||||
taskId: prepared.replay?.taskId ?? runtime.taskId,
|
||||
parentRunId: prepared.replay?.parentRunId ?? request.parentRunId,
|
||||
createdAt: prepared.taskCreatedAt,
|
||||
});
|
||||
if (runtime.signal.aborted) {
|
||||
throw runtime.signal.reason ?? new Error('Subagent task was cancelled.');
|
||||
}
|
||||
|
|
@ -533,8 +566,9 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
'This child thread is already being continued by another run.',
|
||||
);
|
||||
}
|
||||
const preparedThread = prepared;
|
||||
const result = await runWithDetachedSubagentUsage(detachedUsage, () =>
|
||||
request.run(runtime, prepared.initialMessages),
|
||||
request.run(runtime, preparedThread.initialMessages),
|
||||
);
|
||||
if (runtime.signal.aborted) {
|
||||
throw runtime.signal.reason ?? new Error('Subagent task was cancelled.');
|
||||
|
|
@ -555,6 +589,11 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
);
|
||||
return result;
|
||||
} catch (error) {
|
||||
/** A replay is already terminal in Mongo. A temporary wakeup-queue
|
||||
* outage must not overwrite that canonical result with a new error. */
|
||||
if (prepared?.replay != null) {
|
||||
throw error;
|
||||
}
|
||||
const mayPersist =
|
||||
lease.shared == null || (await this.renewSharedLease(scope, threadId, lease));
|
||||
const terminalTask = this.get(request.scopeId, runtime.taskId);
|
||||
|
|
@ -778,6 +817,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
userId,
|
||||
conversationId: threadId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: invocationId,
|
||||
});
|
||||
} catch (error) {
|
||||
|
|
@ -1553,13 +1593,29 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
.reverse()
|
||||
.find((message) => message.subagentTask?.status !== 'running');
|
||||
if (terminal?.subagentTask != null) {
|
||||
const canonicalTaskId = terminal.messageId.endsWith(':assistant')
|
||||
? terminal.messageId.slice(0, -':assistant'.length)
|
||||
: '';
|
||||
if (canonicalTaskId === '') {
|
||||
throw new Error('The prior subagent result has an invalid task identity.');
|
||||
}
|
||||
const canonicalStart = priorAttempt.find(
|
||||
(message) => message.messageId === `${canonicalTaskId}:user`,
|
||||
);
|
||||
const taskCreatedAt = durableMessageTime(
|
||||
canonicalStart ?? terminal,
|
||||
'The prior subagent result has no durable occurrence time.',
|
||||
);
|
||||
return {
|
||||
conversation,
|
||||
initialMessages: [],
|
||||
initialStoredMessages: [],
|
||||
attemptKey,
|
||||
taskCreatedAt,
|
||||
replay: {
|
||||
status: terminal.subagentTask.status as 'completed' | 'error' | 'cancelled',
|
||||
taskId: canonicalTaskId,
|
||||
parentRunId: terminal.subagentTask.parentRunId ?? request.parentRunId,
|
||||
content:
|
||||
terminal.text ??
|
||||
(terminal.subagentTask.status === 'completed'
|
||||
|
|
@ -1588,6 +1644,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
error: true,
|
||||
subagentTask: {
|
||||
attemptKey,
|
||||
parentRunId: request.parentRunId,
|
||||
...(requestFingerprint == null ? {} : { requestFingerprint }),
|
||||
status: 'error',
|
||||
},
|
||||
|
|
@ -1604,7 +1661,16 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
initialMessages: [],
|
||||
initialStoredMessages: [],
|
||||
attemptKey,
|
||||
replay: { status: 'error', content: abandonedMessage },
|
||||
taskCreatedAt: durableMessageTime(
|
||||
savedAbandoned,
|
||||
'The abandoned subagent result has no durable occurrence time.',
|
||||
),
|
||||
replay: {
|
||||
status: 'error',
|
||||
content: abandonedMessage,
|
||||
taskId,
|
||||
parentRunId: request.parentRunId,
|
||||
},
|
||||
};
|
||||
}
|
||||
const branch = selectLatestBranch(allMessages);
|
||||
|
|
@ -1631,6 +1697,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
isCreatedByUser: true,
|
||||
subagentTask: {
|
||||
attemptKey,
|
||||
parentRunId: request.parentRunId,
|
||||
...(requestFingerprint == null ? {} : { requestFingerprint }),
|
||||
status: 'running',
|
||||
},
|
||||
|
|
@ -1650,6 +1717,10 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
initialMessages,
|
||||
initialStoredMessages: mapChatMessagesToStoredMessages(initialMessages),
|
||||
attemptKey,
|
||||
taskCreatedAt: durableMessageTime(
|
||||
savedUserMessage,
|
||||
'The child-thread input has no durable occurrence time.',
|
||||
),
|
||||
userMessageId,
|
||||
};
|
||||
} catch (error) {
|
||||
|
|
@ -1732,6 +1803,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
...(subagentTranscript == null ? {} : { subagentTranscript }),
|
||||
subagentTask: {
|
||||
attemptKey: prepared.attemptKey,
|
||||
parentRunId: request.parentRunId,
|
||||
...(normalizedRequestFingerprint(request) == null
|
||||
? {}
|
||||
: { requestFingerprint: normalizedRequestFingerprint(request) }),
|
||||
|
|
@ -1775,6 +1847,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
error: true,
|
||||
subagentTask: {
|
||||
attemptKey: createSubagentAttemptKey(request.scopeId, request.idempotencyKey),
|
||||
parentRunId: request.parentRunId,
|
||||
...(normalizedRequestFingerprint(request) == null
|
||||
? {}
|
||||
: { requestFingerprint: normalizedRequestFingerprint(request) }),
|
||||
|
|
@ -1791,6 +1864,28 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
await this.touchAfterMessage(scope, threadId, taskId, 'failed');
|
||||
}
|
||||
|
||||
private async registerTaskWakeup(
|
||||
scope: SubagentThreadScope,
|
||||
threadId: string,
|
||||
request: SubagentTaskStartRequest,
|
||||
task: { taskId: string; parentRunId: string; createdAt: number },
|
||||
): Promise<void> {
|
||||
if (this.onTaskPrepared == null) {
|
||||
return;
|
||||
}
|
||||
await this.onTaskPrepared({
|
||||
userId: scope.userId,
|
||||
parentConversationId: scope.parentConversationId,
|
||||
parentMessageId: task.parentRunId,
|
||||
...(request.parentAgentId == null ? {} : { parentAgentId: request.parentAgentId }),
|
||||
...(scope.tenantId == null ? {} : { tenantId: scope.tenantId }),
|
||||
taskId: task.taskId,
|
||||
threadId,
|
||||
subagentType: request.subagentType,
|
||||
createdAt: task.createdAt,
|
||||
});
|
||||
}
|
||||
|
||||
private async persistCancellation(
|
||||
scope: SubagentThreadScope,
|
||||
threadId: string,
|
||||
|
|
@ -1816,6 +1911,7 @@ export class SubagentThreadTaskStore extends InMemorySubagentTaskStore {
|
|||
unfinished: false,
|
||||
subagentTask: {
|
||||
attemptKey: createSubagentAttemptKey(request.scopeId, request.idempotencyKey),
|
||||
parentRunId: request.parentRunId,
|
||||
...(normalizedRequestFingerprint(request) == null
|
||||
? {}
|
||||
: { requestFingerprint: normalizedRequestFingerprint(request) }),
|
||||
|
|
|
|||
|
|
@ -11,6 +11,9 @@ envelope and calls `enqueueAgentTrigger`; the adapter does not invoke an agent r
|
|||
- Give each source event a stable `event.id`, and keep `deliveryId` stable for retries to one
|
||||
target. A retry may use a fresh `requestId` and `receivedAt`.
|
||||
- Render bounded model input on the host. Infrastructure and routing remain server-controlled.
|
||||
- Use `continue` only with a persisted `conversationId` and exact `parentMessageId`. The host defers
|
||||
that delivery while the parent generation is still running or paused, so it cannot replace the
|
||||
generation it is meant to follow.
|
||||
- Use `orderingKey` only when deliveries must remain ordered across different event sources.
|
||||
Without an override, ordering is scoped to the user, source, mode, agent, and conversation.
|
||||
|
||||
|
|
@ -43,7 +46,7 @@ await enqueueAgentTrigger(
|
|||
|
||||
- Mongo owns queue state, leases, retry history, and dead letters across restarts and replicas.
|
||||
- A fresh token fences every claim, including reclaims by the same process.
|
||||
- A delivery is at-least-once. Fire and steer admission reuse the envelope's stable idempotency
|
||||
- A delivery is at-least-once. Fire, continue, and steer admission reuse the envelope's stable idempotency
|
||||
identity, so ambiguous retries do not duplicate accepted work.
|
||||
- Retryable failures use bounded exponential backoff and honor `Retry-After`. Invalid envelopes,
|
||||
permanent authorization failures, and exhausted retries become durable dead letters.
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ function orderingIdentity(
|
|||
envelope.event.source.id,
|
||||
envelope.mode,
|
||||
envelope.target.agentId,
|
||||
envelope.mode === 'steer' ? envelope.target.conversationId : '',
|
||||
envelope.mode === 'fire' ? '' : envelope.target.conversationId,
|
||||
];
|
||||
}
|
||||
return `trigger_lane_${digest([
|
||||
|
|
|
|||
|
|
@ -24,11 +24,16 @@ describe('dispatchAgentTrigger', () => {
|
|||
it('routes fire deliveries with their stable idempotency identity and abort signal', async () => {
|
||||
const controller = new AbortController();
|
||||
const fire = jest.fn(async () => ({ status: 'accepted' as const }));
|
||||
const continueRun = jest.fn(async () => ({ status: 'accepted' as const }));
|
||||
const steer = jest.fn(async () => ({ status: 'accepted' as const }));
|
||||
const envelope = fireEnvelope();
|
||||
|
||||
await expect(
|
||||
dispatchAgentTrigger(envelope, { fire, steer }, { signal: controller.signal }),
|
||||
dispatchAgentTrigger(
|
||||
envelope,
|
||||
{ continue: continueRun, fire, steer },
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
).resolves.toEqual({ status: 'accepted' });
|
||||
|
||||
expect(fire).toHaveBeenCalledWith(envelope, {
|
||||
|
|
@ -36,10 +41,12 @@ describe('dispatchAgentTrigger', () => {
|
|||
signal: controller.signal,
|
||||
});
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
expect(continueRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('routes steer deliveries without requiring a fire implementation detail', async () => {
|
||||
const fire = jest.fn(async () => 'fire');
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
const envelope = createAgentTriggerEnvelope({
|
||||
...createFireInput(),
|
||||
|
|
@ -51,64 +58,99 @@ describe('dispatchAgentTrigger', () => {
|
|||
},
|
||||
});
|
||||
|
||||
await expect(dispatchAgentTrigger(envelope, { fire, steer })).resolves.toBe('steer');
|
||||
await expect(
|
||||
dispatchAgentTrigger(envelope, { continue: continueRun, fire, steer }),
|
||||
).resolves.toBe('steer');
|
||||
expect(steer).toHaveBeenCalledWith(envelope, {
|
||||
idempotencyKey: expect.stringMatching(/^trigger_[a-f0-9]{64}$/),
|
||||
});
|
||||
expect(fire).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('routes continue deliveries to the exact existing conversation branch', async () => {
|
||||
const fire = jest.fn(async () => 'fire');
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
const envelope = createAgentTriggerEnvelope({
|
||||
...createFireInput(),
|
||||
mode: 'continue',
|
||||
target: {
|
||||
agentId: 'agent-1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
},
|
||||
});
|
||||
|
||||
await expect(
|
||||
dispatchAgentTrigger(envelope, { continue: continueRun, fire, steer }),
|
||||
).resolves.toBe('continue');
|
||||
expect(continueRun).toHaveBeenCalledWith(envelope, {
|
||||
idempotencyKey: expect.stringMatching(/^trigger_[a-f0-9]{64}$/),
|
||||
});
|
||||
expect(fire).not.toHaveBeenCalled();
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('propagates handler failures without falling back to another mode', async () => {
|
||||
const error = new Error('fire rejected');
|
||||
const fire = jest.fn(async () => Promise.reject(error));
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
|
||||
await expect(dispatchAgentTrigger(fireEnvelope(), { fire, steer })).rejects.toBe(error);
|
||||
await expect(
|
||||
dispatchAgentTrigger(fireEnvelope(), { continue: continueRun, fire, steer }),
|
||||
).rejects.toBe(error);
|
||||
expect(fire).toHaveBeenCalledTimes(1);
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects unknown modes before deriving identity or calling a handler', () => {
|
||||
const fire = jest.fn(async () => 'fire');
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
const envelope = {
|
||||
...fireEnvelope(),
|
||||
mode: 'resume',
|
||||
mode: 'launch',
|
||||
} as unknown as AgentTriggerEnvelope;
|
||||
|
||||
expect(() => dispatchAgentTrigger(envelope, { fire, steer })).toThrow(
|
||||
new AgentTriggerDispatchError('Unsupported agent trigger mode: resume'),
|
||||
expect(() => dispatchAgentTrigger(envelope, { continue: continueRun, fire, steer })).toThrow(
|
||||
new AgentTriggerDispatchError('Unsupported agent trigger mode: launch'),
|
||||
);
|
||||
expect(fire).not.toHaveBeenCalled();
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
expect(continueRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects unknown envelope versions before deriving identity or calling a handler', () => {
|
||||
const fire = jest.fn(async () => 'fire');
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
const envelope = {
|
||||
...fireEnvelope(),
|
||||
version: 2,
|
||||
} as unknown as AgentTriggerEnvelope;
|
||||
|
||||
expect(() => dispatchAgentTrigger(envelope, { fire, steer })).toThrow(
|
||||
expect(() => dispatchAgentTrigger(envelope, { continue: continueRun, fire, steer })).toThrow(
|
||||
new AgentTriggerDispatchError('Unsupported agent trigger envelope version: 2'),
|
||||
);
|
||||
expect(fire).not.toHaveBeenCalled();
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
expect(continueRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects malformed v1 envelopes before deriving identity or calling a handler', () => {
|
||||
const fire = jest.fn(async () => 'fire');
|
||||
const continueRun = jest.fn(async () => 'continue');
|
||||
const steer = jest.fn(async () => 'steer');
|
||||
const malformed = { ...fireEnvelope() };
|
||||
Reflect.deleteProperty(malformed, 'target');
|
||||
const envelope = malformed as unknown as AgentTriggerEnvelope;
|
||||
|
||||
expect(() => dispatchAgentTrigger(envelope, { fire, steer })).toThrow(
|
||||
expect(() => dispatchAgentTrigger(envelope, { continue: continueRun, fire, steer })).toThrow(
|
||||
new AgentTriggerDispatchError('target must be an object'),
|
||||
);
|
||||
expect(fire).not.toHaveBeenCalled();
|
||||
expect(steer).not.toHaveBeenCalled();
|
||||
expect(continueRun).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import type {
|
||||
AgentContinueTriggerEnvelope,
|
||||
AgentFireTriggerEnvelope,
|
||||
AgentSteerTriggerEnvelope,
|
||||
AgentTriggerEnvelope,
|
||||
|
|
@ -21,11 +22,15 @@ export class AgentTriggerDispatchError extends TypeError {
|
|||
* Host-owned execution adapters. Each handler must enforce current authorization,
|
||||
* limits, persistence, and the supplied idempotency identity before accepting work.
|
||||
*/
|
||||
export interface AgentTriggerDispatchHandlers<FireResult, SteerResult> {
|
||||
export interface AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerResult> {
|
||||
fire: (
|
||||
envelope: AgentFireTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
) => Promise<FireResult>;
|
||||
continue: (
|
||||
envelope: AgentContinueTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
) => Promise<ContinueResult>;
|
||||
steer: (
|
||||
envelope: AgentSteerTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
|
|
@ -33,11 +38,11 @@ export interface AgentTriggerDispatchHandlers<FireResult, SteerResult> {
|
|||
}
|
||||
|
||||
/** Routes a normalized trigger without coupling its source to an execution transport. */
|
||||
export function dispatchAgentTrigger<FireResult, SteerResult>(
|
||||
export function dispatchAgentTrigger<FireResult, ContinueResult, SteerResult>(
|
||||
envelope: unknown,
|
||||
handlers: AgentTriggerDispatchHandlers<FireResult, SteerResult>,
|
||||
handlers: AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerResult>,
|
||||
options?: { signal?: AbortSignal },
|
||||
): Promise<FireResult | SteerResult> {
|
||||
): Promise<ContinueResult | FireResult | SteerResult> {
|
||||
let normalized: AgentTriggerEnvelope;
|
||||
try {
|
||||
normalized = parseAgentTriggerEnvelope(envelope);
|
||||
|
|
@ -51,5 +56,8 @@ export function dispatchAgentTrigger<FireResult, SteerResult>(
|
|||
if (normalized.mode === 'fire') {
|
||||
return handlers.fire(normalized, context);
|
||||
}
|
||||
if (normalized.mode === 'continue') {
|
||||
return handlers.continue(normalized, context);
|
||||
}
|
||||
return handlers.steer(normalized, context);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -280,6 +280,41 @@ describe('createAgentTriggerDeliveryEngine', () => {
|
|||
expect(store.dead).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('defers a continuation until its parent generation settles without consuming an attempt', async () => {
|
||||
const store = storeWith();
|
||||
const engine = createAgentTriggerDeliveryEngine(
|
||||
{
|
||||
store,
|
||||
dispatch: async () =>
|
||||
Promise.reject(
|
||||
new AgentTriggerExecutionError('parent generation is still running', {
|
||||
mode: 'continue',
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
deferWithoutAttempt: true,
|
||||
code: 'PARENT_NOT_READY',
|
||||
status: 409,
|
||||
}),
|
||||
),
|
||||
now: () => START,
|
||||
workerId: 'worker-1',
|
||||
},
|
||||
{ concurrency: 1, maxAttempts: 1 },
|
||||
);
|
||||
|
||||
await engine.runTick();
|
||||
|
||||
expect(store.defer).toHaveBeenCalledWith({
|
||||
id: 'delivery-row-1',
|
||||
workerId: 'worker-1',
|
||||
claimToken: 'claim-1',
|
||||
attempt: 1,
|
||||
availableAt: new Date(START.getTime() + 5_000),
|
||||
});
|
||||
expect(store.retry).not.toHaveBeenCalled();
|
||||
expect(store.dead).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not shorten Retry-After to the exponential backoff cap', async () => {
|
||||
const store = storeWith();
|
||||
const error = new AgentTriggerExecutionError('maintenance', {
|
||||
|
|
|
|||
|
|
@ -211,6 +211,10 @@ function isAccountDeletionDeferral(error: unknown): boolean {
|
|||
);
|
||||
}
|
||||
|
||||
function isRuntimeReadinessDeferral(error: unknown): boolean {
|
||||
return error instanceof AgentTriggerExecutionError && error.deferWithoutAttempt;
|
||||
}
|
||||
|
||||
/** Durable, lease-fenced delivery runner shared by every trusted event source. */
|
||||
export function createAgentTriggerDeliveryEngine(
|
||||
deps: AgentTriggerDeliveryEngineDeps,
|
||||
|
|
@ -326,10 +330,12 @@ export function createAgentTriggerDeliveryEngine(
|
|||
const attemptedAt = now();
|
||||
const deletionCancelled = controller.signal.aborted && cancelledUsers.has(userId);
|
||||
const deletionRejected = isAccountDeletionDeferral(error);
|
||||
const runtimeNotReady = isRuntimeReadinessDeferral(error);
|
||||
if (
|
||||
error instanceof AgentTriggerDeliveryDeferredError ||
|
||||
deletionCancelled ||
|
||||
deletionRejected
|
||||
deletionRejected ||
|
||||
runtimeNotReady
|
||||
) {
|
||||
const delayMs =
|
||||
error instanceof AgentTriggerDeliveryDeferredError ? error.delayMs : DEFAULT_DEFER_MS;
|
||||
|
|
@ -342,9 +348,15 @@ export function createAgentTriggerDeliveryEngine(
|
|||
availableAt,
|
||||
});
|
||||
if (deferred) {
|
||||
let reason = 'pre_dispatch';
|
||||
if (deletionCancelled || deletionRejected) {
|
||||
reason = 'account_deletion';
|
||||
} else if (runtimeNotReady) {
|
||||
reason = 'runtime_readiness';
|
||||
}
|
||||
logger.info('[agent-triggers] delivery deferred without consuming an attempt', {
|
||||
deliveryKey: delivery.deliveryKey,
|
||||
reason: deletionCancelled || deletionRejected ? 'account_deletion' : 'pre_dispatch',
|
||||
reason,
|
||||
availableAt: availableAt.toISOString(),
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -93,6 +93,26 @@ describe('createAgentTriggerEnvelope', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('requires an exact existing branch for continue deliveries', () => {
|
||||
const envelope = createAgentTriggerEnvelope({
|
||||
...createFireInput(),
|
||||
mode: 'continue',
|
||||
target: {
|
||||
agentId: 'agent-1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
},
|
||||
});
|
||||
|
||||
expect(envelope.mode).toBe('continue');
|
||||
expect(envelope.target).toEqual({
|
||||
agentId: 'agent-1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
});
|
||||
expect(parseAgentTriggerEnvelope(JSON.parse(JSON.stringify(envelope)))).toEqual(envelope);
|
||||
});
|
||||
|
||||
it('builds a stable generation-compatible idempotency key per delivery target', () => {
|
||||
const first = createAgentTriggerEnvelope(createFireInput());
|
||||
const retry = createAgentTriggerEnvelope({
|
||||
|
|
@ -210,8 +230,8 @@ describe('createAgentTriggerEnvelope', () => {
|
|||
expect(() =>
|
||||
createAgentTriggerEnvelope({
|
||||
...createFireInput(),
|
||||
mode: 'resume',
|
||||
mode: 'launch',
|
||||
} as unknown as CreateAgentTriggerEnvelopeInput),
|
||||
).toThrow('Unsupported agent trigger mode: resume');
|
||||
).toThrow('Unsupported agent trigger mode: launch');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ import { cloneJsonValue } from '../json';
|
|||
export const AGENT_TRIGGER_ENVELOPE_VERSION = 1 as const;
|
||||
export const AGENT_TRIGGER_IDEMPOTENCY_PREFIX = 'trigger_';
|
||||
|
||||
export type AgentTriggerMode = 'fire' | 'steer';
|
||||
export type AgentTriggerMode = 'continue' | 'fire' | 'steer';
|
||||
|
||||
export interface AgentTriggerSource {
|
||||
/** Stable identity of the configured source, such as a webhook or schedule id. */
|
||||
|
|
@ -36,6 +36,13 @@ interface AgentTriggerTarget {
|
|||
*/
|
||||
export type AgentFireTarget = AgentTriggerTarget;
|
||||
|
||||
export interface AgentContinueTarget extends AgentTriggerTarget {
|
||||
/** Existing conversation that receives a new host-authored turn. */
|
||||
conversationId: string;
|
||||
/** Persisted branch leaf below which the new turn is appended. */
|
||||
parentMessageId: string;
|
||||
}
|
||||
|
||||
export interface AgentSteerTarget extends AgentTriggerTarget {
|
||||
/** Existing conversation whose active generation receives the input. */
|
||||
conversationId: string;
|
||||
|
|
@ -63,12 +70,20 @@ export interface AgentFireTriggerEnvelope extends AgentTriggerEnvelopeBase {
|
|||
target: AgentFireTarget;
|
||||
}
|
||||
|
||||
export interface AgentContinueTriggerEnvelope extends AgentTriggerEnvelopeBase {
|
||||
mode: 'continue';
|
||||
target: AgentContinueTarget;
|
||||
}
|
||||
|
||||
export interface AgentSteerTriggerEnvelope extends AgentTriggerEnvelopeBase {
|
||||
mode: 'steer';
|
||||
target: AgentSteerTarget;
|
||||
}
|
||||
|
||||
export type AgentTriggerEnvelope = AgentFireTriggerEnvelope | AgentSteerTriggerEnvelope;
|
||||
export type AgentTriggerEnvelope =
|
||||
| AgentContinueTriggerEnvelope
|
||||
| AgentFireTriggerEnvelope
|
||||
| AgentSteerTriggerEnvelope;
|
||||
|
||||
interface CreateAgentTriggerEnvelopeBase {
|
||||
requestId: string;
|
||||
|
|
@ -84,6 +99,10 @@ export type CreateAgentTriggerEnvelopeInput =
|
|||
mode: 'fire';
|
||||
target: AgentFireTarget;
|
||||
})
|
||||
| (CreateAgentTriggerEnvelopeBase & {
|
||||
mode: 'continue';
|
||||
target: AgentContinueTarget;
|
||||
})
|
||||
| (CreateAgentTriggerEnvelopeBase & {
|
||||
mode: 'steer';
|
||||
target: AgentSteerTarget;
|
||||
|
|
@ -191,6 +210,18 @@ export function createAgentTriggerEnvelope(
|
|||
};
|
||||
}
|
||||
|
||||
if (input.mode === 'continue') {
|
||||
return {
|
||||
...base,
|
||||
mode: input.mode,
|
||||
target: {
|
||||
agentId: requireString(input.target?.agentId, 'target.agentId'),
|
||||
conversationId: requireString(input.target?.conversationId, 'target.conversationId'),
|
||||
parentMessageId: requireString(input.target?.parentMessageId, 'target.parentMessageId'),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
throw error(`Unsupported agent trigger mode: ${receivedMode}`);
|
||||
}
|
||||
|
||||
|
|
@ -205,7 +236,7 @@ export function parseAgentTriggerEnvelope(input: unknown): AgentTriggerEnvelope
|
|||
}
|
||||
|
||||
const mode = envelope.mode;
|
||||
if (mode !== 'fire' && mode !== 'steer') {
|
||||
if (mode !== 'continue' && mode !== 'fire' && mode !== 'steer') {
|
||||
throw error(`Unsupported agent trigger mode: ${String(mode)}`);
|
||||
}
|
||||
|
||||
|
|
@ -252,6 +283,18 @@ export function parseAgentTriggerEnvelope(input: unknown): AgentTriggerEnvelope
|
|||
};
|
||||
}
|
||||
|
||||
if (mode === 'continue') {
|
||||
return {
|
||||
...base,
|
||||
mode,
|
||||
target: {
|
||||
agentId: requireString(target.agentId, 'target.agentId'),
|
||||
conversationId: requireString(target.conversationId, 'target.conversationId'),
|
||||
parentMessageId: requireString(target.parentMessageId, 'target.parentMessageId'),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (target.preempt != null && typeof target.preempt !== 'boolean') {
|
||||
throw error('target.preempt must be a boolean');
|
||||
}
|
||||
|
|
@ -288,7 +331,8 @@ export function getAgentTriggerIdempotencyKey(envelope: AgentTriggerEnvelope): s
|
|||
envelope.deliveryId,
|
||||
envelope.mode,
|
||||
envelope.target.agentId,
|
||||
envelope.mode === 'steer' ? envelope.target.conversationId : '',
|
||||
envelope.mode === 'fire' ? '' : envelope.target.conversationId,
|
||||
envelope.mode === 'continue' ? envelope.target.parentMessageId : '',
|
||||
]),
|
||||
)
|
||||
.digest('hex');
|
||||
|
|
|
|||
|
|
@ -44,6 +44,27 @@ const createSteerEnvelope = () =>
|
|||
input: 'The opponent moved. Take your turn.',
|
||||
});
|
||||
|
||||
const createContinueEnvelope = () =>
|
||||
createAgentTriggerEnvelope({
|
||||
mode: 'continue',
|
||||
requestId: 'request-3',
|
||||
deliveryId: 'delivery-3',
|
||||
receivedAt: 35,
|
||||
principal: { id: 'user-1', role: 'member', tenantId: 'tenant-1' },
|
||||
target: {
|
||||
agentId: 'agent-1',
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'response-1',
|
||||
},
|
||||
event: {
|
||||
id: 'event-3',
|
||||
type: 'subagent.completed',
|
||||
occurredAt: 31,
|
||||
source: { id: 'subagent-completion', type: 'internal' },
|
||||
},
|
||||
input: 'Collect the completed child task.',
|
||||
});
|
||||
|
||||
function response(payload: unknown, init?: ResponseInit): Response {
|
||||
return new Response(JSON.stringify(payload), {
|
||||
status: 200,
|
||||
|
|
@ -294,6 +315,48 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
|
|||
expect(fetcher).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('releases a prepared result that arrives after setup has timed out', async () => {
|
||||
let finishPreparation!: (value: {
|
||||
status: 'ready';
|
||||
input: string;
|
||||
parentMessageId: string;
|
||||
releaseOnDefiniteFailure: () => Promise<void>;
|
||||
}) => void;
|
||||
const preparation = new Promise<{
|
||||
status: 'ready';
|
||||
input: string;
|
||||
parentMessageId: string;
|
||||
releaseOnDefiniteFailure: () => Promise<void>;
|
||||
}>((resolve) => {
|
||||
finishPreparation = resolve;
|
||||
});
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const fetcher = fetchMock(async () => response({}));
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(fetcher, {
|
||||
prepareContinue: () => preparation,
|
||||
timeoutMs: 10,
|
||||
}),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
mode: 'continue',
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
code: 'TIMEOUT',
|
||||
});
|
||||
finishPreparation({
|
||||
status: 'ready',
|
||||
input: 'late durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
});
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
|
||||
expect(releaseOnDefiniteFailure).toHaveBeenCalledTimes(1);
|
||||
expect(fetcher).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('starts independent token, timezone, and origin setup concurrently', async () => {
|
||||
let resolveToken!: (value: string) => void;
|
||||
let resolveTimezone!: (value: string) => void;
|
||||
|
|
@ -455,6 +518,219 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('createAgentTriggerExecutionHost continue adapter', () => {
|
||||
it('appends an idempotent turn to the exact existing conversation branch', async () => {
|
||||
const envelope = createContinueEnvelope();
|
||||
const idempotencyKey = getAgentTriggerIdempotencyKey(envelope);
|
||||
const fetcher = fetchMock(async () =>
|
||||
response({
|
||||
streamId: 'conversation-1',
|
||||
conversationId: 'conversation-1',
|
||||
generationCreatedAt: 50,
|
||||
status: 'started',
|
||||
}),
|
||||
);
|
||||
const host = createAgentTriggerExecutionHost(deps(fetcher));
|
||||
|
||||
await expect(host.dispatch(envelope)).resolves.toEqual({
|
||||
mode: 'continue',
|
||||
streamId: 'conversation-1',
|
||||
conversationId: 'conversation-1',
|
||||
generationCreatedAt: 50,
|
||||
status: 'started',
|
||||
});
|
||||
const [input, init] = fetcher.mock.calls[0];
|
||||
expect(String(input)).toBe('http://127.0.0.1:3080/api/agents/chat/agents');
|
||||
expect(JSON.parse(String(init?.body))).toEqual({
|
||||
text: envelope.input,
|
||||
endpoint: EModelEndpoint.agents,
|
||||
agent_id: 'agent-1',
|
||||
parentMessageId: 'response-1',
|
||||
conversationId: 'conversation-1',
|
||||
isContinued: false,
|
||||
isRegenerate: false,
|
||||
clientRequestId: idempotencyKey,
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it('retries without consuming the logical delivery when the parent is not settled', async () => {
|
||||
expect.hasAssertions();
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () =>
|
||||
response(
|
||||
{ code: 'PARENT_NOT_READY', error: 'The parent is still running.' },
|
||||
{ status: 409 },
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
await host.dispatch(createContinueEnvelope()).catch((error: unknown) => {
|
||||
expectExecutionError(error, {
|
||||
mode: 'continue',
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
deferWithoutAttempt: true,
|
||||
code: 'PARENT_NOT_READY',
|
||||
status: 409,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it('releases a prepared durable result after a definite admission rejection', async () => {
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () => response({ code: 'AGENT_NOT_FOUND' }, { status: 404 })),
|
||||
{
|
||||
prepareContinue: async () => ({
|
||||
status: 'ready',
|
||||
input: 'durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
}),
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
certainty: 'definite',
|
||||
code: 'AGENT_NOT_FOUND',
|
||||
});
|
||||
expect(releaseOnDefiniteFailure).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('retains a prepared durable result after an ambiguous admission outcome', async () => {
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () => Promise.reject(new Error('connection reset'))),
|
||||
{
|
||||
prepareContinue: async () => ({
|
||||
status: 'ready',
|
||||
input: 'durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
}),
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
certainty: 'ambiguous',
|
||||
code: 'NETWORK_ERROR',
|
||||
});
|
||||
expect(releaseOnDefiniteFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('retains a prepared durable result when a retry gets a definite 5xx response', async () => {
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () =>
|
||||
response(
|
||||
{ code: 'SERVER_NOT_READY', error: 'Generation is finalizing.' },
|
||||
{ status: 503, headers: { 'retry-after': '1' } },
|
||||
),
|
||||
),
|
||||
{
|
||||
prepareContinue: async () => ({
|
||||
status: 'ready',
|
||||
input: 'durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
}),
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
code: 'SERVER_NOT_READY',
|
||||
status: 503,
|
||||
});
|
||||
expect(releaseOnDefiniteFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('releases a prepared durable result when parent state fails before admission', async () => {
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () =>
|
||||
response(
|
||||
{ code: 'PARENT_STATE_UNAVAILABLE', error: 'Parent state is unavailable.' },
|
||||
{ status: 503, headers: { 'retry-after': '1' } },
|
||||
),
|
||||
),
|
||||
{
|
||||
prepareContinue: async () => ({
|
||||
status: 'ready',
|
||||
input: 'durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
}),
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
code: 'PARENT_STATE_UNAVAILABLE',
|
||||
status: 503,
|
||||
});
|
||||
expect(releaseOnDefiniteFailure).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('retains a prepared durable result when an earlier admitted run was replaced', async () => {
|
||||
const releaseOnDefiniteFailure = jest.fn(async () => undefined);
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () => response({ code: 'RUN_REPLACED' }, { status: 409 })),
|
||||
{
|
||||
prepareContinue: async () => ({
|
||||
status: 'ready',
|
||||
input: 'durable child result',
|
||||
parentMessageId: 'response-1',
|
||||
releaseOnDefiniteFailure,
|
||||
}),
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
|
||||
certainty: 'definite',
|
||||
retryable: false,
|
||||
code: 'RUN_REPLACED',
|
||||
status: 409,
|
||||
});
|
||||
expect(releaseOnDefiniteFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects a mismatched continued conversation as an ambiguous outcome', async () => {
|
||||
expect.hasAssertions();
|
||||
const host = createAgentTriggerExecutionHost(
|
||||
deps(
|
||||
fetchMock(async () =>
|
||||
response({ streamId: 'other', conversationId: 'other', status: 'started' }),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
await host.dispatch(createContinueEnvelope()).catch((error: unknown) => {
|
||||
expectExecutionError(error, {
|
||||
mode: 'continue',
|
||||
certainty: 'ambiguous',
|
||||
retryable: true,
|
||||
code: 'INVALID_RESPONSE',
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('createAgentTriggerExecutionHost steer adapter', () => {
|
||||
it('steers through the authenticated admission route with a strict v2 receipt', async () => {
|
||||
const envelope = createSteerEnvelope();
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
import { tenantStorage } from '@librechat/data-schemas';
|
||||
import { logger, tenantStorage } from '@librechat/data-schemas';
|
||||
import { Constants, EModelEndpoint } from 'librechat-data-provider';
|
||||
import type {
|
||||
AgentContinueTriggerEnvelope,
|
||||
AgentFireTriggerEnvelope,
|
||||
AgentSteerTriggerEnvelope,
|
||||
AgentTriggerEnvelope,
|
||||
AgentTriggerMode,
|
||||
} from './envelope';
|
||||
import type { AgentTriggerDispatchContext } from './dispatch';
|
||||
import type { AgentRunPrincipal } from '../envelope';
|
||||
|
|
@ -30,12 +32,25 @@ type FireStatus = 'started' | 'resumed' | 'replaced' | 'settled';
|
|||
|
||||
export type AgentTriggerFetch = (input: string | URL, init?: RequestInit) => Promise<Response>;
|
||||
|
||||
export type AgentTriggerContinuePreparation =
|
||||
| {
|
||||
status: 'ready';
|
||||
input: string;
|
||||
parentMessageId: string;
|
||||
/** Compensates a durable pre-admission claim only when the host knows
|
||||
* that no generation was admitted. Ambiguous outcomes retain the claim. */
|
||||
releaseOnDefiniteFailure?: () => MaybePromise<void>;
|
||||
}
|
||||
| { status: 'settled' };
|
||||
|
||||
export type AgentTriggerFailureCertainty = 'definite' | 'ambiguous';
|
||||
|
||||
export interface AgentTriggerExecutionErrorOptions {
|
||||
mode: 'fire' | 'steer';
|
||||
mode: AgentTriggerMode;
|
||||
certainty: AgentTriggerFailureCertainty;
|
||||
retryable: boolean;
|
||||
/** Release the delivery lease without consuming its logical retry budget. */
|
||||
deferWithoutAttempt?: boolean;
|
||||
code?: string;
|
||||
status?: number;
|
||||
retryAfter?: string;
|
||||
|
|
@ -47,9 +62,10 @@ export interface AgentTriggerExecutionErrorOptions {
|
|||
* remains unchanged.
|
||||
*/
|
||||
export class AgentTriggerExecutionError extends Error {
|
||||
readonly mode: 'fire' | 'steer';
|
||||
readonly mode: AgentTriggerMode;
|
||||
readonly certainty: AgentTriggerFailureCertainty;
|
||||
readonly retryable: boolean;
|
||||
readonly deferWithoutAttempt: boolean;
|
||||
readonly code?: string;
|
||||
readonly status?: number;
|
||||
readonly retryAfter?: string;
|
||||
|
|
@ -60,6 +76,7 @@ export class AgentTriggerExecutionError extends Error {
|
|||
this.mode = options.mode;
|
||||
this.certainty = options.certainty;
|
||||
this.retryable = options.retryable;
|
||||
this.deferWithoutAttempt = options.deferWithoutAttempt === true;
|
||||
this.code = options.code;
|
||||
this.status = options.status;
|
||||
this.retryAfter = options.retryAfter;
|
||||
|
|
@ -87,7 +104,18 @@ export interface AgentTriggerSteerResult {
|
|||
leftover?: boolean;
|
||||
}
|
||||
|
||||
export type AgentTriggerExecutionResult = AgentTriggerFireResult | AgentTriggerSteerResult;
|
||||
export interface AgentTriggerContinueResult {
|
||||
mode: 'continue';
|
||||
status: FireStatus;
|
||||
conversationId: string;
|
||||
streamId?: string;
|
||||
generationCreatedAt?: number;
|
||||
}
|
||||
|
||||
export type AgentTriggerExecutionResult =
|
||||
| AgentTriggerContinueResult
|
||||
| AgentTriggerFireResult
|
||||
| AgentTriggerSteerResult;
|
||||
|
||||
export interface AgentTriggerExecutionHostDeps {
|
||||
/** Trusted root URL for this LibreChat server. */
|
||||
|
|
@ -97,8 +125,14 @@ export interface AgentTriggerExecutionHostDeps {
|
|||
/** Optional user-timezone resolver for dynamic date variables in a new run. */
|
||||
getTimezone?: (
|
||||
principal: AgentRunPrincipal,
|
||||
envelope: AgentFireTriggerEnvelope,
|
||||
envelope: AgentContinueTriggerEnvelope | AgentFireTriggerEnvelope,
|
||||
) => MaybePromise<string | undefined>;
|
||||
/** Optional server-owned resolver for durable internal continuation inputs.
|
||||
* External/source-neutral envelopes remain unchanged when this returns undefined. */
|
||||
prepareContinue?: (
|
||||
envelope: AgentContinueTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
) => MaybePromise<AgentTriggerContinuePreparation | undefined>;
|
||||
fetch?: AgentTriggerFetch;
|
||||
/** Total bound for setup, admission, and the bounded response read. */
|
||||
timeoutMs?: number;
|
||||
|
|
@ -163,7 +197,7 @@ function abortScope(parent: AbortSignal | undefined, timeoutMs: number): AbortSc
|
|||
}
|
||||
|
||||
function abortError(
|
||||
mode: 'fire' | 'steer',
|
||||
mode: AgentTriggerMode,
|
||||
scope: AbortScope,
|
||||
parent: AbortSignal | undefined,
|
||||
stage: string,
|
||||
|
|
@ -183,15 +217,18 @@ function abortError(
|
|||
|
||||
function observeAbort<T>(
|
||||
operation: () => MaybePromise<T>,
|
||||
mode: 'fire' | 'steer',
|
||||
mode: AgentTriggerMode,
|
||||
scope: AbortScope,
|
||||
parent: AbortSignal | undefined,
|
||||
onLateValue?: (value: T) => MaybePromise<void>,
|
||||
): Promise<T> {
|
||||
if (scope.signal.aborted) {
|
||||
return Promise.reject(abortError(mode, scope, parent, 'before dispatch', 'definite'));
|
||||
}
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
let abandoned = false;
|
||||
const onAbort = () => {
|
||||
abandoned = true;
|
||||
scope.signal.removeEventListener('abort', onAbort);
|
||||
reject(abortError(mode, scope, parent, 'during setup', 'definite'));
|
||||
};
|
||||
|
|
@ -201,6 +238,12 @@ function observeAbort<T>(
|
|||
.then(
|
||||
(value) => {
|
||||
scope.signal.removeEventListener('abort', onAbort);
|
||||
if (abandoned) {
|
||||
Promise.resolve(onLateValue?.(value)).catch((error: unknown) => {
|
||||
logger.error('[agentTriggers] Failed to compensate late setup result', error);
|
||||
});
|
||||
return;
|
||||
}
|
||||
resolve(value);
|
||||
},
|
||||
(error: unknown) => {
|
||||
|
|
@ -213,12 +256,13 @@ function observeAbort<T>(
|
|||
|
||||
async function setupValue<T>(
|
||||
operation: () => MaybePromise<T>,
|
||||
mode: 'fire' | 'steer',
|
||||
mode: AgentTriggerMode,
|
||||
scope: AbortScope,
|
||||
parent: AbortSignal | undefined,
|
||||
onLateValue?: (value: T) => MaybePromise<void>,
|
||||
): Promise<T> {
|
||||
try {
|
||||
return await observeAbort(operation, mode, scope, parent);
|
||||
return await observeAbort(operation, mode, scope, parent, onLateValue);
|
||||
} catch (error) {
|
||||
if (error instanceof AgentTriggerExecutionError) {
|
||||
throw error;
|
||||
|
|
@ -342,7 +386,7 @@ function abortCode(scope: AbortScope, parent: AbortSignal | undefined, fallback:
|
|||
return fallback;
|
||||
}
|
||||
|
||||
function requireToken(value: unknown, mode: 'fire' | 'steer'): string {
|
||||
function requireToken(value: unknown, mode: AgentTriggerMode): string {
|
||||
if (typeof value !== 'string' || value.length === 0 || /\s/.test(value)) {
|
||||
throw executionError('Agent trigger token mint returned an invalid token', {
|
||||
mode,
|
||||
|
|
@ -354,7 +398,7 @@ function requireToken(value: unknown, mode: 'fire' | 'steer'): string {
|
|||
return value;
|
||||
}
|
||||
|
||||
function triggerUrl(baseUrl: string, path: string, mode: 'fire' | 'steer'): string {
|
||||
function triggerUrl(baseUrl: string, path: string, mode: AgentTriggerMode): string {
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(baseUrl);
|
||||
|
|
@ -384,6 +428,10 @@ function fireUrl(baseUrl: string): string {
|
|||
return triggerUrl(baseUrl, `/api/agents/chat/${EModelEndpoint.agents}`, 'fire');
|
||||
}
|
||||
|
||||
function continueUrl(baseUrl: string): string {
|
||||
return triggerUrl(baseUrl, `/api/agents/chat/${EModelEndpoint.agents}`, 'continue');
|
||||
}
|
||||
|
||||
function steerUrl(baseUrl: string): string {
|
||||
return triggerUrl(baseUrl, '/api/agents/chat/steer/deliver', 'steer');
|
||||
}
|
||||
|
|
@ -402,7 +450,10 @@ function fireStatus(value: unknown): FireStatus | undefined {
|
|||
: undefined;
|
||||
}
|
||||
|
||||
function parseFireResult(payload: unknown): AgentTriggerFireResult | undefined {
|
||||
function parseStartResult(
|
||||
payload: unknown,
|
||||
mode: 'continue' | 'fire',
|
||||
): AgentTriggerContinueResult | AgentTriggerFireResult | undefined {
|
||||
if (payload == null || typeof payload !== 'object') {
|
||||
return undefined;
|
||||
}
|
||||
|
|
@ -421,7 +472,7 @@ function parseFireResult(payload: unknown): AgentTriggerFireResult | undefined {
|
|||
'generationCreatedAt' in payload ? payload.generationCreatedAt : undefined,
|
||||
);
|
||||
return {
|
||||
mode: 'fire',
|
||||
mode,
|
||||
status,
|
||||
conversationId,
|
||||
...(streamId != null && { streamId }),
|
||||
|
|
@ -429,33 +480,101 @@ function parseFireResult(payload: unknown): AgentTriggerFireResult | undefined {
|
|||
};
|
||||
}
|
||||
|
||||
async function fire(
|
||||
function resolveParentMessageId(
|
||||
preparation: AgentTriggerContinuePreparation | undefined,
|
||||
envelope: AgentContinueTriggerEnvelope | AgentFireTriggerEnvelope,
|
||||
): string {
|
||||
if (preparation?.status === 'ready') {
|
||||
return preparation.parentMessageId;
|
||||
}
|
||||
if (envelope.mode === 'continue') {
|
||||
return envelope.target.parentMessageId;
|
||||
}
|
||||
return Constants.NO_PARENT;
|
||||
}
|
||||
|
||||
/** A response can be definite at the HTTP layer while the idempotent logical
|
||||
* generation is still outcome-ambiguous. In particular, a retry can receive a
|
||||
* 5xx while the first request owns the generation claim or is finalizing its
|
||||
* accepted run. Compensate the prepared durable result only for failures that
|
||||
* prove admission did not happen. */
|
||||
function canReleasePreparedResult(error: AgentTriggerExecutionError): boolean {
|
||||
if (error.certainty !== 'definite') {
|
||||
return false;
|
||||
}
|
||||
if (error.code === 'START_ABORTED' || error.status == null) {
|
||||
return true;
|
||||
}
|
||||
if (error.code === 'PARENT_NOT_READY' || error.code === 'PARENT_STATE_UNAVAILABLE') {
|
||||
return true;
|
||||
}
|
||||
return error.status >= 400 && error.status < 500 && error.status !== 408 && error.status !== 409;
|
||||
}
|
||||
|
||||
function startRun(
|
||||
envelope: AgentFireTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
deps: AgentTriggerExecutionHostDeps,
|
||||
timeoutMs: number,
|
||||
): Promise<AgentTriggerFireResult> {
|
||||
): Promise<AgentTriggerFireResult>;
|
||||
function startRun(
|
||||
envelope: AgentContinueTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
deps: AgentTriggerExecutionHostDeps,
|
||||
timeoutMs: number,
|
||||
): Promise<AgentTriggerContinueResult>;
|
||||
async function startRun(
|
||||
envelope: AgentContinueTriggerEnvelope | AgentFireTriggerEnvelope,
|
||||
context: AgentTriggerDispatchContext,
|
||||
deps: AgentTriggerExecutionHostDeps,
|
||||
timeoutMs: number,
|
||||
): Promise<AgentTriggerContinueResult | AgentTriggerFireResult> {
|
||||
const mode = envelope.mode;
|
||||
const scope = abortScope(context.signal, timeoutMs);
|
||||
let preparation: AgentTriggerContinuePreparation | undefined;
|
||||
try {
|
||||
preparation =
|
||||
mode === 'continue' && deps.prepareContinue != null
|
||||
? await setupValue(
|
||||
() => deps.prepareContinue?.(envelope, context),
|
||||
mode,
|
||||
scope,
|
||||
context.signal,
|
||||
async (latePreparation) => {
|
||||
if (latePreparation?.status === 'ready') {
|
||||
await latePreparation.releaseOnDefiniteFailure?.();
|
||||
}
|
||||
},
|
||||
)
|
||||
: undefined;
|
||||
if (preparation?.status === 'settled' && envelope.mode === 'continue') {
|
||||
return {
|
||||
mode: 'continue',
|
||||
status: 'settled',
|
||||
conversationId: envelope.target.conversationId,
|
||||
};
|
||||
}
|
||||
const input = preparation?.status === 'ready' ? preparation.input : envelope.input;
|
||||
const parentMessageId = resolveParentMessageId(preparation, envelope);
|
||||
const [token, timezone, baseUrl] = await Promise.all([
|
||||
setupValue(
|
||||
() => deps.mintToken(envelope.principal, envelope),
|
||||
'fire',
|
||||
mode,
|
||||
scope,
|
||||
context.signal,
|
||||
).then((value) => requireToken(value, 'fire')),
|
||||
).then((value) => requireToken(value, mode)),
|
||||
setupValue(
|
||||
() => deps.getTimezone?.(envelope.principal, envelope),
|
||||
'fire',
|
||||
mode,
|
||||
scope,
|
||||
context.signal,
|
||||
),
|
||||
setupValue(() => deps.getBaseUrl(), 'fire', scope, context.signal),
|
||||
setupValue(() => deps.getBaseUrl(), mode, scope, context.signal),
|
||||
]).catch((error: unknown) => {
|
||||
scope.abort();
|
||||
throw error;
|
||||
});
|
||||
const url = fireUrl(baseUrl);
|
||||
const url = mode === 'fire' ? fireUrl(baseUrl) : continueUrl(baseUrl);
|
||||
const fetcher: AgentTriggerFetch = deps.fetch ?? globalThis.fetch;
|
||||
let response: Response;
|
||||
try {
|
||||
|
|
@ -472,10 +591,13 @@ async function fire(
|
|||
[GENERATION_PROTOCOL_HEADER]: '2',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
text: envelope.input,
|
||||
text: input,
|
||||
endpoint: EModelEndpoint.agents,
|
||||
agent_id: envelope.target.agentId,
|
||||
parentMessageId: Constants.NO_PARENT,
|
||||
parentMessageId,
|
||||
...(envelope.mode === 'continue' && {
|
||||
conversationId: envelope.target.conversationId,
|
||||
}),
|
||||
isContinued: false,
|
||||
isRegenerate: false,
|
||||
clientRequestId: context.idempotencyKey,
|
||||
|
|
@ -489,9 +611,9 @@ async function fire(
|
|||
const definite = isDefiniteConnectFailure(error);
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
throw executionError(
|
||||
`Agent trigger fire ${definite ? 'could not connect' : 'has an unknown outcome'}: ${message}`,
|
||||
`Agent trigger ${mode} ${definite ? 'could not connect' : 'has an unknown outcome'}: ${message}`,
|
||||
{
|
||||
mode: 'fire',
|
||||
mode,
|
||||
certainty: definite ? 'definite' : 'ambiguous',
|
||||
retryable: true,
|
||||
code: abortCode(scope, context.signal, 'NETWORK_ERROR'),
|
||||
|
|
@ -505,11 +627,11 @@ async function fire(
|
|||
} catch (error) {
|
||||
if (response.ok) {
|
||||
throw executionError(
|
||||
`Agent trigger fire response has an unknown outcome: ${
|
||||
`Agent trigger ${mode} response has an unknown outcome: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
{
|
||||
mode: 'fire',
|
||||
mode,
|
||||
certainty: 'ambiguous',
|
||||
retryable: true,
|
||||
code: abortCode(scope, context.signal, 'INVALID_RESPONSE'),
|
||||
|
|
@ -522,11 +644,19 @@ async function fire(
|
|||
if (!response.ok) {
|
||||
const message =
|
||||
errorMessage(payload) ?? (boundedBody.text.slice(0, 300) || 'request rejected');
|
||||
throw executionError(`Agent trigger fire was rejected (${response.status}): ${message}`, {
|
||||
mode: 'fire',
|
||||
throw executionError(`Agent trigger ${mode} was rejected (${response.status}): ${message}`, {
|
||||
mode,
|
||||
certainty: 'definite',
|
||||
retryable: isRetryableStatus(response.status),
|
||||
code: errorCode(payload) ?? 'FIRE_REJECTED',
|
||||
retryable:
|
||||
isRetryableStatus(response.status) ||
|
||||
(mode === 'continue' &&
|
||||
response.status === 409 &&
|
||||
errorCode(payload) === 'PARENT_NOT_READY'),
|
||||
deferWithoutAttempt:
|
||||
mode === 'continue' &&
|
||||
response.status === 409 &&
|
||||
errorCode(payload) === 'PARENT_NOT_READY',
|
||||
code: errorCode(payload) ?? (mode === 'fire' ? 'FIRE_REJECTED' : 'CONTINUE_REJECTED'),
|
||||
status: response.status,
|
||||
...(response.headers.get('retry-after') != null && {
|
||||
retryAfter: response.headers.get('retry-after') ?? undefined,
|
||||
|
|
@ -534,8 +664,8 @@ async function fire(
|
|||
});
|
||||
}
|
||||
if (boundedBody.truncated) {
|
||||
throw executionError('Agent trigger fire returned an oversized success response', {
|
||||
mode: 'fire',
|
||||
throw executionError(`Agent trigger ${mode} returned an oversized success response`, {
|
||||
mode,
|
||||
certainty: 'ambiguous',
|
||||
retryable: true,
|
||||
code: 'RESPONSE_TOO_LARGE',
|
||||
|
|
@ -548,18 +678,27 @@ async function fire(
|
|||
'status' in payload &&
|
||||
payload.status === 'aborted'
|
||||
) {
|
||||
throw executionError('Agent trigger fire was aborted before generation started', {
|
||||
mode: 'fire',
|
||||
throw executionError(`Agent trigger ${mode} was aborted before generation started`, {
|
||||
mode,
|
||||
certainty: 'definite',
|
||||
retryable: false,
|
||||
code: 'START_ABORTED',
|
||||
status: response.status,
|
||||
});
|
||||
}
|
||||
const result = parseFireResult(payload);
|
||||
const result = parseStartResult(payload, mode);
|
||||
if (result == null) {
|
||||
throw executionError('Agent trigger fire returned an invalid success response', {
|
||||
mode: 'fire',
|
||||
throw executionError(`Agent trigger ${mode} returned an invalid success response`, {
|
||||
mode,
|
||||
certainty: 'ambiguous',
|
||||
retryable: true,
|
||||
code: 'INVALID_RESPONSE',
|
||||
status: response.status,
|
||||
});
|
||||
}
|
||||
if (mode === 'continue' && result.conversationId !== envelope.target.conversationId) {
|
||||
throw executionError('Agent trigger continue returned a mismatched conversation', {
|
||||
mode,
|
||||
certainty: 'ambiguous',
|
||||
retryable: true,
|
||||
code: 'INVALID_RESPONSE',
|
||||
|
|
@ -567,6 +706,30 @@ async function fire(
|
|||
});
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
if (
|
||||
preparation?.status === 'ready' &&
|
||||
preparation.releaseOnDefiniteFailure != null &&
|
||||
error instanceof AgentTriggerExecutionError &&
|
||||
canReleasePreparedResult(error)
|
||||
) {
|
||||
try {
|
||||
await preparation.releaseOnDefiniteFailure();
|
||||
} catch (releaseError) {
|
||||
throw executionError(
|
||||
`Agent trigger ${mode} could not release its rejected preparation: ${
|
||||
releaseError instanceof Error ? releaseError.message : String(releaseError)
|
||||
}`,
|
||||
{
|
||||
mode,
|
||||
certainty: 'definite',
|
||||
retryable: true,
|
||||
code: 'PREPARATION_RELEASE_FAILED',
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
scope.cleanup();
|
||||
}
|
||||
|
|
@ -800,7 +963,13 @@ export function createAgentTriggerExecutionHost(
|
|||
envelope,
|
||||
{
|
||||
fire: (normalized, context) =>
|
||||
runAsPrincipal(normalized, context, () => fire(normalized, context, deps, timeoutMs)),
|
||||
runAsPrincipal(normalized, context, () =>
|
||||
startRun(normalized, context, deps, timeoutMs),
|
||||
),
|
||||
continue: (normalized, context) =>
|
||||
runAsPrincipal(normalized, context, () =>
|
||||
startRun(normalized, context, deps, timeoutMs),
|
||||
),
|
||||
steer: (normalized, context) =>
|
||||
runAsPrincipal(normalized, context, () =>
|
||||
steer(normalized, context, deps, timeoutMs),
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ export interface AgentTriggerServiceOptions {
|
|||
export interface AgentTriggerServiceDeps {
|
||||
fetch?: AgentTriggerExecutionHostDeps['fetch'];
|
||||
getTimezone?: AgentTriggerExecutionHostDeps['getTimezone'];
|
||||
prepareContinue?: AgentTriggerExecutionHostDeps['prepareContinue'];
|
||||
mintToken?: AgentTriggerExecutionHostDeps['mintToken'];
|
||||
timeoutMs?: number;
|
||||
methods?: AgentTriggerDeliveryPersistence;
|
||||
|
|
@ -201,6 +202,7 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
|
|||
((principal) => generateAgentTriggerToken(principal.userId, AGENT_TRIGGER_TOKEN_TTL)),
|
||||
...(deps.fetch != null && { fetch: deps.fetch }),
|
||||
...(deps.getTimezone != null && { getTimezone: deps.getTimezone }),
|
||||
...(deps.prepareContinue != null && { prepareContinue: deps.prepareContinue }),
|
||||
...(deps.timeoutMs != null && { timeoutMs: deps.timeoutMs }),
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -465,6 +465,9 @@ export interface CreateGenerationJobOptions {
|
|||
* status result. Creation may proceed only if that exact epoch is still
|
||||
* current or the stream has no durable job. */
|
||||
expectedPredecessorCreatedAt?: number;
|
||||
/** Atomically refuse to replace a running/paused predecessor while allowing
|
||||
* an absent or terminal predecessor. Used by automatic continuations. */
|
||||
rejectActivePredecessor?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1973,6 +1976,12 @@ class GenerationJobManagerClass {
|
|||
) {
|
||||
throw new Error('Invalid expected generation predecessor');
|
||||
}
|
||||
if (
|
||||
options.rejectActivePredecessor != null &&
|
||||
typeof options.rejectActivePredecessor !== 'boolean'
|
||||
) {
|
||||
throw new Error('Invalid active generation predecessor policy');
|
||||
}
|
||||
|
||||
const tenantId = getTenantId();
|
||||
const safeTenantId = tenantId && tenantId !== SYSTEM_TENANT_ID ? tenantId : undefined;
|
||||
|
|
@ -2010,6 +2019,7 @@ class GenerationJobManagerClass {
|
|||
options.recoveredSteerPayload,
|
||||
creationAttemptId,
|
||||
options.expectedPredecessorCreatedAt,
|
||||
options.rejectActivePredecessor,
|
||||
);
|
||||
} catch (error) {
|
||||
if (error instanceof JobPredecessorMismatchError) {
|
||||
|
|
|
|||
|
|
@ -32,8 +32,8 @@ async function waitFor(predicate: () => boolean): Promise<void> {
|
|||
|
||||
function jobHashFromCreationCall(call: unknown[]): Record<string, string> {
|
||||
const keyCount = Number(call[1]);
|
||||
// JOB_CREATE_LUA receives twelve scalar arguments before its HSET pairs.
|
||||
const fields = call.slice(14 + keyCount);
|
||||
// JOB_CREATE_LUA receives thirteen scalar arguments before its HSET pairs.
|
||||
const fields = call.slice(15 + keyCount);
|
||||
const hash = Object.fromEntries(
|
||||
Array.from({ length: fields.length / 2 }, (_, index) => [
|
||||
String(fields[index * 2]),
|
||||
|
|
|
|||
|
|
@ -24,7 +24,100 @@ function createConditionalJob(
|
|||
);
|
||||
}
|
||||
|
||||
/** An automatic continuation must never replace a parent turn that is still live. */
|
||||
function createWakeupJob(store: InMemoryJobStore, streamId: string) {
|
||||
return store.createJob(
|
||||
streamId,
|
||||
'owner-1',
|
||||
streamId,
|
||||
undefined,
|
||||
{ generationProtocolVersion: 2 },
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
'wakeup-create-attempt',
|
||||
undefined,
|
||||
true,
|
||||
);
|
||||
}
|
||||
|
||||
describe('generation predecessor create fence', () => {
|
||||
test('in-memory admission refuses an active predecessor and admits a settled one', async () => {
|
||||
const store = new InMemoryJobStore();
|
||||
const streamId = 'in-memory-active-predecessor-fence';
|
||||
try {
|
||||
const parent = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
|
||||
await expect(createWakeupJob(store, streamId)).rejects.toBeInstanceOf(
|
||||
JobPredecessorMismatchError,
|
||||
);
|
||||
/** The controller needs the live state to answer a finite PARENT_NOT_READY. */
|
||||
await expect(createWakeupJob(store, streamId)).rejects.toMatchObject({
|
||||
currentJob: { active: true, verified: true, status: 'running' },
|
||||
});
|
||||
|
||||
await expect(
|
||||
store.transitionStatus(streamId, {
|
||||
from: 'running',
|
||||
to: 'requires_action',
|
||||
expectCreatedAt: parent.createdAt,
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
await expect(createWakeupJob(store, streamId)).rejects.toMatchObject({
|
||||
currentJob: { active: true, status: 'requires_action' },
|
||||
});
|
||||
|
||||
await expect(
|
||||
store.transitionStatus(streamId, {
|
||||
from: 'requires_action',
|
||||
to: 'aborted',
|
||||
expectCreatedAt: parent.createdAt,
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
await store.updateJob(streamId, { terminalPersistencePending: true }, parent.createdAt);
|
||||
await expect(createWakeupJob(store, streamId)).rejects.toMatchObject({
|
||||
currentJob: { active: true, status: 'aborted' },
|
||||
});
|
||||
await store.updateJob(streamId, { terminalPersistencePending: false }, parent.createdAt);
|
||||
const wakeup = await createWakeupJob(store, streamId);
|
||||
expect(wakeup.createdAt).toBeGreaterThanOrEqual(parent.createdAt);
|
||||
} finally {
|
||||
await store.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
test('in-memory admission accepts an absent predecessor', async () => {
|
||||
const store = new InMemoryJobStore();
|
||||
const streamId = 'in-memory-absent-predecessor-fence';
|
||||
try {
|
||||
const wakeup = await createWakeupJob(store, streamId);
|
||||
expect(wakeup.createdAt).toEqual(expect.any(Number));
|
||||
} finally {
|
||||
await store.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
test('in-memory ordinary turns still replace an active predecessor', async () => {
|
||||
const store = new InMemoryJobStore();
|
||||
const streamId = 'in-memory-ordinary-replacement-unchanged';
|
||||
try {
|
||||
const first = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
/** Without the policy a user turn keeps replacing a running generation. */
|
||||
const second = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
expect(second.createdAt).toBeGreaterThanOrEqual(first.createdAt);
|
||||
} finally {
|
||||
await store.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
test('in-memory ordinary appends accept active states and reject retained terminal epochs', async () => {
|
||||
const store = new InMemoryJobStore();
|
||||
const streamId = 'in-memory-terminal-append-fence';
|
||||
|
|
|
|||
|
|
@ -26,6 +26,25 @@ function createConditionalJob(
|
|||
);
|
||||
}
|
||||
|
||||
/** An automatic continuation must never replace a parent turn that is still live. */
|
||||
function createWakeupJob(store: RedisJobStore, streamId: string, attempt: string) {
|
||||
return store.createJob(
|
||||
streamId,
|
||||
'owner-1',
|
||||
streamId,
|
||||
undefined,
|
||||
{ generationProtocolVersion: 2 },
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
attempt,
|
||||
undefined,
|
||||
true,
|
||||
);
|
||||
}
|
||||
|
||||
describe('Redis generation predecessor create fence', () => {
|
||||
const keyPrefix = `Predecessor-Fence-${process.pid}-${Date.now()}:`;
|
||||
let redis: RedisTestClient;
|
||||
|
|
@ -50,6 +69,83 @@ describe('Redis generation predecessor create fence', () => {
|
|||
await redis.quit();
|
||||
});
|
||||
|
||||
test('admission refuses an active predecessor without mutating it, and admits a settled one', async () => {
|
||||
const streamId = 'redis-active-predecessor-fence';
|
||||
const parent = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
await expect(
|
||||
store.appendChunk(
|
||||
streamId,
|
||||
{ event: 'on_message_delta', data: { delta: 'parent still writing' } },
|
||||
parent.createdAt,
|
||||
),
|
||||
).resolves.toBe(true);
|
||||
const chunksKey = `stream:{${streamId}}:chunks`;
|
||||
const chunksBefore = await redis.xrange(chunksKey, '-', '+');
|
||||
|
||||
await expect(createWakeupJob(store, streamId, 'redis-wakeup-running')).rejects.toMatchObject({
|
||||
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
||||
currentJob: { createdAt: parent.createdAt, active: true, verified: true, status: 'running' },
|
||||
});
|
||||
/** The refused continuation must leave the live parent turn exactly as it was. */
|
||||
await expect(store.getJob(streamId)).resolves.toMatchObject({
|
||||
createdAt: parent.createdAt,
|
||||
status: 'running',
|
||||
});
|
||||
await expect(redis.xrange(chunksKey, '-', '+')).resolves.toEqual(chunksBefore);
|
||||
|
||||
await expect(
|
||||
store.transitionStatus(streamId, {
|
||||
from: 'running',
|
||||
to: 'requires_action',
|
||||
expectCreatedAt: parent.createdAt,
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
await expect(
|
||||
createWakeupJob(store, streamId, 'redis-wakeup-requires-action'),
|
||||
).rejects.toMatchObject({
|
||||
code: 'GENERATION_PREDECESSOR_MISMATCH',
|
||||
currentJob: { active: true, status: 'requires_action' },
|
||||
});
|
||||
|
||||
await expect(
|
||||
store.transitionStatus(streamId, {
|
||||
from: 'requires_action',
|
||||
to: 'aborted',
|
||||
expectCreatedAt: parent.createdAt,
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
await store.updateJob(streamId, { terminalPersistencePending: true }, parent.createdAt);
|
||||
await expect(
|
||||
createWakeupJob(store, streamId, 'redis-wakeup-terminal-persistence'),
|
||||
).rejects.toMatchObject({
|
||||
currentJob: { active: true, status: 'aborted' },
|
||||
});
|
||||
await store.updateJob(streamId, { terminalPersistencePending: false }, parent.createdAt);
|
||||
const wakeup = await createWakeupJob(store, streamId, 'redis-wakeup-settled');
|
||||
expect(wakeup.createdAt).toBeGreaterThanOrEqual(parent.createdAt);
|
||||
});
|
||||
|
||||
test('admission accepts an absent predecessor', async () => {
|
||||
const streamId = 'redis-absent-predecessor-fence';
|
||||
const wakeup = await createWakeupJob(store, streamId, 'redis-wakeup-absent');
|
||||
expect(wakeup.createdAt).toEqual(expect.any(Number));
|
||||
});
|
||||
|
||||
test('ordinary turns still replace an active predecessor', async () => {
|
||||
const streamId = 'redis-ordinary-replacement-unchanged';
|
||||
const first = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
/** Without the policy a user turn keeps replacing a running generation. */
|
||||
const second = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
generationProtocolVersion: 2,
|
||||
});
|
||||
expect(second.createdAt).toBeGreaterThanOrEqual(first.createdAt);
|
||||
await expect(store.getJob(streamId)).resolves.toMatchObject({ createdAt: second.createdAt });
|
||||
});
|
||||
|
||||
test('mismatch returns the exact current generation without mutating durable state', async () => {
|
||||
const streamId = 'redis-predecessor-fence-no-mutation';
|
||||
const predecessor = await store.createJob(streamId, 'owner-1', streamId, undefined, {
|
||||
|
|
|
|||
|
|
@ -264,6 +264,7 @@ export class InMemoryJobStore implements IJobStoreV2 {
|
|||
recoveredSteerPayload?: RecoveredSteerPayload,
|
||||
creationAttemptId?: string,
|
||||
expectedPredecessorCreatedAt?: number,
|
||||
rejectActivePredecessor?: boolean,
|
||||
): Promise<CreatedJobData> {
|
||||
if (typeof userId !== 'string' || userId.length === 0) {
|
||||
throw new Error('Generation job requires a non-empty user id');
|
||||
|
|
@ -285,6 +286,9 @@ export class InMemoryJobStore implements IJobStoreV2 {
|
|||
) {
|
||||
throw new Error('Invalid expected generation predecessor');
|
||||
}
|
||||
if (rejectActivePredecessor != null && typeof rejectActivePredecessor !== 'boolean') {
|
||||
throw new Error('Invalid active generation predecessor policy');
|
||||
}
|
||||
const providerExecutionId = initialMetadata.providerExecutionId;
|
||||
if (
|
||||
providerExecutionId != null &&
|
||||
|
|
@ -341,6 +345,20 @@ export class InMemoryJobStore implements IJobStoreV2 {
|
|||
const assertExpectedPredecessorCompatible = (): void => {
|
||||
const current = this.jobs.get(streamId);
|
||||
const currentCreatedAt = current?.createdAt ?? this.getRetainedGenerationEpoch(streamId);
|
||||
if (
|
||||
rejectActivePredecessor === true &&
|
||||
(current?.status === 'running' ||
|
||||
current?.status === 'requires_action' ||
|
||||
current?.terminalPersistencePending === true)
|
||||
) {
|
||||
throw new JobPredecessorMismatchError({
|
||||
createdAt: current.createdAt,
|
||||
active: true,
|
||||
verified: true,
|
||||
status: current.status,
|
||||
...(current.conversationId !== undefined && { conversationId: current.conversationId }),
|
||||
});
|
||||
}
|
||||
if (
|
||||
expectedPredecessorCreatedAt == null ||
|
||||
currentCreatedAt === expectedPredecessorCreatedAt
|
||||
|
|
|
|||
|
|
@ -325,7 +325,7 @@ const REPLACEMENT_RECEIPT_ACK_LUA =
|
|||
* recoveredSteerPayloadJson | "",
|
||||
* generationProtocolVersion,
|
||||
* creationAttemptId | "",
|
||||
* expectedPredecessorCreatedAt | "",
|
||||
* expectedPredecessorCreatedAt | "", rejectActivePredecessor ("1" | "0"),
|
||||
* ...hsetPairs]
|
||||
* Returns: [previousUserId | "", previousTenantId | "", createdAt, "",
|
||||
* replacedCreatedAt | "", replacedStatus | "", replacedConversationId | "",
|
||||
|
|
@ -354,6 +354,7 @@ const JOB_CREATE_LUA =
|
|||
'local replacedProviderAbortReady = redis.call("HGET", KEYS[1], "providerAbortReady") ' +
|
||||
'local replacedProviderExecutionId = redis.call("HGET", KEYS[1], "providerExecutionId") ' +
|
||||
'local replacedProviderDrained = redis.call("HGET", KEYS[1], "providerDrained") ' +
|
||||
'local replacedTerminalPersistencePending = redis.call("HGET", KEYS[1], "terminalPersistencePending") ' +
|
||||
'local replacedProtocol = redis.call("HGET", KEYS[1], "generationProtocolVersion") ' +
|
||||
'local MAX_SAFE_EPOCH = 9007199254740991 ' +
|
||||
'local function isSafeEpoch(value) return type(value) == "number" and value >= 0 ' +
|
||||
|
|
@ -375,10 +376,14 @@ const JOB_CREATE_LUA =
|
|||
'local observedCreatedAt = replacedCreatedAt local observedStatus = replacedStatus ' +
|
||||
'local observedConversationId = replacedConversationId ' +
|
||||
'local observedActive = previousJobExists == 1 and ' +
|
||||
'(replacedStatus == "running" or replacedStatus == "requires_action") ' +
|
||||
'(replacedStatus == "running" or replacedStatus == "requires_action" ' +
|
||||
'or replacedTerminalPersistencePending == "1") ' +
|
||||
'if retainedEpoch and (not previousCreatedAt or retainedEpoch > previousCreatedAt) then ' +
|
||||
'previousCreatedAt = retainedEpoch observedCreatedAt = retainedEpochRaw ' +
|
||||
'observedStatus = nil observedConversationId = nil observedActive = false end ' +
|
||||
'if ARGV[13] == "1" and observedActive then ' +
|
||||
'return { previousUserId or "", previousTenantId or "", "0", "predecessor_mismatch", ' +
|
||||
'observedCreatedAt, observedStatus or "", observedConversationId or "", "1", "1" } end ' +
|
||||
'if ARGV[12] ~= "" and (not observedCreatedAt or observedCreatedAt ~= ARGV[12]) then ' +
|
||||
'return { previousUserId or "", previousTenantId or "", "0", "predecessor_mismatch", ' +
|
||||
'observedCreatedAt or ARGV[12], observedStatus or "", observedConversationId or "", ' +
|
||||
|
|
@ -526,7 +531,7 @@ const JOB_CREATE_LUA =
|
|||
'local ttl = tonumber(ARGV[1]) ' +
|
||||
'local generationEpochGraceTtl = tonumber(ARGV[3]) ' +
|
||||
'local hset = {} ' +
|
||||
'for i = 13, #ARGV do hset[#hset + 1] = ARGV[i] end ' +
|
||||
'for i = 14, #ARGV do hset[#hset + 1] = ARGV[i] end ' +
|
||||
'redis.call("HSET", KEYS[1], unpack(hset)) ' +
|
||||
'redis.call("HSET", KEYS[1], "createdAt", tostring(createdAt)) ' +
|
||||
'if ARGV[11] ~= "" then redis.call("HSET", KEYS[1], "__creationAttemptId", ARGV[11]) end ' +
|
||||
|
|
@ -1769,6 +1774,7 @@ export class RedisJobStore implements IJobStoreV2 {
|
|||
recoveredSteerPayload?: RecoveredSteerPayload,
|
||||
creationAttemptId?: string,
|
||||
expectedPredecessorCreatedAt?: number,
|
||||
rejectActivePredecessor?: boolean,
|
||||
): Promise<CreatedJobData> {
|
||||
if (typeof userId !== 'string' || userId.length === 0) {
|
||||
throw new Error('Generation job requires a non-empty user id');
|
||||
|
|
@ -1790,6 +1796,9 @@ export class RedisJobStore implements IJobStoreV2 {
|
|||
) {
|
||||
throw new Error('Invalid expected generation predecessor');
|
||||
}
|
||||
if (rejectActivePredecessor != null && typeof rejectActivePredecessor !== 'boolean') {
|
||||
throw new Error('Invalid active generation predecessor policy');
|
||||
}
|
||||
const providerExecutionId = initialMetadata.providerExecutionId;
|
||||
if (
|
||||
providerExecutionId != null &&
|
||||
|
|
@ -1868,6 +1877,7 @@ export class RedisJobStore implements IJobStoreV2 {
|
|||
String(job.generationProtocolVersion),
|
||||
creationAttemptId ?? '',
|
||||
expectedPredecessorCreatedAt == null ? '' : String(expectedPredecessorCreatedAt),
|
||||
rejectActivePredecessor === true ? '1' : '0',
|
||||
...hsetPairs,
|
||||
);
|
||||
if (Array.isArray(previousOwner) && previousOwner[3] === 'claim_lost') {
|
||||
|
|
|
|||
|
|
@ -741,6 +741,7 @@ export interface IJobStoreV2 extends IJobStore {
|
|||
recoveredSteerPayload?: RecoveredSteerPayload,
|
||||
creationAttemptId?: string,
|
||||
expectedPredecessorCreatedAt?: number,
|
||||
rejectActivePredecessor?: boolean,
|
||||
): Promise<CreatedJobData>;
|
||||
|
||||
/** Remove transaction-time predecessor receipts after their handoff was
|
||||
|
|
|
|||
|
|
@ -29,6 +29,9 @@ let updateMessageText: ReturnType<typeof createMessageMethods>['updateMessageTex
|
|||
let deleteMessagesSince: ReturnType<typeof createMessageMethods>['deleteMessagesSince'];
|
||||
let recordMessage: ReturnType<typeof createMessageMethods>['recordMessage'];
|
||||
let claimSubagentTaskResult: ReturnType<typeof createMessageMethods>['claimSubagentTaskResult'];
|
||||
let releaseSubagentTaskResultClaim: ReturnType<
|
||||
typeof createMessageMethods
|
||||
>['releaseSubagentTaskResultClaim'];
|
||||
|
||||
beforeAll(async () => {
|
||||
mongoServer = await MongoMemoryServer.create();
|
||||
|
|
@ -49,6 +52,7 @@ beforeAll(async () => {
|
|||
deleteMessagesSince = methods.deleteMessagesSince;
|
||||
recordMessage = methods.recordMessage;
|
||||
claimSubagentTaskResult = methods.claimSubagentTaskResult;
|
||||
releaseSubagentTaskResultClaim = methods.releaseSubagentTaskResultClaim;
|
||||
|
||||
await mongoose.connect(mongoUri);
|
||||
});
|
||||
|
|
@ -1569,6 +1573,7 @@ describe('Message Operations', () => {
|
|||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
});
|
||||
expect(first.status).toBe('acquired');
|
||||
|
|
@ -1579,14 +1584,136 @@ describe('Message Operations', () => {
|
|||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
});
|
||||
expect(retried.status).toBe('acquired');
|
||||
|
||||
/** Another invocation is told it was collected instead of handed a copy. */
|
||||
await expect(
|
||||
claimSubagentTaskResult({ userId: 'user123', conversationId, taskId, claimId: 'poll-2' }),
|
||||
).resolves.toEqual({ status: 'claimed' });
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-2',
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'claimed' });
|
||||
});
|
||||
|
||||
it('elects either a manual poll or one idempotent automatic wakeup', async () => {
|
||||
const manualTaskId = uuidv4();
|
||||
const wakeupTaskId = uuidv4();
|
||||
const conversationId = uuidv4();
|
||||
await terminalResult(manualTaskId, conversationId, 'completed');
|
||||
await terminalResult(wakeupTaskId, conversationId, 'completed');
|
||||
|
||||
await expect(
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId: manualTaskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'acquired' });
|
||||
await expect(
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId: manualTaskId,
|
||||
kind: 'wakeup',
|
||||
claimId: 'delivery-1',
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'claimed' });
|
||||
|
||||
const wakeupClaim = {
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId: wakeupTaskId,
|
||||
kind: 'wakeup' as const,
|
||||
claimId: 'delivery-2',
|
||||
};
|
||||
await expect(claimSubagentTaskResult(wakeupClaim)).resolves.toMatchObject({
|
||||
status: 'acquired',
|
||||
});
|
||||
await expect(claimSubagentTaskResult(wakeupClaim)).resolves.toMatchObject({
|
||||
status: 'acquired',
|
||||
});
|
||||
await expect(
|
||||
claimSubagentTaskResult({ ...wakeupClaim, claimId: 'delivery-3' }),
|
||||
).resolves.toMatchObject({ status: 'claimed' });
|
||||
await expect(
|
||||
claimSubagentTaskResult({ ...wakeupClaim, kind: 'manual', claimId: 'poll-2' }),
|
||||
).resolves.toMatchObject({ status: 'claimed' });
|
||||
});
|
||||
|
||||
it('preserves and upgrades retries of legacy manual claims without a kind', async () => {
|
||||
const taskId = uuidv4();
|
||||
const conversationId = uuidv4();
|
||||
await terminalResult(taskId, conversationId, 'completed');
|
||||
await claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'legacy-poll',
|
||||
});
|
||||
await Message.collection.updateOne(
|
||||
{ user: 'user123', conversationId, messageId: `${taskId}:assistant` },
|
||||
{ $unset: { 'subagentTask.resultClaim.kind': '' } },
|
||||
);
|
||||
|
||||
await expect(
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'legacy-poll',
|
||||
}),
|
||||
).resolves.toMatchObject({
|
||||
status: 'acquired',
|
||||
message: { subagentTask: { resultClaim: { kind: 'manual', claimId: 'legacy-poll' } } },
|
||||
});
|
||||
await expect(
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'wakeup',
|
||||
claimId: 'legacy-poll',
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'claimed' });
|
||||
});
|
||||
|
||||
it('releases only the exact rejected wakeup so manual collection can take over', async () => {
|
||||
const taskId = uuidv4();
|
||||
const conversationId = uuidv4();
|
||||
await terminalResult(taskId, conversationId, 'completed');
|
||||
const wakeup = {
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'wakeup' as const,
|
||||
claimId: 'delivery-1',
|
||||
};
|
||||
await expect(claimSubagentTaskResult(wakeup)).resolves.toMatchObject({
|
||||
status: 'acquired',
|
||||
});
|
||||
await expect(
|
||||
releaseSubagentTaskResultClaim({ ...wakeup, claimId: 'another-delivery' }),
|
||||
).resolves.toBe(false);
|
||||
await expect(releaseSubagentTaskResultClaim(wakeup)).resolves.toBe(true);
|
||||
await expect(
|
||||
claimSubagentTaskResult({
|
||||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-after-rejection',
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'acquired' });
|
||||
});
|
||||
|
||||
it('reports a result that is missing or still running as not found', async () => {
|
||||
|
|
@ -1599,6 +1726,7 @@ describe('Message Operations', () => {
|
|||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId: runningTaskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
}),
|
||||
).resolves.toEqual({ status: 'not_found' });
|
||||
|
|
@ -1608,6 +1736,7 @@ describe('Message Operations', () => {
|
|||
userId: 'user123',
|
||||
conversationId,
|
||||
taskId: uuidv4(),
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
}),
|
||||
).resolves.toEqual({ status: 'not_found' });
|
||||
|
|
@ -1623,6 +1752,7 @@ describe('Message Operations', () => {
|
|||
userId: 'other-user',
|
||||
conversationId,
|
||||
taskId,
|
||||
kind: 'manual',
|
||||
claimId: 'poll-1',
|
||||
}),
|
||||
).resolves.toEqual({ status: 'not_found' });
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ interface MessageQueryOptions {
|
|||
|
||||
export type SubagentTaskResultClaim =
|
||||
| { status: 'not_found' }
|
||||
| { status: 'claimed' }
|
||||
| { status: 'claimed'; message: IMessage }
|
||||
| { status: 'acquired'; message: IMessage };
|
||||
|
||||
export interface MessageMethods {
|
||||
|
|
@ -91,8 +91,16 @@ export interface MessageMethods {
|
|||
userId: string;
|
||||
conversationId: string;
|
||||
taskId: string;
|
||||
kind: 'manual' | 'wakeup';
|
||||
claimId: string;
|
||||
}): Promise<SubagentTaskResultClaim>;
|
||||
releaseSubagentTaskResultClaim(params: {
|
||||
userId: string;
|
||||
conversationId: string;
|
||||
taskId: string;
|
||||
kind: 'manual' | 'wakeup';
|
||||
claimId: string;
|
||||
}): Promise<boolean>;
|
||||
deleteMessagesSince(
|
||||
userId: string,
|
||||
params: { messageId: string; conversationId: string },
|
||||
|
|
@ -529,21 +537,19 @@ export function createMessageMethods(mongoose: typeof import('mongoose')): Messa
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assigns one durable terminal child result to the polling invocation that collects
|
||||
* it. The same invocation may re-acquire, so a poll whose response was lost recovers
|
||||
* the result it never received; a different invocation is told it was already
|
||||
* collected rather than handed a second copy.
|
||||
*/
|
||||
/** Atomically assigns one durable terminal child result to either its
|
||||
* explicit poller or one idempotent automatic wakeup delivery. */
|
||||
async function claimSubagentTaskResult({
|
||||
userId,
|
||||
conversationId,
|
||||
taskId,
|
||||
kind,
|
||||
claimId,
|
||||
}: {
|
||||
userId: string;
|
||||
conversationId: string;
|
||||
taskId: string;
|
||||
kind: 'manual' | 'wakeup';
|
||||
claimId: string;
|
||||
}): Promise<SubagentTaskResultClaim> {
|
||||
if (
|
||||
|
|
@ -551,38 +557,112 @@ export function createMessageMethods(mongoose: typeof import('mongoose')): Messa
|
|||
taskId.length > 256 ||
|
||||
conversationId.length === 0 ||
|
||||
conversationId.length > 256 ||
|
||||
(kind !== 'manual' && kind !== 'wakeup') ||
|
||||
claimId.length === 0 ||
|
||||
claimId.length > 128
|
||||
) {
|
||||
throw new TypeError('Invalid subagent task result claim');
|
||||
}
|
||||
const Message = mongoose.models.Message as Model<IMessage>;
|
||||
const filter = {
|
||||
user: userId,
|
||||
conversationId,
|
||||
messageId: `${taskId}:assistant`,
|
||||
'subagentTask.status': { $in: ['completed', 'error', 'cancelled'] },
|
||||
const messageId = `${taskId}:assistant`;
|
||||
const terminal = ['completed', 'error', 'cancelled'];
|
||||
const claim = {
|
||||
kind,
|
||||
claimId,
|
||||
claimedAt: new Date(),
|
||||
};
|
||||
const claimable = {
|
||||
$or: [
|
||||
{ 'subagentTask.resultClaim': { $exists: false } },
|
||||
{
|
||||
'subagentTask.resultClaim.kind': kind,
|
||||
'subagentTask.resultClaim.claimId': claimId,
|
||||
},
|
||||
...(kind === 'manual'
|
||||
? [
|
||||
{
|
||||
'subagentTask.resultClaim.kind': { $exists: false },
|
||||
'subagentTask.resultClaim.claimId': claimId,
|
||||
},
|
||||
]
|
||||
: []),
|
||||
],
|
||||
};
|
||||
const projection = {
|
||||
messageId: 1,
|
||||
conversationId: 1,
|
||||
parentMessageId: 1,
|
||||
sender: 1,
|
||||
text: 1,
|
||||
error: 1,
|
||||
createdAt: 1,
|
||||
updatedAt: 1,
|
||||
subagentTask: 1,
|
||||
};
|
||||
const acquired = await Message.findOneAndUpdate(
|
||||
{
|
||||
...filter,
|
||||
$or: [
|
||||
{ 'subagentTask.resultClaim': { $exists: false } },
|
||||
{ 'subagentTask.resultClaim.claimId': claimId },
|
||||
],
|
||||
},
|
||||
{ $set: { 'subagentTask.resultClaim': { claimId, claimedAt: new Date() } } },
|
||||
{
|
||||
new: true,
|
||||
timestamps: false,
|
||||
projection: { messageId: 1, conversationId: 1, text: 1, subagentTask: 1 },
|
||||
user: userId,
|
||||
conversationId,
|
||||
messageId,
|
||||
'subagentTask.status': { $in: terminal },
|
||||
...claimable,
|
||||
},
|
||||
{ $set: { 'subagentTask.resultClaim': claim } },
|
||||
{ new: true, projection },
|
||||
).lean<IMessage | null>();
|
||||
if (acquired != null) {
|
||||
return { status: 'acquired', message: acquired };
|
||||
}
|
||||
const existing = await Message.exists(filter);
|
||||
return existing == null ? { status: 'not_found' } : { status: 'claimed' };
|
||||
const existing = await Message.findOne({
|
||||
user: userId,
|
||||
conversationId,
|
||||
messageId,
|
||||
'subagentTask.status': { $in: terminal },
|
||||
})
|
||||
.select(projection)
|
||||
.lean<IMessage | null>();
|
||||
return existing == null ? { status: 'not_found' } : { status: 'claimed', message: existing };
|
||||
}
|
||||
|
||||
/** Releases only the exact consumer assignment. This is used when a
|
||||
* pre-admission automatic continuation is definitively rejected, allowing a
|
||||
* later manual poll (or the same delivery retry) to claim the durable result. */
|
||||
async function releaseSubagentTaskResultClaim({
|
||||
userId,
|
||||
conversationId,
|
||||
taskId,
|
||||
kind,
|
||||
claimId,
|
||||
}: {
|
||||
userId: string;
|
||||
conversationId: string;
|
||||
taskId: string;
|
||||
kind: 'manual' | 'wakeup';
|
||||
claimId: string;
|
||||
}): Promise<boolean> {
|
||||
if (
|
||||
taskId.length === 0 ||
|
||||
taskId.length > 256 ||
|
||||
conversationId.length === 0 ||
|
||||
conversationId.length > 256 ||
|
||||
(kind !== 'manual' && kind !== 'wakeup') ||
|
||||
claimId.length === 0 ||
|
||||
claimId.length > 128
|
||||
) {
|
||||
throw new TypeError('Invalid subagent task result claim release');
|
||||
}
|
||||
const Message = mongoose.models.Message as Model<IMessage>;
|
||||
const result = await Message.updateOne(
|
||||
{
|
||||
user: userId,
|
||||
conversationId,
|
||||
messageId: `${taskId}:assistant`,
|
||||
'subagentTask.resultClaim.kind': kind,
|
||||
'subagentTask.resultClaim.claimId': claimId,
|
||||
},
|
||||
{ $unset: { 'subagentTask.resultClaim': 1 } },
|
||||
);
|
||||
return result.modifiedCount === 1;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -723,6 +803,7 @@ export function createMessageMethods(mongoose: typeof import('mongoose')): Messa
|
|||
updateToolCallResult,
|
||||
updateMessage,
|
||||
claimSubagentTaskResult,
|
||||
releaseSubagentTaskResultClaim,
|
||||
deleteMessagesSince,
|
||||
getMessages,
|
||||
getMessage,
|
||||
|
|
|
|||
|
|
@ -138,6 +138,7 @@ const messageSchema: Schema<IMessage> = new Schema(
|
|||
subagentTask: {
|
||||
type: {
|
||||
attemptKey: { type: String, required: true },
|
||||
parentRunId: { type: String },
|
||||
requestFingerprint: { type: String },
|
||||
status: {
|
||||
type: String,
|
||||
|
|
@ -146,6 +147,7 @@ const messageSchema: Schema<IMessage> = new Schema(
|
|||
},
|
||||
resultClaim: {
|
||||
type: {
|
||||
kind: { type: String, enum: ['manual', 'wakeup'], required: true },
|
||||
claimId: { type: String, required: true },
|
||||
claimedAt: { type: Date, required: true },
|
||||
},
|
||||
|
|
|
|||
|
|
@ -51,10 +51,12 @@ export interface IMessage extends Document {
|
|||
/** Server-private durable idempotency marker for one detached subagent turn. */
|
||||
subagentTask?: {
|
||||
attemptKey: string;
|
||||
/** Parent response that initiated this exact child task. */
|
||||
parentRunId?: string;
|
||||
requestFingerprint?: string;
|
||||
status: 'running' | 'completed' | 'error' | 'cancelled';
|
||||
/** Records which polling invocation collected this terminal result. */
|
||||
resultClaim?: {
|
||||
kind: 'manual' | 'wakeup';
|
||||
claimId: string;
|
||||
claimedAt: Date;
|
||||
};
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue