mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-08-27 04:07:05 +00:00
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks * fix: fence event actor checkpoint uncertainty * fix: satisfy event actor type contracts * fix: make actor reconciliation recoverable * fix: preserve event actor lifecycle transitions * fix: fence event actor lifecycle outcomes * fix: enforce event actor lifecycle ownership * fix: retain event actor settlement proof * 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal Repair and compensation deleted the reconciliation row they resolved, which was the only durable proof that the invocation had already applied an external action. A delayed duplicate owner could then reacquire the same invocation id and repeat that action. Both resolutions now retire their receipt to `settled` and record how it settled, so the same-id tombstone survives; `history_repaired` and `action_compensated` still force a cold rebuild. Compensation undoes the effect without re-authorizing the delivery, so a legitimate retry must arrive under a new invocation id. A retried repair converges on its own receipt. Bound the journal so a long-lived actor cannot grow its conversation document without limit: a new fence is admitted only when no active lifecycle row exists, so a capped push can evict nothing but the oldest settled receipts. Stop shipping the unbounded source payload on every bound-child continuation. It rode the delivery body regardless of the feature flag while the sibling `fire` body deliberately sends event identity alone, so a large webhook payload could push a previously working delivery past the chat route's body limit. The actor binds an invocation from identity and never builds the prompt from it. Blank the positional token map on warm continuations. It is derived from the full DB history, while a warm run executes on checkpoint-restored state, so its indices address different messages and the pruner never recounts them — misattributing cached counts to the wrong messages in both directions. Keep the replaced-claim exit on its cleanup path when preserving reconciliation fails: the committing CAS already left a blocking row, so the failed status upgrade costs provenance, not safety. * 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry Give the warm-continuation client test a populated token map and a real cross-run summary so its assertions bite: the positional map must arrive blank (checkpoint-restored state no longer matches DB-derived indices, and the pruner never recounts a populated entry) while `initialSummary` must pass through unchanged — it rides the system tail and summarizes pre-boundary turns that were excluded from the very history the committed checkpoint was built from, so blanking it would silently drop context no warm run can recover. * ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal A compensated receipt still tombstones its invocation id, but its external effect was explicitly undone — the terminal handler nonetheless replayed every settled lifecycle's stored action as authoritative and settled the public outcome as applied, telling an action-aware source the operation stands and suppressing the new-invocation retry compensation requires. The handler now settles a compensated invocation as failed with an explicit compensation error, overriding even fresh applied run evidence from a replayed generation; verified and repaired receipts continue to replay applied. Receipt eviction is now primarily age-based: a stale same-id owner is bounded by time, not by how many newer invocations settle, so the previous count-only slice let a high-rate actor evict a tombstone while its delayed duplicate owner could still wake and repeat the action. Admission prunes only settled receipts older than a retention window that dwarfs every generation, job, and delivery-retry lifetime, and the count cap is demoted to a raised document-size backstop. * 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts The terminal handler read its lifecycle snapshot, verified history, and then settled the public outcome — so a compensation resolving the same receipt during that window lost: the handler settled applied from its stale snapshot and no retry could ever change the replay-identity-locked outcome. The receipt's status CAS is now the serialization point: verification resolves the receipt BEFORE settling, whichever transition wins determines the public outcome, and a crash between resolve and settle converges through the retained receipt's replay. The verified-replay probe requires the receipt's own resolution, so a compensated receipt can never satisfy a verification retry. This inverts the settle-before-receipt ordering deliberately: that ordering guarded proof that resolution used to delete, and the receipt now retains its full action proof through resolution. The document-size cap is no longer an eviction quota. A receipt inside its retention window is never discarded: when the journal holds a full cap of unexpired receipts, new invocations are refused fail-closed until receipts age out, making duplicate protection and document integrity simultaneous invariants instead of a rate-dependent trade. * 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path Skill primes are spliced into the message list directly ahead of the newest message, and a warm continuation forwards only that newest message — so a checkpoint-restored actor would keep serving the prime bodies baked in at its last cold start and never observe an edited or newly attached skill. Until the actor head carries a context fingerprint that forces a cold rebuild when the agent's skill context changes, agents with always-apply or manual skill primes stay on the legacy path, which re-primes fresh bodies every turn: correct on every event, just never warm. * 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes History-derived re-priming was a third path into the same staleness class: an actor that previously invoked a skill carries no request-time prime arrays, yet primeInvokedSkills re-resolves that skill's current body from history each turn and the warm slice drops the reconstruction — leaving the checkpoint's old body active after edits. The fork gate now keys on the priming hook itself (present exactly when the skills capability is enabled) alongside the request-time arrays, so every skill-body path routes to the legacy rebuild until #15235's context fingerprint restores warm continuation for skill-bearing actors. * 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage The executor read applied-action evidence from the run-step collection the instant sendMessage resolved, but that collection is populated asynchronously — an applied invocation could classify as actionless (runSteps still empty while the tool result already streamed), discarding its fork and stranding the actor cold while the terminal handler later settled the same delivery as applied from the persisted evidence. Authoritative proof is now recorded in graph context the moment the expected tool executes: the request-owned recorder observes the tool-end chain (which ToolNode dispatches synchronously with both input and output) and applies the same fences as run-step evidence — exact tool name with the MCP-suffixed form, the declared argument subset against the execution input, an error-free result, and the background non-execution receipt exclusion. readAppliedAction consults the receipt first; run-step inspection remains the fallback for paths that bypass the tool-end chain. Regression coverage reproduces the observed ordering: the real executor commits the head from the receipt while run steps are empty, warm-continues the next event, and never re-executes the action; recorder fences and the receipt-first controller wiring are covered separately. * 🎯 fix: Supply Execution Arguments to the Tool End Callback The live Vertex + MCP canary exposed a contract mismatch the synthetic fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end callback with output only, while the action recorder must verify the declared argument subset against the execution input. The receipt never qualified, every turn fell back to cold history rebuilds, and the tournament advanced with zero actor heads and zero retained checkpoints while looking successful. The execution handler owns both halves at the same moment, so the fix is at the source rather than a correlation store: ToolEndCallbackData gains the executed call's input and every callback site passes tc.args. A handler-level regression drives the real createToolExecuteHandler and asserts the callback receives both fields; recorder regressions pin the production shapes — an output-only tool end must starve an argument-fenced receipt rather than trust an unfenced match, and still qualifies a name-only expected action. * 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions The background-claim callback reports the ORIGINAL tool's name for artifact attribution on the poll turn that harvests a completed task. A name-only expected action could therefore be impersonated by work some earlier turn dispatched: the recorder would attribute that delivery to the current invocation and commit a head whose state never contained the invocation's own action. The run-step evidence path never had this hole — it sees the poll tool's name — so the recorder must match its provenance discipline. Delivery callbacks now carry an explicit backgroundDelivery marker set at the one site that rewrites the name, and the recorder ignores marked deliveries outright. Regressions pin both halves of the contract: the delivery callback must carry the marker with the poll call's arguments, and a marked delivery can never qualify even a name-only expected action. * 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions Three closeout-round findings, each converted into an invariant. Every legacy-path invalidation now advances a durable epoch — including for headless and already cold-marked actors, where the marker alone leaves no CAS-visible trace — and the actor-head CAS requires the epoch observed at preparation. A concurrently prepared fork whose history predates an intervening legacy turn can no longer commit past it; the commit reports an ordinary conflict and journals. Execution identity is now emitted before post-execution output policy: when a side-effecting tool succeeds but its returned content is withheld by the output filter, the callback delivers an outputFiltered receipt with blank content — the recorder accepts it as proof (rejecting model-detached calls it cannot distinguish through the blank shape), the artifact path never sees it, and an applied action is no longer reclassified actionless and re-executed on retry. Background-capable expected actions stay off the fork path: dispatch returns a launch handle every evidence fence correctly rejects, and the completion is provenance-marked as another turn's work, so a fork would settle actionless before the external effect lands with nothing to stop a retry from dispatching it again. The gate mirrors the MCP-suffix name matching of the evidence path. * 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance The epoch fenced only the legacy turn's start: a fork preparing after the begin invalidation but before the turn's message persistence observed the new epoch and cold marker, rebuilt from history that did not yet contain the turn, and committed cleanly because nothing advanced the epoch again — making the incomplete rebuild authoritative and clearing the marker. Every legacy event turn now seals its invalidation at terminal persistence with a second epoch advance, on the success, replaced-claim, and error exits alike. Sealing deliberately carries no quiescence requirement — it must succeed while a fork fence is active, because that is exactly the mid-turn race it defeats — and a fork that already committed against the begin epoch is healed the same way: the seal re-marks the head cold, so the next event rebuilds with complete history. Seal failure never diverts the turn's own exit; the begin bump still fences everything prepared before the turn. * 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence The second epoch advance could not make a legacy turn atomic, and three findings shared that root cause: two conditional updates left a headless gap a fork could create the head inside; the error exit sealed before saveErrorTurn made the error history durable; and any crash or failure between persistence and sealing left an incomplete fork authoritative, because the seal was best-effort and its failure was swallowed. A legacy turn now carries one durable fence. A token is written before execution by a single update-pipeline write — no two-write gap, and the cold marker is applied only where a head exists via $cond/$$REMOVE. While the token is present no fork may prepare (the adapter refuses) or commit (the CAS requires its absence), because the turn's messages are not yet durable. One atomic write clears the exact token and advances the epoch once history is persisted — after saveErrorTurn on the error exit — and success, replacement, and error exits all route through it. Failure is now fail-closed rather than silent: a failed seal leaves the token set, which keeps blocking forks and is logged as such, and a fence abandoned by a crashed turn is reclaimed only once stale, advancing the epoch and marking any head cold so the next event rebuilds from whatever history actually survived. * fix: serialize legacy event actor turns * fix: close legacy actor fence ownership gaps * fix: preserve legacy actor persistence fences
This commit is contained in:
parent
56f0cde9a5
commit
68fc46a055
35 changed files with 5759 additions and 54 deletions
|
|
@ -9,4 +9,6 @@
|
|||
- **Subagent activity stream**: an observational, task-scoped live projection of bounded child progress for the currently open private panel. It may cross API replicas through Redis, never carries hidden reasoning text, and never controls or settles execution. The durable child thread remains canonical and its existing polling view is the fallback for missed or unavailable live events.
|
||||
- **Agent event handling outcome**: the durable, generation-fenced result of a previously accepted event delivery. `started` proves generation admission; terminal states distinguish verified tool application, clean completion without action, failure, and cancellation. Transport success remains separate so an accepted event cannot masquerade as completed work.
|
||||
- **Agent event expected action**: an optional source-declared tool name and bounded argument subset evaluated against host-observed completed run steps. It is evidence policy, not authorization and not a model-authored success claim.
|
||||
- **Event actor head**: the private, durable pointer on an event-bound child conversation to its latest committed LangGraph checkpoint, plus one previous checkpoint for safe cleanup. Only a qualifying applied action advances it through compare-and-swap; failed, cancelled, or no-action invocations leave it unchanged. A legacy-path event marks the head for a cold rebuild from durable message history before fork mode can resume. Every applied commit conflict, unverified commit, or post-commit persistence failure is retained in a private reconciliation journal that blocks later actor turns instead of continuing from stale state; an exact marker can be cleared only after its checkpoint is verified authoritative, its history is repaired, or its external action is explicitly compensated.
|
||||
- **Event actor invocation fork**: a delivery-owned checkpoint namespace copied from the event actor head. A warm invocation receives only the new trusted event, then commits its terminal checkpoint when the expected action is observed or deletes the fork otherwise. Pause-capable actors remain on the existing resumable path until forked HITL has an explicit contract.
|
||||
- **Theme definition**: a versioned, data-only description of LibreChat semantic colors and shared appearance roles, optionally specialized by light or dark mode. The theme module validates and resolves partial definitions against bundled defaults before adapters apply them. A theme definition does not contain arbitrary CSS, application behavior, or alternate feature layouts.
|
||||
|
|
|
|||
|
|
@ -72,6 +72,31 @@ const mockExemptFromConcurrencyLimiter = jest.fn();
|
|||
const mockRecordScheduleOutcome = jest.fn();
|
||||
const mockIsScheduleLive = jest.fn();
|
||||
const mockDeleteAgentCheckpoint = jest.fn();
|
||||
const mockExecuteAgentEventActor = jest.fn();
|
||||
const mockFindAgentEventAppliedAction = jest.fn();
|
||||
/** Faithful stand-in for the graph-context action recorder: first observed
|
||||
* tool end with a name becomes the receipt. Matching fences are unit-tested
|
||||
* against the real implementation in packages/api. */
|
||||
const mockCreateAgentEventActionRecorder = jest.fn(() => {
|
||||
let receipt;
|
||||
return {
|
||||
observeToolEnd: (data) => {
|
||||
if (receipt == null && data?.output?.name != null) {
|
||||
receipt = {
|
||||
toolName: data.output.name,
|
||||
...(data.output.tool_call_id == null ? {} : { toolCallId: data.output.tool_call_id }),
|
||||
};
|
||||
}
|
||||
},
|
||||
read: () => receipt,
|
||||
};
|
||||
});
|
||||
const mockGetAgentEventActorSnapshot = jest.fn();
|
||||
const mockCommitAgentEventActorState = jest.fn();
|
||||
const mockBeginAgentEventActorLegacyTurn = jest.fn();
|
||||
const mockCompleteAgentEventActorLegacyTurn = jest.fn();
|
||||
const mockRecordAgentEventActorReconciliation = jest.fn();
|
||||
const mockResolveAgentEventActorReconciliation = jest.fn();
|
||||
const mockStartupTelemetry = {
|
||||
mark: jest.fn(),
|
||||
setStreamId: jest.fn(),
|
||||
|
|
@ -198,6 +223,12 @@ jest.mock('@librechat/api', () => ({
|
|||
return messages.length === 0;
|
||||
},
|
||||
deleteAgentCheckpoint: (...args) => mockDeleteAgentCheckpoint(...args),
|
||||
executeAgentEventActor: (...args) => mockExecuteAgentEventActor(...args),
|
||||
findAgentEventAppliedAction: (...args) => mockFindAgentEventAppliedAction(...args),
|
||||
createAgentEventActionRecorder: (...args) => mockCreateAgentEventActionRecorder(...args),
|
||||
isHITLEnabled: (policy) => policy?.enabled === true,
|
||||
agentRequestsAskUserQuestion: (agent) =>
|
||||
agent?.toolDefinitions?.some((tool) => tool?.name === 'ask_user_question') === true,
|
||||
isAgentEventRetentionActive: (expiredAt) =>
|
||||
expiredAt == null || new Date(expiredAt).getTime() > Date.now(),
|
||||
createMCPRuntimeRequestBody: ({ messageId, conversationId, parentMessageId }) => ({
|
||||
|
|
@ -228,6 +259,14 @@ jest.mock('~/models', () => ({
|
|||
saveConvo: (...args) => mockSaveConvo(...args),
|
||||
getMessages: (...args) => mockGetMessages(...args),
|
||||
getConvo: (...args) => mockGetConvo(...args),
|
||||
getAgentEventActorSnapshot: (...args) => mockGetAgentEventActorSnapshot(...args),
|
||||
commitAgentEventActorState: (...args) => mockCommitAgentEventActorState(...args),
|
||||
beginAgentEventActorLegacyTurn: (...args) => mockBeginAgentEventActorLegacyTurn(...args),
|
||||
completeAgentEventActorLegacyTurn: (...args) => mockCompleteAgentEventActorLegacyTurn(...args),
|
||||
recordAgentEventActorReconciliation: (...args) =>
|
||||
mockRecordAgentEventActorReconciliation(...args),
|
||||
resolveAgentEventActorReconciliation: (...args) =>
|
||||
mockResolveAgentEventActorReconciliation(...args),
|
||||
isAgentTriggerPrincipalActive: (...args) => mockIsAgentTriggerPrincipalActive(...args),
|
||||
isSubagentOwnerAdmissible: (...args) => mockIsSubagentOwnerAdmissible(...args),
|
||||
}));
|
||||
|
|
@ -343,6 +382,12 @@ describe('ResumableAgentController resume metadata', () => {
|
|||
mockSaveMessage.mockResolvedValue({});
|
||||
mockSaveConvo.mockResolvedValue({});
|
||||
mockDeleteAgentCheckpoint.mockResolvedValue(undefined);
|
||||
mockGetAgentEventActorSnapshot.mockResolvedValue({ state: null, reconciliations: [] });
|
||||
mockCommitAgentEventActorState.mockResolvedValue({ status: 'stale' });
|
||||
mockBeginAgentEventActorLegacyTurn.mockResolvedValue(true);
|
||||
mockCompleteAgentEventActorLegacyTurn.mockResolvedValue(true);
|
||||
mockRecordAgentEventActorReconciliation.mockResolvedValue(true);
|
||||
mockResolveAgentEventActorReconciliation.mockResolvedValue(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
|
|
@ -3933,6 +3978,810 @@ describe('ResumableAgentController resume metadata', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('routes an enabled authenticated event through the checkpoint-fork executor', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
});
|
||||
const client = {
|
||||
options: {},
|
||||
sendMessage: jest.fn(async () => {
|
||||
throw new Error('stop after event actor invocation started');
|
||||
}),
|
||||
};
|
||||
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => {
|
||||
await input.invoke({
|
||||
checkpointNamespace: 'event-actor/fork',
|
||||
checkpointId: 'checkpoint-base',
|
||||
invocationId: 'req-event-fork',
|
||||
continuation: 'warm',
|
||||
signal: input.signal,
|
||||
});
|
||||
});
|
||||
const event = {
|
||||
id: 'game-1:ply-8',
|
||||
type: 'chess.turn',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'speed-chess', type: 'mcp' },
|
||||
payload: { gameId: 'game-1', expectedPly: 8 },
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: '' },
|
||||
body: {
|
||||
text: 'Play the next move.',
|
||||
clientRequestId: 'req-event-fork',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-fork',
|
||||
event,
|
||||
expectedAction: { toolName: 'submit_move', argumentSubset: { expectedPly: 8 } },
|
||||
},
|
||||
},
|
||||
config: {
|
||||
endpoints: { agents: { eventDriven: { checkpointForks: true } } },
|
||||
},
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: undefined,
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
await nextTick();
|
||||
|
||||
expect(mockExecuteAgentEventActor).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
conversationId: 'child-conversation',
|
||||
invocationId: 'req-event-fork',
|
||||
event,
|
||||
expectedAction: { toolName: 'submit_move', argumentSubset: { expectedPly: 8 } },
|
||||
}),
|
||||
{
|
||||
getSnapshot: expect.any(Function),
|
||||
commitState: expect.any(Function),
|
||||
recordReconciliation: expect.any(Function),
|
||||
resolveReconciliation: expect.any(Function),
|
||||
},
|
||||
);
|
||||
expect(mockExecuteAgentEventActor.mock.calls[0][0]).not.toHaveProperty('tenantId');
|
||||
expect(client).toMatchObject({
|
||||
checkpointNamespace: 'event-actor/fork',
|
||||
eventActorCheckpointId: 'checkpoint-base',
|
||||
eventActorInvocationId: 'req-event-fork',
|
||||
eventActorContinuation: 'warm',
|
||||
});
|
||||
});
|
||||
|
||||
it('prefers the execution-time action receipt over lagging run steps', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
});
|
||||
mockFindAgentEventAppliedAction.mockReturnValue(undefined);
|
||||
let observedRead;
|
||||
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => {
|
||||
await input.invoke({
|
||||
checkpointNamespace: 'event-actor/fork',
|
||||
checkpointId: 'checkpoint-base',
|
||||
invocationId: 'req-event-receipt',
|
||||
continuation: 'warm',
|
||||
signal: input.signal,
|
||||
});
|
||||
observedRead = input.readAppliedAction();
|
||||
throw new Error('stop after evidence read');
|
||||
});
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: '' },
|
||||
body: {
|
||||
text: 'Play the next move.',
|
||||
clientRequestId: 'req-event-receipt',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-receipt',
|
||||
event: {
|
||||
id: 'game-1:ply-9',
|
||||
type: 'chess.turn',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'speed-chess', type: 'mcp' },
|
||||
},
|
||||
expectedAction: { toolName: 'submit_move', argumentSubset: { expectedPly: 9 } },
|
||||
},
|
||||
},
|
||||
config: {
|
||||
endpoints: { agents: { eventDriven: { checkpointForks: true } } },
|
||||
},
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: undefined,
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
/** Reproduces the observed race: the tool executed (graph context fires
|
||||
* the observer during sendMessage) but the run-step collection is still
|
||||
* empty when the executor reads evidence right after resolution. */
|
||||
const client = {
|
||||
options: {},
|
||||
sendMessage: jest.fn(async () => {
|
||||
req._agentEventActionObserver({
|
||||
input: { expectedPly: 9 },
|
||||
output: { name: 'submit_move', tool_call_id: 'call-9', content: '{"ok":true}' },
|
||||
});
|
||||
return {};
|
||||
}),
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
await nextTick();
|
||||
|
||||
expect(mockExecuteAgentEventActor).toHaveBeenCalledTimes(1);
|
||||
expect(mockCreateAgentEventActionRecorder).toHaveBeenCalledWith({
|
||||
toolName: 'submit_move',
|
||||
argumentSubset: { expectedPly: 9 },
|
||||
});
|
||||
expect(typeof req._agentEventActionObserver).toBe('function');
|
||||
expect(observedRead).toEqual({ toolName: 'submit_move', toolCallId: 'call-9' });
|
||||
expect(mockFindAgentEventAppliedAction).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('records reconciliation when persistence fails after an event action commits', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
const checkpoint = {
|
||||
threadId: 'child-conversation',
|
||||
checkpointId: 'checkpoint-applied',
|
||||
checkpointNs: 'event-actor/applied',
|
||||
};
|
||||
mockExecuteAgentEventActor.mockResolvedValueOnce({
|
||||
value: {
|
||||
messageId: 'req-event-persistence:assistant',
|
||||
databasePromise: Promise.resolve().then(() => {
|
||||
throw new Error('response persistence unavailable');
|
||||
}),
|
||||
},
|
||||
execution: {
|
||||
status: 'applied',
|
||||
continuation: 'warm',
|
||||
head: { actorThreadId: 'child-conversation', generation: 2, checkpoint },
|
||||
result: { action: { toolName: 'submit_move', toolCallId: 'call-move' } },
|
||||
},
|
||||
});
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Play the next move.',
|
||||
clientRequestId: 'req-event-persistence',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-persistence',
|
||||
event: {
|
||||
id: 'event-persistence',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client: { options: {} } }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockRecordAgentEventActorReconciliation.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
|
||||
expect(mockExecuteAgentEventActor).toHaveBeenCalledTimes(1);
|
||||
expect(mockLogger.info).toHaveBeenCalledWith(
|
||||
'[event-actor] Bound child event completed',
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(mockRecordAgentEventActorReconciliation).toHaveBeenCalledWith({
|
||||
user: 'user-123',
|
||||
tenantId: 'tenant-1',
|
||||
conversationId: 'child-conversation',
|
||||
reconciliation: expect.objectContaining({
|
||||
invocationId: 'req-event-persistence',
|
||||
status: 'persistence_failed',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-move' },
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
it('records the applied actor history barrier only after both deterministic messages persist', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
const checkpoint = {
|
||||
threadId: 'child-conversation',
|
||||
checkpointId: 'checkpoint-applied',
|
||||
checkpointNs: 'event-actor/applied',
|
||||
};
|
||||
const userMessage = {
|
||||
messageId: 'req-event-success:user',
|
||||
parentMessageId: 'parent-message',
|
||||
conversationId: 'child-conversation',
|
||||
text: 'Play the next move.',
|
||||
};
|
||||
const client = {
|
||||
options: {},
|
||||
skipSaveUserMessage: false,
|
||||
sendMessage: jest.fn(async (_text, options) => {
|
||||
options.onStart(userMessage, 'req-event-success:assistant');
|
||||
return {
|
||||
messageId: 'req-event-success:assistant',
|
||||
text: 'Move submitted.',
|
||||
databasePromise: Promise.resolve({
|
||||
conversation: { conversationId: 'child-conversation' },
|
||||
}),
|
||||
};
|
||||
}),
|
||||
};
|
||||
mockExecuteAgentEventActor.mockImplementationOnce(async (input) => ({
|
||||
value: await input.invoke({
|
||||
checkpointNamespace: checkpoint.checkpointNs,
|
||||
checkpointId: checkpoint.checkpointId,
|
||||
invocationId: 'req-event-success',
|
||||
continuation: 'warm',
|
||||
signal: input.signal,
|
||||
}),
|
||||
execution: {
|
||||
status: 'applied',
|
||||
continuation: 'warm',
|
||||
head: { actorThreadId: 'child-conversation', generation: 2, checkpoint },
|
||||
result: { action: { toolName: 'submit_move', toolCallId: 'call-move' } },
|
||||
},
|
||||
}));
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: userMessage.text,
|
||||
clientRequestId: 'req-event-success',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-success',
|
||||
event: {
|
||||
id: 'event-success',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockRecordAgentEventActorReconciliation.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
|
||||
expect(mockSaveMessage).toHaveBeenCalledWith(
|
||||
expect.any(Object),
|
||||
expect.objectContaining({ messageId: 'req-event-success:user' }),
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(mockSaveMessage).toHaveBeenCalledWith(
|
||||
expect.any(Object),
|
||||
expect.objectContaining({ messageId: 'req-event-success:assistant' }),
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(mockRecordAgentEventActorReconciliation).toHaveBeenCalledWith({
|
||||
user: 'user-123',
|
||||
tenantId: 'tenant-1',
|
||||
conversationId: 'child-conversation',
|
||||
reconciliation: {
|
||||
invocationId: 'req-event-success',
|
||||
status: 'history_persisted',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-move' },
|
||||
observedAt: expect.any(Date),
|
||||
},
|
||||
});
|
||||
const lastMessageWrite = Math.max(...mockSaveMessage.mock.invocationCallOrder);
|
||||
expect(lastMessageWrite).toBeLessThan(
|
||||
mockRecordAgentEventActorReconciliation.mock.invocationCallOrder[0],
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
'tool approval',
|
||||
{ toolDefinitions: [] },
|
||||
{ toolApproval: { enabled: true }, eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'primary ask_user_question',
|
||||
{ toolDefinitions: [{ name: 'ask_user_question' }] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'added-agent ask_user_question',
|
||||
{ toolDefinitions: [] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
new Map([['added-agent', { toolDefinitions: [{ name: 'ask_user_question' }] }]]),
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'memory-checkpointer',
|
||||
{ toolDefinitions: [] },
|
||||
{ eventDriven: { checkpointForks: true }, checkpointer: { type: 'memory' } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'always-apply skill prime',
|
||||
{ toolDefinitions: [], alwaysApplySkillPrimes: [{ name: 'playbook', body: '# playbook' }] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'manual skill prime',
|
||||
{ toolDefinitions: [], manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
[
|
||||
'skills-capable (history-derived re-priming)',
|
||||
{ toolDefinitions: [] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
{ primeInvokedSkills: async () => undefined },
|
||||
],
|
||||
[
|
||||
'background-capable expected action',
|
||||
{ toolDefinitions: [], backgroundToolNames: ['submit_move_mcp_chess'] },
|
||||
{ eventDriven: { checkpointForks: true } },
|
||||
undefined,
|
||||
undefined,
|
||||
],
|
||||
])(
|
||||
'keeps %s event actors on the existing resumable path',
|
||||
async (_label, agent, config, agentConfigs, clientOptions) => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({
|
||||
streamId: 'child-conversation',
|
||||
conversationId: 'child-conversation',
|
||||
}),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
const client = {
|
||||
options: { agent, ...(clientOptions ?? {}) },
|
||||
agentConfigs,
|
||||
sendMessage: jest.fn(async () => {
|
||||
throw new Error('stop after legacy event invocation started');
|
||||
}),
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Continue with a pause-capable actor.',
|
||||
clientRequestId: 'req-event-hitl',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-hitl',
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
event: {
|
||||
id: 'event-hitl',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: config } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
await nextTick();
|
||||
|
||||
expect(mockExecuteAgentEventActor).not.toHaveBeenCalled();
|
||||
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
||||
user: 'user-123',
|
||||
tenantId: 'tenant-1',
|
||||
conversationId: 'child-conversation',
|
||||
token: expect.any(String),
|
||||
});
|
||||
expect(client.sendMessage).toHaveBeenCalledTimes(1);
|
||||
/** The fence must open before execution and close only after this
|
||||
* turn's history is durable, under the same token. */
|
||||
const fenceToken = mockBeginAgentEventActorLegacyTurn.mock.calls[0][0].token;
|
||||
expect(mockBeginAgentEventActorLegacyTurn.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
client.sendMessage.mock.invocationCallOrder[0],
|
||||
);
|
||||
expect(mockGenerationJobManager.updateMetadata).toHaveBeenCalledWith(
|
||||
'child-conversation',
|
||||
{ agentEventLegacyTurnToken: fenceToken },
|
||||
1000,
|
||||
);
|
||||
expect(mockGenerationJobManager.updateMetadata.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
client.sendMessage.mock.invocationCallOrder[0],
|
||||
);
|
||||
expect(mockCompleteAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
||||
user: 'user-123',
|
||||
tenantId: 'tenant-1',
|
||||
conversationId: 'child-conversation',
|
||||
token: fenceToken,
|
||||
});
|
||||
expect(mockCompleteAgentEventActorLegacyTurn.mock.invocationCallOrder[0]).toBeGreaterThan(
|
||||
client.sendMessage.mock.invocationCallOrder[0],
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it('retains local fence ownership when Redis metadata persistence fails', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
mockGenerationJobManager.updateMetadata.mockRejectedValueOnce(
|
||||
new Error('redis metadata unavailable'),
|
||||
);
|
||||
const client = {
|
||||
options: {
|
||||
agent: {
|
||||
toolDefinitions: [],
|
||||
manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }],
|
||||
},
|
||||
},
|
||||
sendMessage: jest.fn(),
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Persist ownership before Redis metadata.',
|
||||
clientRequestId: 'req-event-metadata-failure',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-metadata-failure',
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
event: {
|
||||
id: 'event-metadata-failure',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockCompleteAgentEventActorLegacyTurn.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
|
||||
const fenceToken = mockBeginAgentEventActorLegacyTurn.mock.calls[0][0].token;
|
||||
expect(client.sendMessage).not.toHaveBeenCalled();
|
||||
expect(mockCompleteAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
||||
user: 'user-123',
|
||||
tenantId: 'tenant-1',
|
||||
conversationId: 'child-conversation',
|
||||
token: fenceToken,
|
||||
});
|
||||
});
|
||||
|
||||
it('does not execute a legacy event turn when its durable fence is not acquired', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
mockBeginAgentEventActorLegacyTurn.mockResolvedValueOnce(false);
|
||||
const client = {
|
||||
options: {
|
||||
agent: {
|
||||
toolDefinitions: [],
|
||||
manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }],
|
||||
},
|
||||
},
|
||||
sendMessage: jest.fn(async () => ({ messageId: 'must-not-run' })),
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Do not run without the fence.',
|
||||
clientRequestId: 'req-event-fence-lost',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-fence-lost',
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
event: {
|
||||
id: 'event-fence-lost',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockBeginAgentEventActorLegacyTurn.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
|
||||
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
||||
expect(client.sendMessage).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('keeps the legacy event fence open when failed-turn persistence cannot complete', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
mockGenerationJobManager.completeJob.mockRejectedValueOnce(new Error('job store unavailable'));
|
||||
const client = {
|
||||
options: {
|
||||
agent: {
|
||||
toolDefinitions: [],
|
||||
manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }],
|
||||
},
|
||||
},
|
||||
sendMessage: jest.fn(async () => {
|
||||
throw new Error('provider failed');
|
||||
}),
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Persist the failed turn before releasing the fence.',
|
||||
clientRequestId: 'req-event-error-persistence',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-error-persistence',
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
event: {
|
||||
id: 'event-error-persistence',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockGenerationJobManager.completeJob.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
await nextTick();
|
||||
|
||||
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
||||
expect(mockGenerationJobManager.completeJob).toHaveBeenCalledTimes(1);
|
||||
expect(mockCompleteAgentEventActorLegacyTurn).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not replay a stale ambiguous legacy fence based only on age', async () => {
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
wonGenerationClaim({ streamId: 'child-conversation', conversationId: 'child-conversation' }),
|
||||
);
|
||||
mockGetConvo.mockResolvedValue({
|
||||
conversationId: 'parent-conversation',
|
||||
agent_id: 'parent-agent',
|
||||
tenantId: 'tenant-1',
|
||||
});
|
||||
mockBeginAgentEventActorLegacyTurn.mockResolvedValueOnce(false);
|
||||
mockGetAgentEventActorSnapshot.mockResolvedValueOnce({
|
||||
state: null,
|
||||
reconciliations: [],
|
||||
epoch: 0,
|
||||
legacyTurn: {
|
||||
token: 'crashed-legacy-turn',
|
||||
startedAt: new Date(Date.now() - 2 * 60 * 60 * 1000),
|
||||
},
|
||||
});
|
||||
const client = {
|
||||
options: {
|
||||
agent: {
|
||||
toolDefinitions: [],
|
||||
manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }],
|
||||
},
|
||||
},
|
||||
sendMessage: jest.fn(async () => {
|
||||
throw new Error('stop after recovered execution starts');
|
||||
}),
|
||||
};
|
||||
const req = {
|
||||
user: { id: 'user-123', tenantId: 'tenant-1' },
|
||||
body: {
|
||||
text: 'Recover the abandoned legacy fence.',
|
||||
clientRequestId: 'req-event-stale-legacy',
|
||||
conversationId: 'child-conversation',
|
||||
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
|
||||
agentEventDelivery: {
|
||||
deliveryKey: 'req-event-stale-legacy',
|
||||
expectedAction: { toolName: 'submit_move' },
|
||||
event: {
|
||||
id: 'event-stale-legacy',
|
||||
type: 'test.event',
|
||||
occurredAt: Date.now(),
|
||||
source: { id: 'test', type: 'api' },
|
||||
payload: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
|
||||
_isAgentTrigger: true,
|
||||
_agentEventBindingParentConversationId: 'parent-conversation',
|
||||
_agentEventBindingParentAgentId: 'parent-agent',
|
||||
_agentEventBindingTenantId: 'tenant-1',
|
||||
_agentEventBindingRetention: { expiredAt: new Date(Date.now() + 60_000) },
|
||||
};
|
||||
|
||||
await AgentController(
|
||||
req,
|
||||
createResumableResponse(),
|
||||
jest.fn(),
|
||||
jest.fn().mockResolvedValue({ client }),
|
||||
null,
|
||||
);
|
||||
for (
|
||||
let attempt = 0;
|
||||
attempt < 20 && mockBeginAgentEventActorLegacyTurn.mock.calls.length === 0;
|
||||
attempt++
|
||||
) {
|
||||
await nextTick();
|
||||
}
|
||||
|
||||
expect(mockBeginAgentEventActorLegacyTurn).toHaveBeenCalledTimes(1);
|
||||
expect(client.sendMessage).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('uses the guard-normalized tenant for a legacy untenanted event actor', async () => {
|
||||
const expiredAt = new Date(Date.now() + 60_000);
|
||||
mockGenerationJobManager.claimGeneration.mockResolvedValue(
|
||||
|
|
|
|||
|
|
@ -102,6 +102,7 @@ const mockReleaseScheduleResumeFence = jest.fn();
|
|||
const mockAcquireEventChildGenerationLease = jest.fn();
|
||||
const mockReleaseEventChildLease = jest.fn();
|
||||
const mockIsSubagentOwnerAdmissible = jest.fn();
|
||||
const mockCompleteAgentEventActorLegacyTurn = jest.fn();
|
||||
|
||||
jest.mock('@librechat/data-schemas', () => ({
|
||||
...jest.requireActual('@librechat/data-schemas'),
|
||||
|
|
@ -135,6 +136,7 @@ jest.mock('~/models', () => ({
|
|||
getUserMemories: (...args) => mockGetUserMemories(...args),
|
||||
getRoleByName: (...args) => mockGetRoleByName(...args),
|
||||
isSubagentOwnerAdmissible: (...args) => mockIsSubagentOwnerAdmissible(...args),
|
||||
completeAgentEventActorLegacyTurn: (...args) => mockCompleteAgentEventActorLegacyTurn(...args),
|
||||
}));
|
||||
|
||||
jest.mock('~/server/services/Endpoints/agents/eventChildLease', () => ({
|
||||
|
|
@ -351,6 +353,7 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {
|
|||
mockAcquireEventChildGenerationLease.mockResolvedValue(mockReleaseEventChildLease);
|
||||
mockReleaseEventChildLease.mockResolvedValue(undefined);
|
||||
mockIsSubagentOwnerAdmissible.mockResolvedValue(true);
|
||||
mockCompleteAgentEventActorLegacyTurn.mockResolvedValue(true);
|
||||
endpointAgent = {
|
||||
_id: 'mongo-agent-abc',
|
||||
id: AGENT_ID,
|
||||
|
|
@ -2089,6 +2092,27 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {
|
|||
});
|
||||
|
||||
describe('happy path: approve -> reconstruct -> resume -> finalize', () => {
|
||||
it('seals the exact paused legacy-event fence only after resumed history persists', async () => {
|
||||
mockGenerationJobManager.getJob.mockResolvedValue(
|
||||
makeToolApprovalJob({ metadata: { agentEventLegacyTurnToken: 'legacy-hitl-token' } }),
|
||||
);
|
||||
|
||||
const res = await post(approveBody());
|
||||
expect(res.status).toBe(200);
|
||||
await settled;
|
||||
await flush();
|
||||
|
||||
expect(mockCompleteAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
||||
user: USER_ID,
|
||||
tenantId: TENANT_ID,
|
||||
conversationId: CONVO_ID,
|
||||
token: 'legacy-hitl-token',
|
||||
});
|
||||
expect(mockSaveMessage.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
mockCompleteAgentEventActorLegacyTurn.mock.invocationCallOrder[0],
|
||||
);
|
||||
});
|
||||
|
||||
it('ACKs immediately and claims the action atomically with the submitted actionId', async () => {
|
||||
mockGenerationJobManager.getJob.mockResolvedValue(makeToolApprovalJob());
|
||||
const res = await post(approveBody());
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ const {
|
|||
isContentFilterError,
|
||||
deleteAgentCheckpoint,
|
||||
LIBRECHAT_CHECKPOINT_NAMESPACE_KEY,
|
||||
LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY,
|
||||
agentRequestsAskUserQuestion,
|
||||
attachAskUserQuestionArgs,
|
||||
hydrateResumeRunSteps,
|
||||
|
|
@ -217,6 +218,11 @@ class AgentClient extends BaseClient {
|
|||
/** Generation-scoped LangGraph checkpoint namespace. Legacy paused jobs
|
||||
* intentionally use the historical empty namespace. @type {string} */
|
||||
this.checkpointNamespace = options.checkpointNamespace ?? '';
|
||||
/** Bound-event invocation state is assigned immediately before sendMessage,
|
||||
* after the SDK has prepared its isolated fork. */
|
||||
this.eventActorCheckpointId = undefined;
|
||||
this.eventActorInvocationId = undefined;
|
||||
this.eventActorContinuation = undefined;
|
||||
|
||||
/** @type {AgentRun} */
|
||||
this.run;
|
||||
|
|
@ -3297,6 +3303,16 @@ class AgentClient extends BaseClient {
|
|||
// into its physical namespace while tools keep the conversation id.
|
||||
checkpoint_ns: '',
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: this.checkpointNamespace,
|
||||
...(this.eventActorCheckpointId == null
|
||||
? {}
|
||||
: { checkpoint_id: this.eventActorCheckpointId }),
|
||||
...(this.eventActorInvocationId == null
|
||||
? {}
|
||||
: {
|
||||
[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY]: this.eventActorInvocationId,
|
||||
event_actor_invocation_id: this.eventActorInvocationId,
|
||||
event_actor_depth: 1,
|
||||
}),
|
||||
last_agent_index: this.agentConfigs?.size ?? 0,
|
||||
user_id: this.user ?? this.options.req.user?.id,
|
||||
hide_sequential_outputs: this.options.agent.hide_sequential_outputs,
|
||||
|
|
@ -3547,6 +3563,7 @@ class AgentClient extends BaseClient {
|
|||
// below still guarantees it completes before the graph is exposed or run.
|
||||
const shouldPruneCheckpoint =
|
||||
streamId &&
|
||||
this.eventActorInvocationId == null &&
|
||||
(isHITLEnabled(agentsEConfig?.toolApproval) || agents.some(agentRequestsAskUserQuestion));
|
||||
let checkpointPrunePromise = Promise.resolve();
|
||||
if (shouldPruneCheckpoint && this.checkpointNamespace !== '') {
|
||||
|
|
@ -3617,7 +3634,16 @@ class AgentClient extends BaseClient {
|
|||
steering: this.buildSteerWiring(streamId),
|
||||
activityLabel,
|
||||
activityPhase,
|
||||
indexTokenCountMap,
|
||||
eventActorCheckpointing: this.eventActorInvocationId != null,
|
||||
// The token map is positional over the DB-derived history. A warm
|
||||
// continuation runs on checkpoint-restored state (restored messages
|
||||
// plus the one new event), so those indices address different
|
||||
// messages and the pruner would never recount them. Hand it an empty
|
||||
// map so every count is derived from the messages actually in state.
|
||||
// `initialSummary` deliberately stays: it rides the system tail, and
|
||||
// the pre-boundary turns it summarizes were excluded from the very
|
||||
// history the committed checkpoint was built from.
|
||||
indexTokenCountMap: this.eventActorContinuation === 'warm' ? {} : indexTokenCountMap,
|
||||
initialSummary,
|
||||
initialSessions,
|
||||
calibrationRatio,
|
||||
|
|
@ -3690,7 +3716,9 @@ class AgentClient extends BaseClient {
|
|||
await this.activityLabelsMarkedPromise;
|
||||
}
|
||||
try {
|
||||
await run.processStream({ messages }, config, {
|
||||
const invocationMessages =
|
||||
this.eventActorContinuation === 'warm' ? messages.slice(-1) : messages;
|
||||
await run.processStream({ messages: invocationMessages }, config, {
|
||||
callbacks: {
|
||||
[Callback.TOOL_ERROR]: logToolError,
|
||||
},
|
||||
|
|
|
|||
|
|
@ -992,6 +992,84 @@ describe('AgentClient - startup telemetry', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('rehydrates a warm event actor from its fork and injects only the new event message', async () => {
|
||||
jest.clearAllMocks();
|
||||
const history = { _getType: () => 'human', content: 'old turn' };
|
||||
const currentEvent = { _getType: () => 'human', content: 'new event' };
|
||||
mockFormatAgentMessages.mockReturnValueOnce({
|
||||
messages: [history, currentEvent],
|
||||
indexTokenCountMap: { 0: 11, 1: 22 },
|
||||
summary: { text: 'summary of earlier turns', tokenCount: 40 },
|
||||
boundaryTokenAdjustment: undefined,
|
||||
});
|
||||
const processStream = jest.fn().mockResolvedValue();
|
||||
mockCreateRun.mockResolvedValue({
|
||||
Graph: null,
|
||||
processStream,
|
||||
getCalibrationRatio: jest.fn(() => 0),
|
||||
});
|
||||
const client = new AgentClient({
|
||||
req: {
|
||||
user: { id: 'user-123' },
|
||||
body: {},
|
||||
config: { endpoints: { [EModelEndpoint.agents]: {} } },
|
||||
_resumableStreamId: 'conversation-123',
|
||||
},
|
||||
res: {},
|
||||
agent: {
|
||||
id: 'agent-123',
|
||||
endpoint: EModelEndpoint.openAI,
|
||||
provider: EModelEndpoint.openAI,
|
||||
model_parameters: { model: 'gpt-4' },
|
||||
hide_sequential_outputs: false,
|
||||
},
|
||||
endpointTokenConfig: {},
|
||||
eventHandlers: {},
|
||||
contentParts: [],
|
||||
collectedUsage: [],
|
||||
artifactPromises: [],
|
||||
});
|
||||
client.conversationId = 'conversation-123';
|
||||
client.responseMessageId = 'response-123';
|
||||
client.parentMessageId = 'parent-123';
|
||||
client.checkpointNamespace = 'event-actor/fork';
|
||||
client.eventActorCheckpointId = 'checkpoint-base';
|
||||
client.eventActorInvocationId = 'event-2';
|
||||
client.eventActorContinuation = 'warm';
|
||||
client.recordCollectedUsage = jest.fn().mockResolvedValue();
|
||||
|
||||
await client.chatCompletion({ payload: [] });
|
||||
|
||||
expect(mockCreateRun).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
messages: [history, currentEvent],
|
||||
eventActorCheckpointing: true,
|
||||
/** The DB-derived token map is positional over full history, which a
|
||||
* checkpoint-restored graph state no longer matches. It must be blank
|
||||
* so the pruner recounts against the messages actually in state. The
|
||||
* cross-run summary stays: it summarizes pre-boundary turns that were
|
||||
* excluded from the history the committed checkpoint was built from. */
|
||||
indexTokenCountMap: {},
|
||||
initialSummary: { text: 'summary of earlier turns', tokenCount: 40 },
|
||||
}),
|
||||
);
|
||||
expect(processStream).toHaveBeenCalledWith(
|
||||
{ messages: [currentEvent] },
|
||||
expect.objectContaining({
|
||||
configurable: expect.objectContaining({
|
||||
thread_id: 'conversation-123',
|
||||
checkpoint_id: 'checkpoint-base',
|
||||
__librechat_checkpoint_ns: 'event-actor/fork',
|
||||
__librechat_event_actor_invocation_id: 'event-2',
|
||||
event_actor_invocation_id: 'event-2',
|
||||
event_actor_depth: 1,
|
||||
}),
|
||||
}),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(mockDeleteAgentCheckpoint).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not expose or process a fresh graph when strict checkpoint pruning fails', async () => {
|
||||
jest.clearAllMocks();
|
||||
const checkpointGeneration = {
|
||||
|
|
|
|||
|
|
@ -30,6 +30,11 @@ const {
|
|||
getAttachmentTitleText,
|
||||
createMCPRuntimeRequestBody,
|
||||
isAgentEventRetentionActive,
|
||||
executeAgentEventActor,
|
||||
findAgentEventAppliedAction,
|
||||
createAgentEventActionRecorder,
|
||||
isHITLEnabled,
|
||||
agentRequestsAskUserQuestion,
|
||||
} = require('@librechat/api');
|
||||
const { disposeClient } = require('~/server/cleanup');
|
||||
const {
|
||||
|
|
@ -43,6 +48,12 @@ const {
|
|||
saveConvo,
|
||||
getMessages,
|
||||
getConvo,
|
||||
getAgentEventActorSnapshot,
|
||||
commitAgentEventActorState,
|
||||
beginAgentEventActorLegacyTurn,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
recordAgentEventActorReconciliation,
|
||||
resolveAgentEventActorReconciliation,
|
||||
isAgentTriggerPrincipalActive,
|
||||
isSubagentOwnerAdmissible,
|
||||
} = require('~/models');
|
||||
|
|
@ -1782,6 +1793,60 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
});
|
||||
return titleEventPromise;
|
||||
};
|
||||
const eventActorTenantId = req._agentEventBindingTenantId;
|
||||
let appliedEventActor;
|
||||
let eventActorPersistenceComplete = false;
|
||||
let legacyEventActorTurnToken;
|
||||
/** Closes the durable legacy-turn fence once this turn's history is
|
||||
* persisted, clearing the token and advancing the epoch in one write. A
|
||||
* failure here leaves the token SET, which keeps blocking forks until an
|
||||
* explicit reconciliation proves the ambiguous turn safe to release. */
|
||||
const sealLegacyEventActorTurn = async () => {
|
||||
if (legacyEventActorTurnToken == null) {
|
||||
return;
|
||||
}
|
||||
const token = legacyEventActorTurnToken;
|
||||
legacyEventActorTurnToken = undefined;
|
||||
try {
|
||||
const sealed = await completeAgentEventActorLegacyTurn({
|
||||
user: userId,
|
||||
conversationId,
|
||||
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
|
||||
token,
|
||||
});
|
||||
if (!sealed) {
|
||||
logger.error(
|
||||
`[event-actor] Legacy turn fence ${token} was not sealed; forks stay blocked pending reconciliation`,
|
||||
);
|
||||
}
|
||||
} catch (sealError) {
|
||||
logger.error(
|
||||
`[event-actor] Failed to seal legacy turn fence ${token}; forks stay blocked pending reconciliation`,
|
||||
sealError,
|
||||
);
|
||||
}
|
||||
};
|
||||
const recordEventActorPersistenceFailure = async (error) => {
|
||||
if (appliedEventActor == null || eventActorPersistenceComplete) {
|
||||
return;
|
||||
}
|
||||
const recorded = await recordAgentEventActorReconciliation({
|
||||
user: userId,
|
||||
conversationId,
|
||||
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
|
||||
reconciliation: {
|
||||
invocationId: appliedEventActor.invocationId,
|
||||
status: 'persistence_failed',
|
||||
checkpoint: appliedEventActor.checkpoint,
|
||||
action: appliedEventActor.action,
|
||||
error: String(error?.message ?? error).slice(0, 1024),
|
||||
observedAt: new Date(),
|
||||
},
|
||||
});
|
||||
if (!recorded) {
|
||||
throw new Error('Failed to preserve applied event actor persistence reconciliation');
|
||||
}
|
||||
};
|
||||
|
||||
try {
|
||||
const onStart = (userMsg, respMsgId, _isNewConvo) => {
|
||||
|
|
@ -1878,7 +1943,152 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
},
|
||||
};
|
||||
|
||||
const sendPromise = client.sendMessage(text, messageOptions);
|
||||
const checkpointForksEnabled =
|
||||
req.config?.endpoints?.[EModelEndpoint.agents]?.eventDriven?.checkpointForks === true;
|
||||
const agentsConfig = req.config?.endpoints?.[EModelEndpoint.agents];
|
||||
const eventActorAgents = [
|
||||
client?.options?.agent,
|
||||
...(client?.agentConfigs?.values?.() ?? []),
|
||||
].filter(Boolean);
|
||||
const eventActorMayPause =
|
||||
isHITLEnabled(agentsConfig?.toolApproval) ||
|
||||
eventActorAgents.some(agentRequestsAskUserQuestion);
|
||||
/** Skill bodies are spliced or reconstructed into the message list
|
||||
* ahead of the newest message, and a warm continuation forwards only
|
||||
* that newest message — a checkpoint-restored actor would keep serving
|
||||
* the bodies baked in at its last cold start and never observe an
|
||||
* edited or newly attached skill. That covers request-time primes AND
|
||||
* history-derived re-priming: `primeInvokedSkills` re-resolves any
|
||||
* skill the actor previously invoked, so its mere presence (the skills
|
||||
* capability) keeps the actor on the legacy path, which rebuilds fresh
|
||||
* bodies every turn. #15235 tracks the context fingerprint that will
|
||||
* restore warm continuation for skill-bearing actors. */
|
||||
const eventActorHasSkillPrimes =
|
||||
client?.options?.primeInvokedSkills != null ||
|
||||
(client?.options?.agent?.alwaysApplySkillPrimes?.length ?? 0) > 0 ||
|
||||
(client?.options?.agent?.manualSkillPrimes?.length ?? 0) > 0;
|
||||
/** A background-opted expected action can detach: dispatch returns a
|
||||
* launch handle the evidence fences correctly reject, and the later
|
||||
* completion is provenance-marked as another turn's work — so a fork
|
||||
* would classify actionless and settle before the external effect
|
||||
* lands, with no receipt to stop a retry from dispatching it again.
|
||||
* The name comparison mirrors matchesExpectedAction's MCP suffix. */
|
||||
const expectedActionToolName = agentEventDelivery?.expectedAction?.toolName;
|
||||
const eventActorActionMayDetach =
|
||||
typeof expectedActionToolName === 'string' &&
|
||||
eventActorAgents.some((agent) =>
|
||||
(agent.backgroundToolNames ?? []).some(
|
||||
(name) =>
|
||||
name === expectedActionToolName ||
|
||||
name.startsWith(`${expectedActionToolName}_mcp_`),
|
||||
),
|
||||
);
|
||||
const canUseEventActorFork =
|
||||
checkpointForksEnabled &&
|
||||
agentsConfig?.checkpointer?.type !== 'memory' &&
|
||||
!eventActorMayPause &&
|
||||
!eventActorHasSkillPrimes &&
|
||||
!eventActorActionMayDetach &&
|
||||
agentEventDelivery?.event != null &&
|
||||
agentEventDelivery.expectedAction != null &&
|
||||
typeof eventTaskId === 'string' &&
|
||||
req._agentEventBindingParentConversationId != null;
|
||||
/** Authoritative action proof is captured in graph context the moment
|
||||
* the expected tool executes (see the observer tee in initialize.js);
|
||||
* run-step inspection stays only as a fallback, because the run-step
|
||||
* collection is populated asynchronously and can still be empty the
|
||||
* instant sendMessage resolves — misreading an applied invocation as
|
||||
* actionless would discard its fork and strand the actor cold. */
|
||||
const eventActorActionRecorder = canUseEventActorFork
|
||||
? createAgentEventActionRecorder(agentEventDelivery.expectedAction)
|
||||
: undefined;
|
||||
if (eventActorActionRecorder != null) {
|
||||
req._agentEventActionObserver = eventActorActionRecorder.observeToolEnd;
|
||||
}
|
||||
const sendPromise = canUseEventActorFork
|
||||
? executeAgentEventActor(
|
||||
{
|
||||
user: userId,
|
||||
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
|
||||
conversationId,
|
||||
invocationId: eventTaskId,
|
||||
event: agentEventDelivery.event,
|
||||
expectedAction: agentEventDelivery.expectedAction,
|
||||
signal: job.abortController.signal,
|
||||
checkpointer: req.config?.endpoints?.[EModelEndpoint.agents]?.checkpointer,
|
||||
invoke: async (actorContext) => {
|
||||
client.checkpointNamespace = actorContext.checkpointNamespace;
|
||||
client.eventActorCheckpointId = actorContext.checkpointId;
|
||||
client.eventActorInvocationId = actorContext.invocationId;
|
||||
client.eventActorContinuation = actorContext.continuation;
|
||||
return client.sendMessage(text, messageOptions);
|
||||
},
|
||||
readAppliedAction: () =>
|
||||
eventActorActionRecorder.read() ??
|
||||
findAgentEventAppliedAction(
|
||||
agentEventDelivery.expectedAction,
|
||||
client?.run?.getRunSteps?.() ?? [],
|
||||
client?.contentParts ?? [],
|
||||
),
|
||||
},
|
||||
{
|
||||
getSnapshot: getAgentEventActorSnapshot,
|
||||
commitState: commitAgentEventActorState,
|
||||
recordReconciliation: recordAgentEventActorReconciliation,
|
||||
resolveReconciliation: resolveAgentEventActorReconciliation,
|
||||
},
|
||||
).then(({ value, execution }) => {
|
||||
if (execution.status === 'applied') {
|
||||
appliedEventActor = {
|
||||
invocationId: eventTaskId,
|
||||
checkpoint: execution.head.checkpoint,
|
||||
action: execution.result.action,
|
||||
};
|
||||
}
|
||||
logger.info('[event-actor] Bound child event completed', {
|
||||
conversationId,
|
||||
invocationId: eventTaskId,
|
||||
status: execution.status,
|
||||
continuation: execution.continuation,
|
||||
});
|
||||
return value;
|
||||
})
|
||||
: (async () => {
|
||||
if (
|
||||
agentEventDelivery?.event != null &&
|
||||
req._agentEventBindingParentConversationId != null
|
||||
) {
|
||||
const token = crypto.randomUUID();
|
||||
/** Open the fence BEFORE execution so a fork can neither run
|
||||
* nor commit while this turn's messages are not yet durable. */
|
||||
const legacyTurnOwner = {
|
||||
user: userId,
|
||||
conversationId,
|
||||
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
|
||||
};
|
||||
const acquiredLegacyTurn = await beginAgentEventActorLegacyTurn({
|
||||
...legacyTurnOwner,
|
||||
token,
|
||||
});
|
||||
if (!acquiredLegacyTurn) {
|
||||
throw Object.assign(new Error('The event actor is temporarily unavailable'), {
|
||||
code: 'EVENT_ACTOR_NOT_READY',
|
||||
status: 409,
|
||||
});
|
||||
}
|
||||
/** The same exact token must survive a HITL pause. Retain local
|
||||
* ownership before the Redis write so a metadata failure can
|
||||
* still seal this token after its durable error turn is saved;
|
||||
* provider execution starts only after metadata persists. */
|
||||
legacyEventActorTurnToken = token;
|
||||
await GenerationJobManager.updateMetadata(
|
||||
streamId,
|
||||
{ agentEventLegacyTurnToken: token },
|
||||
jobCreatedAt,
|
||||
);
|
||||
}
|
||||
return client.sendMessage(text, messageOptions);
|
||||
})();
|
||||
|
||||
if (titleEligible && titleTiming === 'immediate') {
|
||||
immediateTitlePromise = addTitle(req, {
|
||||
|
|
@ -2112,6 +2322,24 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
job.abortController.signal.removeEventListener('abort', abortTitleOnJobAbort);
|
||||
acceptsTitleEvents = false;
|
||||
resolveConvoReady();
|
||||
try {
|
||||
await recordEventActorPersistenceFailure(
|
||||
new Error('Event actor terminal persistence claim was replaced'),
|
||||
);
|
||||
} catch (reconciliationError) {
|
||||
/** The committing CAS already left a non-settled row that blocks
|
||||
* later actor turns, so a failed status upgrade costs provenance,
|
||||
* not safety. Never divert this clean exit past its cleanup. */
|
||||
logger.error(
|
||||
'[event-actor] Failed to preserve replaced-claim reconciliation',
|
||||
reconciliationError,
|
||||
);
|
||||
}
|
||||
/** This controller lost terminal persistence ownership, so it cannot
|
||||
* prove the winning Stop/replacement has written the unfinished
|
||||
* response yet. Keep the conversation fence closed; a HITL resume
|
||||
* carries the exact token, while every other orphan is handled by
|
||||
* bounded stale reclaim. */
|
||||
await finishResumableRequest(req, userId);
|
||||
disposeBackgroundClient();
|
||||
startupTelemetry?.end(job.abortController.signal.aborted ? 'aborted' : 'replaced');
|
||||
|
|
@ -2185,6 +2413,25 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
: 'Response message could not be persisted before terminal publication',
|
||||
);
|
||||
}
|
||||
if (appliedEventActor != null) {
|
||||
const recorded = await recordAgentEventActorReconciliation({
|
||||
user: userId,
|
||||
conversationId,
|
||||
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
|
||||
reconciliation: {
|
||||
invocationId: appliedEventActor.invocationId,
|
||||
status: 'history_persisted',
|
||||
checkpoint: appliedEventActor.checkpoint,
|
||||
action: appliedEventActor.action,
|
||||
observedAt: new Date(),
|
||||
},
|
||||
});
|
||||
if (!recorded) {
|
||||
throw new Error('Applied event actor history barrier could not be durably recorded');
|
||||
}
|
||||
}
|
||||
await sealLegacyEventActorTurn();
|
||||
eventActorPersistenceComplete = true;
|
||||
|
||||
// If the user stopped this turn — or an empty preempt boundary truncated
|
||||
// it, which persists under the same honest `unfinished` contract — cancel
|
||||
|
|
@ -2323,12 +2570,21 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
job.abortController.signal.removeEventListener('abort', abortTitleOnJobAbort);
|
||||
acceptsTitleEvents = false;
|
||||
resolveConvoReady();
|
||||
try {
|
||||
await recordEventActorPersistenceFailure(error);
|
||||
} catch (reconciliationError) {
|
||||
logger.error(
|
||||
'[event-actor] Failed to preserve terminal persistence reconciliation',
|
||||
reconciliationError,
|
||||
);
|
||||
}
|
||||
|
||||
// Once this controller owns terminal persistence, no competing error
|
||||
// transition can win. Settle its pending marker with conservative
|
||||
// reconciliation on any required-write/final-construction failure,
|
||||
// then release exactly that claim.
|
||||
let ownsScheduledFailure = false;
|
||||
let legacyEventActorErrorHistoryDurable = false;
|
||||
if (terminalClaim && !terminalClaimFinished) {
|
||||
ownsScheduledFailure = true;
|
||||
try {
|
||||
|
|
@ -2385,6 +2641,11 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
sender: client?.sender,
|
||||
}),
|
||||
})) === true;
|
||||
/** A true completion means this owner won the terminal CAS and
|
||||
* the beforeErrorPublication barrier above finished. Only that
|
||||
* combination proves the failed-turn rows are durable enough to
|
||||
* let a checkpoint fork rebuild past this legacy turn. */
|
||||
legacyEventActorErrorHistoryDurable = ownsScheduledFailure;
|
||||
} catch (completeErr) {
|
||||
logger.warn(
|
||||
'[ResumableAgentController] completeJob failed during generation-error cleanup',
|
||||
|
|
@ -2395,6 +2656,13 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
|
|||
}
|
||||
}
|
||||
|
||||
/** Leave the fence set when terminal persistence loses ownership or
|
||||
* fails. Time cannot prove whether an external action occurred, so an
|
||||
* ambiguous fence remains fail-closed pending explicit reconciliation. */
|
||||
if (legacyEventActorErrorHistoryDurable) {
|
||||
await sealLegacyEventActorTurn();
|
||||
}
|
||||
|
||||
if (ownsScheduledFailure && !scheduleTerminalOutcomeRecorded) {
|
||||
const scheduledFailure = classifyScheduledFailure(
|
||||
error,
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ const {
|
|||
getUserMemories,
|
||||
getRoleByName,
|
||||
isSubagentOwnerAdmissible,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
} = require('~/models');
|
||||
const {
|
||||
acquireEventChildGenerationLease,
|
||||
|
|
@ -113,6 +114,31 @@ function deleteResumedGenerationCheckpoint({
|
|||
return deleteAgentCheckpoint(conversationId, checkpointerCfg, checkpointGeneration);
|
||||
}
|
||||
|
||||
async function sealResumedLegacyEventActorTurn({ userId, conversationId, metadata }) {
|
||||
const token = metadata?.agentEventLegacyTurnToken;
|
||||
if (typeof token !== 'string' || token === '') {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const sealed = await completeAgentEventActorLegacyTurn({
|
||||
user: userId,
|
||||
conversationId,
|
||||
...(metadata?.tenantId == null ? {} : { tenantId: metadata.tenantId }),
|
||||
token,
|
||||
});
|
||||
if (!sealed) {
|
||||
logger.error(
|
||||
`[event-actor] Resumed legacy turn fence ${token} was not sealed; forks stay blocked until bounded reclaim`,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
`[event-actor] Failed to seal resumed legacy turn fence ${token}; forks stay blocked until bounded reclaim`,
|
||||
getSafeErrorMetadata(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Error-path checkpoint cleanup runs after the HTTP ACK. A storage failure
|
||||
* must be observable, but must not escape the controller catch and bypass the
|
||||
* remaining request-context/concurrency/client cleanup in `finally`. */
|
||||
|
|
@ -472,6 +498,14 @@ async function finalizeResumedTurn({
|
|||
if (!savedResponseMessage) {
|
||||
throw new Error('Resumed response could not be persisted before terminal publication');
|
||||
}
|
||||
/** The response row is now the durable history barrier for the resumed
|
||||
* legacy turn. Seal its exact pre-pause token before publishing FINAL; a
|
||||
* failed seal remains fail-closed and is recovered by the bounded path. */
|
||||
await sealResumedLegacyEventActorTurn({
|
||||
userId,
|
||||
conversationId,
|
||||
metadata: meta,
|
||||
});
|
||||
|
||||
const convo = await getConvo(userId, conversationId);
|
||||
const conversation = { ...(convo ?? {}), conversationId };
|
||||
|
|
|
|||
|
|
@ -308,6 +308,22 @@ const initializeClient = async ({
|
|||
* server-owned per-agent map. This covers both traditional TOOL_END events
|
||||
* and event-driven ON_TOOL_EXECUTE callbacks. */
|
||||
const toolEndCallback = async (data, metadata = {}) => {
|
||||
/** Event-actor action receipt: recorded in graph context at execution time
|
||||
* so fork classification never races the asynchronously populated run-step
|
||||
* collection. Observational only — a recorder failure must not disturb the
|
||||
* tool result path. */
|
||||
if (typeof req._agentEventActionObserver === 'function') {
|
||||
try {
|
||||
req._agentEventActionObserver(data);
|
||||
} catch (observerError) {
|
||||
logger.warn('[toolEndCallback] Event actor action observer failed', observerError);
|
||||
}
|
||||
}
|
||||
/** Policy-withheld outputs exist solely as execution evidence for the
|
||||
* observer above; nothing may flow to artifact processing. */
|
||||
if (data?.outputFiltered === true) {
|
||||
return;
|
||||
}
|
||||
const node = typeof metadata.langgraph_node === 'string' ? metadata.langgraph_node : '';
|
||||
const nodeAgentId = node.startsWith(GraphNodeKeys.TOOLS)
|
||||
? node.slice(GraphNodeKeys.TOOLS.length)
|
||||
|
|
|
|||
|
|
@ -568,6 +568,15 @@ endpoints:
|
|||
# eventDriven:
|
||||
# childTurns: false
|
||||
# completionWakeups: false
|
||||
# # Reuse bound child state through isolated checkpoint forks. Enable only
|
||||
# # after every API replica supports this lifecycle. Requires the default
|
||||
# # Mongo checkpointer; it is incompatible with checkpointer.type: memory.
|
||||
# # The committed head is an ordinary checkpoint under `checkpointer.ttl`,
|
||||
# # so an actor idle for longer than that TTL rebuilds from message history
|
||||
# # on its next event. Lowering the TTL below an actor's event interval
|
||||
# # leaves it correct but never warm. Agents with skill primes also stay
|
||||
# # on the legacy path for now, so edited skills always reach the actor.
|
||||
# checkpointForks: false
|
||||
# # Optional trusted origin for internal trigger delivery. By default, LibreChat
|
||||
# # uses its own bound listener; override only for an internal TLS/front-door route.
|
||||
# # selfUrl: 'https://librechat.internal'
|
||||
|
|
|
|||
|
|
@ -8,9 +8,12 @@ import {
|
|||
captureAgentCheckpointGeneration,
|
||||
deleteAgentCheckpoint,
|
||||
deleteAgentCheckpoints,
|
||||
forkAgentEventCheckpoint,
|
||||
captureAgentEventCheckpoint,
|
||||
LazyMongoSaver,
|
||||
CheckpointTooLargeError,
|
||||
LIBRECHAT_CHECKPOINT_NAMESPACE_KEY,
|
||||
LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY,
|
||||
__resetCheckpointerForTests,
|
||||
} from './checkpointer';
|
||||
|
||||
|
|
@ -105,6 +108,149 @@ describe('checkpointer (mongodb-memory-server integration)', () => {
|
|||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it('persists clean event-actor exits and forks only the committed checkpoint', async () => {
|
||||
const saver = await getAgentCheckpointer(MONGO_CFG);
|
||||
const threadId = `actor-${new mongoose.Types.ObjectId().toString()}`;
|
||||
const checkpoint = emptyCheckpoint();
|
||||
const config = {
|
||||
configurable: {
|
||||
thread_id: threadId,
|
||||
checkpoint_ns: '',
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: 'event-actor/base',
|
||||
[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY]: 'event-1',
|
||||
},
|
||||
};
|
||||
await saver!.put(config, checkpoint, {
|
||||
source: 'input',
|
||||
step: -1,
|
||||
parents: {},
|
||||
});
|
||||
|
||||
await expect(
|
||||
captureAgentEventCheckpoint(threadId, 'event-actor/base', 'event-1', MONGO_CFG),
|
||||
).resolves.toEqual({
|
||||
threadId,
|
||||
checkpointId: checkpoint.id,
|
||||
checkpointNs: 'event-actor/base',
|
||||
});
|
||||
await expect(
|
||||
forkAgentEventCheckpoint(
|
||||
{ threadId, checkpointId: checkpoint.id, checkpointNs: 'event-actor/base' },
|
||||
'event-actor/fork',
|
||||
'event-2',
|
||||
MONGO_CFG,
|
||||
),
|
||||
).resolves.toEqual({
|
||||
threadId,
|
||||
checkpointId: checkpoint.id,
|
||||
checkpointNs: 'event-actor/fork',
|
||||
});
|
||||
expect(
|
||||
await mongoose.connection
|
||||
.db!.collection('agent_checkpoints')
|
||||
.countDocuments({ thread_id: threadId }),
|
||||
).toBe(2);
|
||||
});
|
||||
|
||||
it('warm-continues from a copied actor head without mutating the committed base', async () => {
|
||||
const { StateGraph, START, END, Annotation } = await import('@langchain/langgraph');
|
||||
const saver = await getAgentCheckpointer(MONGO_CFG);
|
||||
const threadId = `actor-${new mongoose.Types.ObjectId().toString()}`;
|
||||
const State = Annotation.Root({
|
||||
events: Annotation<string[]>({
|
||||
reducer: (left, right) => [...left, ...right],
|
||||
default: () => [],
|
||||
}),
|
||||
});
|
||||
const graph = new StateGraph(State)
|
||||
.addNode('observe', (state: { events: string[] }) => ({
|
||||
events: [`seen:${state.events[state.events.length - 1]}`],
|
||||
}))
|
||||
.addEdge(START, 'observe')
|
||||
.addEdge('observe', END)
|
||||
.compile({ checkpointer: saver as never });
|
||||
const config = (checkpointNamespace: string, invocationId: string, checkpointId?: string) => ({
|
||||
configurable: {
|
||||
thread_id: threadId,
|
||||
checkpoint_ns: '',
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: checkpointNamespace,
|
||||
[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY]: invocationId,
|
||||
...(checkpointId == null ? {} : { checkpoint_id: checkpointId }),
|
||||
},
|
||||
durability: 'exit' as const,
|
||||
});
|
||||
|
||||
await graph.invoke({ events: ['event-1'] }, config('event-actor/base', 'event-1'));
|
||||
const base = await captureAgentEventCheckpoint(
|
||||
threadId,
|
||||
'event-actor/base',
|
||||
'event-1',
|
||||
MONGO_CFG,
|
||||
);
|
||||
expect(base).not.toBeNull();
|
||||
const fork = await forkAgentEventCheckpoint(base!, 'event-actor/fork', 'event-2', MONGO_CFG);
|
||||
expect(fork).not.toBeNull();
|
||||
|
||||
const warm = await graph.invoke(
|
||||
{ events: ['event-2'] },
|
||||
config('event-actor/fork', 'event-2', fork!.checkpointId),
|
||||
);
|
||||
expect(warm.events).toEqual(['event-1', 'seen:event-1', 'event-2', 'seen:event-2']);
|
||||
const committedBase = await saver!.getTuple({
|
||||
configurable: {
|
||||
thread_id: threadId,
|
||||
checkpoint_ns: '',
|
||||
checkpoint_id: base!.checkpointId,
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: 'event-actor/base',
|
||||
},
|
||||
});
|
||||
expect(committedBase?.checkpoint.channel_values.events).toEqual(['event-1', 'seen:event-1']);
|
||||
});
|
||||
|
||||
it('cold-rebuilds in a fresh namespace when an invalidated head id is not copied', async () => {
|
||||
const { StateGraph, START, END, Annotation } = await import('@langchain/langgraph');
|
||||
const saver = await getAgentCheckpointer(MONGO_CFG);
|
||||
const threadId = `actor-${new mongoose.Types.ObjectId().toString()}`;
|
||||
const State = Annotation.Root({
|
||||
events: Annotation<string[]>({
|
||||
reducer: (left, right) => [...left, ...right],
|
||||
default: () => [],
|
||||
}),
|
||||
});
|
||||
const graph = new StateGraph(State)
|
||||
.addNode('observe', (state: { events: string[] }) => ({
|
||||
events: [`seen:${state.events[state.events.length - 1]}`],
|
||||
}))
|
||||
.addEdge(START, 'observe')
|
||||
.addEdge('observe', END)
|
||||
.compile({ checkpointer: saver as never });
|
||||
const config = (checkpointNamespace: string, invocationId: string, checkpointId?: string) => ({
|
||||
configurable: {
|
||||
thread_id: threadId,
|
||||
checkpoint_ns: '',
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: checkpointNamespace,
|
||||
[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY]: invocationId,
|
||||
...(checkpointId == null ? {} : { checkpoint_id: checkpointId }),
|
||||
},
|
||||
durability: 'exit' as const,
|
||||
});
|
||||
|
||||
await graph.invoke({ events: ['legacy-before'] }, config('event-actor/base', 'event-1'));
|
||||
const base = await captureAgentEventCheckpoint(
|
||||
threadId,
|
||||
'event-actor/base',
|
||||
'event-1',
|
||||
MONGO_CFG,
|
||||
);
|
||||
expect(base).not.toBeNull();
|
||||
|
||||
const rebuilt = await graph.invoke(
|
||||
{ events: ['event-after-legacy'] },
|
||||
config('event-actor/cold', 'event-2', base!.checkpointId),
|
||||
);
|
||||
expect(rebuilt.events).toEqual(['event-after-legacy', 'seen:event-after-legacy']);
|
||||
});
|
||||
|
||||
it('deleteAgentCheckpoint prunes a thread’s persisted checkpoint', async () => {
|
||||
const saver = await getAgentCheckpointer(MONGO_CFG);
|
||||
expect(saver).toBeDefined();
|
||||
|
|
|
|||
|
|
@ -20,6 +20,9 @@ import type { RunnableConfig } from '@langchain/core/runnables';
|
|||
* graph-visible conversation `thread_id`.
|
||||
*/
|
||||
export const LIBRECHAT_CHECKPOINT_NAMESPACE_KEY = '__librechat_checkpoint_ns';
|
||||
/** Marks a checkpoint write as belonging to an isolated event-actor attempt.
|
||||
* Unlike ordinary clean chat exits, these exits are durable candidate heads. */
|
||||
export const LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY = '__librechat_event_actor_invocation_id';
|
||||
|
||||
const CHECKPOINT_NAMESPACE_SEPARATOR = '|';
|
||||
|
||||
|
|
@ -28,6 +31,11 @@ function generationCheckpointNamespace(config: RunnableConfig): string | undefin
|
|||
return typeof value === 'string' && value.length > 0 ? value : undefined;
|
||||
}
|
||||
|
||||
function isEventActorInvocation(config: RunnableConfig): boolean {
|
||||
const value = config.configurable?.[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY];
|
||||
return typeof value === 'string' && value.length > 0;
|
||||
}
|
||||
|
||||
/** Prefix every root/subgraph storage namespace with the immutable generation. */
|
||||
function toStorageCheckpointConfig(config: RunnableConfig): RunnableConfig {
|
||||
const generationNamespace = generationCheckpointNamespace(config);
|
||||
|
|
@ -248,10 +256,10 @@ export const CHECKPOINT_HARD_LIMIT_BYTES: number =
|
|||
export const CHECKPOINT_WARN_BYTES: number = 8 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* A HITL checkpoint whose serialized state exceeds {@link CHECKPOINT_HARD_LIMIT_BYTES} — more
|
||||
* A durable checkpoint whose serialized state exceeds {@link CHECKPOINT_HARD_LIMIT_BYTES} — more
|
||||
* than MongoDB can hold in a single document. Thrown BEFORE the doomed write so the run fails
|
||||
* with a clear, typed message instead of a raw driver `BSONObjectTooLarge`. The pause cannot be
|
||||
* persisted regardless of how it is handled upstream; a durable resume is impossible for this turn.
|
||||
* with a clear, typed message instead of a raw driver `BSONObjectTooLarge`. The checkpoint cannot
|
||||
* be persisted regardless of how it is handled upstream, so a durable resume is impossible.
|
||||
*/
|
||||
export class CheckpointTooLargeError extends Error {
|
||||
readonly code = 'CHECKPOINT_TOO_LARGE';
|
||||
|
|
@ -262,9 +270,9 @@ export class CheckpointTooLargeError extends Error {
|
|||
) {
|
||||
const mb = (n: number): string => (n / 1024 / 1024).toFixed(1);
|
||||
super(
|
||||
`Checkpoint state is ${mb(bytes)} MB, over the ${mb(limit)} MB limit for a durable pause. ` +
|
||||
'This conversation carries too much state to pause for input — large tool outputs or ' +
|
||||
'inlined media are the usual cause. Start a new conversation or reduce context.',
|
||||
`Checkpoint state is ${mb(bytes)} MB, over the ${mb(limit)} MB durable limit. ` +
|
||||
'This conversation carries too much state to resume — large tool outputs or inlined ' +
|
||||
'media are the usual cause. Start a new conversation or reduce context.',
|
||||
);
|
||||
this.name = 'CheckpointTooLargeError';
|
||||
}
|
||||
|
|
@ -334,6 +342,9 @@ export class LazyMongoSaver extends MongoDBSaver {
|
|||
taskId: string,
|
||||
): Promise<void> {
|
||||
const storageConfig = toStorageCheckpointConfig(config);
|
||||
if (isEventActorInvocation(config)) {
|
||||
return super.putWrites(storageConfig, writes, taskId);
|
||||
}
|
||||
const checkpointId = config.configurable?.checkpoint_id as string | undefined;
|
||||
if (!checkpointId) {
|
||||
// No checkpoint id to tie a fate to — forward untouched (the base saver's contract).
|
||||
|
|
@ -389,6 +400,11 @@ export class LazyMongoSaver extends MongoDBSaver {
|
|||
checkpoint: Checkpoint,
|
||||
metadata: CheckpointMetadata,
|
||||
): Promise<RunnableConfig> {
|
||||
if (isEventActorInvocation(config)) {
|
||||
await this.assertCheckpointFitsDocument(config, checkpoint, metadata);
|
||||
const persisted = await super.put(toStorageCheckpointConfig(config), checkpoint, metadata);
|
||||
return fromStorageCheckpointConfig(persisted, config);
|
||||
}
|
||||
if (this.writeAnchorIds.delete(checkpoint.id)) {
|
||||
// Carries a resumable write (interrupt / real-channel delta anchor) — persist so resume
|
||||
// can read it, and remember the id briefly so any bookkeeping batch dispatched after
|
||||
|
|
@ -453,8 +469,8 @@ export class LazyMongoSaver extends MongoDBSaver {
|
|||
* `warn` past {@link warnBytes}, and throw {@link CheckpointTooLargeError} past
|
||||
* {@link hardLimitBytes} — BEFORE the write, so an oversize pause fails legibly rather than as a
|
||||
* raw `BSONObjectTooLarge`. Serializes with the same `serde` the base `put` uses, so the measured
|
||||
* bytes match what would be stored. The extra serialization runs only on the (rare) HITL pause
|
||||
* path — never the clean-exit common path, which is discarded before reaching here.
|
||||
* bytes match what would be stored. The extra serialization runs only when a checkpoint is
|
||||
* selected for durable retention: HITL pauses and event-actor invocation heads.
|
||||
*/
|
||||
private async assertCheckpointFitsDocument(
|
||||
config: RunnableConfig,
|
||||
|
|
@ -485,18 +501,18 @@ export class LazyMongoSaver extends MongoDBSaver {
|
|||
// TTL reclaim it. Drop any parked bookkeeping so it doesn't linger in memory.
|
||||
this.bufferedBookkeeping.delete(checkpoint.id);
|
||||
logger.error(
|
||||
`[checkpointer] HITL checkpoint for thread ${threadId ?? 'unknown'} is ${mb(bytes)} MB, over the ${mb(this.hardLimitBytes)} MB durable-pause limit; refusing the write (a document past 16 MB cannot be stored in MongoDB).`,
|
||||
`[checkpointer] Durable checkpoint for thread ${threadId ?? 'unknown'} is ${mb(bytes)} MB, over the ${mb(this.hardLimitBytes)} MB limit; refusing the write (a document past 16 MB cannot be stored in MongoDB).`,
|
||||
);
|
||||
throw new CheckpointTooLargeError(bytes, this.hardLimitBytes, threadId);
|
||||
}
|
||||
if (bytes >= this.warnBytes) {
|
||||
logger.warn(
|
||||
`[checkpointer] HITL checkpoint for thread ${threadId ?? 'unknown'} is ${mb(bytes)} MB, past the ${mb(this.warnBytes)} MB soft threshold (hard limit ${mb(this.hardLimitBytes)} MB) — approaching MongoDB's single-document ceiling.`,
|
||||
`[checkpointer] Durable checkpoint for thread ${threadId ?? 'unknown'} is ${mb(bytes)} MB, past the ${mb(this.warnBytes)} MB soft threshold (hard limit ${mb(this.hardLimitBytes)} MB) — approaching MongoDB's single-document ceiling.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
logger.debug(
|
||||
`[checkpointer] Persisting HITL checkpoint for thread ${threadId ?? 'unknown'}: ${bytes} bytes`,
|
||||
`[checkpointer] Persisting durable checkpoint for thread ${threadId ?? 'unknown'}: ${bytes} bytes`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -599,7 +615,7 @@ export async function getAgentCheckpointer(
|
|||
}
|
||||
if (mongoose.connection.readyState !== 1) {
|
||||
logger.warn(
|
||||
'[checkpointer] Mongoose not connected; HITL runs will use an in-process checkpointer this turn (paused runs will not survive a restart or resolve on another replica).',
|
||||
'[checkpointer] Mongoose not connected; durable agent continuations will use an in-process checkpointer this turn and will not survive a restart or resolve on another replica.',
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
|
|
@ -612,6 +628,78 @@ export async function getAgentCheckpointer(
|
|||
return saverPromise;
|
||||
}
|
||||
|
||||
export interface AgentEventCheckpointReference {
|
||||
threadId: string;
|
||||
checkpointId: string;
|
||||
checkpointNs: string;
|
||||
}
|
||||
|
||||
function eventActorRunnableConfig(
|
||||
reference: Pick<AgentEventCheckpointReference, 'threadId' | 'checkpointNs'>,
|
||||
invocationId: string,
|
||||
checkpointId?: string,
|
||||
): RunnableConfig {
|
||||
return {
|
||||
configurable: {
|
||||
thread_id: reference.threadId,
|
||||
checkpoint_ns: '',
|
||||
[LIBRECHAT_CHECKPOINT_NAMESPACE_KEY]: reference.checkpointNs,
|
||||
[LIBRECHAT_EVENT_ACTOR_INVOCATION_KEY]: invocationId,
|
||||
...(checkpointId == null ? {} : { checkpoint_id: checkpointId }),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Copies one committed actor head into an invocation-owned namespace. */
|
||||
export async function forkAgentEventCheckpoint(
|
||||
source: AgentEventCheckpointReference,
|
||||
checkpointNs: string,
|
||||
invocationId: string,
|
||||
cfg?: TCheckpointerConfig,
|
||||
): Promise<AgentEventCheckpointReference | null> {
|
||||
const saver = await getAgentCheckpointer(cfg);
|
||||
if (!saver || checkpointNs.length === 0 || invocationId.length === 0) {
|
||||
return null;
|
||||
}
|
||||
const tuple = await saver.getTuple(
|
||||
eventActorRunnableConfig(source, invocationId, source.checkpointId),
|
||||
);
|
||||
if (!tuple || tuple.metadata == null || (tuple.pendingWrites?.length ?? 0) > 0) {
|
||||
return null;
|
||||
}
|
||||
const target = { threadId: source.threadId, checkpointNs };
|
||||
const persisted = await saver.put(
|
||||
eventActorRunnableConfig(target, invocationId),
|
||||
tuple.checkpoint,
|
||||
tuple.metadata,
|
||||
);
|
||||
const checkpointId = persisted.configurable?.checkpoint_id;
|
||||
if (typeof checkpointId !== 'string' || checkpointId.length === 0) {
|
||||
throw new Error('Event actor checkpoint fork did not return a checkpoint id');
|
||||
}
|
||||
return { ...target, checkpointId };
|
||||
}
|
||||
|
||||
/** Reads the terminal checkpoint produced inside one invocation namespace. */
|
||||
export async function captureAgentEventCheckpoint(
|
||||
threadId: string,
|
||||
checkpointNs: string,
|
||||
invocationId: string,
|
||||
cfg?: TCheckpointerConfig,
|
||||
): Promise<AgentEventCheckpointReference | null> {
|
||||
const saver = await getAgentCheckpointer(cfg);
|
||||
if (!saver) {
|
||||
return null;
|
||||
}
|
||||
const tuple = await saver.getTuple(
|
||||
eventActorRunnableConfig({ threadId, checkpointNs }, invocationId),
|
||||
);
|
||||
const checkpointId = tuple?.checkpoint.id;
|
||||
return typeof checkpointId === 'string' && checkpointId.length > 0
|
||||
? { threadId, checkpointId, checkpointNs }
|
||||
: null;
|
||||
}
|
||||
|
||||
async function buildMongoSaver(
|
||||
resolved: ResolvedCheckpointerConfig,
|
||||
): Promise<MongoDBSaver | undefined> {
|
||||
|
|
@ -636,7 +724,7 @@ async function buildMongoSaver(
|
|||
errors,
|
||||
);
|
||||
}
|
||||
logger.info('[checkpointer] Durable Mongo checkpointer ready for HITL resume');
|
||||
logger.info('[checkpointer] Durable Mongo checkpointer ready for agent continuation');
|
||||
return saver;
|
||||
} catch (err) {
|
||||
// Reset so a later run can retry rather than being stuck on a failed build.
|
||||
|
|
|
|||
|
|
@ -381,8 +381,13 @@ describe('createToolExecuteHandler — background tool calls', () => {
|
|||
});
|
||||
expect(JSON.parse(allowedPoll.content).result).toBe('safe historical result');
|
||||
expect(toolEndCallback).toHaveBeenCalledTimes(1);
|
||||
/** Delivery callbacks must carry provenance: they report the ORIGINAL
|
||||
* tool's name with the poll call's arguments, so identity-fencing
|
||||
* consumers (the event-actor action recorder) can exclude them. */
|
||||
expect(toolEndCallback).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
backgroundDelivery: true,
|
||||
input: { background_task_id: taskId },
|
||||
output: expect.objectContaining({ artifact }),
|
||||
}),
|
||||
expect.any(Object),
|
||||
|
|
|
|||
|
|
@ -661,7 +661,57 @@ describe('createToolExecuteHandler', () => {
|
|||
expect(result.errorMessage).toContain('content_filter_block');
|
||||
expect(result.errorMessage).not.toContain(protectedValue);
|
||||
expect(result.artifact).toBeUndefined();
|
||||
expect(toolEndCallback).not.toHaveBeenCalled();
|
||||
/** The execution already happened, so identity-only evidence flows —
|
||||
* with blank content and no artifact, never the blocked output. */
|
||||
expect(toolEndCallback).toHaveBeenCalledTimes(1);
|
||||
expect(toolEndCallback).toHaveBeenCalledWith(
|
||||
{
|
||||
input: {},
|
||||
outputFiltered: true,
|
||||
output: {
|
||||
name: 'filtered_output_tool',
|
||||
tool_call_id: 'call_filtered_output',
|
||||
content: '',
|
||||
},
|
||||
},
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(JSON.stringify(toolEndCallback.mock.calls)).not.toContain(protectedValue);
|
||||
});
|
||||
|
||||
it('supplies the executed arguments alongside the output to the tool end callback', async () => {
|
||||
const toolEndCallback = jest.fn();
|
||||
const tool = {
|
||||
name: 'submit_move',
|
||||
invoke: jest.fn(async () => ({ content: '{"ok":true}' })),
|
||||
};
|
||||
const loadTools: ToolExecuteOptions['loadTools'] = jest.fn(async () => ({
|
||||
loadedTools: [tool] as never[],
|
||||
}));
|
||||
const handler = createToolExecuteHandler({ loadTools, toolEndCallback });
|
||||
|
||||
const [result] = await invokeHandler(handler, [
|
||||
{ id: 'call_submit_move', name: 'submit_move', args: { gameId: 'game-1', expectedPly: 8 } },
|
||||
]);
|
||||
|
||||
expect(result.content).toBe('{"ok":true}');
|
||||
/** The stream-consumer tool-end path cannot reconstruct execution input,
|
||||
* so the execution handler — which owns both halves — must supply it.
|
||||
* The event-actor action recorder fences its declared argument subset
|
||||
* against exactly this field; without it, warm continuation silently
|
||||
* degrades to cold history rebuilds (proven by live canary). */
|
||||
expect(toolEndCallback).toHaveBeenCalledTimes(1);
|
||||
expect(toolEndCallback).toHaveBeenCalledWith(
|
||||
{
|
||||
input: { gameId: 'game-1', expectedPly: 8 },
|
||||
output: expect.objectContaining({
|
||||
name: 'submit_move',
|
||||
tool_call_id: 'call_submit_move',
|
||||
content: '{"ok":true}',
|
||||
}),
|
||||
},
|
||||
expect.any(Object),
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
|
|
@ -767,7 +817,13 @@ describe('createToolExecuteHandler', () => {
|
|||
expect(result.errorMessage).toContain('content_filter_block');
|
||||
expect(result.errorMessage).not.toContain(protectedValue);
|
||||
expect(result.artifact).toBeUndefined();
|
||||
expect(toolEndCallback).not.toHaveBeenCalled();
|
||||
/** Execution identity flows despite the blocked output; the protected
|
||||
* content itself never reaches the callback. */
|
||||
expect(toolEndCallback).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ outputFiltered: true }),
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(JSON.stringify(toolEndCallback.mock.calls)).not.toContain(protectedValue);
|
||||
});
|
||||
|
||||
it('fails closed when tool output cannot be completely traversed', async () => {
|
||||
|
|
@ -819,7 +875,14 @@ describe('createToolExecuteHandler', () => {
|
|||
expect(result.status).toBe('error');
|
||||
expect(result.errorMessage).toContain('could not be completely inspected');
|
||||
expect(result.artifact).toBeUndefined();
|
||||
expect(toolEndCallback).not.toHaveBeenCalled();
|
||||
/** The tool did execute; only its uninspectable output is withheld. */
|
||||
expect(toolEndCallback).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
outputFiltered: true,
|
||||
output: expect.objectContaining({ content: '' }),
|
||||
}),
|
||||
expect.any(Object),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -76,6 +76,26 @@ import { instrumentPtcToolMap } from './ptc';
|
|||
import { markSandboxReady } from './prewarm';
|
||||
|
||||
export interface ToolEndCallbackData {
|
||||
/** The executed call's arguments. The stream-consumer tool-end path cannot
|
||||
* reconstruct these, so the execution handler — which owns both halves —
|
||||
* must supply them for consumers that fence on the input (the event-actor
|
||||
* action recorder validates its declared argument subset against this). */
|
||||
input?: unknown;
|
||||
/** True when this callback delivers the harvested completion of a
|
||||
* previously dispatched background task on a poll turn. `output.name` then
|
||||
* reports the ORIGINAL tool for artifact attribution while `input` carries
|
||||
* the poll call's arguments — consumers that fence on execution identity
|
||||
* (the event-actor action recorder) must ignore these deliveries, or a
|
||||
* name-only expected action could be impersonated by work another turn
|
||||
* dispatched. */
|
||||
backgroundDelivery?: boolean;
|
||||
/** True when the tool executed successfully but its returned content was
|
||||
* withheld by post-execution output policy. `output.content` is blank and
|
||||
* no artifact rides the callback — this delivery exists solely so
|
||||
* execution-identity consumers (the event-actor action recorder) can prove
|
||||
* the side effect occurred; a retry of an "actionless" turn would otherwise
|
||||
* repeat an external action whose output was merely filtered. */
|
||||
outputFiltered?: boolean;
|
||||
output: {
|
||||
name: string;
|
||||
tool_call_id: string;
|
||||
|
|
@ -4641,6 +4661,8 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand
|
|||
try {
|
||||
await toolEndCallback(
|
||||
{
|
||||
input: tc.args,
|
||||
backgroundDelivery: true,
|
||||
output: {
|
||||
name: pending.toolName,
|
||||
tool_call_id: tc.id,
|
||||
|
|
@ -4893,6 +4915,30 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand
|
|||
errorMessage: handlerResult.errorMessage,
|
||||
});
|
||||
if (filteredOutput != null) {
|
||||
/** The side effect already happened; only the returned
|
||||
* content is being withheld. Emit execution identity so
|
||||
* an applied action is never reclassified as actionless
|
||||
* and re-executed — the blocked output stays blank. */
|
||||
if (toolEndCallback && handlerResult.errorMessage == null) {
|
||||
try {
|
||||
await toolEndCallback(
|
||||
{
|
||||
input: tc.args,
|
||||
outputFiltered: true,
|
||||
output: { name: tc.name, tool_call_id: tc.id, content: '' },
|
||||
},
|
||||
{
|
||||
...(metadata ?? {}),
|
||||
executingAgentId: agentId,
|
||||
} as ToolEndCallbackMetadata,
|
||||
);
|
||||
} catch (evidenceError) {
|
||||
logger.warn(
|
||||
`[ON_TOOL_EXECUTE] Filtered-output evidence delivery failed for ${tc.name}`,
|
||||
evidenceError,
|
||||
);
|
||||
}
|
||||
}
|
||||
return filteredOutput;
|
||||
}
|
||||
|
||||
|
|
@ -4900,6 +4946,7 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand
|
|||
try {
|
||||
await toolEndCallback(
|
||||
{
|
||||
input: tc.args,
|
||||
output: {
|
||||
name: tc.name,
|
||||
tool_call_id: tc.id,
|
||||
|
|
@ -5146,12 +5193,37 @@ export function createToolExecuteHandler(options: ToolExecuteOptions): EventHand
|
|||
},
|
||||
);
|
||||
if (filteredOutput != null) {
|
||||
/** The side effect already happened; only the returned
|
||||
* content is being withheld. Emit execution identity so
|
||||
* an applied action is never reclassified as actionless
|
||||
* and re-executed — the blocked output stays blank. */
|
||||
if (toolEndCallback) {
|
||||
try {
|
||||
await toolEndCallback(
|
||||
{
|
||||
input: tc.args,
|
||||
outputFiltered: true,
|
||||
output: { name: tc.name, tool_call_id: tc.id, content: '' },
|
||||
},
|
||||
{
|
||||
...(metadata ?? {}),
|
||||
executingAgentId: agentId,
|
||||
} as ToolEndCallbackMetadata,
|
||||
);
|
||||
} catch (evidenceError) {
|
||||
logger.warn(
|
||||
`[ON_TOOL_EXECUTE] Filtered-output evidence delivery failed for ${tc.name}`,
|
||||
evidenceError,
|
||||
);
|
||||
}
|
||||
}
|
||||
return filteredOutput;
|
||||
}
|
||||
|
||||
if (toolEndCallback) {
|
||||
await toolEndCallback(
|
||||
{
|
||||
input: tc.args,
|
||||
output: {
|
||||
name: tc.name,
|
||||
tool_call_id: tc.id,
|
||||
|
|
|
|||
|
|
@ -1342,6 +1342,7 @@ export async function createRun({
|
|||
steering,
|
||||
activityLabel,
|
||||
activityPhase,
|
||||
eventActorCheckpointing = false,
|
||||
hitlCapable = false,
|
||||
toolInputValidationErrors,
|
||||
sessionStartSource,
|
||||
|
|
@ -1430,6 +1431,8 @@ export async function createRun({
|
|||
activityLabel?: { hook: HookCallback<'PostToolBatch'> };
|
||||
/** Run-wide parent phase collector; registered after child batch labels. */
|
||||
activityPhase?: { hook: HookCallback<'PostToolBatch'> };
|
||||
/** Persist clean terminal checkpoints for an isolated bound-event invocation. */
|
||||
eventActorCheckpointing?: boolean;
|
||||
/**
|
||||
* Whether the caller implements the HITL pause/resume lifecycle (inspects
|
||||
* `run.getInterrupt()`, persists a pending action, exposes a resume route). Gates the
|
||||
|
|
@ -1834,7 +1837,7 @@ export async function createRun({
|
|||
*/
|
||||
const asksUserQuestions =
|
||||
hitlCapable && !askToolAdminDisabled && agents.some(agentRequestsAskUserQuestion);
|
||||
if (hitl || asksUserQuestions) {
|
||||
if (hitl || asksUserQuestions || eventActorCheckpointing) {
|
||||
const checkpointer = await getAgentCheckpointer(agentsEndpointConfig?.checkpointer);
|
||||
graphConfig.compileOptions = { ...graphConfig.compileOptions, checkpointer };
|
||||
}
|
||||
|
|
|
|||
756
packages/api/src/agents/triggers/actor.spec.ts
Normal file
756
packages/api/src/agents/triggers/actor.spec.ts
Normal file
|
|
@ -0,0 +1,756 @@
|
|||
import type {
|
||||
IAgentEventActorReconciliation,
|
||||
IAgentEventActorState,
|
||||
} from '@librechat/data-schemas';
|
||||
import {
|
||||
captureAgentEventCheckpoint,
|
||||
deleteAgentCheckpoint,
|
||||
forkAgentEventCheckpoint,
|
||||
getAgentCheckpointer,
|
||||
} from '../checkpointer';
|
||||
import { createAgentEventActionRecorder, findAgentEventAppliedAction } from './outcome';
|
||||
import { executeAgentEventActor } from './actor';
|
||||
|
||||
jest.mock('../checkpointer', () => ({
|
||||
...jest.requireActual('../checkpointer'),
|
||||
captureAgentEventCheckpoint: jest.fn(),
|
||||
deleteAgentCheckpoint: jest.fn(),
|
||||
forkAgentEventCheckpoint: jest.fn(),
|
||||
getAgentCheckpointer: jest.fn(),
|
||||
}));
|
||||
|
||||
const mockedCapture = jest.mocked(captureAgentEventCheckpoint);
|
||||
const mockedDelete = jest.mocked(deleteAgentCheckpoint);
|
||||
const mockedFork = jest.mocked(forkAgentEventCheckpoint);
|
||||
const mockedGetCheckpointer = jest.mocked(getAgentCheckpointer);
|
||||
|
||||
describe('event actor host adapter', () => {
|
||||
const conversationId = 'actor-thread';
|
||||
let state: IAgentEventActorState | null;
|
||||
let epoch = 0;
|
||||
let legacyTurn: { token: string; startedAt: Date } | null = null;
|
||||
let nextCheckpoint = 1;
|
||||
|
||||
beforeEach(() => {
|
||||
state = null;
|
||||
epoch = 0;
|
||||
legacyTurn = null;
|
||||
nextCheckpoint = 1;
|
||||
jest.clearAllMocks();
|
||||
mockedGetCheckpointer.mockResolvedValue({} as never);
|
||||
mockedFork.mockImplementation(async (source, checkpointNs) => ({
|
||||
...source,
|
||||
checkpointNs,
|
||||
}));
|
||||
mockedCapture.mockImplementation(async (threadId, checkpointNs) => ({
|
||||
threadId,
|
||||
checkpointNs,
|
||||
checkpointId: `checkpoint-${nextCheckpoint++}`,
|
||||
}));
|
||||
mockedDelete.mockResolvedValue();
|
||||
});
|
||||
|
||||
const deps = () => ({
|
||||
getSnapshot: jest.fn(async () => ({
|
||||
state,
|
||||
reconciliations: [] as IAgentEventActorReconciliation[],
|
||||
legacyTurn,
|
||||
epoch,
|
||||
})),
|
||||
commitState: jest.fn(async ({ expected, expectedEpoch, checkpoint }) => {
|
||||
if (
|
||||
expectedEpoch !== epoch ||
|
||||
(state == null && expected != null) ||
|
||||
(state != null &&
|
||||
(expected == null ||
|
||||
expected.generation !== state.generation ||
|
||||
expected.checkpoint.checkpointId !== state.checkpoint.checkpointId ||
|
||||
(expected.requiresColdStart === true) !== (state.requiresColdStart === true)))
|
||||
) {
|
||||
return { status: 'stale' as const, ...(state == null ? {} : { state }) };
|
||||
}
|
||||
const previous = state?.checkpoint;
|
||||
state = {
|
||||
generation: (state?.generation ?? 0) + 1,
|
||||
checkpoint,
|
||||
...(previous == null ? {} : { previousCheckpoint: previous }),
|
||||
};
|
||||
return { status: 'committed' as const, state };
|
||||
}),
|
||||
recordReconciliation: jest.fn(async () => true),
|
||||
resolveReconciliation: jest.fn(async () => true),
|
||||
});
|
||||
|
||||
it('cold-starts once, then forks and warm-continues only the next event', async () => {
|
||||
const dependencies = deps();
|
||||
const invocations: Array<{ continuation: string; checkpointId?: string }> = [];
|
||||
const run = async (invocationId: string) =>
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId,
|
||||
event: { id: invocationId, type: 'turn' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async (context) => {
|
||||
invocations.push({
|
||||
continuation: context.continuation,
|
||||
...(context.checkpointId == null ? {} : { checkpointId: context.checkpointId }),
|
||||
});
|
||||
return `response-${invocationId}`;
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move', toolCallId: invocationId }),
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
|
||||
const first = await run('event-1');
|
||||
const second = await run('event-2');
|
||||
|
||||
expect(first).toMatchObject({ value: 'response-event-1', execution: { status: 'applied' } });
|
||||
expect(second).toMatchObject({ value: 'response-event-2', execution: { status: 'applied' } });
|
||||
expect(invocations).toEqual([
|
||||
{ continuation: 'cold' },
|
||||
{ continuation: 'warm', checkpointId: 'checkpoint-1' },
|
||||
]);
|
||||
expect(mockedFork).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ checkpointId: 'checkpoint-1' }),
|
||||
expect.stringMatching(/^event-actor\//),
|
||||
'event-2',
|
||||
undefined,
|
||||
);
|
||||
expect(state).toMatchObject({ generation: 2, checkpoint: { checkpointId: 'checkpoint-2' } });
|
||||
});
|
||||
|
||||
it('commits from the execution-time receipt when run steps lag sendMessage', async () => {
|
||||
const dependencies = deps();
|
||||
const expectedAction = { toolName: 'submit_move', argumentSubset: { gameId: 'game-1' } };
|
||||
const invocations: Array<{ continuation: string }> = [];
|
||||
let toolExecutions = 0;
|
||||
const run = async (invocationId: string) => {
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
return executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId,
|
||||
event: { id: invocationId, type: 'turn' },
|
||||
expectedAction,
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async (context) => {
|
||||
invocations.push({ continuation: context.continuation });
|
||||
toolExecutions += 1;
|
||||
recorder.observeToolEnd({
|
||||
input: { gameId: 'game-1', move: 'e4' },
|
||||
output: {
|
||||
name: 'submit_move_mcp_chess',
|
||||
tool_call_id: `call-${invocationId}`,
|
||||
content: '{"ok":true}',
|
||||
},
|
||||
});
|
||||
return `response-${invocationId}`;
|
||||
},
|
||||
/** Reproduces the observed race: the run-step collection is still
|
||||
* empty the instant sendMessage resolves, so only the graph-context
|
||||
* receipt carries the applied-action proof. */
|
||||
readAppliedAction: () =>
|
||||
recorder.read() ?? findAgentEventAppliedAction(expectedAction, [], []),
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
};
|
||||
|
||||
const first = await run('event-1');
|
||||
const second = await run('event-2');
|
||||
|
||||
expect(first.execution).toMatchObject({
|
||||
status: 'applied',
|
||||
result: { action: { toolName: 'submit_move_mcp_chess', toolCallId: 'call-event-1' } },
|
||||
});
|
||||
expect(second.execution).toMatchObject({ status: 'applied' });
|
||||
expect(invocations).toEqual([{ continuation: 'cold' }, { continuation: 'warm' }]);
|
||||
expect(toolExecutions).toBe(2);
|
||||
expect(state).toMatchObject({ generation: 2, checkpoint: { checkpointId: 'checkpoint-2' } });
|
||||
});
|
||||
|
||||
it('refuses to prepare while a legacy turn fence is open', async () => {
|
||||
state = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-fenced',
|
||||
checkpointNs: 'event-actor/fenced',
|
||||
},
|
||||
};
|
||||
legacyTurn = { token: 'legacy-live', startedAt: new Date() };
|
||||
const dependencies = deps();
|
||||
let invoked = false;
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-during-legacy',
|
||||
event: { id: 'event-during-legacy' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => {
|
||||
invoked = true;
|
||||
return 'response';
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('blocked on an in-flight legacy turn');
|
||||
|
||||
expect(invoked).toBe(false);
|
||||
expect(mockedFork).not.toHaveBeenCalled();
|
||||
expect(dependencies.commitState).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('preserves an old ambiguous fence instead of replaying based on age', async () => {
|
||||
legacyTurn = { token: 'legacy-crashed', startedAt: new Date(Date.now() - 120_000) };
|
||||
const dependencies = deps();
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-after-crash',
|
||||
event: { id: 'event-after-crash' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('blocked on an in-flight legacy turn');
|
||||
|
||||
/** Elapsed time does not establish whether the external action ran. */
|
||||
expect(dependencies.commitState).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('cold-starts after a legacy fallback invalidates the committed head', async () => {
|
||||
state = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-before-legacy',
|
||||
checkpointNs: 'event-actor/before-legacy',
|
||||
},
|
||||
requiresColdStart: true,
|
||||
};
|
||||
const dependencies = deps();
|
||||
let continuation: 'warm' | 'cold' | undefined;
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-after-legacy',
|
||||
event: { id: 'event-after-legacy' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async (context) => {
|
||||
continuation = context.continuation;
|
||||
return 'response';
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).resolves.toMatchObject({ execution: { status: 'applied', continuation: 'cold' } });
|
||||
|
||||
expect(continuation).toBe('cold');
|
||||
expect(mockedFork).not.toHaveBeenCalled();
|
||||
expect(dependencies.commitState).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
expected: expect.objectContaining({ requiresColdStart: true }),
|
||||
}),
|
||||
);
|
||||
expect(state?.requiresColdStart).toBeUndefined();
|
||||
});
|
||||
|
||||
it('cannot commit a cold rebuild past a legacy turn its history predates', async () => {
|
||||
const dependencies = deps();
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-stale-cold',
|
||||
event: { id: 'event-stale-cold' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => {
|
||||
/** A concurrent legacy delivery lands after this cold rebuild
|
||||
* loaded its history. With no head to mark and nothing else to
|
||||
* change, the invalidation epoch is its only durable trace. */
|
||||
epoch += 1;
|
||||
return 'response';
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('commit_conflict reconciliation');
|
||||
|
||||
expect(state).toBeNull();
|
||||
expect(dependencies.commitState).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ expectedEpoch: 0 }),
|
||||
);
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({ status: 'commit_conflict' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('cannot clear a cold-start marker written after warm preparation', async () => {
|
||||
state = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-warm',
|
||||
checkpointNs: 'event-actor/warm',
|
||||
},
|
||||
};
|
||||
const dependencies = deps();
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-raced-by-legacy',
|
||||
event: { id: 'event-raced-by-legacy' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => {
|
||||
state = { ...state!, requiresColdStart: true };
|
||||
return 'response';
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('requires commit_conflict reconciliation');
|
||||
|
||||
expect(dependencies.recordReconciliation).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({ status: 'commit_conflict' }),
|
||||
}),
|
||||
);
|
||||
|
||||
expect(dependencies.commitState).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
expected: expect.not.objectContaining({ requiresColdStart: true }),
|
||||
}),
|
||||
);
|
||||
expect(state?.requiresColdStart).toBe(true);
|
||||
});
|
||||
|
||||
it('discards a no-action fork without advancing the actor head', async () => {
|
||||
state = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-base',
|
||||
checkpointNs: 'event-actor/base',
|
||||
},
|
||||
};
|
||||
const dependencies = deps();
|
||||
const result = await executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-no-action',
|
||||
event: { id: 'event-no-action' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => undefined,
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
|
||||
expect(result.execution.status).toBe('completed_no_action');
|
||||
expect(dependencies.commitState).not.toHaveBeenCalled();
|
||||
expect(mockedDelete).toHaveBeenCalledWith(
|
||||
conversationId,
|
||||
undefined,
|
||||
undefined,
|
||||
expect.objectContaining({
|
||||
throwOnError: true,
|
||||
checkpointNamespace: expect.stringMatching(/^event-actor\//),
|
||||
}),
|
||||
);
|
||||
expect(state.generation).toBe(1);
|
||||
});
|
||||
|
||||
it('preserves action evidence when the provider fails after the tool completed', async () => {
|
||||
const dependencies = deps();
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-action-then-error',
|
||||
event: { id: 'event-action-then-error' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => {
|
||||
throw new Error('provider stream failed after tool');
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('provider stream failed after tool');
|
||||
|
||||
expect(dependencies.commitState).toHaveBeenCalledTimes(1);
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({
|
||||
invocationId: 'event-action-then-error',
|
||||
status: 'invocation_pending',
|
||||
}),
|
||||
}),
|
||||
);
|
||||
expect(dependencies.resolveReconciliation).not.toHaveBeenCalled();
|
||||
expect(mockedDelete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('retains an applied fork when its terminal checkpoint cannot be observed', async () => {
|
||||
mockedCapture.mockResolvedValueOnce(null);
|
||||
const dependencies = deps();
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-checkpoint-indeterminate',
|
||||
event: { id: 'event-checkpoint-indeterminate' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('requires commit_indeterminate reconciliation');
|
||||
|
||||
expect(dependencies.commitState).not.toHaveBeenCalled();
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({ status: 'commit_indeterminate' }),
|
||||
}),
|
||||
);
|
||||
expect(mockedDelete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('recovers an indeterminate cleanup after the actor head was committed', async () => {
|
||||
state = {
|
||||
generation: 2,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-base',
|
||||
checkpointNs: 'event-actor/base',
|
||||
},
|
||||
previousCheckpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-old',
|
||||
checkpointNs: 'event-actor/old',
|
||||
},
|
||||
};
|
||||
mockedDelete.mockRejectedValueOnce(new Error('checkpoint cleanup unavailable'));
|
||||
const dependencies = deps();
|
||||
const result = await executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-commit-then-cleanup-error',
|
||||
event: { id: 'event-commit-then-cleanup-error' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
|
||||
expect(result.execution.status).toBe('applied');
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledTimes(1);
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({ status: 'invocation_pending' }),
|
||||
}),
|
||||
);
|
||||
expect(state.generation).toBe(3);
|
||||
});
|
||||
|
||||
it('persists and surfaces a checkpoint conflict after the action was applied', async () => {
|
||||
state = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-base',
|
||||
checkpointNs: 'event-actor/base',
|
||||
},
|
||||
};
|
||||
const dependencies = deps();
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-conflict',
|
||||
event: { id: 'event-conflict' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => {
|
||||
state = {
|
||||
generation: 2,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-competing',
|
||||
checkpointNs: 'event-actor/competing',
|
||||
},
|
||||
};
|
||||
return 'response';
|
||||
},
|
||||
readAppliedAction: () => ({ toolName: 'submit_move', toolCallId: 'call-conflict' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('requires commit_conflict reconciliation');
|
||||
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({
|
||||
invocationId: 'event-conflict',
|
||||
status: 'commit_conflict',
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-conflict' },
|
||||
}),
|
||||
}),
|
||||
);
|
||||
expect(mockedDelete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('still records reconciliation when an indeterminate commit cannot be read back', async () => {
|
||||
const baseState: IAgentEventActorState = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-base',
|
||||
checkpointNs: 'event-actor/base',
|
||||
},
|
||||
};
|
||||
const dependencies = {
|
||||
getSnapshot: jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
state: baseState,
|
||||
reconciliations: [],
|
||||
legacyTurn: null,
|
||||
epoch: 0,
|
||||
})
|
||||
.mockRejectedValueOnce(new Error('readback unavailable')),
|
||||
commitState: jest.fn(async () => {
|
||||
throw new Error('commit result unavailable');
|
||||
}),
|
||||
recordReconciliation: jest.fn(async () => true),
|
||||
resolveReconciliation: jest.fn(async () => true),
|
||||
};
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-ambiguous-commit',
|
||||
event: { id: 'event-ambiguous-commit' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('requires commit_indeterminate reconciliation');
|
||||
|
||||
expect(dependencies.recordReconciliation).toHaveBeenCalledTimes(2);
|
||||
expect(dependencies.recordReconciliation).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({
|
||||
reconciliation: expect.objectContaining({ status: 'commit_indeterminate' }),
|
||||
}),
|
||||
);
|
||||
expect(mockedDelete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not clear a checkpoint marker before durable history is verified', async () => {
|
||||
const authoritative: IAgentEventActorState = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-authoritative',
|
||||
checkpointNs: 'event-actor/authoritative',
|
||||
},
|
||||
};
|
||||
state = authoritative;
|
||||
const marker = {
|
||||
invocationId: 'event-recovered',
|
||||
status: 'commit_indeterminate' as const,
|
||||
checkpoint: authoritative.checkpoint,
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
};
|
||||
const dependencies = deps();
|
||||
dependencies.getSnapshot.mockResolvedValueOnce({
|
||||
state: authoritative,
|
||||
reconciliations: [marker],
|
||||
legacyTurn: null,
|
||||
epoch: 0,
|
||||
});
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-next',
|
||||
event: { id: 'event-next' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('blocked on commit_indeterminate reconciliation');
|
||||
|
||||
expect(dependencies.resolveReconciliation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not clear a persistence failure merely because its checkpoint is authoritative', async () => {
|
||||
const authoritative: IAgentEventActorState = {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointId: 'checkpoint-with-missing-history',
|
||||
checkpointNs: 'event-actor/missing-history',
|
||||
},
|
||||
};
|
||||
const dependencies = {
|
||||
...deps(),
|
||||
getSnapshot: jest.fn(async () => ({
|
||||
state: authoritative,
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'event-persistence-failed',
|
||||
status: 'persistence_failed' as const,
|
||||
checkpoint: authoritative.checkpoint,
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
legacyTurn: null,
|
||||
epoch: 0,
|
||||
})),
|
||||
};
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-after-persistence-failure',
|
||||
event: { id: 'event-after-persistence-failure' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('blocked on persistence_failed reconciliation');
|
||||
expect(dependencies.resolveReconciliation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('blocks new invocations while a prior applied fork needs reconciliation', async () => {
|
||||
const dependencies = {
|
||||
getSnapshot: jest.fn(async () => ({
|
||||
state: null,
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'event-conflict',
|
||||
status: 'commit_conflict' as const,
|
||||
checkpoint: {
|
||||
threadId: conversationId,
|
||||
checkpointNs: 'event-actor/conflict',
|
||||
},
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
legacyTurn: null,
|
||||
epoch: 0,
|
||||
})),
|
||||
commitState: jest.fn(),
|
||||
recordReconciliation: jest.fn(),
|
||||
resolveReconciliation: jest.fn(),
|
||||
};
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-after-conflict',
|
||||
event: { id: 'event-after-conflict' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('blocked on commit_conflict reconciliation');
|
||||
expect(mockedGetCheckpointer).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('refuses a cold start after its bound child disappeared', async () => {
|
||||
const dependencies = {
|
||||
getSnapshot: jest.fn(async () => undefined),
|
||||
commitState: jest.fn(),
|
||||
recordReconciliation: jest.fn(),
|
||||
resolveReconciliation: jest.fn(),
|
||||
};
|
||||
|
||||
await expect(
|
||||
executeAgentEventActor(
|
||||
{
|
||||
user: 'user-1',
|
||||
conversationId,
|
||||
invocationId: 'event-after-delete',
|
||||
event: { id: 'event-after-delete' },
|
||||
signal: new AbortController().signal,
|
||||
legacyTurnStaleMs: 60_000,
|
||||
invoke: async () => 'response',
|
||||
readAppliedAction: () => ({ toolName: 'submit_move' }),
|
||||
},
|
||||
dependencies,
|
||||
),
|
||||
).rejects.toThrow('Event actor binding is no longer active');
|
||||
expect(mockedGetCheckpointer).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
450
packages/api/src/agents/triggers/actor.ts
Normal file
450
packages/api/src/agents/triggers/actor.ts
Normal file
|
|
@ -0,0 +1,450 @@
|
|||
import { logger } from '@librechat/data-schemas';
|
||||
import {
|
||||
createEventActorExecutor,
|
||||
type EventActorEvent,
|
||||
type EventActorExecutionResult,
|
||||
type EventActorHead,
|
||||
type EventActorHostAdapter,
|
||||
} from '@librechat/agents';
|
||||
import type { ConversationMethods, IAgentEventActorState } from '@librechat/data-schemas';
|
||||
import type { TCheckpointerConfig } from 'librechat-data-provider';
|
||||
import type { AgentTriggerExpectedAction } from './envelope';
|
||||
import type { AgentEventAppliedAction } from './outcome';
|
||||
import {
|
||||
captureAgentEventCheckpoint,
|
||||
deleteAgentCheckpoint,
|
||||
forkAgentEventCheckpoint,
|
||||
getAgentCheckpointer,
|
||||
getApprovalTtlMs,
|
||||
} from '../checkpointer';
|
||||
|
||||
interface EventActorResult extends Record<string, EventActorEvent> {
|
||||
action: AgentEventAppliedAction;
|
||||
checkpointCaptureError: string | null;
|
||||
}
|
||||
|
||||
export interface AgentEventActorInvocationContext {
|
||||
checkpointNamespace: string;
|
||||
checkpointId?: string;
|
||||
invocationId: string;
|
||||
continuation: 'warm' | 'cold';
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
export interface ExecuteAgentEventActorInput<T> {
|
||||
user: string;
|
||||
tenantId?: string;
|
||||
conversationId: string;
|
||||
invocationId: string;
|
||||
event: EventActorEvent;
|
||||
expectedAction?: AgentTriggerExpectedAction;
|
||||
signal: AbortSignal;
|
||||
checkpointer?: TCheckpointerConfig;
|
||||
/** Deprecated compatibility input. Elapsed time never proves replay safety. */
|
||||
legacyTurnStaleMs?: number;
|
||||
invoke(context: AgentEventActorInvocationContext): Promise<T>;
|
||||
readAppliedAction(): AgentEventAppliedAction | undefined;
|
||||
}
|
||||
|
||||
export interface ExecuteAgentEventActorResult<T> {
|
||||
value: T;
|
||||
execution: EventActorExecutionResult<EventActorResult>;
|
||||
}
|
||||
|
||||
export interface AgentEventActorDependencies {
|
||||
getSnapshot: ConversationMethods['getAgentEventActorSnapshot'];
|
||||
commitState: ConversationMethods['commitAgentEventActorState'];
|
||||
recordReconciliation: ConversationMethods['recordAgentEventActorReconciliation'];
|
||||
resolveReconciliation: ConversationMethods['resolveAgentEventActorReconciliation'];
|
||||
}
|
||||
|
||||
function toHead(actorThreadId: string, state: IAgentEventActorState | null): EventActorHead {
|
||||
return state == null
|
||||
? { actorThreadId, generation: 0 }
|
||||
: { actorThreadId, generation: state.generation, checkpoint: state.checkpoint };
|
||||
}
|
||||
|
||||
function asError(value: unknown): Error {
|
||||
return value instanceof Error ? value : new Error(String(value));
|
||||
}
|
||||
|
||||
function checkpointMatches(
|
||||
state: IAgentEventActorState,
|
||||
checkpoint: { threadId: string; checkpointId?: string; checkpointNs: string },
|
||||
): boolean {
|
||||
return (
|
||||
typeof checkpoint.checkpointId === 'string' &&
|
||||
state.checkpoint.threadId === checkpoint.threadId &&
|
||||
state.checkpoint.checkpointId === checkpoint.checkpointId &&
|
||||
state.checkpoint.checkpointNs === checkpoint.checkpointNs
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Executes one authenticated bound-child event through the SDK's checkpoint-fork lifecycle.
|
||||
* The request controller still owns generation admission and terminal receipts; this adapter owns
|
||||
* only checkpoint preparation, invocation isolation, CAS commit, and bounded cleanup.
|
||||
*/
|
||||
export async function executeAgentEventActor<T>(
|
||||
input: ExecuteAgentEventActorInput<T>,
|
||||
deps: AgentEventActorDependencies,
|
||||
): Promise<ExecuteAgentEventActorResult<T>> {
|
||||
let value: T | undefined;
|
||||
let invocationError: unknown;
|
||||
let observedState: IAgentEventActorState | null | undefined;
|
||||
let observedEpoch: number | undefined;
|
||||
const adapter: EventActorHostAdapter<EventActorEvent, EventActorResult> = {
|
||||
async prepare(request, context) {
|
||||
if (context.signal.aborted) {
|
||||
throw context.signal.reason;
|
||||
}
|
||||
const snapshot = await deps.getSnapshot({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
});
|
||||
if (snapshot === undefined) {
|
||||
throw new Error('Event actor binding is no longer active');
|
||||
}
|
||||
const unresolved = snapshot.reconciliations.filter((item) => item.status !== 'settled');
|
||||
if (unresolved.length > 0) {
|
||||
throw new Error(
|
||||
`Event actor is blocked on ${unresolved.map((item) => item.status).join(', ')} reconciliation`,
|
||||
);
|
||||
}
|
||||
/** A legacy turn is or may have been mid-flight: its external action and
|
||||
* durable-history outcome are unknown, so no amount of elapsed time can
|
||||
* prove that replay is safe. Keep the fence closed until a terminal owner
|
||||
* proves persistence and seals its exact token, or an operator performs
|
||||
* an explicit reconciliation. */
|
||||
const legacyTurn = snapshot.legacyTurn;
|
||||
if (legacyTurn != null) {
|
||||
throw new Error('Event actor is blocked on an in-flight legacy turn');
|
||||
}
|
||||
const state = snapshot.state;
|
||||
observedState = state;
|
||||
observedEpoch = snapshot.epoch;
|
||||
const head = toHead(input.conversationId, state);
|
||||
if (state == null || state.requiresColdStart === true) {
|
||||
return { status: 'checkpoint_unavailable', head };
|
||||
}
|
||||
const fork = await forkAgentEventCheckpoint(
|
||||
state.checkpoint,
|
||||
request.checkpointNs,
|
||||
request.invocationId,
|
||||
input.checkpointer,
|
||||
);
|
||||
if (fork == null) {
|
||||
return { status: 'checkpoint_unavailable', head };
|
||||
}
|
||||
return {
|
||||
status: 'ready',
|
||||
invocation: {
|
||||
...request,
|
||||
continuation: 'warm',
|
||||
base: head,
|
||||
fork: { ...fork, invocationId: request.invocationId },
|
||||
},
|
||||
};
|
||||
},
|
||||
async coldContinue(request, head, context) {
|
||||
if (context.signal.aborted) {
|
||||
throw context.signal.reason;
|
||||
}
|
||||
if (!(await getAgentCheckpointer(input.checkpointer))) {
|
||||
throw new Error('Event actor checkpoint forks require a durable Mongo checkpointer');
|
||||
}
|
||||
return {
|
||||
...request,
|
||||
continuation: 'cold',
|
||||
base: head,
|
||||
fork: {
|
||||
threadId: input.conversationId,
|
||||
checkpointNs: request.checkpointNs,
|
||||
...(head.checkpoint == null ? {} : { checkpointId: head.checkpoint.checkpointId }),
|
||||
invocationId: request.invocationId,
|
||||
},
|
||||
};
|
||||
},
|
||||
async invoke(invocation, context) {
|
||||
const fenced = await deps.recordReconciliation({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
reconciliation: {
|
||||
invocationId: invocation.invocationId,
|
||||
status: 'invocation_pending',
|
||||
checkpoint: {
|
||||
threadId: invocation.fork.threadId,
|
||||
checkpointNs: invocation.fork.checkpointNs,
|
||||
...(invocation.fork.checkpointId == null
|
||||
? {}
|
||||
: { checkpointId: invocation.fork.checkpointId }),
|
||||
},
|
||||
action: { toolName: input.expectedAction?.toolName ?? 'expected_action' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
});
|
||||
if (!fenced) {
|
||||
throw new Error('Event actor invocation could not acquire its durable lifecycle fence');
|
||||
}
|
||||
try {
|
||||
value = await input.invoke({
|
||||
checkpointNamespace: invocation.fork.checkpointNs,
|
||||
...(invocation.fork.checkpointId == null
|
||||
? {}
|
||||
: { checkpointId: invocation.fork.checkpointId }),
|
||||
invocationId: invocation.invocationId,
|
||||
continuation: invocation.continuation,
|
||||
signal: context.signal,
|
||||
});
|
||||
} catch (error) {
|
||||
invocationError = error;
|
||||
}
|
||||
const action = input.readAppliedAction();
|
||||
if (action == null) {
|
||||
if (invocationError != null) {
|
||||
throw invocationError;
|
||||
}
|
||||
return { status: 'completed_no_action' };
|
||||
}
|
||||
let checkpoint: Awaited<ReturnType<typeof captureAgentEventCheckpoint>>;
|
||||
try {
|
||||
checkpoint = await captureAgentEventCheckpoint(
|
||||
input.conversationId,
|
||||
invocation.fork.checkpointNs,
|
||||
invocation.invocationId,
|
||||
input.checkpointer,
|
||||
);
|
||||
} catch (error) {
|
||||
return {
|
||||
status: 'applied',
|
||||
result: { action, checkpointCaptureError: asError(error).message },
|
||||
checkpoint: invocation.fork,
|
||||
};
|
||||
}
|
||||
if (checkpoint == null) {
|
||||
return {
|
||||
status: 'applied',
|
||||
result: {
|
||||
action,
|
||||
checkpointCaptureError: 'Applied turn has no observable terminal checkpoint',
|
||||
},
|
||||
checkpoint: invocation.fork,
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: 'applied',
|
||||
result: { action, checkpointCaptureError: null },
|
||||
checkpoint: { ...checkpoint, invocationId: invocation.invocationId },
|
||||
};
|
||||
},
|
||||
async commit(request) {
|
||||
if (request.result.checkpointCaptureError != null) {
|
||||
throw new Error(request.result.checkpointCaptureError);
|
||||
}
|
||||
const expectedCheckpointId = request.expectedHead.checkpoint?.checkpointId;
|
||||
if (
|
||||
request.expectedHead.checkpoint != null &&
|
||||
(typeof expectedCheckpointId !== 'string' || expectedCheckpointId.length === 0)
|
||||
) {
|
||||
throw new Error('Event actor head is missing its checkpoint id');
|
||||
}
|
||||
const appliedCheckpointId = request.checkpoint.checkpointId;
|
||||
if (typeof appliedCheckpointId !== 'string' || appliedCheckpointId.length === 0) {
|
||||
throw new Error('Applied event actor checkpoint is missing its id');
|
||||
}
|
||||
if (observedState === undefined || observedEpoch === undefined) {
|
||||
throw new Error('Event actor commit is missing its prepared host state');
|
||||
}
|
||||
const expectedHeadCheckpoint = request.expectedHead.checkpoint;
|
||||
if (observedState != null && expectedHeadCheckpoint == null) {
|
||||
throw new Error('Event actor commit lost its prepared checkpoint head');
|
||||
}
|
||||
/** The SDK head intentionally contains only portable checkpoint identity.
|
||||
* Retain the host-private cold-start observation from prepare so the CAS
|
||||
* cannot clear a legacy-path invalidation that races before acquisition. */
|
||||
const expected =
|
||||
observedState == null
|
||||
? undefined
|
||||
: {
|
||||
generation: request.expectedHead.generation,
|
||||
checkpoint: {
|
||||
threadId: expectedHeadCheckpoint!.threadId,
|
||||
checkpointId: expectedCheckpointId!,
|
||||
checkpointNs: expectedHeadCheckpoint!.checkpointNs,
|
||||
},
|
||||
...(observedState.requiresColdStart === true ? { requiresColdStart: true } : {}),
|
||||
};
|
||||
const committed = await deps.commitState({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
invocationId: request.invocation.invocationId,
|
||||
action: request.result.action,
|
||||
...(expected == null ? {} : { expected }),
|
||||
/** Legacy-path invalidations against headless or already cold-marked
|
||||
* actors are visible ONLY through the epoch; the CAS must require the
|
||||
* exact epoch observed at preparation. */
|
||||
expectedEpoch: observedEpoch,
|
||||
checkpoint: {
|
||||
threadId: request.checkpoint.threadId,
|
||||
checkpointId: appliedCheckpointId,
|
||||
checkpointNs: request.checkpoint.checkpointNs,
|
||||
},
|
||||
});
|
||||
if (committed.status === 'stale') {
|
||||
/** A host-private cold marker can invalidate the CAS without advancing
|
||||
* the portable SDK head. Omit that non-advanced head so the SDK reports
|
||||
* an ordinary conflict instead of misclassifying it as indeterminate. */
|
||||
const advanced =
|
||||
committed.state != null && committed.state.generation > request.expectedHead.generation;
|
||||
return {
|
||||
status: 'stale',
|
||||
...(advanced ? { head: toHead(input.conversationId, committed.state!) } : {}),
|
||||
};
|
||||
}
|
||||
if (committed.prunableCheckpoint != null) {
|
||||
await deleteAgentCheckpoint(
|
||||
committed.prunableCheckpoint.threadId,
|
||||
input.checkpointer,
|
||||
undefined,
|
||||
{
|
||||
throwOnError: true,
|
||||
checkpointNamespace: committed.prunableCheckpoint.checkpointNs,
|
||||
},
|
||||
);
|
||||
}
|
||||
return { status: 'committed', head: toHead(input.conversationId, committed.state) };
|
||||
},
|
||||
async discard(request) {
|
||||
await deleteAgentCheckpoint(request.invocation.fork.threadId, input.checkpointer, undefined, {
|
||||
throwOnError: true,
|
||||
checkpointNamespace: request.invocation.fork.checkpointNs,
|
||||
});
|
||||
const released = await deps.resolveReconciliation({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
invocationId: request.invocation.invocationId,
|
||||
checkpoint: {
|
||||
threadId: request.invocation.fork.threadId,
|
||||
checkpointNs: request.invocation.fork.checkpointNs,
|
||||
...(request.invocation.fork.checkpointId == null
|
||||
? {}
|
||||
: { checkpointId: request.invocation.fork.checkpointId }),
|
||||
},
|
||||
resolution: 'invocation_abandoned',
|
||||
});
|
||||
if (!released) {
|
||||
const snapshot = await deps.getSnapshot({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
});
|
||||
if (
|
||||
snapshot?.reconciliations.some(
|
||||
(item) => item.invocationId === request.invocation.invocationId,
|
||||
) === true
|
||||
) {
|
||||
throw new Error('Event actor invocation lifecycle fence could not be released');
|
||||
}
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
const executor = createEventActorExecutor(adapter, {
|
||||
maxDepth: 1,
|
||||
dormantCheckpointTtlMs: getApprovalTtlMs(input.checkpointer),
|
||||
});
|
||||
let execution: EventActorExecutionResult<EventActorResult> = await executor.execute({
|
||||
actorThreadId: input.conversationId,
|
||||
invocationId: input.invocationId,
|
||||
event: input.event,
|
||||
depth: 1,
|
||||
signal: input.signal,
|
||||
});
|
||||
if (execution.status === 'failed') {
|
||||
throw execution.error;
|
||||
}
|
||||
if (execution.status === 'cancelled') {
|
||||
throw asError(input.signal.reason ?? 'Event actor invocation cancelled');
|
||||
}
|
||||
if (
|
||||
execution.status === 'commit_indeterminate' &&
|
||||
typeof execution.checkpoint.checkpointId === 'string'
|
||||
) {
|
||||
let snapshot: Awaited<ReturnType<AgentEventActorDependencies['getSnapshot']>>;
|
||||
try {
|
||||
snapshot = await deps.getSnapshot({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
});
|
||||
} catch (error) {
|
||||
logger.warn('[event-actor] Could not verify an indeterminate checkpoint commit', {
|
||||
conversationId: input.conversationId,
|
||||
invocationId: input.invocationId,
|
||||
error: asError(error).message,
|
||||
});
|
||||
}
|
||||
if (
|
||||
snapshot?.reconciliations.some(
|
||||
(item) => item.invocationId === input.invocationId && item.status === 'persistence_pending',
|
||||
) === true &&
|
||||
snapshot?.state != null &&
|
||||
checkpointMatches(snapshot.state, execution.checkpoint) &&
|
||||
execution.result != null
|
||||
) {
|
||||
execution = {
|
||||
status: 'applied',
|
||||
result: execution.result,
|
||||
head: toHead(input.conversationId, snapshot.state),
|
||||
continuation: execution.continuation,
|
||||
};
|
||||
}
|
||||
}
|
||||
if (execution.status === 'commit_conflict' || execution.status === 'commit_indeterminate') {
|
||||
const action = execution.result?.action ?? input.readAppliedAction();
|
||||
if (action == null) {
|
||||
throw new Error(`Event actor ${execution.status} did not retain applied-action evidence`);
|
||||
}
|
||||
const error =
|
||||
execution.status === 'commit_indeterminate'
|
||||
? execution.error.message
|
||||
: 'A competing checkpoint advanced the actor head';
|
||||
const recorded = await deps.recordReconciliation({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
|
||||
reconciliation: {
|
||||
invocationId: input.invocationId,
|
||||
status: execution.status,
|
||||
checkpoint: {
|
||||
threadId: execution.checkpoint.threadId,
|
||||
checkpointNs: execution.checkpoint.checkpointNs,
|
||||
...(execution.checkpoint.checkpointId == null
|
||||
? {}
|
||||
: { checkpointId: execution.checkpoint.checkpointId }),
|
||||
},
|
||||
action,
|
||||
error: error.slice(0, 1024),
|
||||
observedAt: new Date(),
|
||||
},
|
||||
});
|
||||
if (!recorded) {
|
||||
throw new Error(`Failed to persist event actor ${execution.status} reconciliation`);
|
||||
}
|
||||
logger.error('[event-actor] Applied action blocked the actor pending reconciliation', {
|
||||
conversationId: input.conversationId,
|
||||
invocationId: input.invocationId,
|
||||
status: execution.status,
|
||||
error,
|
||||
});
|
||||
throw new Error(`Event actor action requires ${execution.status} reconciliation`);
|
||||
}
|
||||
if (invocationError != null) {
|
||||
throw invocationError;
|
||||
}
|
||||
return { value: value as T, execution };
|
||||
}
|
||||
|
|
@ -582,6 +582,7 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
|
|||
}
|
||||
const envelope = {
|
||||
...base,
|
||||
event: { ...base.event, payload: { blob: 'x'.repeat(4096) } },
|
||||
expectedAction: {
|
||||
toolName: 'submit_move',
|
||||
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
|
||||
|
|
@ -605,12 +606,23 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
|
|||
const headers = fetcher.mock.calls[0][1]?.headers as Record<string, string>;
|
||||
expect(headers['x-lc-agent-event-binding']).toBe(`evtbind_${'a'.repeat(48)}`);
|
||||
expect(headers['x-lc-agent-event-source-key']).toBe('source-key');
|
||||
/** The unbounded source payload never rides the delivery body; the actor
|
||||
* binds an invocation from event identity alone. */
|
||||
expect(JSON.parse(String(fetcher.mock.calls[0][1]?.body))).toMatchObject({
|
||||
agentEventDelivery: {
|
||||
deliveryKey: getAgentTriggerIdempotencyKey(envelope),
|
||||
event: {
|
||||
id: envelope.event.id,
|
||||
type: envelope.event.type,
|
||||
occurredAt: envelope.event.occurredAt,
|
||||
source: envelope.event.source,
|
||||
},
|
||||
expectedAction: envelope.expectedAction,
|
||||
},
|
||||
});
|
||||
expect(
|
||||
JSON.parse(String(fetcher.mock.calls[0][1]?.body)).agentEventDelivery.event,
|
||||
).not.toHaveProperty('payload');
|
||||
});
|
||||
|
||||
it.each(['PARENT_NOT_READY', 'EVENT_ACTOR_NOT_READY'])(
|
||||
|
|
|
|||
|
|
@ -619,6 +619,16 @@ async function startRun(
|
|||
envelope.target.bindingId != null && {
|
||||
agentEventDelivery: {
|
||||
deliveryKey: context.idempotencyKey,
|
||||
/** Identity only, matching the `fire` body: the actor uses this
|
||||
* to bind an invocation, never to build the turn's prompt. The
|
||||
* source payload is unbounded and would push large deliveries
|
||||
* past the chat route's body limit. */
|
||||
event: {
|
||||
id: envelope.event.id,
|
||||
type: envelope.event.type,
|
||||
occurredAt: envelope.event.occurredAt,
|
||||
source: envelope.event.source,
|
||||
},
|
||||
...(envelope.expectedAction != null && {
|
||||
expectedAction: envelope.expectedAction,
|
||||
}),
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
export * from './dispatch';
|
||||
export * from './bindings';
|
||||
export * from './actor';
|
||||
export * from './bindingResolver';
|
||||
export * from './delivery';
|
||||
export * from './envelope';
|
||||
|
|
|
|||
|
|
@ -1,7 +1,25 @@
|
|||
import { StepTypes } from 'librechat-data-provider';
|
||||
import type { Agents } from 'librechat-data-provider';
|
||||
import type { SerializableJobData } from '~/stream';
|
||||
import { createAgentEventTerminalHandler } from './outcome';
|
||||
import {
|
||||
createAgentEventTerminalHandler as createAgentEventTerminalHandlerImpl,
|
||||
createAgentEventActionRecorder,
|
||||
} from './outcome';
|
||||
|
||||
const createAgentEventTerminalHandler = (
|
||||
methods: Pick<
|
||||
Parameters<typeof createAgentEventTerminalHandlerImpl>[0],
|
||||
'settleAgentTriggerHandlingOutcome'
|
||||
> &
|
||||
Partial<Parameters<typeof createAgentEventTerminalHandlerImpl>[0]>,
|
||||
) =>
|
||||
createAgentEventTerminalHandlerImpl({
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue(undefined),
|
||||
getMessage: jest.fn().mockResolvedValue(null),
|
||||
resolveAgentEventActorReconciliation: jest.fn().mockResolvedValue(true),
|
||||
completeAgentEventActorLegacyTurn: jest.fn().mockResolvedValue(true),
|
||||
...methods,
|
||||
});
|
||||
|
||||
function job(overrides: Partial<SerializableJobData> = {}): SerializableJobData {
|
||||
return {
|
||||
|
|
@ -83,6 +101,100 @@ describe('agent event terminal outcomes', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('seals a persisted legacy turn from a non-resume terminal owner', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const completeAgentEventActorLegacyTurn = jest.fn().mockResolvedValue(true);
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: null,
|
||||
reconciliations: [],
|
||||
legacyTurn: { token: 'legacy-terminal-token', startedAt: new Date() },
|
||||
}),
|
||||
getMessage: jest.fn().mockImplementation(({ messageId }) =>
|
||||
Promise.resolve(
|
||||
messageId.endsWith(':user')
|
||||
? {
|
||||
messageId,
|
||||
conversationId: 'conversation-1',
|
||||
isCreatedByUser: true,
|
||||
}
|
||||
: {
|
||||
messageId,
|
||||
conversationId: 'conversation-1',
|
||||
parentMessageId: 'trigger_1:user',
|
||||
isCreatedByUser: false,
|
||||
},
|
||||
),
|
||||
),
|
||||
});
|
||||
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({ status: 'aborted', agentEventLegacyTurnToken: 'legacy-terminal-token' }),
|
||||
[],
|
||||
);
|
||||
|
||||
expect(completeAgentEventActorLegacyTurn).toHaveBeenCalledWith({
|
||||
user: 'user-1',
|
||||
conversationId: 'conversation-1',
|
||||
token: 'legacy-terminal-token',
|
||||
});
|
||||
expect(completeAgentEventActorLegacyTurn.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
settleAgentTriggerHandlingOutcome.mock.invocationCallOrder[0],
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps a terminal legacy fence closed when required message history is missing', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const completeAgentEventActorLegacyTurn = jest.fn().mockResolvedValue(true);
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: null,
|
||||
reconciliations: [],
|
||||
legacyTurn: { token: 'legacy-terminal-token', startedAt: new Date() },
|
||||
}),
|
||||
getMessage: jest.fn().mockResolvedValue(null),
|
||||
});
|
||||
|
||||
await expect(
|
||||
handler(
|
||||
'conversation-1',
|
||||
job({ status: 'aborted', agentEventLegacyTurnToken: 'legacy-terminal-token' }),
|
||||
[],
|
||||
),
|
||||
).rejects.toThrow('invalid durable message history');
|
||||
|
||||
expect(completeAgentEventActorLegacyTurn).not.toHaveBeenCalled();
|
||||
expect(settleAgentTriggerHandlingOutcome).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('replays terminal settlement after the same legacy token was already sealed', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const completeAgentEventActorLegacyTurn = jest.fn().mockResolvedValue(true);
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: null,
|
||||
reconciliations: [],
|
||||
legacyTurn: null,
|
||||
}),
|
||||
});
|
||||
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({ status: 'aborted', agentEventLegacyTurnToken: 'legacy-terminal-token' }),
|
||||
[],
|
||||
);
|
||||
|
||||
expect(completeAgentEventActorLegacyTurn).not.toHaveBeenCalled();
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('matches nested action arrays structurally', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });
|
||||
|
|
@ -402,4 +514,487 @@ describe('agent event terminal outcomes', () => {
|
|||
expect.objectContaining({ status: 'cancelled' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('defers terminal settlement until the post-commit history barrier is durable', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const getMessage = jest.fn().mockResolvedValue(null);
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: {
|
||||
generation: 1,
|
||||
checkpoint: {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
},
|
||||
},
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'persistence_pending',
|
||||
checkpoint: {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
},
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
getMessage,
|
||||
resolveAgentEventActorReconciliation: jest.fn().mockResolvedValue(true),
|
||||
});
|
||||
|
||||
await expect(
|
||||
handler('conversation-1', job({ agentEventExpectedAction: { toolName: 'submit_move' } }), [
|
||||
completedToolStep(),
|
||||
]),
|
||||
).rejects.toThrow('requires persistence_pending reconciliation');
|
||||
expect(getMessage).not.toHaveBeenCalled();
|
||||
expect(settleAgentTriggerHandlingOutcome).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('repairs a lost actor acknowledgement from deterministic durable messages before settling', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: { generation: 1, checkpoint },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'history_persisted',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
getMessage: jest.fn(async ({ messageId }) => {
|
||||
const isUser = messageId.endsWith(':user');
|
||||
return {
|
||||
messageId,
|
||||
conversationId: 'conversation-1',
|
||||
isCreatedByUser: isUser,
|
||||
parentMessageId: isUser ? 'parent-message' : 'trigger_1:user',
|
||||
} as never;
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({
|
||||
status: 'error',
|
||||
error: 'run evidence was lost after the actor commit',
|
||||
agentEventExpectedAction: { toolName: 'submit_move' },
|
||||
}),
|
||||
[],
|
||||
);
|
||||
|
||||
expect(resolveAgentEventActorReconciliation).toHaveBeenCalledWith({
|
||||
user: 'user-1',
|
||||
conversationId: 'conversation-1',
|
||||
invocationId: 'trigger_1',
|
||||
checkpoint,
|
||||
resolution: 'checkpoint_verified',
|
||||
});
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
status: 'applied',
|
||||
action: { toolName: 'submit_move' },
|
||||
}),
|
||||
);
|
||||
expect(settleAgentTriggerHandlingOutcome.mock.calls[0][0]).not.toHaveProperty('error');
|
||||
/** The receipt's status CAS is the serialization point against concurrent
|
||||
* compensation, so verification must resolve BEFORE the public settle. */
|
||||
expect(resolveAgentEventActorReconciliation.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
settleAgentTriggerHandlingOutcome.mock.invocationCallOrder[0],
|
||||
);
|
||||
});
|
||||
|
||||
it('retains authoritative action proof until a failed settlement retry succeeds', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce(false)
|
||||
.mockResolvedValueOnce(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const action = { toolName: 'submit_move', toolCallId: 'call-1' };
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
/** The first attempt resolves the receipt but dies on the settle write;
|
||||
* the retry then observes the already-settled verified receipt. */
|
||||
getAgentEventActorSnapshot: jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
state: { generation: 1, checkpoint },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'history_persisted',
|
||||
checkpoint,
|
||||
action,
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
})
|
||||
.mockResolvedValue({
|
||||
state: { generation: 1, checkpoint },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'settled',
|
||||
resolution: 'checkpoint_verified',
|
||||
checkpoint,
|
||||
action,
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
getMessage: jest.fn(async ({ messageId }) => {
|
||||
const isUser = messageId.endsWith(':user');
|
||||
return {
|
||||
messageId,
|
||||
conversationId: 'conversation-1',
|
||||
isCreatedByUser: isUser,
|
||||
parentMessageId: isUser ? 'parent-message' : 'trigger_1:user',
|
||||
} as never;
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
await expect(
|
||||
handler('conversation-1', job({ status: 'error', error: 'lost run evidence' }), []),
|
||||
).rejects.toThrow('Failed to settle agent event delivery trigger_1');
|
||||
expect(resolveAgentEventActorReconciliation).toHaveBeenCalledTimes(1);
|
||||
|
||||
await handler('conversation-1', job({ status: 'error', error: 'lost run evidence' }), []);
|
||||
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
expect.objectContaining({
|
||||
status: 'applied',
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-1' },
|
||||
}),
|
||||
);
|
||||
expect(resolveAgentEventActorReconciliation).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('honors a compensation that wins the receipt CAS during settlement', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(false);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const action = { toolName: 'submit_move', toolCallId: 'call-1' };
|
||||
const getAgentEventActorSnapshot = jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
state: { generation: 1, checkpoint },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'history_persisted',
|
||||
checkpoint,
|
||||
action,
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
})
|
||||
.mockResolvedValue({
|
||||
state: { generation: 1, checkpoint, requiresColdStart: true },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'settled',
|
||||
resolution: 'action_compensated',
|
||||
checkpoint,
|
||||
action,
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
});
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot,
|
||||
getMessage: jest.fn(async ({ messageId }) => {
|
||||
const isUser = messageId.endsWith(':user');
|
||||
return {
|
||||
messageId,
|
||||
conversationId: 'conversation-1',
|
||||
isCreatedByUser: isUser,
|
||||
parentMessageId: isUser ? 'parent-message' : 'trigger_1:user',
|
||||
} as never;
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({ agentEventExpectedAction: { toolName: 'submit_move' } }),
|
||||
[completedToolStep()],
|
||||
);
|
||||
|
||||
/** Verification lost the receipt CAS to a concurrent compensation, so the
|
||||
* public outcome must honor the compensation, not the stale snapshot. */
|
||||
expect(getAgentEventActorSnapshot).toHaveBeenCalledTimes(2);
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
status: 'failed',
|
||||
error: 'Applied event actor action was explicitly compensated',
|
||||
}),
|
||||
);
|
||||
expect(settleAgentTriggerHandlingOutcome.mock.calls[0][0]).not.toHaveProperty('action');
|
||||
});
|
||||
|
||||
it('replays an applied settlement from its durable resolved lifecycle receipt', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: { generation: 1, checkpoint },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'settled',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-1' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({ status: 'error', error: 'lost local evidence after settlement' }),
|
||||
[],
|
||||
);
|
||||
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
status: 'applied',
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-1' },
|
||||
}),
|
||||
);
|
||||
expect(resolveAgentEventActorReconciliation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('never replays a compensated receipt as applied even over fresh run evidence', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointId: 'checkpoint-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: { generation: 1, checkpoint, requiresColdStart: true },
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'settled',
|
||||
resolution: 'action_compensated',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move', toolCallId: 'call-1' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
/** The replayed generation still carries the original applied run step,
|
||||
* but compensation explicitly undid that effect: the public outcome must
|
||||
* not tell the source the operation stands. */
|
||||
await handler(
|
||||
'conversation-1',
|
||||
job({ status: 'error', agentEventExpectedAction: { toolName: 'submit_move' } }),
|
||||
[completedToolStep()],
|
||||
);
|
||||
|
||||
expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
status: 'failed',
|
||||
error: 'Applied event actor action was explicitly compensated',
|
||||
}),
|
||||
);
|
||||
expect(settleAgentTriggerHandlingOutcome.mock.calls[0][0]).not.toHaveProperty('action');
|
||||
expect(resolveAgentEventActorReconciliation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('preserves an ambiguous pre-action fence when terminal evidence is incomplete', async () => {
|
||||
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
|
||||
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
|
||||
const checkpoint = {
|
||||
threadId: 'conversation-1',
|
||||
checkpointNs: 'event-actor/trigger_1',
|
||||
};
|
||||
const handler = createAgentEventTerminalHandler({
|
||||
settleAgentTriggerHandlingOutcome,
|
||||
getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
|
||||
state: null,
|
||||
reconciliations: [
|
||||
{
|
||||
invocationId: 'trigger_1',
|
||||
status: 'invocation_pending',
|
||||
checkpoint,
|
||||
action: { toolName: 'submit_move' },
|
||||
observedAt: new Date(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
resolveAgentEventActorReconciliation,
|
||||
});
|
||||
|
||||
await expect(handler('conversation-1', job(), [])).rejects.toThrow(
|
||||
'requires invocation_pending reconciliation',
|
||||
);
|
||||
|
||||
expect(resolveAgentEventActorReconciliation).not.toHaveBeenCalled();
|
||||
expect(settleAgentTriggerHandlingOutcome).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('createAgentEventActionRecorder', () => {
|
||||
const expectedAction = { toolName: 'submit_move', argumentSubset: { gameId: 'game-1' } };
|
||||
const successEnd = {
|
||||
input: { gameId: 'game-1', move: 'e4' },
|
||||
output: { name: 'submit_move', tool_call_id: 'call-1', content: '{"ok":true}' },
|
||||
};
|
||||
|
||||
it('records the first qualifying execution and keeps it', async () => {
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
recorder.observeToolEnd(successEnd);
|
||||
recorder.observeToolEnd({
|
||||
input: { gameId: 'game-1' },
|
||||
output: { name: 'submit_move', tool_call_id: 'call-2', content: '{"ok":true}' },
|
||||
});
|
||||
expect(recorder.read()).toEqual({ toolName: 'submit_move', toolCallId: 'call-1' });
|
||||
});
|
||||
|
||||
it('accepts the MCP-suffixed form of the expected tool', async () => {
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
recorder.observeToolEnd({
|
||||
...successEnd,
|
||||
output: { ...successEnd.output, name: 'submit_move_mcp_chess' },
|
||||
});
|
||||
expect(recorder.read()).toEqual({ toolName: 'submit_move_mcp_chess', toolCallId: 'call-1' });
|
||||
});
|
||||
|
||||
it('enforces the fenced argument subset against the execution input', async () => {
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
recorder.observeToolEnd({ ...successEnd, input: { gameId: 'other-game', move: 'e4' } });
|
||||
recorder.observeToolEnd({ ...successEnd, input: undefined });
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('never qualifies an argument-fenced action from an output-only tool end', async () => {
|
||||
/** Live-canary shape: the stream-consumer tool-end path delivers no
|
||||
* execution input, so a declared argument subset can never be verified —
|
||||
* the receipt must starve rather than trust an unfenced match. The
|
||||
* execution handler is required to supply the input (see handlers.spec). */
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
recorder.observeToolEnd({ output: successEnd.output });
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('qualifies a name-only expected action from an output-only tool end', async () => {
|
||||
const recorder = createAgentEventActionRecorder({ toolName: 'submit_move' });
|
||||
recorder.observeToolEnd({ output: successEnd.output });
|
||||
expect(recorder.read()).toEqual({ toolName: 'submit_move', toolCallId: 'call-1' });
|
||||
});
|
||||
|
||||
it('accepts policy-withheld output as proof of a successful foreground execution', async () => {
|
||||
/** Output filtering blanks the returned content AFTER the side effect
|
||||
* happened; reclassifying the turn as actionless would re-execute an
|
||||
* applied external action on retry. */
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
recorder.observeToolEnd({
|
||||
input: { gameId: 'game-1', move: 'e4' },
|
||||
outputFiltered: true,
|
||||
output: { name: 'submit_move', tool_call_id: 'call-filtered', content: '' },
|
||||
});
|
||||
expect(recorder.read()).toEqual({ toolName: 'submit_move', toolCallId: 'call-filtered' });
|
||||
});
|
||||
|
||||
it('never qualifies a withheld output whose call the model detached', async () => {
|
||||
const recorder = createAgentEventActionRecorder({ toolName: 'submit_move' });
|
||||
recorder.observeToolEnd({
|
||||
input: { gameId: 'game-1', run_in_background: true },
|
||||
outputFiltered: true,
|
||||
output: { name: 'submit_move', tool_call_id: 'call-detached', content: '' },
|
||||
});
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('never lets a background-task delivery impersonate a name-only action', async () => {
|
||||
/** The poll turn's delivery callback reports the ORIGINAL tool's name for
|
||||
* artifact attribution — evidence of work another turn dispatched, not
|
||||
* proof this invocation performed its expected action. */
|
||||
const recorder = createAgentEventActionRecorder({ toolName: 'submit_move' });
|
||||
recorder.observeToolEnd({
|
||||
input: { background_task_id: 'task-1' },
|
||||
backgroundDelivery: true,
|
||||
output: { name: 'submit_move', tool_call_id: 'call-poll', content: '{"ok":true}' },
|
||||
});
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('never records name mismatches, errored results, or malformed outputs', async () => {
|
||||
const recorder = createAgentEventActionRecorder(expectedAction);
|
||||
recorder.observeToolEnd({ ...successEnd, output: { ...successEnd.output, name: 'resign' } });
|
||||
recorder.observeToolEnd({
|
||||
...successEnd,
|
||||
output: { ...successEnd.output, status: 'error' },
|
||||
});
|
||||
recorder.observeToolEnd({ ...successEnd, output: { ...successEnd.output, content: null } });
|
||||
recorder.observeToolEnd({ ...successEnd, output: undefined });
|
||||
recorder.observeToolEnd({ input: successEnd.input, output: 'plain-string' });
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('excludes background non-execution receipts', async () => {
|
||||
const recorder = createAgentEventActionRecorder({ toolName: 'submit_move' });
|
||||
recorder.observeToolEnd({
|
||||
input: { gameId: 'game-1', run_in_background: true },
|
||||
output: {
|
||||
name: 'submit_move',
|
||||
tool_call_id: 'call-bg',
|
||||
content: JSON.stringify({ status: 'running', background_task_id: 'task-1' }),
|
||||
},
|
||||
});
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
|
||||
it('records nothing without a declared expected action', async () => {
|
||||
const recorder = createAgentEventActionRecorder(undefined);
|
||||
recorder.observeToolEnd(successEnd);
|
||||
expect(recorder.read()).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import type { ConversationMethods, MessageMethods } from '@librechat/data-schemas';
|
||||
import type { Agents } from 'librechat-data-provider';
|
||||
import type { AgentTriggerExpectedAction } from './envelope';
|
||||
import type { SerializableJobData } from '~/stream';
|
||||
|
|
@ -18,8 +19,34 @@ interface CompletedToolEvidence {
|
|||
arguments?: unknown;
|
||||
}
|
||||
|
||||
export interface AgentEventRunOutcome {
|
||||
status: 'applied' | 'completed_no_action' | 'failed' | 'cancelled';
|
||||
action?: { toolName: string; toolCallId?: string };
|
||||
}
|
||||
|
||||
export type AgentEventAppliedAction = NonNullable<AgentEventRunOutcome['action']>;
|
||||
|
||||
const MAX_RECEIPT_ID_LENGTH = 256;
|
||||
|
||||
async function hasDurableAgentEventHistory(input: {
|
||||
getMessage: MessageMethods['getMessage'];
|
||||
user: string;
|
||||
conversationId: string;
|
||||
deliveryKey: string;
|
||||
}): Promise<boolean> {
|
||||
const [userMessage, responseMessage] = await Promise.all([
|
||||
input.getMessage({ user: input.user, messageId: `${input.deliveryKey}:user` }),
|
||||
input.getMessage({ user: input.user, messageId: `${input.deliveryKey}:assistant` }),
|
||||
]);
|
||||
return (
|
||||
userMessage?.conversationId === input.conversationId &&
|
||||
userMessage.isCreatedByUser === true &&
|
||||
responseMessage?.conversationId === input.conversationId &&
|
||||
responseMessage.isCreatedByUser === false &&
|
||||
responseMessage.parentMessageId === userMessage.messageId
|
||||
);
|
||||
}
|
||||
|
||||
function isBackgroundNonExecutionReceipt(value: unknown, argumentsValue: unknown): boolean {
|
||||
const parsedArguments = parseArguments(argumentsValue);
|
||||
if (
|
||||
|
|
@ -96,7 +123,7 @@ function toolEvidence(
|
|||
}
|
||||
|
||||
function nonExecutedHITLToolCallIds(
|
||||
job: SerializableJobData,
|
||||
job: Pick<SerializableJobData, 'userSubmittedMessageFieldPaths'>,
|
||||
content: Agents.MessageContentComplex[],
|
||||
): Set<string> {
|
||||
const ids = new Set<string>();
|
||||
|
|
@ -129,6 +156,49 @@ function parseArguments(value: unknown): unknown {
|
|||
}
|
||||
}
|
||||
|
||||
/** Classifies terminal run evidence once for both checkpoint commit and public receipt. */
|
||||
export function classifyAgentEventRunOutcome(
|
||||
job: SerializableJobData,
|
||||
runSteps: Agents.RunStep[],
|
||||
content: Agents.MessageContentComplex[] = [],
|
||||
): AgentEventRunOutcome {
|
||||
const action = findAgentEventAppliedAction(job.agentEventExpectedAction, runSteps, content, job);
|
||||
if (action != null) {
|
||||
return { status: 'applied', action };
|
||||
}
|
||||
if (job.status === 'error') {
|
||||
return { status: 'failed' };
|
||||
}
|
||||
if (job.status === 'aborted') {
|
||||
return { status: 'cancelled' };
|
||||
}
|
||||
return { status: 'completed_no_action' };
|
||||
}
|
||||
|
||||
/** Finds qualifying action evidence without requiring the generation to be terminal yet. */
|
||||
export function findAgentEventAppliedAction(
|
||||
expectedAction: AgentTriggerExpectedAction | undefined,
|
||||
runSteps: Agents.RunStep[],
|
||||
content: Agents.MessageContentComplex[] = [],
|
||||
provenance: Pick<SerializableJobData, 'userSubmittedMessageFieldPaths'> = {},
|
||||
): AgentEventAppliedAction | undefined {
|
||||
if (expectedAction == null) {
|
||||
return undefined;
|
||||
}
|
||||
const nonExecutedToolCallIds = nonExecutedHITLToolCallIds(provenance, content);
|
||||
const action = runSteps
|
||||
.flatMap((step) => toolEvidence(step, nonExecutedToolCallIds))
|
||||
.find((item) => matchesExpectedAction(item, expectedAction));
|
||||
return action == null
|
||||
? undefined
|
||||
: {
|
||||
toolName: action.toolName.slice(0, MAX_RECEIPT_ID_LENGTH),
|
||||
...(action.toolCallId == null
|
||||
? {}
|
||||
: { toolCallId: action.toolCallId.slice(0, MAX_RECEIPT_ID_LENGTH) }),
|
||||
};
|
||||
}
|
||||
|
||||
function containsSubset(value: unknown, subset: unknown): boolean {
|
||||
if (Array.isArray(subset)) {
|
||||
return (
|
||||
|
|
@ -162,10 +232,107 @@ function matchesExpectedAction(
|
|||
);
|
||||
}
|
||||
|
||||
export interface AgentEventActionRecorder {
|
||||
observeToolEnd(data: {
|
||||
input?: unknown;
|
||||
backgroundDelivery?: boolean;
|
||||
outputFiltered?: boolean;
|
||||
output?: unknown;
|
||||
}): void;
|
||||
read(): AgentEventAppliedAction | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Captures qualifying applied-action evidence at tool-execution time, in graph
|
||||
* context, instead of trusting the asynchronously populated run-step
|
||||
* collection to be observable the instant `sendMessage` resolves. The recorder
|
||||
* applies the SAME fences as run-step evidence — exact tool name (with the MCP
|
||||
* suffix form), the declared argument subset, an error-free result, and the
|
||||
* background non-execution receipt exclusion. HITL never reaches the fork path
|
||||
* and non-executed approvals never emit a tool end, so the non-execution id
|
||||
* set has no equivalent here. Only the first qualifying execution is retained;
|
||||
* run-step evidence remains the fallback for paths that bypass the tool-end
|
||||
* chain (e.g. programmatic tool calling).
|
||||
*/
|
||||
export function createAgentEventActionRecorder(
|
||||
expectedAction: AgentTriggerExpectedAction | undefined,
|
||||
): AgentEventActionRecorder {
|
||||
let receipt: AgentEventAppliedAction | undefined;
|
||||
return {
|
||||
observeToolEnd(data) {
|
||||
if (expectedAction == null || receipt != null || data == null) {
|
||||
return;
|
||||
}
|
||||
/** A background-task delivery reports the ORIGINAL tool's name on a
|
||||
* later poll turn — evidence of work some earlier turn dispatched,
|
||||
* never proof that THIS invocation performed its action. */
|
||||
if (data.backgroundDelivery === true) {
|
||||
return;
|
||||
}
|
||||
/** Policy-withheld output is still proof of a successful foreground
|
||||
* execution — but with the content blank, a background launch handle
|
||||
* would be indistinguishable from a real result, so a call the model
|
||||
* detached can never qualify through this shape. */
|
||||
if (data.outputFiltered === true) {
|
||||
const parsedInput = parseArguments(data.input);
|
||||
if (
|
||||
parsedInput != null &&
|
||||
typeof parsedInput === 'object' &&
|
||||
!Array.isArray(parsedInput) &&
|
||||
(parsedInput as Record<string, unknown>).run_in_background === true
|
||||
) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
const output = data.output as
|
||||
| { name?: unknown; tool_call_id?: unknown; content?: unknown; status?: unknown }
|
||||
| null
|
||||
| undefined;
|
||||
if (
|
||||
output == null ||
|
||||
typeof output !== 'object' ||
|
||||
typeof output.name !== 'string' ||
|
||||
output.name.length === 0 ||
|
||||
output.content == null ||
|
||||
output.status === 'error'
|
||||
) {
|
||||
return;
|
||||
}
|
||||
if (
|
||||
typeof output.content === 'string' &&
|
||||
isBackgroundNonExecutionReceipt(output.content, data.input)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const toolCallId =
|
||||
typeof output.tool_call_id === 'string' && output.tool_call_id.length > 0
|
||||
? output.tool_call_id
|
||||
: undefined;
|
||||
const evidence: CompletedToolEvidence = {
|
||||
toolName: output.name,
|
||||
...(toolCallId == null ? {} : { toolCallId }),
|
||||
arguments: data.input,
|
||||
};
|
||||
if (!matchesExpectedAction(evidence, expectedAction)) {
|
||||
return;
|
||||
}
|
||||
receipt = {
|
||||
toolName: evidence.toolName.slice(0, MAX_RECEIPT_ID_LENGTH),
|
||||
...(toolCallId == null ? {} : { toolCallId: toolCallId.slice(0, MAX_RECEIPT_ID_LENGTH) }),
|
||||
};
|
||||
},
|
||||
read: () => receipt,
|
||||
};
|
||||
}
|
||||
|
||||
export function createAgentEventTerminalHandler(methods: {
|
||||
settleAgentTriggerHandlingOutcome: (
|
||||
input: SettleAgentTriggerHandlingOutcomeInput,
|
||||
) => Promise<boolean>;
|
||||
getAgentEventActorSnapshot: ConversationMethods['getAgentEventActorSnapshot'];
|
||||
resolveAgentEventActorReconciliation: ConversationMethods['resolveAgentEventActorReconciliation'];
|
||||
completeAgentEventActorLegacyTurn: ConversationMethods['completeAgentEventActorLegacyTurn'];
|
||||
getMessage: MessageMethods['getMessage'];
|
||||
}): (
|
||||
streamId: string,
|
||||
job: SerializableJobData,
|
||||
|
|
@ -181,36 +348,129 @@ export function createAgentEventTerminalHandler(methods: {
|
|||
if (job.agentEventDeliveryKey == null) {
|
||||
return;
|
||||
}
|
||||
const nonExecutedToolCallIds = nonExecutedHITLToolCallIds(job, content);
|
||||
const evidence = runSteps.flatMap((step) => toolEvidence(step, nonExecutedToolCallIds));
|
||||
const action =
|
||||
job.agentEventExpectedAction == null
|
||||
? undefined
|
||||
: evidence.find((item) => matchesExpectedAction(item, job.agentEventExpectedAction!));
|
||||
const settledAt = new Date(job.completedAt ?? Date.now());
|
||||
let status: SettleAgentTriggerHandlingOutcomeInput['status'] = 'completed_no_action';
|
||||
if (action != null) {
|
||||
status = 'applied';
|
||||
} else if (job.status === 'error') {
|
||||
status = 'failed';
|
||||
} else if (job.status === 'aborted') {
|
||||
status = 'cancelled';
|
||||
const conversationId = job.conversationId ?? streamId;
|
||||
const outcome = classifyAgentEventRunOutcome(job, runSteps, content);
|
||||
let committedAction: AgentEventAppliedAction | undefined;
|
||||
let compensated = false;
|
||||
const snapshot = await methods.getAgentEventActorSnapshot({
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
|
||||
});
|
||||
const lifecycle = snapshot?.reconciliations.find(
|
||||
(item) => item.invocationId === job.agentEventDeliveryKey,
|
||||
);
|
||||
if (lifecycle != null) {
|
||||
if (lifecycle.status === 'settled') {
|
||||
/** A compensated receipt still tombstones its invocation id, but its
|
||||
* external effect was explicitly undone — replaying it as applied
|
||||
* would tell the source the operation stands and suppress the
|
||||
* new-invocation retry that compensation requires. */
|
||||
if (lifecycle.resolution === 'action_compensated') {
|
||||
compensated = true;
|
||||
} else {
|
||||
committedAction = lifecycle.action;
|
||||
}
|
||||
} else if (lifecycle.status !== 'history_persisted') {
|
||||
throw new Error(
|
||||
`Agent event actor ${job.agentEventDeliveryKey} requires ${lifecycle.status} reconciliation`,
|
||||
);
|
||||
} else {
|
||||
const historyIsDurable = await hasDurableAgentEventHistory({
|
||||
getMessage: methods.getMessage,
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
deliveryKey: job.agentEventDeliveryKey,
|
||||
});
|
||||
if (!historyIsDurable) {
|
||||
throw new Error(
|
||||
`Agent event actor ${job.agentEventDeliveryKey} has invalid durable message history`,
|
||||
);
|
||||
}
|
||||
/** The persistence lifecycle was created by the same CAS that advanced
|
||||
* the actor head, and history_persisted is written only after the
|
||||
* controller's post-commit message barrier. Resolving BEFORE settlement
|
||||
* makes the receipt's status CAS the serialization point against a
|
||||
* concurrent compensation: whichever transition wins determines the
|
||||
* public outcome, and a crash between this resolve and the settle
|
||||
* below converges through the retained receipt's replay. The receipt
|
||||
* keeps its full action proof either way, so nothing is lost if the
|
||||
* settle write never lands. */
|
||||
const resolved = await methods.resolveAgentEventActorReconciliation({
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
|
||||
invocationId: lifecycle.invocationId,
|
||||
checkpoint: lifecycle.checkpoint,
|
||||
resolution: 'checkpoint_verified',
|
||||
});
|
||||
if (resolved) {
|
||||
committedAction = lifecycle.action;
|
||||
} else {
|
||||
const reread = await methods.getAgentEventActorSnapshot({
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
|
||||
});
|
||||
const raced = reread?.reconciliations.find(
|
||||
(item) => item.invocationId === job.agentEventDeliveryKey,
|
||||
);
|
||||
if (raced?.status === 'settled' && raced.resolution === 'action_compensated') {
|
||||
compensated = true;
|
||||
} else {
|
||||
throw new Error(
|
||||
`Agent event actor ${job.agentEventDeliveryKey} settlement receipt was not retained`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let settlementOutcome: AgentEventRunOutcome = outcome;
|
||||
if (compensated) {
|
||||
settlementOutcome = { status: 'failed' };
|
||||
} else if (committedAction != null) {
|
||||
settlementOutcome = { status: 'applied', action: committedAction };
|
||||
}
|
||||
if (job.agentEventLegacyTurnToken != null && snapshot?.legacyTurn != null) {
|
||||
if (snapshot.legacyTurn.token !== job.agentEventLegacyTurnToken) {
|
||||
throw new Error(
|
||||
`Legacy event actor turn ${job.agentEventLegacyTurnToken} lost token ownership`,
|
||||
);
|
||||
}
|
||||
const historyIsDurable = await hasDurableAgentEventHistory({
|
||||
getMessage: methods.getMessage,
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
deliveryKey: job.agentEventDeliveryKey,
|
||||
});
|
||||
if (!historyIsDurable) {
|
||||
throw new Error(
|
||||
`Legacy event actor ${job.agentEventDeliveryKey} has invalid durable message history`,
|
||||
);
|
||||
}
|
||||
const sealed = await methods.completeAgentEventActorLegacyTurn({
|
||||
user: job.userId,
|
||||
conversationId,
|
||||
...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
|
||||
token: job.agentEventLegacyTurnToken,
|
||||
});
|
||||
if (!sealed) {
|
||||
throw new Error(`Failed to seal legacy event actor turn ${job.agentEventLegacyTurnToken}`);
|
||||
}
|
||||
}
|
||||
const settledAt = new Date(job.completedAt ?? Date.now());
|
||||
const settled = await methods.settleAgentTriggerHandlingOutcome({
|
||||
deliveryKey: job.agentEventDeliveryKey,
|
||||
conversationId: job.conversationId ?? streamId,
|
||||
conversationId,
|
||||
generationCreatedAt: job.createdAt,
|
||||
status,
|
||||
status: settlementOutcome.status,
|
||||
settledAt,
|
||||
...(status === 'failed' && { error: job.error ?? 'Generation failed' }),
|
||||
...(action != null && {
|
||||
action: {
|
||||
toolName: action.toolName.slice(0, MAX_RECEIPT_ID_LENGTH),
|
||||
...(action.toolCallId != null && {
|
||||
toolCallId: action.toolCallId.slice(0, MAX_RECEIPT_ID_LENGTH),
|
||||
}),
|
||||
},
|
||||
...(settlementOutcome.status === 'failed' && {
|
||||
error: compensated
|
||||
? 'Applied event actor action was explicitly compensated'
|
||||
: (job.error ?? 'Generation failed'),
|
||||
}),
|
||||
...(settlementOutcome.action != null && { action: settlementOutcome.action }),
|
||||
});
|
||||
if (!settled) {
|
||||
throw new Error(`Failed to settle agent event delivery ${job.agentEventDeliveryKey}`);
|
||||
|
|
|
|||
|
|
@ -2626,6 +2626,7 @@ class GenerationJobManagerClass {
|
|||
providerDrained: jobData.providerDrained,
|
||||
steersClosed: jobData.steersClosed,
|
||||
idempotencyClientRequestId: jobData.idempotencyClientRequestId,
|
||||
agentEventLegacyTurnToken: jobData.agentEventLegacyTurnToken,
|
||||
terminalPersistencePending: jobData.terminalPersistencePending,
|
||||
terminalPersistenceStartedAt: jobData.terminalPersistenceStartedAt,
|
||||
// Surface the pending review so status/resume routes built on the
|
||||
|
|
|
|||
|
|
@ -1303,6 +1303,7 @@ describe('RedisJobStore Integration Tests', () => {
|
|||
toolName: 'submit_move',
|
||||
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
|
||||
},
|
||||
agentEventLegacyTurnToken: 'legacy-hitl-token',
|
||||
discoveredTools: ['deep_tool'],
|
||||
userSubmittedPaths: ['/content/0/tool_call/args'],
|
||||
userSubmittedMessageFieldPaths: [{ path: '/content/0/tool_call/output', field: 'answer' }],
|
||||
|
|
@ -1315,6 +1316,7 @@ describe('RedisJobStore Integration Tests', () => {
|
|||
toolName: 'submit_move',
|
||||
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
|
||||
});
|
||||
expect(turn1?.agentEventLegacyTurnToken).toBe('legacy-hitl-token');
|
||||
expect(turn1?.discoveredTools).toEqual(['deep_tool']);
|
||||
expect(turn1?.userSubmittedPaths).toEqual(['/content/0/tool_call/args']);
|
||||
expect(turn1?.userSubmittedMessageFieldPaths).toEqual([
|
||||
|
|
@ -1332,6 +1334,7 @@ describe('RedisJobStore Integration Tests', () => {
|
|||
expect(turn2?.isTemporary).toBeUndefined();
|
||||
expect(turn2?.agentEventDeliveryKey).toBeUndefined();
|
||||
expect(turn2?.agentEventExpectedAction).toBeUndefined();
|
||||
expect(turn2?.agentEventLegacyTurnToken).toBeUndefined();
|
||||
expect(turn2?.discoveredTools).toBeUndefined();
|
||||
expect(turn2?.userSubmittedPaths).toBeUndefined();
|
||||
expect(turn2?.userSubmittedMessageFieldPaths).toBeUndefined();
|
||||
|
|
|
|||
|
|
@ -1484,6 +1484,18 @@ describe('GenerationJobManager HITL resume metadata (round 19)', () => {
|
|||
expect(job?.metadata.discoveredTools).toEqual(['deep_tool', 'other_tool']);
|
||||
});
|
||||
|
||||
test('updateMetadata exposes a paused legacy-event fence through the job facade', async () => {
|
||||
const streamId = 'stream-legacy-event';
|
||||
await manager.createJob(streamId, 'user-1');
|
||||
await manager.updateMetadata(streamId, {
|
||||
agentEventLegacyTurnToken: 'legacy-hitl-token',
|
||||
});
|
||||
|
||||
const job = await manager.getJob(streamId);
|
||||
|
||||
expect(job?.metadata.agentEventLegacyTurnToken).toBe('legacy-hitl-token');
|
||||
});
|
||||
|
||||
// H4: a pause that lands AFTER the resume snapshot but before the subscription must
|
||||
// still reach the client. subscribeWithResume re-reads the live job and surfaces it.
|
||||
test('subscribeWithResume surfaces a pause that the resume snapshot missed', async () => {
|
||||
|
|
|
|||
|
|
@ -4732,6 +4732,7 @@ export class RedisJobStore implements IJobStoreV2 {
|
|||
agentEventExpectedAction: data.agentEventExpectedAction
|
||||
? JSON.parse(data.agentEventExpectedAction)
|
||||
: undefined,
|
||||
agentEventLegacyTurnToken: data.agentEventLegacyTurnToken || undefined,
|
||||
scheduleId: data.scheduleId || undefined,
|
||||
scheduledFor: data.scheduledFor || undefined,
|
||||
scheduleConfigRevision: data.scheduleConfigRevision
|
||||
|
|
|
|||
|
|
@ -280,6 +280,8 @@ export interface SerializableJobData {
|
|||
isTemporary?: boolean;
|
||||
agentEventDeliveryKey?: string;
|
||||
agentEventExpectedAction?: import('~/agents/triggers/types').AgentTriggerExpectedAction;
|
||||
/** Exact durable legacy-turn fence carried across a HITL pause/resume. */
|
||||
agentEventLegacyTurnToken?: string;
|
||||
|
||||
/**
|
||||
* Set when status is `requires_action`. Describes the human review the
|
||||
|
|
@ -399,6 +401,7 @@ export type JobMetadataPatch = Partial<
|
|||
| 'isTemporary'
|
||||
| 'agentEventDeliveryKey'
|
||||
| 'agentEventExpectedAction'
|
||||
| 'agentEventLegacyTurnToken'
|
||||
| 'scheduleId'
|
||||
| 'scheduledFor'
|
||||
| 'scheduleConfigRevision'
|
||||
|
|
|
|||
|
|
@ -42,6 +42,9 @@ export function sanitizeJobMetadata(metadata: Partial<GenerationJobMetadata>): J
|
|||
if (metadata.agentEventExpectedAction) {
|
||||
patch.agentEventExpectedAction = metadata.agentEventExpectedAction;
|
||||
}
|
||||
if (metadata.agentEventLegacyTurnToken) {
|
||||
patch.agentEventLegacyTurnToken = metadata.agentEventLegacyTurnToken;
|
||||
}
|
||||
if (metadata.scheduleId) {
|
||||
patch.scheduleId = metadata.scheduleId;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -46,6 +46,8 @@ export interface GenerationJobMetadata {
|
|||
agentEventDeliveryKey?: string;
|
||||
/** Optional action evidence contract declared by the authenticated event source. */
|
||||
agentEventExpectedAction?: AgentTriggerExpectedAction;
|
||||
/** Exact durable legacy-turn fence carried across a HITL pause/resume. */
|
||||
agentEventLegacyTurnToken?: string;
|
||||
/** Trusted scheduled-occurrence identity. These fields are accepted only from a
|
||||
* verified agent-trigger request and let pause/resume/reconciliation keep the
|
||||
* occurrence attached to the exact generation that owns it. */
|
||||
|
|
|
|||
|
|
@ -21,12 +21,17 @@ const endpointsConfig: TEndpointsConfig = {
|
|||
};
|
||||
|
||||
describe('excludedKeys', () => {
|
||||
it.each(['_id', 'user', 'conversationId', 'agentEventBinding', '__v'])(
|
||||
'excludes system field "%s"',
|
||||
(field) => {
|
||||
expect(excludedKeys.has(field)).toBe(true);
|
||||
},
|
||||
);
|
||||
it.each([
|
||||
'_id',
|
||||
'user',
|
||||
'conversationId',
|
||||
'agentEventBinding',
|
||||
'agentEventActor',
|
||||
'agentEventActorReconciliations',
|
||||
'__v',
|
||||
])('excludes system field "%s"', (field) => {
|
||||
expect(excludedKeys.has(field)).toBe(true);
|
||||
});
|
||||
|
||||
it('does not exclude tenantId (plugin-level guard owns this)', () => {
|
||||
expect(excludedKeys.has('tenantId')).toBe(false);
|
||||
|
|
@ -71,6 +76,7 @@ describe('agent event runtime config', () => {
|
|||
childTurns: true,
|
||||
completionWakeups: false,
|
||||
coalescing: true,
|
||||
checkpointForks: true,
|
||||
selfUrl: 'https://triggers.internal',
|
||||
},
|
||||
},
|
||||
|
|
@ -88,6 +94,7 @@ describe('agent event runtime config', () => {
|
|||
childTurns: true,
|
||||
completionWakeups: false,
|
||||
coalescing: true,
|
||||
checkpointForks: true,
|
||||
selfUrl: 'https://triggers.internal',
|
||||
});
|
||||
expect(result.data.rateLimits?.agentEvents).toEqual({
|
||||
|
|
@ -95,6 +102,27 @@ describe('agent event runtime config', () => {
|
|||
userWindowInMinutes: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects checkpoint forks with the process-local memory checkpointer', () => {
|
||||
const result = configSchema.safeParse({
|
||||
version: '1.0',
|
||||
endpoints: {
|
||||
agents: {
|
||||
eventDriven: { checkpointForks: true },
|
||||
checkpointer: { type: 'memory' },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.error?.issues).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
path: ['endpoints', 'agents', 'eventDriven', 'checkpointForks'],
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('speechTab schema', () => {
|
||||
|
|
|
|||
|
|
@ -70,6 +70,10 @@ export const defaultRetrievalModels = [
|
|||
export const excludedKeys = new Set([
|
||||
'conversationId',
|
||||
'agentEventBinding',
|
||||
'agentEventActor',
|
||||
'agentEventActorReconciliations',
|
||||
'agentEventActorEpoch',
|
||||
'agentEventActorLegacyTurn',
|
||||
'subagentThread',
|
||||
'title',
|
||||
'iconURL',
|
||||
|
|
@ -1047,6 +1051,10 @@ export const agentsEndpointSchema = baseEndpointSchema
|
|||
completionWakeups: z.boolean().optional(),
|
||||
/** Enable only after every API worker can consume coalesced deliveries. */
|
||||
coalescing: z.boolean().optional(),
|
||||
/** Reuse a bound event actor's committed checkpoint through isolated
|
||||
* per-invocation forks. Keep off until every API worker runs an SDK
|
||||
* and host adapter that understand the fork lifecycle. */
|
||||
checkpointForks: z.boolean().optional(),
|
||||
/** Optional trusted origin for in-process trigger delivery. The bound
|
||||
* listener remains the default and is safer for most deployments. */
|
||||
selfUrl: z.string().url().optional(),
|
||||
|
|
@ -1065,6 +1073,15 @@ export const agentsEndpointSchema = baseEndpointSchema
|
|||
checkpointer: checkpointerSchema,
|
||||
}),
|
||||
)
|
||||
.superRefine((config, ctx) => {
|
||||
if (config.eventDriven?.checkpointForks === true && config.checkpointer?.type === 'memory') {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['eventDriven', 'checkpointForks'],
|
||||
message: 'Event actor checkpoint forks require the Mongo checkpointer',
|
||||
});
|
||||
}
|
||||
})
|
||||
.default({
|
||||
disableBuilder: false,
|
||||
capabilities: defaultAgentCapabilities,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -2,6 +2,10 @@ import { RetentionMode } from 'librechat-data-provider';
|
|||
import type { FilterQuery, Model, SortOrder, Types } from 'mongoose';
|
||||
import type { DeleteResult } from 'mongoose';
|
||||
import type {
|
||||
IAgentEventActorCheckpoint,
|
||||
IAgentEventActorReconciliation,
|
||||
IAgentEventActorSnapshot,
|
||||
IAgentEventActorState,
|
||||
IAgentEventBindingRecord,
|
||||
AppConfig,
|
||||
IChatProjectDocument,
|
||||
|
|
@ -54,6 +58,34 @@ export type SubagentThreadReadRecord = Pick<
|
|||
|
||||
export type ParentSubagentThreadRecord = SubagentThreadReadRecord;
|
||||
|
||||
export type AgentEventActorCommitResult =
|
||||
| {
|
||||
status: 'committed';
|
||||
state: IAgentEventActorState;
|
||||
prunableCheckpoint?: IAgentEventActorCheckpoint;
|
||||
}
|
||||
| { status: 'stale'; state?: IAgentEventActorState };
|
||||
|
||||
/**
|
||||
* How long a settled receipt keeps tombstoning its invocation id. A stale
|
||||
* same-id owner is bounded by time, not by how many newer invocations settle,
|
||||
* so eviction is primarily age-based: a receipt becomes prunable only once no
|
||||
* delayed duplicate of its invocation could still be admitted — far beyond any
|
||||
* generation, job, or delivery-retry lifetime.
|
||||
*/
|
||||
const AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS = 7 * 24 * 60 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Absolute size bound for the private invocation journal so a pathological
|
||||
* actor cannot grow its conversation document toward Mongo's document limit.
|
||||
* Never an eviction quota: a receipt inside its retention window is never
|
||||
* discarded. When the journal holds this many unexpired receipts, admission
|
||||
* of NEW invocations is refused (fail closed) until receipts age out, so
|
||||
* document integrity and same-id duplicate protection are both invariants
|
||||
* rather than a trade.
|
||||
*/
|
||||
export const AGENT_EVENT_ACTOR_RECEIPT_LIMIT = 1024;
|
||||
|
||||
const ARCHIVE_CONVERSATION_BATCH_SIZE = 500;
|
||||
const PROJECT_STATS_REFRESH_CONCURRENCY = 10;
|
||||
const PROJECT_STATS_REFRESH_MAX_PASSES = 2;
|
||||
|
|
@ -211,6 +243,51 @@ export interface ConversationMethods {
|
|||
sourceKeyId: string;
|
||||
tenantId?: string;
|
||||
}): Promise<IAgentEventBindingRecord | null>;
|
||||
getAgentEventActorSnapshot(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
}): Promise<IAgentEventActorSnapshot | undefined>;
|
||||
commitAgentEventActorState(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
invocationId: string;
|
||||
action: IAgentEventActorReconciliation['action'];
|
||||
expected?: IAgentEventActorState;
|
||||
expectedEpoch: number;
|
||||
checkpoint: IAgentEventActorCheckpoint;
|
||||
}): Promise<AgentEventActorCommitResult>;
|
||||
beginAgentEventActorLegacyTurn(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
token: string;
|
||||
}): Promise<boolean>;
|
||||
completeAgentEventActorLegacyTurn(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
token: string;
|
||||
}): Promise<boolean>;
|
||||
recordAgentEventActorReconciliation(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
reconciliation: IAgentEventActorReconciliation;
|
||||
}): Promise<boolean>;
|
||||
resolveAgentEventActorReconciliation(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
invocationId: string;
|
||||
checkpoint: IAgentEventActorReconciliation['checkpoint'];
|
||||
resolution:
|
||||
| 'checkpoint_verified'
|
||||
| 'action_compensated'
|
||||
| 'history_repaired'
|
||||
| 'invocation_abandoned';
|
||||
}): Promise<boolean>;
|
||||
reserveSubagentThread(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
|
|
@ -414,6 +491,482 @@ export function createConversationMethods(
|
|||
};
|
||||
}
|
||||
|
||||
/** Reads the private actor head and every fail-closed reconciliation marker. */
|
||||
async function getAgentEventActorSnapshot(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
}): Promise<IAgentEventActorSnapshot | undefined> {
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
const conversation = await Conversation.findOne({
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
})
|
||||
.select(
|
||||
'+agentEventActor +agentEventActorReconciliations +agentEventActorEpoch +agentEventActorLegacyTurn',
|
||||
)
|
||||
.lean<IConversation>();
|
||||
return conversation == null
|
||||
? undefined
|
||||
: {
|
||||
state: conversation.agentEventActor ?? null,
|
||||
reconciliations: conversation.agentEventActorReconciliations ?? [],
|
||||
legacyTurn: conversation.agentEventActorLegacyTurn ?? null,
|
||||
epoch: conversation.agentEventActorEpoch ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
/** Advances one actor head only when its complete prior identity still matches. */
|
||||
async function commitAgentEventActorState(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
invocationId: string;
|
||||
action: IAgentEventActorReconciliation['action'];
|
||||
expected?: IAgentEventActorState;
|
||||
expectedEpoch: number;
|
||||
checkpoint: IAgentEventActorCheckpoint;
|
||||
}): Promise<AgentEventActorCommitResult> {
|
||||
if (input.checkpoint.threadId !== input.conversationId) {
|
||||
throw new Error('Event actor checkpoint changed its logical thread');
|
||||
}
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
/** A legacy turn against a headless or already cold-marked actor leaves
|
||||
* the head fields unchanged, so the CAS must also require the invalidation
|
||||
* epoch observed at preparation. `null` matches a document that has never
|
||||
* been invalidated. */
|
||||
const expectedFilter: FilterQuery<IConversation> = {
|
||||
agentEventActorEpoch: input.expectedEpoch === 0 ? null : input.expectedEpoch,
|
||||
/** A legacy turn in flight has not yet persisted its messages, so no
|
||||
* fork rebuild can be complete — refuse the commit outright rather than
|
||||
* relying on the epoch, which only moves once that turn seals. */
|
||||
agentEventActorLegacyTurn: { $exists: false },
|
||||
...(input.expected == null
|
||||
? { agentEventActor: { $exists: false } }
|
||||
: {
|
||||
'agentEventActor.generation': input.expected.generation,
|
||||
'agentEventActor.checkpoint.threadId': input.expected.checkpoint.threadId,
|
||||
'agentEventActor.checkpoint.checkpointId': input.expected.checkpoint.checkpointId,
|
||||
'agentEventActor.checkpoint.checkpointNs': input.expected.checkpoint.checkpointNs,
|
||||
'agentEventActor.requiresColdStart':
|
||||
input.expected.requiresColdStart === true ? true : { $ne: true },
|
||||
}),
|
||||
};
|
||||
const nextState: IAgentEventActorState = {
|
||||
generation: (input.expected?.generation ?? 0) + 1,
|
||||
checkpoint: input.checkpoint,
|
||||
...(input.expected == null ? {} : { previousCheckpoint: input.expected.checkpoint }),
|
||||
};
|
||||
const previous = await Conversation.findOneAndUpdate(
|
||||
{
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
agentEventActorReconciliations: {
|
||||
$elemMatch: {
|
||||
invocationId: input.invocationId,
|
||||
status: 'invocation_pending',
|
||||
},
|
||||
},
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
...expectedFilter,
|
||||
},
|
||||
{
|
||||
$set: {
|
||||
agentEventActor: nextState,
|
||||
'agentEventActorReconciliations.$.status': 'persistence_pending',
|
||||
'agentEventActorReconciliations.$.checkpoint': input.checkpoint,
|
||||
'agentEventActorReconciliations.$.action': input.action,
|
||||
'agentEventActorReconciliations.$.observedAt': new Date(),
|
||||
},
|
||||
$unset: { 'agentEventActorReconciliations.$.error': 1 },
|
||||
},
|
||||
{ new: false, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActor')
|
||||
.lean<IConversation>();
|
||||
if (previous != null) {
|
||||
return {
|
||||
status: 'committed',
|
||||
state: nextState,
|
||||
...(previous.agentEventActor?.previousCheckpoint == null
|
||||
? {}
|
||||
: { prunableCheckpoint: previous.agentEventActor.previousCheckpoint }),
|
||||
};
|
||||
}
|
||||
const current = await getAgentEventActorSnapshot(input);
|
||||
return {
|
||||
status: 'stale',
|
||||
...(current?.state == null ? {} : { state: current.state }),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Opens the durable fence for one legacy-path turn BEFORE it executes. One
|
||||
* mutually exclusive classic update sets the token and, only when a valid
|
||||
* head exists, marks it cold — no partial fence state is externally visible.
|
||||
* Refuses while another legacy turn or fork lifecycle is active. Abandoned
|
||||
* legacy tokens are reclaimed through the bounded recovery operation before
|
||||
* admission is retried.
|
||||
*/
|
||||
async function beginAgentEventActorLegacyTurn(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
token: string;
|
||||
}): Promise<boolean> {
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
const ownership: FilterQuery<IConversation> = {
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
agentEventActorLegacyTurn: { $exists: false },
|
||||
agentEventActorReconciliations: {
|
||||
$not: { $elemMatch: { status: { $ne: 'settled' } } },
|
||||
},
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
};
|
||||
const legacyTurn = { token: input.token, startedAt: new Date() };
|
||||
/** DocumentDB does not support aggregation-pipeline updates. Split the
|
||||
* headful/headless shapes into mutually exclusive classic updates. A head
|
||||
* cannot appear between them without first creating an unresolved fork
|
||||
* lifecycle, which the shared ownership filter rejects. */
|
||||
const openedWithHead = await Conversation.findOneAndUpdate(
|
||||
{ ...ownership, 'agentEventActor.generation': { $exists: true } },
|
||||
{
|
||||
$set: {
|
||||
agentEventActorLegacyTurn: legacyTurn,
|
||||
'agentEventActor.requiresColdStart': true,
|
||||
},
|
||||
},
|
||||
{ new: true, timestamps: false },
|
||||
).lean<IConversation>();
|
||||
if (openedWithHead != null) {
|
||||
return true;
|
||||
}
|
||||
const openedHeadless = await Conversation.findOneAndUpdate(
|
||||
{ ...ownership, 'agentEventActor.generation': { $exists: false } },
|
||||
{ $set: { agentEventActorLegacyTurn: legacyTurn } },
|
||||
{ new: true, timestamps: false },
|
||||
).lean<IConversation>();
|
||||
return openedHeadless != null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Closes the fence in ONE atomic write once the legacy turn's history is
|
||||
* durable: clears this exact token and advances the epoch together, so a
|
||||
* fork can never observe a cleared fence at an unchanged epoch. Matching the
|
||||
* token keeps a later turn's seal from closing an earlier turn's fence.
|
||||
*/
|
||||
async function completeAgentEventActorLegacyTurn(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
token: string;
|
||||
}): Promise<boolean> {
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
const sealed = await Conversation.findOneAndUpdate(
|
||||
{
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
'agentEventActorLegacyTurn.token': input.token,
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
},
|
||||
{
|
||||
$unset: { agentEventActorLegacyTurn: 1 },
|
||||
$inc: { agentEventActorEpoch: 1 },
|
||||
},
|
||||
{ new: true, timestamps: false },
|
||||
).lean<IConversation>();
|
||||
return sealed != null;
|
||||
}
|
||||
|
||||
/** Acquires or advances one invocation lifecycle fence and blocks competing turns. */
|
||||
async function recordAgentEventActorReconciliation(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
reconciliation: IAgentEventActorReconciliation;
|
||||
}): Promise<boolean> {
|
||||
if (input.reconciliation.checkpoint.threadId !== input.conversationId) {
|
||||
throw new Error('Event actor reconciliation changed its logical thread');
|
||||
}
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
const ownership = {
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
};
|
||||
const existing = await Conversation.findOne(ownership)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
const journal = existing?.agentEventActorReconciliations ?? [];
|
||||
const current = journal.find((item) => item.invocationId === input.reconciliation.invocationId);
|
||||
if (current != null) {
|
||||
const canTransition =
|
||||
(current?.status === 'invocation_pending' &&
|
||||
input.reconciliation.status !== 'invocation_pending') ||
|
||||
(current?.status === 'persistence_pending' &&
|
||||
(input.reconciliation.status === 'persistence_failed' ||
|
||||
input.reconciliation.status === 'history_persisted')) ||
|
||||
(current?.status === 'persistence_failed' &&
|
||||
input.reconciliation.status === 'history_persisted');
|
||||
const sameIdentity =
|
||||
current?.status === input.reconciliation.status &&
|
||||
current.checkpoint.threadId === input.reconciliation.checkpoint.threadId &&
|
||||
current.checkpoint.checkpointId === input.reconciliation.checkpoint.checkpointId &&
|
||||
current.checkpoint.checkpointNs === input.reconciliation.checkpoint.checkpointNs &&
|
||||
current.action.toolName === input.reconciliation.action.toolName &&
|
||||
current.action.toolCallId === input.reconciliation.action.toolCallId;
|
||||
/** A pending record is an exclusive ownership fence, not an idempotent
|
||||
* receipt. A second executor must not inherit the first owner's claim. */
|
||||
if (input.reconciliation.status === 'invocation_pending') {
|
||||
return false;
|
||||
}
|
||||
if (!canTransition) {
|
||||
return sameIdentity;
|
||||
}
|
||||
const transitioned = await Conversation.findOneAndUpdate(
|
||||
{
|
||||
...ownership,
|
||||
agentEventActorReconciliations: {
|
||||
$elemMatch: {
|
||||
invocationId: input.reconciliation.invocationId,
|
||||
status: current!.status,
|
||||
},
|
||||
},
|
||||
},
|
||||
{ $set: { 'agentEventActorReconciliations.$': input.reconciliation } },
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
return transitioned != null;
|
||||
}
|
||||
/** Every post-acquisition state must transition the exact pending lifecycle.
|
||||
* Never recreate a marker after terminal recovery has already removed it. */
|
||||
if (input.reconciliation.status !== 'invocation_pending') {
|
||||
return false;
|
||||
}
|
||||
const receiptCutoff = new Date(Date.now() - AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS);
|
||||
if (
|
||||
journal.some(
|
||||
(item) => item.status === 'settled' && item.observedAt.getTime() < receiptCutoff.getTime(),
|
||||
)
|
||||
) {
|
||||
await Conversation.updateOne(
|
||||
ownership,
|
||||
{
|
||||
$pull: {
|
||||
agentEventActorReconciliations: {
|
||||
status: 'settled',
|
||||
observedAt: { $lt: receiptCutoff },
|
||||
},
|
||||
},
|
||||
},
|
||||
{ timestamps: false },
|
||||
);
|
||||
}
|
||||
/** A receipt inside its retention window may NEVER be evicted — a stale
|
||||
* same-id owner is bounded by time, and count-based eviction would reopen
|
||||
* duplicate execution at high event rates. When the journal is full of
|
||||
* unexpired receipts, refuse admission (fail closed) instead of trading
|
||||
* away tombstone protection; the actor resumes once receipts age out.
|
||||
* Admission is serialized by the no-active-row filter, so the journal can
|
||||
* exceed the cap by at most the single row admitted after this check. */
|
||||
const retainedReceipts = journal.filter(
|
||||
(item) => item.status === 'settled' && item.observedAt.getTime() >= receiptCutoff.getTime(),
|
||||
);
|
||||
if (retainedReceipts.length >= AGENT_EVENT_ACTOR_RECEIPT_LIMIT) {
|
||||
logger.error(
|
||||
`[conversation] Event actor receipt journal for ${input.conversationId} is full of unexpired receipts; refusing new invocation ${input.reconciliation.invocationId} until receipts age out`,
|
||||
);
|
||||
return false;
|
||||
}
|
||||
const recorded = await Conversation.findOneAndUpdate(
|
||||
{
|
||||
...ownership,
|
||||
agentEventActorLegacyTurn: { $exists: false },
|
||||
agentEventActorReconciliations: {
|
||||
$not: { $elemMatch: { status: { $ne: 'settled' } } },
|
||||
},
|
||||
'agentEventActorReconciliations.invocationId': { $ne: input.reconciliation.invocationId },
|
||||
},
|
||||
{ $push: { agentEventActorReconciliations: input.reconciliation } },
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
if (
|
||||
recorded?.agentEventActorReconciliations?.some(
|
||||
(item) => item.invocationId === input.reconciliation.invocationId,
|
||||
) === true
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
/** Another writer won the same invocation fence while this write raced. */
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Resolves exactly one lifecycle after settlement, abandonment, or repair. */
|
||||
async function resolveAgentEventActorReconciliation(input: {
|
||||
user: string;
|
||||
conversationId: string;
|
||||
tenantId?: string;
|
||||
invocationId: string;
|
||||
checkpoint: IAgentEventActorReconciliation['checkpoint'];
|
||||
resolution:
|
||||
| 'checkpoint_verified'
|
||||
| 'action_compensated'
|
||||
| 'history_repaired'
|
||||
| 'invocation_abandoned';
|
||||
}): Promise<boolean> {
|
||||
if (input.checkpoint.threadId !== input.conversationId) {
|
||||
throw new Error('Event actor reconciliation changed its logical thread');
|
||||
}
|
||||
const Conversation = mongoose.models.Conversation as Model<IConversation>;
|
||||
const exactCheckpoint: Record<string, unknown> = {
|
||||
invocationId: input.invocationId,
|
||||
'checkpoint.threadId': input.checkpoint.threadId,
|
||||
'checkpoint.checkpointNs': input.checkpoint.checkpointNs,
|
||||
...(input.checkpoint.checkpointId == null
|
||||
? { 'checkpoint.checkpointId': { $exists: false } }
|
||||
: { 'checkpoint.checkpointId': input.checkpoint.checkpointId }),
|
||||
};
|
||||
if (
|
||||
input.resolution === 'checkpoint_verified' &&
|
||||
(typeof input.checkpoint.checkpointId !== 'string' ||
|
||||
input.checkpoint.checkpointId.length === 0)
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
const owner = {
|
||||
user: input.user,
|
||||
conversationId: input.conversationId,
|
||||
subagentThread: { $exists: true },
|
||||
agentEventBinding: { $exists: true },
|
||||
...subagentLeaseTenantFilter(input.tenantId),
|
||||
...activeExpirationFilter<IConversation>(),
|
||||
};
|
||||
if (input.resolution === 'checkpoint_verified') {
|
||||
const settled = await Conversation.findOneAndUpdate(
|
||||
{
|
||||
...owner,
|
||||
'agentEventActor.checkpoint.threadId': input.checkpoint.threadId,
|
||||
'agentEventActor.checkpoint.checkpointId': input.checkpoint.checkpointId!,
|
||||
'agentEventActor.checkpoint.checkpointNs': input.checkpoint.checkpointNs,
|
||||
agentEventActorReconciliations: {
|
||||
$elemMatch: { ...exactCheckpoint, status: 'history_persisted' },
|
||||
},
|
||||
},
|
||||
{
|
||||
$set: {
|
||||
'agentEventActorReconciliations.$.status': 'settled',
|
||||
'agentEventActorReconciliations.$.resolution': 'checkpoint_verified',
|
||||
'agentEventActorReconciliations.$.observedAt': new Date(),
|
||||
},
|
||||
$unset: { 'agentEventActorReconciliations.$.error': 1 },
|
||||
},
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
if (settled != null) {
|
||||
return true;
|
||||
}
|
||||
/** A verification retry may replay only a receipt that verification
|
||||
* itself settled. A compensated receipt must fail this resolve so the
|
||||
* caller re-reads and honors the compensation instead. */
|
||||
const replay = await Conversation.exists({
|
||||
...owner,
|
||||
'agentEventActor.checkpoint.threadId': input.checkpoint.threadId,
|
||||
'agentEventActor.checkpoint.checkpointId': input.checkpoint.checkpointId!,
|
||||
'agentEventActor.checkpoint.checkpointNs': input.checkpoint.checkpointNs,
|
||||
agentEventActorReconciliations: {
|
||||
$elemMatch: { ...exactCheckpoint, status: 'settled', resolution: 'checkpoint_verified' },
|
||||
},
|
||||
});
|
||||
return replay != null;
|
||||
}
|
||||
const checkpointFilter: Record<string, unknown> = {
|
||||
...exactCheckpoint,
|
||||
...(input.resolution === 'invocation_abandoned'
|
||||
? { status: 'invocation_pending' }
|
||||
: { status: { $ne: 'settled' } }),
|
||||
};
|
||||
const ownership = {
|
||||
...owner,
|
||||
agentEventActorReconciliations: { $elemMatch: checkpointFilter },
|
||||
};
|
||||
const mustRebuild =
|
||||
input.resolution === 'action_compensated' || input.resolution === 'history_repaired';
|
||||
if (!mustRebuild) {
|
||||
const abandoned = await Conversation.findOneAndUpdate(
|
||||
ownership,
|
||||
{ $pull: { agentEventActorReconciliations: checkpointFilter } },
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
return abandoned != null;
|
||||
}
|
||||
/** Repair and compensation both act on an invocation whose action already
|
||||
* reached the outside world, so the receipt must survive as the durable
|
||||
* same-invocation tombstone. Deleting it would let a delayed duplicate
|
||||
* owner reacquire the same id and repeat that action. Compensation undoes
|
||||
* the effect; it does not re-authorize the delivery, so a legitimate retry
|
||||
* must arrive under a new invocation id. */
|
||||
const retainedReceipt = {
|
||||
'agentEventActorReconciliations.$.status': 'settled',
|
||||
'agentEventActorReconciliations.$.resolution': input.resolution,
|
||||
'agentEventActorReconciliations.$.observedAt': new Date(),
|
||||
};
|
||||
const resolved = await Conversation.findOneAndUpdate(
|
||||
{ ...ownership, agentEventActor: { $exists: true } },
|
||||
{ $set: { ...retainedReceipt, 'agentEventActor.requiresColdStart': true } },
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
if (resolved != null) {
|
||||
return true;
|
||||
}
|
||||
/** Compensation before a first committed head must retire the exact fence
|
||||
* without creating a partial actor state that cannot be resumed. */
|
||||
const resolvedWithoutHead = await Conversation.findOneAndUpdate(
|
||||
{ ...ownership, agentEventActor: { $exists: false } },
|
||||
{ $set: retainedReceipt },
|
||||
{ new: true, timestamps: false },
|
||||
)
|
||||
.select('+agentEventActorReconciliations')
|
||||
.lean<IConversation>();
|
||||
if (resolvedWithoutHead != null) {
|
||||
return true;
|
||||
}
|
||||
/** A retried repair finds its own retained receipt already settled. */
|
||||
const replayed = await Conversation.exists({
|
||||
...owner,
|
||||
agentEventActorReconciliations: {
|
||||
$elemMatch: { ...exactCheckpoint, status: 'settled', resolution: input.resolution },
|
||||
},
|
||||
});
|
||||
return replayed != null;
|
||||
}
|
||||
|
||||
/** Creates immutable child lineage exactly once without overwriting a concurrent winner. */
|
||||
async function reserveSubagentThread(input: {
|
||||
user: string;
|
||||
|
|
@ -1786,6 +2339,12 @@ export function createConversationMethods(
|
|||
getSubagentThreadForParent,
|
||||
listSubagentThreadsForParent,
|
||||
getAgentEventBinding,
|
||||
getAgentEventActorSnapshot,
|
||||
commitAgentEventActorState,
|
||||
beginAgentEventActorLegacyTurn,
|
||||
completeAgentEventActorLegacyTurn,
|
||||
recordAgentEventActorReconciliation,
|
||||
resolveAgentEventActorReconciliation,
|
||||
reserveSubagentThread,
|
||||
acquireSubagentThreadLease,
|
||||
renewSubagentThreadLease,
|
||||
|
|
|
|||
|
|
@ -66,6 +66,103 @@ const convoSchema: Schema<IConversation> = new Schema(
|
|||
default: undefined,
|
||||
select: false,
|
||||
},
|
||||
/** Committed event-actor state. Invocation forks live in the checkpointer;
|
||||
* only the current and previous committed identities live with the binding. */
|
||||
agentEventActor: {
|
||||
type: {
|
||||
generation: { type: Number, min: 1, required: true },
|
||||
checkpoint: {
|
||||
type: {
|
||||
threadId: { type: String, required: true },
|
||||
checkpointId: { type: String, required: true },
|
||||
checkpointNs: { type: String, required: true },
|
||||
},
|
||||
_id: false,
|
||||
required: true,
|
||||
},
|
||||
previousCheckpoint: {
|
||||
type: {
|
||||
threadId: { type: String, required: true },
|
||||
checkpointId: { type: String, required: true },
|
||||
checkpointNs: { type: String, required: true },
|
||||
},
|
||||
_id: false,
|
||||
default: undefined,
|
||||
},
|
||||
requiresColdStart: { type: Boolean, default: undefined },
|
||||
},
|
||||
_id: false,
|
||||
default: undefined,
|
||||
select: false,
|
||||
},
|
||||
/** Fail-closed invocation proof. Active records block later turns through checkpoint,
|
||||
* history, and outcome settlement; settled receipts no longer block new IDs but keep
|
||||
* delayed owners from reacquiring an invocation that already applied its action. */
|
||||
agentEventActorReconciliations: {
|
||||
type: [
|
||||
{
|
||||
invocationId: { type: String, required: true },
|
||||
status: {
|
||||
type: String,
|
||||
enum: [
|
||||
'invocation_pending',
|
||||
'persistence_pending',
|
||||
'history_persisted',
|
||||
'commit_conflict',
|
||||
'commit_indeterminate',
|
||||
'persistence_failed',
|
||||
'settled',
|
||||
],
|
||||
required: true,
|
||||
},
|
||||
checkpoint: {
|
||||
type: {
|
||||
threadId: { type: String, required: true },
|
||||
checkpointId: { type: String, default: undefined },
|
||||
checkpointNs: { type: String, required: true },
|
||||
},
|
||||
_id: false,
|
||||
required: true,
|
||||
},
|
||||
action: {
|
||||
type: {
|
||||
toolName: { type: String, required: true },
|
||||
toolCallId: { type: String, default: undefined },
|
||||
},
|
||||
_id: false,
|
||||
required: true,
|
||||
},
|
||||
error: { type: String, default: undefined },
|
||||
resolution: {
|
||||
type: String,
|
||||
enum: ['checkpoint_verified', 'action_compensated', 'history_repaired'],
|
||||
default: undefined,
|
||||
},
|
||||
observedAt: { type: Date, required: true },
|
||||
_id: false,
|
||||
},
|
||||
],
|
||||
default: undefined,
|
||||
select: false,
|
||||
},
|
||||
/** Bumped by every legacy-path event so a concurrently prepared fork can
|
||||
* never commit past an invalidation the head fields alone cannot show. */
|
||||
agentEventActorEpoch: {
|
||||
type: Number,
|
||||
default: undefined,
|
||||
select: false,
|
||||
},
|
||||
/** In-flight legacy turn. Blocks fork execution and commit until the
|
||||
* turn's history is durable; a crash leaves it set, failing closed. */
|
||||
agentEventActorLegacyTurn: {
|
||||
type: {
|
||||
token: { type: String, required: true },
|
||||
startedAt: { type: Date, required: true },
|
||||
},
|
||||
_id: false,
|
||||
default: undefined,
|
||||
select: false,
|
||||
},
|
||||
tags: {
|
||||
type: [String],
|
||||
default: [],
|
||||
|
|
|
|||
|
|
@ -14,6 +14,62 @@ export interface IAgentEventBinding {
|
|||
actorId: string;
|
||||
}
|
||||
|
||||
export interface IAgentEventActorCheckpoint {
|
||||
threadId: string;
|
||||
checkpointId: string;
|
||||
checkpointNs: string;
|
||||
}
|
||||
|
||||
/** Private committed checkpoint state for one event-bound child actor. */
|
||||
export interface IAgentEventActorState {
|
||||
generation: number;
|
||||
checkpoint: IAgentEventActorCheckpoint;
|
||||
previousCheckpoint?: IAgentEventActorCheckpoint;
|
||||
/** Forces the next qualifying event to rebuild from durable message history. */
|
||||
requiresColdStart?: boolean;
|
||||
}
|
||||
|
||||
export interface IAgentEventActorReconciliation {
|
||||
invocationId: string;
|
||||
status:
|
||||
| 'invocation_pending'
|
||||
| 'persistence_pending'
|
||||
| 'history_persisted'
|
||||
| 'commit_conflict'
|
||||
| 'commit_indeterminate'
|
||||
| 'persistence_failed'
|
||||
| 'settled';
|
||||
checkpoint: Omit<IAgentEventActorCheckpoint, 'checkpointId'> & { checkpointId?: string };
|
||||
action: { toolName: string; toolCallId?: string };
|
||||
error?: string;
|
||||
/** How a retained receipt reached `settled`. Absent on active lifecycle rows. */
|
||||
resolution?: 'checkpoint_verified' | 'action_compensated' | 'history_repaired';
|
||||
observedAt: Date;
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable fence covering one legacy-path turn from before its execution until
|
||||
* its history is persisted. While present, no fork may execute or commit — the
|
||||
* turn's messages are not yet durable, so any rebuild would be incomplete. A
|
||||
* crash leaves the token in place (fail-closed) until it is reclaimed.
|
||||
*/
|
||||
export interface IAgentEventActorLegacyTurn {
|
||||
token: string;
|
||||
startedAt: Date;
|
||||
}
|
||||
|
||||
export interface IAgentEventActorSnapshot {
|
||||
state: IAgentEventActorState | null;
|
||||
reconciliations: IAgentEventActorReconciliation[];
|
||||
legacyTurn: IAgentEventActorLegacyTurn | null;
|
||||
/** Durable invalidation epoch. Every legacy-path event bumps it — including
|
||||
* for headless or already cold-marked actors, where the marker alone leaves
|
||||
* no CAS-visible trace — and the commit CAS requires the epoch observed at
|
||||
* preparation, so a stale fork can never commit state built from history
|
||||
* read before an intervening legacy turn. */
|
||||
epoch: number;
|
||||
}
|
||||
|
||||
export interface IAgentEventBindingRecord {
|
||||
conversationId: string;
|
||||
agentId: string;
|
||||
|
|
@ -75,6 +131,14 @@ export interface IConversation extends Document {
|
|||
subagentThreadLease?: ISubagentThreadLease;
|
||||
/** Internal event-source identity. Excluded from ordinary conversation reads. */
|
||||
agentEventBinding?: IAgentEventBinding;
|
||||
/** Internal event-actor checkpoint head. Excluded from ordinary conversation reads. */
|
||||
agentEventActor?: IAgentEventActorState;
|
||||
/** Private invocation proof: active lifecycle fences plus settled same-ID receipts. */
|
||||
agentEventActorReconciliations?: IAgentEventActorReconciliation[];
|
||||
/** Private invalidation epoch; see {@link IAgentEventActorSnapshot.epoch}. */
|
||||
agentEventActorEpoch?: number;
|
||||
/** Private in-flight legacy-turn fence; see {@link IAgentEventActorLegacyTurn}. */
|
||||
agentEventActorLegacyTurn?: IAgentEventActorLegacyTurn;
|
||||
assistant_id?: string;
|
||||
instructions?: string;
|
||||
stop?: string[];
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue