Commit graph

6 commits

Author SHA1 Message Date
Danny Avila
2d794a0439
💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539)
* fix: Detect silently dead Redis sockets with deadline-bounded heartbeats

A Redis peer can vanish without a FIN or RST (a dropped NAT entry, a proxy
failover, a migrated VM). The socket stays `ready` from ioredis's point of
view, no error fires, and every command on it waits for the kernel's
retransmission timeout — about fifteen minutes at Linux defaults.

Observed in production: while the main client's socket was dead,
POST /api/agents hung until Cloudflare returned a 524; while the subscriber's
socket was dead, every attach failed with "Timed out synchronizing Redis
subscription" for sixteen minutes across five conversations.

`REDIS_PING_INTERVAL` could not help — it awaits ping() with no deadline, so
the probe just joins everything else that is waiting.

- Add startRedisHeartbeat: races a PING against a deadline and destroys the
  socket when the probe goes unanswered, so ioredis reconnects, replays queued
  commands, and re-subscribes a subscriber's channels.
- Destroy the stream rather than disconnect(true), whose FIN a vanished peer
  never acknowledges; cluster clients fall back to the regular reconnect.
- Add createIoRedisSubscriber so every dedicated subscriber gets an error
  listener and its own heartbeat, and route the generation-stream and both
  subagent-routing subscribers through it.
- Set ioredis keepAlive from REDIS_KEEP_ALIVE (default 10s).
- Add REDIS_PING_TIMEOUT, REDIS_SUBSCRIBER_PING_INTERVAL and REDIS_KEEP_ALIVE,
  documented in .env.example.
- Give the fake RESP server a silent mode and a SUBSCRIBE reply so the
  dead-peer case can be tested against real clients.

* fix: Probe each cluster node and refuse a non-positive heartbeat deadline

A cluster-level PING is routed to one node, so a healthy reply proved nothing
about the other sockets. The heartbeat now resolves its targets each tick —
every node for a cluster, the client itself otherwise — and races an
independent deadline per target, tearing down only the socket that stops
answering. A non-positive deadline disables the heartbeat with a warning
instead of forcing a reconnect on every tick, and the single-probe test no
longer depends on wall-clock slack.
2026-09-03 09:45:16 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Danny Avila
d641c398d5
🧳 fix: Port Subagent Control Receipt Writes to DocumentDB-Safe Operators (#15171)
* fix: harden subagent control receipt persistence

* fix: harden durable subagent control replay

* fix: await terminal control receipts on shutdown

* fix: close subagent control replay races

* test: type stale-owner transport fixture

* fix: quiesce durable subagent controls

* test: await subagent shutdown durability boundary

* fix: serialize durable subagent controls

* fix: fail shutdown on cleanup errors

* fix: report cancellable result availability accurately

* fix: fence subagent control receipt ownership

* fix: close distributed control receipt races

* test: type control receipt race fixture

* chore: require authoritative control receipts

* fix: close subagent control lifecycle races

* style: separate control reservation member

* test: harden subagent settlement wait

* fix: preserve authoritative control replay state
2026-08-24 20:05:39 -04:00
Danny Avila
a997275902
🧾 feat: Persist Authoritative Subagent Control Receipts (#15168)
* feat: persist subagent control receipts

* fix: require control receipt persistence

* fix: preserve authoritative control history
2026-08-24 11:39:56 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
d3e70159ca
📡 feat: Stream Detached Subagent Activity (#15111)
* feat: stream detached subagent activity

* fix: annotate activity stream limits

* fix: isolate subagent activity imports

* fix: harden detached subagent activity lifecycle

* test: cover synchronous activity transport failure

* test: include required subagent activity identity

* fix: identify and reconnect subagent activity events

* fix: bound subagent activity lifecycles

* fix: close subagent activity handoff races

* fix: bind and synchronize activity subscriptions

* fix: detect fresh activity attachment

* fix: complete activity synchronization handoff

* fix: bind activity sync and failure circuits

* fix: expose subscription-bound synchronization

* fix: fence activity reconnect publications

* test: make detached timeout settlement deterministic

* fix: fence Redis activity attachments

* fix: close failed activity streams

* perf: reuse fenced activity frontier

* style: sort subagent thread imports

* fix: preserve queued subagent activity

* test: type activity publication counter

* fix: disconnect subagent activity subscriber

* fix: close background activity lifecycle gaps

* fix: preserve streamed activity spacing

* fix: preserve bounded live subagent activity

* fix: merge durable subagent activity safely

* fix: model detached activity coverage

* fix: type detached activity inputs

* fix: order overlapping subagent activity

* chore: sort activity test imports

* fix: buffer subagent activity handoff gaps

* fix: flush activity after parent close

* fix: advance closed activity suffixes

* fix: preserve detached activity ordering

* fix: close detached activity delivery races

* fix: bound shared Redis subscriber readiness

* fix: expire shared Redis subscription readiness

* fix: clean up late Redis subscriptions

* fix: preserve late Redis subscription fallback
2026-08-22 09:45:27 -04:00