LibreChat/packages/api/src/cache
Danny Avila 2d794a0439
💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539)
* fix: Detect silently dead Redis sockets with deadline-bounded heartbeats

A Redis peer can vanish without a FIN or RST (a dropped NAT entry, a proxy
failover, a migrated VM). The socket stays `ready` from ioredis's point of
view, no error fires, and every command on it waits for the kernel's
retransmission timeout — about fifteen minutes at Linux defaults.

Observed in production: while the main client's socket was dead,
POST /api/agents hung until Cloudflare returned a 524; while the subscriber's
socket was dead, every attach failed with "Timed out synchronizing Redis
subscription" for sixteen minutes across five conversations.

`REDIS_PING_INTERVAL` could not help — it awaits ping() with no deadline, so
the probe just joins everything else that is waiting.

- Add startRedisHeartbeat: races a PING against a deadline and destroys the
  socket when the probe goes unanswered, so ioredis reconnects, replays queued
  commands, and re-subscribes a subscriber's channels.
- Destroy the stream rather than disconnect(true), whose FIN a vanished peer
  never acknowledges; cluster clients fall back to the regular reconnect.
- Add createIoRedisSubscriber so every dedicated subscriber gets an error
  listener and its own heartbeat, and route the generation-stream and both
  subagent-routing subscribers through it.
- Set ioredis keepAlive from REDIS_KEEP_ALIVE (default 10s).
- Add REDIS_PING_TIMEOUT, REDIS_SUBSCRIBER_PING_INTERVAL and REDIS_KEEP_ALIVE,
  documented in .env.example.
- Give the fake RESP server a silent mode and a SUBSCRIBE reply so the
  dead-peer case can be tested against real clients.

* fix: Probe each cluster node and refuse a non-positive heartbeat deadline

A cluster-level PING is routed to one node, so a healthy reply proved nothing
about the other sockets. The heartbeat now resolves its targets each tick —
every node for a cluster, the client itself otherwise — and races an
independent deadline per target, tearing down only the socket that stops
answering. A non-positive deadline disables the heartbeat with a warning
instead of forcing a reconnect on every tick, and the single-probe test no
longer depends on wall-clock slack.
2026-09-03 09:45:16 -04:00
..
__tests__ 💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539) 2026-09-03 09:45:16 -04:00
cacheConfig.ts 💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539) 2026-09-03 09:45:16 -04:00
cacheFactory.ts 🚢 fix: Reconnect Keyv Redis After Replica Failover (#15502) 2026-09-02 08:57:40 -04:00
heartbeat.ts 💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539) 2026-09-03 09:45:16 -04:00
index.ts feat: add low-noise Redis observability (#14309) 2026-07-16 10:22:03 -04:00
keyvFiles.ts
keyvMongo.ts
principals.ts feat: add low-noise Redis observability (#14309) 2026-07-16 10:22:03 -04:00
recovery.ts 🚢 fix: Reconnect Keyv Redis After Replica Failover (#15502) 2026-09-02 08:57:40 -04:00
redisClients.tls.spec.ts 🧬 chore: Upgrade Redis Dependencies to Dodge the ElastiCache BigInt Cursor (#15068) 2026-08-21 11:14:37 -04:00
redisClients.ts 💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539) 2026-09-03 09:45:16 -04:00
redisTelemetry.spec.ts feat: add low-noise Redis observability (#14309) 2026-07-16 10:22:03 -04:00
redisTelemetry.ts feat: add low-noise Redis observability (#14309) 2026-07-16 10:22:03 -04:00
redisUtils.spec.ts 🧬 chore: Upgrade Redis Dependencies to Dodge the ElastiCache BigInt Cursor (#15068) 2026-08-21 11:14:37 -04:00
redisUtils.ts 💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539) 2026-09-03 09:45:16 -04:00