Redis Streams vs SQS for Event Backbones

At Edmingle, we needed an event backbone for course enrollment workflows, notification dispatch, and analytics aggregation. SQS was the obvious AWS default. But we already ran ElastiCache Redis for caching and rate limiting, and our team had deep Redis operational experience. We chose Redis Streams and built a worker framework on top of it.

Three years later, that framework processed millions of events with sub-second latency. It also taught me exactly where Redis Streams shines and where I would reach for SQS instead.

Why We Chose Redis Streams at Edmingle

The decision was pragmatic, not ideological. We already operated Redis in production with monitoring, backup, and on-call runbooks. Adding Streams to an existing cluster had near-zero operational overhead. SQS would have been a new service to monitor, a new IAM policy surface, and a new billing line item for a team of eight engineers.

Our consumer pattern was worker pools pulling jobs with acknowledgment - classic competing consumers. Redis Streams consumer groups map directly to this model. We did not need SNS fan-out, FIFO ordering across unrelated queues, or dead-letter queues with automatic redrive.

Producer → XADD stream:events * type enrollment payload {...}
                    │
                    ▼
         ┌── Consumer Group: enrollment-workers ──┐
         │  Worker 1 (pending: 0)                  │
         │  Worker 2 (pending: 2) ← reclaim idle     │
         │  Worker 3 (pending: 0)                  │
         └─────────────────────────────────────────┘
                    │
                    ▼ XACK on success · XPENDING on timeout

The Worker Framework We Built

We built a Node.js worker framework on top of Redis Streams that handled what SQS gives you for free: consumer group management, pending message reclaim, retry with backoff, dead-letter routing, and graceful shutdown.

// Worker framework (conceptual - Edmingle pattern)
class StreamWorker {
  constructor({ redis, stream, group, consumer, handler }) {
    this.redis = redis;
    this.stream = stream;
    this.group = group;
    this.consumer = consumer;
    this.handler = handler;
  }

  async start() {
    await this.ensureGroup();
    while (this.running) {
      const messages = await this.redis.xreadgroup(
        "GROUP", this.group, this.consumer,
        "COUNT", 10, "BLOCK", 5000,
        "STREAMS", this.stream, ">",
      );
      for (const [id, fields] of messages) {
        try {
          await this.handler(fields);
          await this.redis.xack(this.stream, this.group, id);
        } catch (err) {
          await this.handleFailure(id, fields, err);
        }
      }
      await this.reclaimIdle(60_000);
    }
  }
}

The framework added structured logging, OpenTelemetry spans per message, and a /health endpoint that checked consumer group lag. This was the real investment - not choosing Redis over SQS, but building the operational layer.

Where Redis Streams Won

Sub-millisecond enqueue latency. Our enrollment events needed to trigger notification workers within seconds, not minutes. Redis Streams consistently delivered single-digit millisecond publish latency on the same cluster we used for caching.

Atomic multi-stream operations. We used MULTI/EXEC to write an event and update a related cache key in one round trip. SQS cannot do this.

Consumer group introspection. XPENDING, XINFO GROUPS, and XLEN gave us real-time lag visibility without a separate CloudWatch dashboard. During incidents, redis-cli was faster than the AWS console.

Cost at our volume. Millions of events per month on an existing r6g.large node cost us nothing incremental. Equivalent SQS volume would have been roughly $40-80/month - not huge, but the operational simplicity of one less service mattered more.

Where SQS Would Have Been Better

Durability guarantees. SQS persists to disk across AZs by default. Redis Streams durability depends on AOF/RDB persistence configuration and cluster topology. We accepted this risk because our events were recoverable from MySQL source records - the stream was a delivery mechanism, not the source of truth.

Dead-letter handling. SQS DLQ with automatic redrive is turnkey. Our framework implemented DLQ as a separate stream (stream:dlq) with manual replay tooling. It worked, but it was code we had to maintain.

Scale ceiling. Redis Streams throughput is bounded by single-node memory and CPU. At Edmingle's volume (low millions/day), we were fine. At tens of thousands of messages per second, I would split streams across cluster shards or move to SQS/Kafka.

Cross-account and cross-region. SQS integrates natively with Lambda, SNS, and cross-account IAM. Redis Streams requires VPN or VPC peering for cross-account access - fine within one VPC, painful across accounts.

Operational Lessons from Production

Set MAXLEN ~ on streams to prevent unbounded memory growth. We used approximate trimming at 100,000 entries per stream - old messages were already processed and logged to MySQL.

Run reclaim loops aggressively. XPENDING messages with idle time > 60 seconds get reclaimed by a background process. Without this, a crashed worker silently stalls the consumer group.

Monitor stream length and consumer group lag as first-class metrics. I alert when lag exceeds 1,000 messages or oldest pending message age exceeds 5 minutes.

Treat the stream as a delivery pipe, not a database. Every event should be reconstructable from the source system. If Redis loses data, you replay from MySQL - not from the stream.

Engineering Takeaways

• If you already operate Redis at scale, Streams is a credible event backbone with near-zero incremental ops cost.
• Build a worker framework for reclaim, retry, DLQ, and graceful shutdown - SQS gives these for free, Redis does not.
• Redis Streams wins on latency, atomicity, and introspection; SQS wins on durability, DLQ, and scale ceiling.
• Always set MAXLEN ~ on streams and run pending message reclaim loops.
• Treat streams as delivery pipes - source of truth stays in MySQL or S3.
• At tens of thousands of msg/sec or cross-account fan-out, migrate to SQS or Kafka regardless of Redis comfort.

Trade-offs I Would Consider

Option: Redis Streams on existing ElastiCache cluster

• Pros: Zero new infrastructure, sub-ms latency, atomic ops with cache, team already knows Redis ops.
• Cons: Memory-bound retention, manual DLQ/reclaim, durability depends on persistence config.

Option: SQS with Lambda or ECS workers

• Pros: Managed durability, native DLQ, infinite scale, AWS-native integrations.
• Cons: Higher latency, new service to monitor, per-message cost, no atomic multi-operation.

What I Would Do in Production

Before choosing, inventory your existing Redis operations maturity. If you already have Redis on-call runbooks and monitoring, Streams is cheap to add. If Redis is a black box, start with SQS.

Implement pending message reclaim from day one - not after your first silent stall incident.

Export stream length, consumer group lag, and pending count to CloudWatch via a sidecar or cron. redis-cli is great for debugging but not for alerting.

Cap stream length with MAXLEN ~ and document the replay procedure from source records.

Re-evaluate at 10x current volume. The Edmingle framework worked at millions/day; I would revisit the decision at tens of millions.

FAQ

Q: Can I use Redis Streams with Redis Cluster mode?

Yes, but all consumers for a stream must connect to the same shard. Design stream keys with hash tags ({enrollment}:events) to co-locate related streams on one shard.

Q: How do I handle poison messages in Redis Streams?

Track retry count in message fields. After N failures, XACK the original and XADD to a DLQ stream. Build a replay CLI that reads DLQ and re-publishes to the main stream.

Q: Is Redis Streams durable enough for payment events?

Not as the sole record. Use MySQL or S3 as source of truth for financial events. Redis Streams delivers the notification; if it loses a message, you replay from the authoritative store.

Q: Should I use Bull/BullMQ instead of raw Streams?

BullMQ is built on Redis and gives you DLQ, retries, and dashboards out of the box. I used raw Streams at Edmingle because we wanted full control. For most teams, BullMQ is the faster path.

Q: When should I skip both and use Kafka?

When you need event replay as a first-class feature, multiple consumer groups reading the same events independently at high throughput, or cross-team event contracts with schema registry. That is Kafka territory.