High-Concurrency Event Processing Framework
Edmingle's platform handles sharp traffic spikes during enrollments and signups. This framework moves burst-write work off synchronous APIs into a staged Redis Streams pipeline with asynchronous workers, retry semantics, and operational alerting - a pattern the backend team could reuse across domains.
Problem
Edmingle enrollment, signup, and social features generate burst-write traffic that cannot stay on synchronous API threads without DB lock contention, elevated P95 latency, and fragile manual retries during peak class hours.
Before and After
Before
- Heavy writes executed inline on API request path
- Enrollment spikes caused lock contention and slow responses
- No reusable pattern for new async domains
- Failure handling mixed with HTTP timeout limits
- Limited visibility into backlog depth during incidents
After
- Redis Streams backbone with domain-specific consumer groups
- Sub-millisecond enqueue; workers absorb burst writes
- Reusable Node.js worker framework with reclaim + retry
- Slack alerts and structured logs per stream/group
- XPENDING / XINFO lag introspection without new tooling
Redis Streams Worker Pipeline
Burst-write domains enqueue to Redis Streams; consumer groups pull with XREADGROUP, acknowledge on success, reclaim idle pending messages, and route failures to retry or dead-letter handling.
API
Stream
Workers
Ops
What I Built
Stream producer contract
- APIs XADD typed events after validation - return 202-style fast path once durably enqueued.
- Enrollment, signup, and social each use dedicated streams to isolate failure domains.
- MULTI/EXEC pairs event write with cache invalidation in one round trip.
Consumer group workers
- XREADGROUP BLOCK with competing consumers per domain worker pool.
- XACK on success; exponential backoff + dead-letter routing on repeated handler failure.
- Idle reclaim (XAUTOCLAIM / XPENDING) recovers messages from crashed workers during deploys.
Operational layer
- Health endpoint exposes consumer group lag and pending counts.
- Slack notifications on sustained pending growth or handler error storms.
- Framework documented so backend team could add new consumers without reinventing retry semantics.
Engineering Decisions
- Redis Streams over SQS - team already operated ElastiCache; Streams added near-zero ops overhead vs a new AWS service.
- Build worker framework - SQS gives DLQ/retry for free; we invested in reclaim, logging, and domain templates on Redis.
- Separate streams per domain - blast-radius isolation beats one mega-stream with complex routing logic.
- Keep enqueue on API tier - workers run on existing compute; no new cluster required for v1.
By the Numbers
Event volume
Millions/month
Enqueue latency
Sub-ms on ElastiCache
Domains
Signup · enrollment · social
Ops tooling
redis-cli XINFO / XPENDING
Incremental infra cost
$0 on existing node
Framework runtime
Node.js workers
Reliability and Ops
- Consumer reclaim prevents stuck pending messages after deploys or OOM kills.
- Handler idempotency required for domains with at-least-once delivery semantics.
- Blog post documents when to choose SQS instead for orgs without existing Redis ops.
Testing
- Integration tests against Redis test container for XADD/XREADGROUP/XACK flows.
- Staging soak with synthetic enrollment bursts before peak season.
- Runbook: check pending count, reclaim idle, inspect Slack error thread before scaling workers.
What I Did
- 01Built a staged Redis Streams worker framework for signup, enrollment, and social workloads with locking semantics and retry handling.
- 02Added logging and Slack-based error alerts to reduce database-lock risk under burst traffic.
- 03Documented the architecture so additional consumers could be implemented by the backend team.