Go SQS Worker Platform
Some platform workloads - large transfers, batch requests, migration jobs - cannot share the synchronous API path without risking timeouts. This project introduced Go workers backed by Amazon SQS, raising throughput for high-concurrency operations while keeping user-facing APIs responsive under peak load.
Problem
Large batch transfers, migration jobs, and progress-tracking endpoints exceeded ALB idle timeouts when executed synchronously on Node.js API servers - users saw 504s while work continued orphaned in the background.
Before and After
Before
- Batch progress API blocked until job finished (timeouts)
- Peak enrollment windows starved API thread pools
- No standard pattern for long-running platform jobs
- Retry logic duplicated across endpoints
- Support could not see job state without engineer queries
After
- Enqueue to SQS; Go workers process with parallel I/O
- API returns job ID immediately; poll reads progress row
- Batch-progress endpoint under one second
- DLQ for poison messages after max receives
- CloudWatch queue age alarms for backlog growth
Go SQS Worker Platform
Long-running batch and transfer workloads enqueue to SQS; Go workers poll with visibility timeouts, parallelize I/O, and report progress asynchronously so API threads stay under one second.
API
Queue
Workers
Storage
What I Built
API enqueue path
- Validate auth, write job row to MySQL, send SQS message with job ID correlation.
- Idempotency keys prevent duplicate batch submissions from retries.
- Progress poll endpoint reads MySQL only - no long poll on request thread.
Go worker pool
- Goroutines parallelize S3 transfers and external API calls Node struggled to bound.
- Visibility timeout aligned to p99 handler duration.
- Graceful shutdown completes in-flight messages before exit.
Operations
- Terminal states: success, failed, cancelled - visible to support tooling.
- Structured logs with job ID across API and worker tiers.
- Peak-load work isolated from warm-pool API EC2 fleet.
Engineering Decisions
- Go over Node for workers - higher concurrency I/O for transfer-heavy jobs without blocking the event loop.
- SQS over Redis Streams here - longer job durations and AWS-native DLQ fit batch workloads better than stream pending reclaim.
- MySQL progress store - poll UX and admin dashboards need queryable state, not stream offsets alone.
- Separate worker fleet - scaling Go consumers does not require scaling user-facing API capacity.
By the Numbers
Batch progress API
<1s (was timing out)
Worker language
Go
Queue
Amazon SQS + DLQ
API response time cut
70% on daily-stats path
Job visibility
MySQL progress rows
Alerting
CloudWatch queue age
Reliability and Ops
- DLQ captures poison messages; manual replay after root-cause fix.
- Visibility timeout extension for jobs exceeding default window.
- API fleet protected during enrollment peaks - async tier absorbs burst.
Testing
- Staging load test with parallel batch submits and progress polling.
- Chaos: kill worker mid-job, verify message returns after visibility timeout.
- Integration test: enqueue -> worker completes -> poll returns success.
What I Did
- 01Introduced Go workers with Amazon SQS and queue-based processing for high-concurrency transfers and requests.
- 02Raised throughput and isolated peak-load work from synchronous APIs.
- 03Cut a batch-progress API from timeouts to under one second by moving work off the request path.