Distributed System Design: High-Throughput Fan-Out Patterns & Event Messaging

In distributed architectures, fan-out is the core design pattern used to decouple a single event producer from multiple downstream processing pipelines. Whether you are broadcasting order creation events, handling real-time social feed ingestion, or triggering parallel audit logs, fan-out guarantees that a single write can trigger $N$ independent asynchronous operations concurrently.

1. System Topology: SNS to SQS Distributed Fan-Out

2. Architectural Comparison: Fan-Out on Write vs. Fan-Out on Read

Selecting the wrong fan-out strategy is one of the most common causes of cascading database failures under traffic spikes.

Fan-Out on Write (Push Model)

When an event occurs, the producer (or dispatcher) immediately broadcasts the payload to every subscriber's queue. Each worker materializes the result into localized storage.

• Pros: Read requests are near-instant ($O(1)$ lookup time) because data is pre-computed.
• Cons: Write amplification ($O(N)$ writes for $N$ subscribers). High-cardinality producers (e.g. users with millions of followers) create extreme write spikes.

Fan-Out on Read (Pull Model)

Events are recorded into a centralized append-only log. Downstream services fetch and aggregate data only when a read request is received.

• Pros: Write phase is extremely fast ($O(1)$ append).
• Cons: High read latency ($O(K)$ query cost to aggregate $K$ sources at query time).

3. Mathematical Model of Buffer Capacity & Consumer Scale

To prevent message loss during burst traffic, queue depth $Q(t)$ is governed by the rate of producer emission $\lambda(t)$ minus total worker consumption capacity:

Where:

• $\lambda(t)$ is the incoming message rate per second.
• $\mu$ is the processing capacity of a single worker instance ($\text{msgs/sec}$).
• $W$ is the current auto-scaled worker count.

If $\lambda(t) > \mu \cdot W$, the queue buffers messages without dropping requests. To prevent infinite memory growth, visibility timeouts and Dead Letter Queues (DLQs) isolate poison-pill messages after $K$ failed retries.

4. Implementing Idempotent Fan-Out Dispatchers

Because message queues guarantee at-least-once delivery, workers MUST be strictly idempotent. Below are production implementations in TypeScript, Python, and Go:

1import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";
2import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";
3import { Redis } from "ioredis";
4
5const sns = new SNSClient({ region: "us-east-1" });
6const redis = new Redis(process.env.REDIS_URL!);
7
8export interface OrderEvent {
9  eventId: string;
10  orderId: string;
11  amount: number;
12  currency: string;
13  timestamp: string;
14}
15
16// 1. Publish Event to Fan-Out Topic
17export async function publishOrderEvent(event: OrderEvent, topicArn: string): Promise<string> {
18  const command = new PublishCommand({
19    TopicArn: topicArn,
20    Message: JSON.stringify(event),
21    MessageDeduplicationId: event.eventId,
22    MessageGroupId: event.orderId,
23  });
24
25  const response = await sns.send(command);
26  return response.MessageId!;
27}
28
29// 2. Idempotent Consumer Worker with Redis Deduplication
30export async function processBillingQueueMessage(message: { Body?: string; ReceiptHandle?: string }): Promise<void> {
31  if (!message.Body) return;
32  
33  const event: OrderEvent = JSON.parse(message.Body);
34  const lockKey = `idempotency:billing:${event.eventId}`;
35
36  // Atomic SETNX (Set if Not Exists) with 24h TTL
37  const acquired = await redis.set(lockKey, "PROCESSING", "EX", 86400, "NX");
38  
39  if (!acquired) {
40    console.log(`[Idempotency] Event ${event.eventId} already processed or in-flight. Skipping.`);
41    return;
42  }
43
44  try {
45    // Process payment transaction
46    await executePaymentCharge(event);
47    await redis.set(lockKey, "COMPLETED", "EX", 86400);
48  } catch (err) {
49    await redis.del(lockKey); // Release lock for retry
50    throw err;
51  }
52}
53
54async function executePaymentCharge(event: OrderEvent): Promise<void> {
55  // Payment gateway logic...
56}

5. Key Resilience Principles

1. Dead Letter Queue (DLQ) Isolation: Route poison-pill messages to a DLQ after $K=3$ consecutive failures to prevent blocking worker pools.
2. Exponential Backoff with Jitter: Avoid thundering herd problems on downstream databases when recovering from an outage: $$T_{\text{sleep}} = \min\left(T_{\text{max}}, \; T_{\text{base}} \times 2^{\text{attempt}}\right) + \text{rand}(0, \text{jitter})$$
3. Idempotency Key Enforcement: Store processed message IDs in Redis or Postgres with TTL to prevent duplicate execution during network retries.