The problem#
The product's analytics backend was a single Node.js service writing rows into PostgreSQL. At 40k events per second it was fine. At 300k it started dropping writes, and the ops team was paged weekly during traffic spikes.
We needed a pipeline that could absorb order-of-magnitude growth, survive backpressure without losing data, and keep query latency flat as volume grew.
Architecture#
EventFlow is a three-stage pipeline: stateless ingestion servers validate and batch events, Kafka buffers them with replication, and consumer groups deduplicate, enrich, and commit to ClickHouse.
Design constraint
Nothing upstream of the consumer could be allowed to block. Every stage had to degrade independently — if ClickHouse slowed down, consumers lag; ingestion never stops accepting writes.
The hot path#
The ingestion server is deliberately boring. No locks, no shared state — each request is validated, assigned a deterministic partition key, and appended to a per-partition batch that flushes on size or deadline:
type Batcher struct {
mu sync.Mutex
buf []Event
topic string
writer *kafka.Writer
}
func (b *Batcher) Add(e Event) error {
b.mu.Lock()
b.buf = append(b.buf, e)
b.mu.Unlock()
return nil
}
// Flush runs on a timer and on batch-size thresholds.
func (b *Batcher) Flush() error {
b.mu.Lock()
defer b.mu.Unlock()
if len(b.buf) == 0 {
return nil
}
msgs := make([]kafka.Message, 0, len(b.buf))
for _, e := range b.buf {
msgs = append(msgs, kafka.Message{
Key: []byte(e.PartitionKey),
Value: e.Bytes(),
})
}
b.buf = b.buf[:0]
return b.writer.WriteMessages(context.Background(), msgs...)
}Two details matter more than they look. First, PartitionKey is the account
ID, so every event for one tenant lands on a single partition — ordering is
free and per-tenant hot shards are trivial to find. Second, the flush path is
the only place that can fail, and it retries with exponential backoff while
the batch stays in memory.
Exactly-once, practically#
"Exactly-once" is a contract between the producer and the consumer, not a property of the broker. We implement it with idempotent keys:
CREATE TABLE events (
event_id FixedString(32),
account_id UInt64,
kind LowCardinality(String),
payload String,
ingested_at DateTime64(3),
PRIMARY KEY (event_id, ingested_at)
) ENGINE = ReplacingMergeTree
ORDER BY (account_id, ingested_at);Consumers insert with ReplacingMergeTree, so replays after a crash collapse
to the same row. The event_id is a content hash computed at ingestion —
deterministic, so retries converge.
What I'd do differently
Kafka gave us ordering and replay for free, but running it well is a real operational commitment. For a v2 I'd seriously evaluate a managed streaming layer (or NATS JetStream for smaller deployments) and spend the reclaimed effort on consumer-side testing.
What made it work#
- Backpressure everywhere. Every stage has explicit limits — bounded batches, bounded channel queues, bounded connection pools.
- Deterministic partitioning made debugging a ten-thousand-foot problem into a single-partition problem.
- Load testing in production shape. We replayed a month of real traffic at 10x before cutting over, which surfaced three bugs that unit tests never would.