Skip to content
Abdifatah Artan
All posts
GoDistributed SystemsPerformance

Building a High-Throughput Ingestion Pipeline in Go: Backpressure, Batching, and Exactly-Once

Jan 27, 20264 min read

A million events per second is not a big number until you have to receive, validate, buffer, and commit each one without losing any. This post walks through the patterns that matter — the ones that survive contact with production traffic — using Go as the language of choice.

Start with the failure model#

Before writing any code, write down what you're allowed to lose and what you're not:

  • Validation failures — discard, but count them.
  • Broker unavailable — buffer in memory, bounded, with a hard cap.
  • Consumer crash mid-batch — replays must converge (idempotency).
  • Slow downstream — degrade by lagging, never by dropping.

That last point is the entire thesis: every stage must degrade independently. Slow consumers lag; ingestion keeps accepting. Nothing upstream may block on anything downstream.

Worker pools with bounded queues#

The classic mistake is a goroutine per request with an unbounded channel. Under spike traffic, memory grows until the OOM killer decides your uptime for you. Bound everything:

pool.go
type Pool struct {
    jobs    chan Job
    results chan error
    wg      sync.WaitGroup
}
 
func NewPool(workers, queueSize int) *Pool {
    p := &Pool{
        jobs:    make(chan Job, queueSize),
        results: make(chan error, queueSize),
    }
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go p.work()
    }
    return p
}
 
func (p *Pool) work() {
    defer p.wg.Done()
    for job := range p.jobs {
        p.results <- job.Process()
    }
}
 
// Submit returns ErrOverloaded instead of blocking forever.
func (p *Pool) Submit(j Job) error {
    select {
    case p.jobs <- j:
        return nil
    default:
        return ErrOverloaded
    }
}

Two decisions here do the heavy lifting. First, Submit uses a non-blocking send — under overload the caller gets an explicit error it can retry or shed. Second, the queue bound turns "unbounded memory growth" into a measurable, alertable metric: queue depth.

Shedding is a feature, not a failure

Returning ErrOverloaded to 0.01% of requests during a spike is a business decision you should make deliberately — with a 503, a retry, and a metric — rather than accidentally, with an OOM kill.

Batching: latency for throughput#

Per-message writes are how you get 400 events/sec out of hardware that should do 40,000. Batch on two triggers — size and deadline — so you're never waiting long and never writing small:

batcher.go
type Batcher struct {
    mu      sync.Mutex
    buf     []Event
    maxSize int
    flushFn func([]Event) error
}
 
func (b *Batcher) Add(e Event) error {
    b.mu.Lock()
    b.buf = append(b.buf, e)
    full := len(b.buf) >= b.maxSize
    b.mu.Unlock()
 
    if full {
        return b.Flush()
    }
    return nil
}
 
// Flush is also called by a ticker every 50ms.
func (b *Batcher) Flush() error {
    b.mu.Lock()
    defer b.mu.Unlock()
    if len(b.buf) == 0 {
        return nil
    }
    batch := b.buf
    b.buf = b.buf[:0]
    return b.flushFn(batch)
}

The deadline flush is the part people forget: without it, low-traffic partitions hold events for seconds and p99 latency explodes while average looks great.

Exactly-once, practically#

Exactly-once is a contract between producer and consumer, not a property of the broker. The standard play is idempotent writes keyed by a deterministic event ID — a content hash computed at ingestion:

schema.sql
CREATE TABLE events (
  event_id   FixedString(32),
  account_id UInt64,
  kind       LowCardinality(String),
  payload    String,
  ingested_at DateTime64(3)
) ENGINE = ReplacingMergeTree
ORDER BY (account_id, ingested_at);
 
-- Replays collapse to one row; retries converge.

And on the consumer side, the rule is simply: commit the offset and the row in the same logical operation. With ClickHouse's ReplacingMergeTree, re-processing a batch after a crash overwrites rather than duplicates.

Test convergence, not just correctness

Property-test the consumer: run the same batch through it twice, crash it at random points, and assert the table converges. Idempotency bugs are almost impossible to find by reading the code.

What we measured#

After the rework, the same hardware that topped out at 60k events/sec was sustaining:

1.2MThroughputevents / sec, peak
87msp99 latencyingest → queryable
0Dropped eventsunder load tests
<1msGC pausep99, no tuning

The boring part that saved us#

The biggest operational win wasn't any of the above — it was per-stage queue-depth dashboards. Every channel in the system exports its depth as a gauge. When the consumer tier lagged, we could see exactly which stage was the bottleneck in one glance, usually before the first alert fired.

Wrapping up#

The patterns scale down as well as up. Bounded queues, deadline batching, and idempotent writes are maybe a hundred lines of Go combined — and they're the difference between a pipeline that runs and a pipeline that runs at 3 a.m. without you.