Moving from request-response architectures to event-driven systems is a significant shift. You gain decoupling, resilience, and the ability to process massive event volumes, but you also inherit complexity around ordering, exactly-once semantics, and operational visibility. Go’s lightweight concurrency model and Kafka’s partition-based design are a natural fit for this challenge.
In this guide, I’ll walk through the architectural decisions and implementation patterns that let you build microservices capable of handling millions of events per second, with concrete Go code examples and the trade-offs you’ll face in production.
Understanding Kafka Partitions and Consumer Groups
Kafka’s power comes from its partition model. Each topic is split into partitions, and each partition is an ordered, immutable log. Messages within a partition maintain order, but messages across partitions do not. This is fundamental to everything that follows.
A consumer group is a set of consumers that work together to read from a topic. Kafka automatically assigns partitions to consumers in the group. If you have 4 partitions and 3 consumers, one consumer will read from 2 partitions while the others read from 1 each. The key insight: only one consumer in a group reads from any given partition at a time.
This design means:
- Horizontal scaling is straightforward: add more consumers to a group to process faster.
- Order is guaranteed within a partition, but not globally across the topic.
- You can have multiple consumer groups reading the same topic independently.
If global ordering matters for your use case, you’d typically use a single partition (which limits parallelism) or accept that you’ll need to handle out-of-order events in your business logic.
Partition Strategy: Choosing Your Key
How you partition your events determines your throughput ceiling and your ordering guarantees. The partition key is the critical decision.
If you send a message to Kafka without a key, it goes to a random partition. If you include a key, Kafka hashes it and always sends messages with the same key to the same partition. This ensures ordering for related events.
Common strategies:
- User ID or Account ID: All events for a user go to the same partition. Good for maintaining user-level ordering. Scales as long as your event distribution is reasonably balanced.
- Entity ID: All events for an order, transaction, or resource go to the same partition. Similar benefits and trade-offs to user ID.
- Null key (random): Maximum throughput, no ordering guarantees. Use when you don’t need ordering or can handle it downstream.
- Composite key: Combine multiple fields (e.g., region + tenant + entity). Adds complexity but can improve load distribution.
The risk with any fixed key strategy is hot partitions. If one user generates 90% of your events, that partition becomes a bottleneck while others sit idle. Monitor partition lag and event volume per partition in production.
Building a Consumer in Go
Go’s goroutines make concurrent message consumption elegant. Here’s a minimal but production-ready consumer:
package main
import (
"context"
"fmt"
"log"
"sync"
"time"
"github.com/segmentio/kafka-go"
)
func main() {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: []string{"localhost:9092"},
Topic: "events",
GroupID: "my-service",
StartOffset: kafka.LastOffset,
CommitInterval: time.Second,
MaxBytes: 10e6,
})
defer reader.Close()
ctx := context.Background()
var wg sync.WaitGroup
// Start 4 concurrent message processors
for i := 0; i < 4; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for {
msg, err := reader.ReadMessage(ctx)
if err != nil {
log.Printf("read error: %v", err)
break
}
if err := processEvent(msg); err != nil {
log.Printf("consumer %d: process error: %v", id, err)
// Decide: retry, DLQ, or continue?
}
}
}(i)
}
wg.Wait()
}
func processEvent(msg kafka.Message) error {
fmt.Printf("partition: %d, offset: %d, key: %s, value: %s
",
msg.Partition, msg.Offset, string(msg.Key), string(msg.Value))
return nil
}
This example uses segmentio/kafka-go, a pure Go client. Key points:
- The
GroupIDparameter registers this consumer with a consumer group. Kafka tracks offsets per group. - Multiple goroutines call
ReadMessageconcurrently. The reader handles partition assignment and load balancing internally. - Offsets are committed automatically every second (configurable). This records “we’ve processed up to offset X” so if the consumer restarts, it resumes from there.
StartOffsetcontrols where to start if no committed offset exists.kafka.LastOffsetmeans start from the newest message.
This pattern scales easily: spawn more goroutines to process faster, or add more consumer instances to the group to spread the load across machines.
Exactly-Once Semantics and Offset Management
By default, Kafka offers at-least-once delivery. A message might be processed twice if a consumer crashes after processing but before committing its offset. For many use cases this is acceptable, but if you need exactly-once, you have options.
Option 1: Idempotent processing. Make your business logic idempotent so processing the same message twice produces the same result. Add a unique message ID, store processed IDs in a database, and skip if you’ve seen it before. Simple and often practical.
Option 2: Transactions. Kafka supports transactional writes and reads, but this adds latency and complexity. You’d coordinate offset commits with your business logic in a single transaction. Use when idempotency isn’t feasible.
Option 3: Manual offset management. Instead of auto-commit, explicitly commit offsets only after you’re certain the message has been fully processed and persisted. This gives you fine-grained control:
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: []string{"localhost:9092"},
Topic: "events",
GroupID: "my-service",
CommitInterval: 0, // Disable auto-commit
})
msg, _ := reader.ReadMessage(ctx)
if err := processAndStore(msg); err != nil {
// Don't commit; on restart we'll re-process this message
return err
}
// Only commit after successful processing
if err := reader.CommitMessages(ctx, msg); err != nil {
log.Printf("commit error: %v", err)
}
In production, I’d recommend starting with idempotent processing and auto-commit. It’s simpler, performs better, and handles 95% of real-world cases. Move to manual commits only if you hit a specific scenario where duplicates cause measurable problems.
Handling Failures and Consumer Lag
Failures are inevitable. A consumer might crash, a database might be slow, or an external API might timeout. How you handle this affects both reliability and performance.
Dead-letter queues are a common pattern: if processing fails after N retries, send the message to a separate topic for manual inspection. Here’s a sketch:
const maxRetries = 3
for attempt := 0; attempt < maxRetries; attempt++ {
if err := processEvent(msg); err == nil {
break // Success
}
if attempt == maxRetries-1 {
// Last attempt failed, send to DLQ
if err := sendToDLQ(msg, err); err != nil {
log.Printf("DLQ send failed: %v", err)
}
} else {
time.Sleep(time.Duration(math.Pow(2, float64(attempt))) * time.Second) // Exponential backoff
}
}
Monitor consumer lag: the difference between the latest offset in the topic and the latest offset your consumer has processed. High lag means you’re falling behind. It could indicate slow processing, a crashed consumer, or an undersized consumer group.
In Go, you can check lag programmatically:
stats := reader.Stats()
fmt.Printf("Lag: %d messages
", stats.Lag)
Set up alerts if lag exceeds a threshold (e.g., more than 100k messages). This often signals a problem before it becomes critical.
Scaling Patterns for Production
Once your consumer works locally, scaling to production involves several decisions.
Vertical vs. Horizontal Scaling
Vertical scaling means adding more goroutines or CPU to a single consumer instance. It’s simple and works well up to a point. Horizontal scaling means adding more instances to your consumer group.
Kafka handles horizontal scaling automatically through consumer group rebalancing. When you add a new consumer instance, Kafka detects it and redistributes partitions. This is transparent but does cause a brief pause in processing (seconds, typically).
In production, I’d use both: run 4 to 8 goroutines per instance for CPU-bound work, and add more instances as volume grows. This balances simplicity with resilience.
Batch Processing vs. Real-Time
Processing one message at a time is simple but can be inefficient if your downstream (database, API) supports batching. Go makes batching straightforward with channels:
batch := make([]kafka.Message, 0, 100)
ticker := time.NewTicker(5 * time.Second)
for {
select {
case msg := <-msgChan:
batch = append(batch, msg)
if len(batch) >= 100 {
if err := processBatch(batch); err != nil {
log.Printf("batch error: %v", err)
}
batch = batch[:0]
}
case <-ticker.C:
if len(batch) > 0 {
if err := processBatch(batch); err != nil {
log.Printf("batch error: %v", err)
}
batch = batch[:0]
}
}
}
Batching reduces latency per message but increases end-to-end latency. For real-time analytics, process one at a time. For daily reports or bulk inserts, batch aggressively.
Deployment and Configuration
In Kubernetes or Docker, your consumer is typically a stateless deployment. Kafka handles state (offsets) for you. This means you can scale up and down freely.
Configuration to tune:
- Fetch size: How much data the consumer pulls per request. Larger means more throughput but higher memory. Default is 1MB; adjust based on message size and available memory.
- Fetch min bytes: Wait until at least this many bytes are available before responding to a fetch request. Reduces CPU but increases latency.
- Session timeout: How long before Kafka considers a consumer dead and rebalances. Too short causes unnecessary rebalances; too long delays failure detection. Default 10 seconds is usually fine.
- Commit interval: How often to commit offsets. More frequent means better recovery but more broker load. 1 second is a common default.
Start with defaults and measure. Profiling is your friend: Go’s pprof tool makes it easy to identify CPU and memory bottlenecks in your consumer.
Monitoring and Observability
You can’t fix what you can’t see. Instrument your consumer with metrics:
- Messages processed per second: Throughput indicator.
- Processing latency (p50, p95, p99): How long does each message take to process?
- Consumer lag: Are you keeping up with producers?
- Rebalance events: How often is Kafka redistributing partitions?
- Error rate: What fraction of messages fail to process?
Use Prometheus or a similar metrics system. Here’s a basic example with the Prometheus Go client:
import "github.com/prometheus/client_golang/prometheus"
var (
messagesProcessed = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "kafka_messages_processed_total",
},
[]string{"topic", "group"},
)
processingDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "kafka_processing_duration_seconds",
},
[]string{"topic"},
)
)
func processEvent(msg kafka.Message) error {
start := time.Now()
err := doWork(msg)
duration := time.Since(start).Seconds()
if err == nil {
messagesProcessed.WithLabelValues(msg.Topic, "my-group").Inc()
}
processingDuration.WithLabelValues(msg.Topic).Observe(duration)
return err
}
Pair metrics with structured logging. Log the offset, partition, and any error context. This makes debugging production issues much faster.
Real-World Trade-Offs
Throughput vs. Latency: Batching and larger fetch sizes improve throughput but add latency. If you need sub-100ms processing, keep batches small. If you’re processing millions of events for a daily report, batch aggressively.
Ordering vs. Scalability: Per-partition ordering (via partition keys) limits your parallelism. If you need strict global ordering, you’re essentially limited to one partition, which limits Kafka’s scaling model. Accept eventual consistency or distribute ordering to your application layer.
Consistency vs. Availability: Exactly-once semantics add complexity and latency. At-least-once with idempotent processing is simpler and usually sufficient. Only pursue exactly-once if you have a specific, measurable requirement.
Operational Complexity vs. Performance: Manual offset management, custom retry logic, and transactional semantics all add operational overhead. Start simple, measure, and only add complexity when you hit a concrete problem.
Wrapping Up
Event-driven microservices with Go and Kafka are powerful but require thoughtful design. Start by understanding your partition strategy and consumer group model. Build a simple consumer, test it, and measure. Use Go’s concurrency primitives to scale horizontally. Monitor lag and errors. And remember: the best architecture is the one you can understand and operate in production.
The patterns here (idempotent processing, DLQs, lag monitoring, batching) apply to any event streaming system. Once you’ve built one solid consumer, the next ones are faster.
What’s the difference between a partition key and a consumer group?
A partition key determines which partition a message goes to within a topic. Messages with the same key always go to the same partition, ensuring ordering. A consumer group is a set of consumers that work together to read from a topic. Kafka automatically assigns partitions to consumers in the group. They serve different purposes: the key controls message routing, while the group controls consumption parallelism.
Can I have more consumers than partitions in my consumer group?
Yes, but the extra consumers will be idle. Kafka assigns partitions to consumers one-to-one. If you have 4 partitions and 6 consumers in a group, 2 consumers will have no partitions assigned and won’t process any messages. To utilize all consumers, increase your partition count, but this also increases operational complexity.
How do I handle slow or failing message processing?
Use a combination of retries with exponential backoff and a dead-letter queue. If a message fails after N retries (typically 3), send it to a separate topic for manual inspection. Monitor your DLQ regularly. For slow processing, consider batching messages if your downstream system supports it, or add more consumer instances to your group.
What’s consumer lag and why does it matter?
Consumer lag is the difference between the latest offset available in the topic and the latest offset your consumer has processed. High lag means you’re falling behind. It indicates slow processing, a crashed consumer, or an undersized consumer group. Monitor lag continuously and set alerts if it exceeds your SLA. A lag of 100k messages might be acceptable for a daily batch job but unacceptable for real-time analytics.
Should I use exactly-once or at-least-once semantics?
Start with at-least-once delivery and make your processing idempotent. This is simpler, faster, and handles 95% of real-world cases. Only move to exactly-once (via transactions or deduplication) if you have a specific requirement and can measure the trade-off. Exactly-once adds latency and operational complexity that often isn’t worth it.
