20/05/2024 - GO, KAFKA
One of the ways for preserving message ordering with Kafka is use of a single partition topic with idempotency enabled in producers. Idempotent producers aim to deliver messages once without duplications even after retries. The broker assigns a unique Producer ID (PID) to each producer. Additionally, each message gets an increasing sequence number. You can achieve idempotent producers with setting below.
enable.idempotence = true
acks = all
max.in.flight.requests.per.connection = 1
If you use Sarama client for Golang, it looks like below.
cfg := sarama.NewConfig()
cfg.Producer.Idempotent = true
cfg.Producer.RequiredAcks = sarama.WaitForAll
cfg.Net.MaxOpenRequests = 1
...
If you have a topic with multiple partitions, you can't configure Kafka to maintain ordering across partitions. Kafka does not guarantee ordering within a topic, only within a partition. In other words, for the consumer, message order is not maintained across topic partitions, it is only maintained per partition. You will have to find a way to accomplish ordering.