Kafka Message Size Calculator

Kafka Record V2 format adds approximately 7-21 bytes of metadata per record (varint-encoded fields for attributes, timestamp delta, offset delta, key length, value length, and optional headers). There's also a 61-byte RecordBatch header added once per batch. The exact overhead depends on the magnitudes of these fields and can vary significantly based on your message structure.

Kafka applies compression to entire batches, not individual messages. The compression ratio varies by algorithm: gzip (typically 30% of original), zstd (50%), lz4 (80%), and snappy (85%). The producer estimates compressed size using a sliding-window weighted average of recent compression ratios (KIP-126). Real-world ratios depend on data compressibility and may differ from defaults.

Message size is the size of a single record (payload + key + headers + ~14 bytes overhead). Batch size is multiple messages grouped together for efficiency, with an additional 61-byte batch header added once. Batching reduces overhead per message and improves throughput, but larger batches increase latency. The batch size determines network packet efficiency and storage allocation.

Network throughput = messages per second × compressed message size. For example, 10,000 msg/s × 500 bytes = 5,000,000 bytes/s = 40 Mbps. The calculator accounts for compression effects. Remember that actual network utilization may be higher due to TCP/IP overhead, Kafka protocol framing, and network protocol inefficiencies.

The default limit is 1 MiB (1,048,576 bytes) for `message.max.bytes` (broker), `max.request.size` (producer), and `max.message.bytes` (topic). These can be increased but larger messages consume more memory and network bandwidth. Messages larger than these limits will be rejected. Consider using external storage (S3, blob storage) for large payloads and keeping only references in Kafka.

Use the producer performance tool (`kafka-producer-perf-test`) with your actual data to measure real message sizes. Monitor the `record-size-avg` metric in Kafka metrics. For development, measure the serialized size of your message objects before encoding. Consider using a sample of production traffic to measure real-world sizes.