Kafka Message Size Calculator
Calculate Kafka message size, batch size and throughput with compression, then the disk a topic needs with replication and retention.
Compression
More optionsbatch 100 · key 0 B · hdr 0 B
Ratio = compressed ÷ original. Defaults are rough figures for JSON/log-like data (zstd 0.28, gzip 0.30, lz4 and snappy 0.45): measure your own payloads.
Disk for 7 days at replication factor 3
~4.2 TBon disk
18.6 Mbps produce · 55.7 Mbps incl. replication
- Payload 500 B
- Overhead 14 B
- Per batch
- 23.19 KB
- Written per day (one copy)
- 200.4 GB
With the default batch.size (16,384 B) batches close at ~70 records, not 100: raise batch.size (and linger.ms) to match.
How it’s worked outDecimal units: 1 MB = 1,000,000 B, so MB/s × 8 = Mbps
- Record = payload + key + headers + ~14 B of varint metadata. A batch adds a 61 B header once.
- Kafka compresses whole batches, so the ratio is applied to the records in a batch; the batch header stays uncompressed.
- Produce throughput = messages/s × the per-message share of the compressed batch. Followers fetch a full copy each, so with replication the cluster takes in RF × that.
- Disk = bytes written per day × retention days × replication factor, before index files and free-space headroom (plan for ~30–50% spare).
How this works: Method, 3 sources,
How this works
Method
A record is payload + key + headers + about 14 bytes of metadata (plus 1–2 bytes for the key length), and each batch adds a 61-byte header; the compression ratio (yours, or a default per codec) applies to the records in a batch, since Kafka compresses whole batches. Throughput is messages per second × each message’s share of the compressed batch, and disk is bytes per day × retention days × replication factor.
Sources
- Kafka record batch format (Record v2) (Apache Kafka)
- Squeezing the firehose (Kafka compression measurements) (Cloudflare, 2018)
- facebook/zstd README, Silesia corpus benchmarks
Changes
- Redesigned with a record-to-disk picture; compression now applies per batch as Kafka does, default ratios come from published measurements, and replication and retention size the disk.
- Fixed the request-size check to use the compressed batch, which is what the producer actually sends.
- Results now update as you type, starting from a filled-in example.
- First published.
Frequently Asked Questions
What is the overhead in Kafka messages?
Kafka Record V2 format adds approximately 7-21 bytes of metadata per record (varint-encoded fields for attributes, timestamp delta, offset delta, key length, value length, and optional headers). There's also a 61-byte RecordBatch header added once per batch. The exact overhead depends on the magnitudes of these fields and can vary significantly based on your message structure.
How does compression affect message size?
Kafka applies compression to entire record batches, not individual messages, so the batch header stays uncompressed and bigger batches compress better. Rough ratios for JSON/log-like data: zstd ~28% of original, gzip ~30%, lz4 and snappy ~45% (from Cloudflare's Kafka measurements and the zstd benchmark tables); your data may differ a lot. The producer estimates compressed size using a sliding-window weighted average of recent compression ratios (KIP-126). Real-world ratios depend on data compressibility and may differ from defaults.
What's the difference between message size and batch size?
Message size is the size of a single record (payload + key + headers + ~14 bytes overhead). Batch size is multiple messages grouped together for efficiency, with an additional 61-byte batch header added once. Batching reduces overhead per message and improves throughput, but larger batches increase latency. The batch size determines network packet efficiency and storage allocation.
How do I calculate network throughput?
Network throughput = messages per second × compressed message size. For example, 10,000 msg/s × 500 bytes = 5,000,000 bytes/s = 40 Mbps. The calculator accounts for compression effects. Remember that actual network utilization may be higher due to TCP/IP overhead, Kafka protocol framing, and network protocol inefficiencies.
What are the default Kafka message size limits?
The default limit is 1 MiB (1,048,576 bytes) for `message.max.bytes` (broker), `max.request.size` (producer), and `max.message.bytes` (topic). These can be increased but larger messages consume more memory and network bandwidth. Messages larger than these limits will be rejected. Consider using external storage (S3, blob storage) for large payloads and keeping only references in Kafka.
How can I estimate my actual message sizes?
Use the producer performance tool (`kafka-producer-perf-test`) with your actual data to measure real message sizes. Monitor the `record-size-avg` metric in Kafka metrics. For development, measure the serialized size of your message objects before encoding. Consider using a sample of production traffic to measure real-world sizes.