Question 1

What is the overhead in Kafka messages?

Accepted Answer

Kafka Record V2 format adds approximately 7-21 bytes of metadata per record (varint-encoded fields for attributes, timestamp delta, offset delta, key length, value length, and optional headers). There's also a 61-byte RecordBatch header added once per batch. The exact overhead depends on the magnitudes of these fields and can vary significantly based on your message structure.

Question 2

How does compression affect message size?

Accepted Answer

Kafka applies compression to entire batches, not individual messages. The compression ratio varies by algorithm: gzip (typically 30% of original), zstd (50%), lz4 (80%), and snappy (85%). The producer estimates compressed size using a sliding-window weighted average of recent compression ratios (KIP-126). Real-world ratios depend on data compressibility and may differ from defaults.

Question 3

What's the difference between message size and batch size?

Accepted Answer

Message size is the size of a single record (payload + key + headers + ~14 bytes overhead). Batch size is multiple messages grouped together for efficiency, with an additional 61-byte batch header added once. Batching reduces overhead per message and improves throughput, but larger batches increase latency. The batch size determines network packet efficiency and storage allocation.

Question 4

How do I calculate network throughput?

Accepted Answer

Network throughput = messages per second × compressed message size. For example, 10,000 msg/s × 500 bytes = 5,000,000 bytes/s = 40 Mbps. The calculator accounts for compression effects. Remember that actual network utilization may be higher due to TCP/IP overhead, Kafka protocol framing, and network protocol inefficiencies.

Question 5

What are the default Kafka message size limits?

Accepted Answer

The default limit is 1 MiB (1,048,576 bytes) for `message.max.bytes` (broker), `max.request.size` (producer), and `max.message.bytes` (topic). These can be increased but larger messages consume more memory and network bandwidth. Messages larger than these limits will be rejected. Consider using external storage (S3, blob storage) for large payloads and keeping only references in Kafka.

Question 6

How can I estimate my actual message sizes?

Accepted Answer

Use the producer performance tool (`kafka-producer-perf-test`) with your actual data to measure real message sizes. Monitor the `record-size-avg` metric in Kafka metrics. For development, measure the serialized size of your message objects before encoding. Consider using a sample of production traffic to measure real-world sizes.

Kafka Message Size Calculator

Message Configuration

Compression Settings

Kafka Message Size Analysis

Size Breakdown (Per Message)

Kafka Config Limits Check

Frequently Asked Questions

Related Tools

API Rate Limit Calculator

System Latency Budget Calculator

Jira Story Points Calculator

Notepad

Share with friends