Binary protocols encode messages as compact byte sequences, achieving lower overhead, faster parsing, and smaller payloads than text-based formats (JSON, XML). Designing a binary protocol requires decisions about framing (how to delimit messages), encoding (how to represent data types), versioning (how to evolve the protocol), and error detection. Binary protocols power gRPC, Kafka, Redis RESP, and database wire protocols.
Framing
Framing delimits message boundaries in a byte stream. Length-prefixed framing: each message is prefixed with its length (4-byte big-endian int32). The receiver reads the length first, then reads exactly that many bytes. Simple, efficient, and the dominant approach in binary protocols (gRPC HTTP/2 DATA frames, Kafka message protocol). Fixed-size messages: simpler but inflexible. Delimiter-based (scan for sentinel bytes): fragile if payload contains the delimiter. Always prefer length-prefixed framing.
Type Encoding
Primitive types: integers encoded as fixed-width little-endian or big-endian (4 bytes for int32, 8 bytes for int64, big-endian is network standard). Variable-length integers (varint): LEB128 or Protocol Buffer varint encoding uses 1 byte for values 0-127, 2 bytes for 128-16383, etc. Varints reduce payload size for small integers at the cost of variable-size parsing. Strings: length-prefixed UTF-8 bytes. Booleans: 1 byte (0 or 1). Floating-point: IEEE 754 binary32/binary64.
Protocol Versioning
Include a version field in the message header: magic number (identifies the protocol) + version byte + message type byte + length + payload. The magic number distinguishes your protocol from random bytes on a port. The version field allows receivers to handle multiple protocol versions simultaneously during upgrades. Reserve a range of message types for future use. Use Protobuf or Thrift for schema-based binary encoding that handles versioning automatically through field tagging.
Protocol Buffers (Protobuf)
Protobuf encodes messages as a sequence of (field_tag, wire_type, value) triples. Field tags are integers defined in the .proto schema. Missing fields are omitted from the encoding (default values are not transmitted). Unknown fields (from a newer schema version) are preserved by receivers, enabling forward compatibility. Adding new optional fields with new tags is safe. Removing fields requires marking them as reserved to prevent tag reuse. Protobuf is more compact than JSON and parses 5-10x faster.
Error Detection
TCP provides error detection for in-transit bit errors (TCP checksum). For storage protocols and protocols over unreliable transports, add checksums: CRC32 (fast, good error detection), CRC32C (hardware-accelerated on modern CPUs), or xxHash (faster than CRC, no hardware acceleration required). Append the checksum to each message frame. The receiver computes the checksum over the received bytes and compares to the appended checksum. Mismatches indicate data corruption.
Request-Response Correlation
In multiplexed protocols (multiple in-flight requests on one connection), correlation IDs match responses to requests. Include a request_id (int32 or UUID) in every request header. The response carries the same request_id. The client maps request_id → pending future/callback and resolves it on response receipt. This is how Kafka, gRPC over HTTP/2 (stream IDs), and Redis pipelining work. Without correlation IDs, requests must be processed strictly in order (head-of-line blocking).
Compression
Compress message payloads when network bandwidth is the bottleneck (large payloads, low-bandwidth links). Common algorithms: LZ4 (fastest compression/decompression, moderate ratio), Snappy (fast, moderate ratio, Google), Zstd (best ratio at moderate speed, preferred for new systems). Indicate compression in the message header: a compression_type field (0=none, 1=lz4, 2=snappy, 3=zstd). Do not compress already-compressed data (images, videos, encrypted content) — it grows the payload. Compress only above a minimum payload size threshold (e.g., 1KB).