(2023-04-13) On serialization of binary chunks
----------------------------------------------
Today is a Clean Thursday, and my post will be much shorter and cleaner than
any of the recent ones.

If you need a really safe way to store or transfer arbitrary binary data
chunks, just use djb's netstrings ([1]) which are constructed like this:

[length]:[content],

Here, the trailing comma is mandatory and the [length] is decimal, with
starting zero prohibited unless the length is 0 (so, an empty string is
encoded into three characters '0:,').

The decoding algorithm is dead simple as well:

1. Read everything until the first : byte as a decimal number L. If L is not
a valid non-negative decimal integer number (or starts with 0 while being
positive), report an error and halt.
2. Allocate a buffer B which is L bytes long. If allocation is impossible,
report an error and halt.
3. Read the next L bytes after the first : byte into the buffer B. If reading
L bytes is impossible, report an error and halt.
4. Read the (L+1)-th byte. If it's not equal to the ',' byte, report an error
and halt.
5. Return the buffer B as the result. End of algorithm.

Why is the trailing comma needed if we already specify how many bytes to
read? I asked this question myself some time ago. The comma being in the
expected place after the chunk is an additional (although not fully
surefire) marker that no bytes were lost from the chunk during the transfer
and no extra bytes were added, and the actual chunk length is exactly as
declared.

I really can't praise this format high enough. I recommend using it as a
top-level wrapper for whatever message format you employ in your scripting,
desktop, mobile, server, embedded applications. Even if your messages are in
plaintext, they will remain in plaintext, but now much safer. Netstrings
introduce so little overhead while adding so much value that they really
help to make the world a bit better place. Just use them wherever you can.

And no, I'm not confusing marshalling with serialization. And neither should
you.

--- Luxferre ---

[1]: https://cr.yp.to/proto/netstrings.txt

You are viewing proxied material from hoi.st. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.