Title: Port of the week: mbuffer | |
Author: Solène | |
Date: 31 May 2016 | |
Tags: portoftheweek networking | |
Description: | |
This Port of the week is a bit special because sadly, the port isn't | |
available on OpenBSD. The port is **mbuffer** (which you can find in | |
misc/mbuffer). | |
I discovered it while looking for a way to enhance one of my network | |
stream scripts. I have some scripts that get a dump of a postgresql | |
base through SSH, copy it from stdin to a file with tee and send it | |
out to the local postgres, the command line looks like | |
$ ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip | |
-f | tee dumps/my_base.dump | psql my_base | |
I also use the same kind of command to receive a ZFS snapshot from | |
another server. | |
But there is an issue, the end server is relatively slow, postgresql | |
and ZFS will eat lot of data from stdin and then it will stop for | |
sometimes writing on the disk, when they are ready to take new data, | |
it's slow to fill them. This is where **mbuffer** takes places. This | |
tool permit to add a buffer that will take data from stdin and fill | |
its memory (that you set on the command line), so when the slowest | |
part of the command is ready to take data, mbuffer will empty its | |
memory into the pipe, so the slowlest command isn't waiting to get | |
filled before working again. | |
The new command looks like that for a buffer of 300 Mb | |
ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip | |
-f | tee dumps/my_base.dump | mbuffer -s 8192 -m 300M | psql my_base | |
mbuffer also comes with a nice console output, showing | |
- bandwith in | |
- bandwith out | |
- percentage/consumption of memory filled | |
- total transfered | |
in @ 1219 KiB/s, out @ 1219 KiB/s, 906 MiB total, buffer 0% full | |
In this example the server is too fast so there is no wait, the buffer | |
isn't used (_0% full_). | |
mbuffer can also listen on TCP, unix socket and have a lot of | |
parameters that I didn't try, if you think that can be useful for you, | |
just go for it ! | |