Title: Fair Internet bandwidth management on a network using OpenBSD | |
Author: Solène | |
Date: 30 August 2021 | |
Tags: openbsd bandwidth | |
Description: | |
# Introduction | |
I have a simple DSL line with a 15 Mb/s download and 900 kb/s upload | |
rates and there are many devices using the Internet and two people in | |
remote work. Some poorly designed software (mostly on windows) will | |
auto update without allowing to reduce the bandwidth or some huge | |
bloated website will require lot of download and will impact workers | |
using the network. | |
The point of this article is to explain how to use OpenBSD as a router | |
on your network to allow the Internet access to be used fairly by | |
devices on the network to guarantee everyone they will have at least a | |
bit of Internet to continue working flawlessly. | |
I will use the queuing features from the OpenBSD firewall PF (Packet | |
Filter) which relies on the CoDel network scheduler algorithm, which | |
seems to bring all the features we need to do what we want. | |
pf.conf manual page: QUEUEING section | |
Wikipedia page about the CoDel network scheduler algorithm | |
# Important | |
I'm writing this in a separate section of the article because it is | |
important to understand. | |
It is not possible to limit the download bandwidth, because once the | |
data are already in the router, this mean they came from the modem and | |
it's too late to try to do anything. But there is still hope, if the | |
router receives data from the Internet it's that some devices on the | |
network asked to receive it, you can act on the uploaded data to | |
throttle what we receive. This is not obvious at first but it makes | |
totally sense once you get the idea. | |
The biggest point to understand is that you can throttle download speed | |
through the ACK packets. Think of two people on a phone, let's say | |
Alice and Bob, Alice is your network and calls Bob who is very happy to | |
tell his life to Alice. Bob speaking is data you download. In a | |
normal conversation, Bob will talk and will hear some sounds from Alice | |
who acknowledge what Bob is saying. If Alice stops or shut her | |
microphone, Bob may ask if Alice is still listening and will wait for | |
an answer. When Alice is making a sound (like "hmmhm or yes"), this is | |
an acknowledgement for Bob to continue. Literally, Bob is sending a | |
voice stream to Alice who is sending ACK (acknowledgement short name) | |
packets to Bob so he can continue. | |
This is exactly where you can control bandwidth, if we reduce the | |
bandwidth used by ACK packets for a download, you can reduce the given | |
download. If you can allow multiple systems to fairly send their share | |
of ACK, they should have a fair share of the downloaded data. | |
What's even more important is that you absolutely don't use all the | |
upload bandwidth with ACK packets to reach your maximum download | |
bandwidth. We will have to separate ACK from uploaded data so we don't | |
limit file upload or similar flows. | |
# Setup | |
For the setup I used a laptop with two network cards, one was connected | |
to the ISP box and the other was on the LAN side. I've enabled a DHCP | |
server on the OpenBSD router to automatically give IP addresses and | |
gateway and name servers addresses to devices on the network. | |
Basically, you can just plug an equivalent router on your current LAN, | |
disable DHCP on your ISP router and enable DHCP on your OpenBSD system | |
using a different subnet, both subnets will be available on the network | |
but for tests it requires little changes, when you want to switch from | |
a router to another by default, toggle the DHCP service on both and | |
renew DHCP leases on your devices. This is extremely easy. | |
```ASCII network diagram | |
+---------+ | |
| ISP | | |
| router | | |
+---------+ | |
| | |
| | |
| re0 | |
+---------+ | |
| OpenBSD | | |
| router | | |
+---------+ | |
| em0 | |
| | |
| | |
+---------+ | |
| network | | |
| switch | | |
+---------+ | |
``` | |
# Configuration explained | |
## Line by line | |
I'll explain first all the config lines from my /etc/pf.conf file, and | |
later in this article you will find a block with the complete rules | |
set. | |
The following lines are default and can be kept as-is except if you | |
want to filter what's going in or out, but it's another topic as we | |
only want to apply queues. Filtering would be as usual. | |
```pf.conf configuration line | |
set skip on lo | |
block return # block stateless traffic | |
pass # establish keep-state | |
``` | |
This is where it get interesting. The upstream router is accessed | |
through the interface re0, so we create a queue of the speed of the | |
link of that interface, which is 1 Gb/s. pf.conf syntax requires to | |
use bits per second (b/s or bps) and not bytes per second (Bps or B/s) | |
which can be misleading. | |
```pf.conf configuration line | |
queue std on re0 bandwidth 1G | |
``` | |
Then, we create a queue that inherits from the parent created before, | |
this represent the whole upload bandwidth to reach the Internet. We | |
will make all the traffic reaching the Internet to go through this | |
queue. | |
I've set a bandwidth of 900K with a max of 900K, this mean, that this | |
queue can't let pass more than 900 kilo bits per second (which | |
represent 900/8 = 112.5 kB/s or kilo Bytes per second). This is the | |
extreme maximum my Internet access allows me. | |
```pf.conf configuration line | |
queue internet parent std bandwidth 900K max 900K | |
``` | |
The following lines are all sub queues to divide the upload usage, we | |
want to have a separate queue for DNS request which must not be delayed | |
to keep responsiveness, but also voip or VPN queues to guarantee a | |
minimum available for the users. | |
The web queue is the one which is likely to pass the most data, if you | |
upload a file through a website, it will pass through the web queue. | |
The unknown queue is the outgoing traffic that is not known, it's up to | |
you to put a maximum or not. | |
Finally, the ackp queue that is split into two other queues, it's the | |
most important part of the setup. | |
The "bandwidth xxxK" values should sum up to something around the 900K | |
defined as a maximum in the parent, this only mean we target to keep | |
this amount for this queue, this doesn't enforce a minimum or a maximum | |
which can be defined with min and max keywords. | |
As explained earlier, you can control the downloading speed by | |
regulating the sent ACK packets, all ACK will go through the queues | |
ack_web and ack. | |
ack_web is a queue dedicated for http/https downloads and the other ack | |
queue is used for other protocol, I preferred to divide it in two so | |
other protocol will have a bit more room for themselves to | |
counterbalance a huge http download (Steam game platform like to make | |
things hard on this topic by making downloads to simultaneous server | |
for maximum bandwidth usage). | |
The two ack queues accumulated can't get over the parent queue set as | |
406K here. Finding the correct value is empirical, I'll explain later. | |
All these queues created will allow each queue to guarantee a minimum | |
from the router point of view, roughly said per protocol here. | |
Unfortunately, this won't guarantee computers on the network will have | |
a fair share of the queues! This is a crucial understanding I lacked | |
at first when trying to do this a few years ago. The solution is to | |
use the "flow" scheduler by using the flow keyword in the queue, this | |
will give some slot to every session on the network, guarantying (at | |
least theoretically) every session have the same time passed to send | |
data. | |
I used "flows" only for ACK, it proved to work perfectly fine for me as | |
it's the most critical part but in fact, it could be applied to every | |
leaf queues. | |
```pf.conf configuration line | |
queue web parent internet bandwidth 220K qlimit 100 | |
queue dns parent internet bandwidth 5K | |
queue unknown parent internet bandwidth 150K min 100K qlimit 1… | |
queue vpn parent internet bandwidth 150K min 200K qlimit 1… | |
queue voip parent internet bandwidth 150K min 150K | |
queue ping parent internet bandwidth 10K min 10K | |
queue ackp parent internet bandwidth 200K max 406K | |
queue ack_web parent ackp bandwidth 200K flows 256 | |
queue ack parent ackp bandwidth 200K flows 256 | |
``` | |
Because packets aren't magically assigned to queues, we need some match | |
rules for the job. You may notice the notation with parenthesis, this | |
mean the second member of the parenthesis is the queue dedicated for | |
ACK packets. | |
The VOIP queuing is done a bit wide, it seems Microsoft Teams and | |
Discord VOIP goes through these port ranges, it worked fine from my | |
experience but may depend of protocols. | |
```pf.conf configuration line | |
match proto tcp from em0:network to any queue (unknown,ack) | |
match proto tcp from em0:network to any port { 80 443 8008 8080 } queue (web,ac… | |
match proto tcp from em0:network to any port { 53 } queue (dns,ack) | |
match proto udp from em0:network to any port { 53 } queue dns | |
# VPN (wireguard, ssh, openvpn) | |
match proto udp from em0:network to any port { 4443 1194 } queue vpn | |
match proto tcp from em0:network to any port { 1194 22 } queue (vpn,ack) | |
# voip (teams) | |
match proto tcp from em0:network to any port { 3479 50000:50060 } queue voip | |
match proto udp from em0:network to any port { 3479 50000:50060 } queue voip | |
# keep some bandwidth for ping packets | |
match proto icmp from em0:network to any queue ping | |
``` | |
Simple rule to enable NAT so devices from the LAN network can reach the | |
Internet. | |
```pf.conf configuration line | |
# NAT to the outside | |
pass out on egress from !(egress:network) nat-to (egress) | |
``` | |
Default OpenBSD rules that can be kept here. | |
```pf.conf configuration line | |
# By default, do not permit remote connections to X11 | |
block return in on ! lo0 proto tcp to port 6000:6010 | |
# Port build user does not need network | |
block return out log proto {tcp udp} user _pbuild | |
``` | |
## How to choose values | |
In the previous section I used absolute values, like 900K or even 406K. | |
A simple way to define them is to upload a big file to the Internet | |
and check the upload rate, I use bwm-ng but vnstat or even netstat | |
(with the correct combination of flags) could work, see your average | |
bandwidth over 10 or 20 seconds while transferring, and use that value | |
as a maximum in BITS as a maximum for the internet queue. | |
As for the ACK queue, it's a bit more tricky and you may tweak it a | |
lot, this is a balance between full download mode or conservative | |
download speed. I've lost a bit of download rate for the benefit of | |
keeping room for more overall responsiveness. Like previously, monitor | |
your upload rate when you download a big file (or even multiples files | |
to be sure to fill your download link) and you will see how much will | |
be used for ACK. It will certainly be a few try and guesses before you | |
get the perfect value, too low and the maximum download rate will be | |
reduced, and too high and your link will be filled entirely when | |
downloading. | |
## Full configuration | |
```pf.conf configuration file | |
set skip on lo | |
block return # block stateless traffic | |
pass # establish keep-state | |
queue std on re0 bandwidth 1G | |
queue internet parent std bandwidth 900K min 900K max 900K | |
queue web parent internet bandwidth 220K qlimit 100 | |
queue dns parent internet bandwidth 5K | |
queue unknown parent internet bandwidth 150K min 100K qlimit 1… | |
queue vpn parent internet bandwidth 150K min 200K qlimit 100 | |
queue voip parent internet bandwidth 150K min 150K | |
queue ping parent internet bandwidth 10K min 10K | |
queue ackp parent internet bandwidth 200K max 406K | |
queue ack_web parent ackp bandwidth 200K flows 256 | |
queue ack parent ackp bandwidth 200K flows 256 | |
match proto tcp from em0:network to any queue (unknown,ack) | |
match proto tcp from em0:network to any port { 80 443 8008 8080 } queue (web,ac… | |
match proto tcp from em0:network to any port { 53 } queue (dns,ack) | |
match proto udp from em0:network to any port { 53 } queue dns | |
# VPN (ssh, wireguard, openvpn) | |
match proto udp from em0:network to any port { 4443 1194 } queue vpn | |
match proto tcp from em0:network to any port { 1194 22 } queue (vpn,ack) | |
# voip (teams) | |
match proto tcp from em0:network to any port { 3479 50000:50060 } queue voip | |
match proto udp from em0:network to any port { 3479 50000:50060 } queue voip | |
# ICMP | |
match proto icmp from em0:network to any queue ping | |
# NAT | |
pass out on egress from !(egress:network) nat-to (egress) | |
# default OpenBSD rules | |
# By default, do not permit remote connections to X11 | |
block return in on ! lo0 proto tcp to port 6000:6010 | |
# Port build user does not need network | |
block return out log proto {tcp udp} user _pbuild | |
``` | |
# How to monitor | |
There is an excellent tool to monitor the queues in OpenBSD which is | |
systat in its queue view. Simply call it with "systat queue", you can | |
define the refresh rate by pressing "s" and a number. If you see | |
packets being dropped in a queue, you can try to increase the qlimit of | |
the queue which is the amount of packets kept in the queue and delayed | |
(it's a FIFO) before dropping them. The default qlimit is 50 and may | |
be too low. | |
systat man page anchored to the queues parameter | |
# Conclusion | |
I've spent a week scrutinizing pf.conf manual and doing many tests with | |
many hardware until I understand that ACK were the key and that the | |
flow queuing mode was what I was looking for. As a result, my network | |
is much more responsive and still usable even when someone/some device | |
is using the network without any kind of limit. | |
The setup can appear a bit complicated but in the end it's only a few | |
pf.conf lines and using the correct values for your internet access. I | |
chose to make a lot of queues, but simply separating ack from the | |
default queue may be enough. |