# State of the Homelab

# State of the Homelab

_Published: September 12, 2022_

I've been sinking some time recently into organizing my
homelab, spurred by the recent addition of a NAS, and thought
it might be a good time to write about it.

## Birds Eye View

Here's the network topology:

┌──────────────┐ ┌───────┐
│ Wifi Clients │ │ Wired │
└──────────────┘ │Clients│
: └───┬───┘
: │
┌───────▼───────┐ ┌────▼─────┐
Internet │ Verizon ├────► OpenBSD │
─────────► Router │ │ Router │
│ (FiosGateway) ◄────┤ (apu2e4) │
└──────┬──▲─────┘ └──┬───▲───┘
│ │ │ │
┌───▼──┴──┐ ┌───▼───┴────┐
│ pi.hole │ │ NAS/Git │
│ (Rpi4b) │ │(Odroid HC4)│
└─────────┘ └────────────┘

Excluding the Fios router, that's 3 servers hosting
the following services:

- A git server for private repos (public repos hosted on
[git.alexkarle.com](https://git.alexkarle.com), and
[sourcehut](https://git.sr.ht)).
- A NAS (network attached storage) for backups, photos, music, etc
- A network wide ad-blocker ([pi-hole](https://pi-hole.net/))
- An OpenBSD router/firewall (dhcp / pf) to provide extra security
to my wired devices.

## Why Bother with a Homelab, Anyways?

Before I dive into each component, I want to take a step back and
ask _why_.

In a world where you can pay `$HIP_COMPANY` $5/mo to
run or host anything, it may seem like a homelab
is a waste of time and effort. Looking at what I'm running,
a lot of it could even be hosted for free!

Despite the time cost of tending to this digital garden,
I've found that
running my homelab has been an incredible source of learning from
hands-on experimentation. At this stage in my career, this type
of experience is invaluable, especially because a lot of it
(hardware tinkering, sysadmin tasks, linux distros, etc) doesn't
come across my desk often.

As an added bonus, I really enjoy the feeling of digital ownership
I get from hosting my private data. It certainly comes with the
weight of responsibility that I need to keep (and test!) backups,
but the learning and ownership feel worthwhile for now.

## The Nitty Gritty

### Fios Gateway

I have no special attachment to Verizon--I wouldn't go so far as
to endorse them, but my coworkers don't complain about lag during
video calls, so I haven't mustered the courage to switch providers.

It's on my long todo list to switch to a more local ISP, but with
both me and my fiance working from home it's the last place
I want an outage.

### Pi Hole

Pi-hole is a network wide adblocker. It works by acting as the DNS
server for your network and responding with localhost (0.0.0.0) for
known spammy domains.

As a concrete example, with pi-hole running right now, I can't access
doubleclick.net (Google ads):

# Response from my router
$ host doubleclick.net 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:

doubleclick.net has address 0.0.0.0
doubleclick.net has IPv6 address ::

# Response from Google's DNS resolver
$ host doubleclick.net 192.168.1.1
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

doubleclick.net has address 142.251.41.14
[...]

The default configuration for the Gateway router is to tell clients
to use it (192.168.1.1) as a DNS server. By updating Gateway to use
pi-hole as _it's_ server (instead of the Verizon supplied ones), all
clients on the network receive pi-hole's filtering.

What's brilliant about this is that no clients need updating. As far
as they're aware, they _really are_ trying to reach out to doubleclick.net.
It's just a network failure that 0.0.0.0 isn't listening on 443!

The results are most noticeable on mobile since I use uBlock Origin on
all my browsers. It's amazing how much faster and less cluttered
certain mobile apps are without ads (cough nytimes cough please stop
it with the full screen banners).

Finally, as the name might imply, the pi-hole was designed to run on the
Raspberry Pi. I'm running mine on a
[Raspberry Pi 4b](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/),
which is definitely overkill for the resources it needs.

### Git Server

Running a private git server is incredibly easy. In fact, there isn't
really any separate git daemon that needs running. So long as you have
ssh access to the host, you can clone/push/pull from:

$ git clone user-on-host@host:/path/to/server

I like to get a bit fancy and `adduser` a `git` user on the server so
that the repos can be stored in its home directory and ssh access can
be managed separately from the user account I would normally use.

Assuming the `git` user has the home directory `/home/git`, the following
will clone `/home/git/repo.git`:

$ git clone git@host:repo.git

Much cleaner!

A lot more detail, including securing the git user by assigning it the
`git-shell` for the login shell, can be found in the amazing
[Pro Git](https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server)
book.

As far as _what_ I keep on my personal server (that I wouldn't trust
to sourcehut, or even really my own git.alexkarle.com), I host the following
repos:

- My password repository (for [`pass(1)`](https://www.passwordstore.org/))
- Personal notes (public notes go on [gopher://alexkarle.com/1/notes]!)

In the past (mostly for fun), I've hosted [cgit](https://git.zx2c4.com/cgit/about/),
[gitea](https://gitea.io/) and even GitLab (reverse chronologically
and also least to most heavy). I've found that for the few private
repos I host, I rarely want a web UI (let alone forking/user accounts/etc).

### NAS

The most recent upgrade to my homelab was the addition of a purpose-built
NAS using the [Odroid HC4](https://www.hardkernel.com/shop/odroid-hc4/)
toaster-style dual hard drive board.

Previously, my backups were distributed across multiple drives and
frequently offline (with all my operating system tinkering I have 5
SSD's of which only 2 are in use at any time...). Things that I
needed frequent access to were stored on a (rather fragile) Raspberry Pi 3b
on a 64GB thumbdrive! Needless to say, the HC4 is a step up.

It's only really been online for ~24hrs so I don't have a solid review
of the hardware yet, but initial impressions are:

- I was a bit bummed that the Linux they support is an old forked kernel,
which strikes me as probably missing security patches (didn't poke around
hard enough to confirm though--it may be up to date and just old!)
- Despite being on the OpenBSD hardware list, I couldn't get it to boot.
Given the need to unscrew the plastic top to get the UART serial connection
in, I stopped trying after a few hours :(. I might email the mailing list
in the future to see what I overlooked.
- I was pleased to find that the [Armbian](https://www.armbian.com/odroid-hc4/)
project supported it (with a newer kernel!). This is what I ended up
installing (with no issues so far)
- I stumbled across [Chandler Swift's](https://chandlerswift.com/2021/05/07/odroid-hc4-nas)
awesome writeup about his experience with the HC4 and his golang
[odroidhc4-display](https://github.com/ChandlerSwift/odroidhc4-display) tool
is the perfect easy-to-deploy service for the little display

Overall, operating system quirks aside, I'm really happy with how it turned
out. I put in two 2TB western digital drives (whatever BestBuy had on sale
a few weekends ago) and encrypted both of them with `cryptsetup` using the
LUKS encryption mode as described in the
[Arch Wiki](https://wiki.archlinux.org/title/Dm-crypt/Device_encryption).
I intentionally did not RAID-1 the drives together because I'm more worried
about accidentally `rm`-ing a file than I am not having access to the data
in case of drive failure. Instead, I have only one drive always online. A
cron job mounts the offline drive daily, `rsync`'s over the data, and
unmounts it when done. This should give me hopefully a few hours or more
if I realize I deleted a file. (I'll eventually also cycle in a third drive
for offsite storage somewhere trusted like my parents' house).

In the future I'd love to use this NAS as an excuse to explore fancier
filesystems like ZFS, but I stuck to ext4 for now.

### OpenBSD Firewall/Router

*Puts tinfoil hat on.*

The final piece of the topology is maybe the least functional in terms
of hosting required services but the best learning tool: the OpenBSD router.

There are arguably some security wins here by bisecting my network between
wifi and wired clients. For one, the Verizon router itself may or may not
be receiving security patches (it's proprietary, who knows?). By setting
up a firewall so that the only traffic going in to wired clients is the
traffic expected, the wired clients are a tad safer.

While the security angle is certainly appealing, the much bigger reason
to have this in my homelab has been experimenting with router technologies.
In setting this up I had to grok `pf` (packet filtering for the firewall),
`dhcpd` (to give clients IP addresses), and just basic networking
(how _does_ a machine in one network talk to another?). There's no better
way to learn networking than having my wifi laptop trying to ping my
wired desktop and `tcpdump`ing the traffic.

This is running on a PC Engines [apu2e4](https://www.pcengines.ch/apu2e4.htm),
mostly since it seemed popular with the community and I wanted to make sure
the device had good OpenBSD support. It's been running since April 2021
without issues, so I'd recommend it!

I would eventually love to write my own pi-hole using the DNS tools in
base, but for now it's low on my todo list.

## Conclusion

If you made it this far, thanks! I hope you learned something or found
something of interest.

I'll hopefully write a similar "state of the cloud" post to cover the
services I'm running outside home, but I think this post might just be
long enough for now :)

*Update:* [State of the Cloud](/blog/state-of-the-cloud-2022-09.html)
post has been written!