Author: Josuah

monitoring suite
==================

$ while read line; do $line >/dev/null 2>&1 || logger -cs -p local0.alert -t monitor "$line" & done </etc/monitor.conf; wait

Explanation: It is a monitoring suite, that reads configuration in
this form (which can be generated or edited manually):

/etc/monitor.conf
~~~
check-http name=http-suckless host=nuchalis url=https://home.suckless.z0.is/
check-http name=http-suckless host=nuchalis url=https://ev.suckless.org/
....

For each input line, it execute the check-xxx with all arguments, and
if it fails, it prints and logs to syslog and standard error (-s flag).
If it is ran by cron, you also get an email every time something fails
(setup an alias for root to your@mail in /etc/mail/aliases or alike).

Here is an example of check command:

~~~
check-http
~~~
export "$@"
: ${method:=GET} # default value
exec curl -Ls -X "$method" "$url" >/dev/null

The mails will look like this:
~~~
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <root@olor> monitor-run
Date: Mon, 26 Oct 2020 03:35:03 +0000 (UTC)

monitower: check-http name=http-suckless host=olor vhost=suckless url=https://home.suckless.z0.is/
monitower: check-mail name=smtp host=olor ip=45.32.148.148 [email protected]
monitower: check-mail name=smtp host=olor ip=45.32.148.148 [email protected]
~~~

If there is a VPN between the production machines and admins laptops or
if there is a jump host, then syslog can be configured to forward all
local0.alert log events to that host, which can then decide to print
them out on the terminal and emmit a beep on local machine to get some
/dev/speaker beep sound.