## Shoot Footgun
Delivering Server Failure
by
Josuah Demangeon <
[email protected]>
## How to mess up a server
* Delete a random binary in /bin
* "command not found"
* scripts that randomly crash
## How to mess up a server really bad
* Move off /lib
* "command not found" even though the binaries are there
* dynamic linking <3
#pause
This is still not what I did...
## How to comletely destroy a server
#pause
* Upload /bin from Linux to an OpenBSD system
#pause
* "^?ELF^B^A^A: command not found"
#pause
* Upload the whole / while we are at it
## Self doom recipe
#!/bin/sh -e
# export all local objects to a remote system
host=$1
exec rsync -Pa --size-only "$snap/" "$host:$snap/"
## Self doom recipe
#!/bin/sh -e
# export all local objects to a remote system
host=${1:?}
exec rsync -Pa --size-only "$snap/" "$host:$snap/"
## Self doom recipe
#!/bin/sh -e
# export all local objects to a remote system
host=${1:?}
snap=${SNAPSHOT:-/var/lib/snapshot
exec rsync -Pa --size-only "$snap/" "$host:$snap/"
## How to comletely destroy a server
* Joyfully uploading / to the remote system
* OpenBSD/musl-libc/busybox/Linux
## How to comletely destroy a server forever
#pause
* The backup script uploading / on all of my servers (2 of them)
* Everything down: smtpd, gopherd, gitd, httpd, dnsd, irc client...
## EMERGENCY REBOOTSTRAP (1/3)
* Custom packages
* Custom service ./run scripts
* No server to fetch them from (both are down now)
#pause
* Must get it from previous system
## EMERGENCY REBOOTSTRAP (2/3)
* Boot into install .iso then (U) update
* Only OpenBSD 6.4 .iso available on VPS
* OpenBSD 6.6/6.4/musl-libc/busybox/Linux
* Download
## EMERGENCY REBOOTSTRAP (3/3)
* Book another VPS
* Download packages
* Run services
## Facts
* Things can still go wrong when you backup your backups
* No "password recovery" possible when DNS and SMTP servers all down
* The most important data to backup: registrar admin panel password
