Title: Configuration deployment made easy with drist | |
Author: Solène | |
Date: 29 November 2018 | |
Tags: unix drist automation | |
Description: | |
Hello, in this article I will present you my deployement tool **drist** | |
(if you | |
speak Russian, I am already aware of what you think). It reached a | |
feature | |
complete status today and now I can write about it. | |
As a system administrator, I started using *salt* a few years ago. And | |
honestly, I can not cope with it anymore. It is slow, it can get very | |
complicated for some tasks like correctly ordering commands and a | |
configuration file can become a nightmare when you start using | |
condition in it. | |
### History | |
I also tried alternatives like *ansible*, *puppet*, *Rex* etc... One | |
day, when | |
lurking in the ports tree, I found **sysutils/radmind** which got a lot | |
interest from me even if it is really poorly documented. It is a | |
project from | |
1995 if I remember correctly, but I liked the base idea. *Radmind* | |
works with | |
files, you create a known working set of files for your system, and you | |
can | |
propagate that whole set to other machines, or see differences between | |
the | |
reference and the current system. Sets could be negative, meaning that | |
the | |
listed files should not be present on the system, but it was also | |
possible to | |
add extra sets for specific hosts. The whole thing is really really | |
cumbersome, | |
this requires a lot of work, I found little documentation etc... so I | |
did not | |
used it but, that lead me to write my own deployment tool using ideas | |
from | |
*radmind* (working with files) and from *Rex* (using a script for doing | |
changes). | |
### Concept | |
**drist** aims at being simple to understand and pluggable with | |
standard tools. | |
There is no special syntax to learn, no daemon to run, no agent, and it | |
relies | |
on base tools like awk, sed, ssh and rsync. | |
**drist** is cross platform as it has a few requirements but it is not | |
well | |
suited for deploying on too much differents operating systems. | |
When executed, **drist** will execute six steps in a specific order, | |
you can | |
use only steps you need. | |
Shamelessly copied from the man page, explanations after: | |
1. If folder **files** exists, its content is copied to server | |
rsync(1). | |
2. If folder **files-HOSTNAME** exists, its content is copied to server | |
using rsync(1). | |
3. If folder **absent** exists, filenames in it are deleted on server. | |
4. If folder **absent-HOSTNAME** exists, filenames in it are deleted on | |
server. | |
5. If file **script** exists, it is copied to server and executed | |
there. | |
6. If file **script-HOSTNAME** exists, it is copied to server and | |
executed there. | |
In the previous list, all the existences checks are done from the | |
current | |
working directory where drist is started. The text **HOSTNAME** is | |
replaced by | |
the output of `uname -n` of the remote server, and files are copied | |
starting from | |
the root directory. | |
drist does not do anything more. In a more litteral manner, it copies | |
files to | |
the remote server, using a local filesystem tree (folder **files**). It | |
will | |
delete on the remote server all files present in the local filesystem | |
tree | |
(folder **absent**), and it will run on the remote server a script | |
named | |
**script**. | |
Each of theses can be customized per-host by adding a "-HOSTNAME" | |
suffix to the | |
folder or file name, because experience taught me that some hosts does | |
require | |
specific configuration. | |
If a folder or a file does not exist, **drist** will skip it. So it is | |
possible | |
to only copy files, or only execute a script, or delete files and | |
execute a | |
script after. | |
### Drist usage | |
The usage is pretty simple. **drist** has 3 flags which are optionals. | |
- -n flag will show what happens (simuation mode) | |
- -s flag tells drist to use sudo on the remote host | |
- -e flag with a parameter will tell drist to use a specific path for | |
the sudo | |
program | |
The remote server address (ssh format like user@host) is mandatory. | |
$ drist my_user@my_remote_host | |
drist will look at files and folders in the current directory when | |
executed, | |
this allow to organize as you want using your filesystem and a revision | |
control | |
system. | |
### Simple examples | |
Here are two examples to illustrate its usage. The examples are easy, | |
for | |
learning purpose. | |
#### Deploying ssh keys | |
I want to easily copy my users ssh keys to a remote server. | |
$ mkdir drist_deploy_ssh_keys | |
$ cd drist_deploy_ssh_keys | |
$ mkdir -p files/home/my_user1/.ssh | |
$ mkdir -p files/home/my_user2/.ssh | |
$ cp -fr /path/to/key1/id_rsa files/home/my_user1/.ssh/ | |
$ cp -fr /path/to/key2/id_rsa files/home/my_user2/.ssh/ | |
$ drist user@remote-host | |
Copying files from folder "files": | |
/home/my_user1/.ssh/id_rsa | |
/home/my_user2/.ssh/id_rsa | |
#### Deploying authorized_keys file | |
We can easily create the authorized_key file by using cat. | |
$ mkdir drist_deploy_ssh_authorized | |
$ cd drist_deploy_ssh_authorized | |
$ mkdir -p files/home/user/.ssh/ | |
$ cat /path/to/user/keys/*.pub > | |
files/home/user/.ssh/authorized_keys | |
$ drist user@remote-host | |
Copying files from folder "files": | |
/home/user/.ssh/authorized_keys | |
This can be automated using a makefile running the cat command and then | |
running | |
drist. | |
all: | |
cat /path/to/keys/*.pub > | |
files/home/user.ssh/authorized_keys | |
drist user@remote-host | |
#### Installing nginx on FreeBSD | |
This module (aka a folder which contain material for drist) will | |
install nginx | |
on FreeBSD and start it. | |
$ mkdir deploy_nginx | |
$ cd deploy_nginx | |
$ cat >script <<EOF | |
#!/bin/sh | |
test -f /usr/local/bin/nginx | |
if [ $? -ne 0 ]; then | |
pkg install -y nginx | |
fi | |
sysrc nginx_enable=yes | |
service nginx restart | |
EOF | |
$ drist user@remote-host | |
Executing file "script": | |
Updating FreeBSD repository catalogue... | |
FreeBSD repository is up to date. | |
All repositories are up to date. | |
The following 1 package(s) will be affected (of 0 checked): | |
nginx: 1.14.1,2 | |
421 KiB to be downloaded. | |
[1/1] Fetching nginx-1.14.1,2.txz: 100% 421 KiB 430.7kB/s | |
00:01 | |
Checking integrity... done (0 conflicting) | |
[1/1] Installing nginx-1.14.1,2... | |
===> Creating groups. | |
Using existing group 'www'. | |
===> Creating users | |
Using existing user 'www'. | |
[1/1] Extracting nginx-1.14.1,2: 100% | |
Message from nginx-1.14.1,2: | |
=================================================================== | |
Recent version of the NGINX introduces dynamic modules | |
support. In | |
FreeBSD ports tree this feature was enabled by default with | |
the DSO | |
knob. Several vendor's and third-party modules have been | |
converted | |
to dynamic modules. Unset the DSO knob builds an NGINX | |
without | |
dynamic modules support. | |
directive in the main context, specifying the path to the | |
shared | |
object file for the module, enclosed in quotation marks. | |
When you | |
reload the configuration or restart NGINX, the module is | |
loaded in. | |
It is possible to specify a path relative to the source | |
directory, | |
or a full path, please see | |
https://www.nginx.com/blog/dynamic-modules-nginx-1-9-11/ | |
and | |
http://nginx.org/en/docs/ngx_core_module.html#load_module | |
for | |
details. | |
=================================================================== | |
nginx_enable: -> yes | |
Performing sanity check on nginx configuration: | |
nginx: the configuration file | |
/usr/local/etc/nginx/nginx.conf syntax is ok | |
nginx: configuration file /usr/local/etc/nginx/nginx.conf | |
test is successful | |
nginx not running? (check /var/run/nginx.pid). | |
Performing sanity check on nginx configuration: | |
nginx: the configuration file | |
/usr/local/etc/nginx/nginx.conf syntax is ok | |
nginx: configuration file /usr/local/etc/nginx/nginx.conf | |
test is successful | |
Starting nginx. | |
### More complex example | |
Now I will show more complexes examples, with host specific steps. I | |
will not | |
display the output because the previous output were sufficient enough | |
to give a | |
rough idea of what drist does. | |
#### Removing someone ssh access | |
We will reuse an existing module here, a user should not be able to | |
login | |
anymore on its account on the servers using the ssh key. | |
$ cd ssh | |
$ mkdir -p absent/home/user/.ssh/ | |
$ touch absent/home/user/.ssh/authorized_keys | |
$ drist user@server | |
#### Installing php on FreeBSD | |
The following module will install php and remove the opcache.ini file, | |
and will | |
install php72-pdo_pgsql if it is run on server | |
*production.domain.private*. | |
$ mkdir deploy_php && cd deploy_php | |
$ mkdir -p files/usr/local/etc | |
$ cp /some/correct/config.ini files/usr/local/etc/php.ini | |
$ cat > script <<EOF | |
#!/bin/sh | |
test -f /usr/local/etc/php-fpm.conf || pkg install -f | |
php-extensions | |
sysrc php_fpm_enable=yes | |
service php-fpm restart | |
test -f /usr/local/etc/php/opcache.ini || rm | |
/usr/local/etc/php/opcache.ini | |
EOF | |
$ cat > script-production.domain.private <<EOF | |
#!/bin/sh | |
test -f /usr/local/etc/php/pdo_pgsql.ini || pkg install -f | |
php72-pdo_pgsql | |
service php-fpm restart | |
EOF | |
#### The monitoring machine | |
This one is unique and I would like to avoid applying its configuration | |
against | |
another server (that happened to me once with salt and it was really | |
really | |
bad). So I will just do all the job using the hostname specific cases. | |
$ mkdir my_unique_machine && cd my_unique_machine | |
$ mkdir -p | |
files-unique-machine.private/usr/local/etc/{smokeping,munin} | |
$ cp /good/config | |
files-unique-machine.private/usr/local/etc/smokeping/config | |
$ cp /correct/conf | |
files-unique-machine.private/usr/local/etc/munin/munin.conf | |
$ cat > script-unique-machine.private <<EOF | |
#!/bin/sh | |
pkg install -y smokeping munin-master munin-node | |
munin-configure --shell --suggest | sh | |
sysrc munin_node_enable=yes | |
sysrc smokeping_enable=yes | |
service munin-node restart | |
service smokeping restart | |
EOF | |
$ drist user@incorrect-host | |
$ drist [email protected] | |
Copying files from folder "files-unique-machine.private": | |
/usr/local/etc/smokeping/config | |
/usr/local/etc/munin/munin.conf | |
Executing file "script-unique-machine.private": | |
[...] | |
Nothing happened on the wrong system. | |
#### Be creative | |
Everything can be automated easily. I have some makefile in a lot of my | |
drist | |
modules, because I just need to type "make" to run it correctly. | |
Sometimes it | |
requires concatenating files before being run, sometimes I do not want | |
to make | |
mistake or having to remember on which module apply on which server (if | |
it's | |
specific), so the makefile does the job for me. | |
One of my drist module will look at all my SSL certificates from | |
another | |
module, and make a reed-alert configuration file using awk and | |
deploying it on | |
the monitoring server. All I do is typing "make" and enjoy my free | |
time. | |
### How to get it and install it | |
- Drist can be downloaded [at this | |
address](ftp://ftp.bitreich.org/releases/drist/drist-v1.02.tgz). | |
- Sources can be cloned using `git clone git://bitreich.org/drist` | |
In the sources folder, type "make install" as root, that will copy | |
drist binary | |
to /usr/bin/drist and its man page to /usr/share/man/man1/drist.1 | |
For copying files, drist requires rsync on both local and remote hosts. | |
For running the script file, a sh compatible shell is required (csh is | |
not working). |