Title: Port of the week: rclone | |
Author: Solène | |
Date: 28 October 2020 | |
Tags: portoftheweek | |
Description: | |
New **Port of the Week** after 3 years! I never thought it was so long | |
since last blog post about slrn. | |
This post is about the awesome **rclone** program, written in Go and | |
available on most popular platforms (including OpenBSD!). I will | |
explain how to configure it from the interactive command, from file | |
and what you can do with rclone. | |
**rclone** can be see as a rsync on steroids which supports lot of | |
Cloud backend and also support creating an encrypted data repository | |
over any backend (local file, ftp, sftp, webdav, Dropbox, AWS S3, | |
etc...). | |
It's **not** a automatic synchronization tool or a **backup** | |
software. It can copy files from A to B, synchronize two places | |
(can be harmful if you don't pay attention). | |
Let's see how to use it with an ssh server on which we will | |
create an encrypted repository to store important data. | |
[Official documentation](https://rclone.org/) | |
## Installation | |
Most of the time, run your package manager to install `rclone`. | |
It's a single binary. | |
## Interactive configuration | |
*You can skip this LONG section if you want to learn what rclone | |
can do and how to configure it in a 10 lines files.* | |
There is a parameter to have a question / answer interface to | |
configure your repository, using `rclone config`. | |
I'll make a full walkthrough to enable an encrypted repository | |
because I struggled to understand the logic behind rclone when I | |
started using it. | |
Let's start. I'll create an encrypted destination on my local NAS | |
which doesn't have full disk encryption, so anyone who access the | |
system won't be able to read my data. First, this will require to | |
set up an sftp repository and then an encrypted repository using the | |
previous one as a backend. | |
Let's create a new config named `home_nas`. | |
$ rclone config | |
2020/10/27 21:30:48 NOTICE: Config file | |
"/home/solene/.config/rclone/rclone.conf" not found - using defaults | |
No remotes found - make a new one | |
n) New remote | |
s) Set configuration password | |
q) Quit config | |
n/s/q> n | |
name> home_nas | |
We want the storage type 29, "SSH/SFTP" (I removed all 50+ others | |
storages for readability). | |
Type of storage to configure. | |
Enter a string value. Press Enter for the default (""). | |
Choose a number from below, or type in your own value | |
[...] | |
29 / SSH/SFTP Connection | |
\ "sftp" | |
[...] | |
Storage> 29 | |
My host is 192.168.1.200 | |
** See help for sftp backend at: https://rclone.org/sftp/ ** | |
Enter a string value. Press Enter for the default (""). | |
Choose a number from below, or type in your own value | |
1 / Connect to example.com | |
\ "example.com" | |
host> 192.168.1.200 | |
I will connect with the username `solene`. | |
SSH username, leave blank for current username, solene | |
Enter a string value. Press Enter for the default (""). | |
user> solene | |
Standard port 22, which is the default | |
SSH port, leave blank to use default (22) | |
Enter a string value. Press Enter for the default (""). | |
port> | |
I answer **n** because I want rclone to use ssh agent, this could | |
be the ssh password to the remote user, but I highly discourage | |
everyone from using password authentication on SSH! | |
SSH password, leave blank to use ssh-agent. | |
y) Yes type in my own password | |
g) Generate random password | |
n) No leave this optional password blank (default) | |
y/g/n> n | |
Leave this except if you want to provide a private key. | |
Raw PEM-encoded private key, If specified, will override key_file | |
parameter. | |
Enter a string value. Press Enter for the default (""). | |
key_pem> | |
Leave this except if you want to provide a PEM-encoded private key. | |
Path to PEM-encoded private key file, leave blank or set | |
key-use-agent to use ssh-agent. | |
variables such as `${RCLONE_CONFIG_DIR}`. | |
key_file> | |
Leave this except if you need to use a password to unlock your | |
private key. I use ssh agent so I don't need it. | |
The passphrase to decrypt the PEM-encoded private key file. | |
Encrypted keys | |
in the new OpenSSH format can't be used. | |
y) Yes type in my own password | |
g) Generate random password | |
n) No leave this optional password blank (default) | |
y/g/n> n | |
If your user agent manage multiples keys, you should enter the | |
correct value here, I only have one key so I leave it empty. | |
When set forces the usage of the ssh-agent. | |
key-file is read and only the associated key is | |
requested from the ssh-agent. This allows to avoid `Too many | |
authentication failures for *username*` errors | |
when the ssh-agent contains many keys. | |
Enter a boolean value (true or false). Press Enter for the default | |
("false"). | |
key_use_agent> | |
This is a question about crypto, accept the default except if you | |
have to connect to old servers. | |
Enable the use of insecure ciphers and key exchange methods. | |
exchange methods: | |
- aes192-cbc | |
- aes256-cbc | |
- 3des-cbc | |
- diffie-hellman-group-exchange-sha256 | |
- diffie-hellman-group-exchange-sha1 | |
recovered by an attacker. | |
Enter a boolean value (true or false). Press Enter for the default | |
("false"). | |
Choose a number from below, or type in your own value | |
1 / Use default Cipher list. | |
\ "false" | |
2 / Enables the use of the aes128-cbc cipher and | |
diffie-hellman-group-exchange-sha256, | |
diffie-hellman-group-exchange-sha1 key exchange. | |
\ "true" | |
use_insecure_cipher> | |
We want to keep hashcheck feature so just skip the answer to keep | |
the default value. | |
Disable the execution of SSH commands to determine if remote file | |
hashing is available. | |
Leave blank or set to false to enable hashing (recommended), set to | |
true to disable hashing. | |
Enter a boolean value (true or false). Press Enter for the default | |
("false"). | |
disable_hashcheck> | |
We are at the end of the configuration, we are proposed to change | |
more parameters but we don't need to. | |
Edit advanced config? (y/n) | |
y) Yes | |
n) No (default) | |
y/n> n | |
Now we can see the output of the configuration file of rclone in | |
regards to my `home_nas` destination. I agree with the configuration | |
to continue. | |
Remote config | |
-------------------- | |
[home_nas] | |
type = sftp | |
host = 192.168.1.200 | |
user = solene | |
-------------------- | |
y) Yes this is OK (default) | |
e) Edit this remote | |
d) Delete this remote | |
y/e/d> y | |
Here is a summary of the configuration, we have only one remote | |
here. | |
Current remotes: | |
==== ==== | |
home_nas sftp | |
In the menu, I will choose to add another remote. Let's name it | |
`home_nas_encrypted` | |
e) Edit existing remote | |
n) New remote | |
d) Delete remote | |
r) Rename remote | |
c) Copy remote | |
s) Set configuration password | |
q) Quit config | |
e/n/d/r/c/s/q> n | |
name> home_nas_encrypted | |
We will choose the special storage `crypt` which work on an existing | |
backend. | |
Type of storage to configure. | |
Enter a string value. Press Enter for the default (""). | |
Choose a number from below, or type in your own value | |
10 / Encrypt/Decrypt a remote | |
\ "crypt" | |
Storage> 10 | |
To this question, we will define we want the data stored to | |
`home_nas_encrypted` being saved in `home_nas` remote in the | |
`encrypted_repo` directory. | |
** See help for crypt backend at: https://rclone.org/crypt/ ** | |
Normally should contain a ':' and a path, eg | |
"myremote:path/to/dir", | |
"myremote:bucket" or maybe "myremote:" (not recommended). | |
Enter a string value. Press Enter for the default (""). | |
remote> home_nas:encrypted_repo | |
Depending on the level of obfuscation your choice may vary. The | |
simple filename obfuscation is fine for me. | |
How to encrypt the filenames. | |
Enter a string value. Press Enter for the default ("standard"). | |
Choose a number from below, or type in your own value | |
1 / Encrypt the filenames see the docs for the details. | |
\ "standard" | |
2 / Very simple filename obfuscation. | |
\ "obfuscate" | |
3 / Don't encrypt the file names. Adds a ".bin" extension only. | |
\ "off" | |
filename_encryption> 2 | |
As for the directory names obfuscation, I recommend to enable it, | |
otherwise that leave the whole directory tree readable! | |
Option to either encrypt directory names or leave them intact. | |
nothing. | |
Enter a boolean value (true or false). Press Enter for the default | |
("true"). | |
Choose a number from below, or type in your own value | |
1 / Encrypt directory names. | |
\ "true" | |
2 / Don't encrypt directory names, leave them intact. | |
\ "false" | |
directory_name_encryption> 1 | |
Type the password that will be used to encrypt the data. | |
Password or pass phrase for encryption. | |
y) Yes type in my own password | |
g) Generate random password | |
y/g> y | |
Enter the password: | |
password: | |
Confirm the password: | |
password: | |
You can add a salt to the passphrase, I choose not too. | |
Password or pass phrase for salt. Optional but recommended. | |
Should be different to the previous password. | |
y) Yes type in my own password | |
g) Generate random password | |
n) No leave this optional password blank (default) | |
y/g/n> | |
No need to change advanced parameters. | |
Edit advanced config? (y/n) | |
y) Yes | |
n) No (default) | |
y/n> n | |
Here is a summary of the configuration of this remote backend. | |
I'm fine with it. | |
Remote config | |
-------------------- | |
[home_nas_encrypted] | |
type = crypt | |
remote = home_nas:encrypted_repo | |
directory_name_encryption = true | |
password = *** ENCRYPTED *** | |
-------------------- | |
y) Yes this is OK (default) | |
e) Edit this remote | |
d) Delete this remote | |
y/e/d> y | |
We see we have now two remote backends, one with the crypt type. | |
Current remotes: | |
==== ==== | |
home_nas sftp | |
home_nas_encrypted crypt | |
Quit rclone, the configuration is done. | |
e) Edit existing remote | |
n) New remote | |
d) Delete remote | |
r) Rename remote | |
c) Copy remote | |
s) Set configuration password | |
q) Quit config | |
e/n/d/r/c/s/q> q | |
## Configuration file | |
The previous configuration process only produced this short | |
configuration file, so you may copy/paste from it and adapt to add | |
more backends if you want, instead of doing the tedious `config` | |
process. | |
Here is my file `~/.config/rclone/rclone.conf` on my desktop. | |
[home_nas] | |
type = sftp | |
host = 192.168.1.200 | |
user = solene | |
type = crypt | |
remote = home_nas:encrypted_repo | |
directory_name_encryption = true | |
password = GDS9B1B1LrBa3ltQrSbLf1Vq5C6VbaA1AJVlSZ8 | |
## First usage | |
Now we defined our configuration, we need to create the remote | |
directory that will be used as a backend, this is important to avoid | |
errors when using rclone, this is a simple step required only once. | |
$ rclone mkdir home_nas_encrypted: | |
On the remote server, I can see a `/home/solene/encryted_repo` | |
directory. It's now ready to use! | |
## A few commands | |
**rclone** has a LOT of commands available, I will present a few | |
of them. | |
### Copying files to/from backend | |
Let's say I want to copy files to the encrypted repository. There | |
is a `copy` command. | |
$ rclone copy /home/solene/log/templates | |
home_nas_encrypted:blog_template | |
There are no output by default when the program runs fine. You can | |
use `-v` flag to have some verbose output (I prefer it). | |
### List files on a remote backend | |
Now, we want to see if the files were copied correctly, we will use | |
the `ls` command. | |
$ rclone ls home_nas_encrypted: | |
299 blog_template/article.tpl | |
700 blog_template/gopher_head.tpl | |
2505 blog_template/layout.tpl | |
295 blog_template/more.tpl | |
236 blog_template/navigation.tpl | |
57 blog_template/one-tag.tpl | |
34 blog_template/page.tpl | |
189 blog_template/rss-item.tpl | |
326 blog_template/rss.tpl | |
We can also use `ncdu` to mimic the **ncdu** program displaying a | |
curses interfaces to visualize disk usage in a nice browsing tree. | |
$ rclone ncdu home_nas_encrypted | |
-- home_nas_encrypted: ------------------ | |
6.379k [##########] /blog_template | |
### The sync command | |
Files and directories can also be copied with the `sync` command, | |
but this must be used with care because it makes sure the destination | |
matches exactly the origin when you use it. It's the equivalent of | |
`rsync -a --delete origin/ destination/`, so any extra files will | |
be removed! Note that you can use `--dry-run` to see what will | |
happen. | |
### Filters | |
When you copy files using the various available method, instead of | |
using a path, you can provide a filter file or a list of paths to | |
transfers. This can be very efficient when you want to recover | |
specifics data. | |
The documentation about | |
[filtering is available here](https://rclone.org/filtering/) | |
### Parameters | |
**rclone** supports a lot of parameters, like to limit upload | |
bandwidth, copy multiples files at once, enable an interactive mode | |
in case of file deletion/overwriting. | |
### Mount | |
On Linux, FreeBSD and MacOS, **rclone** can use a FUSE filesystem | |
to mount the remote repository on the filesystem, making its uses | |
totally transparent. | |
This is extremely useful, avoiding the tediousness of the get/put | |
paradigm of **rclone**. | |
**This can even be used to make an encrypted repository on the local | |
filesystem! :)** | |
### Create a webdav/sftp/ftp server | |
**rclone** has the capability of act as a server and expose a | |
configured remote backend on various network protocol like webdav, | |
sftp, ftp, s3 (minio) ! | |
The | |
[serv document is available | |
here](https://rclone.org/commands/rclone_serve/) | |
Example running a simple webdav server with hardcoded login/password: | |
$ rclone serv webdav --user solene --password ANicePassword | |
home_nas_encrypted: | |