Title: Introduction to IPFS | |
Author: Solène | |
Date: 17 April 2021 | |
Tags: openbsd ipfs | |
Description: | |
# introduction to IPFS | |
IPFS is a distributed storage network protocol that comes with a public | |
network. Anyone can run a peer and access content from IPFS and then | |
relay the content while it's in your cache. | |
Gateways are websites used to allow accessing content of IPFS through | |
http, there are several public gateways allowing to get data from IPFS | |
without being a peer. | |
Every publish content has an unique CID to identify it, we usually add | |
a /ipfs/ to it like in | |
/ipfs/QmRVD1V8eYQuNQdfRzmMVMA6cy1WqJfzHu3uM7CZasD7j1. The CID is unique | |
and if someone add the same file from another peer, they will get the | |
same hash as you. | |
If you add a whole directory in IPFS, the top directory hash will | |
depend on the hash of its content, this mean if you want to share a | |
directory like a blog, you will need to publish the CID every time you | |
change the content, as it's not practical at all, there is an | |
alternative for making the process more dynamic. | |
A peer can publish data in a long name called an IPNS. The IPNS string | |
will never change (it's tied to a private key) but you can associate a | |
CID to it and update the value when you want and then tell other peers | |
the value changed (it's called publishing). The IPNS notation used is | |
looking like | |
/ipns/k51qzi5uqu5dmebzq75vx3z23lsixir3cxi26ckl409ylblbjigjb1oluj3f2z.ip | |
ns, you can access an IPNS content with public gateways with a | |
different notation. | |
- IPNS gateway use example: | |
https://k51qzi5uqu5dmebzq75vx3z23lsixir3cxi26ckl409ylblbjigjb1oluj3f2z. | |
ipns.dweb.link/ | |
- IPFS gateway use example: | |
https://ipfs.io/ipfs/QmRVD1V8eYQuNQdfRzmMVMA6cy1WqJfzHu3uM7CZasD7j1/ | |
The IPFS link will ALWAYS return the same content because it's a | |
defined hash to a specific resource. The IPNS link can be updated to | |
have a newer CID over time, allowing people to bookmark the location | |
and browse it for updates later. | |
# Using a public gateway | |
There are many public gateways you can use to fetch content. | |
Health check of public gateways, useful to pick one | |
You will find two kind of gateways url, one like "https://$domain/" and | |
other like "https://$something_very_long.ipfs.$domain/", for the first | |
one you need to append your /ipfs/something or /ipns/something requests | |
like in the previous examples. For the latter, in web browser it only | |
works with ipns because web browsers think the CID is a domain and will | |
change the case of the letters and it's not long valid. When using an | |
ipns like this, be careful to change the .ipfs. by .ipns. in the url to | |
tell the gateway what kind of request you are doing. | |
# Using your own node | |
First, be aware that there is no real bandwidth control mechanism and | |
that IPFS is known to create too many connections that small routers | |
can't handle. On OpenBSD it's possible to mitigate this behavior using | |
queuing. It's possible to use a "lowpower" profile that will be less | |
demanding on network and resources but be aware this will degrade IPFS | |
performance. I found that after a few hours of bootstrapping and | |
reaching many peers, the bandwidth usage becomes less significant but | |
it's may be an issue for DSL connections like mine. | |
When you create your own node, you can use its own gateway or the | |
command line client. When you request a data that doesn't belong to | |
your node, it will be downloaded from known peers able to distribute | |
the blocks and then you will keep it in cache until your cache reach | |
the defined limited and the garbage collector comes to make some room. | |
This mean when you get a content, you will start distributing it, but | |
nobody will use your node for content you never fetched first. | |
When you have data, you can "pin" it so it will never be removed from | |
cache, and if you pin a directory CID, the content will be downloaded | |
so you have a whole mirror of it. When you add data to your node, it's | |
automatically pinned by default. | |
The default ports are 4001 (the one you need to expose over the | |
internet and potentially forwarding if you are behind a NAT), the Web | |
GUI is available at http://localhost:5001/ and the gateway is available | |
at http://localhost:8080/ | |
## Installing the node on OpenBSD | |
To make it brief, there are instructions in the provided pkg-readme but | |
I will give a few advice (that I may add to the pkg-readme later). | |
```OpenBSD installation instructions | |
pkg_add go-ipfs | |
su -l -s /bin/sh _go-ipfs -c "IPFS_PATH=/var/go-ipfs /usr/local/bin/ipfs init" | |
rcctl enable go_ipfs | |
# recommended settings | |
rcctl set go_ipfs flags --routing=dhtclient --enable-namesys-pubsub | |
cat <<EOF >> /etc/login.conf | |
go_ipfs:\ | |
:openfiles=2048:\ | |
:tc=daemon: | |
EOF | |
rcctl start go_ipfs | |
``` | |
You can change the profile to lowpower with "env | |
IPFS_PATH=/var/go-ipfs/ ipfs config profile apply lowpower", you can | |
also list profiles with the ipfs command. | |
I recommend using queues in PF to limit the bandwidth usage, for my DSL | |
connection I've set a maximum of 450K and it doesn't disrupt my network | |
anymore. I explained how to proceed with queuing and bandwidth | |
limitations in a previous article. | |
## Installing the node on NixOS | |
Installing IPFS is easy on NixOS thanks to its declarative way. The | |
system has a local IPv4 of 192.168.1.150 and a public IP of | |
136.214.64.44 (fake IP here). it is started with a 50GB maximum for | |
cache. The gateway will be available on the local network on | |
http://192.168.1.150:8080/. | |
```configuration code for NixOS to deploy IPFS | |
services.ipfs.enable = true; | |
services.ipfs.enableGC = true; | |
services.ipfs.gatewayAddress = "/ip4/192.168.1.150/tcp/8080"; | |
services.ipfs.extraFlags = [ "--enable-namesys-pubsub" ]; | |
services.ipfs.extraConfig = { | |
Datastore = { StorageMax = "50GB"; }; | |
Routing = { Type = "dhtclient"; }; | |
}; | |
services.ipfs.swarmAddress = [ | |
"/ip4/0.0.0.0/tcp/4001" | |
"/ip4/136.214.64.44/tcp/4001" | |
"/ip4/136.214.64.44/udp/4001/quic" | |
"/ip4/0.0.0.0/udp/4001/quic" | |
]; | |
``` | |
## Testing your gateway | |
Let's say your gateway is http://localhost:8080/ for making simpler | |
incoming examples. If you want to request the data | |
/ipfs/QmRVD1V8eYQuNQdfRzmMVMA6cy1WqJfzHu3uM7CZasD7j1 , you just have to | |
add this to your gateway, like this: | |
http://localhost:8080/ipfs/QmRVD1V8eYQuNQdfRzmMVMA6cy1WqJfzHu3uM7CZasD7 | |
j1 and you will get access to your file. | |
When using ipns, it's quite the same, for /ipns/blog.perso.pw/ you can | |
request http://localhost:8080/ipns/blog.perso.pw/ and then you can | |
browse my blog. | |
# OpenBSD experiment | |
To make all of this really useful, I started an experiment: | |
distributing OpenBSD amd64 -current and 6.9 both with sets and packages | |
over IPFS. Basically, I have a server making a rsync of both sets once | |
a day, will add them to the local IPFS node, get the CID of the top | |
directory and then publish the CID under an IPNS. Note that I have to | |
create an index.html file in the packages sets because IPFS doesn't | |
handle directory listing very well. | |
The following examples will have to be changed if you don't use a local | |
gateway, replace localhost:8080 by your favorite IPFS gateway. | |
You can upgrade your packages with this command: | |
```upgrading OpenBSD packages by fetching data over IPFS | |
env PKG_PATH=http://localhost:8080/ipns/k51qzi5uqu5dmebzq75vx3z23lsixir3cxi26ck… | |
``` | |
You can switch to latest snapshot: | |
```upgrading to latest snapshot by fetching data over IPFS | |
sysupgrade -s http://localhost:8080/ipns/k51qzi5uqu5dmebzq75vx3z23lsixir3cxi26c… | |
``` | |
While it may be slow to update at first, if you have many systems, | |
running a local gateway used by all your computers will allow to have a | |
cache of downloaded packages, making the whole process faster. | |
I made a "versions.txt" file in the top directory of the repository, it | |
contains the date and CID of every publication, this can be used to | |
fetch a package from an older set if it's still available on the | |
network (because I don't plan to keep all sets, I have a limited disk). | |
You can simply use the url | |
http://localhost:8080/ipns/k51qzi5uqu5dmebzq75vx3z23lsixir3cxi26ckl409y | |
lblbjigjb1oluj3f2z/pub/OpenBSD/ in the file /etc/installurl to globally | |
use IPFS for pkg_add or sysupgrade without specifying the url every | |
time. | |
# Using DNS | |
It's possible to use a DNS entry to associate an IPFS resource to a | |
domain name by using dnslink. The entry would look like: | |
```dns entry for a dnslink TXT record | |
_dnslink.blog IN TXT "dnslink=/ipfs/somehashhere" | |
``` | |
Using an /ipfs/ syntax will be faster to resolve for IPFS nodes but you | |
will need to update your DNS every time you update your content over | |
IPFS. | |
To avoid manipulating your DNS every so often (you could use an API to | |
automate this by the way), you can use an /ipns/ record. | |
```dns entry for a dnslink TXT record using IPNS | |
_dnslink.blog IN TXT "dnslink=/ipns/something" | |
``` | |
This way, I made my blog available under the hostname blog.perso.pw but | |
it has no A or CNAME so it work only in an IPFS context (like a web | |
browser with IPFS companion extension). Using a public gateway, the | |
url becomes https://ipfs.io/ipns/blog.perso.pw/ and it will download | |
the last CID associated to blog.perso.pw. | |
# Conclusion | |
IPFS is a wonderful piece of technology but in practice it's quite slow | |
for DSL users and may not work well if you don't need a local cache. I | |
do really love it though so I will continue running the OpenBSD | |
experiment. | |
Please write me if you have any feedback or that you use my OpenBSD | |
IPFS repository. I would be interested to know about people's | |
experiences. | |
# Interesting IPFS resources | |
dweb-primer tutorials for IPFS (very well written) | |
Official IPFS documentation | |
IPFS companion for Firefox and Chrom·ium·e | |
Pinata.cloud is offering IPFS hosting (up to 1 GB for free) for pinned content | |
Wikipedia over IPFS | |
OpenBSD website/faq over IPFS (maintained by solene@) |