Title: Using Netdata on NixOS and connecting to Netdata cloud | |
Author: Solène | |
Date: 16 September 2022 | |
Tags: nixos monitoring netdata cloud | |
Description: In this article, you will learn about Netdata monitoring | |
tool, how to use it on NixOS and how to connect it to the free Netdata | |
privacy friendly cloud solution | |
# Introduction | |
I'm still playing with monitoring programs, and I've been remembered | |
about Netdata. What an improvement over the last 8 years! | |
This tutorial explains how to get Netdata installed on NixOS, and how | |
to register your node in Netdata cloud. | |
Netdata GitHub project page | |
Netdata live demo | |
# What's Netdata? | |
This program is a simple service to run on a computer, it will | |
automatically gather a ton of metrics and make them easily available | |
over the local TCP port 19999. You just need to run Netdata and | |
nothing else, and you will have every metrics you can imagine from your | |
computer, and some explanations for each of them! | |
That's pretty cool because Netdata is very efficient, it draws nearly | |
no CPU while gathering a few thousands metrics every few seconds, and | |
is memory efficient and can be constrained to a dozen of megabytes. | |
While you can export its metrics to something like graphite or | |
Prometheus, you lose the nice display which is absolutely a blast | |
compare to Grafana (in my opinion). | |
Update: as pointed out by a reader (thanks!), it's possible to connect | |
Netdata instances to only one used for viewing metrics. I'll | |
investigate this soon. | |
Netdata documentation about streaming. | |
Netdata also added some machine learning anomaly detection, it's simple | |
and doesn't use many resources or require a GPU, it only builds | |
statistical models to be able to report if some metrics have an unusual | |
trend. It takes some time to gather enough data, and after a few days | |
it's starting to work. | |
# Installing Netdata on NixOS | |
As usual, it's simple, add this to your NixOS configuration and | |
reconfigure the system. | |
``` | |
services.netdata = { | |
enable = true; | |
config = { | |
global = { | |
# uncomment to reduce memory to 32 MB | |
#"page cache size" = 32; | |
# update interval | |
"update every" = 15; | |
}; | |
ml = { | |
# enable machine learning | |
"enabled" = "yes"; | |
}; | |
}; | |
}; | |
``` | |
You should have Netdata dashboard available on http://localhost:19999 . | |
## Streaming mode | |
Here is a simple configuration on NixOS to connect a headless node | |
without persistency to send all on a main Netdata server storing data | |
but also displaying them. | |
You need to generate an UUID with uuidgen, replace UUID in the text | |
with the result. It can be per system or shared by multiple Netdata | |
instances. | |
My networks are 10.42.42.0/24 and 10.43.43.0/24, I'll allow everything | |
matching 10.* on the receiver, I don't open port 19999 on a public | |
interface. | |
### Senders | |
``` | |
services.netdata.enable = true; | |
services.netdata.config = { | |
global = { | |
"default memory mode" = "none"; # can be used to disable local data s… | |
}; | |
}; | |
services.netdata.configDir = { | |
"stream.conf" = pkgs.writeText "stream.conf" '' | |
[stream] | |
enabled = yes | |
destination = 10.42.42.42:19999 | |
api key = UUID | |
[UUID] | |
enabled = yes | |
''; | |
}; | |
``` | |
### Receiver | |
``` | |
networking.firewall.allowedTCPPorts = [19999]; | |
services.netdata.enable = true; | |
services.netdata.configDir = { | |
"stream.conf" = pkgs.writeText "stream.conf" '' | |
[UUID] | |
enabled = yes | |
default history = 3600 | |
default memory mode = dbengine | |
health enabled by default = auto | |
allow from = 10.* | |
''; | |
}; | |
``` | |
# Netdata cloud | |
Netdata company started a "cloud" offer that is free, but they plan to | |
keep it free but also propose more services for paying subscribers. | |
The free plan is just a convenience to see metrics from multiple nodes | |
at the same place, they don't store any metrics apart metadata (server | |
name, OS version, kernel, etc..), when you look at your metrics, they | |
just relay from your server to your web browser without storing the | |
data. | |
The free cloud plan offers a correlating feature, but I still didn't | |
have the opportunity to try it, and also email alerting when an alarm | |
is triggered. | |
Netdata cloud website | |
Netdata cloud data privacy information | |
## Adding a node | |
The official way to connect a Netdata agent to the Netdata cloud is to | |
use a script downloaded on the internet and run it with some parameter. | |
Connecting a Linux agent | |
I strongly dislike this method as I'm not a huge fan of downloading | |
script to run as root that are not provided by my system. | |
When you want to add a new node, you will be given a long command line | |
and a token, keep that token somewhere. NixOS Netdata package offers a | |
script named `netdata-claim.sh` (which seems to be part of Netdata | |
source code) that will generate a pair of RSA keys, and look for the | |
token in a file. | |
Netdata data page: Add a node | |
Once you got the token, we will claim it to associate it to a node: | |
1. create /var/lib/netdata/cloud.d/token and write the token in it | |
2. run nix-shell -p netdata --run "netdata-claim.sh" as root | |
3. your node should be registered in Netdata cloud | |
# Conclusion | |
Netdata is really a wonderful tool, ideally I'd like it to replace all | |
the Grafana + storage + agent stack, but it doesn't provide persistent | |
centralized storage compatible with its dashboard. I'm going to | |
experiment with their Netdata cloud service, I'm not sure if it would | |
add value for me, and while they have a very correct data privacy | |
policy, I prefer to self-host everything. |