Title: Using Netdata on NixOS and connecting to Netdata cloud

	Title: Using Netdata on NixOS and connecting to Netdata cloud
	Author: Solène
	Date: 16 September 2022
	Tags: nixos monitoring netdata cloud
	Description: In this article, you will learn about Netdata monitoring
	tool, how to use it on NixOS and how to connect it to the free Netdata
	privacy friendly cloud solution

	# Introduction

	I'm still playing with monitoring programs, and I've been remembered
	about Netdata. What an improvement over the last 8 years!

	This tutorial explains how to get Netdata installed on NixOS, and how
	to register your node in Netdata cloud.

	Netdata GitHub project page
	Netdata live demo

	# What's Netdata?

	This program is a simple service to run on a computer, it will
	automatically gather a ton of metrics and make them easily available
	over the local TCP port 19999. You just need to run Netdata and
	nothing else, and you will have every metrics you can imagine from your
	computer, and some explanations for each of them!

	That's pretty cool because Netdata is very efficient, it draws nearly
	no CPU while gathering a few thousands metrics every few seconds, and
	is memory efficient and can be constrained to a dozen of megabytes.

	While you can export its metrics to something like graphite or
	Prometheus, you lose the nice display which is absolutely a blast
	compare to Grafana (in my opinion).

	Update: as pointed out by a reader (thanks!), it's possible to connect
	Netdata instances to only one used for viewing metrics. I'll
	investigate this soon.

	Netdata documentation about streaming.

	Netdata also added some machine learning anomaly detection, it's simple
	and doesn't use many resources or require a GPU, it only builds
	statistical models to be able to report if some metrics have an unusual
	trend. It takes some time to gather enough data, and after a few days
	it's starting to work.

	# Installing Netdata on NixOS

	As usual, it's simple, add this to your NixOS configuration and
	reconfigure the system.

	```
	services.netdata = {
	enable = true;

	config = {
	global = {
	# uncomment to reduce memory to 32 MB
	#"page cache size" = 32;

	# update interval
	"update every" = 15;
	};
	ml = {
	# enable machine learning
	"enabled" = "yes";
	};
	};
	};
	```

	You should have Netdata dashboard available on http://localhost:19999 .

	## Streaming mode

	Here is a simple configuration on NixOS to connect a headless node
	without persistency to send all on a main Netdata server storing data
	but also displaying them.

	You need to generate an UUID with uuidgen, replace UUID in the text
	with the result. It can be per system or shared by multiple Netdata
	instances.

	My networks are 10.42.42.0/24 and 10.43.43.0/24, I'll allow everything
	matching 10.* on the receiver, I don't open port 19999 on a public
	interface.

	### Senders

	```
	services.netdata.enable = true;
	services.netdata.config = {
	global = {
	"default memory mode" = "none"; # can be used to disable local data s…
	};
	};
	services.netdata.configDir = {
	"stream.conf" = pkgs.writeText "stream.conf" ''
	[stream]
	enabled = yes
	destination = 10.42.42.42:19999
	api key = UUID
	[UUID]
	enabled = yes
	'';
	};
	```

	### Receiver

	```
	networking.firewall.allowedTCPPorts = [19999];
	services.netdata.enable = true;
	services.netdata.configDir = {
	"stream.conf" = pkgs.writeText "stream.conf" ''
	[UUID]
	enabled = yes
	default history = 3600
	default memory mode = dbengine
	health enabled by default = auto
	allow from = 10.*
	'';
	};
	```


	# Netdata cloud

	Netdata company started a "cloud" offer that is free, but they plan to
	keep it free but also propose more services for paying subscribers.
	The free plan is just a convenience to see metrics from multiple nodes
	at the same place, they don't store any metrics apart metadata (server
	name, OS version, kernel, etc..), when you look at your metrics, they
	just relay from your server to your web browser without storing the
	data.

	The free cloud plan offers a correlating feature, but I still didn't
	have the opportunity to try it, and also email alerting when an alarm
	is triggered.

	Netdata cloud website
	Netdata cloud data privacy information

	## Adding a node

	The official way to connect a Netdata agent to the Netdata cloud is to
	use a script downloaded on the internet and run it with some parameter.

	Connecting a Linux agent

	I strongly dislike this method as I'm not a huge fan of downloading
	script to run as root that are not provided by my system.

	When you want to add a new node, you will be given a long command line
	and a token, keep that token somewhere. NixOS Netdata package offers a
	script named `netdata-claim.sh` (which seems to be part of Netdata
	source code) that will generate a pair of RSA keys, and look for the
	token in a file.

	Netdata data page: Add a node

	Once you got the token, we will claim it to associate it to a node:

	1. create /var/lib/netdata/cloud.d/token and write the token in it
	2. run nix-shell -p netdata --run "netdata-claim.sh" as root
	3. your node should be registered in Netdata cloud

	# Conclusion

	Netdata is really a wonderful tool, ideally I'd like it to replace all
	the Grafana + storage + agent stack, but it doesn't provide persistent
	centralized storage compatible with its dashboard. I'm going to
	experiment with their Netdata cloud service, I'm not sure if it would
	add value for me, and while they have a very correct data privacy
	policy, I prefer to self-host everything.