An exec plugin for collectd: mqttcollect

Ben and Alexander recently twisted my arm until I promised to create
something which would consume [1]Mosquitto's $SYS/broker/# topic
branch and write those values to [2]collectd. You will know that that
topic branch emits broker statistics [3]via MQTT:
$SYS/broker/bytes/received 79665672
$SYS/broker/bytes/sent 27887950
$SYS/broker/load/messages/received/1min 70.10

My first thought was: let me add a service to [4]mqttwarn for that (if
you don't know [5]mqttwarn you may be interested in [6]these [7]two
articles introducing it), but for the systems it was to run on, it was
to be a standalone thing in C. The first cut of the program was
operational within a couple of hours: it is hooked into collectd via
[8]collectd's exec plugin which launches a long-lived process and
reads metrics it issues from stdin.

It then occurred to me I could also handle [9]JSON payloads easily
enough, extracting an element's value from the JSON to use as a
number. One thing led to another, and I then wanted elements from the
JSON payload to be interpolated into the metric names [10]collectd is
given, so I added that as well. The result is a rather fast minimal
[11]mqttwarn which hands metrics to [12]collectd.

An ini-type configuration file provides the settings required to run
mqttcollect, instead of a dozen command-line options. Hostname, port,
CA certificate file for TLS, TLS-PSK are all supported. The nodename
given to collectd is configurable (it defaults to the short uname) as
is an optional prefix which is, well, prefixed to a metric to
differentiate instances of a plugin.
[defaults]
host = localhost
port = 1883
username = jane
password = s1c#ret
; psk_key =
; psk_identity =
; ca_file =
; nodename = foob
; progname = mqttcollect
; prefix   = PREFIX

You configure any number of topics which mqttcollect will subscribe
to, and you specify the [13]type of metric as well as whether or not
the MQTT topic should be translated before being handed off. There are
three possibilities:
1. No translation. E.g a topic temperature/arduino is passed through
unchanged.
2. Rewrite a topic (e.g. temperature/arduino) to hot/in/here.
3. Rewrite with JSON interpolation.

The first two mechanisms are described in the
[14]mqttcollect.ini.example file we provide. The third is a bit more
difficult to grasp, so let me elaborate.

metric configuration

Assume for instance I subscribe to the wildcarded topic branch
arduino/temp/+ and that the payload of messages which are published on
that branch contains [15]JSON with a celsius and fahrenheit
temperatures as well as the name of the room in which the temperature
was measured.
{"fahrenheit": 53.26, "celsius": 11.81, "room": "kitchen"}

I can have mqttcollect use each of the elements in the JSON (e.g.
celsius, room, etc.) to construct the name of a metric. So, if I
configure, say, a metric name of heat.{room} and, as shown above, the
JSON payload has a { "room" : "kitchen" } in it, the metric name will
be rewritten to heat.kitchen.

You'll notice the < character followed by a word; this indicates that
mqttcollect should retrieve the value of the metric from said JSON
element. So, for example, <celsius means: the value of the metric
should be taken from the payload's { "celsius" : 11.81 } element.
Likewise, <fahrenheit would consume that value, but who'd want that?
;-)

Putting this together, if I configure mqttcollect with this section:
[arduino/temp/+]
gauge = heat.{room}<celsius

mqttcollect will rewrite the metric name from the JSON payload and
obtain the value for the metric from the same JSON payload, handing
collectd the following line:
PUTVAL tiggr/mqttcollect/gauge-heat.kitchen 1431548550:11.81

A practical example we're using this for is for our [16]OwnTracks
Greenwich devices which publish JSON payloads like this example
(shortened for clarity):
{
   "_type": "location",
   "alt": 53,
   "tid": "BB",
   "vel": 62
}

mqttcollect will, when configured as below, produce three metrics per
message it receives.
[owntracks/+/+]
gauge = vehicle/{tid}/speed<vel
gauge = vehicle/{tid}/altitude<alt
counter = vehicle/{tid}/odometer<trip
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/speed 1431543655:62.00
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/altitude 1431543655:53.00
PUTVAL tiggr/mqttcollect/counter-vehicle/BB/odometer 1431543655:672798
00

This lets us produce nice graphs with all sorts of useless
information, such as at which altitude vehicles are driving at using
[17]InfluxDB and Grafana.

Viewed in Grafana

I've put the source code to mqttcollect [18]up on this repository.

References

  1. http://mosquitto.org/
  2. https://collectd.org/
  3. http://jpmens.net/2013/02/25/lots-of-messages-mqtt-pub-sub-and-the-mosquitto-broker/
  4. https://github.com/jpmens/mqttwarn
  5. https://github.com/jpmens/mqttwarn
  6. https://jpmens.net/2014/02/17/introducing-mqttwarn-a-pluggable-mqtt-notifier/
  7. https://jpmens.net/2014/04/03/how-do-your-servers-talk-to-you/
  8. https://collectd.org/wiki/index.php/Plugin:Exec
  9. http://json.org/
 10. https://collectd.org/
 11. https://github.com/jpmens/mqttwarn
 12. https://collectd.org/
 13. https://github.com/astro/collectd/blob/master/src/types.db
 14. https://github.com/jpmens/mqttcollect/blob/master/mqttcollect.ini.example
 15. http://json.org/
 16. https://jpmens.net/2014/09/18/choral-greenwich-owntracks-edition-goes-into-production/
 17. https://jpmens.net/2014/07/10/metrics-in-a-home/
 18. https://github.com/jpmens/mqttcollect

You are viewing proxied material from serf.jpmens.net. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.