Introduction
Introduction Statistics Contact Development Disclaimer Help
Add mqtt hackathon - brcon2024-hackathons - Bitreichcon 2024 Hackathons
git clone git://bitreich.org/brcon2024-hackathons git://enlrupgkhuxnvlhsf6lc3fz…
Log
Files
Refs
Tags
Submodules
---
commit 7ac09da5aedd5b5e3fbbdcc57adadb263f09d011
parent 657b359aba0a48bc973a0cffe152a941297f6543
Author: Christoph Lohmann <[email protected]>
Date: Thu, 8 Aug 2024 23:01:46 +0200
Add mqtt hackathon
Diffstat:
A mqtt/README.md | 8 ++++++++
A mqtt/brcon2024_adc.md | 483 +++++++++++++++++++++++++++++…
A mqtt/mqtt-wrapper | 59 +++++++++++++++++++++++++++++…
A mqtt/tamagotchi | 1 +
4 files changed, 551 insertions(+), 0 deletions(-)
---
diff --git a/mqtt/README.md b/mqtt/README.md
@@ -0,0 +1,8 @@
+# MQTT
+
+## Following the talk of Thursday
+
+brcon2024_adc.md has the challenges
+tamagotchi/ for the tamagotchi source
+mqtt-wrapper for running some command
+
diff --git a/mqtt/brcon2024_adc.md b/mqtt/brcon2024_adc.md
@@ -0,0 +1,483 @@
+## brcon2024 - 2024-08-08
+
+ title: Deep C programming - UNIX philosophy in distributed offshore systems
+
+ author: Anders Damsgaard (adc)
+
+ contact: [email protected]
+ gopher://adamsgaard.dk
+ https://adamsgaard.dk
+
+ slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
+
+## brcon2024 - 2024-08-08
+
+ title: Deep C programming - UNIX philosophy in distributed offshore systems
+
+ author: Anders Damsgaard (adc)
+
+ contact: [email protected]
+ gopher://adamsgaard.dk
+ https://adamsgaard.dk
+ <marquee><blink>
+ slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
+ </blink></marquee>
+
+
+## Previously..
+* brcon2020: Energy efficient programming in science
+* brcon2021: Unix principles for science simulations
+* brcon2022: Est. Bitreich Arctic Vault
+* brcon2023: Using Unix and Bitreich tools for preparing scientific papers
+* Today: UNIX philosophy in distributed offshore systems
+#pause
+* Tomorrow: UNOX philosophy in Vitré
+
+## Outline
+* How we can benefit from hyped technology with true and tried principles
+* Overview of the offshore system
+* The talk transitions into hands-on exercises
+* Ask questions along the way in #bitreich-con
+
+## Danish Geotechnical Institute (geo.dk)
+* ~250 employees in Lyngby and Aarhus, Denmark
+* Geotechnical investigations (onshore/offshore)
+* Environmental consultancy
+* Laboratory tests
+* Geodata analysis
+* Geoscientific software development
+
+#pause
+-> New offshore rig built from the ground up
+
+## Offshore geotechnical investigations
+ .* +------------------+
+ \|\ | ,_____. |
+ _============. | | | |
+ \\# # # #/ | | ._______|--|--| |
+ | +.,| | Geo .--|--| |
+ _---------------------|-+-------|--|--|--|\
+ \ MS/Bitreich | +--|--+ | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~|~|~~~~~~~~~~~~
+ \ +------------------+'' ~
+ \_________________________________/--+ ~ . ~~
+ ~
+
+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo .--|--|
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich +--|--+ |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~
+ \ .-'' ~
+ \_________________________________/--+ ~ . ~~
+ ~
+
+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'' ~
+ \_____________________________|___/--x~ ~ . ~~
+ .--|--. ~
+ | v |
+ +-----+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'~ ~ . ~~
+ \_____________________________|___/--x ~
+ |
+ |
+ |
+ ><> |
+ |
+ <>< |
+ |
+ .--|--.
+ | v |
+----------------------------------------------+-----+-----------------
+
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'' ~
+ \_____________________________|___/--x
+ |
+ |
+ |
+ ><> |
+ |
+ <>< |
+ |
+ .--|--.
+ | | |
+----------------------------------------------+--|--+-----------------
+ |
+ |
+ |
+ v
+
+## Offshore geotechnical investigations
+ | |
+ | |
+ | |
+ | |
+ | |
+ | |
+ | |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~|~~~~~~~~~~~~
+ | |
+ | |
+ | |
+ | |
+ ? | |
+ ><> | |
+ | |
+ | | !
+ | | <><
+ | |
+ | |
+-----------------------------------------|---------------|------------
+ | |
+ | |
+ | |
+ | |
+ |_______________|
+
+https://www.mdpi.com/energies/energies-17-01964/article_deploy/html/images/ene…
+
+## Offshore systems overview
+* Up to 800 sensors, usually 1-10 Hz
+* SCADA, PLCs, environmental sensors, navigation, scientific results,
+ calibration data, ...
+* Control messages
+* Asynchronous data streams
+* Event driven logic
+* Raw sensor output into physical units
+* Aggregation of time series and combination of sources
+* Time series databases
+* Development project with many involved partners
+
+## Approaches: Unix philosophy vs. Microsoft mentality
+* Does the client know what they want?
+* Do we know everything about what we want to solve?
+* Do we know if our assumptions are valid, in particular in complex systems?
+* Do we have full specifications?
+* Should we produce a big integrated C# monolith to handle computations,
+ flow, storage, logic?
+
+#pause
+ > yes no | head -n 5
+
+
+## Data routing system
+* MQTT: Message Queuing Telemetry Transport
+* Started as a proprietary protocol for SCADA systems in O&G
+* Lightweight, publish-subscribe network protocol
+* Constrained devices and low-bandwidth, high-latency networks
+* Publish/Subscribe Model: Decouples message producers from consumers
+* Data accessible via network, not living internally in an obscure program
+
+Specifications:
+- MQTT v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
+- MQTT v3.1.1: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.…
+- https://github.com/mqtt/mqtt.org/wiki/Differences-between-3.1.1-and-5.0
+- MQTT and TLS: https://www.rfc-editor.org/rfc/rfc9431.txt
+
+## Central concepts
+* Broker: Central server that routes messages
+* Clients: Devices that publish or subscribe to one or more topics
+* Messages: Data packets sent between clients via the broker
+* Topics: Hierarchical and dynamic name space for message
+
+Libs for many languages, e.g.,
+ - https://github.com/eclipse/paho.mqtt.c
+ - https://github.com/eclipse/paho.mqtt.golang
+
+## DISCLAIMER
+
+The coming exercises mainly to demonstrate fundamentals and principles
+
+Most are suboptimal or bad practice
+
+But, let's have some #fun ...
+
+## MQTT use cases
+* Home thermometers and weather stations (if you ask the web)
+* IoT Devices: Smart homes, wearables, and industrial automation
+* Mobile Applications: Real-time messaging and notifications
+* Remote Monitoring: Environmental sensors, health monitoring
+* Connected Vehicles: Telematics and fleet management
+
+## MQTT messages
+* TCP/IP
+* Limit to message size configurable
+
+* New subscribers receive only new messages (no backlog)
+* Retained Messages: Last known good value for new subscribers
+* Last Will and Testament (LWT): Notifies other clients about an ungraceful di…
+
+* Quality of Service (QoS): Three levels to ensure message delivery
+* QoS 0: At most once delivery (fire and forget)
+* QoS 1: At least once delivery (acknowledged delivery)
+* QoS 2: Exactly once delivery (assured delivery)
+
+## Why MQTT for the offshore project?
+* Hierarchial data structure through topics
+* A node in the topic tree exists as soon as someone publishes or
+ subscribes to it
+* Data accessible via network and all clients can read or publish
+ (by default)
+
+#pause
+ rig/sensor1/raw
+ rig/sensor1/phys
+ rig/sensor2/raw
+ rig/sensor2/phys
+ ...
+ ship/navigation/easting
+ ship/navigation/northing
+ ...
+
+## Topic examples
+Wildcards: #, + (subscribe only)
+
+Subscription examples:
+ 1) rig/sensor1/raw
+ 2) ship/#: Subscribe to everything from the ship
+ 3) rig/+/phys: Subscribe to all physical values
+
+## Example: Subscribe and publish
+$ cat bin/timestamp # in case you don't have ts(1)
+#!/bin/sh
+while read -r REPLY
+do
+ printf '%s %s\n' "$(date '+%Y-%m-%d %Z %H:%M:%S')" "$REPLY"
+done
+#pause
+$ export MQTTURI='mqtt://bitreich:[email protected]:65431/'
+#pause
+$ mosquitto_sub -L "${MQTTURI}#" -v -t '#' | timestamp | grep -v hatesensor
+#pause
+
+$ date | mosquitto_pub -L "${MQTTURI}$(hostname)/$(whoami)" -l
+
+## Reporting online status via last will and retain, reporting some metrics
+
+gophers://duckless.org/0/tmp/mqtt-online
+
+#!/bin/sh
+online_topic="${ONLINE_TOPIC:-$(hostname)/online}"
+mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
+mqtt_port="${MQTT_PORT:-65431}"
+mqtt_user="${MQTT_USER:-bitreich}"
+mqtt_password="${MQTT_PASSWORD:-bitreich}"
+args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_password}"
+mosquitto_pub ${args} -t "${online_topic}" -r -m "1"
+while :
+do
+ printf '%s\t%s\t%s\t%s\n' "$(date +%s)" "$(date)" "$(hostname)" "$(uptime)"
+ sleep 360
+done | mosquitto_pub ${args} --will-topic "${online_topic}" \
+ --will-retain --will-payload "0" -t "${online_topic}" -r -l
+
+## Data structure on the network
+Interact with data hierarchy via small modular clients.
+Brings all benefits of UNIX design into play.
+
+The tenents of the UNIX philosophy:
+
+ - Small is beautiful
+ - Make each program do one thing well
+ - Build a prototype as soon as possible
+ - Choose portability over efficiency
+ - Store numerical data in flat ASCII files
+ - Use software leverage to your advantage
+ - Use shell scripts to increase leverage and portability
+ - Avoid captive user interfaces
+ - Make every program a filter
+
+src: Make Gancarz 1995 "The UNIX Philosophy", ISBN 1-55558-123-4
+
+## Distributed computing
+Assign small modular programs to read stdin from topics and post their
+result to another topic
+
+* Rudimentary distributed computing
+* Each program acts as a text filter
+* Each program can contribute local data
+
+(disclaimer: for 1:1 connections, netcat or named pipes over SSH is simpler)
+
+Broker statistics: $SYS/broker
+ mosquitto_sub -L "${MQTTURI}\$SYS/broker/#" -v
+
+## Connecting stdin, stdout, and stderr with MQTT topics
+Full script: gophers://duckless.org/0/tmp/mqtt-wrapper
+
+ fifodir="$(mktemp -d)" || die 'mktemp'
+ fin="${fifodir}/in"
+ fout="${fifodir}/out"
+ ferr="${fifodir}/err"
+ mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
+
+#pause
+ mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
+ mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
+ mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
+ trap -- 'cleanup' ERR INT HUP EXIT CHLD
+
+#pause
+ while :
+ do
+ $runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
+ done
+
+## Connecting stdin, stdout, and stderr with MQTT topics (cont.)
+
+Works if input is continuously evaluated (i.e., does not wait for EOF)
+
+ RUNTIME=bc ARGS="-l" sh -x ./mqtt-wrapper
+ RUNTIME=sh ARGS="your-favorite-cgi-script" sh -x ./mqtt-wrapper
+ RUNTIME=sh ARGS="your-sd-program" sh -x ./mqtt-wrapper
+
+## Outlook
+* Lots of possibilities coupling pub/sub with text streams
+* Use existing client/broker implementations and libraries
+* Minimal C implementation would be nice, aiming at a subset of features
+* Represent topic structure with virtual file system representation
+ (i.e., plan9 or /proc in linux)
+
+## Challenge 0: Installing a MQTT broker/client
+Mosquitto (mostly C, BSD-like license).
+Pretty bloated, a minimal alternative would be nice.
+You just need the client for these exercises.
+
+ https://github.com/eclipse/mosquitto
+
+Package names in common OS:
+
+ OpenBSD mosquitto broker+client
+ Arch mosquitto broker+client
+ Gentoo app-misc/mosquitto broker+client
+ Debian mosquitto-clients client
+
+## Challenge agenda
+First one to post a working solution will get points towards a
+price.
+
+## Challenge 1: Let's write a chat
+Hints:
+
+ mosquitto_pub(1): Publish a message to a named topic
+ Useful flags:
+ -d: Show debug info
+ -l: Read messages from stdin (each line 1 message)
+ mosquitto_sub(1): Subscribe to a path in the topic hierarchy
+ -d: Show debug info
+ -v: Print topic for each received message as first field
+
+ Connection URI:
+ -L 'mqtt://bitreich:[email protected]:65431/testtopic'
+
+Let's use the message format: "nick\tmessage"
+
+## Challenge 2: Write a script that records the chat
+Hint: The '#' wild card
+
+What is the best choice for a time stamp?
+* Time as the sender reports it?
+* Time as the subscriber sees it?
+
+## Challenge 3: Let the data flow
+Anyone has a funky data source?
+
+* USB body thermometer?
+* spoon -t
+* while :; do printf '%s\t%s' "$(hostname)" "$(uptime)"; sleep 10; done
+* ii/annna?
+* vmstat(8)
+
+Task: Publish some data and let the others guess what it is.
+The funkier, the better.
+
+Message format ideas:
+ TSV, JSON, XLSX? sfeed(5)?
+ VT100 control codes??
+
+## Challenge 4: Raise an alert!
+Set up a script that monitors one of the data sources, and triggers
+an alert upon a condition (email, xmessage, log line, ...).
+
+## Challenge 5: Make some gnuplot
+Monitor one of the sensors from the previous, and create a continuously
+updating gnuplot (-t dumb) with a rolling window.
+
+## Challenge 6: Health check reporting
+Write a script that monitors some gopher and reports their status to
+
+ gopherstatus/<uri>
+ up|down
+
+## Challenge 7: Tamagotchi
+Make script that imitates a virtual pet, expecting to be feed
+"cookies" as messages on stdin every 30 s. It should die if it gets to much o…
+
+Report its happyness to a topic with ASCII smileys.
+
diff --git a/mqtt/mqtt-wrapper b/mqtt/mqtt-wrapper
@@ -0,0 +1,59 @@
+#!/bin/sh
+set -x
+#
+# $input_topic | $runtime $runtime_args | $output_topic
+# 2>$error_topic
+#
+# application to run
+runtime="${RUNTIME:-cat}"
+# arguments to $runtime
+runtime_args="${ARGS}"
+# stdin for $runtime; where others send to
+input_topics="${INPUT_TOPICS:-$$/in}"
+# stdout of $runtime; where others need to subscribe
+output_topic="${OUTPUT_TOPIC:-$$/out}"
+# stderr of $runtime; where others can subscribe
+error_topic="${ERROR_TOPIC:-$$/err}"
+
+mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
+mqtt_port="${MQTT_BROKER:-65431}"
+mqtt_user="${MQTT_USER:-bitreich}"
+mqtt_password="${MQTT_PASSWORD:-bitreich}"
+mosquitto_args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_pa…
+
+cleanup() {
+ kill -TERM -- -$$
+ rm -f "$fin" "$fout" "$ferr"
+ rmdir "$fifodir"
+ exit 1
+}
+
+die() {
+ printf '%s: %s\n' "${0##*/}" "${1}" >&2
+ cleanup
+ exit "${2:-1}"
+}
+
+fifodir="$(mktemp -d)" || die 'mktemp'
+fin="${fifodir}/in"
+fout="${fifodir}/out"
+ferr="${fifodir}/err"
+mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
+
+# split multiple input topics separated by "," into separate args
+subscribe_args="$(printf '%s' "$input_topics" | sed 's/,/ -t /g')"
+
+# subscribe to input topic(s), add -v option if you want message topic as firs…
+mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
+
+# create named pipes for output streams
+mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
+mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
+trap -- 'cleanup' ERR INT HUP EXIT CHLD
+
+# run the program (lossy restart if it fails), connecting to named pipes
+while :
+do
+ $runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
+done
+
diff --git a/mqtt/tamagotchi b/mqtt/tamagotchi
@@ -0,0 +1 @@
+Subproject commit 61808fed51f116f4c1b5ab622a9e3e76738b303b
You are viewing proxied material from bitreich.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.