Title: Nushell: Introduction to a new kind of shell | |
Author: Solène | |
Date: 31 October 2022 | |
Tags: openbsd nixos nushell shell | |
Description: This article covers a gentle introduction to the shell | |
nushell | |
# What is nushell | |
Let me introduce you to a nice project I found while lurking on the | |
Internet. It's called nushell and is a non-POSIX shell, so most of | |
your regular shells knowledge (zsh, bash, ksh, etc…) can't be applied | |
on it, and using it feels like doing functional programming. | |
It's a good tool for creating robust data manipulation pipelines, you | |
can think of it like a mix of a shell which would include awk's power, | |
behave like a SQL database, and which knows how to import/export | |
XML/JSON/YAML/TOML natively. | |
You may want to try nushell only as a tool, and not as your main shell, | |
it's perfectly fine. | |
With a regular shell, iterating over a command output can be complex | |
when it involves spaces or newlines, for instance, that's why `find` | |
and `xargs` have a `-print0` parameter to have a special delimited | |
between "items", but it doesn't compose well with other tools. Nushell | |
handles correctly this situation as its manipulates the data using | |
indexed entries, given you correctly parsed the input at the beginning. | |
Nushell official project page | |
Nushell documentation website | |
# How to get it | |
Nushell is a rust program, so it should work on every platform where | |
Rust/Cargo are supported. I packaged it for OpenBSD, so it's available | |
on -current (and will be in releases after 7.3 is out), the port could | |
be used on 7.2 with no effort. | |
With Nix, it's packaged under the name `nushell`, the binary name is | |
`nu`. | |
For other platforms, it's certainly already packaged, otherwise you can | |
find installation instructions to build it from sources. | |
Nushell documentation: Building nushell from sources | |
# Configuration | |
At first run, you are prompted to use default configuration files, I'd | |
recommend accepting, you will have files created in | |
`~/.config/nushell/`. | |
The only change I made from now is to make Tab completion | |
case-sensitive, so `D[TAB]` completes to `Downloads` instead of asking | |
between `dev` and `Downloads`. Look for `case_sensitive_completions` in | |
`.config/nushell/config.nu` and set it to `true`. | |
# Examples | |
If you are like me, and you prefer learning by doing instead of reading | |
a lot of documentation, I prepared a bunch of real world use case you | |
can experiment with. The documentation is still required to learn the | |
many commands and syntax, but examples are a nice introduction. | |
## Getting help | |
Help from nushell can be parsed directly with nu commands, it's | |
important to understand where to find information about commands. | |
Use `help a-command` to learn from a single command: | |
```script | |
> help help | |
Display help information about commands. | |
Usage: | |
> help {flags} ...(rest) | |
Flags: | |
-h, --help - Display this help message | |
-f, --find <String> - string to find in command names, usage, and search terms | |
[cut so it's not too long] | |
``` | |
Use `help commands` to list all available commands (I'm limiting to 5 | |
between there are a lot of commands) | |
```script | |
help commands | last 5 | |
╭───┬─────────────┬───────�… | |
│ # │ name │ category │ is_plugin │ is_custom �… | |
├───┼─────────────┼───────�… | |
│ 0 │ window │ filters │ false │ false �… | |
│ 1 │ with-column │ dataframe or lazyframe │ false │ false �… | |
│ 2 │ with-env │ env │ false │ false �… | |
│ 3 │ wrap │ filters │ false │ false �… | |
│ 4 │ zip │ filters │ false │ false �… | |
╰───┴─────────────┴───────�… | |
``` | |
Add `sort-by category` to list them... sorted by category. | |
``` | |
help commands | sort-by category | |
``` | |
Use `where category == filters` to only list commands from the | |
`filters` category. | |
``` | |
help commands | where category == filters | |
``` | |
Use `find foobar` to return lines containing `foobar`. | |
``` | |
help commands | find insert | |
``` | |
## General examples | |
### Converting a data structure into another | |
This is just an example from YAML to JSON, but you can convert much | |
more formats into other formats. | |
``` | |
open dev/home-impermanence/tests/impermanence.yml | to json | |
{ | |
"directories": | |
[ | |
"Documents", | |
"Downloads", | |
"Datastore/Music", | |
"Datastore", | |
"Datastore/", | |
"Datastore/Music/Band1", | |
".config", | |
"foo/bar", | |
"foo/bar/hello" | |
], | |
"size": "500m", | |
"files": | |
[ | |
".Xdefaults", | |
".profile", | |
".xsession", | |
] | |
} | |
``` | |
### Parsing sysctl output | |
``` | |
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" | |
``` | |
Because the output would be too long, here is how you get 10 random | |
keys from sysctl. | |
``` | |
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" | shuffle | last 10 | sort-by k… | |
╭───┬─────────────────────�… | |
│ # │ key │ value │ | |
├───┼─────────────────────�… | |
│ 0 │ fs.quota.reads │ 0 │ | |
│ 1 │ net.core.high_order_alloc_disable │ 0 │ | |
│ 2 │ net.ipv4.conf.all.drop_gratuitous_arp │ 0 │ | |
│ 3 │ net.ipv4.conf.default.rp_filter │ 2 │ | |
│ 4 │ net.ipv4.conf.lo.disable_xfrm │ 1 │ | |
│ 5 │ net.ipv4.conf.lo.forwarding │ 0 │ | |
│ 6 │ net.ipv4.ipfrag_low_thresh │ 3145728 │ | |
│ 7 │ net.ipv6.conf.all.ioam6_id │ 65535 │ | |
│ 8 │ net.ipv6.conf.all.router_solicitation_interval │ 4 │ | |
│ 9 │ net.mptcp.enabled │ 1 │ | |
╰───┴─────────────────────�… | |
``` | |
### Recursively convert FLAC files to OPUS | |
A complicated task using a regular shell, recursively find files | |
matching a pattern and then run a given command on each of them, in | |
parallel. Which is exactly what you need if you want to convert your | |
music library into another format, let's convert everything from FLAC | |
to OPUS in this example. | |
In the following command line, we will look for every `.flac` file in | |
the subdirectories, then run in parallel using `par-each` the command | |
`ffmpeg` on it, from its current name to the old name with `.flac` | |
changed to `.opus`. | |
The `let convert` and `| complete` commands are used to store the | |
output of each command into a result table, and store it in the | |
variable `convert` so we can query it after the job is done. | |
``` | |
let convert = (ls **/*flac | par-each { |file| do -i { ffmpeg -i $file.name ($f… | |
``` | |
Now, we have a structure in `convert` that contains the columns | |
`stdout`, `stderr` and `exit_code`, so we can look if all the commands | |
did run correctly using the following query. | |
``` | |
$convert | where exit_code != 0 | |
``` | |
### Synchronize a music library to a compressed one | |
I had a special need for my phone and my huge music library, I wanted | |
to have a lower quality version of it synced with syncthing, but I | |
needed this to be easy to update when adding new files. | |
It takes all the music files in `/home/user/Music/` and creates a 64K | |
opus file in `/home/user/Stream/` by keeping the same file tree | |
hierarchy, and if the opus destination file exists it's skipped. | |
```nushell | |
cd /home/user/Music/ | |
let dest = "/home/user/Stream/" | |
let convert = (ls **/* | | |
where name =~ ".(mp3|flac|opus|ogg)$" | | |
where name !~ "(Audiobook|Piano)" | | |
par-each { | |
|file| do -i { | |
let new_name = ($file.name | str replac… | |
if (not ([$dest, $new_name] | str join … | |
mkdir ([$dest, ($file.name | pa… | |
ffmpeg -i $file.name -b:a 64K (… | |
} | complete | |
} | |
}) | |
$convert | |
``` | |
### Convert PDF/CBR/CBZ pages into webp and CBZ archives | |
I have a lot of digitalized books/mangas/comics, this conversion is a | |
handy operation reducing the size of the files by 40% (up to 70%). | |
``` | |
def conv [] { | |
if (ls | first | get name | str contains ".jpg") { | |
ls *jpg | par-each {|file| do -i { cwebp $file.name -o ($file.name | … | |
rm *jpg | |
} | |
if (ls | first | get name | str contains ".ppm") { | |
ls *ppm | par-each {|file| do -i { cwebp $file.name -o ($file.name | … | |
rm *ppm | |
} | |
} | |
ls * | each {|file| do -i { | |
if ($file.name | str contains ".cbz") { unzip $file.name -d ./pages/ } ; | |
if ($file.name | str contains ".cbr") { unrar e -op./pages/ $file.name … | |
if ($file.name | str contains ".pdf") { mkdir pages ; pdfimages $file.n… | |
cd pages ; conv ; cd ../ ; ^zip -r $"($file.name).webp.cbz" pages ; rm … | |
} } | |
``` | |
### Parse gnu tar output | |
``` | |
〉tar vtf nushell.tgz | parse -r "(.*?) (.*?)\/(.*?)\\s+(.*?) (.*?) (.*?) (.*… | |
╭───┬────────────┬────────�… | |
│ # │ mode │ owner │ group │ size │ date │ time �… | |
├───┼────────────┼────────�… | |
│ 0 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 16:45 �… | |
│ 1 │ -rw-r--r-- │ solene │ wheel │ 519 │ 2022-10-30 │ 13:41 �… | |
│ 2 │ -rw-r--r-- │ solene │ wheel │ 29304 │ 2022-10-29 │ 18:49 �… | |
│ 3 │ -rw-r--r-- │ solene │ wheel │ 75003 │ 2022-10-29 │ 13:16 �… | |
│ 4 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 00:00 �… | |
│ 5 │ -rw-r--r-- │ solene │ wheel │ 337 │ 2022-10-29 │ 18:52 �… | |
│ 6 │ -rw-r--r-- │ solene │ wheel │ 14 │ 2022-10-29 │ 18:53 �… | |
╰───┴────────────┴────────�… | |
``` | |
### Opening spreadsheets | |
``` | |
〉open --raw freq.ods | from ods | get Sheet1 | headers | |
╭───┬─────────────┬───────�… | |
│ # │ Policy │ Compile time │ Idle time │ column3 │ Compile po… | |
├───┼─────────────┼───────�… | |
│ 0 │ powersaving │ 1123.00 │ 0.00 │ │ 5… | |
│ 1 │ auto │ 871.00 │ 252.00 │ │ 5… | |
╰───┴─────────────┴───────�… | |
``` | |
We can format new strings from columns values. | |
``` | |
〉open --raw freq.ods | from ods | get Sheet1 | headers | each {|row| do { ech… | |
╭───┬─────────────────────�… | |
│ 0 │ powersaving = 5.9 Watts │ | |
│ 1 │ auto = 6.34 Watts │ | |
╰───┴─────────────────────�… | |
``` | |
### Filter and sort a JSON | |
There is a website listing packages that can be updated on OpenBSD at | |
https://portroach.openbsd.org, it provides json of data for rendering. | |
We can use this data to sort which maintainer has the most up to date | |
percentage, but only if they manage more than 30 packages. | |
``` | |
fetch https://portroach.openbsd.org/json/totals.json | get results | where tota… | |
``` | |
## NixOS examples | |
### Query profiles packages | |
``` | |
nix profile list | parse "{index} {flake} {source} {store}" | |
╭───┬─────────────────────�… | |
│ # │ flake │ … | |
├───┼─────────────────────�… | |
│ 0 │ flake:nixpkgs#legacyPackages.x86_64-linux.libreoffice │ path:/nix/s… | |
│ │ │ narHash=sha… | |
│ │ │ b5e2a934894… | |
│ 1 │ flake:nixpkgs#legacyPackages.x86_64-linux.dino │ path:/nix/s… | |
│ │ │ narHash=sha… | |
│ │ │ 4afca5fd9f5… | |
╰───┴─────────────────────�… | |
``` | |
### Query flakes | |
``` | |
nix flake show --json | from json | |
╭────────────────┬────────�… | |
│ defaultPackage │ {record 5 fields} │ | |
│ packages │ {record 5 fields} │ | |
╰────────────────┴────────�… | |
nix flake show --json | from json | get packages | |
╭────────────────┬────────�… | |
│ aarch64-darwin │ {record 2 fields} │ | |
│ aarch64-linux │ {record 2 fields} │ | |
│ i686-linux │ {record 2 fields} │ | |
│ x86_64-darwin │ {record 2 fields} │ | |
│ x86_64-linux │ {record 2 fields} │ | |
╰────────────────┴────────�… | |
nix flake show --json | from json | get packages.x86_64-linux | |
╭───────────────┬─────────�… | |
│ nix-dev-html │ {record 2 fields} │ | |
│ nix-dev-pyenv │ {record 3 fields} │ | |
╰───────────────┴─────────�… | |
``` | |
### Parse a flake.lock file | |
``` | |
> open flake.lock | from json | get nodes.nixpkgs.locked | |
╭──────────────┬──────────�… | |
│ lastModified │ 1663494472 │ | |
│ narHash │ sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb/tFi/qM= │ | |
│ path │ /nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source │ | |
│ rev │ f677051b8dc0b5e2a9348941c99eea8c4b0ff28f │ | |
│ type │ path │ | |
╰──────────────┴──────────�… | |
``` | |
## OpenBSD examples | |
### Parse /etc/fstab | |
``` | |
> open /etc/fstab | from ssv -m 1 -n | rename device mountpoint fs options freq… | |
_────┬────────────────────┬… | |
│ # │ device │ mountpoint │ fs │ … | |
├────┼────────────────────�… | |
│ 0 │ 55a6c21017f858cb.b │ none │ swap │ sw … | |
│ 1 │ 55a6c21017f858cb.a │ / │ ffs │ rw,noatime,softd… | |
│ 2 │ 55a6c21017f858cb.l │ /home │ ffs │ rw,noatime,wxall… | |
│ 3 │ 55a6c21017f858cb.d │ /tmp │ ffs │ rw,noatime,softd… | |
│ 4 │ 55a6c21017f858cb.f │ /usr │ ffs │ rw,noatime,softd… | |
│ 5 │ 55a6c21017f858cb.g │ /usr/X11R6 │ ffs │ rw,noatime,softd… | |
│ 6 │ 55a6c21017f858cb.h │ /usr/local │ ffs │ rw,noatime,softd… | |
│ 7 │ 55a6c21017f858cb.k │ /usr/obj │ ffs │ rw,noatime,softd… | |
│ 8 │ 55a6c21017f858cb.j │ /usr/src │ ffs │ rw,noatime,softd… | |
│ 9 │ 55a6c21017f858cb.e │ /var │ ffs │ rw,noatime,softd… | |
│ 10 │ afebb2a83a449265.b │ /build │ ffs │ rw,noatime,softd… | |
│ 11 │ afebb2a83a449265.a │ /build/pobj │ ffs │ rw,noatime,softd… | |
│ 12 │ 55a6c21017f858cb.b │ /build/pobj_mfs │ mfs │ -s1G,wxallowed,n… | |
╰────┴────────────────────�… | |
``` | |
### Parse /var/log/messages | |
``` | |
open /var/log/messages | parse -r "(?<date>\\w+ \\d+ \\d+:\\d+:\\d+) (?<hostnam… | |
╭───┬─────────────────┬───�… | |
│ # │ date │ hostname │ program │ pid │ … | |
├───┼─────────────────┼───�… | |
│ 0 │ Oct 31 10:27:32 │ fx6 │ collectd │ 55258 │ uc_update: … | |
│ 1 │ Oct 31 10:43:02 │ fx6 │ collectd │ 55258 │ uc_update: … | |
│ 2 │ Oct 31 11:00:01 │ fx6 │ syslogd │ 4629 │ restart … | |
│ 3 │ Oct 31 11:05:26 │ fx6 │ pkg_delete │ │ Removed hel… | |
│ 4 │ Oct 31 11:05:29 │ fx6 │ pkg_add │ │ Added helix… | |
│ 5 │ Oct 31 11:16:49 │ fx6 │ pkg_add │ │ Added llvm-… | |
│ 6 │ Oct 31 11:20:18 │ fx6 │ pkg_add │ │ Added clang… | |
│ 7 │ Oct 31 11:20:32 │ fx6 │ pkg_add │ │ Added bash-… | |
│ 8 │ Oct 31 11:20:34 │ fx6 │ pkg_add │ │ Added fzf-0… | |
│ 9 │ Oct 31 11:21:01 │ fx6 │ pkg_delete │ │ Removed fzf… | |
╰───┴─────────────────┴───�… | |
``` | |
### Parse pkg_info output | |
``` | |
pkg_info | str trim | parse -r "(?<package>.*?)-(?<version>[a-zA-Z0-9\\.]*?) (… | |
╭────┬───────────────────┬�… | |
│ # │ package │ version │ description … | |
├────┼───────────────────┼�… | |
│ 0 │ athn-firmware │ 1.1p4 │ firmware binary images for athn… | |
│ 1 │ collectd │ 5.12.0 │ system metrics collection engin… | |
│ 2 │ curl │ 7.85.0 │ transfer files with FTP, HTTP, … | |
│ 3 │ gettext-runtime │ 0.21p1 │ GNU gettext runtime libraries a… | |
│ 4 │ intel-firmware │ 20220809v0 │ microcode update binaries for I… | |
│ 5 │ inteldrm-firmware │ 20220913 │ firmware binary images for inte… | |
│ 6 │ kakoune │ 2021.11.08 │ modal code editor with a focus … | |
│ 7 │ libgcrypt │ 1.10.1p0 │ crypto library based on code us… | |
│ 8 │ libgpg-error │ 1.46 │ error codes for GnuPG related s… | |
│ 9 │ libiconv │ 1.17 │ character set conversion librar… | |
│ 10 │ libstatgrab │ 0.91p5 │ system statistics gathering lib… | |
│ 11 │ libxml │ 2.10.3 │ XML parsing library … | |
│ 12 │ libyajl │ 2.1.0 │ small JSON library written in A… | |
│ 13 │ nghttp2 │ 1.50.0 │ library for HTTP/2 … | |
│ 14 │ nushell │ 0.70.0 │ a new kind of shell … | |
│ 15 │ obsdfreqd │ 1.0.3 │ userland daemon to manage CPU f… | |
│ 16 │ quirks │ 6.42 │ exceptions to pkg_add rules and… | |
│ 17 │ rsync │ 3.2.5pl0 │ mirroring/synchronization over … | |
│ 18 │ ttyplot │ 1.4p0 │ realtime plotting utility for t… | |
│ 19 │ vmm-firmware │ 1.14.0p0 │ firmware binary images for vmm(… | |
│ 20 │ xz │ 5.2.7 │ LZMA compression and decompress… | |
│ 21 │ yash │ 2.52 │ POSIX-compliant command line sh… | |
╰────┴───────────────────┴�… | |
``` | |
# Conclusion | |
Nushell is very fun, it's terribly different from regular shells, but | |
it comes with a powerful language and tooling. I always liked shells | |
because of pipes commands, allowing to construct a complex | |
transformation/analysis step by step, and easily inspect any step, or | |
be able to replace a step by another. | |
With nushell, it feels like I finally have a better tool to create more | |
reliable, robust, portable and faster command pipelines. The learning | |
curve didn't feel too hard, but maybe it's because I'm already used to | |
functional programming. |