 |
Title: Nushell: Introduction to a new kind of shell |
|
Author: Solène |
|
Date: 31 October 2022 |
|
Tags: openbsd nixos nushell shell |
|
Description: This article covers a gentle introduction to the shell |
|
nushell |
|
|
|
# What is nushell |
|
|
|
Let me introduce you to a nice project I found while lurking on the |
|
Internet. It's called nushell and is a non-POSIX shell, so most of |
|
your regular shells knowledge (zsh, bash, ksh, etc…) can't be applied |
|
on it, and using it feels like doing functional programming. |
|
It's a good tool for creating robust data manipulation pipelines, you |
|
can think of it like a mix of a shell which would include awk's power, |
|
behave like a SQL database, and which knows how to import/export |
|
XML/JSON/YAML/TOML natively. |
|
|
|
You may want to try nushell only as a tool, and not as your main shell, |
|
it's perfectly fine. |
|
|
|
With a regular shell, iterating over a command output can be complex |
|
when it involves spaces or newlines, for instance, that's why `find` |
|
and `xargs` have a `-print0` parameter to have a special delimited |
|
between "items", but it doesn't compose well with other tools. Nushell |
|
handles correctly this situation as its manipulates the data using |
|
indexed entries, given you correctly parsed the input at the beginning. |
|
|
 |
Nushell official project page |
|
Nushell documentation website |
|
|
|
# How to get it |
|
|
|
Nushell is a rust program, so it should work on every platform where |
|
Rust/Cargo are supported. I packaged it for OpenBSD, so it's available |
|
on -current (and will be in releases after 7.3 is out), the port could |
|
be used on 7.2 with no effort. |
|
With Nix, it's packaged under the name `nushell`, the binary name is |
|
`nu`. |
|
|
|
For other platforms, it's certainly already packaged, otherwise you can |
|
find installation instructions to build it from sources. |
|
|
|
Nushell documentation: Building nushell from sources |
|
|
|
# Configuration |
|
|
|
At first run, you are prompted to use default configuration files, I'd |
|
recommend accepting, you will have files created in |
|
`~/.config/nushell/`. |
|
|
|
The only change I made from now is to make Tab completion |
|
case-sensitive, so `D[TAB]` completes to `Downloads` instead of asking |
|
between `dev` and `Downloads`. Look for `case_sensitive_completions` in |
|
`.config/nushell/config.nu` and set it to `true`. |
|
|
|
# Examples |
|
|
|
If you are like me, and you prefer learning by doing instead of reading |
|
a lot of documentation, I prepared a bunch of real world use case you |
|
can experiment with. The documentation is still required to learn the |
|
many commands and syntax, but examples are a nice introduction. |
|
|
|
## Getting help |
|
|
|
Help from nushell can be parsed directly with nu commands, it's |
|
important to understand where to find information about commands. |
|
|
|
Use `help a-command` to learn from a single command: |
|
|
|
```script |
|
> help help |
|
Display help information about commands. |
|
|
|
Usage: |
|
> help {flags} ...(rest) |
|
|
|
Flags: |
|
-h, --help - Display this help message |
|
-f, --find <String> - string to find in command names, usage, and search terms |
|
|
|
[cut so it's not too long] |
|
``` |
|
|
|
Use `help commands` to list all available commands (I'm limiting to 5 |
|
between there are a lot of commands) |
|
|
|
```script |
|
help commands | last 5 |
|
╭───┬─────────────┬───────�… |
|
│ # │ name │ category │ is_plugin │ is_custom �… |
|
├───┼─────────────┼───────�… |
|
│ 0 │ window │ filters │ false │ false �… |
|
│ 1 │ with-column │ dataframe or lazyframe │ false │ false �… |
|
│ 2 │ with-env │ env │ false │ false �… |
|
│ 3 │ wrap │ filters │ false │ false �… |
|
│ 4 │ zip │ filters │ false │ false �… |
|
╰───┴─────────────┴───────�… |
|
``` |
|
|
|
Add `sort-by category` to list them... sorted by category. |
|
|
|
``` |
|
help commands | sort-by category |
|
``` |
|
|
|
Use `where category == filters` to only list commands from the |
|
`filters` category. |
|
|
|
``` |
|
help commands | where category == filters |
|
``` |
|
|
|
Use `find foobar` to return lines containing `foobar`. |
|
|
|
``` |
|
help commands | find insert |
|
``` |
|
|
|
## General examples |
|
|
|
### Converting a data structure into another |
|
|
|
This is just an example from YAML to JSON, but you can convert much |
|
more formats into other formats. |
|
|
|
``` |
|
open dev/home-impermanence/tests/impermanence.yml | to json |
|
{ |
|
"directories": |
|
[ |
|
"Documents", |
|
"Downloads", |
|
"Datastore/Music", |
|
"Datastore", |
|
"Datastore/", |
|
"Datastore/Music/Band1", |
|
".config", |
|
"foo/bar", |
|
"foo/bar/hello" |
|
], |
|
"size": "500m", |
|
"files": |
|
[ |
|
".Xdefaults", |
|
".profile", |
|
".xsession", |
|
] |
|
} |
|
``` |
|
|
|
### Parsing sysctl output |
|
|
|
``` |
|
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" |
|
``` |
|
|
|
Because the output would be too long, here is how you get 10 random |
|
keys from sysctl. |
|
|
|
``` |
|
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" | shuffle | last 10 | sort-by k… |
|
╭───┬─────────────────────�… |
|
│ # │ key │ value │ |
|
├───┼─────────────────────�… |
|
│ 0 │ fs.quota.reads │ 0 │ |
|
│ 1 │ net.core.high_order_alloc_disable │ 0 │ |
|
│ 2 │ net.ipv4.conf.all.drop_gratuitous_arp │ 0 │ |
|
│ 3 │ net.ipv4.conf.default.rp_filter │ 2 │ |
|
│ 4 │ net.ipv4.conf.lo.disable_xfrm │ 1 │ |
|
│ 5 │ net.ipv4.conf.lo.forwarding │ 0 │ |
|
│ 6 │ net.ipv4.ipfrag_low_thresh │ 3145728 │ |
|
│ 7 │ net.ipv6.conf.all.ioam6_id │ 65535 │ |
|
│ 8 │ net.ipv6.conf.all.router_solicitation_interval │ 4 │ |
|
│ 9 │ net.mptcp.enabled │ 1 │ |
|
╰───┴─────────────────────�… |
|
``` |
|
|
|
### Recursively convert FLAC files to OPUS |
|
|
|
A complicated task using a regular shell, recursively find files |
|
matching a pattern and then run a given command on each of them, in |
|
parallel. Which is exactly what you need if you want to convert your |
|
music library into another format, let's convert everything from FLAC |
|
to OPUS in this example. |
|
|
|
In the following command line, we will look for every `.flac` file in |
|
the subdirectories, then run in parallel using `par-each` the command |
|
`ffmpeg` on it, from its current name to the old name with `.flac` |
|
changed to `.opus`. |
|
The `let convert` and `| complete` commands are used to store the |
|
output of each command into a result table, and store it in the |
|
variable `convert` so we can query it after the job is done. |
|
|
|
``` |
|
let convert = (ls **/*flac | par-each { |file| do -i { ffmpeg -i $file.name ($f… |
|
``` |
|
|
|
Now, we have a structure in `convert` that contains the columns |
|
`stdout`, `stderr` and `exit_code`, so we can look if all the commands |
|
did run correctly using the following query. |
|
|
|
``` |
|
$convert | where exit_code != 0 |
|
``` |
|
|
|
### Synchronize a music library to a compressed one |
|
|
|
I had a special need for my phone and my huge music library, I wanted |
|
to have a lower quality version of it synced with syncthing, but I |
|
needed this to be easy to update when adding new files. |
|
|
|
It takes all the music files in `/home/user/Music/` and creates a 64K |
|
opus file in `/home/user/Stream/` by keeping the same file tree |
|
hierarchy, and if the opus destination file exists it's skipped. |
|
|
|
```nushell |
|
cd /home/user/Music/ |
|
let dest = "/home/user/Stream/" |
|
let convert = (ls **/* | |
|
where name =~ ".(mp3|flac|opus|ogg)$" | |
|
where name !~ "(Audiobook|Piano)" | |
|
par-each { |
|
|file| do -i { |
|
let new_name = ($file.name | str replac… |
|
if (not ([$dest, $new_name] | str join … |
|
mkdir ([$dest, ($file.name | pa… |
|
ffmpeg -i $file.name -b:a 64K (… |
|
} | complete |
|
} |
|
}) |
|
$convert |
|
``` |
|
|
|
### Convert PDF/CBR/CBZ pages into webp and CBZ archives |
|
|
|
I have a lot of digitalized books/mangas/comics, this conversion is a |
|
handy operation reducing the size of the files by 40% (up to 70%). |
|
|
|
``` |
|
def conv [] { |
|
if (ls | first | get name | str contains ".jpg") { |
|
ls *jpg | par-each {|file| do -i { cwebp $file.name -o ($file.name | … |
|
rm *jpg |
|
} |
|
if (ls | first | get name | str contains ".ppm") { |
|
ls *ppm | par-each {|file| do -i { cwebp $file.name -o ($file.name | … |
|
rm *ppm |
|
} |
|
} |
|
ls * | each {|file| do -i { |
|
if ($file.name | str contains ".cbz") { unzip $file.name -d ./pages/ } ; |
|
if ($file.name | str contains ".cbr") { unrar e -op./pages/ $file.name … |
|
if ($file.name | str contains ".pdf") { mkdir pages ; pdfimages $file.n… |
|
cd pages ; conv ; cd ../ ; ^zip -r $"($file.name).webp.cbz" pages ; rm … |
|
} } |
|
``` |
|
|
|
### Parse gnu tar output |
|
|
|
``` |
|
〉tar vtf nushell.tgz | parse -r "(.*?) (.*?)\/(.*?)\\s+(.*?) (.*?) (.*?) (.*… |
|
╭───┬────────────┬────────�… |
|
│ # │ mode │ owner │ group │ size │ date │ time �… |
|
├───┼────────────┼────────�… |
|
│ 0 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 16:45 �… |
|
│ 1 │ -rw-r--r-- │ solene │ wheel │ 519 │ 2022-10-30 │ 13:41 �… |
|
│ 2 │ -rw-r--r-- │ solene │ wheel │ 29304 │ 2022-10-29 │ 18:49 �… |
|
│ 3 │ -rw-r--r-- │ solene │ wheel │ 75003 │ 2022-10-29 │ 13:16 �… |
|
│ 4 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 00:00 �… |
|
│ 5 │ -rw-r--r-- │ solene │ wheel │ 337 │ 2022-10-29 │ 18:52 �… |
|
│ 6 │ -rw-r--r-- │ solene │ wheel │ 14 │ 2022-10-29 │ 18:53 �… |
|
╰───┴────────────┴────────�… |
|
``` |
|
|
|
### Opening spreadsheets |
|
|
|
``` |
|
〉open --raw freq.ods | from ods | get Sheet1 | headers |
|
╭───┬─────────────┬───────�… |
|
│ # │ Policy │ Compile time │ Idle time │ column3 │ Compile po… |
|
├───┼─────────────┼───────�… |
|
│ 0 │ powersaving │ 1123.00 │ 0.00 │ │ 5… |
|
│ 1 │ auto │ 871.00 │ 252.00 │ │ 5… |
|
╰───┴─────────────┴───────�… |
|
``` |
|
|
|
We can format new strings from columns values. |
|
|
|
``` |
|
〉open --raw freq.ods | from ods | get Sheet1 | headers | each {|row| do { ech… |
|
╭───┬─────────────────────�… |
|
│ 0 │ powersaving = 5.9 Watts │ |
|
│ 1 │ auto = 6.34 Watts │ |
|
╰───┴─────────────────────�… |
|
``` |
|
|
|
### Filter and sort a JSON |
|
|
|
There is a website listing packages that can be updated on OpenBSD at |
|
https://portroach.openbsd.org, it provides json of data for rendering. |
|
|
|
We can use this data to sort which maintainer has the most up to date |
|
percentage, but only if they manage more than 30 packages. |
|
|
|
``` |
|
fetch https://portroach.openbsd.org/json/totals.json | get results | where tota… |
|
``` |
|
|
|
## NixOS examples |
|
|
|
### Query profiles packages |
|
|
|
``` |
|
nix profile list | parse "{index} {flake} {source} {store}" |
|
╭───┬─────────────────────�… |
|
│ # │ flake │ … |
|
├───┼─────────────────────�… |
|
│ 0 │ flake:nixpkgs#legacyPackages.x86_64-linux.libreoffice │ path:/nix/s… |
|
│ │ │ narHash=sha… |
|
│ │ │ b5e2a934894… |
|
│ 1 │ flake:nixpkgs#legacyPackages.x86_64-linux.dino │ path:/nix/s… |
|
│ │ │ narHash=sha… |
|
│ │ │ 4afca5fd9f5… |
|
╰───┴─────────────────────�… |
|
``` |
|
|
|
### Query flakes |
|
|
|
``` |
|
nix flake show --json | from json |
|
╭────────────────┬────────�… |
|
│ defaultPackage │ {record 5 fields} │ |
|
│ packages │ {record 5 fields} │ |
|
╰────────────────┴────────�… |
|
|
|
nix flake show --json | from json | get packages |
|
╭────────────────┬────────�… |
|
│ aarch64-darwin │ {record 2 fields} │ |
|
│ aarch64-linux │ {record 2 fields} │ |
|
│ i686-linux │ {record 2 fields} │ |
|
│ x86_64-darwin │ {record 2 fields} │ |
|
│ x86_64-linux │ {record 2 fields} │ |
|
╰────────────────┴────────�… |
|
|
|
nix flake show --json | from json | get packages.x86_64-linux |
|
╭───────────────┬─────────�… |
|
│ nix-dev-html │ {record 2 fields} │ |
|
│ nix-dev-pyenv │ {record 3 fields} │ |
|
╰───────────────┴─────────�… |
|
``` |
|
|
|
### Parse a flake.lock file |
|
|
|
``` |
|
> open flake.lock | from json | get nodes.nixpkgs.locked |
|
╭──────────────┬──────────�… |
|
│ lastModified │ 1663494472 │ |
|
│ narHash │ sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb/tFi/qM= │ |
|
│ path │ /nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source │ |
|
│ rev │ f677051b8dc0b5e2a9348941c99eea8c4b0ff28f │ |
|
│ type │ path │ |
|
╰──────────────┴──────────�… |
|
``` |
|
|
|
## OpenBSD examples |
|
|
|
### Parse /etc/fstab |
|
|
|
``` |
|
> open /etc/fstab | from ssv -m 1 -n | rename device mountpoint fs options freq… |
|
_────┬────────────────────┬… |
|
│ # │ device │ mountpoint │ fs │ … |
|
├────┼────────────────────�… |
|
│ 0 │ 55a6c21017f858cb.b │ none │ swap │ sw … |
|
│ 1 │ 55a6c21017f858cb.a │ / │ ffs │ rw,noatime,softd… |
|
│ 2 │ 55a6c21017f858cb.l │ /home │ ffs │ rw,noatime,wxall… |
|
│ 3 │ 55a6c21017f858cb.d │ /tmp │ ffs │ rw,noatime,softd… |
|
│ 4 │ 55a6c21017f858cb.f │ /usr │ ffs │ rw,noatime,softd… |
|
│ 5 │ 55a6c21017f858cb.g │ /usr/X11R6 │ ffs │ rw,noatime,softd… |
|
│ 6 │ 55a6c21017f858cb.h │ /usr/local │ ffs │ rw,noatime,softd… |
|
│ 7 │ 55a6c21017f858cb.k │ /usr/obj │ ffs │ rw,noatime,softd… |
|
│ 8 │ 55a6c21017f858cb.j │ /usr/src │ ffs │ rw,noatime,softd… |
|
│ 9 │ 55a6c21017f858cb.e │ /var │ ffs │ rw,noatime,softd… |
|
│ 10 │ afebb2a83a449265.b │ /build │ ffs │ rw,noatime,softd… |
|
│ 11 │ afebb2a83a449265.a │ /build/pobj │ ffs │ rw,noatime,softd… |
|
│ 12 │ 55a6c21017f858cb.b │ /build/pobj_mfs │ mfs │ -s1G,wxallowed,n… |
|
╰────┴────────────────────�… |
|
``` |
|
|
|
### Parse /var/log/messages |
|
|
|
``` |
|
open /var/log/messages | parse -r "(?<date>\\w+ \\d+ \\d+:\\d+:\\d+) (?<hostnam… |
|
╭───┬─────────────────┬───�… |
|
│ # │ date │ hostname │ program │ pid │ … |
|
├───┼─────────────────┼───�… |
|
│ 0 │ Oct 31 10:27:32 │ fx6 │ collectd │ 55258 │ uc_update: … |
|
│ 1 │ Oct 31 10:43:02 │ fx6 │ collectd │ 55258 │ uc_update: … |
|
│ 2 │ Oct 31 11:00:01 │ fx6 │ syslogd │ 4629 │ restart … |
|
│ 3 │ Oct 31 11:05:26 │ fx6 │ pkg_delete │ │ Removed hel… |
|
│ 4 │ Oct 31 11:05:29 │ fx6 │ pkg_add │ │ Added helix… |
|
│ 5 │ Oct 31 11:16:49 │ fx6 │ pkg_add │ │ Added llvm-… |
|
│ 6 │ Oct 31 11:20:18 │ fx6 │ pkg_add │ │ Added clang… |
|
│ 7 │ Oct 31 11:20:32 │ fx6 │ pkg_add │ │ Added bash-… |
|
│ 8 │ Oct 31 11:20:34 │ fx6 │ pkg_add │ │ Added fzf-0… |
|
│ 9 │ Oct 31 11:21:01 │ fx6 │ pkg_delete │ │ Removed fzf… |
|
╰───┴─────────────────┴───�… |
|
``` |
|
|
|
### Parse pkg_info output |
|
|
|
``` |
|
pkg_info | str trim | parse -r "(?<package>.*?)-(?<version>[a-zA-Z0-9\\.]*?) (… |
|
╭────┬───────────────────┬�… |
|
│ # │ package │ version │ description … |
|
├────┼───────────────────┼�… |
|
│ 0 │ athn-firmware │ 1.1p4 │ firmware binary images for athn… |
|
│ 1 │ collectd │ 5.12.0 │ system metrics collection engin… |
|
│ 2 │ curl │ 7.85.0 │ transfer files with FTP, HTTP, … |
|
│ 3 │ gettext-runtime │ 0.21p1 │ GNU gettext runtime libraries a… |
|
│ 4 │ intel-firmware │ 20220809v0 │ microcode update binaries for I… |
|
│ 5 │ inteldrm-firmware │ 20220913 │ firmware binary images for inte… |
|
│ 6 │ kakoune │ 2021.11.08 │ modal code editor with a focus … |
|
│ 7 │ libgcrypt │ 1.10.1p0 │ crypto library based on code us… |
|
│ 8 │ libgpg-error │ 1.46 │ error codes for GnuPG related s… |
|
│ 9 │ libiconv │ 1.17 │ character set conversion librar… |
|
│ 10 │ libstatgrab │ 0.91p5 │ system statistics gathering lib… |
|
│ 11 │ libxml │ 2.10.3 │ XML parsing library … |
|
│ 12 │ libyajl │ 2.1.0 │ small JSON library written in A… |
|
│ 13 │ nghttp2 │ 1.50.0 │ library for HTTP/2 … |
|
│ 14 │ nushell │ 0.70.0 │ a new kind of shell … |
|
│ 15 │ obsdfreqd │ 1.0.3 │ userland daemon to manage CPU f… |
|
│ 16 │ quirks │ 6.42 │ exceptions to pkg_add rules and… |
|
│ 17 │ rsync │ 3.2.5pl0 │ mirroring/synchronization over … |
|
│ 18 │ ttyplot │ 1.4p0 │ realtime plotting utility for t… |
|
│ 19 │ vmm-firmware │ 1.14.0p0 │ firmware binary images for vmm(… |
|
│ 20 │ xz │ 5.2.7 │ LZMA compression and decompress… |
|
│ 21 │ yash │ 2.52 │ POSIX-compliant command line sh… |
|
╰────┴───────────────────┴�… |
|
``` |
|
|
|
# Conclusion |
|
|
|
Nushell is very fun, it's terribly different from regular shells, but |
|
it comes with a powerful language and tooling. I always liked shells |
|
because of pipes commands, allowing to construct a complex |
|
transformation/analysis step by step, and easily inspect any step, or |
|
be able to replace a step by another. |
|
With nushell, it feels like I finally have a better tool to create more |
|
reliable, robust, portable and faster command pipelines. The learning |
|
curve didn't feel too hard, but maybe it's because I'm already used to |
|
functional programming. |
